Data Engineer

We are looking for candidates with few years of experience (regular level) as well with more experience (senior level).

 Responsibilities:

  •  Design & Develop: Collaborate with the team to build scalable data platforms on Azure Cloud, leveraging Azure Data Factory, Databricks, ADLS, and other Azure services.

  •  Data Management: Integrate diverse data sources, ensuring data quality, consistency, and governance. Implement security measures using Azure Purview, Key Vault, and Databricks Unity Catalog for access control and auditing.

  •  Collaboration: Work closely with data scientists and analysts to deliver efficient data solutions. Support and maintain the data platform, leveraging Azure Monitor and Log Analytics for

troubleshooting.

  •  Code Management & Optimization: Develop and maintain production-grade data pipelines using Spark & PySpark, ensuring version control in Git and following best practices for CI/CD.

  •  Continuous Improvement: Stay up to date with Azure and Databricks technologies to optimize workflows and automate processes for efficiency.

Qualifications:

  •  Education: Bachelor’s in Computer Science or a related field (Master’s is a plus).

  •  Experience: Proven experience working with Azure Cloud data platforms, including Data Factory, Databricks, ADLS, SQL Databases, and Spark/PySpark.

  •  Technical Skills:

  •  Strong experience in Spark & PySpark for big data processing.

  •  Proficiency in Python, Scala, or Java for data engineering tasks.

  •  Knowledge of SQL/NoSQL databases, ETL, and data warehousing principles.

  •  Strong understanding of Git for version control and CI/CD processes in a data engineering environment.

  •  Soft Skills: Strong communication, problem-solving skills, and the ability to collaborate effectively across teams.

Additional Technologies in the Project:

  •  The project will also involve Databricks Asset Bundles (DAB) and Delta Live Tables (DLT) for managing and deploying data pipelines.