Data Engineer

Job Description: Data Engineer, LDWH Migration Programme

Level: Senior / Mid-Senior

Main Responsibilities:

• Develop and maintain data pipelines using Azure Databricks.

• Implement ETL processes, ensuring data quality and integrity.

• Read and comprehend complex SQL stored procedures.

• Translate architecture direction into engineering delivery.

• Conduct data profiling and quality assessments.

• Define acceptance criteria for data transformation.

• Collaborate with the programme team to support overall migration efforts.

• Participate in CI/CD processes for efficient deployment.

Key Requirements:

• Senior hands-on experience with Azure Databricks (jobs, clusters, Unity Catalog, SQL warehouses).

• Expertise in PySpark / Python for transformation logic and ETL pipelines.

• Solid working knowledge of Delta Lake (schema evolution, MERGE, time travel).

• Advanced SQL skills (complex queries, window functions); ability to comprehend legacy ETL.

• Familiarity with Databricks Asset Bundles (DAB) and CI/CD practices (Azure DevOps, Git).

• Working knowledge of Azure Data Lake Storage Gen2.

• Comfortable understanding legacy ETL processes and source data profiling.

• Experience with data reconciliation (defining criteria, interpreting discrepancies).

• Ability to articulate requirements for junior engineers.

• Fundamental knowledge of Power BI / Tableau.

Nice to Have:

• Experience in platform engineering and configuration refactoring using YAML.

• Ability to maintain environment governance across DEV, QAS, and PRD.

• Skills in source feasibility analysis for new data sources.

• Capability in transformation development across various data zones using PySpark and Delta Lake.

• Familiar with implementing acceptance criteria and conducting quality checks.

Other Details:

This position is part of a complex and dynamic migration programme and is pivotal for ensuring the successful transition of data systems. Remote engagement options are available, and collaboration with an internal team is essential.