Data Engineer with Azure

Introduction & Summary: The objective of this engagement is to provide Data Engineers to execute the technical implementation of the analytical platform. This role focuses on covering key aspects such as data ingestion, transformation, and orchestration processes that align with the designed architecture and business requirements. The ideal candidate will have a minimum of 3 years of experience in data inflow and transformation processes in a cloud environment, with practical exposure to advanced technologies.

Main Responsibilities:

• Ingesting data to the bronze layer

• Data anonymization

• Creating orchestration flow

• Data platform initial setup

• Creating and testing ETL/ELT processes in the data platform

• Generating data transformations between the bronze and silver layers (including data validation, duplicate removal, and merging data from different sources)

• Implementing monitoring and alerting for data pipelines

• Optimizing data storage and retrieval for cost and performance

• Documenting all processes and configurations for maintainability

• Recommending necessary changes to the production system to ensure high-quality data for analytics.

Key Requirements:

• Minimum 3 years of experience in creating data inflow and transformation processes (pipelines) in a cloud environment

• Practical experience with technologies:

  • Azure Stack (ADLS, Azure Blob Storage)

  • Databricks, Delta Lakehouse, DBT, GIT

• Knowledge of orchestration tools: Apache Airflow or similar solutions

• Knowledge of PySpark

• Very good knowledge of ETL/ELT processes

• Advanced knowledge of SQL, especially in query optimization and performance

• B2 English level

Nice to Have:

• Experience with big data technologies

• Familiarity with data security practices

• Understanding of data governance principles