Strong Junior/ Middle Data Engineer with Python

Our client is one of the largest betting communities, having pioneered the betting exchange model back in 2000. Powered by cutting-edge technology, they operate the world’s leading online betting exchange.

Position overview

We are looking for a Data Engineer to support the migration and modernization of our existing SQL Server–based data workloads to a cloud-native Lakehouse platform built on AWS and Databricks. In this role, you will design and operate scalable, resilient, high-quality data pipelines and services that empower analytics, real-time streaming, and machine learning use cases across the organization.

Responsibilities

• Migrate legacy SQL Server workloads to a modern Lakehouse architecture on AWS and Databricks.

• Design, build, and maintain data pipelines for batch and real-time processing.

• Ensure data quality, reliability, and scalability across all pipelines and services.

• Collaborate with data scientists, analysts, and business stakeholders to deliver data solutions for analytics and ML use cases.

• Implement best practices for data governance, security, and compliance.

• Optimize performance and cost efficiency in a cloud-native environment.

  
  

Requirements

• Strong proficiency in Python for data engineering tasks.

• Hands-on experience with AWS services (e.g., S3, Glue, Lambda, EMR).

• Expertise in Databricks and Spark for big data processing.

• Solid understanding of SQL and relational database concepts.

• Experience with ETL/ELT frameworks and workflow orchestration tools (e.g., Airflow).

• Knowledge of data modeling, data warehousing, and Lakehouse principles.

  
  

Nice to have

• Familiarity with streaming technologies (Kafka, Kinesis).

• Experience with CI/CD pipelines for data solutions.

• Understanding of data security and compliance in cloud environments.

• Exposure to machine learning workflows and MLOps concepts.