Data Engineer with Databricks

Responsibilities:

• Storage Cleanup
• Implement cost tracking at user-level
• Transition to serverless compute for optimized performance
• Move scheduled jobs from all-purpose clusters to dedicated job clusters
• Proactively monitor costs and performance with dashboards and alerts
• Ofload raw data to DWH landing zone
• Use pre-aggregated datasets to reduce duplicate compute
• Transition Databricks workloads to DBT framework
• Complete migration of scheduled jobs to dedicated job clusters
• Expand serverless compute deployment post-GA

Requirements:

• Minimum 3 years of experience in similar position
• Proficiency in Spark, with a focus on Scala and Python
• Strong experience with Azure, including Azure Data Factory (ADF) and Azure Storage
• Hands-on expertise with Azure Databricks, including IAM, Jobs, Workflows, SQL Warehouses, Delta Lake, and Serverless
• Advanced SQL skills for data transformation and optimization
• Experience with DBT (Data Build Tool) for data modeling and pipeline development
• English level: B2, C1

Nice to have:

• Experience with Kafka for real-time data streaming

Responsibilities:

• Storage Cleanup
• Implement cost tracking at user-level
• Transition to serverless compute for optimized performance
• Move scheduled jobs from all-purpose clusters to dedicated job clusters
• Proactively monitor costs and performance with dashboards and alerts
• Ofload raw data to DWH landing zone
• Use pre-aggregated datasets to reduce duplicate compute
• Transition Databricks workloads to DBT framework
• Complete migration of scheduled jobs to dedicated job clusters
• Expand serverless compute deployment post-GA

Requirements:

• Minimum 3 years of experience in similar position
• Proficiency in Spark, with a focus on Scala and Python
• Strong experience with Azure, including Azure Data Factory (ADF) and Azure Storage
• Hands-on expertise with Azure Databricks, including IAM, Jobs, Workflows, SQL Warehouses, Delta Lake, and Serverless
• Advanced SQL skills for data transformation and optimization
• Experience with DBT (Data Build Tool) for data modeling and pipeline development
• English level: B2, C1

Nice to have:

• Experience with Kafka for real-time data streaming

Offer:

• Private medical care
• Co-financing for the sports card
• Training & learning opportunities
• Constant support of dedicated consultant
• Employee referral program