Data Engineer – Databricks (Performance & Security)

About the Role

We are looking for a Data Engineer focused on Databricks platform excellence, mastering features, security, performance optimization, and operational best practices. You will play a key role in improving how we leverage Databricks across our data platform, ensuring scalable, secure, and efficient workloads. This role suits someone passionate about infrastructure, governance, and performance — and who’s eager to expand knowledge into streaming technologies.

Key Responsibilities

• Optimize Databricks performance across jobs, clusters, and data workloads.

• Implement and maintain governance, security, and permissions via Unity Catalog and Terraform.

• Contribute to Databricks feature enablement — Delta Live Tables, Databricks Asset Bundles (DABs), and cost efficiency improvements.

• Monitor cluster and job performance using Grafana and internal observability tools.

• Support data engineers in troubleshooting and fine-tuning jobs and SQL queries.

• Explore and progressively implement streaming features (Structured Streaming, Autoloader, Kafka integration).

• Manage Databricks environments integrated with Airflow on AKS, Azure ADLS Gen2, and Terraform.

• Collaborate with DataOps to ensure high availability, logging, and monitoring coverage.

• Implement performance and security benchmarks for production workloads.

Key Requirements

• Deep knowledge of Databricks configuration, optimization, and administration.

• Experience with PySpark, SQL, and data performance tuning.

• Familiarity with Unity Catalog, DABs, and Terraform integration.

• Strong understanding of data governance, cost optimization, and cluster policies.

• Basic to intermediate experience with Kafka or interest in learning streaming processing.

Nice to Have

• Experience with modern data orchestration tools.

• Knowledge of additional data lakes or storage solutions.

• Understanding of cloud architecture and services.

• Familiarity with version control systems, particularly Git.

Other Details

This role provides flexibility with remote working options, offering a chance to engage in cutting-edge technologies while ensuring data excellence. Candidates should be prepared for ongoing learning and adaptation to new tools and methodologies, particularly within the Databricks ecosystem.