Senior Databricks Data Engineer

For one of our clients (financial company), we are looking for a specialist for the position of

Senior Databricks Data Engineer.

Form of cooperation: B2B contract.

Rate: 170-200 PLN/hour.

Work mode: 100% remotely.

Recruitment process: 3 stages (screening + internal interview with the client + interview in English with the end client).

The international project (US Client) is in the Databricks environment and may include both new implementation and upgrades of existing solutions.

Your responsibilities:

1. Unity Catalog - access management, configuration, lineage tracking).
2. design and optimisation of pipelines in PySpark and Delta Lake.
3. implementation of Spark tuning (AQE, broadcast joins, partition pruning).
4. integration with CI/CD tools (e.g. Azure DevOps).
5. design and implementation of a scalable lakehouse data architecture, with a focus on: real-time and batch data processing & automation and standardisation of ETL / ELT processes.
6. collaborate with business and technical teams, translating requirements into technical solutions.
7. maintain technical and architectural documentation.
8. support teams in data analysis and process optimisation.

Requirements:

1. minimum 5 years of experience working with data.
2. very good knowledge of Azure Databricks.
3. working knowledge of PySpark, Delta Lake and Lakehouse architecture.
4. experience with configuration and management of Unity Catalog.
5. familiarity with Spark optimization issues.
6. experience with ETL / ELT processes.
7. experience with CI/CD tools (Azure DevOps, Git).
8. independence, high level of seniority.
9. high involvement in design, technical supervision and implementation.
10. Ability to lead topics independently and collaborate with technical and business teams.

Nice to have:

Certifications:

1. Microsoft Certified: Azure Data Engineer Associate.
2. Databricks Certified (e.g. Data Engineer, ML Professional).
3. Familiarity with Microsoft Fabric.

Nice to have:

1. experience with GCP or other clouds.
2. knowledge of Microsoft Fabric.
3. experience in international projects (e.g. USA).
4. Ability to conduct technical discussions with the customer.

Are you interested? Join us!