Data Engineering Architect (Data Bricks)

We are seeking a highly skilled Data Engineering Architect with extensive experience in Databricks and related technologies to join a prominent organization in the automotive industry. The ideal candidate will have 10 years of experience in data engineering, with a strong focus on Databricks and Lakehouse architecture.

Key Responsibilities:

• Design and implement Databricks Lakehouse architecture, ensuring scalability and reliability.
• Ingest and transform both batch and streaming data on the Databricks Lakehouse Platform.
• Utilize Databricks Workflows to monitor dataflow graphs, track health, and analyze performance metrics of production jobs and Delta Live Tables pipelines.
• Build and manage Delta Lake tables to monitor and visualize performance, data quality, and reliability.
• Orchestrate and manage diverse workloads across the full lifecycle, including Delta Live Tables, PySpark, and other tools.
• Lead data modeling efforts to optimize data structures and ensure efficient processing.
• Experience with cluster sizing and performance optimization.
• Implement robust security protocols within Databricks.
• Collaborate with cross-functional teams in an Agile environment to understand business requirements and optimize data workflows.

Required Skills:

• Must-have skills: Databricks, Delta Lake, PySpark, Lakeflow, Delta Live Tables, Delta Sharing, Unity Catalog, and security.
• Nice-to-have skills: Azure Data Factory or Fivetran, PowerBI.
• Strong expertise in data engineering, Databricks as a Lakehouse solution, and data transformation techniques.
• Excellent communication and customer handling skills.
• Experience in an Agile environment and leading efforts to streamline data workflows.

Preferred Qualifications:

• Databricks Champions certification is preferred.

If you are an experienced Data Engineering Architect with a strong Databricks background and are ready to take on exciting challenges in the automotive industry, we would love to hear from you!