Lead Data Engineer (Databricks)

At Grape Up, we transform businesses by unlocking the potential of AI and data through innovative software solutions.

We partner with industry leaders, from the automotive and finance industry, to build sophisticated Data & Analytics platforms that transform how organizations manage and leverage their data assets. Our solutions provide comprehensive capabilities spanning data storage, management, advanced analytics, machine learning, and AI, enabling enterprises to accelerate innovation and make data-driven decisions.

Responsibilities:

• Implement a scalable architecture capable of handling the high volume of simulation data

• Build a flexible data preprocessing pipelines that are extensible and that can be integrated into customer’s existing platform

• Define KPIs to measure the improved reusability and automation of the new pipelines and test their performance in an end-to-end setting with model training

• Develop and implement processes and best practices for data management and governance

• Optimize and enhance system setup and improve data structures following industry best practices

• Collaborate effectively with data engineering team members while partnering closely with analytics and data science teams to meet user needs

• Collaborate with business stakeholders and technical teams to understand data requirements, translate business needs into technical solutions

• Lead technical discussions and solution design sessions with clients or internal stakeholders, presenting complex data engineering concepts in accessible ways

Requirements:

• Master’s degree in computer science, Data Engineering, AI, or a related field

• 6+ years of professional experience in Data Engineering and Big Data, building production-grade data platforms and pipelines

• Proven experience with Databricks platform (Azure Databricks or Databricks on AWS)

• Strong experience with Apache Spark (PySpark), including performance optimization, and large-scale data processing

• Proficiency in Python and SQL for data transformations and analytics workloads

• Experience with data pipeline orchestration (Airflow, Dagster, Prefect or similar)

• Experience with data governance and quality frameworks

• Strong problem-solving skills and ability to work independently

• Fluency in English, both written and spoken

Nice to have:

• PhD degree in computer science, Data Engineering, AI, or a related field (completed or in progress)

• Advanced Databricks expertise (Unity Catalog, Delta Lake and Lakehouse architecture)

• Data Engineer (Databricks) certificate

• Experience processing high-volume Automotive or IoT data (e.g., telemetry, sensor, or event-based data)

• Experience in pre-sales and consulting activities

• Experience with streaming tools like Kafka or Azure Event Hubs, etc.

• Experience with Infrastructure as Code (Terraform, CloudFormation)

Benefits of joining Grape Up:

• Non-corporate work environment among experienced engineers

• Individual growth & development plan supported by cyclical feedback sessions

• Access to knowledge platforms (e.g. Pluralsight)

• Equipment of your choice

• In-house business and tech trainings, and meetups

• Financing of conferences, external trainings, and certifications

• Language lessons (English, German and Polish for foreigners)

• Weekly Lunch & Learn where we meet up in the office, lunch together, and share our knowledge

• Employee referral program

• Rewards for the success of the month and year awarded by our employees

• Integration activities & trips