Senior Data Engineer (Databricks)

At Grape Up, we transform businesses by unlocking the potential of AI and data through innovative software solutions.

We partner with industry leaders, from the automotive and finance industry, to build sophisticated Data & Analytics platforms that transform how organizations manage and leverage their data assets. Our solutions provide comprehensive capabilities spanning data storage, management, advanced analytics, machine learning, and AI, enabling enterprises to accelerate innovation and make data-driven decisions.

Responsibilities

• Implement a scalable architecture capable of handling the high volume of simulation data

• Build a flexible data preprocessing pipelines that are extensible and that can be integrated into customer’s existing platform

• Define KPIs to measure the improved reusability and automation of the new pipelines and test their performance in an end-to-end setting with model training

• Develop and implement processes and best practices for data management and governance

• Optimize and enhance system setup and improve data structures following industry best practices

• Collaborate effectively with data engineering team members while partnering closely with analytics and data science teams to meet user needs

• Collaborate with business stakeholders and technical teams to understand data requirements, translate business needs into technical solutions

• Lead technical discussions and solution design sessions with clients or internal stakeholders, presenting complex data engineering concepts in accessible ways

Requirements:

• Master’s degree in computer science, Data Engineering, AI, or a related field

• 5+ years of professional experience in Data Engineering and Big Data, building production-grade data platforms and pipelines

• Proven experience designing, implementing, and operating production solutions on Databricks (Azure Databricks or Databricks on AWS)

• Strong experience with Apache Spark (PySpark), including performance optimization, and large-scale data processing

• Experience designing and operating data pipeline orchestration using Apache Airflow (or Dagster / Prefect)

• Expert-level proficiency in Python and SQL for data transformations and analytics workloads

• Hands-on experience with Delta Lake and Lakehouse architecture

• Experience designing and implementing data governance and data management frameworks (e.g., data quality, access control, Unity Catalog or equivalent)

• Experience processing high-volume Automotive or IoT data (e.g., telemetry, sensor, or event-based data)

• Experience in pre-sales and consulting activities 

• Strong problem-solving skills and ability to work independently

• Fluency in English, both written and spoken

Nice to have:

• PhD degree in computer science, Data Engineering, AI, or a related field (completed or in progress)

• Data Engineer (Databricks) certificate

• Experience with streaming tools like Kafka or Azure Event Hubs etc

• Experience with Infrastructure as Code (Terraform, CloudFormation)

• Experience with Kubernetes

• Experience with Machine learning and MLOps