Data Engineer Lead (GCP + DBT + Airflow)

Project Overview:

As a Senior Data Engineer/Tech Lead, you will be part of the Operations Data Domain team, focusing on the ingestion, cleansing, and delivery of high-quality data products utilizing Google Cloud Platform (GCP). Your role will involve designing and building scalable data pipelines, ensuring smooth integration, and optimizing the data ecosystem to support advanced analytics. The ideal candidate will bring innovative solutions, creativity, and a positive attitude to tackle complex data challenges.

Key Responsibilities:

• Design, develop, and maintain scalable and reliable big data systems, including data pipelines, warehouses, and lakes.
• Work closely with cross-functional teams such as data product managers, data scientists, analysts, and software engineers to translate business requirements into efficient data solutions.
• Architect and fine-tune systems for managing large-scale datasets, including storage, processing, and retrieval.
• Create and implement automated processes for data analysis, model development, validation, and deployment.
• Uphold data governance and security standards to ensure data integrity and compliance with regulations.
• Develop clean, efficient, and structured software for continuous product delivery.
• Analyze key insights and trends, using statistical techniques to simplify and communicate important business stories.
• Address and resolve performance issues, data quality concerns, and bottlenecks in the data infrastructure.
• Mentor junior engineers and foster a culture of continuous learning and technical excellence.
• Effectively communicate with both technical and non-technical stakeholders.
• Contribute to the development of internal best practices, frameworks, and reusable components to enhance team efficiency.

Required Technical Skills:

• A degree in computer science, engineering, mathematics, or a related field.
• Over 7 years of experience in developing and deploying batch and streaming data pipelines in production.
• Strong experience with various relational SQL and NoSQL databases.
• In-depth knowledge of cloud-native data services, ideally on Google Cloud Platform (BigQuery, Vertex AI, Pub/Sub, Cloud Functions, etc.).
• Expertise with leading data warehousing tools such as Snowflake, BigQuery, or RedShift.
• Hands-on experience with dbt (Data Build Tool) for data transformation.
• Experience with big data frameworks like Hadoop, Spark, and Kafka, with familiarity with Databricks being an advantage.
• Proficiency in programming languages such as Python, Java, C++, or Scala.
• Hands-on experience in the full data engineering lifecycle, including metadata-driven solutions, and building Data Lake/Lake House systems.
• Strong understanding of Apache Airflow for orchestration and workflow management.
• Familiarity with GitHub/Git Toolkit.
• Experience in providing operational support to stakeholders.
• Strong grasp of software engineering best practices, such as unit testing, test automation, CI, and code reviews.
• Experience with CI/CD pipelines using Jenkins and GitHub Actions.

Desirable Skills:

• Familiarity with data visualization tools like Tableau, PowerBI, or Looker.

English Proficiency: B2, C1, C2

Team Setup: Extended Teams

Sprawdź inne ciekawe oferty pracy na: https://antal.pl/dla-kandydata