Senior Data Engineer

Project information:

• Industry: Insurance and IT services

• Rate: up to 180 zł/h netto + VAT

• Location: Warsaw (first 2-3 months of office visits once a week, then occasionally)

• Project language: Polish, English

Summary:

As a Senior Data Engineer, you will play a pivotal role in developing scalable Data Hubs to support various applications, including analytical, reporting, operational, and Generative AI. This position emphasizes key skills such as Python and SQL programming, expertise in cloud environments like Azure, and proficiency in Databricks and Spark. Your responsibilities will include defining architecture standards, mentoring team members, and ensuring the performance and compliance of data solutions.

Responsibilities:

• Data Hub Architecture: Design scalable Data Hub systems for integrating both structured and unstructured data.

• Technical Leadership: Define practices and patterns for data engineering implementations.

• Mentoring & Knowledge Sharing: Provide guidance and support to fellow engineers.

• Data Pipeline Optimization: Create and enhance batch and real-time data workflows.

• Quality & Monitoring: Implement measures for data validation and anomaly detection.

• Automation & CI/CD: Streamline deployment processes through automation and DevOps practices.

• Collaboration & Strategy: Work with diverse teams to align data solutions with business goals.

• Security & Compliance: Ensure data governance and adherence to regulations.

• Documentation & Knowledge Management: Produce quality technical documentation for reference.

Key Requirements:

• Programming Skills: Proficient in Python and SQL.

• Cloud Experience: Familiarity with Azure Data Factory, ADLS, Azure SQL, Synapse.

• Databricks & Spark: Extensive knowledge of Databricks and Apache Spark.

• Data Architecture: Expertise in designing enterprise-scale platforms.

• Streaming Data Processing: Experience with real-time data analysis tools.

• Automation & DevOps: Skills in CI/CD, Terraform, Kubernetes, Docker.

• Data Governance: Knowledge of data compliance practices.

• Leadership: Capability to guide teams effectively.

• Communication: Proficient in technical documentation and fluent in English (minimum B2).

Nice to Have:

• Stream Analytics Skills: Experience with Azure Stream Analytics.

• Event Streaming Tools: Familiarity with Azure Event Hubs.

• Agile Methodologies: Experience working in Agile/Scrum environments.