Senior Data Engineer

Responsibilities

• Implement data-facing backend features: analyze, design, and code services that read/write Delta tables, respecting schema evolution and time travel; document APIs and data contracts.

• Validate feasibility of new requirements within the existing medallion architecture; propose changes to bronze/silver/gold layers and indexing/partitioning where needed.

• Design and maintain interfaces to enterprise systems and BI tools; coordinate with interfacing parties and data governance for schema and access approvals.

• Create unit tests and component integration tests for Spark/SQL and service endpoints; establish data quality checks and reconciliation steps.

• Optimize performance and security: tune Spark jobs, improve query plans, enforce access controls, and ensure auditability.

• Maintain build and configuration: manage YAML pipelines in Azure DevOps, version artifacts, and automate deployments to Databricks environments.

• Drive DevOps for data services: monitoring, alerting, and incident-aware coding; assess release impacts on operations and downstream consumers.

• Conduct code reviews via pull requests; lead bug fixing with root cause analysis and preventive controls at data and code levels.

Qualifications

• Successfully completed degree in Computer Science, Mathematics, Physics, or related field.

• At least 6 years overall engineering experience with 4+ years in Data Engineering with Apache Spark on Azure Databricks; proven delivery of production-grade pipelines.

• With strong communication skills enabling effective collaboration with customers and teams.

• Strong Data Engineering Core: Apache Spark, Delta Lake, Medallion architecture

• Experience with Azure Databricks (workspaces, clusters/pools, secrets), Azure Data Lake Storage Gen2, serving layers on Azure SQL.

• Nice to have: investment/asset management domain knowledge (ESG, alternatives, performance analytics), KPI lineage, event streaming (Azure Event Hub/Kafka).

• Very good communication in English and Polish (German is a plus)

• Openness to visit Kraków or Katowice office up to 2×/month.

About Cluster Dynamics Reply

Cluster Reply is the Reply Group company specialising in consulting and system integration of Microsoft technologies. As a Microsoft partner, Cluster Reply is active in Germany, Austria and Switzerland and works within the Reply network with sister companies in Brazil, Great Britain, Italy as well as the USA. The company focuses on innovation and supports customers in their digital transformation. The solutions range from on-premises to cloud applications in the areas of modern workplace and security, business applications, applications and infrastructure as well as data and artificial intelligence.