Senior Data Engineer with Flink

Information about project:

Location: Fully remote

Rate: up to 170 pln/h net + VAT, B2B

Summary:

The primary goal of this role is to enhance and maintain the Flink platform on a leading audio streaming service, focusing on the development of streaming pipelines, data migration, and the implementation of AI tools for efficient processes.

Main Responsibilities:

• Enhance the Flink platform to improve performance and reliability.

• Perform migrations for DataStream API and upgrade to Flink 2.0.

• Build and improve streaming pipelines for faster data insights.

• Migrate data storage from BigQuery to Google Cloud Storage (GCS).

• Convert datasets from Avro to Parquet format.

• Develop a system for SQL anti-pattern detection and assist teams in fixing them in DBT pipelines.

• Leverage AI tools to automate migration tasks and support scaling operations.

Key Requirements:

• Strong hands-on experience with Apache Flink, especially the DataStream API.

• Proven ability to upgrade and maintain Flink environments, ideally with exposure to Flink 2.0.

• Deep understanding of streaming pipeline architecture, including performance tuning and fault tolerance.

• Experience migrating large-scale datasets from BigQuery (BQ) to Data Cloud Storage (DCS).

• Proficiency in data format conversion, especially Avro to Parquet.

• Ability to scale and automate migration workflows, ensuring data integrity and minimal downtime.

• Solid knowledge of Google Cloud Platform (GCP) and its data services.

• Familiarity with distributed systems, schema evolution, and storage optimization.

• Practical experience using AI-based tools to accelerate data migration and transformation tasks.

• Understanding of how to apply machine learning or intelligent automation to validate and optimize data workflows.

• English proficiency level: min. B2