Data Engineer (Streaming Focus)

About the Role

We are looking for a skilled Data Engineer with a strong focus on streaming to join our team. You will design, build, and maintain data pipelines and streaming solutions that ensure high data availability, scalability, and reliability. This role is ideal for an engineer passionate about real-time data processing, streaming frameworks, and modern cloud-native architectures.

Key Responsibilities

• Design, develop, and optimize real-time data streaming pipelines.

• Implement and maintain Change Data Capture (CDC) and other data extraction processes.

• Work extensively with Confluent (Kafka) and Spark Structured Streaming.

• Build and maintain data pipelines using Python, SQL, Bash, Node.js, and Unix/Linux Shell.

• Ensure best practices in data engineering lifecycle: versioning, CI/CD, deployment, release management.

• Collaborate with cross-functional teams to integrate streaming solutions into wider data platforms.

• Use modern infrastructure and tooling: GitHub Actions, Terraform, Kubernetes, and other DevOps practices.

• Ensure data reliability, scalability, and performance in production environments.

Essential Skills & Experience

• Proven experience in data engineering with a focus on streaming.

• Strong proficiency in Confluent Kafka and Spark Structured Streaming.

• Experience with CDC and other data extraction techniques.

• Hands-on programming experience: Python, SQL, Bash, Node.js, Linux/Unix Shell.

• Familiarity with data engineering best practices: versioning, release management, agile methodologies.

• Experience with modern DevOps & cloud-native tools: GitHub Actions, Terraform, Kubernetes.

Nice to Have

• Knowledge of additional streaming platforms or orchestration frameworks.

• Experience working in multi-cloud environments (AWS, Azure, GCP).

• Exposure to big data ecosystems beyond streaming (e.g., Databricks, Snowflake, Delta Lake).