Cloud Data Engineer
New
Data
Type of work
Full-time
Experience
Senior
Employment Type
B2B
Operating mode
Remote
DCG
Cześć, tu DCG! Jesteśmy ekspertami w obszarze rekrutacji, a naszym konikiem jest branża IT. Mimo to chętnie wykorzystujemy swoje kompetencje również w realizacji projektów na stanowiska z sektorów tj. finanse, HR, marketing i wiele innych. Stawiamy na dobrą relację, zawsze będziemy grać z Tobą do jednej bramki. Przygotujemy Cię do rozmowy z naszym Klientem tak, abyś mógł pokazać się ze swojej najlepszej strony. Zróbmy wspólnie krok w Twoim rozwoju zawodowym. Nie możemy doczekać się, aż Cię poznamy! 😊
Tech stack
Python
AWS
Databricks
ETL
SQL
PySpark
Azure
Job description
Responsibilities:
- Implement and manage data ingestion pipelines from diverse sources such as Kafka, RDBMS (Postgres) using CDC (Change Data Capture), and file systems (CSV) following Medalion Architecture principles
- Develop and optimize data transformations using PySpark and SQL to handle data ranging from MB to GB, depending on the source
- Conduct unit testing and integration testing to ensure the accuracy and reliability of data transformations and pipelines
- Work with AWS technologies, including S3 for data storage and Docker on AWS for containerized applications
- Implement and manage infrastructure using Terraform, such as creating S3 buckets, managing Databricks Service Principals, and deploying infrastructure as code
- Deploy and manage solutions using CI/CD pipelines, particularly with CircleCI, to ensure seamless and automated deployment processes
Requirements:
- Minimum 4-5 years of a professional experience
- Proficiency in SQL and Python
- Strong experience with AWS cloud services
- Hands-on experience with DataBricks
- Knowledge of ETL Processing
- Effective communication skills in English (minimum B2 level)
- Knowledge of system design
- Understanding of Medalion Architecture
Nice to have:
- Familiarity with Kedro and Airbyte
- Knowledge of Machine Learning
Offer:
- Private medical care
- Co-financing for the sport card
- Training & learning opportunities
- Constant support of dedicated consultant
- Team-building events organized by DCG
- Employee referral program
Check similar offers
