Senior Data Engineer

Introduction & Summary

We are seeking a Senior Data Engineer to build data systems and applications that bolster decision-making processes. This role focuses on developing and maintaining data pipelines in public clouds like AWS while participating in data migrations from on-premise environments. Ideal candidates will possess advanced technical expertise and strong communication skills, facilitating team collaboration in an Agile setup.

Main Responsibilities

The Senior Data Engineer will be responsible for:

• Building distributed and highly parallelized big data processing pipelines to handle massive amounts of structured and unstructured data in near real-time.

• Utilizing Spark to enrich and transform corporate data, enabling effective searching, visualization, and advanced analytics.

• Collaborating with analysts and business stakeholders to develop analytics models.

• Ensuring continuous delivery on Hadoop and other Big Data platforms.

• Automating processes to enhance repeatability and reliability.

• Cooperating closely with the QA team.

Key Requirements

• Minimum 5 years of experience in Spark & Scala.

• Minimum 7 years of experience in Python.

• Minimum 5 years of experience in Linux.

• Experience with distributed data processing engines like Spark.

• Experience with Hadoop and related technologies (Hive, Oozie, MapReduce, etc.).

• Strong/advanced SQL skills.

• Proven experience in creating data flows.

• Expertise in BitBucket and GIT.

• AWS certification or experience is a plus.

• Unit testing experience (JUnit 5, Mockito, Spark testing).

• Knowledge of code versioning strategy and branching strategy.

• Advanced English language skills.

Preferable location Trójmiasto / Warszawa or ŁodźHybrid work possible with 1 day/week office presence + once/twice per year physical presence in Gdynia/Gdańsk for quarterly PI Planning (2 days) may be required