Senior Data Engineer

Senior Data Engineer

Gdansk, Poland

Job Description

We are a dedicated team passionate about revolutionising data engineering. Specialising in building Data Lake and leveraging the Databricks platform, we constantly innovate to drive business success. Working primarily in Spark and Python, we collaborate seamlessly to transform raw data into actionable insights, fuelling informed decision-making processes. Our team members bring a diverse range of expertise, ensuring a well-rounded approach to every area of our Data Landscape. We thrive in a dynamic environment, continuously learning and adapting to the evolving landscape of data technology.

Responsibilities

• Design and implement robust data processing pipelines to efficiently ingest, integrate, and transform raw data from diverse sources such as HL internal databases, IoT data, fleet data, and third-party sources.
• Ensure seamless integration of data into the Data Lake, adhering to data quality and consistency standards.
• Utilise your strong expertise in Databricks and Apache Spark to develop and optimise data processing workflows within the Data Lake.
• Leverage Spark's capabilities to process data at scale, perform complex transformations, and facilitate data aggregation for further analysis.
• Take ownership of data storage and management within AWS, including the separation of data into different buckets and zones.
• Work closely with IT administrators to ensure proper access control and data security.
• Implement data tokenization mechanism to secure sensitive information, ensuring compliance with data privacy regulations.
• Continuously monitor the Data Lake's performance and identify opportunities for improvement.
• Optimise data processing pipelines and storage structures to enhance data retrieval speed and overall system efficiency.
• Work closely with the Product Group, who are the data owners, to understand their data needs, provide technical guidance, and ensure data quality and integrity throughout the data life-cycle.
• Collaborate with data scientists to ensure data availability meets business requirements.
• Provide expert technical support to the Data Lake team and data users, addressing issues related to data ingestion, processing, and access.
• Troubleshoot and resolve any data-related problems in a timely manner.
• Promote best practices in data governance, ensuring adherence to data standards and documentation requirements.
• Maintain comprehensive documentation of data processing workflows and data lineage for future reference.

Qualifications

• Bachelor’s or master’s degree in computer science, Data Science, or related fields.
• At least 5+ years of experience as a Data Engineer, with a focus on building and managing Data Lake infrastructures.
• Strong expertise in AWS services, including data storage and processing using S3, Databricks, and Spark.
• Must have hands-on experience with Databricks, Apache Spark and Python for data processing and optimisation.
• Proficiency in working with big data formats like Parquet, Iceberg, Delta Format.
• Solid understanding of data privacy regulations and best practices in handling PII data.
• Experience with data visualisation and reporting tools like Qlik Sense is a plus.
• Strong problem-solving skills, attention to detail, and the ability to work in a collaborative team environment.
• Excellent communication skills, with the ability to convey technical concepts to both technical and non-technical stakeholders.
• Experience with containerisation technologies and IaC (Terraform)
• Proficiency with orchestration tools like Airflow
• English language skills. 

About Us

With a fleet of 264 modern container ships and a total transport capacity of 2.0 million TEU, Hapag-Lloyd is one of the world’s leading liner shipping companies. In the Liner Shipping segment, the Company has around 13,500 employees and 400 offices in 135 countries. Hapag-Lloyd has a container capacity of 2.9 million TEU – including one of the largest and most modern fleets of reefer containers. A total of 113 liner services worldwide ensure fast and reliable connections between more than 600 ports across the world. In the Terminal & Infrastructure segment, Hapag-Lloyd has stakes in 20 terminals in Europe, Latin America, the United States, India, and North Africa. The roughly 2,600 employees assigned to the Terminal & Infrastructure segment handle terminal-related activities and provide complementary logistics services at selected locations.

About The Team

Knowledge Center in Gdansk

One of the goals of Hapag-Lloyd strategy, is to become more agile in the way we work and to create an environment in which we can make faster and innovative decisions. Knowledge Center in Gdansk enables the further accelerated growth, especially in the area of developing innovative digital solutions, agility and business centricity. To learn more, please check our website - https://knowledgecenter.pl/ 

We offer:

• Private medical care (Saltus)
• Gym card (Multisport)
• Attractive annual bonus up to 22,5%! (depending on company performance results)
• Group life insurance and employee capital plan (PPK)
• Cafeteria benefit system (cinema tickets, vouchers etc.)
• Focus on healthy lifestyle (fruit days, bike competitions, football trainings)
• Charity and volunteer initiatives
• Modern and well-connected office (Alchemia complex in Gdańsk Oliwa)
• Relocation support (financial support, covering immigration process for non-Polish citizens)
• Internal learning management system
• Development budget (sharing the costs of certifications and conferences/ IT events)
• Flexible working hours and home office possibility (hybrid work model)