Data Engineer

7N is an agent for high-end IT professionals. For over 30 years of operation we have proven that a clear and transparent financial model, collaboration exclusively with experts in their respective fields, and taking good care of them, comprise the best possible IT consulting model. We act as an individual agent for our consultants and promote their competences to our clients by offering them a wide range of projects in which they may participate. We add wage transparency, career development support and professional stability. Our main goal is a long-term collaboration; therefore the majority of our staff have been with us for many years. 

About project

We are seeking two highly skilled and experienced Data Engineers who will join a team working in a global container liner shipping company.

Team is specialising in building Data Lake and leveraging the Databricks platform.

Possibility of 100% remote work.

Tasks

• Designing and implementing robust data processing pipelines to efficiently ingest, integrate, and transform raw data from diverse sources such as internal databases, IoT data, fleet data, and third-party sources
• Ensuring seamless integration of data into the Data Lake, adhering to data quality and consistency standards
• Utilizing your strong expertise in Databricks and Apache Spark to develop and optimize data processing workflows within the Data Lake
• Leveraging Spark's capabilities to process data at scale, perform complex transformations, and facilitate data aggregation for further analysis
• Taking ownership of data storage and management within AWS, including the separation of data into different buckets and zones
• Working closely with IT administrators to ensure proper access control and data security
• Implementing data tokenization mechanisms to secure sensitive information, ensuring compliance with data privacy regulations
• Continuously monitoring the Data Lake's performance and identifying opportunities for improvement
• Optimizing data processing pipelines and storage structures to enhance data retrieval speed and overall system efficiency
• Working closely with the Product Group, who are the data owners, to understand their data needs, provide technical guidance, and ensure data quality and integrity throughout the data lifecycle
• Collaborating with data scientists to ensure data availability meets business requirements
• Providing expert technical support to the Data Lake team and data users, addressing issues related to data ingestion, processing, and access
• Troubleshooting and resolving any data-related problems in a timely manner
• Promoting best practices in data governance, ensuring adherence to data standards and documentation requirements
• Maintaining comprehensive documentation of data processing workflows and data lineage for future reference

Requirements

• At least 5+ years of experience as a Data Engineer, with a focus on building and managing Data Lake infrastructures
• Strong expertise in AWS services, including data storage and processing using S3, Databricks, and Spark
• Hands-on experience with Databricks, Apache Spark, and Python for data processing and optimization
• Proficiency in working with big data formats like Parquet, Iceberg, Delta Format
• Solid understanding of data privacy regulations and best practices in handling PII data
• Experience with data visualization and reporting tools like Qlik Sense is a plus
• Strong problem-solving skills, attention to detail, and the ability to work in a collaborative team environment
• Excellent communication skills, with the ability to convey technical concepts to both technical and non-technical stakeholders.
• Experience with containerization technologies and Infrastructure as Code (IaC) using Terraform
• Proficiency with orchestration tools like Airflow
• Proficiency in English at the C1 level

We offer

• Financial transparency: the salary is: 26 880 – 33 600 PLN (160 – 200 PLN/h) + VAT
• Clear wage model. 7N Consultants know exactly what’s 7N fee for its agent role on the project.   
• Professional development support. We subsidize trainings, technical certificates, conference participation, and foreign language classes. Our Career Development Program offers consultants advisory support in career planning. Additionally, 7N Consultants have the opportunity to participate in interpersonal skills training through the 7N Secret Code. Learn more about it here: 7N: 7N – The Secret Code   
• 7N Inspiration Team’s support and an opportunity to join their ranks. They are a group of 7N Consultants who consistently and actively share their professional knowledge and expertise.  
• Cooperation with real experts – the average professional experience of a 7N consultant is 10 years. Check out one of our discussion panels here: Does a Scrum Master need a background in programming (in Polish) 
• Comprehensive and personal project support from one of a 7N Agent. Priority regarding project continuity and quality. See what one 7N Consultant said about the role of 7N Agents and project opportunities.  
• Numerous (10-12 each year) high-end events – both online and offline. See the 7N KickOff 2023 clip here: https://www.youtube.com/watch?v=iSFtQO7cOYQ 
• Healthcare, Benefit Multisport and life insurance subsidies regardless of cooperation model. Access to Mindgram, a psychological support and professional development platform.  
• Professional and quality-oriented recruitment process led by the most experienced recruiters in the industry. Read more about it in the 7N candidate recruitment satisfaction survey (in Polish).