DataOps Engineer

7N is an agent for high-end IT professionals. For over 30 years of operation we have proven that a clear and transparent financial model, collaboration exclusively with experts in their respective fields, and taking good care of them, comprise the best possible IT consulting model. We act as an individual agent for our consultants and promote their competences to our clients by offering them a wide range of projects in which they may participate. We add wage transparency, career development support and professional stability. Our main goal is a long-term collaboration; therefore the majority of our staff have been with us for many years. 

About project

We are seeking highly skilled and experienced DataOps Engineer who will join a team working in a global container liner shipping company. Team is specialising in building Data Lake and leveraging the Databricks platform. This role is an exciting opportunity to lead the development and optimization of data orchestration platform, leveraging Astronomer and Airflow, and enabling CI/CD workflows for a cutting-edge, cloud-based Data Warehouse built on Databricks and AWS.

Possibility of 100% remote work.

Tasks

• Developing and maintaining a scalable Astronomer-based scheduling environment for our Data Lakehouse platform
• Designing and implementing CI/CD pipelines to automate deployments, validate data models, and monitor workflows
• Configuring and managing Astronomer workspaces, deployments, and access control
• Integrating dbt Core with Cosmos for orchestrating dbt models in Airflow
• Building and optimizing Cosmos Airflow DAGs for data ingestion and transformation
• Documenting and implementing deployment processes, including ephemeral environments for development
• Collaborating with stakeholders to ensure best practices in pipeline design and Airflow job scheduling
• Leveraging AWS services (e.g., S3) and Databricks for performance, scalability, and security
• Troubleshooting and supporting data scheduling, CI/CD processes, and model deployment pipelines
• Ensuring comprehensive documentation of workflows, CI/CD processes, and operational standards
• Finalizing the migration of the existing Data Lake scheduling from Airflow MWAA to Astronomer

Requirements

• At least 5+ years of experience as a Data Engineer or DataOps Engineer, focusing on CI/CD pipelines and workflow orchestration
• Expertise in configuring and deploying Astronomer or Apache Airflow in cloud environments
• Hands-on experience with Databricks, dbt Core, and AWS services
• Proficiency in Python and Spark for data transformation and pipeline development
• Strong SQL skills and knowledge of relational databases
• Familiarity with GitFlow and CI/CD tools like GitLab or Jenkins
• Knowledge of dimensional modeling, data access patterns, and data storage optimization
• Experience with Terraform for Infrastructure as Code and Airflow DAGs with Cosmos
• Excellent collaboration and communication skills, with a proactive approach to sharing knowledge
• Proficiency in English at the C1 level

We offer

• Financial transparency: the salary is: 26 880 – 31 920 PLN (160 – 190 PLN/h) + VAT
• Clear wage model. 7N Consultants know exactly what’s 7N fee for its agent role on the project.   
• Professional development support. We subsidize trainings, technical certificates, conference participation, and foreign language classes. Our Career Development Program offers consultants advisory support in career planning. Additionally, 7N Consultants have the opportunity to participate in interpersonal skills training through the 7N Secret Code. Learn more about it here: 7N: 7N – The Secret Code   
• 7N Inspiration Team’s support and an opportunity to join their ranks. They are a group of 7N Consultants who consistently and actively share their professional knowledge and expertise.  
• Cooperation with real experts – the average professional experience of a 7N consultant is 10 years. Check out one of our discussion panels here: Does a Scrum Master need a background in programming (in Polish) 
• Comprehensive and personal project support from one of a 7N Agent. Priority regarding project continuity and quality. See what one 7N Consultant said about the role of 7N Agents and project opportunities.  
• Numerous (10-12 each year) high-end events – both online and offline. See the 7N KickOff 2024 clip here: https://www.youtube.com/watch?v=8MjSywOKZnc
• Healthcare, Benefit Multisport and life insurance subsidies regardless of cooperation model. Access to Mindgram, a psychological support and professional development platform.  
• Professional and quality-oriented recruitment process led by the most experienced recruiters in the industry. Read more about it in the 7N candidate recruitment satisfaction survey (in Polish).