Databricks Data Engineer
Data
Databricks Data Engineer
BlueSoft
Warszawa
Type of work
Undetermined
Experience
Senior
Employment Type
B2B
Operating mode
Remote
Tech stack
Apache Spark
Azure Services
Job description
Online interview
BlueSoft was established in 2002. Thanks to our passion and the trust of our clients, it has been developing dynamically since then. Today, our experts build solutions compliant with global standards and meeting the expectations of even the most demanding customers. We love what we do and we do not slow down. We employ ambitious and talented people who are not afraid to take up challenges.
At BlueSoft, we don’t just code, we create bespoke solutions to industry leaders and share our experience to help our clients on their journey of digital transformation. From simple support to complex digital transformations, we effectively address clients technical and business needs.
Working transparent, agile, and forward-looking, we build long-term partnerships with our clients. We approach each challenge holistically, always involve both industry and domain specialists and never lose sight of the end user. We work as technology and vendor agnostics, use open source where possible and combine business domain knowledge with consulting and technology skills.
Responsibilities:
- Experience in designing and implementation of ETL/ELT processes, especially in area of Metadata-Driven Processing Framework for Azure Data Factory
- Designing and implementing extremely efficient data ingestion pipelines from various kind of sources (Azure Databricks and Apache Spark)
- Continuous improvement of the data processing on various stages and choosing right tool to the right task (not to shoot a mosquito with a cannon)
- Providing insights on the latest technology, which might be later used by customer – not only pure PoC, but with a real chance to implement it later
- Dealing with both batch and stream processing tools (based mostly on cloud) and with huge amount of data (hundreds of terabytes or even petabytes) that matter for human’s health
- Substantial discussion and permanent analysis e.g. why and where Spark might be better than Flink (and vice versa)
Requirements:
- Awareness of Data Management/Data Governance principles
- Experience in building ETL / DWH transformation processes
- Current experience in building data pipelines using Azure Data Factory and Apache Spark (Databricks).
- designing and delivering solutions using tools such as Azure Data Analytics platform, Azure Storage, Azure SQL Data Warehouse, Azure Data Lake, Azure Cosmos DB, Azure Stream Analytics etc.
- Knowledge of Apache Kafka, Nifi
- Experience in area of non-relational databases (e.g. MongoDB, Cassandra or other)
- Awareness of working with structured and unstructured data
What we offer
- We offer a team of fun-loving, interdisciplinary technology experts and enthusiasts inspired work in a flat hierarchy
- A secure, permanent full-time job with an unusual company that has been growing dynamically for over 18 years
- Cooperation with experts in their field
- The ability to work with the latest technologies based on the highest standards of development
- Multisport card and medical insurance
- Sea adventures – we regularly sail in the Caribbean, Mediterranean or North Sea
- Remote work possible.