Senior Data Engineer
Data
Senior Data Engineer
Gdynia
Type of work
Undetermined
Experience
Senior
Employment Type
B2B
Operating mode
Remote
Tech stack
Python / Java
Spark
Scala
Data Integration
ETL
Big Data
SQL
Job description
Online interview
We are seeking a highly intelligent, energetic and ambitious team player to help the data engineering team with implementing automated processes to support new data sources integrations, expanding data transformation capabilities, enabling to programmatically monitor processes and data quality across multiple systems as well as facilitating data management reporting automation.
Technical Requirements:
- 7 years relevant work experience
- 5+ years of experience developing in object oriented programming languages, preferably Python or Java
- Highly Competent with Scala, Spark, the Spark Engine, and the Spark Dataframe API
- Experience with development best practices under continuous integration, testing and deployment in an AGILE environment involving source control tools such as GIT
- Familiarity with database tools, Integration Architecture, Data Integration, ETL, Business Intelligence concepts and Big-Data solutions
- Expertise using SQL for acquiring and transforming data
- Ability to maintain, refactor, improve, and test existing code to reduce technical debt
- Knowledge and experience in integrations involving a variety of data providers
- B.S. in Computer Science or related field and industry experience
Responsibilities:
- Build scalable data processing pipelines in Spark
- Debug Spark jobs and do performance tuning
- Write unit and integration tests for all data processing code
- Read specs and translate them into code and design documents
- Perform code reviews and develop processes for improving code quality
- Integrating data from various data sources (e.g CRMs, Messaging, Video conference, Telephony, Group Document management, Contract signature/management, etc); writing in-house data back into source systems
- Reporting data quality and processes execution alerts
- Performing data extraction, transformation, cleansing and loading between different data layers and environments
- Consolidating and loading relevant data for data quality, metadata management, data consumption, UI usage and processes performance related dashboards
A huge plus if you have:
- Experience with Kafka messaging and big data technologies such as Hadoop, HDFS, MongoDB
- Familiarity with NoSQL
- Experience with Spring XD, XT
- Experience with shell scripting
- REST/SOAP APIs in addition to the cloud
- JavaScript with JSON
- CRM Knowledge, Jenkins, Airflow
- AWS S3, AWS Redshift, AWS ECS, AWS SQS
- GSuite, Slack, Jira, Confluence, Git, and Github
Check similar offers
