Big Data Cloud Engineer /Admin
Data
Big Data Cloud Engineer /Admin
Gdynia
Type of work
Undetermined
Experience
Senior
Employment Type
B2B
Operating mode
Remote
Tech stack
Hadoop
Spark
PySpark
Kafka
Python
Azure
Hive
Job description
Online interview
We are looking for a Senior Big Data Cloud Engineer/Admin who is able to translate functional and technical requirements into detail design. Responsible for analyzing large data sets to develop custom data pipelines to drive business solutions.
Chosen perks at Idego:
Chosen perks at Idego:
- 100% remote work, recruitment & onboarding
- 100% paid holidays (24 working days)
- innovative and complex platform that will go live in 2020
- experienced team from 4 to 10+ years in commercial projects
- unique #memes channel
- private medical insurance (basic dental services included) and Multisport
- a lot of common sense, common goal approach and general friendlines
Responsibilities:
- Work with team leads for regular updates, requirement understanding and design discussions
- Develops data pipeline for ingesting & cleansing data, applies data validation & transformation, stores data in various zones using Python, Hive & PySpark
- Gathers the data from various third-party data in to cloud and on-Prem data platforms, optimizing it and joining with internal datasets to gather meaningful information. Stores data in various zones in right file format using appropriate compression techniques (ORC, Parquet and Avro)
- Plan the closure of the solutions and see through the implementation of the solutions
- Prepare and produce releases of data pipelines
- Participates in day to day project and production delivery status meetings and provide technical support for faster resolution of issues
Check similar offers
