Senior Databricks Engineer (Architect)

Databricks Architect

Project: Legacy Migration from SAS to Databricks

Key Responsibilities

• Migration Strategy: Lead the end-to-end migration of complex data pipelines from SAS to Databricks.

• Architecture Design: Design scalable PySpark/Delta Lake solutions to replace legacy SAS Base/Macro code.

• Code Transformation: Deconstruct existing SAS logic and re-engineer it into optimized, modern ETL/ELT workflows.

• Performance Tuning: Optimize Spark jobs and SQL queries within Databricks for maximum cost-efficiency and speed.

• Stakeholder Guidance: Act as a technical bridge between legacy data owners and the modern cloud engineering team.

Technical Requirements

• Databricks Expert: Extensive experience with PySpark, Delta Lake, and Unity Catalog.

• SAS Specialist: Deep understanding of SAS datasets, procedures, and macros (ability to read and translate legacy code).

• Senior+ Experience: Proven track record in architectural roles or leading large-scale data migrations.

• Cloud Proficiency: Hands-on experience with Azure (preferred) or AWS data services.

• Software Craftsmanship: Proficiency in Python and SQL at an advanced level.