Senior Data Architect

Join a Rapidly Growing AI Consumer Intelligence Platform Delivering Insights for the World's Leading Brands

Company Background

Our client's AI-powered consumer intelligence platform synthesizes massive-scale data (billions of Google searches, social conversations, product reviews, and videos) to deliver actionable insights for the world’s leading brands, such as Coca-Cola, Dove, Colgate, Clorox, and Vaseline, among others.

With a newly appointed CTO building the engineering team, this is your opportunity to shape how production AI is built at a high-growth company. Your impact will be immediate and visible. No legacy code politics, no entrenched hierarchies, and no inherited technical debt.

The Opportunity

Our client is seeking a Senior Data Architect to lead the modernization of its data platform. This is a high-impact role where you will redesign a Databricks-based data architecture, establishing patterns that will scale with the company's growth.

You will work directly with the CTO and collaborate with our data engineering team to transform a functional but organically grown platform into a production-grade, automated, and maintainable global system. This role requires someone who has successfully completed similar modernization projects and can mentor junior engineers while implementing best practices.

Success Criteria

• Modular, reusable Databricks notebooks with configuration-driven pipelines replacing hardcoded customer logic

• Canonical data model with clear schema contracts enabling easy onboarding of new customers, segments, and data providers

• End-to-end automation from data ingestion through to dashboard refresh with minimal human intervention

• Production-grade error handling with comprehensive logging, alerting, and recovery mechanisms

• Clear environment separation (dev/staging/prod) with CI/CD pipelines for notebook deployment

• Improved data science workflows making it easier for ML engineers to build and deploy models

• Documented architecture standards and patterns adopted by the broader team Key Responsibilities

Architecture & Design

• Redesign and implement a Medallion Architecture (Bronze/Silver/Gold) with clear data contracts at each layer

• Create a canonical data model that accommodates multi-tenant, multi-segment data (product line + country combinations)

• Establish schema evolution and deprecation strategies using Unity Catalog

• Design integration patterns for multiple data providers (social media, e-commerce, search

data)

Implementation & Migration

• Refactor existing Databricks notebooks into modular, parameterized, reusable components

• Implement comprehensive error handling, logging, and alerting frameworks

• Migrate legacy tables to new canonical schemas with data validation

• Set up CI/CD pipelines for Databricks using GitHub integration

Automation & Operations

• Design orchestration workflows using Databricks Workflows or Azure Data Factory

• Implement monitoring dashboards for pipeline health and data quality

• Create runbooks and documentation for operational procedures

Leadership & Mentorship

• Mentor and upskill data engineers on Databricks best practices and modern data architecture patterns

• Conduct code reviews and establish coding standards for the data engineering team

• Lead architectural discussions and drive consensus on technical decisions

• Collaborate with data scientists to optimize ML workflows and model deployment

Required Qualifications

Experience

• 8-12+ years of experience in data engineering, data architecture, or related roles

• 4+ years of hands-on experience with Databricks (notebooks, Delta Lake, Unity Catalog,

Workflows)

• 3+ years designing and implementing data architectures for multi-tenant SaaS platforms

• 2+ successful data platform modernization or migration projects (legacy to modern

architecture)

• 2+ years of experience mentoring or leading data engineers

Technical Skills

• Azure Cloud Platform: Deep expertise with Azure Data Services (Storage, Data Factory,

Synapse, Key Vault, Monitor)

• Databricks: Advanced proficiency with PySpark, Spark SQL, Delta Lake, Unity Catalog,

DLT (Delta Live Tables),

• Databases: Strong SQL skills with SQL Server; schema design and optimization

• Python: Production-quality Python for data engineering (error handling, logging, testing)

• CI/CD: Experience with GitHub Actions, Azure DevOps, or similar for Databricks

deployment

• BI Tools: Familiarity with Power BI and/or Apache Superset data modeling requirements

• Data Modeling: Expertise in dimensional modeling, Medallion architecture, and schema

evolution strategies

Soft Skills

• Strong communication skills; ability to explain technical concepts to non-technical stakeholders

• Proactive problem-solver who can work independently while keeping leadership informed

• Patient mentor who enjoys helping junior engineers grow Nice to Have

• Experience with LLM/GenAI integration in data pipelines (OpenAI, Azure OpenAI, Gemini)

• Background in consumer insights, CPG, or market research data

• Experience with social media data APIs (TikTok, Instagram, Facebook) or e-commerce

data (Amazon, Walmart)

• Databricks certifications (Data Engineer Associate/Professional)

• Experience with NLP pipelines and text analytics at scale

• PostgreSQL