Solution Architect - Databricks

As a Solution Architect – Databricks, you will design and deliver enterprise‑grade data solutions on Azure Databricks, leading a large‑scale data modernisation programme for a telecommunications client. You will architect migration strategies, define governance frameworks, and provide technical leadership to ensure seamless transformation from legacy data warehousing systems to a modern lakehouse architecture.

What You'll Do and How You'll Succeed

Architecture & Design

• Design and architect end‑to‑end data pipelines using Azure Databricks with Medallion Architecture (Bronze Silver Gold Semantic layers).
• Lead hybrid orchestration frameworks combining DLT‑Lakeflow for CDC ingestion and metadata‑driven frameworks for transformation layers.
• Define and implement Unity Catalog governance strategies for data access, security, and lineage.
• Architect Databricks Workflows for job scheduling, dependency management, and replacing legacy Control‑M orchestration.
• Design SCD Type 2 implementations and complex transformation patterns for EDW migration.

Technical Leadership

• Provide guidance on PySpark and Spark SQL best practices for large‑scale data processing.
• Define coding standards, modular notebook organisation, and configuration‑driven development approaches.
• Lead technical decisions on schema evolution, data quality frameworks, and reconciliation strategies.
• Recommend approaches for surrogate key generation, batch control mechanisms, and selective reprocessing.

Migration & Modernisation

• Architect migration strategies from legacy systems (ODS, EDW, Informatica PowerCenter) to Azure Databricks lakehouse.
• Design CDC ingestion patterns using Oracle GoldenGate and Informatica IDR integration.
• Define approaches for historical data migration and parallel run strategies.
• Ensure minimal refactoring of existing SQL/PySpark code through metadata‑driven frameworks.

Operational Excellence

• Design solutions supporting BAU operations including manual data patching, hold/resume/skip controls, and entity‑level reprocessing.
• Architect job/cycle/batch auditing frameworks aligned with existing operational patterns.
• Define monitoring, alerting, and logging strategies across all pipeline layers.
• Ensure solutions support targeted fixes without requiring full refreshes.

Stakeholder Engagement

• Collaborate with Amdocs and customer architecture teams on design approvals and technical alignment.
• Participate in discovery sessions for EDW entities (DDS, SDS, DNF).
• Present technical solutions and trade‑off analyses to leadership and steering committees.
• Work closely with delivery teams to translate designs into implementable solutions.

We'd Love to Hear From You If...

Experience

• You have 8+ years in data engineering and architecture roles.
• You have 4+ years of hands‑on experience with Databricks (Delta Lake, Spark, Unity Catalog).
• You have 3+ years of experience with Azure cloud services (ADLS Gen2, Azure DevOps, Azure Data Factory).
• You have proven experience in large‑scale data migration projects, ideally EDW modernisation.

Technical Expertise

• You are expert in Azure Databricks (Delta Lake, DLT/Lakeflow, Databricks Workflows, Unity Catalog).
• You are proficient in PySpark and Spark SQL for optimised transformations.
• You have hands‑on experience with Medallion Architecture (Bronze/Silver/Gold).
• You are skilled in dimensional modelling, SCD implementations, and surrogate key strategies.
• You are experienced in metadata‑driven ETL/ELT frameworks.
• You have worked with CDC tools such as Oracle GoldenGate or Kafka.
• You are familiar with data quality frameworks, reconciliation, and exception handling.
• You are experienced with Git, CI/CD pipelines, and Databricks Asset Bundles (DAB).
• Preferred: Power BI, Informatica PowerCenter/IDR, Control‑M, Terraform/IaC, telecommunications domain.

Ways of Working

• You have strong analytical and problem‑solving abilities.
• You communicate complex technical concepts clearly to diverse audiences.
• You collaborate effectively across distributed teams and time zones.
• You mentor and guide junior team members.
• You document design decisions thoroughly (HLSD, LLD).
• You bring a collaborative mindset when working with customer stakeholders.

Certifications (Preferred)

• Databricks Certified Data Engineer Professional
• Databricks Certified Solution Architect Professional
• Microsoft Azure Data Engineer Associate (DP‑203)
• Microsoft Azure Solutions Architect Expert (AZ‑305)