Johannesburg, South Africa

Data Engineer | DISA-DIA-OMDE-JHB-0091

 Job Description:


1. Azure Databricks Hands-On Experience

  • Status: Exceeded
  • Notes: She brings 3 dedicated years of hands-on Databricks experience, specifically architecting and implementing ETL/ELT pipelines using Azure Databricks, Spark clusters, high concurrency performance tuning, and Databricks Asset Bundles (DABs) in her current role at Investec and previous work with Standard Bank.

2. Apache Spark, PySpark, and SQL

  • Status: Met
  • Notes: Highly proficient across Python, PySpark, SparkSQL, and T-SQL. Her experience includes developing complex transformations and automated notebooks for enterprise-scale data workloads.

3. Delta Lake & Medallion Architecture (Bronze, Silver, Gold)

  • Status: Met
  • Notes: Extensively experienced in implementing Lakehouse architecture using Delta Lake, delta tables, parquet files, and executing full on-prem to cloud data migrations utilizing the Medallion Architecture across multiple financial sector roles.

4. Data Pipelines & Data Integration Solutions

  • Status: Met
  • Notes: Extensive enterprise background building batch processing data pipelines, integrating disparate sources (REST APIs, Cosmos NoSQL, Azure SQL, on-prem databases) using Azure Data Factory (ADF), SSIS, and custom orchestration tools like Control-M.

5. Azure DevOps and CI/CD Pipelines

  • Status: Met
  • Notes: Strong 5-year track record managing CI/CD pipeline workflows for schema migrations, infrastructure deployments (using Terraform and Bicep/ARM templates), and code promotion via Azure DevOps and Git/GitHub.

6. Data Quality, Performance Tuning, and Cost Optimization

  • Status: Met
  • Notes: Leverages data governance frameworks like Unity Catalog and data quality best practices. She also brings strong relational database and cloud performance tuning experience (indexing, partitioning, cluster optimization) gained across 11+ years of database development..

7. Advantageous Skills (Generative AI / LLMs & SharePoint Integration)

  • Status: Partial / Present
  • Notes: Her CV notes exposure to Azure ML and AI services (retraining/deploying models), although explicit Generative AI/LLM project deployment and specific SharePoint integrations are not highlighted and should be verified in conversation if needed.
  Required Skills:

Data