Senior Data Engineer

Tech Lead Data Engineering

Position Overview:
As a Tech Lead Data Engineering, you will act as the key technical advisor for stakeholders, take ownership of data engineering projects, manage technical debt, and mentor junior team members. You will contribute to architecture, design, and development of robust data solutions.

Key Responsibilities

Leadership & Collaboration:

• Serve as the primary technical contact for stakeholders and oversee data project execution.
• Mentor junior engineers and lead design discussions.

Data Engineering:

• Develop batch and real-time data pipelines using Databricks, PySpark, and DBT.
• Clean, transform, and optimize raw data for analytics, ensuring cost and performance efficiency.
• Design and manage database schemas and optimize storage formats (e.g., Parquet, Delta Lake).
• Implement data quality checks, lineage documentation, and advanced partitioning/indexing strategies.

Collaboration:

• Partner with data scientists and analysts to ensure seamless data access for analytics and machine learning.
• Work with Platform Engineers to align on best practices.

Security & Compliance:

• Enforce role-based access controls, data encryption, and compliance with data privacy regulations.

Infrastructure & Configuration:

• Manage Azure cloud infrastructure, including Data Lake, Blob Storage, and Databricks clusters.
• Build and maintain pipelines using Azure Data Factory.

Documentation & Continuous Improvement:

• Maintain detailed documentation of pipelines, processes, and tools.
• Propose and implement optimizations for systems and workflows.

Education & Experience

• Masters or PhD in Engineering, Computer Science, or related fields.
• 5+ years of experience in data engineering, particularly in large-scale environments.
• Proficiency with cloud ecosystems, especially Azure.

Technical Skills

Programming & Tools:

• Advanced Python (e.g., Pandas, dependency management) and SQL.
• Strong software engineering practices (e.g., design patterns, clean code, testing).

Data Processing & Storage:

• Expertise in PySpark, Delta Lake, and storage formats (e.g., Parquet, JSON).
• Proficiency with Azure Data Factory, Databricks, and database management (SQL and NoSQL).

Data Architecture:

• Strong skills in schema design, ETL workflows, and modeling (Dimensional, 3NF, Data Vault).

DevOps & Automation:

• Experience with CI/CD tools (Azure DevOps) and Infrastructure as Code (ARM templates, Terraform preferred).

Security & Monitoring:

• Knowledge of role-based access, encryption, and monitoring tools (e.g., Azure Monitor).