Senior Data Engineer

The Senior Data Engineer plays a critical role in designing, developing, and maintaining scalable data infrastructure and pipelines. This position supports data-driven decision-making across the organization by ensuring data quality, accessibility, and performance.

Roles and Responsibilities

• Data Pipeline Development: Design, build, and maintain robust, scalable data pipelines using tools like Apache Airflow, Kafka, and Spark.
• Data Architecture: Develop and optimize data models, data warehouses, and ETL processes to support analytics and machine learning initiatives.
• Collaboration: Work closely with data scientists, analysts, and software engineers to understand data requirements and deliver tailored solutions.
• Data Quality & Governance: Ensure data integrity, implement validation checks, and enforce data governance policies.
• Performance Optimization: Monitor and improve data processing speed, system uptime, and query performance.
• Leadership & Mentorship: Guide junior engineers, promote best practices, and foster a culture of continuous learning.
• Innovation & Research: Stay updated with emerging technologies in AI, automation, and data streaming; lead initiatives to integrate them into existing systems.
• Other duties as required.

Desired Skills and Experience

• Bachelor's or master's degree in computer science, data science, information technology, or a related field, with 6+ years of experience.
• Proficiency in SQL, Python, and ETL/ELT tools
• Experience with big data technologies (Hadoop, Spark)
• Familiarity with cloud platforms (Azure, AWS, and GCP)
• Strong understanding of data modelling, data warehousing, and stream processing
• Knowledge of data security, compliance, and governance
• Knowledge of CI/CD pipelines using Azure DevOps
• Understanding of data security and compliance (e.g., GDPR, HIPAA)

Experience in Microsoft solutions are as follows:

• Azure Data Factory
• Microsoft Fabric
• Azure Synapse Analytics
• Azure Databricks
• Power BI
• Azure SQL Database / SQL Server
• Azure Purview