M06 - Data Engineer

Responsibilities

• Support data engineering tasks, including the implementation, enhancement, and setup of data pipelines, as well as the rectification of broken pipelines.
• Manage the data platform and perform regular software version upgrades across environments, ensuring thorough testing and clear documentation.
• Support daily operational needs by handling application configuration changes, managing user access, addressing functional queries, and troubleshooting issues.
• Perform source code reviews and configuration reviews periodically to ensure code quality and to verify that sensitive information is not hardcoded or embedded in source code or configuration files.
• Conduct periodic patching of Azure Cloud Servers.
• Engage stakeholders to understand use cases for building new data pipelines, performing data modeling, completing data collection forms, documenting use cases, defining data attributes within various data quality zones, and establishing dataset naming conventions.
• Demonstrate a proactive and diligent approach to issue resolution, improvements, and ownership of data engineering tasks.
• Collaborate effectively within the team and communicate clearly with stakeholders.

Technical Skills

• Programming & Data Manipulation - Python, PySpark, or similar programming languages. Python packages such as Pandas, GeoPandas, Shapely.
• Databases & SQL - Experience with database design and management, e.g. PostgreSQL, MS SQL, Oracle, Geodatabase. Strong SQL programming skills (complex queries, performance tuning, data manipulation).
• ETL & Scripting - ETL (Extract, Transform, Load) experience. Scripting and version control (Bash, PowerShell, Git).
• Analytics & Tools - Technologies such as JupyterHub, RStudio, PowerBI. CI/CD tools such as Jenkins, GitLab, YAML.
• Integration & Cloud - API development and SFTP for secure data transfer. Cloud Platforms: Microsoft Azure, AWS. Cloud Technologies: Azure Data Factory, Databricks, Azure Functions, Azure Key Vault, AWS Lambda.
• Containerization - Experience with Docker, Kubernetes (added advantage).

Requirements

• Degree in Computer Science, Engineering, or related disciplines.
• Strong interest in pure Data Engineering work (not AI research or Business Analytics roles).
• Experienced in data engineering, including:
• Implementation and enhancement of data pipelines
• Development of data models and schemas
• Pipeline monitoring and management
• Setting up data pipelines on Azure, Databricks, and using Python scripting
• Experienced in architecting, designing, and developing data platforms.
• Good communication skills and ability to quickly understand technical and business requirements.
• Proactive, diligent, and able to work independently as well as collaboratively.