G06 - Data Engineer

Responsibilities

• Support data engineering tasks, including the implementation and enhancement of data pipelines, as well as the rectification of broken pipelines.
• Manage the data platform and perform regular software version upgrades across all environments, ensuring thorough testing and detailed documentation.
• Support daily operational needs by handling application configuration changes, managing user access request to application, addressing general function queries, and troubleshooting on issues.
• Perform source code review and configuration review periodically to ensure code quality and verify that sensitive information, such as secrets, is not hardcoded or embedded in source codes or configuration files.

Periodic patching of Azure Cloud Servers

• Develop business processes by engaging stakeholders to understand use cases for building data pipelines, performing data modeling, completing data collection forms, documenting use cases, defining data attributes within various data quality zones, and establishing naming conventions for datasets.

Requirements:

• Degree in Computer Science, Engineering, or related disciplines
• Experienced in data engineering, including the implementation of data pipelines, development of data models and schemas, and pipeline monitoring and management.
• Experienced in architecting, designing, and developing data platform.

Experience with:

• Python, PySpark, or similar programming languages
• Python packages for data manipulation and analysis (e.g. Pandas, GeoPandas, Shapely)
• Database design and management (e.g. PostgreSQL, MS SQL, Oracle, Geodatabase)
• SQL programming (e.g. writing complex queries, optimizing performance, data manipulation)
• Scripting and version control (e.g. Bash, PowerShell, Git)
• ETL (Extract, Transform, Load) processes 
• Technologies such as JupyterHub, RStudio, PowerBI 
• CI/CD tools such as Jenkins, GitLab, YAML
• GIS technology (e.g. ArcGIS Server, PostGIS)
• API development and SFTP for secure data transfer
• Cloud Platforms (e.g. Microsoft Azure, AWS)
• Cloud Technologies (e.g. Azure Data Factory, Databricks, Azure Functions, Azure Key Vault, AWS Lambda)
• Containerization technologies (e.g. Docker, Kubernetes)

Added advantage with the following:

• Infrastructure-as-code Tools Terraform, CloudFormation
• Log Management tools (e.g. Azure Monitor, AWS CloudWatch, Splunk)
• Agile Management Tools Confluence, Jira, Kanban board