ETL Cloud Data Engineer

Job Summary:

We are looking for a highly skilled Python / ETL Data Engineer to design, develop, and optimize data pipelines using AWS Glue, AWS Athena, AWS Data Pipeline, and other ETLtools. The ideal candidate will have strong expertise in Python, SQL, AWS, and datapipeline management, ensuring smooth data extraction, transformation, and loading (ETL) processes.

Key Responsibilities

Develop and maintain data pipelines using AWS Glue, AWS Athena, AWS Data Pipeline, and other ETL tools.

Implement monitoring & early detection processes for pipeline issues such as missing data or lagging data.

Extract, transform, and load (ETL) data from various sources, including relational databases, non-relational databases, and flat files.

Optimize and transform data to meet business requirements and load it into target data stores.

Troubleshoot & optimize performance of data pipelines for scalability and efficiency.

Collaborate with Data Scientists and Business Analysts to understand data requirements and implement solutions.

Build and maintain REST APIs for data access and integration with external systems.

Ensure data security & compliance best practices across all pipelines.

Document & test ETL processes to maintain high-quality data flows.

Required Skills & Qualifications

Strong Python programming skills for data processing and ETL automation.

Proficiency in SQL and database design (relational and non-relational databases).

In-depth knowledge of AWS services, including AWS Glue, Redshift, S3, and EMR.

Experience in building ETL pipelines on cloud-based environments (preferably AWS).

Hands-on experience with REST API development and integration.

Familiarity with data warehousing concepts and best practices.

Understanding of data modeling, API testing, and troubleshooting.

Experience with data governance, data security, and compliance frameworks.

Exposure to big data technologies like Spark, Hadoop, or Kafka.

Knowledge of containerization (Docker, Kubernetes) and DevOps practices.

Qualifications

Minimum 5 years experience as Data Engineer with ETL Cloud Experience preferably AWS or Azure certification prefered

Strong Python Programming sills and experience building API

Malaysian Citizen or Permanent Resident

Oil & Gas Industry experience a plus

Degree in Computer Science, Engineering or equivalent