Data Engineer

Data Engineer

We are seeking an experienced Data Engineer to join the Digital Memory (DIMSUM) pod within OTA. The ideal candidate is proficient in Python, PySpark, and AWS tooling, specifically EMR, Lambda functions, DynamoDB, Athena, OpenSearch, and API development. Familiarity with Sagemaker and Elasticache is a plus.

Responsibilities:

• Update and enhance existing PySpark scripts running on EMR on EKS.
• Develop and optimize Lambda functions (Python).
• Create new PySpark scripts for data processing.
• Support the development of business insights through data analysis.
• Collaborate on Tableau dashboards for data quality monitoring.
• Contribute to iterative development for Customer Frustration Score and Predictive Intent models.
• Engage in Agile processes, working closely with a 4-member team, using JIRA for project management.

Qualifications:

• Expertise in Python, PySpark, AWS EMR, Lambda, DynamoDB, Athena, OpenSearch.
• Experience with API development and optimization.
• Familiarity with data-driven solutions, Sagemaker, and Elasticache.
• Experience working in an Agile environment with team communication tools like JIRA.
• Exposure to ML models and algorithms is a plus, though not mandatory.

This is an exciting opportunity to work on innovative digital memory initiatives in a collaborative and agile team environment!