Data Engineer

Job Description: 

• Create and manage data pipelines for diverse sources in our data lake/warehouse. 
• Collaborate with analysts to implement necessary data transformations. Research relevant data sources for property development. 
• Develop web scraping scripts for data acquisition. 
• Automate deployment and testing of data pipelines using CI/CD. 
• Utilize Azure, AWS, and other technologies for data solutions. 
• Implement data integration between systems. 
• Monitor and optimize pipeline performance. 
• Ensure data security, compliance, and quality. 
• Stay updated on data engineering trends.

Requirements:

• Bachelor's or Master's degree in Computer Science, Engineering, or related field.
• Proven experience as a Data Engineer, with strong understanding of data engineering concepts.
• Hands-on experience with Azure (Azure Function, Azure Data Factory, Azure Databricks) and/or AWS (AWS Glue, Lambda, Amazon Redshift).
• Proficiency in Python, Java, or Scala, and experience with data processing frameworks (e.g., Spark, Hadoop).
• Knowledge of data modelling, warehousing, and database technologies.