Senior Data Engineer

Job Title: Senior Data Engineer

About the Company:

Our client is a leading big data consulting company, providing cutting-edge solutions to help businesses harness the power of their data. With a focus on delivering high-impact analytics and AI-driven insights, they empower organizations to make data-driven decisions that drive growth and innovation.

Position Overview:

We are looking for a Senior Data Engineer who is passionate about designing and implementing scalable data solutions. In this role, you will collaborate with cross-functional teams to build efficient data pipelines, ensure data quality, and support the development of innovative analytics and AI projects.

Qualifications

• Bachelors Degree in Computer Science, Software Engineering, Information Technology, or equivalent industry experience.
• Experience in programming languages such as Python, SQL, or Scala.
• Understand the concepts of Data Lake, Data Warehouse, and Data Mart.
• Experience in Big Data technologies and their ecosystem such as Hadoop, Spark and Airflow.
• Experience in building and maintaining reliable and scalable ETL on big data platforms as well as experience working with varied forms of data as structured, semi-structured or unstructured data.
• Familiar with Cloud Computing Services such as AWS, Azure, or GCP.
• Understand the process of data pipeline such as collecting, transforming, and publishing data.
• Understand the tools and design platforms that allow processing of data from multiple heterogeneous sources with different frequencies (batch/real-time).
• Innovative problem-solving skills with the ability to identify and resolve complex architectural issues.


Key responsibilities

• Architect, build, and maintain scalable, high-performance data pipelines and infrastructure to collect, process, store, and analyze large datasets.
• Implement data security measures, ensuring compliance with data privacy laws and protecting sensitive information.
• Optimize and maintain data pipelines and databases for optimal performance and scalability, including tuning big data processing jobs.
• Lead cross-functional teams in data-related projects, collaborating closely with data scientists, analysts, IT teams, and business stakeholders to achieve data-driven goals.
• Research and implement new technologies and approaches for big data processing, storage, and analytics to drive innovation and improve efficiency.
• Ensure high data quality and integrity by implementing proper data models, validation rules, and data cleaning processes.
• Mentor junior data engineers, providing guidance and support in their professional development.
• Continuously evaluate and improve existing systems and processes to adapt to changing needs and new data sources.