685 - Sr. Data Engineer (Python, SQL, Spark, Databricks, Azure)

Job Summary:

We are looking for a Principal Data Engineer with strong programming expertise in Python and experience in building large-scale data processing pipelines using technologies such as Databricks, Spark, Python, and Postgres SQL. Exceptional communication skills and a proactive approach to challenges are essential. Candidates with experience in the CPG or Retail industry will have an advantage, though this is not a requirement.

Responsibilities

• Programming Expertise: Design, code, and maintain large-scale data processing pipelines using Databricks, Spark, Python, and SQL.

• Data Processing: Architect and optimize data pipelines to ensure high efficiency, scalability, and reliability.

• Cloud Platform Management: Develop and deploy data solutions on cloud platforms, with a preference for Azure.

• Quality Assurance: Implement processes to maintain data accuracy, consistency, and reliability.

• Data Integration: Seamlessly integrate data from diverse sources and formats into processing pipelines.

• Data Governance: Collaborate with data governance teams to establish and enforce best practices and quality standards.

Requirements

• Programming Skills: Advanced proficiency in Python.

• Databricks Experience: Minimum of 2 years with Databricks (4+ years preferred).

• Apache Spark: Expertise in using Spark for efficient data processing.

• SQL Knowledge: Advanced skills in SQL for data analysis and transformations.

• Cloud Expertise: Strong knowledge of at least one cloud platform, preferably Azure.

• Communication: Ability to effectively communicate and challenge assumptions to drive solutions.

• Industry Knowledge: Experience in the CPG or Retail industry is advantageous but not required.

Core Skills

• Programming in Python and SQL.

• Building and optimizing pipelines using Databricks and Spark.

• Strong understanding of cloud platforms, particularly Azure.

• Quality assurance and data governance best practices.

Preferred Skills/Experience

• Familiarity with other data-centric technologies outside of Databricks, such as Data Warehousing, ETL, Analytics and Reporting.

• Previous experience in working with multiple collaborative teams, especially Data Science and Engineering delivery teams.

• Experience in the Consumer Packaged Goods (CPG) or Retail industry.

• A bachelors or masters degree in computer science, data engineering, or related fields.

• 4 years of experience in data engineering roles, focusing on large-scale data processing pipelines.