Data Engineering Analyst 

Job Summary

We are looking for a Data Engineer with experience in Databricks to develop, maintain, and optimize data pipelines and ETL in an Azure Databricks. You will be part of a data engineering team ensuring the quality, scalability, and automation of cloud-based data processes.

Responsibilities

Design, develop, and maintain data pipelines in Databricks.
Automate processes using Airflow.
Implement data validation and schema enforcement with Avro and Delta Tables.
Optimize query performance and storage using PySpark.
Create data products and dashboard in PowerBI.

Requirements

Experience with Databricks and Spark (PySpark).
Knowledge of Azure EventHub or Confluent Kafka.
Experience in ETL and data modeling using Delta Tables/Parquet.
Strong SQL skills for optimizing queries in distributed environments.
Hands-on experience with Avro and Delta Tables.
Skills in automation using Airflow.
Skills in create dashboard.


Nice to Have

Experience create dashboard on PowerBI.
Experience with Airflow.
Knowledge of Unity Catalog.
Familiarity with monitoring tools like Prometheus, Datadog.