Senior Data Engineer

Senior Data Engineer

Experience: 6+ Years

Location: WFH / Remote (India)

Open Positions: Multiple

Qualification: B.Tech / M.Tech / MCA or equivalent

Work Timings: 1:30 PM IST – 10:30 PM IST

Functional Area: Data Engineering

About the Role

We are seeking an experienced Senior Data Engineer to design, develop, and maintain scalable, reliable, and high-performance data platforms and pipelines that support business intelligence, analytics, and data-driven decision-making.

The ideal candidate will have strong hands-on expertise in Python, Pyspark, Apache Airflow, Amazon Redshift, SQL, AWS Glue, and modern data engineering practices, along with mandatory experience in the Healthcare domain. The candidate should have experience building end-to-end ETL/ELT pipelines, processing large-scale datasets, implementing data quality frameworks, and supporting enterprise analytics and reporting needs.

This role requires a technically strong professional who can collaborate with business stakeholders, data analysts, architects, and application teams to deliver robust and scalable data solutions.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT workflows for structured and unstructured data.
• Build and optimize data ingestion, transformation, and data integration processes using Python, Pyspark and SQL.
• Develop, schedule, and monitor workflows using Apache Airflow.
• Design and manage enterprise data warehouse solutions using Amazon Redshift.
• Implement data models, data marts, and reporting datasets to support analytics and business intelligence requirements.
• Optimize data processing jobs and database performance for scalability and efficiency.
• Work with large healthcare datasets, ensuring data accuracy, consistency, and compliance with industry standards.
• Collaborate with business stakeholders to understand reporting and data requirements.
• Implement data quality checks, validation frameworks, and monitoring processes.
• Troubleshoot data pipeline issues and perform root cause analysis.
• Ensure data security, governance, and compliance requirements are adhered to.
• Participate in code reviews, development best practices, and continuous improvement initiatives.
• Create and maintain technical documentation, data flow diagrams, and data mapping documents.
• Work closely with cross-functional teams including Data Science, Analytics, Product, and Engineering teams.

Skills, Knowledge & Experience

Mandatory Skills

• 6+ years of experience in Data Engineering, Big Data Engineering, or related roles.
• Mandatory experience working in the Healthcare domain.
• Strong hands-on experience in Python development for data engineering solutions.
• Strong hands-on experience with Apache Spark (PySpark) for large-scale data processing.
• Hands-on experience with Apache Airflow for workflow orchestration and scheduling.
• Strong experience designing, developing, and optimizing solutions on Amazon Redshift.
• Advanced SQL skills with experience in query tuning and performance optimization.
• Experience building ETL/ELT pipelines and modern data integration frameworks.
• Strong understanding of data warehousing concepts, dimensional modelling, and data architecture.
• Experience handling large-scale datasets and complex data transformation requirements.
• Strong debugging, troubleshooting, and performance tuning skills.
• Experience working with cloud-based data platforms and services (AWS or Azure) is preferred.
• Excellent analytical and problem-solving skills.
• Strong communication and stakeholder management capabilities.

Preferred Skills

• Experience with AWS data services such as S3, Glue, EMR, Athena, Lambda, or Kinesis
• Experience with Azure Data Factory, Azure Databricks, ADLS, Synapse Analytics, or other Azure data services
• Exposure to Databricks, Snowflake, or modern Lakehouse architectures
• Experience with CI/CD for Data Engineering pipelines
• Knowledge of Healthcare data standards such as HL7, FHIR, EHR/EMR, Claims, Provider, Member, or Clinical Data
• Experience with data governance, data quality, and compliance frameworks
• Experience with Git and DevOps best practices for data engineering

This job description is not intended to cover or contain a comprehensive listing of all responsibilities, duties, or activities that are required. Responsibilities, duties, and/or activities may change, or new ones may be added at any time with or without notice.

If you are a motivated professional with a passion for delivering impactful solutions, we'd love to hear from you. Apply today to be part of a dynamic and forward-thinking team at NextGen Invent.