Senior data Engineer

Job Description:

The ideal candidate is adept at using large data sets to find opportunities for product and process optimization and using models to test the effectiveness of different courses of action. Must have strong experience using a variety of data mining/data analysis methods, using a variety of data tools, building and implementing models, using/creating algorithms and creating/running simulations. They must have a proven ability to drive business results with their data-based insights. They must be comfortable working with a wide range of stakeholders and functional teams. The right candidate will have a passion for discovering solutions hidden in large data sets and working with stakeholders to improve business outcomes.

• Strong experience with various Data Management architectures like Data Warehouse, Data Lake, Data Hub and the supporting processes like Data Integration, Governance, Metadata Management

• Strong ability to design, build and manage data pipelines for data structures encompassing data transformation, data models, logics, schemas, metadata and workload management.

• Strong experience in working with large, heterogeneous datasets in building and optimizing data pipelines, pipeline architectures and integrated datasets using traditional data integration technologies. These should include ETL/ELT, data replication/CDC, message-oriented data movement, API design and access and upcoming data ingestion and integration technologies such as stream data integration

• At least three years or more of work experience in data management disciplines including data integration, modeling, optimization and data quality, and/or other areas directly relevant to data engineering responsibilities and tasks.

• Collaborating with various teams and providing periodic updates through presentations and prototype demonstrations. The role will require working on multiple projects simultaneously. This position will also be involved in the formulation of key business requirements to be solved, rationalizing the various analytical approaches to solve those problems.

• Maintains state-of-the-art knowledge and contributes to technical discussions and reviews as an expert in related areas of responsibility

• Communicates ideas, plans, and results effectively via oral and written reports - Works effectively with peers, management, operations groups, and outside organizations

• Participate in technical forums and other appropriate events and conferences

• This position is involved in helping to develop, analyze and draw conclusions, and presenting the results back to business users. Depending on the candidate’s qualification, this position can be either a Data Engineer or a Data Scientist.

Qualifications:

• Strong ability to design, build, and manage data pipelines for data structures encompassing data transformation, data models, schemas, metadata and workload management.

• The ability to work with both IT and business in integrating analytics and data science output into business processes and workflows.

• Strong experience with popular database programming languages including [SQL, NO SQL

• Strong experience in working with large, heterogeneous datasets in building and optimizing data pipelines, pipeline architectures and integrated datasets using traditional data integration technologies. These should include [ETL/ELT, data replication/CDC, message-oriented data movement, API design and access] and upcoming data ingestion and integration technologies such as [stream data integration, CEP and data virtualization].

• Strong experience in working with SQL on Microsoft SQL Server.

• Strong experience in working with and optimizing existing ETL processes and data integration and data preparation flows and helping to move them in production.

• Basic experience working with Power BI for semantic-layer-based data discovery.

• Strong experience of using Alteryx as ETL tool and Qlik as BI tool.

Other Skills and Abilities:

• Strong ability to work in a fast-paced environment

• Strong ability to work both autonomously and in a team

• Strong ability to communicate complex quantitative analysis in a clear, precise, and actionable manner to both technical and non-technical audiences

• Excellent time-management, multi-tasking, and communication skills. Capable of juggling multiple projects and related complexities at the same time

• Good interpersonal and communication skills

• Knowledge in Hadoop ecosystems will be a plus

• Have a good understanding of latest technology trends