Data Engineer

Education Requirements, Skills, and Knowledge

• Good problem solving and decision-making skills; ability to understand and analyze complex issues
• Self-motivated, detail orientated, highly organized and able to handle a variety of tasks and responsibilities in an efficient manner with a high level of quality
• Exceptional analytical and conceptual thinking skills.
• The ability to influence stakeholders and work closely with them to determine acceptable solutions.
• Bachelor's Degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent experience).
• 5+ years of experience as a Data Engineer or in a similar role, with a strong focus on Microsoft Azure technologies.
• Proficiency with Azure Data Tools: Azure Data Factory, Azure SQL, Azure Data Lake, Azure Synapse Analytics.
• Expertise in SQL for data manipulation, optimization, and query building.
• Knowledge of programming languages such as Python or .NET for data processing and automation tasks.
• Experience with cloud-based infrastructure management and infrastructure-as-code (e.g., ARM templates, Terraform).
• Familiarity with CI/CD pipelines and version control tools (e.g., Git, Azure DevOps).
• Strong understanding of data modeling, data architecture, and data warehousing concepts.
• Knowledge of Azure security best practices and experience with data governance in cloud environments.
• Excellent problem-solving skills and a keen attention to detail.

Responsibilities:

· Data Pipeline Development: Design, develop, and maintain scalable and reliable ETL (Extract, Transform, Load) processes using Azure Data Factory, Data Lake, and other Microsoft tools.

· Data Warehousing: Implement and optimize data storage solutions using Azure SQL Data Warehouse, Azure Synapse Analytics, and/or Azure Data Lake Storage.

· Data Integration: Collaborate with other teams to integrate data from various sources, including on-premise systems, cloud services, and third-party APIs into Azure cloud infrastructure.

· Performance Optimization: Monitor and optimize the performance of data pipelines and databases to ensure minimal downtime, reduced latency, and efficient resource use.

· Data Governance & Security: Ensure data governance standards, including data quality, security, and privacy, are adhered to by utilizing Azure security tools such as Azure Key Vault and Azure Active Directory.

· Collaboration: Work with cross-functional teams, including data scientists, analysts, and software engineers, to understand business needs and deliver tailored data solutions.

· Automation & Monitoring: Set up automated data workflows, monitoring dashboards, and alerts using Azure Monitor, Azure Logic Apps, and PowerShell scripts.

· Documentation: Maintain thorough documentation of data models, pipeline configurations, and operational processes.