Intermediate Data Engineer

Responsibilities:

• Develop algorithms to transform data into useful, actionable information
• Design, construct, install, test and maintain highly scalable data management systems
• Ensure systems meet business requirements and industry practices
• Build high-performance prototypes and proof of concepts
• Research opportunities for data acquisition and new uses of existing data
• Develop data set processes for data modelling, mining and production
• Integrate new data management technologies and software engineering tools into existing structures
• Create custom software components (e.g. specialized UDFs) and analytics applications
• Employ a variety of languages and tools (e.g. scripting languages) to marry systems together
• Install and update disaster recovery procedures
• Recommend ways to improve data reliability, efficiency and quality
• Support the achievement of the business strategy, objectives and values
• Participate and support corporate responsibility initiatives for the achievement of business strategy
• Build, test, and maintain database pipeline architectures
• Building data products for the data analytics team to ingest and train models on
• Ensure data systems (e.g. Clickhouse, MySQL) are functioning correctly and performing to expectations
• Collaborate with management to understand company objectives
• Create new data validation methods and data analysis tools
• Ensure compliance with data governance and security policies
• Document all created artefacts, create ERDs and any other needed documentation to make sure data analytics are working at capacity
• Explore new technology and look into new ways of reaching specific objectives.
• Work closely with our in-house team as well as outsourced service providers and partners.

Requirements:

• Degree / Diploma in Data Engineering preferred but not required.
• 2+ years of cloud experience
• 3+ years of experience building, maintaining and administering a DBMS, such as MySQL or Postgres.
• 2+ years experience in SQL development, including writing CTEs, and creating schemas and tables.
• A successful history of manipulating, processing and extracting value from large disconnected datasets.
• Strong analytic skills related to working with unstructured datasets.
• Fullstack development skills Front end (JavaScript) / Backend (PHP, Python) / Data administration and security (MYSQL)
• Python ORM programming experience (SQLalchemy) would be a bonus
• Proficiency in both written and spoken English
• A fundamental understanding of Basic Machine Learning
• Knowledge about server and data security on the web