Data Engineer Airbyte/Databricks

Mission Overview:

Keystone Solutions is recruiting a Data Engineer consultant to deliver a central monitoring solution for a Modern Data Platform (MDP) at our client. The mission will be executed under Keystone Solutions consultancy model, working closely with the clients teams. The MDP has been established as a center of excellence for multiple data teams and comprises the following components: Data ingestion: Airbyte; Orchestration: Dagster (or a comparable orchestrator); Transformations: dbt; Lakehouse: Databricks; Visualization: Power BI; Metadata: OpenMetadata. The clients executive team requests a central monitoring solution to track MDP usage and performance (uptime, errors, usage, costs, etc.). This assignment is split into several phases.

Current phase (Data ingestion and integration) goal: Technically expose and make all necessary monitoring data centrally available in the Databricks lakehouse, based on the analysis from phase 1.

Consultancy Model at Keystone Solutions:

As a Keystone Solutions consultant, you will be engaged on a consultancy mission embedded with the clients teams. You will deliver the responsibilities listed below on-site at the client location (hybrid/on-site as required), collaborating closely with stakeholders across data engineering, platform, and analytics. You remain a Keystone Solutions employee, benefiting from our collective expertise, community of practice, coaching, and knowledge-sharing while executing high-impact client work.

Key Responsibilities (performed as a Keystone Solutions consultant):

• Implement data-ingestion flows for all selected sources, including:
  • Set up and configure Airbyte connectors (standard and/or custom)
  • Set up API calls to the various tools (Airbyte, Dagster, dbt, Databricks, OpenMetadata, among others)
• Implement authentication and connections, including:
  • API keys, service principals, VPN/network configuration if needed
• Load raw monitoring data into Databricks (source layers / raw zones)
• Model and transform data, including:
  • Unnesting/exploding JSON structures
  • Building normalized and aggregated tables for monitoring
  • Potential use of dbt for transformations
• Ensure repeatable, robust pipelines (logging, error handling, restartability)
• Handover and documentation to the dashboarding profile, including:
  • Clear description of tables, columns, refresh logic, and dependencies

Required Experience and Skills:

• Proven experience as a Data Engineer with:
  • Data ingestion via Airbyte (ideally including custom connectors)
  • Working with Databricks (Delta Lake, notebooks, SQL/PySpark)
  • API integrations (REST, JSON) and basic network concepts
• Knowledge of dbt is a strong asset
• Experience setting up monitoring/logging pipelines is a plus
• Certifications such as DataCamp – Data Analyst Associate and dbt Fundamentals are considered a plus

Additional Tools and Technologies (valued by the client):

• AWS SageMaker
• dbt
• Google Analytics and Google Tag Manager
• Google BigQuery
• Looker
• Python
• Snowplow
• SQL

Languages:

• Active knowledge of Dutch
• Active knowledge of English
• Active knowledge of French

Mandatory Candidate Question:

Together with your CV, please submit your answer to the following question. Failure to submit an answer, or if the answer is insufficient, will result in the candidate not being retained: How have you previously worked with Airbyte and Databricks? Provide a concrete example.

Consultancy Advantages with Keystone Solutions:

• Consultancy nature of work:
  • Operate as a Keystone Solutions consultant embedded within the clients environment, delivering tangible outcomes while drawing on Keystones expert community.
  • On-site collaboration at client locations (hybrid/on-site as required) to accelerate decision-making and impact.
• Dynamic projects:
  • Opportunity to tackle diverse data engineering challenges across multiple client contexts over time, from modern data platforms to analytics enablement and observability.
• Turbo-charged learning and development:
  • Access to mentorship, peer reviews, internal tech talks, and curated learning paths that reinforce Databricks, Airbyte, dbt, orchestration, and observability best practices.
• Ambition skyrocketing within a consultancy framework:
  • Clear growth paths that reward delivery excellence, client impact, and thought leadership across engagements.

Our Values in Every Client Engagement:

Being a K-Stone means you bring integrity, craftsmanship, knowledge sharing, and client impact to every project. You will uphold these values while working side-by-side with the clients teams, ensuring robust engineering standards, transparency, and sustainable solutions.

Ideal Background for This Consultancy Mission:

This role suits a hands-on Data Engineer (or a Data Analyst with strong engineering skills) who thrives on building reliable data ingestion and monitoring foundations using Airbyte, Databricks, dbt, and an orchestrator such as Dagster. You will apply your skills in a production-grade lakehouse setting and ensure the monitoring data is ready for downstream dashboarding in tools like Power BI.

Apply Now:

If you are ready to tackle technical and strategic challenges in a dynamic consultancy environment, apply today at Keystone Solutions Career Portal.