Senior Data Engineer

About Source Edge

Source Edge is a Tokyo-based, technology recruitment and career consulting company. We work with select number of client partners, helping them to find both local and global talent as they tackle the unique challenges of hiring in the Japan market. For our candidate partners, we strive to go the extra mile and provide transparency, timely updates and in-depth insights in to our clients and their hiring processes.

Who is our Client?

Our client is a Japanese technology platform that is working to not only transform, but revolutionise, the way that domestic industries utilise data and technology.  By processing vast amounts of data from companies in industrial and manufacturing industries, they are able to transform it in to forms that are easier to utilize and quickly implement in to the business and commercial business.


What would you will do...

• Design and implementation of data infrastructure
  • Manufacturing cost estimation data
  • Transaction performance data
  • Manufacturing control process data
  • Usage data for various products
• Analysis of base data to obtain hypotheses concerning business and operational improvement
• Build data processing pipelines in cooperation with algorithm designers

Technical Environment

• Development Language
  • Frontend: HTML, CSS, TypeScript
  • Backend: TypeScript, Rust, Kotlin
• Frameworks
  • Frontend: React, Apollo, Next.js, styled-components, WebGL, WebAssembly
  • Backend: Rust (tokio, tonic, diesel), Kotlin (Micronaut, Exposed), Node.js (express, NestJS)
• Infrastructure: GCP, GKE
• Database: PostgreSQL, Firestore
• API: GraphQL, REST, gRPC
• Authentication: Auth0
• Data Pipeline: GCP Vertex AI Pipeline, Dataflow
• Development tools: GitHub, CircleCI, Figma, Sentry, DataDog, Storybook
• Communication tools: Slack, Discord, JIRA, Miro

What You'll Need...

• 3+ years of hands-on experience with server-side technologies
• 3+ years of experience with data pipelines or data processing frameworks
• Experience using SQL for analysis
• General knowledge of Linux-based infrastructure and public cloud architecture
• Familiarity with container technologies such as Docker
• Experience operating container-related technologies such as Kubernetes, Tekton, Argo, etc.
• Experience with data pipeline technologies such as Airflow, Apache Beam, Spark, etc.
• Hands-on experience with declarative infrastructure-as-code technologies (terraform, puppet, ansible, etc)
• Experience using version control systems such as Git
• Proficiency in English

(Preferred Qualifications)

• General knowledge of computer networking
• Experience with cloud providers such as AWS, GCP, or Azure
• Experience monitoring data pipelines with modern solutions such as Datadog, New Relic, Grafana, etc
• Experience with production use cases of Elasticsearch
• Experience with kNN (ANN) frameworks (faiss, nmslib, etc) and associated database technologies (Milvus, Vald, etc)
• Experience with machine learning frameworks (Chainer, PyTorch, Tensorflow, etc)
• Experience in application development or analysis work with Rust, Python, MATLAB, or R

What Our Client Can Offer You...

• A competitive compensation and benefits package
• An opportunity to work on engineering problems that could serve to revolutionise traditional, multi-billion dollar industries
• A supportive Japan team and an environment committed to internationalisation
• Relocation to Japan (when COVID restrictions allow)

For overseas candidates

• Relocation support is available
• Visa support is available
• Japanese language educational support is available