Job Description

Overview

We are seeking skilled Data Engineers to join our Data & Digital Twin Foundation team. You will design, build, and maintain data pipelines that power digital twin platforms, real-time operational systems, and AI/ML workloads. Working closely with data architects, simulation engineers, and ML teams, you will transform raw operational data into high-quality, governed datasets that drive intelligent decision-making.

Our core data platform stack includes:

Data Platform & Lakehouse

Databricks (PySpark, Databricks SQL) for unified analytics and data engineering
Delta Lake for ACID-compliant lakehouse architecture
Unity Catalog for data governance, lineage, and access control

Stream & Event Processing

Apache Kafka for real-time event ingestion
Structured Streaming for continuous data processing
Delta Live Tables for declarative, quality-enforced pipelines

Specialized Data Stores

Neo4j for graph data modeling and network topology
Python and SQL for data transformation

Data Quality

Delta Live Tables expectations for data validation
Data profiling and anomaly detection

Key Responsibilities

Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Delta Lake
Build real-time and batch data ingestion pipelines from diverse operational systems
Implement data transformations that serve digital twin platforms and operational analytics
Develop and maintain graph data models in Neo4j for network topology and relationship modeling
Integrate Kafka event streams with Databricks for real-time operational state updates
Implement data quality checks using Delta Live Tables expectations
Ensure data governance compliance through Unity Catalog (lineage, access control, metadata)
Optimize pipeline performance, reliability, and cost efficiency
Write clean, well-documented, and testable code following engineering best practices
Collaborate with ML engineers to deliver feature-engineered datasets
Participate in code reviews, knowledge sharing, and continuous improvement initiatives
Support production data systems through monitoring, troubleshooting, and incident resolution

Preferred Qualifications

7+ years of hands-on data engineering experience
Track record of building and maintaining production-grade data pipelines
Experience with Delta Live Tables for declarative pipeline development
Experience working in agile, cross-functional teams
Familiarity with time-series data patterns and operational data modeling

Highly Desirable

Experience building data pipelines for digital twin or simulation platforms
Familiarity with operational state modeling for real-time systems
Exposure to physics-informed or time-series ML feature engineering
Experience working with distributed, multidisciplinary teams
Exposure to industrial domains such as Manufacturing, Logistics, or Transportation is a plus