Job Description

Technical Core

Databricks Mastery: Expert-level knowledge of Delta Lake and the Medallion Architecture (Bronze/Silver/Gold layers).
Apache Spark (PySpark/SQL): Ability to write optimized Spark code. For coming POC, Python is usually preferred over Scala/R for its flexibility and ecosystem.
AWS Infrastructure: Deep understanding of S3 (Bucket policies/storage), IAM (Roles/Policies) for secure Databricks access, and VPC/Networking (Good to have)
Data Ingestion: Experience with Databricks Autoloader or Unity Catalog for managed data governance.

Role Summary:

We are seeking a hands-on Data Engineer to spearhead our Databricks POC on AWS. You will be responsible for the initial environment setup, security configuration, and designing the framework for our future data platform.
You will connect diverse AWS and external data sources into a unified Databricks environment.

Key Responsibilities:

Configure Databricks workspace integration with AWS (S3, IAM, VPC).
Cleanse and transform raw data from S3, RDS, and APIs into Delta tables.
Design and implement a scalable Medallion Architecture using Delta Lake.
Build automated ingestion pipelines using Databricks Autoloader.
Optimize Spark jobs for performance and reliability.
Establish data governance standards using Unity Catalog. (Good to have)
Evaluate POC success metrics (performance, cost, ease of use).

Requirements: 3-5+ years in Data Engineering with PySpark/SQL; strong experience with AWS Glue or EMR is a plus.; Databricks Certified Data Engineer Professional preferred.