Associate Data Architect I / Senior Data Engineer

Role Description

Job Description: Lead Data EngineerRole Summary

We are looking for a Lead Data Engineer with strong expertise in Azure Databricks, PySpark, and modern data engineering practices. The role involves building scalable data pipelines, optimizing large data workloads, and contributing to Lakehouse platform development within a banking and financial services environment.

Key Responsibilities

Develop and maintain batch and streaming data pipelines using Azure Databricks, PySpark, Delta Lake.

Build and optimize ETL/ELT workflows for banking datasets (payments, transactions, customer, fraud, regulatory).

Implement Lakehouse best practices including Delta tables, schema evolution, and data quality checks.

Optimize Spark jobs using caching, partitioning, Z-ordering, and performance tuning.

Manage ingestion using Auto Loader, ADF/Synapse, Event Hubs/Kafka.

Ensure security, governance, and compliance using Unity Catalog/Purview, RBAC, encryption, and secure workspace setups.

Collaborate with architects, data scientists, and business teams to deliver reliable and scalable data solutions.

Drive DevOps automation using Azure DevOps, CI/CD, and IaC (Terraform/ARM/Bicep).

Provide technical guidance, code reviews, and mentorship to junior engineers.

Mandatory Skills

Strong hands-on experience with Azure Databricks, PySpark, Delta Lake, Structured Streaming.

Expertise in Azure data services: ADF, Synapse, ADLS Gen2, Event Hubs/Kafka.

Solid understanding of distributed computing and Lakehouse architecture.

Strong SQL/NoSQL and data modeling knowledge (Dimensional, Data Vault, Lakehouse).

Experience with Spark performance tuning and large-scale data pipeline optimization.

Familiarity with governance tools (Unity Catalog/Purview).

Good experience with CI/CD, DevOps, and IaC.

Preferred Skills

Experience in banking/financial services (fraud, risk, payments, regulatory reporting).

Exposure to Databricks on multi-cloud (AWS/GCP).

Understanding of event-driven architectures and microservices.

Experience supporting ML pipelines with data scientists.

Skills

azure data factory,azure databricks,pyspark,delta lake,