Senior Data Engineer – Databricks

Job Summary

We are looking for an experienced Senior Data Engineer – Databricks to design, develop, and maintain scalable data pipelines on the Databricks platform. The role requires strong expertise in PySpark, Databricks, and modern data engineering practices, along with experience modernizing legacy data pipelines and building high-performance data solutions in cloud environments.

Key Responsibilities

• Design, build, and maintain scalable data pipelines using Databricks and PySpark.
• Develop end-to-end data workflows including data ingestion, transformation, and consumption.
• Optimize Spark jobs, cluster configurations, and pipeline performance.
• Manage and orchestrate workflows using Databricks Jobs and notebooks.
• Refactor legacy ETL pipelines into modern PySpark-based ELT frameworks.
• Implement data quality checks, monitoring, and error handling mechanisms.
• Design and maintain Delta Lake tables with proper optimization strategies.
• Collaborate with data architects, analysts, and infrastructure teams to deliver reliable data solutions.
• Troubleshoot and resolve production data pipeline issues.

Mandatory Technical Skills

• Strong Data Engineering fundamentals including ETL/ELT pipeline design.
• Hands-on experience with PySpark (DataFrames API, Spark SQL, performance tuning).
• Experience with Databricks platform including workspace, clusters, notebooks, and job orchestration.
• Knowledge of Delta Lake features such as ACID transactions and schema evolution.
• Strong Python programming for data processing and automation.
• Experience with data modelling (Dimensional modeling, Data Vault, or Lakehouse architecture).
• Strong SQL skills for data transformation and analysis.
• Experience with cloud platforms (Azure, AWS, or GCP).
• Experience with Git version control and CI/CD practices.

Professional Experience

• Minimum 8 years of experience in data engineering or related roles.
• 2–3 years of hands-on experience with Databricks.
• Experience building and maintaining production-grade data pipelines at scale.
• Proven experience modernizing legacy data pipelines.
• Experience working in Agile development environments.

Certifications (Mandatory)

• Databricks Certified Data Engineer Associate
• OR
• Databricks Certified Data Engineer Professional

Preferred Skills

• Knowledge of Spark Structured Streaming.
• Understanding of data governance and data quality frameworks.
• Cloud certifications such as Azure Data Engineer, AWS Data Analytics, or GCP Data Engineer.