Senior Data Engineer (Databricks)

Role Overview

We are seeking an experienced Senior Data Engineer – Databricks to design, build, and deliver scalable lakehouse data platforms. The role involves end-to-end data architecture, governance, and implementation on the Databricks platform, working closely with clients, Databricks Professional Services, and cross-functional teams to deliver enterprise-grade data solutions.

Key Responsibilities

End-to-End Data Architecture

• Collaborate with Databricks Professional Services and customer stakeholders to design end-to-end data architectures on the Databricks platform.
• Design and implement data ingestion strategies from multiple source systems (batch and streaming).
• Architect scalable storage solutions leveraging Delta Lake and Lakehouse architecture.
• Develop efficient data processing frameworks using Apache Spark and Databricks workflows.
• Build data consumption layers for analytics, reporting, and machine learning use cases.
• Ensure seamless and reliable data flow across the entire data lifecycle.

Governance, Security & Compliance

• Implement enterprise data governance frameworks in collaboration with Databricks Professional Services.
• Design and enforce security policies and access controls within Databricks environments.
• Ensure compliance with regulatory requirements and industry standards.
• Implement data lineage, audit trails, monitoring, and observability solutions.
• Optimize platform performance through cluster configuration and query tuning.
• Establish data quality standards and validation processes.

Data Modelling & Design

• Define business-aligned data models supporting enterprise objectives.
• Design Delta Lake table structures optimized for performance and scalability.
• Create dimensional, normalized, and data vault models based on business requirements.
• Implement Medallion Architecture (Bronze, Silver, Gold layers).
• Develop schemas supporting analytical and operational workloads.
• Maintain documentation, data dictionaries, and metadata standards.

Technical Leadership & Collaboration

• Lead technical discovery sessions to understand business requirements.
• Provide technical leadership and best-practice guidance to engineering teams.
• Collaborate with Infrastructure, Applications, and Cybersecurity teams.
• Mentor junior engineers and architects on Databricks technologies.
• Present solution architectures and recommendations to senior stakeholders.

Solution Implementation & Delivery

• Oversee end-to-end implementation of Databricks solutions from design to deployment.
• Ensure solutions meet performance, scalability, and reliability requirements.
• Support proof-of-concepts and pilot implementations.
• Conduct architecture and design reviews to ensure alignment with enterprise standards.

Required Technical Skills

Core Skills

• Proven hands-on experience with Databricks platform (workspace administration, cluster management, workflows).
• Experience implementing Unity Catalog for enterprise data governance.
• Deep knowledge of Delta Lake (ACID transactions, time travel, optimization).
• Strong expertise in Apache Spark (Spark SQL, DataFrames, performance tuning).
• Advanced programming skills in Python, SQL, and/or Scala.
• Experience with cloud platforms: Azure, AWS, or GCP.

Additional Skills

• Data pipeline development (ETL/ELT).
• Data governance and metadata management.
• CI/CD practices for data platforms.
• Performance optimization and troubleshooting.
• Data security and compliance.
• Data quality and validation frameworks.

Experience Requirements

• 8–10+ years of experience in data engineering, analytics, or data architecture.
• 3–5+ years of hands-on Databricks experience.
• Proven experience implementing Unity Catalog in enterprise environments.
• Experience designing and implementing large-scale, complex data models.
• Experience working with Databricks Professional Services or partner ecosystem preferred.
• Exposure to industries such as Public Sector, Financial Services, or Healthcare is advantageous.

Preferred Certifications

• Databricks Certified Data Engineer Professional
• Databricks Certified Associate Developer for Apache Spark
• Cloud certifications (Azure Data Engineer / AWS Data Analytics / GCP Data Engineer)
• Relevant data management or analytics certifications

Soft Skills

• Excellent communication and presentation skills
• Strong analytical and problem-solving abilities
• Ability to collaborate with partners and cross-functional teams
• Client-focused mindset and relationship-building skills
• Self-motivated with the ability to manage multiple priorities
• Experience working in Agile environments