Modern Data Platform (MDP) - Databricks

Position Overview

The Modern Data Platform (MDP) Databricks Architect / Consultant / Developer is responsible for designing, building, and operating modern analytics and data platforms using Databricks Lakehouse Platform. The role enables organizations to deliver scalable, high-performance, and governed data solutions supporting analytics, BI, data engineering, data science, and AI/ML workloads.

Depending on experience and seniority, the role may focus on hands-on development and delivery (Developer / Consultant) or on platform architecture, standards, and data strategy (Architect). The role collaborates closely with business stakeholders, data teams, cloud architects, and security teams to ensure platforms are secure, cost-effective, and enterprise-ready.

Key Responsibilities

Data Platform Design & Delivery

· Design and implement modern data platforms using Databricks Lakehouse

· Build data ingestion and transformation pipelines using:

o Databricks Workflows

o Apache Spark (PySpark / Spark SQL)

o Delta Lake

· Support batch, streaming, and near-real-time data processing

· Enable analytics, BI, and advanced data science workloads

Architecture & Platform Standards (Architect Focus)

· Define Modern Data Platform architecture using Databricks

· Design scalable lakehouse architectures integrated with cloud data lakes

· Establish standards for data ingestion, modeling, quality, and governance

· Ensure alignment with cloud, security, identity, and cost-management standards

· Provide architectural oversight and design governance across teams

Data Engineering, Analytics & AI Enablement

· Develop optimized Spark-based data pipelines and transformations

· Design Delta Lake models and performance-optimized tables

· Support Power BI, Tableau, or other BI tools for data consumption

· Enable data science and ML use cases using Databricks ML and notebooks

Operations, Governance & Optimization

· Implement monitoring, performance tuning, and cluster optimization

· Apply data governance, access controls, and lineage practices

· Support CI/CD for notebooks, jobs, and data pipelines

· Contribute to platform documentation, standards, and operational runbooks

Collaboration & Advisory

· Collaborate with analysts, data scientists, and application teams

· Advise stakeholders on Databricks capabilities and design trade-offs

· Mentor junior engineers and consultants (Senior / Architect level)

Qualifications and Skills

Experience (Role Level Determination)

MDP Databricks Developer

· 4–6 years of overall data / analytics experience

· 2–4 years of hands-on Databricks and Spark experience

· Strong focus on development and implementation

MDP Databricks Consultant

· 6–9 years of data and analytics experience

· 4–6 years delivering modern data platforms using Databricks

· Strong delivery, design, and stakeholder engagement skills

MDP Databricks Architect

· 10–15+ years of data and analytics experience

· 7–10+ years designing enterprise-scale data platforms

· Proven experience defining lakehouse architectures and data strategies

Technical Skills

· Strong expertise in Databricks Lakehouse Platform

· Deep experience with Apache Spark, PySpark, and Spark SQL

· Knowledge of Delta Lake, data modeling, and performance tuning

· Experience with streaming data (Structured Streaming, Kafka)

· Understanding of data security, governance, and cost optimization

Preferred Qualifications (Key Certifications)

· Databricks Certified Data Engineer (Associate / Professional)

· Databricks Certified Data Architect (where applicable)

· Cloud platform certifications (Azure / AWS – desirable)