Job Description

About Position:

We are looking for Sr/ Lead Data Engineer who will design, execute, and maintain advanced data pipeline architectures, supporting key analytics tools. Responsibilities include managing technical documentation, validating data, developing with Delta Live Tables, orchestrating workflows, and dynamically adapting to changing project requirements.

Role: Sr. / Lead Data Engineer
Location: Any Persistent Location
Experience: 5 to 15 Years
Job Type: Full Time Employment

What you will do:

Develop and maintain ETL pipelines primarily using AWS Databricks, excluding Azure Data Factory or Azure Synapse.
Managing data through the Medallion Architecture, including bronze, silver, and gold layers, preferably with Delta Live Tables API.
Expertly utilize Delta Live Tables API in production environments.
Debug Spark logs and manage Databricks workflow errors efficiently.
Commit regularly in Git, respond promptly to comments, and utilize branches effectively.
Implement Databricks Asset Bundles for streamlined CI/CD releases.
Orchestrate data pipelines and manage jobs with Databricks Workflows.
Develop and optimize pipelines with Apache Spark, demonstrating proficiency in PySpark and Spark SQL.
Translate complex business requirements into detailed technical specifications.
Implement data validation with the python library, Pandera.
Create and manage Databricks SQL Alerts.
Manage Fivetran connectors to ensure reliable and efficient data integration from various data sources into the Data Lakehouse, including setup and monitoring.
Manage connections to Tableau Cloud, overseeing the configuration, maintenance, and support of data sources within Tableau.

Expertise You'll Bring:

Databricks Data Engineer Associate certification.
Databricks Data Engineer Professional certification.
1-3 years of experience in data engineering or a related field.

Benefits:

Competitive salary and benefits package
Culture focused on talent development with quarterly promotion cycles and company-sponsored higher education and certifications
Opportunity to work with cutting-edge technologies
Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
Annual health check-ups
Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Our company fosters a values-driven and people-centric work environment that enables our employees to:

Accelerate growth, both professionally and personally
Impact the world in powerful, positive ways, using the latest technologies
Enjoy collaborative innovation, with diversity and work-life wellbeing at the core
Unlock global opportunities to work and learn with the industry’s best

Let’s unleash your full potential at Persistent

“Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind.”