Senior Data Engineer

Job Description for Senior Data Engineer

Experience : 4years to 8years

Required Skills : Aws,Python,Pyspark,Databricks

Notice Period : Immediate to 15days

Databricks (Spark)

· Develop scalable ETL/ELT pipelines using PySpark (RDD/DataFrame APIs), Delta Lake, Auto Loader (cloudFiles), and Structured Streaming.

· Optimize jobs: partitioning, bucketing, Z-Ordering, OPTIMIZE + VACUUM, broadcast joins, AQE, checkpointing.

· Manage Unity Catalog: catalogs/schemas/tables, data lineage, permissions, secrets, tokens, and cluster policies.

· CI/CD for Databricks assets: notebooks, Jobs, Repos, MLflow artifacts.

· Build Medallion Architecture (Bronze/Silver/Gold) with Delta Live Tables (DLT) and expectations for data quality.

· Event-driven ingestion: Kafka/Kinesis → Databricks Streaming

Snowflake (DW & ELT)

· Model and implement star/snowflake schemas, data marts, and secure views.

· Performance tuning: clustering keys, micro-partitions, result caching, warehouse sizing, query profile analysis.

· Implement Task/Stream patterns for CDC; external tables for data lakes (S3); Snowpipe for near-real-time ingestion.

· Python/Snowpark for transformations and UDFs; SQL best practices (CTEs, window functions).

· Security: Row Level Security (RLS), Column Masking, OAuth/SCIM, network policies, data sharing (reader accounts).

AWS Data Engineering

· Storage & compute: S3 (lifecycle, encryption, partitioning), EMR (if needed), Lambda, Glue (ETL/Schema registry), Athena, Kinesis (Data Streams/Firehose), RDS/Aurora, Step Functions.

· Orchestration: MWAA/Airflow or Step Functions (error handling, retries, backfills, SLA alerts).

· Infra-as-code: Terraform/CloudFormation for reproducible environments (Databricks workspace, IAM, S3, networking).

· Security/compliance: IAM least privilege, KMS, VPC endpoints/private links, Secrets Manager, CloudTrail/CloudWatch, GuardDuty.

· Observability: CloudWatch metrics/logs, structured logging, datadog/Prometheus (optional), cost monitoring (tags/budgets).

Data Quality, Governance & Security

· Implement unit/integration tests for pipelines (e.g., pytest + Great Expectations + DLT expectations).

· Data contracts and schema evolution; monitor SLA/SLO; DQ dashboards (missingness, drift, freshness, completeness).

· PII handling: tokenization/pseudonymization, field-level encryption, KYB/KYC data flows adherence; audit trails.

· Cataloging & lineage through Unity Catalog and/or OpenLineage/Purview (if applicable).

DevOps & CI/CD

· Git workflows (branching, PR reviews), Databricks CLI/Terraform modules for jobs/clusters/UC, Snowflake DevOps (object versioning via schemachange or SQL-based migration).

· Automated testing in pipelines; feature flags, canary releases for data jobs; rollback strategies.

Client-Facing PoCs & Delivery

· Rapid PoC build: clearly defined success metrics, benchmark cost/performance, produce a transition plan to production.

· Present architectural decisions, trade-offs (Spark vs Snowflake ELT), and cost projections (Databricks DBU, Snowflake credits, storage egress).

· Produce runbooks, operational playbooks, and knowledge transfer documents for client teams.

Required Technical Skillset

· Databricks: PySpark, Delta Lake, Auto Loader, DLT, Jobs, Unity Catalog, MLflow basics.

· Snowflake: SQL, Snowpipe, Tasks/Streams, Snowpark (Python), warehouse sizing, performance tuning, security policies.

· Python: strong in packages for DE (pandas, pyarrow, pytest), robust error handling, typing, and packaging.

· Orchestration: Airflow DAGs (Sensors, Operators, XCom), Step Functions state machines.

· Streaming & CDC: Kafka/Kinesis, Debezium (nice-to-have), CDC patterns to Delta/Snowflake.

· AWS: S3, Glue, Lambda, Kinesis, IAM/KMS, VPC, CloudWatch; Terraform/CloudFormation.

· Data Modeling: 3NF/Dimensional, slowly changing dimensions (SCD Type 2), surrogate keys, surrogate vs natural debates.

· Security & Compliance: encryption at rest/in transit, tokenization, key rotation, audit logging, governance controls.

· Performance & Cost: Spark job tuning, Snowflake warehouse right-sizing, partitioning/clustering, object storage best practices.

Nice-to-Have:

· dbt (Snowflake) with tests & exposures; Great Expectations.

· Databricks SQL Warehouses and BI connectivity; Photon engine awareness.

· Lakehouse Federation (UC external locations); Delta Sharing; Iceberg experience.

· Kafka Connect/Debezium, NiFi or MuleSoft (for data integrations).

· Experience in financial services

· Exposure to ISO/IEC 27001 controls in data platforms.

Education & Certifications

· Bachelor’s/Master’s in CS/IT/EE or related.

· Certifications (plus): Databricks Data Engineer Associate/Professional, Snowflake SnowPro Core/Advanced, AWS Solutions Architect/Big Data/DP.