Lead Data Engineer

Skill: Lead Data Engineer

Experience: 10 to 12/13 years

Location: Hyderabad/Chennai

Job Description:

We are looking for a seasoned Data Engineer with strong expertise in GCP, Kafka, and PySpark, along with proven experience in leading teams or driving technical initiatives. The ideal candidate will play a key role in designing scalable data platforms, mentoring team members, and delivering high-quality data solutions.

Key Responsibilities:

• Lead the design, development, and deployment of scalable data pipelines and data platforms.
• Architect and implement real-time and batch data processing solutions using Kafka and PySpark.
• Work extensively with Google Cloud Platform (GCP) services for data engineering and analytics.
• Provide technical leadership, guidance, and mentorship to junior and mid-level engineers.
• Collaborate with cross-functional teams including Data Scientists, Analysts, and business stakeholders.
• Ensure high data quality, performance optimization, and system reliability.
• Drive best practices in coding, data architecture, and data governance.
• Participate in solution design discussions and contribute to long-term data strategy.
• Troubleshoot complex data issues and provide scalable solutions.

Mandatory Skills:

• Strong hands-on experience with Kafka (real-time data streaming)
• Expertise in Python (PySpark) for large-scale data processing
• Hands-on experience with Google Cloud Platform (GCP)
• Strong understanding of Big Data ecosystem and distributed systems

Leadership / Additional Skills:

• Experience in leading teams or handling end-to-end project delivery
• Strong problem-solving and decision-making skills
• Ability to interact with stakeholders and drive technical discussions

Good to Have:

• Experience with data orchestration tools (e.g., Airflow)
• Knowledge of BigQuery or other data warehousing tools
• Exposure to CI/CD and DevOps practices
• Experience with data modeling and architecture