Role Description
We are looking for a highly skilled Data Engineer with strong expertise in SQL, PySpark, Python, and Pandas. The ideal candidate will also have proven team leadership capabilities, enabling them to guide and mentor a team while delivering scalable data solutions. Experience with Generative AI and prompt engineering is a strong plus.
Key Responsibilities
- Design, develop, and optimize scalable data pipelines and ETL processes.
- Work extensively with large datasets using SQL and PySpark.
- Write clean, efficient, and maintainable code in Python.
- Perform data manipulation and analysis using Pandas.
- Lead and mentor a team of data engineers, ensuring best practices and high-quality deliverables.
- Collaborate with cross-functional teams including data scientists, analysts, and business stakeholders.
- Ensure data quality, integrity, and governance across systems.
- Optimize data workflows for performance and scalability.
- Participate in architecture discussions and contribute to technical strategy.
- Stay updated with emerging technologies, especially in Generative AI.
Required Skills & Qualifications
- Strong proficiency in SQL for data querying and transformation.
- Hands-on experience with PySpark for big data processing.
- Advanced programming skills in Python.
- Expertise in Pandas for data analysis and manipulation.
- Proven experience in leading or managing a team.
- Strong problem-solving and analytical skills.
- Experience working with cloud platforms (AWS, Azure, or GCP).
- Excellent communication and collaboration skills.
Good to Have
- Experience with Generative AI concepts and applications.
- Knowledge of Prompt Engineering techniques.
- Familiarity with machine learning workflows and tools.
- Exposure to data warehousing
Skills
data engineering,spark,sql,python,