Job Requirements
Role Overview
We are seeking a Lead Software Engineer – Automation Testing with 6–8 years of experience in designing, developing, and executing automation frameworks for AI/ML-driven applications. The ideal candidate brings hands-on experience with GenAI Testing, LangSmith, PyTest, Agentic frameworks, and strong AI/ML evaluation expertise.
Key Responsibilities
AI / GenAI Testing
- Design and execute V&V strategies for classical ML and GenAI/ agentic systems.
- Define test plans, tests cases, and execute them
- Evaluate AI models for accuracy, performance, safety, robustness, hallucinations, bias, and correctness.
- For GenAI, rely on LangSmith for LLM observability, tracing, monitoring, and evaluation, for offline and oline evaluation
- For classical ML UC, ensure V&V tests are done during the whole life cycle of the model (ML monitoring)
Test Automation
- Build scalable automation frameworks using PyTest integrated with CI/CD pipelines.
- Develop automated test suites for APIs, microservices, and AI pipelines.
- Perform sanity, integration, functional, regression, and UAT testing across QA, UAT, and Prod environments.
- Implement automation strategies: Keyword-driven, Data-driven, BDD, Hybrid frameworks.
Quality Assurance & Reporting
- Analyze test results and provide insights to engineering teams.
- Log and track defects using standard bug-reporting tools.
- Maintain high-quality documentation for all test processes and reports.
Required Skills & Experience
- 6–8 years of experience in software testing & automation.
- Strong programming expertise in Python.
- Good understanding of Data Science related metrics (for both classical ML and GenAI UC)Experience with AI/Chatbot products and testing AI applications.
- Experience with Git, GitHub Copilot, and CI/CD pipelines.
- Knowledge of Azure/AWS cloud environments.
- Familiarity with MySQL, PostgreSQL, Oracle, SQL Server, NoSQL databases.
- Strong verbal & written communication skills.
- Experience with multi-stakeholder delivery models including onshore–offshore.
- Qualification: B.E./B.Tech
Preferred / Nice-to-Have
- Experience in AI/ML model evaluation.
- Exposure to AI observability tools and automated prompt validation.
- Hands-on expertise with LangSmith, Agentic frameworks, PyTest, BDD/V&V frameworks.
Soft Skills
- Analytical & Critical Thinking
- Curiosity & Continuous Learning
- Strategic Communication(Active Listening, Technical Translations to business impacts)
Empowering tech talents to make better career decisions
Sign In
Companies
Jobs
Interviews
Salaries
Reviews
Blog
Resume Builder
Resume Checker
AI InterviewPrep
One-Click Job Apply
Check All Tools
Add Salary
Write Review
For Employer
