AI TEST LEAD - Gen AI

Job Description

Required Skills & Qualifications:

?       This role requires deep awareness of quality challenges unique to LLM- and SLM-powered Agentic AI applications, especially in healthcare and education, where correctness, reliability, and compliance are essential.

• 7+ years in Quality Engineering/Automation, with 3 years in QA leadership roles.
• Proven experience transforming teams from manual QA to automation-first.
• Awareness of LLM/SLM quality challenges (latency unpredictability, token inefficiency, hallucinations, SME UAT cycles).
• Strong automation expertise (Playwright, PyTest, Cypress, JUnit, REST API testing).
• Familiarity with Agentic AI frameworks (LangChain, LangGraph, RAG pipelines, Vector DBs).
• Experience in healthcare or education applications with regulatory constraints.
• Solid background in CI/CD, DevOps, and cloud-native systems (Azure, Kubernetes, GitHub Actions).

Nice to Have (Big Plus)

• Experience with Playwright MCP (multi-context automation) for scaling automation.
• Hands-on with AI evaluation tools (Promptfoo, DeepEval, OpenAI Evals).
• Familiarity with AI observability & monitoring (Datadog).
• Background in AI security testing (prompt injection, adversarial robustness).