AI QA Engineer

HIRING ALERT! AI QA ENGINEER !

Role: AI QA Engineer

Work Location: Navi Mumbai/ Bangalore

Experience: 3-6 Years

Preferred Immediate Joiner.

Applicants please share your updated resumes on [HIDDEN TEXT]

Role Overview

We are looking for an AI QA Engineer with hands-on experience working on GenAI platforms and workflows, particularly AI observability, evaluation, testing, or safety platforms similar in nature to Arize, Cekura, Fiddler, Giskard, WhyLabs, Phoenix, or equivalent systems.

This role requires direct exposure to GenAI bots, RAG pipelines, and AI workflows in real platforms, not theoretical knowledge or generic QA experience. You will evaluate AI behavior, analyze failures, and contribute to improving the quality, reliability, and safety of GenAI systems.

This is not a traditional QA role and not suitable for candidates who have only tested UI, APIs, or deterministic systems.

Key Responsibilities

• Execute hands-on AI evaluation and QA on:
• GenAI bots (chatbots, copilots, agents)
• RAG-based workflows
• AI evaluation and observability platforms
• Design and execute:
• Prompt-based test scenarios
• Multi-turn conversation tests
• Edge-case and adversarial evaluations
• Evaluate AI behavior for:
• Hallucinations
• Grounding failures
• Inconsistency and drift
• Safety and reliability issues
• Work within existing AI platforms (evaluation, observability, monitoring) to:
• Analyze traces, metrics, and outputs
• Interpret model behavior across runs
• Document findings clearly in:
• Evaluation reports
• Platform dashboards
• Structured test summaries
• Collaborate with:
• AI QA Lead on methodology and review
• ML and Product teams to provide actionable feedback

Must-Have Skills & Experience

• 3–5+ years of experience in AI QA, AI evaluation, or GenAI testing
• Hands-on experience working on GenAI platforms, such as:
• AI evaluation platforms
• AI observability or monitoring tools
• GenAI safety or reliability platforms
• Direct experience testing:
• GenAI chatbots or agents
• RAG pipelines and workflows
• Multi-step AI interactions
• Strong understanding of:
• Prompt-based testing
• Non-deterministic AI behavior
• Quality vs correctness in GenAI systems
• Ability to clearly explain why an AI response is good or bad

Good-to-Have

• Experience with platforms like Arize, Cekura, Fiddler, Giskard, Phoenix, WhyLabs, or similar
• Familiarity with evaluation metrics and scoring approaches
• Exposure to AI safety, adversarial testing, or red-teaming
• Basic scripting or data analysis skills

Skills Required
AI evaluation platforms, GenAI platforms, Multi-step AI interactions, Non-deterministic AI behavior, Prompt-based testing, Quality vs correctness in GenAI systems, RAG pipelines