AI Conversation Quality Analyst

About the Role

We are seeking a highly motivated and detail-oriented Quality Assurance (QA) Engineer to join our team. You will play a key role in ensuring the reliability, accuracy, and user experience of our Conversational AI systems. This role involves testing dialogue models, integrations with NLP / LLM-based backends, and end-to-end conversational workflows across multiple platforms (chat,voice, and integrations with enterprise systems).

Location : Bengaluru, India

Key Responsibilities

• Test Planning & Strategy

○ Develop and execute comprehensive test plans for conversational AI features, including NLP pipelines, dialogue flows, integrations, and APIs.

○ Design functional, regression, performance, and exploratory test cases specific to conversational interfaces.

○ Validate intent recognition, entity extraction, context handling, and multi-turn dialogue management.

○ Test conversational flows for accuracy, completeness, and user experience. Perform scenario-based and edge-case testing to capture unexpected user inputs.

○ Build automated test scripts for API endpoints, dialogue flows, and ML model outputs.

○ Work with frameworks like Postman, Cypress, PyTest, or custom NLP testing tools.

○ Contribute to CI / CD pipelines with automated regression tests.

○ Partner with Data Scientists, ML Engineers, and Product Managers to define acceptance criteria for AI-driven features.

○ Provide feedback on conversational design and model performance.

○ Collaborate with DevOps on load testing and monitoring production systems.

○ Ensure conversational AI products meet high standards for accuracy, robustness, fairness, and usability.

○ Help define and track KPIs (e.G., intent accuracy, response latency, conversation success rate).

Qualifications

frameworks (e.G., Rasa, Dialogflow, LLM APIs).

Nice-to-Haves