AI Bot Performance Analyst

We are looking for a detail-oriented and analytical Chat and Voice Bot QA Specialist to join our Conversational AI team. In this role, you'll be responsible for evaluating the performance and quality of AI-driven chat and voice assistants. The ideal candidate brings a high level of first-principles thinking to assess bot responses, understands the fundamentals of generative AI, and is passionate about delivering seamless user experiences. This is a non-technical role that doesn't require coding, but does require a foundational understanding of how conversational bots work, how workflows are structured, and how to methodically test and evaluate them.

Key Responsibilities

Evaluate AI Responses : Systematically test chat and voice bot outputs for coherence, relevance, tone, factual accuracy, and alignment with business objectives.

Test Workflows : Assess conversation flows and user journeys for edge cases, logical consistency, and usability across different platforms (web, mobile, IVR).

• First-Principles Problem Solving : Analyze and break down complex issues using first-principles thinking;

question assumptions and propose better alternatives.

Scenario Creation : Develop diverse test scenarios and real-world prompts to validate AI performance across varied contexts and user intents.

work closely with product, design, and engineering teams.

AI Alignment Audits : Ensure the AI responses align with user intent, brand tone, and ethical guidelines.

Collaborate on Evaluation Frameworks : Help define or refine qualitative and quantitative criteria for bot performance and user experience.

Requirements

Qualifications

2-4 years of experience working in chatbot / voice testing, or a related field.

Strong analytical skills and a problem-solving mindset grounded in first-principles thinking.

Prior experience in QA, user research, product evaluation, or content moderation is a plus.

Familiarity with generative AI technologies (like ChatGPT, Bard, or Claude) and LLMs.

Basic understanding of conversation design, chat / voice interfaces, and user journey mapping.

Ability to write clear, concise test documentation and reports.

Excellent communication skills and attention to detail.

Curious, user-centric mindset with a drive for continuous improvement.