QA Engineer III

Overview

As a member of our Agentic AI Engineering team, we seek individuals passionate about advancing the quality and reliability of AI systems through rigorous evaluation, experimentation, and tooling. You will play a key role in assessing the performance, safety, and alignment of AI models, leveraging platforms like OpenAI and LangSmith. Our team thrives in a culture of curiosity, continuous learning, and collaborative innovation, where your contributions directly impact the trustworthiness of AI solutions used globally.

Responsibilities

• Design and execute evaluation strategies for LLM-based systems using LangSmith, OpenAI tools, and custom frameworks.
• Collaborate with product managers, researchers, and engineers to define evaluation metrics aligned with business and ethical goals.
• Develop automated pipelines for regression testing, prompt evaluation, and model comparison.
• Analyze model outputs for correctness, bias, safety, and hallucination using both qualitative and quantitative methods.
• Build reusable test harnesses and datasets for benchmarking AI capabilities across tasks and domains.
• Contribute to the development of internal tools for prompt engineering, evaluation visualization, and feedback loops.
• Act as a quality advocate for AI-driven features across the product lifecycle.
• Mentor junior team members in AI evaluation best practices and tooling.
• Stay current with advancements in LLM evaluation, interpretability, and responsible AI.

Qualifications

Bachelor's or master's degree in computer science or a related technical field.

Skills Required

Github, Jira, Sql, Karate, Pandas, Rest Assured, Protractor, Python