Lead AI QA Engineer

LEAD AI QA ENGINEER

How you will make an impact :

- Design and build evaluation harnesses for agentic systems in Python golden datasets, LLM-as-judge graders, multi-turn regression suites and trace-based assertions. In addition, develop framework to verify generated AI output.

- Author automated test suites for prompts, tools, structured outputs (Pydantic / JSON schema), retrieval pipelines (ETL Experience) and end-to-end agent workflows.

- Validate guardrails around tool execution : auth scoping, input/output validation, PII and prompt-injection protections, and hallucination mitigation.

- Wire evaluations into CI using Dataiku Evaluations, GitHub Actions or Jenkins so every change is graded against quality, safety and cost SLOs before it ships.

- Build observability into testing by instrumenting traces with LangSmith, Langfuse, MLflow or OpenTelemetry and triaging production drift back into the eval harness.

- Own quality end-to-end define release criteria, run pre-prod and shadow tests, and partner with engineering to root-cause and fix regressions quickly.

- Partner with data engineers on Snowflake-backed retrieval testing patterns (Cortex Analyst and Cortex Search Services) and with platform teams on observability, security and cost.

- Help shape internal QA standards for AI & Data engineering as the stack evolves, contributing to design reviews and sharing knowledge across the India and U.S. teams.

- Participate in a collaborative DevOps environment, working closely with developers, AI engineers, Data Engineers, DBAs and product partners across environments.

In your first 90 days :

- By the end of your first 90 days, you will have stood up at least one production-grade evaluation harness golden dataset, LLM-as-judge graders and regression suite wired into CI for an internal agent.

- You will have automated trace-based assertions running against staging traffic, a clear quality scorecard for at least one shipped agent, and a clear opinion about what our next testing investment should be.

What you need to be successful :

- 3+ years of professional QA / SDET experience, with production experience automating tests for backend services or data pipelines.

- 1+ years of hands-on experience testing LLM or AI features in production: prompt regression, tool / function-call validation, structured outputs and RAG correctness.

- Working knowledge of evaluation frameworks such as RAGAS, DeepEval, LangSmith, Langfuse or comparable LLM-as-judge tooling.

- Strong Python and PyTest skills; solid SQL skills and comfort with at least one cloud platform (AWS, Azure or GCP).

- Fluency with Git, Docker, REST APIs and at least one CI tool (GitHub Actions, Jenkins, GitLab CI or CircleCI).

- Solid understanding of data security and responsible AI practices, particularly in PCI-compliant or regulated environments.

- Proven ability to work independently and within a team, managing priorities across concurrent projects and time zones.

- Strong written and verbal communication skills; able to work effectively with both technical and non-technical stakeholders.

- A bachelors degree is not required equivalent practical experience (including bootcamps, self-taught work, career changes or non-CS technical degrees) counts.

Bonus Skills :

- Hands-on experience with Dataiku DSS (Python / SQL recipes, scenarios, code environments, the dataiku and dataikuapi clients) or Dataiku Evaluations.

- Experience with Dataiku LLM Mesh, Knowledge Banks, Prompt Studio, or Visual / Code Agents.

- Experience with Snowflake, Snowpark, or Snowflake Cortex (Search, Analyst, Agents).

- Experience with red-teaming, prompt-injection testing or adversarial test generation for LLMs.

- Familiarity with multi-agent patterns: supervisor / router, subagent / handoff, reflection, human-in-the-loop.

- Experience with performance and load testing tools such as Locust, JMeter or k6.

- ISTQB, AI Testing or comparable QA certification.

- Experience in loyalty, martech, adtech or a comparable data-rich B2B domain.

(ref:hirist.tech)