Principal Quality Assurance Engineer - Automation Testing

Responsibilities :

• Perform functional, performance, and security testing on cloud-native services deployed on Microsoft Azure.
• Design and implement automated test suites for APIs, service components, and AI pipelines.
• Automate the evaluation of AI system outputs to ensure accuracy, consistency, and safety of responses.
• Collaborate with developers and data scientists to establish service-level quality metrics and observability hooks.
• Validate services against AI regulatory frameworks and ensure traceability, fairness, and robustness in outcomes.
• Participate in threat modelling and security validation of exposed APIs and AI services.
• Provide feedback early in the lifecycle to reduce defects and improve design.
• Mentor junior testers, encourage continuous learning, and contribute to a culture of innovation.
• Familiarity with LLM evaluation techniques, output scoring, and validation frameworks.
• Understanding of key concepts such as prompt engineering, RAG, model orchestration, and hallucination detection.
• Experience in testing for accuracy, relevance, and consistency of AI model predictions / generations
• Defining Performance Metrics for AI services and testing for the same
• Awareness of AI safety, bias detection, and explainability techniques.
• Experience ensuring compliance with AI regulations and standards (e.g., NIST AI RMF, EU AI Act).
• Strong belief in ethical AI practices, transparency, and end-user trust.

Qualifications :

(ref : hirist.tech)