Senior AI / LLM Automation QA Engineer

About Company :

• They balance innovation with an open, friendly culture and the backing of a long-established parent company, known for its ethical reputation. We guide customers from what’s now to what’s next by unlocking the value of their data and applications to solve their digital challenges, achieving outcomes that benefit both business and society.

About Client :

Job Summary

Seeking an experienced Automation QA Engineer with strong expertise in evaluating and optimizing Large Language Model (LLM) responses. The role involves building automated evaluation workflows using Ragas , LangSmith , and Python-based frameworks to assess accuracy, relevancy, and faithfulness of model outputs. The ideal candidate has hands-on skills in prompt engineering , understands how to tweak and refine prompts for improved response quality, and contributes to LLM optimization techniques including token efficiency and prompt restructuring. The role also involves conducting performance testing of LLMs by measuring metrics such as Inter-Token Latency , First Token Time, and throughput. Strong knowledge of Python, basic SQL, and automation principles is essential to ensure high-quality, reliable, and efficient GenAI solutions.

Roles & Responsibilities