Data Scientist - Python/LLM

Job Description :

- Design and implement end-to-end evaluation frameworks to assess performance, reliability, and safety of multi-agent AI systems

- Lead experimentation and A/B testing efforts to systematically test hypotheses, validate model improvements, and track performance across agent iterations

- Curate and maintain high-quality ground truth datasets to enable accurate, reproducible evaluation of multi-agent outputs

- Identify and address reliability and accuracy gaps across agent workflows, failure modes, and edge cases in production-like environments

- Stay current on emerging research in agentic AI, LLM evaluation, and multi-agent coordination to continuously improve framework design

Technical Skills :

- Proficiency in Python and ML frameworks

- Hands-on experience with LLM APIs and agentic frameworks (LangChain, LlamaIndex, Semetic KernalI)

- Familiarity with evaluation tooling (Ragas, DeepEval, LangSmith, or similar)

- Experience with data pipelines, experiment tracking (MLflow, W&B), and CI/CD for ML workflows

- Strong foundation in statistics, NLP, prompt engineering, experimental design, and A/B testing methodology

- Proficiency in Azure ML, Azure OpenAI Service, and Azure AI Foundry for model deployment, evaluation, and orchestration

- Familiarity with Azure Monitor and Application Insights for tracking reliability and performance of deployed agent systems

(ref:hirist.tech)