Data Scientist and LLM Operations Specialist

Role : : Data Scientist and LLM Operations Specialist

Location : : Remote / india

Duration : : Contract

Position Overview : We are seeking a highly skilled Data Scientist and LLMOps Resource to play a pivotal role in the development, expansion, operation, and maintenance of generative AI solutions. The primary responsibilities include running an experimental framework to determine the optimal prompt engineering approaches, tuning prompts, and collaborating with subject matter experts (SMEs) for evaluations and results. This role requires a deep understanding of evaluating models output in production, particularly when ground metrics are absent, monitoring for issues such as model drift and hallucinations, and optimizing for offline and online metrics.

Key Responsibilities :

• Scope, develop, expand, operate, and maintain scalable, reliable and safe generative AI solutions.
• Design and execute prompt engineering experiments to optimize Large Language Models (LLMs) for various use cases.
• Collaborate with SMEs to evaluate prompt effectiveness and align AI solutions with business needs.
• Understand and apply offline and online evaluation metrics for LLMs, ensuring continuous model improvements.
• Evaluate production models using live data in the absence of ground metrics, implementing robust monitoring systems.
• Monitor LLM applications for model drift, hallucinations, and performance degradation.
• Ensure smooth integration of LLMs into existing workflows, providing real-time insights and predictive analytics.

Qualifications : For Data Scientist :

For AI / ML Engineer :

Preferred Skills :