LLM Evaluation and Annotation Specialist

Innodata (NASDAQ : INOD) is a leading data engineering company. With more than 2,000 customers and operations in 13 cities around the world, we are an AI technology solutions provider-of-choice for 4 out of 5 of the world’s biggest technology companies, as well as leading companies across financial services, insurance, technology, law, and medicine. By combining advanced machine learning and artificial intelligence (ML / AI) technologies, a global workforce of subject matter experts, and a high-security infrastructure, we’re helping usher in the promise of AI. Innodata offers a powerful combination of both digital data solutions and easy-to-use, high-quality platforms. Our global workforce includes over 5,000 employees in the United States, Canada, United Kingdom, the Philippines, India, Sri Lanka, Israel and Germany.

Job Title : Rater / AI Trainer (LLM Training )

Location : Remote

About the Role :

AI development team as Raters / AI Trainers. In this role, you will use your deep technical knowledge to evaluate, annotate, and provide structured feedback on AI-generated content produced by Large Language Models (LLMs). Your expertise will bring a level of precision, technical depth, and computational reasoning that is essential for training advanced AI. This role allows you to bridge your academic expertise with cutting-edge LLM development—helping to create AI that understands, explains, and reasons about code like a human expert.

Key Responsibilities :

• Evaluate LLM outputs for correctness, coherence, and relevance in technical areas such as programming, algorithms, data science, software engineering, systems design, and theoretical computer science.
• Assess and annotate AI responses for logical consistency, technical accuracy, code quality, computational feasibility, and clarity.
• Develop and test prompts that challenge the model’s understanding of programming concepts, debugging scenarios, and algorithmic logic.
• Provide structured, technical feedback on AI-generated answers, including identifying subtle bugs, inefficient logic, or misleading explanations.
• Ensure annotation consistency, data integrity, and high-quality labelling across assigned rating tasks.

Eligibility Criteria :

Kindly share me your resume at oshakya@innodata.com

Best Regards,

Omkar Shakya