AI System Evaluator

Job Role : AI Evaluation Specialist

iMerit seeks detail-oriented and analytically minded AI Evaluation Specialists to perform highly nuanced evaluations of AI system outputs across different modalities : text, image, video, and multimodal interactions.

Specialists will assess the accuracy, appropriateness, quality, clarity, and cultural alignment of model outputs against complex guidelines, ensuring that results align with project standards and real-world use cases.

Evaluators will assess output from large language models (LLMs), vision models (LVMs), and multimodal AI systems.

Responsibilities :

• Evaluate outputs generated by LLMs across multiple modalities.
• Assess quality against project-specific criteria such as correctness, coherence, completeness, style, cultural appropriateness, and safety.
• Identify subtle errors, hallucinations, or biases in AI responses.
• Apply domain expertise and logical reasoning to resolve ambiguous or unclear outputs.
• Provide detailed written feedback, tagging, and scoring of outputs to ensure consistency across the evaluation team.

Requirements :

Key Skills :

Data annotation, LLM evaluation, content moderation, AI / ML domains, data annotation tools, software platforms, language, multimodal communication.

We Offer :

A dynamic work environment, opportunities for professional growth, and a competitive compensation package.