Talent.com
Mindrift
AI Evaluation Engineer (Python, QA or Security)Mindrift • Hyderabad, TS, IN
AI Evaluation Engineer (Python, QA or Security)

AI Evaluation Engineer (Python, QA or Security)

Mindrift • Hyderabad, TS, IN
5 days ago
Job type
  • Remote
  • Quick Apply
Job description

Please submit your CV in English and indicate your level of English proficiency.

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

What this opportunity involves We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.

You'll create challenging tasks and evaluation criteria within realistic simulated environments:

  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT

  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate

What we look for

  • 5+ years in software development
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
  • Experience writing tests (functional, integration)
  • English proficiency - B2+

Why this is hard

Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

How it works

Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid

Compensation

Up to $30/hr equivalent, depending on level and pace. Tasks are estimated at ~20 hours each; you set your own schedule.

Create a job alert for this search

AI Evaluation Engineer (Python, QA or Security) • Hyderabad, TS, IN

Similar jobs

Firmware Engineer

TechBiz Global GmbHHyderabad, TS, IN

At TechBiz Global, we are providing recruitment service to our TOP clients from our portfolio.If you're looking for an exciting opportunity to grow in a innovative environment, this could be the pe... Show more

Fullstack developer (AIassisted development)

TechBiz Global GmbHHyderabad, TS, IN

At TechBiz Global, we are providing recruitment service to our TOP clients from our portfolio.If you're looking for an exciting opportunity to grow in a innovative environment, this could be the pe... Show more

Rotating Engineer – Offshore Reliability

Hudson ManpowerHyderabad, TS, IN

Rotating Engineer – Offshore Reliability.Bachelor’s Degree in Mechanical Engineering.Oil & Gas / Refinery (Offshore).The Rotating Engineer – Offshore Reliability will be responsible for ensurin... Show more

Senior Verification Engineer

TechBiz Global GmbHHyderabad, TS, IN

At TechBiz Global, we are providing recruitment service to our TOP clients from our portfolio.If you're looking for an exciting opportunity to grow in a innovative environment, this could be the pe... Show more

Senior Software Engineer NodeJS

KC Overseas EducationHyderabad, TS, IN

AI-powered study abroad platform built by KC Overseas Education — one of India's largest and most established overseas education groups.The platform serves students, counsellors, and partner univer... Show more

Mechanical Engineer – Onshore Reliability

Hudson ManpowerHyderabad, TS, IN

Mechanical Engineer – Onshore Reliability.Bachelor’s Degree in Mechanical Engineering.Oil & Gas / Refinery (Onshore).The Mechanical Engineer – Onshore Reliability will be responsible for improv... Show more

AI Tools & Automation Intern (Developer)

Hudson ManpowerHyderabad, TS, IN

Job Role: AI Developer Intern (LLM + MCP + AI Trends).We are hiring an AI Developer Intern who can both:.Build AI systems (LLMs, MCP servers, APIs).Continuously track and evaluate new AI tools &amp... Show more

DFT & PostSilicon Validation

TechBiz Global GmbHHyderabad, TS, IN

TOP clients from our portfolio.We are currently looking for a.DFT & Post-Silicon Validation.Are you passionate about Design for Testability (DFT) for complex SoCs and SoC chiplets in package? W... Show more

HR and Recruitment Specialist (Americas Timezone)

RM Staffing B.V.Hyderabad, TS, IN

Reboot Monkey is a global datacenter services provider headquartered in Haarlem, Netherlands, operating 24 green-powered facilities across 6 continents.We deliver colocation, IP transit, smart hand... Show more

Rotating Engineer – Onshore Reliability

Hudson Manpowerhyderabad, TS, IN

Rotating Engineer – Onshore Reliability.Bachelor’s Degree in Mechanical Engineering.Oil & Gas / Refinery (Onshore).The Rotating Engineer – Onshore Reliability will be responsible for ensuring t... Show more

OCP Networking Architect

TechBiz Global GmbHHyderabad, TS, IN

At TechBiz Global, we are providing recruitment service to our TOP clients from our portfolio.We are currently looking for a highly proactive and detail-oriented.If you're looking for an exciting o... Show more

Technical Recruiter (Remote)

RM Staffing B.V.Hyderabad, TS, IN
Remote

Reboot Monkey is a global datacenter services provider headquartered in Haarlem, Netherlands, operating.We deliver colocation, IP transit, smart hands, remote hands, and managed datacenter services... Show more

IP Verification Engineer

TechBiz Global GmbHHyderabad, TS, IN

At TechBiz Global, we are providing recruitment service to our TOP clients from our portfolio.If you're looking for an exciting opportunity to grow in a innovative environment, this could be the pe... Show more

Infrastructure Verification Engineer

TechBiz Global GmbHHyderabad, TS, IN

At TechBiz Global, we are providing recruitment service to our TOP clients from our portfolio.We are currently seeking a Continuous Integration & Verification Infrastructure Engineer to join on... Show more

Design Engineer – SAS (Switchgear)

Hudson Manpowerhyderabad, TS, IN

Design Engineer – Substation Automation System (SAS).MV/LV switchgear, protection relays, and control systems.The candidate will play a key role in delivering reliable and high-quality automation s... Show more

Technical Project Manager Data Center (Remote)

RM Staffing B.V.Hyderabad, TS, IN
Remote

Clients need a single point of contact who actually understands hardware, not just a relationship manager who has to relay every technical question.Projects span hardware deployment, structured cab... Show more

Mechanical Design Engineer | Distribution Transformer | Saudi Arabia

Hudson Manpowerhyderabad, TS, IN

We are seeking an experienced Distribution Transformer Mechanical Design Engineer to design and develop transformers up to 5 MVA, 33 kV class.The candidate will be responsible for preparing detaile... Show more

Mechanical Engineer – Offshore Reliability

Hudson Manpowerhyderabad, TS, IN

Mechanical Engineer – Offshore Reliability.Bachelor’s Degree in Mechanical Engineering.Oil & Gas / Refinery (Offshore).The Mechanical Engineer – Offshore Reliability will be responsible for imp... Show more

Coherency Verification Engineer

TechBiz Global GmbHHyderabad, TS, IN

At TechBiz Global, we are providing recruitment service to our TOP clients from our portfolio.Coherency Verification Enginee.If you're looking for an exciting opportunity to grow in a innovative en... Show more

Senior Software Architect (100% Remote Worldwide)

Tether Operations LimitedHyderabad, TS, IN
Remote

Join Tether and Shape the Future of Digital Finance.At Tether, we’re not just building products, we’re pioneering a global financial revolution.Our cutting-edge solutions empower businesses—from ex... Show more