Customer AI Product Owner
AccentureAUAccenture is a leading global professional services company, providing a broad range of services in strategy and consulting, interactive, technology and operations, with digital capabilities across... Show more
AI Evaluation Engineer (Python, QA or Security)Mindrift • AU
AI Evaluation Engineer (Python, QA or Security)
Mindrift • AU
6 days ago
Job type
- Remote
- Quick Apply
Job description
Please submit your CV in English and indicate your level of English proficiency.
Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.
What this opportunity involves We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.
You'll create challenging tasks and evaluation criteria within realistic simulated environments:
- Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
- Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
- Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
- Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust
What this is NOT
- Not data labeling
- Not prompt engineering
- Not writing code from scratch - the agent writes most of the code; you guide and evaluate
What we look for
- 5+ years in software development
- Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
- Experience writing tests (functional, integration)
- English proficiency - B2+
Why this is hard
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.
How it works
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid
Compensation
Up to $50/hr equivalent, depending on level and pace. Tasks are estimated at ~20 hours each; you set your own schedule.
Create a job alert for this search
AI Evaluation Engineer (Python, QA or Security) • AU
Similar jobs
Senior AI Engineer (Sydney, Melbourne or Brisbane)
Automic GroupAUAutomic Group is Australia’s market leader in technology-driven investor administration.Combining proprietary cloud-native technology with expert service, Automic simplifies investor administration... Show more
AI Solutions Specialist | NLP, GenAI & LLM | AI Automation & Translation | 100% remote
White BayAURemote
AI Solutions Specialist | NLP, GenAI & LLM | AI Automation & Translation | 100% remote (Canadian-based) | $100,000 - $140,000.They are a recognised leader in multilingual technology and con... Show more
QA Engineer
CotalentAUA scaling tech company who is leveraging machine learning to streamline and automate workflows for its clients.To support its growth, the company is seeking a technically orientated QA to build qua... Show more
Manager, Forward-Deployed AI Engineer — AI Mobilization & Transformation
MasterCardAUMastercard powers economies and empowers people in 200+ countries and territories worldwide.Together with our customers, we’re helping build a sustainable economy where everyone can prosper.We supp... Show more
Analytics Engineer
Octopus DeployAUOctopus Deploy sets the standard for Continuous Delivery, empowering software teams to deliver value in an agile way.Over 4,000 organizations globally – including Ubisoft, Xero, Stack Overflow, NAS... Show more
R&D Associate Engineer – ANZ Foods
PepsiCoAUWork for a global FMCG company behind some of the world’s most trusted brands.We offer a supportive environment that recognises talent, encourages career growth, and empowers you to make a differen... Show more
SENIOR CONSULTANT CYBER & AI - SECURE AI GOVERNANCE FRAMEWORKS
CyberCXAUWork for the Secure AI Practice in CyberCX consulting and delivering on Secure AI Governance Framework projects.Work on Secure AI Governance Projects, Secure AI Strategies, AI Risk Assessments.You ... Show more
Senior Gemini Platform Engineer
AccentureAUFind out more about us at accenture.You will lead the technical architecture and delivery of enterprise-grade AI platforms.Your mission is to move clients from "Experimentation" to "Production" by ... Show more
*AWS Data Engineer*
AccentureAUCombine the best of your ingenuity and Cloud technologies to build a better world and manage clients towards change.Join us and do extraordinary things.We work with one shared purpose: to deliver o... Show more
SECURE AI ENGINEER
CyberCXAUSecure AI, AI-enabled applications, and modern cloud environments.This role is suited to someone who enjoys solving practical technical problems, working closely with clients and teammates, and bui... Show more
Founding Engineer (Staff Level) | Computer Vision and Sensors | Start up | 100% remote | $200,000 -
White BayAURemote
Founding Engineer (Staff Level) | Computer Vision and Sensors | Start up | 100% remote | $200,000 - $250,000 USD + Equity .They’re an early-stage, pre-seed startup developing advanced AI-driven tar... Show more
Product Owner - AI
Employment HeroAUEmployment Hero is on a mission to make employment easier and more valuable for everyone.Our Employment Operating System brings hiring, HR, payroll and benefits into an all-in-one solution.Since ou... Show more
AWS Security Engineer
AccentureAUAccenture is a global professional services company with leading capabilities in digital, cloud and security.This role focuses on AWS security engineering, responsible for configuring cloud-native ... Show more
Test and Evaluation Engineer
Clicks IT RecruitmentAUClicks is partnering with a global technology leader that develops and delivers advanced mining technology solutions used in some of the world's most challenging operational environments.Test &... Show more
Director, Forward-Deployed AI Engineer - AI Mobilization & Transformation
MasterCardAUMastercard powers economies and empowers people in 200+ countries and territories worldwide.Together with our customers, we’re helping build a sustainable economy where everyone can prosper.We supp... Show more
Senior Analytics Engineer
Employment HeroAUEmployment Hero is on a mission to make employment easier and more valuable for everyone.Our Employment Operating System brings hiring, HR, payroll and benefits into an all-in-one solution.Since ou... Show more
SENIOR MACHINE LEARNING ENGINEER
Opus Recruitment SolutionsAUSenior Machine Learning Engineer.Senior Machine Learning Engineer .Partnered with a leading AI software company who specialise in building facial analytics & computer vision solutions across Re... Show more
Software Engineer
AmakaAUThe developer participates in the design and development phase of integration projects.He/She will actively contribute to projects implementation and assist the Solutions Architect/Technical Leader... Show more
Advanced Analytics Lead
Virgin AustraliaAUWe’re the airline that dares to be different, bold, down-to-earth and built around giving our guests real choice.Safety is our backbone, service is our heartbeat, and our people are our superpower.... Show more