Opportunity Snapshot Pay: $25-45/hour Time per task: ~1.5 hours per run (evidence and scoring included) Onboarding: Once you are presented to the hiring manager, onboarding typically takes 24–48 hours before you can start tasking.
AI Evaluation Rubric Reviewer
Job description
Client: Flash
The work
We are testing AI agents on realistic shopping and delivery tasks. You'll review AI-drafted tasks (prompt, grading rubric) along with model attempts, and decide whether each failure is fair:
- Was the information available?
- Does the rubric accurately grade what the prompt asked for?
- Would a valid answer have been marked wrong?
- Is the rubric structure correct per the guidelines?
Fix the criteria that aren't fair, flag the tasks that can't be saved, and write a short note on each call.
Must-haves
- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.
- Can read a long agent transcript and find where it went wrong.
- Precise writing, and you check the data instead of trusting the reference answer.
Who can apply
- Open to
- Albania, Algeria, Angola, Antigua and Barbuda, Argentina, Australia, Austria, Bahrain, Bangladesh, Belgium, Benin, Bolivia and 95 more
- Posted
- November 16, 2026 by Mercor
- Last checked
- November 17, 2026: still open on work.mercor.com, where you apply
Opens work.mercor.com in a new tab. If they ask how you found the role, mention RemoWork.