In this hourly, remote contractor role, you will work as a Malayalam Quality Assurance Lead (QAL) to oversee quality, consistency, and trainer performance across Malayalam AI training projects.
AI Evaluation Rubric Auditor
Job description
Client: Flash
The work
We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair:
- Was the information available?
- Does the rubric accurately grade what the prompt asked for?
- Would a valid answer have been marked wrong?
- Is the rubric structure correct per the guidelines?
This role includes providing feedback to reviewers regarding prompt and rubric issues.
Must-haves
- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.
- Experience providing feedback to writers/reviewers
- Consistent availability
- Can read a long agent transcript and find where it went wrong.
- Precise writing, and you check the data instead of trusting the reference answer.
Who can apply
- Open to
- Canada, United States, Argentina, Bolivia, Brazil, Chile, Colombia, Ecuador, Guyana, Paraguay, Peru, Uruguay and 1 more
- Posted
- November 16, 2026 by Mercor
- Last checked
- November 17, 2026: still open on work.mercor.com, where you apply
Opens work.mercor.com in a new tab. If they ask how you found the role, mention RemoWork.