**Based in San Francisco, California, Turing is the world’s leading research accelerator for frontier AI labs and a trusted partner for global enterprises deploying advanced AI systems.
SWE-Bench Task Auditor
United States
$70 – $90/hr
August 28, 2026
Job description
Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.
Basic Qualifications • 3+ years professional software engineering • Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles) • Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking • Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)
Preferred Qualifications • Familiarity with SWE-Bench (Verified) or similar repository benchmarks • Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.) • Prior code-review or task-grading experience
Apply now
Opens work.mercor.com in a new tab. If they ask how you found the role, mention RemoWork.
**Based in San Francisco, California, Turing is the world’s leading research accelerator for frontier AI labs and a trusted partner for global enterprises deploying advanced AI systems.
Pay: 50–100/hour Role Type: Contractor Location: Anywhere in the world, remote Estimated Time Commitment: Approximately 15 hours per task run Start Timeline: Within 24–48 hours of onboarding We are seeking experienced Software Engineers and Open Source Contributors to help…
Location: Remote – North America preferred Preferred Geographies: United States (approved regions) and Canada Start Date: Immediate Experience Required: 5+ years Compensation: $100–$150 per hour Availability: Minimum 4 hours per day and 20 hours per week, with at least 4 hours…