The quality of AI depends entirely on the quality of the problems used to test it. We're looking for researchers and domain experts with deep machine learning knowledge to design the evaluation challenges that define — and push — the limits of today's most capable AI systems.
ML Challenge Task Auditor
United States
$70 – $90/hr
August 28, 2026
Job description
Evaluate the quality, correctness, and methodological rigor of applied machine-learning tasks used to train and evaluate a frontier AI lab's models. You'll assess experiment design, model-selection reasoning, and evaluation methodology — and provide clear, rubric-based written feedback.
Basic Qualifications • 3+ years hands-on applied/experimental ML (experiment design, model selection, hyperparameter tuning, evaluation methodology) • Strong grasp of data-quality rigor: leakage detection, metric gaming, and train/test/CV hygiene • Proficiency with standard ML frameworks (PyTorch, TensorFlow, scikit-learn, XGBoost) • Ability to critique ML claims against evidence and reproduce results
Preferred Qualifications • Competition / benchmark experience (e.g., Kaggle) • Graduate research or publication record in applied ML • Prior task-grading or peer-review experience
Note: this role evaluates applied/experimental ML rigor — it is not an LLM-application-building or MLOps role.
Apply now
Opens work.mercor.com in a new tab. If they ask how you found the role, mention RemoWork.
To qualify you must work as an ML research engineer, research scientist or applied scientist, and have personally trained, fine-tuned or optimized models in PyTorch or JAX.
Mercor is looking for STEM researchers who hold or are currently pursuing an MSc or PhD to support a research project with a leading AI lab. You will work through real research workflows on your own Mac to help benchmark and improve cutting-edge AI models.
The Human Frontier Collective (HFC) brings researchers and scientists together to tackle consequential problems in AI. In partnership with Scale Labs, we are building a small cohort of researchers to work on the empirical foundations of AI safety.