Evaluate frontier AI systems by testing complex real-world workflows using Model Context Protocol (MCP) and AI plugins/connectors, creating evaluation rubrics, and assessing AI performance across travel, productivity, health, and personal planning tasks.
AI Evaluation Specialist
Remote (Global)
$20 – $35/hr
May 30, 2026
Job description
As an AI Evaluation Specialist, you will help train and evaluate next-generation AI systems by designing evaluation tasks, creating grading rubrics, and assessing AI performance across real-world computer-based workflows.
This role focuses on structured thinking, precise written communication, detailed observation, and objective evaluation methodologies.
Key Responsibilities
Design and implement self-contained AI evaluation tasks, including:
- Prompts
- Supporting files
- Grading rubrics
Define clear and unambiguous evaluation criteria for successful and unsuccessful task completion
Assess AI performance across a variety of administrative and workflow scenarios
Observe and document AI agent behavior with:
- Accuracy
- Precision
- Consistency
Produce concise, high-quality reports and written observations
Refine evaluation tasks and scoring frameworks based on feedback and testing results
Adapt evaluation methodologies across different business and operational domains
Collaborate with customers and internal teams to improve AI evaluation processes
Promote rigorous documentation, structured observation, and objective assessment practices
Required Skills & Qualifications
Minimum 3 years of experience in roles requiring strong written precision and structured thinking, such as:
- Paralegal
- Executive Assistant
- Junior Analyst
- Librarian
- Document Archival Specialist
- Research Assistant
- Technical Writer
- QA Analyst
Native or fluent English writing ability
Experience producing observations that are:
- Clear
- Specific
- Unambiguous
Experience designing or applying:
- Rubric-based evaluations
- Structured scoring systems
- Assessment frameworks
Strong attention to detail and pattern recognition skills
Excellent written and verbal communication abilities
Proficiency with:
- SaaS applications
- Web browsers
- File management systems
- Document editing tools
Ability to independently manage ambiguous or evolving projects
Preferred Qualifications
Experience evaluating AI outputs
Experience contributing to technology-driven process improvement initiatives
Experience developing or refining:
- Evaluation rubrics
- Scoring methodologies
- Assessment frameworks
Ability to work across multiple domains and rapidly adapt to new workflow environments
About micro1
micro1 is a data engine that helps AI labs train foundation models and helps enterprises build AI agents.
The company provides:
- Frontier AI evaluations
- Reinforcement learning environments
- Enterprise AI agent evaluation systems
Its platform includes:
- An AI recruiter agent for sourcing and vetting experts
- A data platform for producing high-quality training data
- Pipeline performance systems focused on quality and scalability
Apply now
You will be redirected to the company's website to complete your application.
Please submit your CV in English and indicate your level of English proficiency. Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems.
Please submit your CV in English and indicate your level of English proficiency. Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems.
Please submit your CV in English and indicate your level of English proficiency. Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems.