Design comprehensive test cases, validate AI evaluation benchmarks, and ensure the quality and reliability of frontier AI model assessments in a full-time remote W-2 role.
Software Engineering Expert
United States
$60 – $90/hr
July 25, 2026
Job description
Role Overview
Cincinnatus LLC is seeking experienced Software Engineering Experts to help develop next-generation evaluation benchmarks for frontier AI models. In this fully remote W-2 role, you will design realistic, multi-step software engineering challenges, build reference solutions, and evaluate how advanced AI coding agents perform on complex engineering tasks.
Working closely with researchers at a leading AI lab, you will help define the next generation of AI evaluation benchmarks by creating authentic engineering problems that require real-world software development, debugging, environment setup, and technical reasoning.
Key Responsibilities
Design Engineering Tasks
- Create realistic, multi-step software engineering challenges based on real-world engineering workflows.
- Develop tasks involving Python implementation, debugging, environment setup, tooling, and documentation.
- Design evaluation benchmarks that challenge state-of-the-art AI coding agents.
Build Reference Solutions
- Implement complete Python reference solutions for authored tasks.
- Develop setup instructions, validation checks, and expected outputs.
- Ensure each benchmark has clear, verifiable success criteria.
Evaluate AI Performance
- Test AI coding assistants on authored engineering tasks.
- Analyze where AI systems succeed and where they fail.
- Provide detailed feedback to researchers on AI reasoning and implementation errors.
Collaborate with Research Teams
- Review tasks created by fellow software engineering experts.
- Improve benchmark quality through peer feedback and refinement.
- Work closely with AI researchers to strengthen evaluation methodologies.
Required Qualifications
- Master's degree or PhD in:
- Computer Science
- Another STEM discipline
- Or equivalent practical experience in research-intensive software engineering
- At least 1 year of experience in:
- Software Engineering
- Research Engineering
- Research roles involving significant coding and data analysis
- Strong hands-on experience with:
- Python
- Debugging
- Clean code practices
- Proficiency with:
- Git
- Integrated Development Environments (IDEs)
- Standard software development workflows
- Excellent written communication skills.
- High attention to detail and ability to work independently.
- Available to work approximately 35 hours per week.
Preferred Qualifications
- Experience with:
- AI coding assistants
- Prompt engineering
- AI agent workflows
- Experience in:
- AI training
- Model evaluation
- Benchmark creation
- Task authoring
- Strong problem-solving skills in ambiguous, open-ended engineering environments.
Compensation
- $60–$90 per hour
- Full-time W-2 employment
Work Arrangement
- Fully Remote (United States)
- W-2 Contingent Role
- Approximately 35 hours per week
- Opportunity to work with a leading AI laboratory through Cincinnatus LLC
Apply now
You will be redirected to the company's website to complete your application.
Role Title: Workflow Annotator — Software Engineering Role Type: Contractor Location: Remote micro1 is engaging Workflow Annotator — Software Engineering experts to contribute to a dynamic customer project focused on building a library of realistic, multi-step engineering…
Job Title: Senior Software Engineer Job Type: Contractor (~15 hrs a week) Location: Remote Job Summary: In this role, you'll apply your expertise to help train next-generation AI systems.
Please submit your CV in English and indicate your level of English proficiency. Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems.