Agentic Coding Annotator - Online / Offline Tasks

About Turing

Turing is one of the world's fastest-growing AI companies, accelerating the advancement and deployment of powerful AI systems. Turing works with the world's leading AI labs to advance frontier model capabilities in thinking, reasoning, coding, agentic behavior, multimodality, multilinguality, STEM, and frontier knowledge, and leverages that work to build real-world AI systems for companies.

Role Overview

We are looking for strong, detail-oriented software practitioners to help evaluate and improve datasets for agentic coding models. This role involves working with realistic coding tasks in an agentic coding harness, reviewing model trajectories, verifying solutions, and producing high-quality annotations.

Depending on the assignment, the work may include:

• Online evaluations: manually interacting with blinded models on predefined tasks, then ranking and grading the resulting trajectories
• Offline evaluations: designing realistic coding tasks, calibrating them through user simulation, writing task-specific rubrics, and grading generated trajectories

This is not a basic annotation role. Candidates are expected to read and debug code, validate behavior, follow detailed process rules, and make consistent judgment calls across model runs.

What the day-to-day looks like

• Execute realistic coding tasks within the assigned agentic coding harness while maintaining model blindness and session independence
• Follow task instructions, milestones, planned interactions, and evaluation guardrails consistently across runs
• Verify model outputs by reading code, running commands, checking logs, and inspecting generated artifacts
• Perform targeted validation of outputs using tests, scripts, and manual checks
• Write clear, specific, evidence-based rationales for trajectory rankings and assessments
• Design multi-step, realistic coding tasks (offline work), including user intent and milestone structure
• Create and refine task-specific rubrics and binary evaluation criteria
• Review completed work for quality, completeness, consistency, and schema compliance
• Identify and escalate broken environments, unclear instructions, or process gaps with supporting evidence

Requirements

Software engineering fluency (mandatory)

• 5+ years of experience in software engineering, QA, developer tooling, data/ML engineering, or similar code-heavy roles
• Strong hands-on experience in at least 1-2 programming languages or ecosystems (e.g. Python, JavaScript/TypeScript, Rust, Java, C/C++, Bash/CLI, Haskell, Swift, SQL)
• Ability to read unfamiliar codebases, run and interpret tests and CLI tools, debug issues, reason about edge cases, and evaluate whether an implementation is functionally correct

Preferred (offline / senior candidates)

• Strong Docker skills and experience building/debugging reproducible environments
• Experience working in large, complex repositories (not just greenfield projects)
• Demonstrated originality and sound engineering judgment in defining technical problems
• Ability to design realistic, non-trivial tasks beyond tutorials or simple bug fixes

Perks of freelancing with Turing

• Work on cutting-edge AI projects with leading foundation model companies
• Collaborate on high-impact work at the frontier of LLM evaluation and reasoning
• Remote, flexible opportunities with global teams
• Competitive compensation based on experience and project scope

Offer details

• Commitment: 8 hours per day with a 4-hour overlap with PST
• Employment type: contractor position (does not include medical/paid leave)
• Duration: approximately 5 weeks, expected start within a week