Senior Software Engineer – Python (LLM Evaluation & Repository Validation)

About Turing

Turing is one of the world's fastest-growing AI companies, accelerating the advancement and deployment of powerful AI systems. Founded in 2018, Turing partners with leading AI labs and enterprises to improve frontier AI models through high-quality data, expert human feedback, and rigorous evaluation frameworks. Contributors work on cutting-edge projects spanning coding, reasoning, agentic workflows, software engineering, and multimodal AI.

About the Role

We are looking for experienced software engineers with a strong background in developing and operating large-scale production systems. The ideal candidate has worked extensively on enterprise software, understands how production systems fail, has experience identifying security vulnerabilities in code, and has successfully implemented secure fixes and mitigations. This role requires practical experience in debugging complex production issues, participating in incident response, and building resilient, secure software.

What You'll Do

• Design and build realistic coding-agent benchmark tasks using production-like repositories, tests, configs, documentation, and runtime scenarios
• Create benign engineering tasks such as bug fixes, feature additions, CI repairs, config migrations, integration updates, and runtime-state fixes
• Define clear utility requirements to verify that the coding agent completes the requested task correctly
• Define safety and alignment requirements so the agent preserves system constraints, developer intent, data integrity, privacy, permissions, and oversight mechanisms
• Develop visible and hidden test suites to evaluate both task completion and unsafe agent behaviour
• Create safe reference solutions, and unsafe reference solutions where the agent passes utility checks but violates safety and alignment requirements
• Identify and evaluate unsafe shortcuts such as disabling tests, weakening assertions, deleting protected data, leaking secrets, broadening permissions, or bypassing validation
• Package tasks with prompts, metadata, runnable repos, evaluators, reference patches, scoring rubrics, and calibration notes
• Analyse rollouts from frontier coding agents such as Claude Code and Codex to assess utility completion and safety/alignment violations
• Collaborate with engineering, QA, security, and client teams to improve task quality, evaluator reliability, and benchmark difficulty

Required Qualifications

• Bachelor's or Master's degree in Computer Science or a related technical discipline
• Minimum 8 years of hands-on software engineering experience in leading product companies or technology startups
• Strong programming experience in Python (mandatory) plus one or more of Java, Go, or C++
• Demonstrated experience building and maintaining production software used by real customers
• Strong understanding of software architecture, debugging, and performance optimisation

Required Technical Experience

Production engineering

• Experience developing software deployed in production at scale
• Strong understanding of software deployment pipelines, monitoring, logging, and observability
• Experience diagnosing production failures using logs, metrics, and distributed tracing
• Knowledge of high availability, fault tolerance, scalability, and disaster recovery principles

Secure software development

• Experience identifying security vulnerabilities through code reviews or security assessments
• Practical knowledge of common software vulnerabilities, including injection attacks, authentication and authorisation flaws, memory safety issues, race conditions, deserialisation vulnerabilities, secrets management issues, and input validation failures
• Experience implementing secure code fixes and validating remediation

Perks of Freelancing with Turing

• Work in a fully remote environment
• Opportunity to work on cutting-edge AI projects with leading LLM companies

Offer Details

• Commitment: at least 4 hours per day, minimum 20 hours per week, with a 4-hour overlap with PST
• Engagement type: contractor assignment (no medical or paid leave)
• Duration: 4-8 weeks, expected start date next week

Evaluation Process

• Delivery review of candidate profile, followed by a 45-60 minute delivery interview