The Role As a Strategic Projects Lead (SPL), you will own the end-to-end delivery of coding data pipelines used to train frontier AI models.
Software Engineer – LLM Evaluation & Repository Validation
Remote (Global)
$50 – $100/hr
October 2, 2026
Job description
We are looking for software engineers to validate real-world repositories used to evaluate large language models. You will triage issues, set up reproducible environments, assess test quality, and check how well models fix real bugs.
Key Responsibilities
- Review and triage issues in open-source GitHub projects.
- Prepare repositories for evaluation, including containerisation and environment automation.
- Assess how thorough and well written the unit tests are.
- Edit and run code locally to see how LLMs perform on bug-fixing tasks.
- Help pick repositories and issues that are hard for LLMs.
- Write up findings clearly for researchers and reviewers.
- Follow project guidelines, take part in calibration and review sessions, and incorporate feedback to keep quality consistent.
Required Qualifications
- 3+ years of software engineering experience.
- Strong experience in at least one of Python, C++, C#, Go, Rust or Ruby.
- Proficiency with Git, Docker and basic pipeline setup.
- Ability to navigate complex code bases and run real-world projects locally.
- Fluent written English.
- Reliable, detail-oriented, and able to work independently and meet deadlines in a remote setting.
Preferred Qualifications
- Prior LLM research or evaluation projects.
- Experience building or testing developer tools or automation agents.
- Open-source contribution or evaluation experience.
Rights, Confidentiality and Data Handling
- Submit only work that you created yourself or that you have the right to share.
- Do not submit material covered by an employer, client or third-party confidentiality agreement, or material you have no right to share.
- Do not submit export-controlled, military or defence-related technical data.
- Remove credentials, secrets and personal data from anything you submit.
- Rex may ask for evidence of ownership or licence, may verify submissions, and may reject or remove any submission.
Legal and Compliance Notice
- All content you submit must be lawful, and you must have the right to share it.
- Do not submit confidential or proprietary material belonging to an employer, client or third party, or personal data without a lawful basis.
- Rex may verify identity, credentials, ownership and licensing; may reject or remove any submission; and may report unlawful submissions to the relevant authorities.
- Engagement and payment are subject to successful verification and to applicable sanctions, export-control, tax and employment laws.
- The contractor agreement will set out ownership and licence terms for the work you deliver (to be confirmed by Rex Legal).
Compensation
USD $50–100 per hour, adjusted by country of residence.
Apply now
Opens portal.rex.zone in a new tab. If they ask how you found the role, mention RemoWork.
## About the Project We build hard, verifiable command-line tasks used to evaluate frontier AI agents. AI agents are good at the textbook. They still fail where experience matters: a default nobody questions, a rule nobody wrote down, a fix that looks right and isn't.
Opportunity Snapshot Pay: $50–100/hour Time per task: ~15 hours per run Onboarding: As soon as you’re selected, Talent Ops will send you everything you need to get started You’ll need: clear open source contributions and a profile to showcase them, such as GitHub (preferably…
Opportunity Snapshot Pay: $50–100/hour Time per task: ~15 hours per run Onboarding: As soon as you’re selected, Talent Ops will send you everything you need to get started You’ll need: clear open source contributions and a profile to showcase them, such as GitHub (preferably…