Mercor opens its Explore page; search for "SWE-Bench Task Auditor" to find this role.

SWE-Bench Task Auditor
Apply NowReferral link · no cost to you
← Back to all jobs
Get new roles the moment they’re postedTelegramDiscord
MC
MercorDomain Experts

SWE-Bench Task Auditor

Remote · US only$70–$90 per hourContract / Freelance

About the Role

Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback. Basic Qualifications • 3+ years professional software engineering • Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles) • Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking • Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++) Preferred Qualifications • Familiarity with SWE-Bench (Verified) or similar repository benchmarks • Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.) • Prior code-review or task-grading experience


Eligibility

Mercor lists this role for residents of the United States.

Similar Opportunities

Recommended AI training and data annotation positions

View all jobs →
Up to $55per hour
SME Careers

Social Media Expert

Remote17 countries onlyContract / FreelanceAdvanced / Expert

2026-10-03
More Domain Expert Jobs