Mercor opens its Explore page; search for "AI Evaluation Rubric Auditor" to find this role.

AI Evaluation Rubric Auditor
Apply NowReferral link · no cost to you
← Back to all jobs
Get new roles the moment they’re postedTelegramDiscord
MC
MercorDomain Experts

AI Evaluation Rubric Auditor

Remote$60–$85 per hourContract / Freelance

About the Role

The work We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair: • Was the information available? • Does the rubric accurately grade what the prompt asked for? • Would a valid answer have been marked wrong? • Is the rubric structure correct per the guidelines? This role includes providing feedback to reviewers regarding prompt and rubric issues. Must-haves • Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application. • Experience providing feedback to writers/reviewers • Consistent availability • Can read a long agent transcript and find where it went wrong. • Precise writing, and you check the data instead of trusting the reference answer.

Similar Opportunities

Recommended AI training and data annotation positions

View all jobs →
Up to $55per hour
SME Careers

Social Media Expert

Remote17 countries onlyContract / FreelanceAdvanced / Expert

2026-10-03
More Domain Expert Jobs