PR & Communications Expert
Remote17 countries onlycontractAdvanced / Expert
Mercor opens its Explore page; search for "AI Evaluation Rubric Auditor" to find this role.
The work We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair: • Was the information available? • Does the rubric accurately grade what the prompt asked for? • Would a valid answer have been marked wrong? • Is the rubric structure correct per the guidelines? This role includes providing feedback to reviewers regarding prompt and rubric issues. Must-haves • Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application. • Experience providing feedback to writers/reviewers • Consistent availability • Can read a long agent transcript and find where it went wrong. • Precise writing, and you check the data instead of trusting the reference answer.
Recommended AI training and data annotation positions
Remote17 countries onlycontractAdvanced / Expert
Remote17 countries onlyContract / FreelanceAdvanced / Expert
RemoteContract / Freelance