AI Prompt Quality Rater
Apply NowReferral link · no cost to you
← Back to all jobs
Get new roles the moment they’re postedTelegramDiscord
CL
Claru.aiAI Training & Evaluation

AI Prompt Quality Rater

Remote$35–$55 per hourContract / Freelance

About the Role

You'll evaluate prompt-response pairs generated by large language models, scoring them across dimensions like factual accuracy, instruction-following, coherence, and appropriate refusal behavior. Each session involves reviewing batches of 20–50 pairs using a structured rubric inside a web-based annotation platform, flagging edge cases, and writing brief justifications for non-obvious scores. Beyond surface-level ratings, you'll identify subtle failure modes — responses that are technically accurate but misleading, answers that follow the letter of a prompt while violating its intent, and outputs that pass a quick read but contain embedded errors. You'll work asynchronously, with a typical batch taking 60–90 minutes, and you're expected to maintain inter-annotator agreement scores above 0.75 kappa. Feedback you submit feeds directly into preference datasets used to fine-tune and align production models. Calibration sessions run bi-weekly, where you'll review disagreements with a senior evaluator and update your scoring approach. Strong performance can lead to specialized tracks covering domain-specific evaluation (legal, medical, code). Skills: Prompt evaluation, Rubric-based scoring, LLM output analysis, Instruction-following assessment, Factual accuracy verification, Annotation platform proficiency.


Eligibility

Not open to residents of: Afghanistan, Belarus, Central African Republic, Côte d'Ivoire, Cuba, Democratic Republic of the Congo, Ghana, Haiti, Iran, Iraq, Libya, Mali, Myanmar (Burma), Nicaragua, Nigeria, North Korea, Pakistan, Russia, Somalia, South Sudan, Sudan, Syria, Venezuela, Yemen, Zimbabwe, and the regions of Crimea, Donetsk and Luhansk.

Similar Opportunities

Recommended AI training and data annotation positions

View all jobs →
Up to $15per hour
SME CareersFeatured

PowerPoint Specialist

RemoteBangladesh, India & Pakistan onlyContract / FreelanceAdvanced / Expert

1w ago
Up to $51per hour
SME Careers

Senior Video Editor

Remote28 countries onlyContract / FreelanceAdvanced / Expert

3w ago
More AI Training and Evaluation Jobs