AI Evaluation & Rating (RLHF)

If you want to get paid as an AI evaluator or AI rater (the RLHF and model evaluation work behind every chatbot), here is the honest headline: most people earn somewhere between $15 and $45 an hour for generalist tasks, credentialed specialists (coders, PhDs, MDs, lawyers) earn $75 to $200+ an hour, and the US national average for reinforcement learning from human feedback (RLHF) work sits around $40.70/hr according to ZipRecruiter. The work is real, it is remote, and the demand is large: OpenAI, Google, Meta, and Anthropic are each spending on the order of $1 billion a year on human-provided training data (HeroHunt). But it is freelance, feast-or-famine, and full of low-effort scams. This guide tells you exactly what the work is, what each platform actually pays, how to get in, and where the traps are.

RLHF and model evaluation is the most valuable and best-paid corner of the broader "get paid to train AI" world. It is not image tagging or transcription. It is judgment work: you read what an AI model produced, decide whether it is good, and tell the lab why. Because that judgment is what actually improves frontier models, the labs pay more for it than for raw annotation, and they pay a large premium when you bring real expertise to the table.

What RLHF and Model Evaluation Work Actually Is

Reinforcement Learning from Human Feedback is the technique that turned raw language models into useful assistants. A base model predicts text; RLHF teaches it which responses humans actually prefer. As Palo Alto Networks explains, humans rate or rank model outputs, those ratings train a separate "reward model," and the reward model then steers the main model toward human-preferred behavior. You are the human in "human feedback." Your ratings are the raw material.

In practice, the day-to-day breaks into a handful of recurring task types.

Preference ranking (the core task)

You are shown a prompt and two or more model responses (often labeled A and B). You pick the better one and usually explain why: which is more accurate, more helpful, better formatted, safer, or more complete. This is the single most common RLHF task. According to industry descriptions of the workflow ( RiseDataLabs), comparison pairs are "generated and ranked at scale, validated through structured quality control." Most tasks take 3 to 15 minutes each.

Rating and scoring on rubrics

Instead of comparing two responses, you score a single response against defined criteria: helpfulness (1 to 7), factual accuracy, instruction-following, tone, safety. The lab supplies a rubric and you apply it consistently across hundreds of examples. Quality is tracked through "gold standard accuracy, inter-annotator agreement, and per-worker trust scores," and low-quality labels get auto-reassigned ( TaskMonk).

Writing demonstrations and corrections

Sometimes the model's answer is wrong and you write the correct one (a "demonstration"), or you edit the model's output into a gold-standard version. This is where strong writers and domain experts earn more: you are not just judging, you are producing the reference answer the model should learn from.

Prompt engineering and adversarial prompting

You write prompts designed to probe the model, elicit specific behaviors, or expose weaknesses, then evaluate how it responds across multi-turn conversations.

Red teaming and safety evaluation

The highest-skill category. Red teaming is "systematic stress-testing for safety, bias, and jailbreak vulnerabilities before public release" ( APxML). You try to make the model behave badly (produce disallowed content, contradict itself, leak information) and document exactly how you did it. These structured findings feed straight into the safety training pipeline. Red teaming, multi-turn conversations, and agentic or RL-environment data are described as the "difficult task types" that command premium rates and longer workforce ramps (HeroHunt).

Two delivery modes exist: live chat evaluation (you interact with the model in real time and rate as you go) and async transcript rating (you review batches of saved conversations). Most generalist work is async; live evaluation is more common on specialized projects.

How Much You Can Actually Make

Pay is driven by four levers, in order of impact: (1) your credentials and domain, (2) task complexity, (3) which platform you are on, and (4) your geography. The single biggest lever is expertise. A generalist and a board-certified physician can be on the same platform earning 10x different rates.

Here is the overall picture from Mercor's rate data, which puts the national average AI-trainer rate at about $31/hr with a full range of $12 to $200+:

Pay by seniority and tier

The percentile breakdown from the same Mercor data is a useful reality check: bottom quartile $18.50 to $21, median $25 to $33, upper quartile $53 to $58, and the top 10% at $100+. So the "$90/hr" you see advertised is real, but it is a top-quartile-plus outcome that generally requires credentials.

Pay by platform

This is the table most people are looking for. Rates below are for the platforms that actually hire for RLHF and model evaluation, drawn from platform pages, Glassdoor, and independent 2026 reviews.

A few clarifications. Mercor, Surge AI, and Handshake AI sit at the top because they are built around credentialed experts. DataAnnotation and Outlier are the most common entry points for generalists. Appen/CrowdGen and TELUS are the legacy "search rater" world (Appen's revenue fell from $270M in 2023 to $231M by end of 2025, and task volume has thinned ( review)). Prolific is a slightly different animal: it is a research-participant panel where the "AI" work is often academic behavioral studies rather than pure RLHF, with a required floor of £6/$8 per hour and a recommended £9/$12 ( Prolific).

Pay by geography

Geography matters enormously, both because some platforms are US-and-a-few-countries only, and because rates in North America and Western Europe run roughly double those in Eastern Europe or Latin America ( The AI Rankings).

The DataAnnotation restriction (US, Canada, UK, Ireland, Australia, New Zealand only) is worth memorizing, because it is one of the best-paying generalist entry points and it turns away most of the world ( The AI Rankings).

What a realistic month looks like

Advertised rates are ceilings. The binding constraint is almost never the rate, it is the number of paid hours available. Multiple sources describe the same pattern: workers often start strong ($1,500 to $2,000/week full-time) and see it decline over months as platforms surge-hire and then run out of tasks ( ZipRecruiter). For a contributor working 15 to 20 hours a week at the $30 to $45 tier, Outlier reviews describe $1,800 to $3,600/month as realistic during active project periods, and $0 when a project pauses. TELUS-style rater work is more stable but caps out around $500 to $1,200/month part-time ( review). Plan for lumpiness, and diversify across two or three platforms so a project drought on one does not zero out your income.

Skills and Qualifications You Actually Need

The barrier to entry is lower than most people assume for generalist work and higher than most people assume for the money tiers.

For generalist RLHF (the $15 to $45 band): No degree is strictly required on DataAnnotation, Outlier, or Alignerr for generalist tracks. What you genuinely need is:

• Native-level (or very strong) reading and writing in the target language, usually English.
• The patience to read long, multi-page instructions and follow them exactly. This is the number-one differentiator. The whole game is consistency against a rubric.
• Basic critical-thinking and fact-checking instincts: spotting when a confident answer is subtly wrong.
• Reliable internet and a computer (most tasks are desktop-only).

For the expert tiers (the $75 to $200+ band): A credential and provable domain depth. Platforms explicitly target PhD students, post-docs, practicing physicians, attorneys, and senior software engineers ( Mercor). Coding is the most accessible premium skill: DataAnnotation coding work pays $40 to $45/hr and up ( review), and you do not need a CS degree, just the ability to evaluate and write correct code in a real language.

The tools are simple: a browser, the platform's web-based labeling interface, and often a rubric document open in another tab. You do not need to know how to code a neural network. You need to be a sharp, consistent judge of quality.

How to Get Started and Pass the Assessments

The legitimate path has a fixed shape: find a platform, apply, complete a sample or qualification task, and if your work is good, you start receiving paid tasks. Money flows one direction only, from the company to you, and there is never a fee to join ( HireFeed).

Step 1: Pick platforms that match your profile

• Strong writer, no special credential, in one of the six countries: Start with DataAnnotation. It is writer-friendly and among the best-paying generalist entry points.
• Generalist, broader country list: Outlier and Alignerr accept a wide range and have high volume.
• You have a PhD, MD, JD, or senior engineering background: Go straight to Mercor, Surge AI, and Handshake AI, where credentials unlock $75 to $200+/hr.
• You want stable, lower-stakes work: TELUS International AI and Appen/CrowdGen search/rater projects.

You can and should apply to several. A profile aggregator like OpenTrain lets you build one expert profile and reuse it, but applying directly on each platform is fine.

Step 2: Pass the qualification test

Most platforms gate paid work behind an unpaid assessment. On DataAnnotation, the Starter Assessment takes about an hour and covers writing, grammar, reading comprehension, and simple logical reasoning, and critically, you can only take it once, with no retakes ( Portfolink). The proven approach:

1. Read the instructions completely before you touch anything. The test is designed to confirm you can follow complex, multi-page instructions accurately. Skimming is the most common failure.
2. Prioritize accuracy over speed. Successful candidates routinely take longer to be thorough. The platforms track quality, not clock speed, on the assessment ( OperationsArmy).
3. Be internally consistent. Apply the same criteria to every example, especially ambiguous ones. Inconsistency is what fails people.
4. Sweat the small details. Grammar, spelling, and formatting in your written justifications are part of the score, not a side note.
5. Explain your reasoning. When you rank A over B, the "why" is often what is actually being graded.

Step 3: Pass the follow-on qualifications

Passing the starter test does not mean paid work yet. On DataAnnotation you then take multiple short, unpaid qualification tests that match you to specific projects ( OperationsArmy). Treat each one like the first: read fully, be consistent, explain your reasoning.

Step 4: Protect your quality score

Once you are working, your per-worker trust/quality score determines whether you keep getting tasks and get promoted to higher-paying projects. Slow down on the first tasks of any new project, re-read the guidelines when a project updates them, and never guess when you can verify.

Payment Logistics and Taxes

You are an independent contractor, not an employee. That has consequences you should plan for from day one.

How you get paid: Most platforms pay weekly or bi-weekly via PayPal, and some use services like Hyperwallet. Surge AI, for example, pays weekly through Hyperwallet or PayPal, with first payment typically 3 to 5 weeks after you start ( review). Handshake and DataAnnotation are known for weekly PayPal payouts.

Taxes (US): All of it is taxable, whether or not you receive a tax form. You will typically get a 1099-NEC (non-employee compensation) or 1099-K (via PayPal). For 2025 and later, the 1099-K threshold reverted to more than $20,000 AND more than 200 transactions ( TurboTax), so many contractors will not receive a 1099-K, but the income is still reportable. The IRS is explicit: all gig income is taxable and self-reported on Schedule C.

Key tax facts for US contractors:

• You owe self-employment tax of 15.3% on net earnings (up to the Social Security wage base), on top of ordinary income tax ( CountryTaxCalc).
• No tax is withheld. You are expected to make quarterly estimated payments (roughly April 15, June 15, September 15, January 15).
• You can deduct legitimate business expenses on Schedule C: a portion of internet, a home-office deduction if eligible, and equipment.
• Set aside 25 to 30% of your earnings for taxes so quarterly deadlines do not hurt.

Pros and Cons: An Honest Ledger

The upside

• Genuinely flexible and remote. You choose your hours. Glassdoor contributors consistently cite work-from-home flexibility as the top pro, and Outlier scores 3.9/5 on work-life balance ( Glassdoor).
• Low barrier for generalists, high ceiling for experts. You can start with no credential, and if you have a PhD/MD/JD you can earn professional-consultant rates from your couch.
• The work is interesting. You are looking at frontier models before the public does, and the variety of projects keeps it from getting stale.
• You get paid to build AI literacy that is increasingly valuable in any career.

The downside

• Feast or famine. The recurring, near-universal complaint. Task availability is unpredictable; income swings from $8,000 a quarter to nearly nothing ( Mercor review context). Platforms over-onboard: Alignerr reportedly adds 15 to 50 new freelancers daily on Slack, many with no work ( Glassdoor).
• Unpaid onboarding. Long tutorials, mandatory webinars, and assessment tasks are often uncompensated or paid at a fraction of the task rate. One Outlier reviewer calculated about 15 hours of work for roughly $120, technically below their state minimum wage ( review).
• No benefits, no stability. No health insurance, no PTO, no guaranteed hours. This is supplemental or bridge income for most, not a salary.
• Legal and payment friction on some platforms. Outlier (operated by Scale AI) has faced multiple lawsuits over wage and misclassification claims ( review), and Handshake's Project HH contributors reported receiving partial payment of 20 to 50% of tracked hours ( review).
• Occasional exposure to disturbing content, especially in red-teaming and safety work.

Red Flags and Scams to Avoid

The RLHF gold rush attracts fraud. The rule is simple: money only ever flows from the company to you. Watch for these ( Digital Biz Talk, HireFeed):

• Any request for payment. "Training materials," "certification," "platform access," or "starter kit" fees are always a scam. Legitimate platforms are free to join.
• The hiring process runs on Telegram or WhatsApp. Almost certainly fraudulent. Real platforms use their own web portals and company email.
• You are asked to download an unknown "annotation tool" or register on a strange, off-brand website to take a "test." Real assessments happen inside the platform's own environment.
• Requests for banking passwords, or "verification" via a check they send you. Never share banking credentials; check-cashing schemes are classic fraud.
• An "assessment" that is suspiciously large and looks exactly like real production work. Some scams extract free labeling under the guise of an unpaid test. A legitimate qualification is a bounded sample, not an open-ended project.
• Impersonation. Scammers use the names of real companies (Appen explicitly warns it never asks you to pay ( Appen review)). Always apply through the official domain you verified independently, never a link from an unsolicited message.

The 2026 Outlook

The demand side is strong and structurally shifting toward exactly the skilled evaluation work this guide covers. The AI data labeling market is estimated at about $2.32 billion in 2026, projected to reach $6.53 billion by 2031 at a ~23% CAGR ( Mordor Intelligence); broader estimates run to $17 billion+ by the early 2030s ( Precedence Research). Meta's $15 billion investment for a 49% stake in Scale AI in 2025 signals how strategic this supply chain has become (HeroHunt).

The important trend for a worker: the market is moving from volume to depth. Simple annotation is being automated and commoditized (and its rates are falling), while RLHF, expert feedback, red teaming, and domain evaluation "command premium rates and longer workforce ramps" (HeroHunt). New credential-first platforms keep launching ( AfterQuery, a Y Combinator lab, raised $30M in April 2026 and pays specialists $100 to $250/hr ( RemoWork)). The takeaway: generalist rates will stay under pressure, but anyone who can bring verifiable expertise (coding, science, medicine, law, advanced math, or genuinely rigorous reasoning) is entering a growing, well-paid market.

Bottom Line

RLHF and model evaluation is legitimate, remote, and pays $15 to $90+ an hour depending mostly on what you bring to it. If you are a careful reader and writer, start with DataAnnotation, Outlier, or Alignerr, pass the assessment by reading every instruction and prioritizing accuracy, and diversify across platforms to smooth out the feast-or-famine cycle. If you have a real credential, skip straight to Mercor, Surge AI, or Handshake AI, where your expertise is worth $75 to $200+ an hour. Treat it as flexible supplemental income, set aside money for self-employment taxes, and never pay a cent to join. Do that, and this is one of the better ways to earn from a laptop in 2026.