Key findings
- 1
Between the week of 3 Aug 2026 and the week of 5 Oct 2026, open roles in AI Evaluation & Rating (RLHF) on the platforms tracked since 3 Aug 2026 rose from about 161 to a peak of 234 in the week of 17 Aug 2026, then fell to 84 (-47.5% over the 10 weeks, against +10.5% for all AI training work on the same platforms); platforms first tracked later, left out of this panel, listed about 60 more in the week of 5 Oct 2026.
- 2
21 platforms listed the 143 open roles in AI Evaluation & Rating (RLHF); SuperAnnotate (SME) held 20.3% and the three largest 53.1% (HHI 1,204, unconcentrated, against 742 for all AI training work).
- 3
116 of the 143 open roles in AI Evaluation & Rating (RLHF) (81.1%) stated an hourly rate, with a median of $55/hr (middle half $30 to $76), 15% below the $65/hr of all AI training work.
- 4
Each open role in AI Evaluation & Rating (RLHF) drew 0.92 apply clicks a week on AITraining.jobs, 2.44x the average AI training role: the 3rd highest of 12 kinds of work with enough clicks to rank.
- 5
34.3% of the 143 open roles in AI Evaluation & Rating (RLHF) accepted applicants from any country, against 28.5% of all AI training roles; of the 20 countries compared, residents of the United States (100), India (75) and Germany (74) could apply to the most, and residents of Palestine to the fewest (49).
- 6
The words used most in the titles of open roles in AI Evaluation & Rating (RLHF) were Evaluator or reviewer (31), Expert or SME (26) and Analyst or researcher (18), of 143 open roles.
- 7
Of the 133 open roles in AI Evaluation & Rating (RLHF) with a full description, 20.3% mention a PhD or master's degree, 6.0% a professional credential and 21.1% an assessment or test; the 51 stating years of experience ask for a median of 5 years.
- 8
Over the 28 days from 20 Sep 2026 to 17 Oct 2026, open roles in AI Evaluation & Rating (RLHF) on the platforms tracked at its start went down 53.2% (186 to 87): platforms listed 156 new roles and removed 263, and a removed role had been listed for a median of 29 days (33 across all AI training work).
Figure 1
Open roles, week by week
Is the number of open roles in AI Evaluation & Rating (RLHF) rising or falling?
| Week | Open roles (fixed panel) | Index |
|---|---|---|
| 3 to 9 Aug | 161 | 100.0 |
| 10 to 16 Aug | 204 | 126.6 |
| 17 to 23 Aug | 234 | 145.6 |
| 24 to 30 Aug | 223 | 138.7 |
| 31 Aug to 6 Sep | 203 | 126.3 |
| 7 to 13 Sep | 193 | 119.8 |
| 14 to 20 Sep | 150 | 93.5 |
| 21 to 27 Sep | 141 | 87.9 |
| 28 Sep to 4 Oct | 122 | 75.6 |
| 5 to 11 Oct | 84 | 52.5 |
Sample. Average open roles in AI Evaluation & Rating (RLHF) over each week's days, on the 17 platforms tracked since 3 Aug 2026 (the fixed panel; 15 of them list this work), 3 Aug 2026 to 11 Oct 2026.
The index is the panel's open roles against the week of 3 Aug 2026 (100), beside the same index for all AI training work on the same platforms.
OpenTrain AI (first tracked 5 Sep 2026), Welo Data (first tracked 8 Sep 2026), Perle Labs (first tracked 10 Oct 2026), SuperAnnotate (SME) (first tracked 10 Oct 2026), Turing (first tracked 10 Oct 2026), RemoExperts (first tracked 10 Oct 2026), Sovrano AI (first tracked 11 Oct 2026) and Anyone AI (first tracked 11 Oct 2026) list this work but were added to the dataset later, so they are left out of the panel and counted only in the last column: a platform's arrival never reads as growth.
Weeks with fewer than 10 open roles on the panel, or cut by the span, are marked.
The series starts in the week of 3 Aug 2026 at the earliest: before it, this dataset's feeds were still being added and tuned.
Source. AITraining.jobs Research, The Market for AI Evaluation & Rating (RLHF) Work, live data.
Figure 2
Listings added and removed
How fast do roles in AI Evaluation & Rating (RLHF) turn over, and is the stock growing on the same platforms?
| Work | Open on 20 Sep 2026 | Open on 17 Oct 2026, same platforms | Change | New listings |
|---|---|---|---|---|
| AI Evaluation & Rating (RLHF) | 186 | 87 | -53.2% | 156 |
| All AI training work | 2,595 | 2,690 | +3.7% | 2,252 |
Sample. Roles listed from 20 Sep 2026 to 17 Oct 2026: 406 in AI Evaluation & Rating (RLHF), 5,849 across all AI training work.
Change compares the roles open on the first and the last day of the window on the platforms tracked when it opened, so a platform added during the window does not read as growth.
A new listing first appeared in the window on a platform already tracked before that week; a platform's first week (its back catalogue) is not counted.
Median days listed runs from a removed role's listing date to its last sighting, for removed roles with a known listing date (at least 10; 256 here).
Source. AITraining.jobs Research, The Market for AI Evaluation & Rating (RLHF) Work, live data.
Figure 3
Who lists it
Which platforms list roles in AI Evaluation & Rating (RLHF), and what does each advertise?
| Platform | Open roles | Share of open roles | Median per hour |
|---|---|---|---|
| SuperAnnotate (SME) | 29 | $50 | |
| micro1 | 28 | $60 | |
| Turing | 19 | n/a | |
| Mercor | 12 | $75 | |
| Alignerr | 9 | $80 | |
| Braintrust | 8 | n/a | |
| Invisible | 8 | $53 | |
| Welo Data | 8 | $20 | |
| Handshake AI | 6 | n/a | |
| AfterQuery | 2 | n/a | |
| Claru | 2 | n/a | |
| OneForma | 2 | n/a | |
| OpenTrain AI | 2 | n/a | |
| Anyone AI | 1 | n/a | |
| Appen | 1 | n/a | |
| Outlier (Scale AI) | 1 | n/a | |
| Perle Labs | 1 | n/a | |
| RemoExperts | 1 | n/a | |
| Terac | 1 | n/a | |
| Vetto | 1 | n/a | |
| xAI (SpaceXAI) | 1 | n/a |
Sample. 143 roles in AI Evaluation & Rating (RLHF) open at the end of 17 Oct 2026, by the platform that lists them: HHI 1,204 (unconcentrated), against 742 for all AI training work.
HHI: the sum of every platform's squared percentage share of the open roles, 0 to 10,000; under 1,500 is unconcentrated, 1,500 to 2,500 moderately and above 2,500 highly concentrated (US merger-guideline thresholds).
Shares are of listings, not of people hired or money paid: a platform posting one role per language or country weighs more here than its hiring may.
A median needs at least 8 roles stating an hourly rate.
Source. AITraining.jobs Research, The Market for AI Evaluation & Rating (RLHF) Work, live data.
Figure 4
What it pays
What do roles in AI Evaluation & Rating (RLHF) advertise per hour, against AI training work as a whole?
| Work | Median per hour | 25th percentile | 75th percentile | Stating an hourly rate |
|---|---|---|---|---|
| AI Evaluation & Rating (RLHF) | $30 | $76 | 81.1% | |
| All AI training work | $36 | $85 | 82.6% |
Sample. Roles open on 17 Oct 2026: 143 in AI Evaluation & Rating (RLHF), 116 of them stating an hourly rate; 3,668 across all AI training work.
Pay is the hourly figure the posting advertises (the middle of a stated range). The middle half runs from the 25th to the 75th percentile.
Roles paid per task or per project, or that state no figure, do not state an hourly rate. A median needs at least 8 roles stating one.
Source. AITraining.jobs Research, The Market for AI Evaluation & Rating (RLHF) Work, live data.
Figure 5
How its rates are distributed
What share of roles in AI Evaluation & Rating (RLHF) pay $50 an hour or more?
| Advertised rate | Open roles | Share of priced roles | All AI training work |
|---|---|---|---|
| Under $15/hr | 6 | 5.2% | 5.7% |
| $15 to $25/hr | 14 | 12.1% | 8.2% |
| $25 to $35/hr | 11 | 9.5% | 9.2% |
| $35 to $50/hr | 20 | 17.2% | 16.4% |
| $50 to $75/hr | 32 | 27.6% | 25.4% |
| $75 to $100/hr | 23 | 19.8% | 14.4% |
| $100 to $150/hr | 6 | 5.2% | 16.8% |
| $150/hr or more | 4 | 3.4% | 3.9% |
Sample. 116 roles in AI Evaluation & Rating (RLHF) open on 17 Oct 2026 that state an hourly rate, against 3,031 across all AI training work.
Each role counts once, at the middle of its advertised range. The bands are those of the pay-transparency report.
Source. AITraining.jobs Research, The Market for AI Evaluation & Rating (RLHF) Work, live data.
Figure 6
Applicant demand per open role
Do roles in AI Evaluation & Rating (RLHF) draw more or fewer applicants per opening than the average AI training role?
| Roles | Clicks per role a week | Index |
|---|---|---|
| AI Evaluation & Rating (RLHF) | 0.92 | 2.44x |
| All AI training work | 0.38 | 1.00x |
| Its roles stating an hourly rate | 0.92 | 2.45x |
| Its roles stating no hourly rate | 0.90 | 2.38x |
| Its roles open worldwide | 0.72 | 1.90x |
| Its roles limited by country | 1.15 | 3.04x |
Sample. Apply clicks on a specific role on AITraining.jobs, from 20 Sep 2026 to 17 Oct 2026: 619 on roles in AI Evaluation & Rating (RLHF), against the days those roles were open.
Clicks per role a week = apply clicks / (open role-days / 7). Index 1.00 is the rate of every role on the site.
Rows with fewer than 30 clicks or 3 roles open on an average day are marked as a small sample.
Apply clicks measure interest among visitors to AITraining.jobs, who see roles open in their country and recent, priced roles first. Clicks are never split by platform.
Source. AITraining.jobs Research, The Market for AI Evaluation & Rating (RLHF) Work, live data.
Figure 7
Applicant demand by listing age
Do newer roles in AI Evaluation & Rating (RLHF) draw more applicants than older ones?
| Listing age | Clicks per role a week | Index (this work = 1) |
|---|---|---|
| Listed during the window | 1.86 | 2.02x |
| Listed up to 4 weeks before it | 1.09 | 1.18x |
| Listed 4 to 12 weeks before it | 0.22 | 0.24x |
| Listed more than 12 weeks before it | 0.72 | 0.78x |
Sample. Apply clicks on roles in AI Evaluation & Rating (RLHF), from 20 Sep 2026 to 17 Oct 2026, by when each role was listed relative to the window's first day (20 Sep 2026).
Listing date: the date the job board gives for the posting, or else the first day the role appeared in the feed. A role that arrived with a platform's back catalogue and gives no date has none.
The index divides each row's rate by the rate of all roles in this kind of work, so it reads within the kind.
Rows with fewer than 30 clicks or 3 roles open on an average day are marked as a small sample.
Source. AITraining.jobs Research, The Market for AI Evaluation & Rating (RLHF) Work, live data.
Figure 8
Who may apply: location rules
Can people apply to roles in AI Evaluation & Rating (RLHF) from anywhere?
| Location rule | Open roles | Share of its open roles | All AI training work |
|---|---|---|---|
| Open worldwide | 48 | 33.6% | 28.5% |
| Worldwide except listed countries | 10 | 7.0% | 9.3% |
| Only listed countries | 84 | 58.7% | 61.9% |
| No rule stated | 1 | 0.7% | 0.4% |
Sample. 143 roles in AI Evaluation & Rating (RLHF) open at the end of 17 Oct 2026, against 3,668 across all AI training work.
The rule is read from each posting's location text, then, where that names no country, from what the posting's own description states, and where both are silent from what the platform states about where it hires.
Source. AITraining.jobs Research, The Market for AI Evaluation & Rating (RLHF) Work, live data.
Figure 9
Who may apply: roles open to each country
How many roles in AI Evaluation & Rating (RLHF) can a resident of each country apply to?
Sample. 143 roles in AI Evaluation & Rating (RLHF) open at the end of 17 Oct 2026, for residents of a fixed set of 20 countries.
A country's count is the roles open worldwide (including roles that state no rule, as on the job board), plus roles whose list names it, plus roles whose exclusion list does not.
The 20 countries are the same in every report of this series: those named most often by the location rules of open AI training roles on 9 Oct 2026, either as the only countries accepted or as excluded.
Median pay is over the roles open to that country that state an hourly rate (at least 8).
Source. AITraining.jobs Research, The Market for AI Evaluation & Rating (RLHF) Work, live data.
Figure 10
What the roles are called
What are roles in AI Evaluation & Rating (RLHF) called: annotator, evaluator, expert or trainer?
| Title word | Open roles | Share of its open roles |
|---|---|---|
| Evaluator or reviewer | 31 | 21.7% |
| Expert or SME | 26 | 18.2% |
| Analyst or researcher | 18 | 12.6% |
| Lead or manager | 18 | 12.6% |
| Specialist | 16 | 11.2% |
| Quality assurance | 14 | 9.8% |
| Trainer | 14 | 9.8% |
| Engineer or developer | 8 | 5.6% |
| Rater | 8 | 5.6% |
| Writer or editor | 3 | 2.1% |
| Red teamer or tester | 2 | 1.4% |
| Tutor | 1 | 0.7% |
Sample. 125 of the 143 roles in AI Evaluation & Rating (RLHF) open on 17 Oct 2026 name one of these words in their title (87.4%); 12 title words are named.
A title can use several of these words ('Expert AI Trainer') and counts for each. The vocabularies are those of the job-titles, languages and expert-fields reports.
Median pay needs at least 8 roles stating an hourly rate.
Source. AITraining.jobs Research, The Market for AI Evaluation & Rating (RLHF) Work, live data.
Figure 11
What the descriptions ask for
Do roles in AI Evaluation & Rating (RLHF) ask for degrees, credentials, experience or tests?
| Requirement | Roles | Share of described roles |
|---|---|---|
| Mentions a PhD | 18 | 13.5% |
| Mentions a PhD or master's | 27 | 20.3% |
| Mentions any degree | 53 | 39.8% |
| Mentions a professional credential | 8 | 6.0% |
| States years of experience | 51 | 38.3% |
| Asks for 5 years or more | 32 | 24.1% |
| States weekly hours | 28 | 21.1% |
| Mentions an assessment or test | 28 | 21.1% |
| Mentions an AI-led or video interview | 10 | 7.5% |
| Mentions a background check | 1 | 0.8% |
Sample. 133 roles in AI Evaluation & Rating (RLHF) open on 17 Oct 2026 whose job board publishes a full description (AfterQuery, Anyone AI, Appen, Braintrust, Claru, Handshake AI, Invisible, Mercor, micro1, OneForma, OpenTrain AI, Outlier (Scale AI), Perle Labs, RemoExperts, SuperAnnotate (SME), Terac, Turing, Vetto, Welo Data and xAI (SpaceXAI)), 93.0% of its open roles; 3,260 described roles across all AI training work.
A mention can be a requirement or a preference ('a PhD is a plus'): the figure counts both. The patterns are those of the requirements report.
Only some platforms publish full descriptions, so this figure describes their roles, not every platform's.
Source. AITraining.jobs Research, The Market for AI Evaluation & Rating (RLHF) Work, live data.
For labs, data vendors and researchers
Buy the data behind this report
The figures above are a summary. Behind them are records we collect ourselves, every day:
9,572 roles tracked, each with its history
Every role on 29 job boards, read daily since 13 Jul 2026: its pay and place, and the first and last day it was open.
6,200 apply clicks in 28 days
Which roles people actually apply to, by field, pay and kind of work. It happens on this site, so nobody else can count it.
39 lab and vendor hiring boards
The job boards of 25 AI labs and 14 human-data vendors, read every day: who is hiring for human data, and who is hiring to buy it.
What you can buy
The records
The rows behind any figure, with their history, not only the totals.
A regular feed
The same figures every week or month, as files.
A custom cut or benchmark
Your countries, languages or fields, or your own platform against the market.
Method and definitions
The public job boards of the AI training platforms this site tracks, read every day. Apply clicks are made on AITraining.jobs; full descriptions come from the platforms whose boards publish them. Every figure is computed by one database query over the window shown, the same query for the live report and for every edition. Each figure states its sample beneath it.
- Kind of work
- Each role is classified from its title and description into one of the twelve kinds of AI training work this site tracks; this report counts the roles classified as AI Evaluation & Rating (RLHF).
- Fixed panel
- The platforms already tracked when the weekly series opens. The index follows their open roles, so a platform added later (such as OpenTrain AI and Welo Data in September 2026) never reads as growth.
- Index
- The panel's average open roles in a week, divided by the same in the base week, times 100; shown beside the same index for all AI training work on the same platforms.
- Open role
- A role listed by a given day and still in its platform's feed that day.
- New listing
- A role that first appeared during the window on a platform already tracked before that week. A platform's first week in this dataset is its back catalogue and is not counted as new.
- Removed listing
- A role that left its platform's feed during the window.
- Days listed
- From a removed role's listing date to the last day it was seen in its feed. A median needs at least 10 removed roles.
- HHI
- Herfindahl-Hirschman Index: the sum of every platform's squared percentage share of open roles, 0 to 10,000. Under 1,500 unconcentrated, 1,500 to 2,500 moderately, above 2,500 highly concentrated.
- Advertised pay
- The hourly figure a posting states (the middle of a stated range). Medians need at least 8 roles stating a rate; the middle half runs from the 25th to the 75th percentile.
- Clicks per open role a week
- Apply clicks on AITraining.jobs on a specific role, divided by the days the roles were open, times 7.
- Demand index
- Clicks per open role a week divided by the same rate across every role on the site. Above 1, the roles draw more applicants than the average role.
- Listing age
- When a role was listed relative to the window's first day: during the window, up to 4 weeks before it, 4 to 12 weeks before or longer. The listing date is the job board's posting date, or else the first day the role appeared in the feed.
- Location rule
- Read from each posting's location text and, where that names no country, from what the posting's own description states: open worldwide, worldwide except listed countries, only listed countries, or no rule stated.
- Open to a country
- A role is open to residents of a country if it is open worldwide, its list names the country, or its exclusion list does not. The 20 countries compared are the same in every report of the series.
- Window
- Live: the last 28 complete days, in UTC, and the last twelve complete ISO weeks (from the week of 3 Aug 2026 at the earliest) for the weekly series. Edition: one calendar month, in UTC, and its weeks.
What this data cannot tell you
- Platforms that recruit without a public job board are not counted. Several large ones invite workers to projects directly.
- A listing is not a hire. Some platforms keep evergreen roles open for months, and some post one listing per language or country.
- Apply clicks measure interest among the people who use AITraining.jobs, and what is shown first gets clicked more. Clicks are never published by platform.
- Full descriptions are published by a few platforms only, so the requirements figure describes their roles and appears only when at least 10 roles of this kind are described.
- A kind of work with few open roles gives figures that move with a handful of postings: rows on too little data are marked as a small sample.
How the job feeds are read and roles are classified: site methodology. How fresh, complete and consistent the data is, checked every day: data quality and coverage.
Editions
The live report covers the last 28 complete days and changes every day. Each complete month is also published as an edition, frozen two days after the month ends, so a figure you cite from an edition never moves.
- Live dataLast 28 complete days, recomputed daily
- September 2026 editionFrozen 10 Oct 2026
- August 2026 editionFrozen 10 Oct 2026
Cite and reuse
Free to quote, chart and republish under CC BY 4.0 with a link to this page. Live figures change daily: cite a monthly edition for numbers that will read the same later, or keep the access date.
Citation
AITraining.jobs (2026). The Market for AI Evaluation & Rating (RLHF) Work (live data, 20 Sep 2026 to 17 Oct 2026). https://aitraining.run/research/ai-evaluation-rlhf-market (accessed 18 Oct 2026)
BibTeX
@misc{aitrainingjobs_ai_evaluation_rlhf_market_live,
author = {{AITraining.jobs}},
title = {The Market for AI Evaluation & Rating (RLHF) Work (live data, 20 Sep 2026 to 17 Oct 2026)},
year = {2026},
howpublished = {\url{https://aitraining.run/research/ai-evaluation-rlhf-market}},
note = {Accessed 18 Oct 2026},
}More research
The same figures for every other kind of work:
- Coding & Software Evaluation market
- Math & STEM market
- Law, Medicine & Finance market
- Agentic Tasks & RL Environments market
- Data Annotation & Labeling market
- Robotics & Physical-AI Data market
- Multilingual & Localization market
- Transcription, Audio & Speech market
- Writing & Content market
- Red-Teaming & AI Safety market
- Research Studies & Surveys market
- Which AI Labs Are Hiring for Human DataWhich AI labs are hiring for the teams that buy, run and evaluate human data, and for what roles?Read the report
- Who Is Buying AI Training DataWhich AI labs and data vendors have people whose job is to source, license or acquire training data right now?Read the report
- Where Human-Data Vendors Are BuildingHow fast are the human-data vendors hiring their own staff, and for which functions?Read the report
- What AI Labs Pay for Human DataWhat do AI labs spend on human data, what do vendors bill, what do the experts receive, and what do RL environments and tasks cost?Read the report
- What AI Labs Are BuyingWhich kinds of training data are in rising demand, and which are cooling?Read the report
- AI Training Data Licensing DealsWho has licensed content and data to which AI companies, on what terms and for how much?Read the report
- The State of AI Training PayWhat does AI training work pay, by kind of work and by platform?Read the report