Coding & Software Evaluation

If you can read code, spot a bug, and explain why an answer is wrong, you can get paid $25 to $200+ per hour to evaluate AI-generated software. "Coding and software evaluation" work in AI training means reviewing the code that large language models write, judging whether it is correct, secure, efficient, and production-ready, ranking competing answers, and writing the corrected reference solution when every model gets it wrong. Frontier AI labs (OpenAI, Anthropic, Google DeepMind, Meta) pay real engineers to do this because a model cannot reliably grade its own code at the level of nuance required, and each of those labs now spends on the order of $1 billion a year on human training data. This guide covers exactly what the work is day to day, what the legitimate platforms actually pay (by platform, seniority, and country), how to pass the assessments, the tax and payment logistics, and the red flags that separate real work from the scams flooding the space.

The short version: this is one of the highest-paying remote gig categories that exists, it is genuinely legitimate at the platforms named below, and it rewards actual engineering skill rather than credentials or resume polish. It is also inconsistent, project-based, and not a replacement for a salary. Both of those things are true at once, and this guide will be honest about both.

What "Coding & Software Evaluation" Work Actually Is

The core loop is the same across almost every platform. You are handed a coding prompt (sometimes a self-contained problem, sometimes a realistic scenario like "debug this production issue" or "implement this feature across these files") together with one or more candidate solutions written by an AI model. Your job is to judge those solutions. According to the day-to-day breakdown from practitioners, you read the prompt and 1 to 4 candidate solutions, run them mentally or in a sandbox, find bugs, evaluate code quality, and write a corrected reference solution if all candidates fail.

That single loop shows up in several named task types, and most contracts mix them:

Pairwise comparison and ranking

You see two (or more) model answers side by side and decide which is better, then explain why in structured feedback. This is the backbone of RLHF (reinforcement learning from human feedback). You are not just picking a winner: you are producing the reasoning that teaches the model what "better" means. As one practitioner guide puts it, you read model outputs side by side, rank them, explain your reasoning, write better versions when both options are bad, and flag things that are subtly wrong in ways that matter.

Code review and bug hunting

You review a single AI-written solution for correctness and quality, catching real defects. DataAnnotation describes this as catching bugs like race conditions and inefficient algorithms, and evaluating across systems programming, algorithms, security, backend design, and frontend. The subtle bugs matter most: an answer that looks right and passes a naive test but has an off-by-one error, a race condition, an O(n squared) approach where O(n) exists, or a security hole.

Reference solution authoring

When every candidate is wrong, or when a project needs a gold-standard answer, you write a clean, correct, production-quality implementation yourself. DataAnnotation calls this "authoring production-quality implementations that serve as training data". This is the highest-value work because your code becomes the target the model is trained toward.

Prompt writing and adversarial testing

You craft precise engineering prompts designed to probe the limits of a model's reasoning. Mercor frames the "test" side of the job as assessing how models handle ambiguity, incomplete requirements, architectural tradeoffs, conflicting constraints, and multi-step engineering workflows. You are effectively trying to break the model in instructive ways.

Rubric design and RL environments

At the senior end, you stop grading individual tasks and start designing the evaluation itself: building rubrics, test harnesses, containerized environments, and reinforcement-learning environments where an agent's code is automatically scored. Surge AI's senior coding role involves "owning end-to-end coding data projects" and setting up technical environments like containers and test harnesses. This is where the money is highest.

The common thread: human judgment is the product. You are being paid for the specific engineering discernment a model cannot yet reliably reproduce.

The Platforms That Hire for Coding Evaluation (and What They Really Pay)

There is a large gap between the headline rate a platform advertises and what a typical contributor nets. The numbers below are drawn from each platform's own pages plus third-party reporting and Glassdoor data. Treat the top of each range as the specialist ceiling and the bottom as generalist or onboarding pay.

Pay by platform

A few honest caveats on these numbers. First, advertised rates are not effective rates. Workers who track their full platform engagement (not just active task time, but time reading instructions, waiting for tasks, and doing calibration) often find their effective hourly rate drops 20 to 30% below the dashboard figure. Second, the highest tiers ($150 to $300/hr) are real but scarce, gated behind specialist domains or senior credentials. Third, several platforms ( Handshake, DataAnnotation) can and do ghost long-term contractors without warning when a project ends, so treat any single platform as impermanent.

Pay by seniority

The single biggest lever on your rate is not which platform you pick, it is how senior and specialized your engineering skill reads in the assessment. Mercor's public tiers show this cleanly.

Sources: Mercor expert tiers, coding evaluator band. The through-line: the expert band runs from roughly $75 to over $200 an hour, with senior specialists averaging about $85 and reaching $200+.

Pay by geography

This work is global, but the pay is not evenly distributed, and this is one of the least-discussed realities of the field. Country of residence affects both your rate and which platforms will even accept you.

The practical takeaway from the geographic data is blunt: US/UK workers access higher-paying platforms restricted by country, while workers in India and the Philippines face substantially lower rates on the platforms that accept them globally. The two escape hatches from geographic pay tiers are the same two that raise anyone's rate: genuine specialization and speed. A distributed-systems specialist in any country out-earns a generalist anywhere.

Skills and Qualifications You Actually Need

The good news for engineers: this work is merit-based to an unusual degree. DataAnnotation is explicit that no resume is required and evaluation is merit-based, assessed through a short exercise rather than credentials. You do not strictly need a computer science degree. What you need is the ability to demonstrate real engineering judgment on a timed test.

The hard requirements

• Working professional fluency in at least one mainstream language. Python is the most in-demand, followed by TypeScript/JavaScript, then Rust, C++, Go, and Java. If you know two or three of these well, you qualify for far more projects.
• The ability to find subtle bugs. Not just "does it compile," but "does this pass the naive test and still fail on an edge case, a concurrency scenario, or a security vector."
• Complexity reasoning. Recognizing when an O(n squared) solution should be O(n log n), or when an approach will not scale.
• Clear written explanation. You must articulate why one answer beats another. Feedback that just says "wrong" is useless; the reasoning is the training signal.

What raises your tier

• 2 to 6+ years of professional software engineering experience (Surge asks for this range explicitly, Mercor wants professionals who can independently reason through production systems).
• Depth in a niche: distributed systems, infrastructure, security, systems/low-level programming, ML systems, or a specific mainstream framework. Niche depth is what moves you from the $50/hr band to the $120 to $200/hr band.
• Open-source contribution history or a demonstrable public track record ( DataAnnotation names open-source contributors as a target group).
• Domain crossover: coding plus law, medicine, or finance credentials pushes you toward the $175 to $300+/hr specialist ceiling.

Students are welcome too. DataAnnotation lists computer science students with strong algorithmic foundations as a target group, so a strong CS undergrad who can crush a bug-hunting exercise can start earning coding rates.

Tools You Will Use

The tooling is deliberately lightweight. Most coding evaluation happens inside a web-based platform interface, not your local IDE. Expect:

• A browser-based task interface presenting the prompt, the model outputs, and the rating rubric or ranking controls.
• A code sandbox or execution environment for actually running candidate code. On senior projects you may set up your own containers and test harnesses.
• Markdown/rich-text feedback fields for your structured reasoning.
• Your own local environment for authoring reference solutions when the task allows it, then pasting in the verified result.
• ID verification tooling. Most legitimate platforms verify identity, and some (like Outlier) require ID verification and sometimes a video selfie to prevent fraud and duplicate accounts.

You do not need special software, a paid license, or a "starter kit." If a platform asks you to buy tools, that is a scam signal (covered below).

How to Get Started and Pass the Assessment

The application funnel is nearly identical across platforms: apply, complete a qualification exercise, get matched to projects, start earning. Money only ever flows from the platform to you. Here is the realistic path.

Step 1: Pick two or three legitimate platforms and apply

Do not apply to just one. Work arrives in waves and any single platform can go quiet or drop you. A sensible starter portfolio for a strong engineer: DataAnnotation (no resume, fast merit-based assessment, high coding pay), Mercor (clear expert tiers, weekly pay), and one of Braintrust / Handshake / micro1 depending on your region and credentials. Apply directly on each platform's own domain, never through a third-party link.

Step 2: Take the assessment seriously

This is the gate, and it is a real test of engineering skill, not a formality.

• DataAnnotation: a roughly 60-minute engineering assessment testing whether you can identify correctness bugs, reason about complexity, and write clean code. No resume, purely merit.
• Outlier: a timed test matching your selected domain, with coding assessments featuring LeetCode-style problems, plus a critical General Reasoning Screening that requires 80%+ to pass. Miss the reasoning screen and you are out regardless of coding skill.
• Surge / senior roles: a short exercise delivered via Google Doc that you email back with your resume, designed to show you can scope a project and communicate technically.

Practical tips that move the needle:

• Read the rubric before you start and grade against it, not against your own taste. The number one reason strong engineers fail is answering the question they wish was asked instead of the one on the screen.
• Justify every judgment in writing. Even when only a rating is requested, clear reasoning is what the platform is actually screening for.
• Do not rush the bug hunt. The tests are built so a fast skim misses the planted subtle defect. Run the code (mentally or in a sandbox), test the edges.
• Write clean reference code. If asked to author a solution, treat it as production code: correct, readable, reasonably efficient, no debug cruft.

Step 3: Get matched and calibrate

Once you pass, you are matched to projects by preferred stack and assessment result. Early tasks are often "calibration" work where your ratings are compared against a gold standard. Your quality score here determines your ongoing task flow and sometimes your rate, so treat calibration as more of the assessment.

Step 4: Manage the reality of project waves

Expect income to be lumpy. AfterQuery-style work typically arrives as 2 to 3 week assignments at 10 to 20 hrs/week, and queues can empty without warning. The contributors who earn consistently are the ones active on several platforms so that when one queue dries up another is filling.

Realistic Earnings: What You Can Actually Make

Here is the honest arithmetic. The advertised rate, the effective rate, and the annual total are three different numbers.

At the coding evaluator level ($40 to $60/hr): A contributor working 15 hours a week at $45/hr grosses about $675/week, or roughly $2,700/month, if the queue stays full. Adjust for a 20 to 30% effective-rate haircut and dry spells, and a realistic sustained figure is $1,800 to $2,400/month as a side income.

At the expert level ($80 to $150/hr): A senior engineer landing steady expert-tier work at $100/hr for 20 hours a week grosses $2,000/week. Full-time-equivalent, that approaches the $150,000+/year that some senior specialists reach, and the very top full-time AI training roles for senior engineers and researchers can pay upwards of $200,000 annually.

Proof it adds up: one widely-cited DataAnnotation contributor documented earning $37,000 over time on the platform, and DataAnnotation reports having paid over $20 million to contractors since 2020 across 100,000+ people.

The realistic mental model: coding evaluation is an excellent high-hourly side income and a plausible full-time income for a senior specialist who treats it like a business (multiple platforms, high quality scores, specialist positioning). It is not a stable salary with benefits, and anyone promising that is selling something.

Payment and Tax Logistics

You are an independent contractor (1099), not an employee. This has concrete consequences you must plan for.

How you get paid

Most platforms pay weekly via PayPal or direct deposit. DataAnnotation, for example, pays weekly and processes coding work task-based. Note the holds: DataAnnotation contributors wait a week to cash out hourly funds or three days for per-task projects, with instant transfer to PayPal thereafter. Mercor and several others also pay weekly. This fast, frequent payout is one of the genuine perks of the category.

The tax reality (US)

• You owe self-employment tax. On top of income tax, contractor income is subject to self-employment tax of 15.3% (12.4% Social Security + 2.9% Medicare), because no employer is paying the other half for you.
• 1099-NEC thresholds. A platform must send you a 1099-NEC if it pays you $600 or more in a year. But you must report all income even below that. For payment apps, the 1099-K reporting threshold is changing (rising toward $2,000 for 2026), so do not rely on receiving a form. Keep your own records of every payment from every platform.
• Quarterly estimated taxes. If you expect to owe more than $1,000 for the year, the IRS expects quarterly estimated payments. Missing them can trigger penalties.
• Set aside 25 to 35% of every payout for federal, state, and self-employment taxes. This is the single most common mistake new contractors make: spending the gross and getting a nasty bill in April.
• Deduct legitimate business expenses. Your computer, home-office portion, internet, and relevant software can offset income. Track them.

Non-US contributors face their own local tax obligations and often currency-conversion haircuts on payout, so check your country's self-employment rules.

Pros and Cons: An Honest Ledger

Pros

• Exceptional hourly pay for remote work. $40 to $200+/hr for coding is far above almost any other remote gig category.
• Merit-based entry. DataAnnotation and others require no resume; your assessment is your application. Skill beats pedigree.
• Genuine flexibility. Asynchronous, work-from-anywhere, set your own hours around a primary job.
• Fast, frequent payment. Weekly payouts, often via PayPal, with short holds.
• You stay sharp. Reading and critiquing frontier-model code keeps your engineering judgment current, and you see how the best models actually reason.
• Real demand. Every major lab needs this continuously; the AI training dataset market is projected to grow from ~$3.9B in 2026 to $16.3B by 2033.

Cons

• Income is lumpy and impermanent. Projects arrive in waves, queues empty without notice, and platforms can ghost even long-term contractors.
• Effective rate is below the headline. Unpaid calibration, instruction-reading, and downtime can shave 20 to 30% off the dashboard rate.
• No benefits, and you owe the full 15.3% self-employment tax.
• Quality pressure. Low quality scores cut your task flow or rate; the calibration bar is real.
• The work can be repetitive once the novelty of grading model code wears off.
• The field is crowded with scams (next section), so vetting every opportunity is on you.

Red Flags and Scams to Avoid

AI training is now a magnet for job scams. Reports to the FTC topped 105,000 in 2024, roughly triple the 2020 total, with reported losses jumping from $90M to over $513M. Deepfake fraud in hiring jumped roughly 1,300% from 2023 to 2024. Learn these signals cold.

• Any upfront payment is a scam. Full stop. No legitimate platform charges for "training materials," "certification," "platform access," equipment, or a background check. Money flows one direction only: from the company to you.
• Hiring conducted entirely over WhatsApp/Telegram/iMessage with no verifiable company presence and no real interview is a serious red flag.
• Vague task descriptions. If a posting says "train AI models" but cannot tell you whether you will rate code, rank text, or label images, be skeptical. Legit platforms are transparent about tasks.
• Fake domains. Watch for lookalikes like scaleai-jobs.com instead of the real scale.com. Always apply on the platform's official domain, cross-checked from its main website.
• "Assessment" that is actually free labor. Some scams disguise unpaid production work as an endless "test." A legitimate qualification exercise is short (about an hour) and clearly bounded.
• Too-good-to-be-true rates with no skill gate. "$90/hr, no experience, start today" with an instant offer is the classic bait.

How to verify a real opportunity: look up the role on the company's official website; if it is not listed there, walk away. Legitimate platforms also have active subreddits, forums, or Discord communities where real contributors discuss payment and tasks. Every platform named in this guide passes these checks.

The 2026 Outlook: Is This Work Going Away?

The obvious worry: if AI is getting better at coding, will it stop needing humans to grade its code? The evidence points the other way for expert-level evaluation, even as it shifts the work upward.

Demand for high-quality human coding feedback is rising, not falling. Every frontier lab needs a continuous stream of expert human judgment to train, refine, and safety-test models, and human judgment is precisely the thing a model cannot yet reliably supply about its own outputs. As models get stronger, the bar for useful feedback rises: catching a model's remaining mistakes now requires more senior engineers, not fewer, which is exactly why the pay tiers have been climbing toward $200/hr.

At the same time, the low end is under pressure. Routine, generalist annotation is being automated and commoditized, and the same LSE and industry analysis that says AI will not replace programmers wholesale but will commoditize routine implementation applies to this work too. The trajectory of the job is moving from "grade this snippet" toward "design the rubric, build the RL environment, adjudicate the hardest cases." That is a more senior, better-paid role, and it favors engineers who invest in specialization now.

The market backdrop supports staying in: multiple analysts project the AI training data segment growing at a 20%+ CAGR through the decade, with the broader data labeling market heading from ~$2.6B in 2026 to $7B+ by 2031. The honest forecast for a skilled coder: the floor of this work will keep eroding, the ceiling will keep rising, and your job is to position toward the ceiling by getting more specialized and maintaining high quality scores.

Bottom Line

Coding and software evaluation in AI training is a rare thing in remote work: genuinely legitimate, genuinely high-paying, and genuinely merit-based. A competent engineer can realistically earn $40 to $60/hr on the entry-coding tier, and a senior specialist can reach $120 to $200/hr, with the very top full-time roles clearing $200,000 a year. The price of admission is passing a real skills assessment, positioning yourself as specialized rather than generalist, working across multiple platforms to smooth out lumpy queues, and handling your own taxes like the small business you now are. Avoid anything that asks you to pay, apply only on official domains, and treat the top of every advertised range as the specialist ceiling, not the default. Do that, and this is one of the best-paid uses of an engineer's spare hours available in 2026.