Yes, you can get paid to write problems that AI models fail. Labs and the programs that staff them pay subject experts for original questions with a worked solution and a grading rubric. On 15 November 2026 public listings paid by the hour or per approved item: xAI listed math tutors at $45 to $75 an hour, micro1 listed attorneys who build legal reasoning tasks at $80 to $150 an hour, and Alignerr listed Lean 4 proof work at $170-200. Most ask for a graduate degree, a license or competition results. You sign the rights to each item over to the program, and the work comes in bursts.
- An item is three things: the problem, a solution another expert can check, and a rule for grading an answer.
- Published pay on 15 November 2026 ran from $45 to $150 an hour on most listings and $170-200 on one formal-proof listing.
- Per-item projects pay only for items a reviewer approves. Ask how long an item takes before you price your time.
- What you submit usually belongs to the program. Never submit a question your university, publisher or employer owns.
What is AI benchmark question writing?
You write original problems in your own field that current AI models answer wrongly, each with a correct solution and a rule for grading it. Labs use the items to measure models, and some to train them.
The clearest description we found is in a Mercor drug discovery listing: "Three things per task: the prompt, the data room, and the grading document." The prompt is the question. The data room is whatever the solver needs to read. The grading document says what a right answer contains.
A micro1 listing for litigators asks them to "Build complex legal reasoning tasks from scenarios like those you handle in practice, with realistic supporting materials" and to "Define what makes an answer right, so another attorney reviewing it would reach the same verdict." A micro1 listing for chemists asks them to "Design and write challenging, expert-level scientific problems in computational chemistry, structural biology, and drug discovery, with clear, verifiable solutions." Outlier's home page lists "Write a Challenging Prompt" and "Create Grading Rubrics" among its common tasks.
The public benchmarks show what the finished product looks like. The FrontierMath paper describes "hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians", and reported that models at the time could "solve under 2% of problems". Humanity's Last Exam "consists of 2,500 questions across dozens of subjects" and was "developed globally by subject-matter experts". Most paid work is for private sets like these that are never published.
Who gets hired to write benchmark questions?
People who can prove depth in one field: a graduate degree, a professional license or results in competitions. Teaching and exam-setting experience helps, because the job is writing hard questions that are still fair.
The table shows public listings read on 15 November 2026 whose work includes writing or auditing test items. Alphabetical by program. Pay is as published, and a range is the listing's own minimum and maximum.
| Program | Listing | What it asks for | Published pay |
|---|---|---|---|
| Alignerr | Formal Verification Scientist (Lean 4 & Mathlib) | The title names the skill: formal proofs in Lean 4 | $170-200 an hour |
| Handshake AI | Math Expert | PhDs and postdocs | Up to $75 an hour, depending on the project |
| Mercor | SWE-Bench Task Auditor | Remote, United States | $70 to $90 an hour |
| micro1 | Computational Chemistry, Structural Biology & Drug Discovery Scientist (PhD) | A PhD and hands-on lab research | $45 to $90 an hour |
| micro1 | Litigation Attorney | J.D., active U.S. bar license, 3 to 15 years of practice | $80 to $150 an hour, paid per approved task |
| xAI | AI Tutor, Math | A master's or PhD in mathematics, or an International Mathematical Olympiad medal | $45 to $75 an hour for US-based candidates |
The xAI listing accepts "a medal in the International Mathematical Olympiad (IMO) or a high-level mathematics competition" in place of a graduate degree. That is the one place we found competition results written into a requirement.
The call for questions for Humanity's Last Exam set its own bar: "It's strongly encouraged that question-writers have 5+ years of experience in a technical industry job", or be a PhD student or above. For pay across all expert work, see AI training jobs for PhDs and postdocs.
How is pay set: per accepted item or by the hour?
Both exist. Hourly listings pay for logged time, and per-item projects pay only when a reviewer approves the item.
The micro1 litigation listing is per item. It shows an hourly equivalent of $80 to $150, says attorneys are paid per approved task, and gives "Time per task: ~8 hours for task creation". Turing has posted coding work at "$25 per approved task". Outlier's FAQ says it "pays for completed work, either hourly or per task." Under per-item terms a rejected item can mean unpaid hours, so ask two things before you start: how long a typical item takes, and whether a returned item can be fixed and resubmitted.
Open calls pay prizes, not wages. The Humanity's Last Exam call offered "a chance to win money from a $500,000 prize pool": "The top 50 questions will earn $5000 each. The next top 500 questions will earn $500 each." Everyone else who submitted earned co-authorship at most. The FrontierMath paper does not say what its problem authors were paid, and we found no public figure, so treat any number you see quoted for it with care.
Why do benchmark items get rejected?
Four faults cause most rejections: a model can already solve it, the answer is ambiguous, the solution cannot be checked, or the item is not original. The reviewer is usually another expert in your field.
The Humanity's Last Exam guidelines put the bar in two lines: "Questions should be difficult for non-experts and not easily answerable via a quick online search" and "Questions must be your own work and not copied from others." The micro1 litigation listing tells authors to "Flag tasks that turn out to be ambiguous or ungradable rather than submitting them." DataAnnotation says it reviews work "for thoroughness and accuracy rather than speed" and that "subpar or problematic contributions may result in loss of access to future projects."
- One defensible answer. If two specialists could reasonably disagree, the item is ungradable.
- Everything needed is inside the item. Define notation and state the jurisdiction, the units or the edition of the standard.
- Hard because of the field, not because of a trick. Hidden wordplay teaches a model nothing.
- A solution a peer can follow step by step, with the authority or derivation for each step.
- Not findable. Search for your own question before you submit it. If the answer is one search away, a model has probably read it.
- Tested, where the project lets you. If the project gives you a model to try the item on, record what it got wrong and why.
Who owns the questions you write?
Usually the program or its client, from the moment you submit. Read the assignment clause before you write anything you might want to reuse.
Outlier's terms of use, read on the same day, are typical of the wording: "you irrevocably assign all right title and interest throughout the world, including Intellectual Property Rights, in and to the Work Product to the Company." The same terms have you state that "you are the sole and exclusive owner of all User Content you submit". Together those two lines mean you cannot later publish the item, put it in your own course, or sell it again, and you must have owned it in the first place.
That second point catches academics and employees. Exam questions written for a university course, problems from a paper under review and anything built from an employer's files may not be yours to assign. The Humanity's Last Exam call allowed "questions from past exams you've given if they're not accessible to the public", which is a rule for that project only. If you have an employer, read what an employer can and cannot stop before your first item.
Is benchmark writing steady work?
No. A project opens when a lab needs a new test set and closes when the set is full.
Programs say this themselves. Handshake AI: "Projects may be paused or closed with little notice based on the needs of our partner labs." Mercor: "We can not guarantee a minimum number of hours of work per week, or when a project may come to a close." Treat it as project income on top of a job, a practice or a research post. No tasks available? covers what each program says about gaps.
How to tell us your field
Origin Data Partners takes inquiries from experts in every field and every country. Say what you could write hard problems about: the subfield, your degree or license, and anything that shows you set questions for a living, such as teaching, exam writing, peer review or competition coaching. We reply by email with what is open to someone with your background.
Tell us your field and the kind of problems you can write, and we'll email you what is open. It's free.
Origin Data Partners is independent of the programs on this page. A program may pay Origin when Origin introduces someone who joins it, which never changes what you are paid (how Origin is paid).
Common questions
Can you get paid to write questions that AI gets wrong?
Yes. On 15 November 2026 public listings from xAI, micro1, Mercor, Handshake AI and Alignerr paid experts to write, solve or audit hard test items, mostly at $45 to $150 an hour. The item must be original, have one checkable answer and come with a grading rule.
What degree do you need for AI benchmark writer jobs?
Most listings ask for a master's or PhD in the subject, or a professional license for law, medicine and similar fields. One xAI math listing also accepts an International Mathematical Olympiad medal or a high-level competition result.
How is pay calculated for writing problems for AI?
Either by the hour or per approved item. A per-item project pays when a reviewer accepts the item, so the real hourly figure depends on how long an item takes you and how many are returned.
Are there AI math problem writer jobs for olympiad medalists?
Yes. An xAI listing for math tutors read on 15 November 2026 accepts a medal in the International Mathematical Olympiad or a high-level mathematics competition, and listed $45 to $75 an hour for US-based candidates.
What is an AI rubric writer?
Someone who writes the grading rule for a test item: what a correct answer must contain, what loses credit and how partial answers score. On most projects the person who writes the problem writes its rubric too.
Is getting paid to stump AI steady work?
No. Projects open and close with each lab's need for a new test set, and programs state that they can pause a project at short notice. It suits people who already have a main income.
What were FrontierMath problem writers paid?
The FrontierMath paper does not publish what authors were paid, and we found no public figure. The one open call with published money was Humanity's Last Exam, which paid prizes of $5000 and $500 for top questions from a $500,000 pool.
