What are AI evals? How to know if your AI feature actually works

1. 1

   Why can’t you just try it a few times?

   AI answers can change from run to run, and a prompt change that fixes one case can quietly break three others. Trying a few questions and saying "looks good" is what OpenAI calls a "vibe-based" eval, and it lists it as an anti-pattern.

   Think of a mock test series before JEE, NEET or the SAT. Same kind of paper, a fixed marking scheme, and a score you track week after week. An eval is a mock test for your AI.

   In his post "Your AI Product Needs Evals", Hamel Husain argues that most unsuccessful AI products share one root cause: no solid way to evaluate them.

   Key idea

   If you are not measuring it, you are guessing.

2. 2

   What goes into an eval?

   An eval has three parts. A golden dataset: a list of realistic inputs, ideally from real users, including tricky edge cases, each with the expected answer or the key facts it must contain. A grader that decides pass or fail. And a score, like 17 out of 20.

   Anthropic’s docs give three design rules: make it task-specific (mirror what real users ask), automate grading where you can, and prefer more test cases with automatic grading over a few hand-graded ones.

   • Code grading: exact or keyword match. Fastest and most reliable, but misses nuance.
   • Human grading: most flexible and high quality, but slow and expensive.
   • LLM grading: another model grades with a rubric. Fast and flexible, but test it before you trust it.
3. 3

   How do you find what to test?

   Don’t start by imagining problems. Start by reading real outputs. This is called error analysis, and the AI Evals FAQ on hamel.dev calls it the most important activity in evals.

   Collect about 100 varied conversations and read at least the first 30 yourself. Write a short note on anything wrong. Then group the notes into categories and count them. Now you know which failure matters most. In the FAQ’s words: "Write evaluators for errors you discover, not errors you imagine."

   Chat  Note                               Category
   #07   Quoted last year’s fee               Outdated info
   #12   Replied in Hindi to English question Wrong language
   #19   Invented a scholarship               Made-up fact
   #23   Invented a hostel rule               Made-up fact
   
   Counts: Made-up fact 9, Outdated info 5, Wrong language 2

4. 4

   What is LLM-as-a-judge, and when can you trust it?

   For answers that code can’t check, you can ask a second model to grade them against a rubric, a short list of what a good answer must do. Anthropic’s tips: write a detailed rubric, ask for a fixed output like PASS or FAIL, let the judge reason before it decides, and use a different model from the one that wrote the answer.

   Judges have biases. OpenAI notes they can favour longer answers and be swayed by the order in which answers appear. So first grade 30 to 100 answers yourself, run the judge on the same ones, and only rely on it if it agrees with you most of the time. Simple pass or fail labels work better than 1 to 5 scores, which are hard to apply consistently.

   You are grading a college help bot.
   PASS only if the answer:
   1) uses only facts from the policy text, and
   2) gives a clear number or date when asked.
   Otherwise FAIL.
   
   Policy: {policy}
   Question: {question}
   Answer: {answer}
   
   Think step by step, then output PASS or FAIL.

5. 5

   A worked example: improving a leave-policy bot

   Here is the loop in action, with made-up numbers. You collect 20 real questions about the leave policy, with the right answers. Version 1 of your prompt passes 14 out of 20 (70%). Error analysis shows 4 failures are made-up facts.

   You change the prompt to answer only from the policy text and to say "I don’t know" otherwise. You rerun the same 20 questions: 18 out of 20 (90%), and none of the 14 that passed before now fail. That last check matters: it catches regressions, things that used to work and broke.

   OpenAI’s advice is to keep evaluating: rerun your evals whenever you change the prompt, model or data, and keep adding new failures you see in real use.

   1. Collect real questions↓→
   2. Read outputs, label failures↓→
   3. Write graders↓→
   4. Run and score↓→
   5. Change prompt or model↓→
   6. Rerun before release

   Key idea

   Same test set, every change, every time. That is how a demo becomes a product.

From the people who built it

Straight from the source

The original idea, then the same idea taken apart, then how you use it today.

Hamel Husain

Machine learning engineer with 20+ years of experience, formerly at Airbnb and GitHub, who now helps teams measure their AI systems and co-teaches a course on evals.

Your AI Product Needs Evals (blog post, March 2024)

1 · Understand

AI products stall when teams judge quality by feel. The teams that improve fast build evals for their own users’ tasks, look at real outputs constantly, and use what they find to change prompts, data and code.

2 · Break it down

• Level 1, unit tests: quick, cheap checks written as code (does the answer include a date? is the JSON valid?), run on every change.
• Level 2, human and model review: log real conversations, have people read them, and train an AI judge to agree with those people.
• Level 3, A/B tests: once the product is mature, check whether changes actually help real users.
• Through all of it, keep reading the data. He writes, "You can never stop looking at data."

3 · Use it

Start at Level 1 today. Write five simple code checks for your bot, run them every time you change the prompt, and spend 20 minutes a week reading real conversations.

Lianmin Zheng, Wei-Lin Chiang, Ion Stoica and colleagues (UC Berkeley, UC San Diego, Carnegie Mellon, Stanford and MBZUAI)

The team behind MT-Bench and Chatbot Arena, who tested at scale whether one AI can fairly grade another.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (paper, 2023)

1 · Understand

Strong models like GPT-4 can grade open-ended answers and agree with human preferences over 80% of the time, about as often as humans agree with each other. But judges have clear biases you must test for.

2 · Break it down

• Position bias: the judge may favour whichever answer it sees first.
• Verbosity bias: it may prefer longer answers even when they are not better.
• Self-enhancement bias: it may favour answers written by itself or a similar model.
• Limited reasoning: judges can get maths and logic questions wrong when grading them, so check those verdicts with extra care.

3 · Use it

Before trusting an AI judge, grade a sample yourself and compare. Swap the order of the answers you show it, and use a different model as judge from the one being tested.

6 terms

Words to know

Eval

A repeatable test that measures how well an AI feature does a specific job.

Golden dataset

A fixed set of realistic inputs with known good answers, reused every time you change something.

Rubric

A short checklist of what a good answer must do, used by a human or a model to grade.

LLM-as-a-judge

Using a second AI model to grade answers against a rubric, checked first against human grades.

Error analysis

Reading real outputs, noting what went wrong, and counting failures by category.

Regression

Something that used to work but broke after a change. Evals help you catch it before users do.

1 video · plays here

Watch

Hand-picked from the people who explain this best. The video loads only when you press play.

Hands-on · 10 minutes or less

Try it

Build a 10-question eval in a spreadsheet

1. 1Open a new Google Sheet. Add columns: Question, Must include, Model answer, Pass or Fail, Note.
2. 2Write 10 questions on a topic you know well, like your college’s admission rules or your favourite app’s FAQ. Include 2 tricky ones and 1 whose answer is not in any official source.
3. 3Paste each question into a chatbot and copy its answer into the sheet.
4. 4Mark Pass or Fail yourself using the Must include column. Add a short note for each fail. Count your score.
5. 5Change your prompt (for example, add "If you are not sure, say so") and rerun all 10. Did the score go up? Did anything that passed now fail?
6. 6Bonus: ask a different chatbot to grade each answer with a PASS or FAIL rubric. How often does it agree with you?

Open a new Google Sheet

3 questions · instant feedback

Check yourself

Pick an answer to see if you are right, and why.

Question 1

Question 1 of 3

What is a golden dataset?

AThe data used to train the modelBA fixed set of realistic inputs with known good answers that you rerun after every changeCA list of your best customersDThe model’s most confident answers

Question 2

Question 2 of 3

Why should you compare an LLM judge with your own grades before trusting it?

AJudges are always wrongBJudges can have biases, like favouring longer answers, and may not match what you mean by goodCIt makes the eval cheaperDBecause judges can only grade code

Question 3

Question 3 of 3

You just launched an AI feature. What is the best first step toward evals?

ABuy an evaluation platformBWrite 50 tests for problems you imagine might happenCRead real outputs, note what went wrong, and count failures by typeDAsk users to rate every answer from 1 to 10

Lesson 24 of 36

Done? Keep going

Mark this lesson complete to keep your place. Progress is saved on this device, no login.

Mark lesson complete

Up next · Lesson 25 · 13 minHow voice AI agents workA voice AI agent listens, thinks and speaks, and can take actions like booking an appointment. Most are built as a cascade: speech-to-text turns your voice into text, a language model writes the reply and calls tools, and text-to-speech reads it out. Newer speech-to-speech models, such as OpenAI’s Realtime API, do all three in one model. The hard parts are speed (replying in about a second), knowing when you have finished speaking, and stopping politely when you interrupt.← Previous: Subagents and multi-agent systems

People also ask

Questions people ask

What are evals in AI?+

Evals are repeatable tests for AI features. You run a fixed set of realistic inputs through your system, grade the outputs with code, people or another model, and track the score every time you change the prompt, model or data.

What is LLM-as-a-judge?+

It means using one AI model to grade another model’s answers against a rubric. It is fast and scales well, but judges have biases, such as preferring longer answers, so check them against human grades before relying on them.

How many test cases do I need for an eval?+

Start small and real. A useful habit is to read about 100 varied real conversations and label at least 30 yourself to find failure types, then turn those into tests. More test cases with automatic grading usually beat a few hand-graded ones.

5 sources · checked Nov 2026

Sources and further reading

• OpenAI docs: Evaluation best practices
• Anthropic docs: Define success criteria and build evaluations
• Hamel Husain: Your AI Product Needs Evals
• hamel.dev: AI Evals FAQ
• Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Related on this site

Previous: What is an AI agent?Why AI makes things upNext: Your first AI automation in n8nAI product manager guideAI engineer career guideFree AI skills resources