An evaluation set is a fixed list of real cases. Each has an input and a verified answer, and you score the system against all of them after every change. Create it before the system exists, using material from future users, while discovery is still running. Start with roughly a hundred. Without one, every later decision about models, prompts or search settings rests on impressions.
What it is, and what it is not
The set works like a regression suite with one difference: AI outputs are not identical from run to run, so you score them instead of matching them exactly. The score shows whether a change helped, hurt or only altered the output.
Size matters less than care. Ten cases prove nothing. A thousand is a separate project. A starting size of about a hundred is easy for a person to check by hand, and you can grow it from there.
Build it early for a practical reason. Once people have seen what the system does, the questions they write drift towards what it already handles well.
Where the first hundred cases come from
Draw them from future users rather than from the builders. Engineers drift into questions the system already handles. Users write what they really send: clumsy wording, missing context, a mistaken assumption, or no right answer at all.
| Source | Strength | Weakness |
|---|---|---|
| The existing queue (tickets, emails, search logs) | Real wording and real volume | Mostly the common path |
| Interviews with the people who answer today | They remember the awkward cases | They recall what hurt, not what is typical |
| A "worst cases" session with the same people | Finds edge cases early | Small numbers; needs a facilitator |
| Policy documents and contracts | Gives checkable answers | Questions are written by the author, so they skew easy |
Use all four, and give the first two the most weight. Aim to have the first draft of the set ready before the architecture is fixed, so the results can shape design choices such as which documents to index.
What each case records
A case is more than a question and an answer. Store these fields so the set can be sorted, audited and retired:
- an ID and the date the case was added;
- the input exactly as a user would send it;
- the expected answer, and any other answer that would also be acceptable;
- the document or record that supports the answer;
- a category (see the next section);
- who verified the answer, and when;
- whether the correct behaviour is to refuse or hand over to a person.
An unverified answer makes a case useless. Assign a named subject expert to confirm each one.
Cover six kinds of case
A hundred cases on the easy path will never teach you anything. Spread them across these categories and avoid letting any one exceed about half the set.
| Category | What it tests | Example prompt, in general terms |
|---|---|---|
| Common path | The baseline: what most queries look like | A routine question with a clear answer |
| Real edge case | Exceptions that actually occur | An unusual account type, or a query spanning two policies |
| No answer exists | Whether the system says so | A question the documents do not cover |
| Needs a person | Whether it hands over | A case above the authority limit |
| False premise | Whether it corrects the user | A question about something you do not offer |
| Out of date | Whether it uses current material | A question about a policy that changed this year |
The "no answer" and "false premise" categories are the ones teams skip. They matter most. A system that answers every question will, at times, answer wrongly and sound sure. Human-in-the-loop design covers what should happen to the hand-over cases.
What to measure
Use several numbers. A retrieval-based system has stages, and one accuracy figure shows a failure without showing its location.
| Stage | Measure | How |
|---|---|---|
| Retrieval | Hit rate: was the right passage returned? | Run retrieval alone, with the language model switched off |
| Answer | Accuracy against the checked answer | Grade each answer against the expected one |
| Citation | Does the cited source back the claim beside it? | Compare each claim with its cited passage |
| Refusal | Did it decline where it should? | Score the "no answer" and "false premise" categories separately |
| Cost and speed | Cost and time per answer | Log both for each run |
Measure retrieval first. If the right passage never arrives, no later stage can recover. Retrieval beyond the demo explains why. A tidy answer citing an unrelated document does more harm than a refusal, since it survives a quick review.
How to grade
Exact matching works for structured extraction and very little else. For open answers, three methods are available, and you will probably combine them.
Structural checks come first. Is a source cited? Is the figure the same as in the record? Is the answer inside the permitted list? They are cheap and catch many failures.
Human grading gives the most accurate result but cannot keep pace with volume. Use it for the baseline and for spot checks on whatever automatic method you adopt.
A language model as judge scales. It has known biases. Zheng and colleagues, in the MT-Bench and Chatbot Arena paper, name position, verbosity and self-enhancement biases, and report that strong judges reached over 80 per cent agreement with human preferences in their tests. Treat that as a reason to use a judge and a reason to check it. Grade a sample by hand, compare, and report the agreement before you trust the automatic score.
Keep a hold-out
Split the set in two. Use the development half while you tune prompts, search settings and models. Keep the other half untouched until a release candidate is ready. If you tune against every case, the score improves while real quality stays flat.
Use it as a gate
The set earns its value only when it runs. Run it on every change to the model, the prompt, the retrieval settings or the index, and block a release that regresses in any category.
The gate also makes the model replaceable. If a provider retires a version or a cheaper one appears, you measure the swap instead of debating it. Teams with no set upgrade and hope. The fields an AI audit trail must hold show how to record which set and which versions each score refers to.
Keep it alive
A set that never changes goes stale. Add every real failure from production as a new case. Retire cases whose policy has changed. Give the set a version number and an owner. Review the category mix each quarter, since users drift towards new questions.
What it will tell you early
Expect the set to show, early, how far the real system sits below the demo. That result is the set working. The other way to learn it is in production, from a user, on a case nobody tried, when the fix costs more and the user has lost confidence. See why enterprise AI agent projects get cancelled for how that pattern ends.
Questions teams ask
How many cases do we need?
Start with about a hundred spread across the six categories, then grow it from production failures.
Who writes the expected answers?
A subject expert on your side, not the engineers. Record who confirmed each one.
Can the model write the cases?
It can draft variations of cases you collected, which is useful for coverage. The original cases must come from real users, and a person must verify every answer.
Does the set need to change when we change models?
The cases stay. The scores and the version record change. If the business policy changes, update the affected cases and record the change.
Where Brihat fits
Brihat Infotech's custom AI and LLM development delivers a scored evaluation set and the scripts that run it with each system, and its MLOps work places evaluation gates in the release pipeline. Start a conversation.