Last updated: September 21, 2026
By Kris Drouet, Engineering Executive, in partnership with KORE1
LLM evaluation is a repeatable scored test suite for model output, run against a fixed set of real examples, so quality becomes a number you can watch instead of a feeling somebody had in a demo. Build it before the feature, not after the incident. Thirty labeled examples and one scoring rule beat any platform you could buy this quarter.
A team I worked with shipped a feature that drafted replies to borrower document requests. Good idea. Real problem. Three engineers clicked around in it for an afternoon, everything looked right, and it went live on a Tuesday.
Friday a servicing rep escalated. The thing had told a borrower, in a confident and beautifully written paragraph, that a pay stub was already on file. It was not on file. That borrower had been waiting nine days.
The bug was not the interesting part. I asked how often it was doing that, and nobody in the room could answer, not the engineers who built it and not the vendor whose model sat under it. One in four thousand. One in twenty-five. Nothing had ever been measured, so the only honest answer available was a shrug.
I have spent twenty-five years asking people to show me the data. It works on cloud spend, on velocity, on the build versus buy call. Then somewhere in the last two years a lot of otherwise disciplined teams stopped applying it to the one component in the stack that makes things up, which is the only place in the whole system where the failure is quiet. If you land on build, you are also hiring AI and machine learning engineers, and the eval suite is what you will interview them against.
An Eval Is Not a Unit Test
An eval is a scored test that runs your AI feature against a fixed set of real inputs and grades each output against a rule you wrote in advance. Unit tests answer yes or no. Evals return a distribution, because the same prompt can produce a different answer on every run, and the number that matters is how often it lands.
Normal testing instinct fails here. Engineers hunt for crashes. This failure mode does not crash.
The 2025 Stack Overflow Developer Survey put a number on it that I have quoted in about nine meetings since. Asked for their biggest frustration with AI tools, 66% of developers picked “AI solutions that are almost right, but not quite.” Not wrong. Almost right. In the same survey 84% use or plan to use AI tools, 46% actively distrust the accuracy of what comes back, and only 33% trust it, which is the whole industry in three numbers.
Everyone is shipping it. Nobody believes it. And the failure that bothers them most is the exact one a pass-fail test cannot see, because almost right passes.
Thirty Examples Beat a Platform
First question I always get is which tool to buy. Wrong first question.
Your first suite is a spreadsheet and a script, and it can be running by Friday:
- Pull 30 real inputs from your own logs, support queue, or ticket history. Real ones. The messy ones your demo dataset quietly excluded.
- Write down what a correct output looks like for each. Expensive part. A few hours of somebody senior, and no way around it.
- Score. Pass, fail, or 1 to 5 if the task has shades.
- Re-run whenever anybody touches the prompt, the model version, the retrieval index, or the system message.
Thirty is not a statistically satisfying number and I will not pretend otherwise. It is enough to catch a regression before a customer does, which is a lower bar and the one that pays. Teams that start at thirty reach two hundred inside a quarter. Teams that start by evaluating vendors have nothing running six weeks later.
Weight the set toward cases that hurt. A wrong answer about a missing document costs you a compliance conversation. A wrong answer about office hours costs nothing.
Who Grades the Grader
Hand-grading works at thirty. It does not work at two thousand, so a model starts doing the grading, and this is where careful teams get careless.
The work that made LLM-as-judge respectable is Zheng et al., 2023. Strong judge models agreed with human preferences more than 80% of the time, roughly the rate two humans agree with each other. Real result. It is why the technique spread.
The follow-up is less comfortable. Abel Yagubyan’s April 2026 preprint “The Coin Flip Judge?” ran two OpenAI judge models across 29 tasks with repeated trials and found pairwise preferences flipped 13.6% of the time on average, with 28% of questions flipping on more than a fifth of runs. One question flipped 56%. Two judges agreed with each other only 76% of the time. Semantically equivalent prompt templates moved the majority outcome in a quarter of cases, and recovering the 50-trial verdict with 95% confidence took 11 repeated runs.
Single author, 29 tasks, two models from one vendor. Treat the exact figures as directional. The direction is the point.
So the judge is a component under test, not an oracle:
- Pin the judge model and its version. A silent provider upgrade moves your quality metric while nothing in your product changed.
- Keep 30 to 50 human-labeled items as a control, measured the same way. If judge agreement slips, the dashboard starts lying before the product does.
- Repeat anything you would act on. One run is a coin flip in a lab coat.
- Rewriting the judge prompt changes the instrument. Re-baseline.
Three Places the Suite Has to Run
A suite that runs when somebody remembers it is a document, not a control. Three homes. Each catches something the others miss.
| Where it runs | What it catches | What it costs you |
|---|---|---|
| Pre-merge, in CI | Prompt edits and retrieval changes that quietly break a case | Minutes per pull request, plus inference spend |
| Pre-release, full offline set | Model version swaps and anything the CI subset skips | An hour of wall clock, once per release |
| Production, sampled traffic | Real inputs nobody imagined, and slow drift | Sampling infrastructure and a standing review slot |
Row three is the one teams skip, and it is the one that would have answered my question on that Friday. Sample a few hundred real outputs a week, score them, plot the line. Drift arrives as a slope. Never as an alert.
If the audience is a risk committee rather than engineers, the language already exists. The NIST AI Risk Management Framework gives measurement its own MEASURE function and expects it to start before deployment and continue across the lifecycle. NIST followed in August 2026 with the draft TEVV-Athlon framework, a four-stage method for building custom AI assessments that is scoped to cover large language models and agentic systems rather than the classifiers most governance language still assumes. Comment closes October 6, 2026. Worth knowing before your auditor knows it.
What Teams Ask Me Before Their First Eval
How Long Does the First Eval Suite Take to Stand Up?
About a week of one engineer’s part-time attention for a thirty-case suite, and most of that week goes to deciding what a correct answer is rather than to writing code.
The labeling is the work, and every team underestimates it. What surprises them is the disagreement that surfaces in the first session, because three people labeling the same thirty outputs differently means you never had a shared definition of quality to begin with.
Can We Just Use Public Benchmark Scores Instead?
No, because a public benchmark measures general capability on somebody else’s data, and your users will fail you on the specific slice that benchmark never contained.
Benchmarks help you narrow a shortlist. They are not evidence about your feature. A model near the top of a leaderboard can still be wrong about your document types, your tone rules, and your edge cases, and the leaderboard cannot warn you, because it never saw any of them.
Our Vendor Says They Handle Evaluation. Is That Enough?
Not on its own. A vendor grades their system against their definition of correct, and the gap between their definition and yours is where your incidents live.
Ask for the eval set. Ask for the rubric. Ask what gets measured in production rather than at release. A vague answer is a finding, and it belongs beside the other findings in the four gates you clear before signing an AI vendor contract.
What Do We Do When the Score Drops After a Model Upgrade?
Roll back first, diagnose second. A pinned older model with a known score beats a newer one whose behavior on your cases is unverified.
This is the scenario the suite exists for. Without it a provider updates a model, quality shifts, and a customer tells you three weeks later. With it, CI tells you the same afternoon and the rollback is a version string.
Who Should Own Evals, Engineering or Product?
Engineering owns the harness and the pipeline. Product or a domain expert owns the rubric, because defining a correct answer is a business judgment wearing a technical costume.
Split it any other way and one of two things happens. Engineers invent a definition of quality the business does not share, or product writes a rubric nobody can compute. Put both names on the document.
Run the First One This Week
Thirty cases. One scoring rule. A script anybody can run in ten minutes. That is the whole starting position, and it is small enough that the only real reason not to have one is that nobody decided to own it.
The teams that survive contact with production are rarely the ones with the best models. They are the ones who can tell you, on a Friday afternoon with a servicing rep on hold, how often the thing gets it wrong. Everything downstream gets easier after that, including the harder conversation about what the feature costs once real volume hits it.
If nobody has the bandwidth to own this, that is a staffing problem wearing a tooling problem’s clothes. KORE1 places AI and ML engineers on contract engagements and direct hire, and its twelve-month retention on placements runs 92%, which matters here because most of the value in this role is context accumulated about your specific failure modes. Talk with a KORE1 recruiter before the job description gets written. The title for this work is not settled, and the wrong one filters out the people you want.
Connect with me on LinkedIn and send me your rubric. I will tell you which of your thirty cases is doing no work.
