Most early benchmarks are wrong. The golden set is too small, the scoring rubric is half-formed, the judge model drifts, and the test distribution barely overlaps with what the system will see in production. That is not a reason to skip benchmarking — it is the reason to do it earlier, more often, and with lower expectations of any single number.
The job of an early benchmark is not to declare a winner. It is to build a measurement instrument that becomes less wrong over time, in a structured way, while the system underneath it is still moving. Teams that treat evaluation as a deliverable to be finished after the model is built tend to ship systems they cannot defend. Teams that treat the eval harness as a first-class artifact — versioned, reviewed, and iterated — tend to find the real failure modes before customers do.
Why early-stage AI benchmarking looks broken
Early benchmarks fail in predictable ways. The first is sample size. A team runs forty hand-picked prompts, sees the new model win on twenty-six of them, and declares a six-percentage-point lift. The confidence interval on that estimate is wide enough to swallow the entire decision. The second is selection bias. The examples in the golden set were chosen because someone found them interesting, which means they over-represent the failure modes the team already knew about and under-represent the ones it did not.
The third is rubric drift. The person scoring outputs in week one is not scoring them the same way in week six, because the team has learned what good looks like in the interim. The fourth is judge instability — when an LLM-as-judge is grading outputs, small changes to the judge prompt, the judge model version, or the temperature can shift the leaderboard by more than the underlying system improvement. Any one of these alone is survivable. Stacked together, they produce a benchmark that confidently points the wrong direction.
The trap is treating these problems as reasons to delay evaluation until the system is mature. By then the team has shipped six revisions on intuition, the failure surface has compounded, and there is no historical signal to recover. Better to build an instrument that is openly flawed but improving than to pretend measurement starts at v1.0.
What a useful eval harness looks like before the system is finished
An eval harness for an unfinished system has different requirements than one for a production model. It needs to be cheap to run, easy to extend, and explicit about what it is not measuring. The harness should produce three classes of artifact every time it runs: per-example scores with traces, aggregate metrics with confidence intervals, and a diff against the previous run on the same examples.
Cheapness matters more than people admit. If the harness takes four hours and two hundred dollars to run, it gets run twice a week and the team flies blind between runs. If it takes twelve minutes and a few dollars, it gets run on every meaningful change, and the regression surface becomes visible. Cost-per-run is a design constraint, not an afterthought.
Extensibility matters because the things worth measuring change as the system matures. Early on, the questions are about basic capability — does the system do the task at all, does it refuse appropriately, does it stay in format. Later, the questions shift to robustness, calibration, and distribution shift. The harness should make it cheap to add a new metric without rewriting the runner.
How to build a golden set that survives contact with reality
A golden set is a curated collection of inputs with known-good behavior. The mistake most teams make is treating it as a fixed asset — built once, frozen, then used forever. In practice the golden set should be a living artifact with structure. Start with three layers: a small set of canonical examples that define the task, a stratified sample drawn from real or realistic traffic, and an adversarial layer that probes known failure modes.
Stratification is the part teams skip. If production traffic has a long tail of input types, a uniformly sampled golden set will under-represent rare-but-important cases. Define the axes that matter — input length, domain, user intent, language, complexity — and sample so that each cell has enough examples to support a per-stratum estimate, not just a single aggregate number.
Hold out a portion of the set from any tuning loop. The moment the team starts iterating on prompts or fine-tuning based on the full golden set, the headline number becomes optimistic. A clean holdout, scored less frequently, is the only honest signal of whether improvements generalize.
- Canonical layer: ten to fifty hand-built examples that define what the task means. These rarely change.
- Stratified layer: drawn from real traffic where possible, sampled across the axes that drive behavioral differences.
- Adversarial layer: known failure modes, edge cases, jailbreaks, distribution-shift probes — grows over time as new failures are discovered.
- Holdout layer: a fraction of each layer that the team commits not to look at during iteration.
Why scoring is harder than running the model
Generating outputs is the easy part. Deciding whether each output is good is the part that quietly determines whether the benchmark is useful. For tasks with a clean reference answer, exact match or structured comparison works. For most enterprise tasks — summarization, extraction with judgment calls, reasoning, agentic workflows — there is no reference, and scoring requires either human judgment or a model-based judge.
Human scoring is the gold standard and the bottleneck. Use it to calibrate, not to scale. A few hundred human-scored examples, with at least two raters per example and an inter-rater agreement check, become the ground truth against which any automated judge is validated. If the automated judge disagrees with humans more than humans disagree with each other, the judge is not yet trustworthy.
LLM-as-judge can scale, but it needs the same discipline as any other model. Pin the judge model and version. Write the rubric down. Run paired comparisons rather than absolute scores where possible — judges are more reliable at picking a winner than at assigning a number. Measure judge-human agreement on a sample every time the judge changes. When the judge drifts, the leaderboard drifts with it, and the team will not notice unless someone is watching.
Reading the numbers without lying to yourself
Report intervals, not point estimates. A benchmark result of sixty-two percent means very little without knowing whether the ninety-five percent interval is plus or minus two points or plus or minus fifteen. For small golden sets, bootstrapping over the examples is straightforward and informative. Without intervals, every reported improvement gets interpreted as real, and the team accumulates a backlog of changes that were probably noise.
Track per-stratum performance, not just the aggregate. A system that improves on average while regressing on the highest-stakes stratum is a system that has gotten worse for the customers who matter most. The aggregate number is for executives. The stratum breakdown is for the people responsible for the system not breaking.
What to do on Monday
Stand up a thin eval harness this week, even if it scores ten examples with a rubric you will throw away. Version it. Run it on every meaningful change. Add a holdout. Add confidence intervals. Add a second rater. The first useful benchmark is the one that exists; every subsequent improvement makes it less wrong.
The teams that ship defensible AI systems are not the ones with the cleanest benchmarks at the start. They are the ones who treated evaluation as an engineering discipline from day one — instrumented early, iterated honestly, and refused to let a single confident number stand in for a structured understanding of where the system works and where it does not.