Why 'it looks better' is not evidence
Most changes to an AI workflow are small: a reworded instruction, a swapped model, a new tool, an adjusted retrieval query. Each one is easy to eyeball against two or three examples and call an improvement.
The problem is that a handful of manual checks cannot show what happened to the cases that were already working. A change that fixes the case you were staring at can quietly break a pattern you were not looking at. Google's own MLOps guidance treats this as a structural requirement rather than a nice-to-have: production machine learning pipelines are expected to run automated data and model validation before anything reaches production serving, with offline validation first and only then a controlled online rollout such as a canary or an A/B test.
What a golden set actually is
A golden set is a fixed list of real inputs the workflow has seen before, each paired with an output a person has already judged acceptable. It is not a benchmark borrowed from a research paper; it is drawn from your own tickets, documents, or requests, because those are the cases the workflow has to handle correctly.
Open evaluation frameworks such as OpenAI's Evals project formalize the same idea at larger scale: a registry of test cases plus a way to run a model or workflow against them and score the results, so a change can be checked against a stable reference instead of a fresh round of manual spot-checks.
- Real inputs, not synthetic or invented ones
- An expected output or an explicit acceptable range, not just a vibe
- Coverage of the common path plus the two or three failure modes that already happened once
- A stable enough set that a score change means the workflow changed, not the test
Building the first version
Start smaller than feels sufficient. Twenty to fifty cases pulled from real history, each with an owner-approved expected output, will catch most obvious regressions and is small enough that someone will actually maintain it.
Weight the set toward what has gone wrong before, not toward easy cases. If a past incident traced back to a misread date format or a missing field, that exact case belongs in the set permanently, so the same failure cannot silently return in a later change.
Running it before every change, not after a complaint
A golden set that only runs when someone remembers to run it is not a control, it is a habit that will lapse. It needs to run automatically whenever a prompt, model, tool, or retrieval source changes, and the result needs to block or flag the change before it reaches production.
This is the same discipline behind continuous delivery for traditional software, applied to a system whose behavior can shift with a one-line prompt edit: treat the prompt, the model version, and the tool configuration as versioned artifacts, and require the test set to pass before any of them moves forward.
What the score does and does not tell you
A passing golden set means the workflow still handles the cases you already know about. It does not mean the workflow is correct on cases you have not seen yet, and it does not replace the human review step for consequential decisions.
Treat it as a floor, not a ceiling: the set should grow every time a new failure mode is found in production, and a change that passes the existing set but causes a new incident is a signal to add that case rather than a reason to distrust the practice.