A golden test set proves the change works. It doesn't prove it's safe.

Evaluation sets, regression suites, and sample-based review all answer the same question: does the new version produce acceptable outputs on the cases we already know about? That's necessary, but it's answered against a fixed, finite set of examples chosen in advance.

Production traffic is not fixed. New document formats arrive, upstream systems change their payloads, and edge cases your test set never sampled show up at volume. The Google SRE workbook makes the underlying point plainly: test environments aren't production, and tests don't cover every scenario, so some defects only surface once real traffic hits the change.

That's the gap a shadow run or a canary closes. Instead of asking whether the change passes a fixed exam, you ask how it behaves on the traffic that's actually arriving right now, before any of that traffic depends on its answer.

Dark-launch the step before anyone sees its output

A dark launch runs the new logic for real users without acting on its output or showing it to anyone. Martin Fowler's description of the pattern is specific: it's for testing the extra load and performance impact of new back-end functionality before that functionality is turned on for real.

For an AI workflow, that means calling the new prompt, model, or retrieval path alongside the current one on live inputs, logging what it would have returned, and routing on the existing logic as if the new path didn't exist. You get real latency numbers, real error rates, and a real sample of outputs to review, without a single decision made by the change you haven't trusted yet.

This step is cheap precisely because nothing is at stake while it runs. If the new path is slow, or it errors on a document shape you didn't anticipate, you find out from logs, not from a customer or an operator noticing something wrong downstream.

  • Log the shadow path's output, latency, and errors alongside the live path's.
  • Never let the shadow path write back, send a message, or change workflow state.
  • Review a sample of disagreements between the two paths before moving to a canary.

Canary the cutover instead of flipping a switch

Once the shadow run looks clean, the next step isn't turning the change on for everyone. It's a canary: routing a small, defined share of real traffic to the new version while the rest keeps running on the current one, then comparing the two.

The SRE workbook's guidance applies directly here: pick a handful of strong, user-visible signals rather than noisy ones. For a workflow, that's usually completion rate, error rate, latency, and the human-correction or escalation rate on reviewed outputs. Avoid judging the canary against last week's numbers; compare it against the control group running at the same time, since time-based noise (volume, document mix, day of week) can make an identical version look different from itself.

Danilo Sato's description of canary releases on Martin Fowler's site is worth keeping in mind: the point is early warning, not a hypothesis test. If the canary's numbers drift from the control's by more than you'd accept, route back to the current version and look at what changed. If they hold, widen the traffic share in stages rather than jumping straight to 100%.

Decide what counts as a regression before the canary starts

Canaries fail at the decision stage more often than the data-collection stage. Teams collect the right metrics, then argue in the moment about whether a 0.4-point rise in the escalation rate is a regression or noise. That argument should happen before the traffic starts moving, not while it's live.

Write down, in advance, the metric, the acceptable range, and who gets paged or notified if the canary crosses it. Keep the list short enough that someone can hold it in their head during a live rollout. A workflow change that needs twelve conditions evaluated to approve isn't ready for a canary; it's ready for more shadow-run time first.

The same discipline applies to rollback. Decide, before cutover, exactly how traffic routes back to the previous version, and confirm that path works before you need it. A canary you can't safely back out of is just an early, smaller version of the full-cutover risk you were trying to avoid.