An overall deadline is not a design

Most workflow specs state a single end-to-end target: the ticket gets triaged in under two minutes, the claim gets a decision within the hour. That number says nothing about where the time is allowed to go.

An AI workflow typically chains retrieval, one or more model calls, a validation step, and a write-back to another system. If any one of those can run long with no limit, the end-to-end target is really just a hope, not a budget.

Split the budget by step, then enforce it

Allocate the overall target across the steps that make it up, leaving headroom for retries and queueing rather than spending the whole budget on the happy path. A retrieval call, a model call, and a write-back each get their own ceiling.

Enforcement means an actual timeout on each call, not a comment saying it should be fast. A step that exceeds its budget should fail fast into a defined fallback rather than hold the rest of the workflow hostage.

  • Per-step timeout, set in code, not just monitored after the fact
  • Headroom reserved for one bounded retry, not unlimited retries
  • A named fallback for every step that can blow its budget: cached answer, simpler model, or route to a person

Measure the tail, not the average

An average latency number hides exactly the behavior that breaks a workflow. Google's Site Reliability Engineering guidance on service level objectives makes this explicit: a typical request might complete quickly while a meaningful share of requests run many times slower, and that tail is what determines whether real handoffs stay on time.

Track P50 for a sense of typical performance, but manage the workflow against P95 and P99. If one in twenty model calls is slow enough to blow the step's budget, that is not a rare edge case in a workflow that runs hundreds of times a day — it is a regular occurrence that needs a designed response.

Decide what happens when the budget is blown

A timeout without a fallback just converts a slow success into a hard failure. Before the workflow goes live, decide explicitly what a budget breach does: retry once with a shorter timeout, fall back to a cheaper or faster model, serve a cached result, or route the case to a person with the partial evidence gathered so far.

This decision belongs with the same people who set the decision boundary and review paths for the workflow. Latency is a production risk alongside incorrect output, and the NIST AI Risk Management Framework treats reliability and predictable behavior under stress as part of trustworthy AI system design, not a separate infrastructure concern.

Revisit the budget as the workflow changes

A latency budget set during a pilot with light traffic will not hold once volume grows, a vendor changes its model, or a new data source is added to the retrieval step. Re-check P95 and P99 against the budget whenever any upstream dependency changes.

Workflows that survive contact with real volume are the ones where a slow step degrades gracefully into a known fallback, instead of surprising the team the first time a dependency has a bad day.