Uptime is not the same as working correctly

A conventional service is either up or down, and uptime monitoring catches most of what matters. An AI workflow can stay fully available while quietly degrading: a model call that keeps hitting its token limit and truncating output, a tool call that fails and silently falls back to a weaker path, a retrieval step that returns stale or empty context. None of that trips a standard health check.

The practical implication is that an AI workflow needs its own monitoring layer, built around what actually happens inside a run — not just whether the endpoint responds. That layer is usually called observability, and for generative and agentic systems it centers on three signal types: traces, metrics, and events.

Three signals worth capturing

OpenTelemetry's guidance on generative AI observability groups what to capture into three categories, and the distinction is useful for scoping a first monitoring pass rather than trying to log everything at once.

  • Traces — the parameters and outcome of each individual model or tool call: settings like temperature, token counts, latency, and errors.
  • Metrics — aggregated performance over time: request volume, average latency, and token consumption, which is what cost and capacity planning actually run on.
  • Events — the granular record of a specific run: the prompt sent, the response received, and the decision the workflow made with it, which is what a debugging session actually needs.

Trace the whole path, not just the model call

An agentic workflow's slowness or failure rarely comes from the model call itself. It comes from a chain of model calls, tool invocations, and retries that a single log line can't show. OpenTelemetry's semantic conventions for generative AI structure this as a trace tree: a top-level span for the agent's run, with child spans for each model interaction and each tool call underneath it.

That structure is what makes a failure diagnosable instead of just visible. Standard attributes — which model served a request, how many input and output tokens it used, why it stopped generating, how long the call took — let a team tell the difference between a slow model, a flaky integration, and a workflow stuck in a retry loop, without guessing from a support ticket.

Feed the signals back into governance, not just a dashboard

Collecting traces, metrics, and events is only half the job. The NIST AI Risk Management Framework's Measure function is explicit that monitoring needs defined thresholds and an assigned owner — signals that nobody is accountable for reviewing don't reduce risk, they just accumulate in a dashboard.

In practice, that means the same person or team responsible for the workflow's human review step should also own its monitoring signals, on a set cadence: what triggered an escalation this week, which tool calls failed and why, whether latency or token cost is drifting. Observability built this way becomes part of the operating discipline, not a separate tool bolted on after the first outage.