A retry is not free once a workflow writes to real systems

In a demo, retrying a failed step just means running the code again. In production, a workflow step is a charge, a support ticket, an outbound email, or a written field in a CRM or ERP. Repeating that action is not neutral — it can double-charge a customer, double-send a message, or create a duplicate record that the next system in line has to reconcile.

The answer is not to avoid retries. Timeouts, rate limits, and dropped connections are routine in distributed systems and are usually short-lived, so retrying is still the right first response to a failure. The real fix is to make each write-back step safe to repeat, so a retry can never turn one intended action into two.

Idempotency keys are the standard answer

The pattern predates AI workflows. Attach a stable idempotency key to every write request, and have the receiving system remember which keys it has already processed. Stripe's API works this way: sending the same request twice with the same idempotency key returns the original result instead of creating a second charge.

The same discipline applies to every external write inside an AI workflow — creating a ticket, sending a notification, updating a record. Each of those needs its own idempotency key, generated once per logical operation rather than once per network attempt.

  • Generate the key when the workflow decides to act, not when it happens to send the request.
  • Store the outcome against that key so a replay returns the same result instead of repeating the action.
  • Scope each key to the specific write, not to the whole workflow run.

AI adds a wrinkle that retries alone do not solve

Conventional retry logic assumes that repeating a call reproduces the same request. A model call can break that assumption: regenerating a draft, a classification, or a recommendation can return a different answer from the same input, especially above zero temperature.

That means a workflow needs a per-step answer to what a retry actually does. Re-running a read-only classification step is usually fine. Re-running a step that already wrote something downstream is not — that step should replay its stored result rather than generate a new one and write it again.

Build the safety in before you wire up retries

Before adding automatic retries to a workflow, list every step that writes outside the workflow: a CRM update, an outbound email, a payment, a ticket creation. For each one, define the idempotency key and confirm the receiving system actually enforces it rather than just accepting it as an extra field.

Then test the failure directly: force an interruption after the write succeeds but before the workflow records that success, and confirm the retry does not repeat the write. Microsoft's guidance on the retry pattern makes the same point for distributed systems generally — retries only improve reliability when the operation being retried is safe to repeat.