Two different limits get conflated

Most API providers enforce a request-rate limit (calls per second or per minute) and a separate concurrency limit (how many requests can be in flight at once). Stripe documents both directly: a global rate limit measured in requests per second, plus a distinct concurrency limit that caps simultaneously active requests and is triggered less often but usually by long-running or resource-heavy calls such as list operations.

An AI workflow that calls a model API, a vector store, and a line-of-business system in the same step can hit either limit independently. Treating every 429 the same way misses that a rate-limit error and a concurrency error call for different fixes: slow down the send rate for one, reduce how many requests run in parallel for the other.

  • Rate limit: too many requests in a time window
  • Concurrency limit: too many requests in flight at once
  • Both can return the same HTTP status but need different responses

Backoff belongs only on retry-safe errors

The standard mitigation is exponential backoff with jitter: wait progressively longer between retries, with randomness added so many callers don't retry in the same instant and cause a new spike. Google Cloud's retry guidance for its storage API states this plainly, and also draws a line most teams skip: only retry requests that are actually idempotent, or that carry a precondition making them safe to repeat.

In an AI workflow, that line matters more than in a typical API client. A step that drafts a message is safe to retry. A step that writes an approval, sends an email, or updates a record is not safe to retry blindly, because a retried write can duplicate the action rather than just resend it. Backoff logic needs to know which steps in the workflow are read-or-draft (safe to retry automatically) and which are consequential writes (need an idempotency key, a dedupe check, or a human confirmation before retry).

Reduce how often the limit gets hit

Retrying gracefully is the fallback, not the plan. OpenAI's rate limit documentation lists batching as a primary mitigation: combining multiple items into fewer requests when you have spare token capacity but limited request capacity, and using an asynchronous batch endpoint for work that doesn't need a synchronous response.

The same principle applies to workflow design generally. If a queue processes documents one at a time and each one calls the model API separately, the workflow will hit its request-rate ceiling well before it hits any real compute limit. Grouping items, right-sizing requested output length, and separating latency-sensitive steps from batch-friendly ones all lower the number of calls a workflow makes for the same amount of work.

Give the workflow somewhere to put pressure

A retry loop without a queue just moves the failure later. When an AI workflow is doing real volume, the integration layer needs a place to hold work that can't run yet: a queue with a depth limit, a scheduler that spaces out calls to match the provider's stated ceiling, or a circuit breaker that pauses a downstream step entirely when errors spike instead of retrying it into a deeper backlog.

This is also where the limit needs to become visible to an operator. A workflow that silently queues, retries, and eventually times out looks the same as one that is working normally, until someone notices delayed output. Surfacing queue depth and retry counts as an operational metric, not just an application log line, is what turns a rate limit from an outage into a known, monitored constraint.