OpenAI rate limits are a workflow design problem, not a retry problem

Automation Engineering · By Caleb Sakala · April 6, 2026

You hit a rate limit error. You add exponential backoff. The error keeps happening. You increase the backoff ceiling. It still happens. You open a support ticket.

This is the standard loop, and it exists because most teams treat rate limits as a retry problem when they are actually a workflow design problem.

The difference matters because retries cannot fix the underlying issue. They can reduce the frequency of failures, but they cannot change the structural conditions that produce them. Understanding what those conditions actually are is the first step toward building workflows that do not fail under load.

What rate limits actually measure

OpenAI enforces limits across four dimensions: requests per minute (RPM), tokens per minute (TPM), requests per day (RPD), and images per minute (IPM). Each one can trigger a 429 independently, and they stack against each other.

A workflow that makes ten concurrent calls, each sending a 4,000-token prompt and receiving a 2,000-token response, consumes 60,000 tokens in that minute. At GPT-4o Tier 1, the TPM ceiling is 30,000. The math does not work, and adding retry logic does not change the math.

The error message will tell you which one, but most teams do not parse it carefully. They see 429, they add backoff, and they move on. The limit that actually fired stays unaddressed.

The three structural causes

Most rate limit failures come from one of three places in the workflow architecture.

The first is sequential prompting with large context windows. Workflows that chain steps together and pass the full conversation history forward accumulate tokens with every call. By step five or six, a workflow that started with a 500-token prompt is sending 8,000-token requests. TPM exhaustion at the end of a chain is almost always this.

The second is fan-out without throttling. Batch jobs that process lists of items by firing parallel requests for each item will spike RPM instantly on any list longer than a few dozen items. The workflow works fine in testing with five items. It fails in production with five hundred.

The third is context window flooding. RAG pipelines that inject full retrieved documents into every prompt are the main culprit here. A pipeline retrieving three 2,000-token documents for each of twenty parallel queries sends 120,000 tokens in a single minute before any generation has started. This is a distinct failure mode from the first two because the tokens are in the input, not the accumulated context, and no amount of output compression fixes it.

Why retries make this worse

Exponential backoff is correct at the function level. If a single call fails, waiting before retrying is the right behavior. The problem is when teams apply it at the workflow level as an architectural pattern.

A workflow step that retries on 429 does not reduce token consumption. It queues the same tokens to be sent again when the rate limit window resets. If the underlying request volume exceeds the limit, retries extend the time the workflow takes to complete without reducing the failure rate. They introduce latency without solving the capacity problem.

For fan-out workflows, retries without throttling produce a thundering herd: all the failed requests retry at the same time, spike the limit again, and produce another wave of 429s. The backoff ceiling rises, the workflow slows to a crawl, and the root cause remains unchanged.

What the fix actually looks like

The architectural answer depends on which limit is firing.

For TPM exhaustion from accumulated context: truncate history before passing it forward. Keep the system prompt and the last two to three exchanges. Summarize earlier context if it is needed. The goal is a flat token footprint per call, not a growing one.

For RPM spikes from fan-out: add a concurrency limit and a request queue. Process batches of ten to twenty items at a time with a delay between batches. This trades latency for reliability, which is almost always the right trade in a production workflow.

For context window flooding from RAG: chunk documents before retrieval, retrieve chunks rather than full documents, and set a hard token budget for the retrieved context block. If the budget is exceeded, truncate or summarize rather than injecting everything.

For workflows that genuinely require high throughput: route across multiple API keys or multiple providers. Anthropic, Google, and OpenAI all offer comparable capability at the top of their model lines. A routing layer that distributes requests across providers turns a single-provider TPM ceiling into a much higher effective ceiling.

How Chase Agents handles this in practice

Chase Agents lets you configure the model and connection at the step level, not at the workflow level. A workflow can use GPT-4o for a classification step, Claude for a generation step, and Gemini for a summarization step, each with its own connection and its own rate limit budget.

That per-step configuration is what makes multi-provider routing practical without custom code. You define the routing logic once in the workflow structure. The platform handles the connection management. When one provider is under load, you switch the step to a different connection and the workflow continues without changes to the surrounding logic.

The same configuration layer handles concurrency. You set a max concurrency for fan-out steps directly in the workflow editor. Items that exceed the concurrency limit queue automatically rather than firing all at once. The thundering herd problem disappears at the configuration level before it can become a retry problem.

Tier upgrades help less than you think

The instinct when hitting rate limits is to request a tier upgrade. Tier upgrades raise the ceiling, but they do not change the workflow structure. A workflow with unthrottled fan-out at Tier 1 has unthrottled fan-out at Tier 5. The limit is higher, so it takes longer to hit, but it still hits.

Tier upgrades are useful when your workflow is correctly structured and you have genuinely outgrown the capacity of your current tier. They are not useful as a substitute for fixing the structural conditions that are causing the failures.

The correct sequence is: identify which limit dimension is firing, fix the structural cause, then request a tier upgrade if the corrected workflow still needs more capacity.

GPT-5 and the token budget shift

GPT-5's Tier 1 TPM limit increased to 500,000 tokens per minute in early 2026, up from 30,000 for GPT-4o. For workflows that were hitting TPM limits on GPT-4o, routing those steps to GPT-5 at Tier 1 effectively removes the TPM constraint for most use cases.

This does not mean the structural fixes above become unnecessary. A workflow that floods context windows will eventually hit any TPM ceiling at sufficient scale. But for teams that have made the structural fixes and are still constrained by GPT-4o Tier 1 limits, GPT-5 routing is a practical path to more headroom without a tier upgrade request.

The diagnostic checklist

Before adding any retry logic, answer these questions:

• Which dimension fired: RPM, TPM, RPD, or IPM?
• Is the token footprint per call growing across workflow steps (accumulated context)?
• Is the workflow firing parallel requests without a concurrency limit (fan-out)?
• Is a RAG step injecting full documents rather than chunks (context window flooding)?
• Is the workflow tied to a single provider with no routing fallback?

If the answer to any of those is yes, fix it before adding backoff. Retries on top of a broken structure produce slower failures, not fewer failures.