Handling LLM API Rate Limits: Retry, Backoff and Queues in Python

To handle LLM API rate limits in Python, retry only the errors that are worth retrying, wait at least as long as the Retry-After header says, add random jitter, cap the number of attempts, and limit how many requests are in flight so you rarely hit a 429 in the first place. This guide shows each layer using the OpenAI Python SDK, and the same pattern applies to other providers.

Who this is for and what you will have

This article is for developers whose LLM calls fail intermittently with 429 Too Many Requests, or who are about to run a batch job that will. By the end you will have a retry helper with backoff and jitter, an async concurrency limiter, a simple queue for bulk work, and a clear rule for which errors to retry. If you are on Azure OpenAI, the limits work differently; see Fix Azure OpenAI 429 Too Many Requests: TPM, RPM and PTU limits for those specifics.

Understand what a rate limit is

OpenAI's rate limit guide lists several metrics: requests per minute (RPM), requests per day (RPD), tokens per minute (TPM), tokens per day (TPD) and images per minute (IPM). You hit a limit on whichever metric reaches its cap first, so a few requests with very large prompts can exhaust TPM while many small requests exhaust RPM. Limits apply at organization and project level, vary by model, and can be shared across model families.

Two details cause surprises:

• The estimate uses max_tokens. Your token usage for rate limiting is calculated from the larger of max_tokens and the estimated size of your request. Setting max_tokens close to the response size you really expect helps you stay inside TPM.
• Failed requests still count. Requests that fail still count toward your per-minute limit, so retrying continuously without waiting makes the problem worse.

The two kinds of 429

The same status code covers different situations, and they need different handling.

OpenAI documents that the slow_down 429 can occur even within your RPM and TPM limits when request rate rises too quickly. As a rough guide from the same page, once traffic passes 1M input tokens per minute, increase it by no more than 50 percent every 15 minutes.

Prerequisites

You need Python 3 with the OpenAI SDK and an API key in the environment.

python -m venv .venv
source .venv/bin/activate
pip install openai
export OPENAI_API_KEY="your-key-here"

Step 1: Know what the SDK already does

Before writing a loop, know that the Python SDK retries for you. Per its README, certain errors are retried 2 times by default with a short exponential backoff: connection errors, 408 Request Timeout, 409 Conflict, 429 Rate Limit and 500 or higher internal errors. Requests are retried only when the body can be safely resent, and consuming a stream is not retried automatically.

You can change the budget on the client or per request:

from openai import OpenAI
 
client = OpenAI(max_retries=5)  # default is 2
 
# Or for a single call:
client.with_options(max_retries=5).chat.completions.create(
    model="gpt-5.5",
    messages=[{"role": "user", "content": "Hello"}],
)

For many applications, raising max_retries is enough. Write your own loop when you need control over the total time, metrics on each retry, a shared queue across workers, or different handling per error type. If you do, the OpenAI guide says to disable SDK retries or count them toward your limits so you don't multiply requests. Set max_retries=0 on the client you use inside your loop.

Step 2: Write a retry helper with backoff and jitter

The guidance from the rate limits page is: treat Retry-After as a minimum wait and add a small random delay; if the header is missing or invalid, use exponential backoff with jitter; and cap both the number of attempts and the total retry time. This helper does exactly that.

import random
import time
 
import openai
from openai import OpenAI
 
client = OpenAI(max_retries=0)  # our loop owns the retries
 
MAX_ATTEMPTS = 6
MAX_TOTAL_SECONDS = 60
BASE_DELAY = 1.0
MAX_DELAY = 20.0
 
 
def retry_after_seconds(exc: openai.APIStatusError):
    value = exc.response.headers.get("retry-after")
    try:
        return float(value) if value is not None else None
    except ValueError:
        return None
 
 
def call_with_retry(fn, *args, **kwargs):
    started = time.monotonic()
    for attempt in range(MAX_ATTEMPTS):
        try:
            return fn(*args, **kwargs)
        except (openai.RateLimitError, openai.InternalServerError) as exc:
            if time.monotonic() - started > MAX_TOTAL_SECONDS:
                raise
            if attempt == MAX_ATTEMPTS - 1:
                raise
            header_delay = retry_after_seconds(exc)
            if header_delay is not None:
                delay = header_delay + random.uniform(0, 1)
            else:
                delay = min(MAX_DELAY, BASE_DELAY * 2**attempt)
                delay = random.uniform(delay / 2, delay)  # jitter
            time.sleep(delay)
 
 
reply = call_with_retry(
    client.chat.completions.create,
    model="gpt-5.5",
    messages=[{"role": "user", "content": "Summarise rate limiting in one line."}],
)
print(reply.choices[0].message.content)

A few design choices are worth explaining:

• openai.RateLimitError is the exception for HTTP 429, and all SDK errors inherit from openai.APIError.
• The loop retries 429 and 5xx only. A bad request or an authentication error should fail immediately.
• Jitter spreads retries out. Without it, many workers that were throttled at the same moment all retry at the same moment and trigger another burst.
• The total time cap makes the failure visible to your caller instead of silently hanging for minutes.

Do not retry quota errors

A RateLimitError can be raised for a quota or billing problem as well as for a rate limit. OpenAI's guidance is not to retry quota or billing errors because they require action on your part. Inspect the error message or code before sleeping; if it refers to quota, raise immediately and alert someone. Printing exc.body while you build the helper shows the error code your account returns.

Step 3: Limit concurrency so you stay under the limit

Retries are a reaction. A concurrency limit prevents most 429s. With async code, a semaphore caps how many requests are in flight at once.

import asyncio
import random
 
import openai
from openai import AsyncOpenAI
 
aclient = AsyncOpenAI(max_retries=0)
semaphore = asyncio.Semaphore(8)  # tune to your limits
 
 
async def acall(prompt: str) -> str:
    async with semaphore:
        for attempt in range(6):
            try:
                r = await aclient.chat.completions.create(
                    model="gpt-5.5",
                    messages=[{"role": "user", "content": prompt}],
                )
                return r.choices[0].message.content
            except openai.RateLimitError:
                delay = min(20.0, 2**attempt)
                await asyncio.sleep(random.uniform(delay / 2, delay))
        raise RuntimeError("rate limited after 6 attempts")
 
 
async def main(prompts):
    return await asyncio.gather(*(acall(p) for p in prompts))

Pick the semaphore size from your limits. Dividing your RPM by the typical request latency in minutes gives a rough ceiling for concurrent requests; start lower and raise it while watching 429 counts. The semaphore size is a tuning value, not a documented number.

Step 4: Put bulk work behind a queue

When requests come from many places, such as web requests, background jobs and cron tasks, a shared queue with a fixed number of workers gives you a single choke point.

import asyncio
 
queue: asyncio.Queue[str] = asyncio.Queue()
results: dict[str, str] = {}
 
 
async def worker():
    while True:
        prompt = await queue.get()
        try:
            results[prompt] = await acall(prompt)
        finally:
            queue.task_done()
 
 
async def run(prompts, workers=4):
    tasks = [asyncio.create_task(worker()) for _ in range(workers)]
    for p in prompts:
        queue.put_nowait(p)
    await queue.join()
    for t in tasks:
        t.cancel()
    return results

In production, replace the in-memory queue with a durable one such as a message broker so work survives restarts. The principle stays the same: producers enqueue quickly, a small pool of workers drains the queue at a rate that matches your limits, and failures are retried with backoff or parked for inspection.

Step 5: Read the rate limit headers

OpenAI returns headers that tell you where you stand:

• x-ratelimit-limit-requests and x-ratelimit-limit-tokens: the maximums.
• x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens: what is left.
• x-ratelimit-reset-requests and x-ratelimit-reset-tokens: time until each limit resets.
• Retry-After: the minimum seconds to wait, when present.

Log the remaining values on a sample of requests. A steady fall toward zero in the remaining-tokens header tells you to slow producers down before the first 429 arrives. Other providers expose similar headers; Anthropic, for example, documents anthropic-ratelimit-* headers and a retry-after header, and uses a token bucket algorithm where capacity is replenished continuously instead of resetting at fixed intervals.

Step 6: Reduce demand instead of absorbing it

Some of the cheapest fixes remove tokens or requests:

• Lower max_tokens to roughly what you need, because the rate limit estimate uses the larger of max_tokens and the estimated request size.
• Use the Batch API for non-urgent work. OpenAI's rate limit guide describes batch limits separately from synchronous limits: tokens from pending batch jobs count against a batch queue limit, which is based on input tokens queued per model. That makes batch a good place for large, non-urgent workloads, but it is not unlimited.
• Cache repeated context where your provider supports it. For Claude, cached reads do not count toward input-token limits on most models; see Anthropic Prompt Caching: Cut Cost and Latency on Repeated Context.
• Request a higher limit from the provider once your usage is steady and well understood.

Troubleshooting

Checklist

• You know which 429 you are seeing: rate limit, ramp-up or quota.
• One retry layer owns the policy, with Retry-After honoured, jitter added and a cap on attempts and total time.
• Concurrency is limited by a semaphore or queue.
• max_tokens is set close to real need.
• Rate limit headers are logged so you can slow down before errors occur.
• Non-urgent bulk work goes through a batch interface.

References

• OpenAI rate limits guide: https://developers.openai.com/api/docs/guides/rate-limits
• OpenAI Python SDK README: https://github.com/openai/openai-python
• Anthropic rate limits: https://platform.claude.com/docs/en/api/rate-limits