# Handling LLM API Rate Limits: Retry, Backoff and Queues in Python > Handle 429 errors from the OpenAI API in Python with Retry-After, exponential backoff with jitter, a concurrency limiter and a queue, without retry storms. Source: https://www.epifive.com/blog/llm-api-rate-limits-retry-backoff-python · Published 2026-10-25 · EpiFive ## Key takeaways - Honour the Retry-After header as a minimum wait, add jitter, and fall back to exponential backoff when the header is missing. - Cap both the number of retries and the total time spent retrying so failures surface instead of hanging. - Do not retry quota or billing errors, because they need a human action rather than a wait. - Limit concurrency with a semaphore or queue so you stay under limits instead of reacting to 429 errors. - Disable SDK retries or count them if you add your own loop, and send non-urgent bulk work to the Batch API. To handle LLM API rate limits in Python, retry only the errors that are worth retrying, wait at least as long as the `Retry-After` header says, add random jitter, cap the number of attempts, and limit how many requests are in flight so you rarely hit a 429 in the first place. This guide shows each layer using the OpenAI Python SDK, and the same pattern applies to other providers. ## Who this is for and what you will have This article is for developers whose LLM calls fail intermittently with `429 Too Many Requests`, or who are about to run a batch job that will. By the end you will have a retry helper with backoff and jitter, an async concurrency limiter, a simple queue for bulk work, and a clear rule for which errors to retry. If you are on Azure OpenAI, the limits work differently; see [Fix Azure OpenAI 429 Too Many Requests: TPM, RPM and PTU limits](https://www.epifive.com/blog/azure-openai-429-rate-limit-fix) for those specifics. ## Understand what a rate limit is OpenAI's rate limit guide lists several metrics: requests per minute (RPM), requests per day (RPD), tokens per minute (TPM), tokens per day (TPD) and images per minute (IPM). You hit a limit on whichever metric reaches its cap first, so a few requests with very large prompts can exhaust TPM while many small requests exhaust RPM. Limits apply at organization and project level, vary by model, and can be shared across model families. Two details cause surprises: - **The estimate uses `max_tokens`.** Your token usage for rate limiting is calculated from the larger of `max_tokens` and the estimated size of your request. Setting `max_tokens` close to the response size you really expect helps you stay inside TPM. - **Failed requests still count.** Requests that fail still count toward your per-minute limit, so retrying continuously without waiting makes the problem worse. ### The two kinds of 429 The same status code covers different situations, and they need different handling. | Situation | How it presents | What to do | | --- | --- | --- | | Rate limit or sudden ramp-up | 429, sometimes with the `slow_down` code, often with `Retry-After` | Wait, then retry; ramp traffic up gradually | | Quota or billing problem | 429 error about quota or billing | Do not retry; add credit or raise limits | | Server overloaded | 503 with `server_is_overloaded` | Retry after `Retry-After` if present and increase the delay if errors continue | OpenAI documents that the `slow_down` 429 can occur even within your RPM and TPM limits when request rate rises too quickly. As a rough guide from the same page, once traffic passes 1M input tokens per minute, increase it by no more than 50 percent every 15 minutes. ## Prerequisites You need Python 3 with the OpenAI SDK and an API key in the environment. ```bash python -m venv .venv source .venv/bin/activate pip install openai export OPENAI_API_KEY="your-key-here" ``` ## Step 1: Know what the SDK already does Before writing a loop, know that the Python SDK retries for you. Per its README, certain errors are retried 2 times by default with a short exponential backoff: connection errors, 408 Request Timeout, 409 Conflict, 429 Rate Limit and 500 or higher internal errors. Requests are retried only when the body can be safely resent, and consuming a stream is not retried automatically. You can change the budget on the client or per request: ```python from openai import OpenAI client = OpenAI(max_retries=5) # default is 2 # Or for a single call: client.with_options(max_retries=5).chat.completions.create( model="gpt-5.5", messages=[{"role": "user", "content": "Hello"}], ) ``` For many applications, raising `max_retries` is enough. Write your own loop when you need control over the total time, metrics on each retry, a shared queue across workers, or different handling per error type. If you do, the OpenAI guide says to disable SDK retries or count them toward your limits so you don't multiply requests. Set `max_retries=0` on the client you use inside your loop. ## Step 2: Write a retry helper with backoff and jitter The guidance from the rate limits page is: treat `Retry-After` as a minimum wait and add a small random delay; if the header is missing or invalid, use exponential backoff with jitter; and cap both the number of attempts and the total retry time. This helper does exactly that. ```python import random import time import openai from openai import OpenAI client = OpenAI(max_retries=0) # our loop owns the retries MAX_ATTEMPTS = 6 MAX_TOTAL_SECONDS = 60 BASE_DELAY = 1.0 MAX_DELAY = 20.0 def retry_after_seconds(exc: openai.APIStatusError): value = exc.response.headers.get("retry-after") try: return float(value) if value is not None else None except ValueError: return None def call_with_retry(fn, *args, **kwargs): started = time.monotonic() for attempt in range(MAX_ATTEMPTS): try: return fn(*args, **kwargs) except (openai.RateLimitError, openai.InternalServerError) as exc: if time.monotonic() - started > MAX_TOTAL_SECONDS: raise if attempt == MAX_ATTEMPTS - 1: raise header_delay = retry_after_seconds(exc) if header_delay is not None: delay = header_delay + random.uniform(0, 1) else: delay = min(MAX_DELAY, BASE_DELAY * 2**attempt) delay = random.uniform(delay / 2, delay) # jitter time.sleep(delay) reply = call_with_retry( client.chat.completions.create, model="gpt-5.5", messages=[{"role": "user", "content": "Summarise rate limiting in one line."}], ) print(reply.choices[0].message.content) ``` A few design choices are worth explaining: - `openai.RateLimitError` is the exception for HTTP 429, and all SDK errors inherit from `openai.APIError`. - The loop retries 429 and 5xx only. A bad request or an authentication error should fail immediately. - Jitter spreads retries out. Without it, many workers that were throttled at the same moment all retry at the same moment and trigger another burst. - The total time cap makes the failure visible to your caller instead of silently hanging for minutes. ### Do not retry quota errors A `RateLimitError` can be raised for a quota or billing problem as well as for a rate limit. OpenAI's guidance is not to retry quota or billing errors because they require action on your part. Inspect the error message or code before sleeping; if it refers to quota, raise immediately and alert someone. Printing `exc.body` while you build the helper shows the error code your account returns. ## Step 3: Limit concurrency so you stay under the limit Retries are a reaction. A concurrency limit prevents most 429s. With async code, a semaphore caps how many requests are in flight at once. ```python import asyncio import random import openai from openai import AsyncOpenAI aclient = AsyncOpenAI(max_retries=0) semaphore = asyncio.Semaphore(8) # tune to your limits async def acall(prompt: str) -> str: async with semaphore: for attempt in range(6): try: r = await aclient.chat.completions.create( model="gpt-5.5", messages=[{"role": "user", "content": prompt}], ) return r.choices[0].message.content except openai.RateLimitError: delay = min(20.0, 2**attempt) await asyncio.sleep(random.uniform(delay / 2, delay)) raise RuntimeError("rate limited after 6 attempts") async def main(prompts): return await asyncio.gather(*(acall(p) for p in prompts)) ``` Pick the semaphore size from your limits. Dividing your RPM by the typical request latency in minutes gives a rough ceiling for concurrent requests; start lower and raise it while watching 429 counts. The semaphore size is a tuning value, not a documented number. ## Step 4: Put bulk work behind a queue When requests come from many places, such as web requests, background jobs and cron tasks, a shared queue with a fixed number of workers gives you a single choke point. ```python import asyncio queue: asyncio.Queue[str] = asyncio.Queue() results: dict[str, str] = {} async def worker(): while True: prompt = await queue.get() try: results[prompt] = await acall(prompt) finally: queue.task_done() async def run(prompts, workers=4): tasks = [asyncio.create_task(worker()) for _ in range(workers)] for p in prompts: queue.put_nowait(p) await queue.join() for t in tasks: t.cancel() return results ``` In production, replace the in-memory queue with a durable one such as a message broker so work survives restarts. The principle stays the same: producers enqueue quickly, a small pool of workers drains the queue at a rate that matches your limits, and failures are retried with backoff or parked for inspection. ## Step 5: Read the rate limit headers OpenAI returns headers that tell you where you stand: - `x-ratelimit-limit-requests` and `x-ratelimit-limit-tokens`: the maximums. - `x-ratelimit-remaining-requests` and `x-ratelimit-remaining-tokens`: what is left. - `x-ratelimit-reset-requests` and `x-ratelimit-reset-tokens`: time until each limit resets. - `Retry-After`: the minimum seconds to wait, when present. Log the remaining values on a sample of requests. A steady fall toward zero in the remaining-tokens header tells you to slow producers down before the first 429 arrives. Other providers expose similar headers; Anthropic, for example, documents `anthropic-ratelimit-*` headers and a `retry-after` header, and uses a token bucket algorithm where capacity is replenished continuously instead of resetting at fixed intervals. ## Step 6: Reduce demand instead of absorbing it Some of the cheapest fixes remove tokens or requests: - **Lower `max_tokens`** to roughly what you need, because the rate limit estimate uses the larger of `max_tokens` and the estimated request size. - **Use the Batch API for non-urgent work.** OpenAI's rate limit guide describes batch limits separately from synchronous limits: tokens from pending batch jobs count against a batch queue limit, which is based on input tokens queued per model. That makes batch a good place for large, non-urgent workloads, but it is not unlimited. - **Cache repeated context** where your provider supports it. For Claude, cached reads do not count toward input-token limits on most models; see [Anthropic Prompt Caching: Cut Cost and Latency on Repeated Context](https://www.epifive.com/blog/anthropic-prompt-caching-cost-latency). - **Request a higher limit** from the provider once your usage is steady and well understood. ## Troubleshooting | Symptom | Likely cause | Fix | | --- | --- | --- | | 429 right after a traffic spike | Sudden ramp-up (`slow_down`) | Ramp up gradually; add a queue | | 429 even though RPM and TPM look low | Large `max_tokens` inflates the token estimate | Set `max_tokens` close to the expected output | | Calls take much longer than expected | SDK retries stacked with your own loop | Set `max_retries=0` and keep one retry layer | | Retries never succeed and the quota message appears | Quota or billing limit | Stop retrying; add credit or raise the limit | | Many workers retry in lockstep | No jitter | Add random jitter to every delay | | Streaming responses fail midway | SDK does not retry stream consumption | Catch the error and restart the request in your own code | ## Checklist - You know which 429 you are seeing: rate limit, ramp-up or quota. - One retry layer owns the policy, with `Retry-After` honoured, jitter added and a cap on attempts and total time. - Concurrency is limited by a semaphore or queue. - `max_tokens` is set close to real need. - Rate limit headers are logged so you can slow down before errors occur. - Non-urgent bulk work goes through a batch interface. ## References - OpenAI rate limits guide: https://developers.openai.com/api/docs/guides/rate-limits - OpenAI Python SDK README: https://github.com/openai/openai-python - Anthropic rate limits: https://platform.claude.com/docs/en/api/rate-limits ## Frequently asked questions ### How long should I wait after a 429 from the OpenAI API? Treat the Retry-After header as a minimum wait and add a small random delay. If the header is missing or invalid, use exponential backoff with jitter, and cap both the number of attempts and the total retry time. ### Should I retry every 429 error? No. A 429 caused by a rate limit or a sudden traffic increase is worth retrying after a wait, but quota and billing errors need action from you, such as adding credit or raising a limit. Retrying them only wastes time. ### Why do I still get 429 errors when my RPM and TPM look fine? OpenAI documents a 429 with the slow_down code that appears when your request rate rises too quickly, even within RPM and TPM limits. Ramp traffic up gradually. Also remember that the token estimate uses the larger of max_tokens and the request size, so a generous max_tokens uses up tokens-per-minute capacity. ### Do the OpenAI SDK retries and my own retry loop conflict? They can multiply requests. The Python SDK retries certain errors, including 429, two times by default. If you write your own loop, set max_retries to 0 on the client or count the SDK retries toward your budget.