LLM Rate Limiting and Retries: Exponential Backoff with Jitter in Python
TL;DR — LLM rate limiting and retries: handle 429s with exponential backoff + full jitter. SpeedTestHQ: "Full jitter: random(0, min(base * 2^attempt, max)). Honor Retry-After header. Token bucket for proactive throttling. RPM and TPM both apply simultaneously." SitePoint: "Anthropic enforces three limits: RPM, TPM, concurrent. Prioritize Retry-After, then reset-header math, then exponential backoff. Circuit breaker fails fast when retries futile." rate-limit-shield: "TokenBucket + CircuitBreaker + RetryPolicy. Closed→Open→Half-Open. Per-model isolation. Designed for OpenAI/Anthropic/Bedrock. Jitter or die — synchronized retries turn 500 into 5,000." llm-retry-py: "6 attempts, 500ms base, 30s cap, full jitter. Presets for Anthropic/OpenAI/Bedrock/Gemini. No HTTP client — wrap any callable." ofox.ai: "retry-after is your best friend. Full jitter widest spread. Categorize errors: retryable (429, 5xx) vs client (400, 401, 403)." Learn more with LLM gateway, LiteLLM tutorial, LLM streaming SSE, and switch provider.
SitePoint frames the problem: "Handling Claude API 429 errors in production Python applications demands more than a basic retry loop. Anthropic's rate-limiting system enforces multiple overlapping constraints simultaneously, and a naive approach to retries can amplify failures rather than absorb them."
rate-limit-shield states the universal pattern: "Every team building on LLM APIs hand-rolls the same three primitives: a token bucket, a circuit breaker, and an exponential-backoff retry. They get it 80% right, ship it, and discover the failure modes in production at 3 AM."
Rate Limiting Architecture
capacity = TPM/RPM
refill = TPM/60 per sec
block when empty"] Headers["RateLimit-Remaining Headers
throttle preemptively
when remaining drops low"] Distributed["Distributed Coordinator
Redis shared counter
multi-instance coordination"] end subgraph Reactive["Reactive Retry"] Check429["Check 429 Response"] RetryAfter["Retry-After Header
use server value directly
add 0.5s jitter"] ResetHeaders["Reset Headers
compute wait from
nearest reset timestamp"] Backoff["Exponential Backoff
ceiling = min(base * 2^attempt, max)
full jitter: random(0, ceiling)"] end subgraph Breaker["Circuit Breaker"] Closed["Closed
normal operation
requests pass through"] Open["Open
fail fast
no retries attempted"] HalfOpen["Half-Open
probe with single request
test if provider recovered"] end subgraph Errors["Error Classification"] Retryable["Retryable (429, 500, 502, 503, 504)
exponential backoff with jitter"] Client["Client Errors (400, 401, 403)
no retry — fix code"] Sustained["Sustained Outage
circuit breaker opens
fail fast to fallback"] end subgraph Limits["Rate Limit Dimensions"] RPM["RPM
Requests Per Minute
rolling 60s window"] TPM["TPM
Tokens Per Minute
input + output tokens"] Concurrent["Concurrent Requests
in-flight requests
at any instant"] end Request --> TokenBucket TokenBucket --> Headers Headers --> Distributed Distributed --> Check429 Check429 --> RetryAfter RetryAfter --> ResetHeaders ResetHeaders --> Backoff Check429 --> Retryable Check429 --> Client Retryable --> Sustained Sustained --> Breaker Closed --> Open Open --> HalfOpen HalfOpen --> Closed Limits --> TokenBucket
Jitter Modes
| Jitter Mode | Formula | Best For |
|---|---|---|
| Full (recommended) | random.uniform(0, capped) |
Many clients, widest spread |
| Equal | capped/2 + random.uniform(0, capped/2) |
Guaranteed minimum wait |
| Decorrelated | random.uniform(base, capped) |
Most uniform spread |
| None | capped |
Testing only — causes thundering herd |
Rate Limit Dimensions
| Dimension | What It Counts | Typical Limits | Binding Constraint |
|---|---|---|---|
| RPM | API calls per 60s | 3 (free) to 10,000 (enterprise) | Many small prompts |
| TPM | Input + output tokens per 60s | 40K (free) to 2M (enterprise) | Large prompts |
| Concurrent | In-flight requests at once | Varies by tier | Parallel processing |
Implementation
from dataclasses import dataclass
from typing import Optional
from enum import Enum
import random
import time
class JitterMode(Enum):
FULL = "full"
EQUAL = "equal"
NONE = "none"
class CircuitState(Enum):
CLOSED = "closed"
OPEN = "open"
HALF_OPEN = "half_open"
@dataclass
class LLMRateLimitingGuide:
"""LLM rate limiting and retries implementation guide."""
def get_retry_config(self) -> str:
"""Retry configuration."""
return (
"from dataclasses import dataclass\n"
"import random\n"
"\n"
"@dataclass\n"
"class RetryConfig:\n"
" max_retries: int = 5\n"
" base_delay: float = 1.0\n"
" max_delay: float = 60.0\n"
" exponential_base: float = 2.0\n"
" jitter_mode: str = 'full'\n"
" retryable_codes: tuple = (\n"
" 429, 500, 502, 503, 504)\n"
"\n"
" def get_delay(\n"
" self, attempt: int\n"
" ) -> float:\n"
" ceiling = min(\n"
" self.base_delay * (\n"
" self.exponential_base\n"
" ** attempt),\n"
" self.max_delay)\n"
" if self.jitter_mode == 'full':\n"
" return random.uniform(\n"
" 0, ceiling)\n"
" elif self.jitter_mode == 'equal':\n"
" return (ceiling / 2 +\n"
" random.uniform(\n"
" 0, ceiling / 2))\n"
" return ceiling"
)
def get_retry_with_backoff(self) -> str:
"""Retry with exponential backoff and jitter."""
return (
"import random\n"
"import time\n"
"from openai import OpenAI\n"
"from openai import (\n"
" RateLimitError,\n"
" APIConnectionError,\n"
" APIError, Timeout)\n"
"\n"
"client = OpenAI()\n"
"\n"
"def call_with_retry(\n"
" messages: list,\n"
" model: str = 'gpt-4o',\n"
" max_retries: int = 5,\n"
" base_delay: float = 1.0,\n"
" max_delay: float = 60.0,\n"
"):\n"
" '''Call LLM with\n"
" exponential backoff\n"
" and full jitter.'''\n"
" for attempt in range(\n"
" max_retries):\n"
" try:\n"
" response = client\\\n"
" .chat.completions\\\n"
" .create(\n"
" model=model,\n"
" messages=messages)\n"
" return response\n"
" except RateLimitError as e:\n"
" # Check Retry-After\n"
" retry_after = None\n"
" if hasattr(e, 'response'):\n"
" headers = e.response\\\n"
" .headers\n"
" retry_after = headers\\\n"
" .get('retry-after')\n"
" if retry_after:\n"
" delay = float(\n"
" retry_after)\n"
" # Add jitter\n"
" delay += random\\\n"
" .uniform(0, 0.5)\n"
" else:\n"
" # Full jitter\n"
" ceiling = min(\n"
" base_delay * (\n"
" 2 ** attempt),\n"
" max_delay)\n"
" delay = random\\\n"
" .uniform(0, ceiling)\n"
" print(f'429: retry in '\n"
" f'{delay:.1f}s '\n"
" f'(attempt {attempt+1})')\n"
" time.sleep(delay)\n"
" except (APIConnectionError,\n"
" Timeout) as e:\n"
" # Transient errors\n"
" ceiling = min(\n"
" base_delay * (\n"
" 2 ** attempt),\n"
" max_delay)\n"
" delay = random\\\n"
" .uniform(0, ceiling)\n"
" time.sleep(delay)\n"
" except APIError as e:\n"
" # Non-retryable\n"
" raise\n"
" raise Exception(\n"
" f'Max retries ({max_retries}) '\n"
" f'exceeded')"
)
def get_token_bucket(self) -> str:
"""Token bucket for proactive throttling."""
return (
"import time\n"
"import threading\n"
"\n"
"class TokenBucket:\n"
" '''Thread-safe token bucket\n"
" for rate limiting.'''\n"
"\n"
" def __init__(\n"
" self, capacity: int,\n"
" refill_rate: float):\n"
" # capacity = TPM or RPM\n"
" # refill_rate = TPM/60\n"
" self.capacity = capacity\n"
" self.refill_rate = refill_rate\n"
" self.tokens = capacity\n"
" self.last_refill = time.time()\n"
" self.lock = threading\\\n"
" .Lock()\n"
"\n"
" def _refill(self):\n"
" now = time.time()\n"
" elapsed = now - \\\n"
" self.last_refill\n"
" new_tokens = (\n"
" elapsed * self.refill_rate)\n"
" self.tokens = min(\n"
" self.capacity,\n"
" self.tokens + \n"
" new_tokens)\n"
" self.last_refill = now\n"
"\n"
" def acquire(\n"
" self, tokens: int = 1\n"
" ) -> bool:\n"
" with self.lock:\n"
" self._refill()\n"
" if self.tokens >= tokens:\n"
" self.tokens -= tokens\n"
" return True\n"
" return False\n"
"\n"
" def wait_for_tokens(\n"
" self, tokens: int = 1\n"
" ):\n"
" '''Block until tokens\n"
" available.'''\n"
" while not self.acquire(\n"
" tokens):\n"
" time.sleep(0.1)\n"
"\n"
"# Usage: TPM = 100000\n"
"bucket = TokenBucket(\n"
" capacity=100000,\n"
" refill_rate=100000/60)\n"
"\n"
"# Before each request\n"
"bucket.wait_for_tokens(\n"
" estimated_tokens)"
)
def get_circuit_breaker(self) -> str:
"""Circuit breaker implementation."""
return (
"import time\n"
"from enum import Enum\n"
"\n"
"class CircuitState(Enum):\n"
" CLOSED = 'closed'\n"
" OPEN = 'open'\n"
" HALF_OPEN = 'half_open'\n"
"\n"
"class CircuitBreaker:\n"
" '''Circuit breaker for\n"
" sustained outage resilience.'''\n"
"\n"
" def __init__(\n"
" self,\n"
" failure_threshold: int = 5,\n"
" recovery_timeout: float = 30.0\n"
" ):\n"
" self.failure_threshold = \\\n"
" failure_threshold\n"
" self.recovery_timeout = \\\n"
" recovery_timeout\n"
" self.state = CircuitState\\\n"
" .CLOSED\n"
" self.failures = 0\n"
" self.last_failure = None\n"
"\n"
" def can_execute(self) -> bool:\n"
" if self.state == \\\n"
" CircuitState.CLOSED:\n"
" return True\n"
" if self.state == \\\n"
" CircuitState.OPEN:\n"
" if (self.last_failure and\n"
" time.time() - \n"
" self.last_failure > \n"
" self.recovery_timeout):\n"
" self.state = \\\n"
" CircuitState\\\n"
" .HALF_OPEN\n"
" return True\n"
" return False\n"
" return True # HALF_OPEN\n"
"\n"
" def record_success(self):\n"
" self.failures = 0\n"
" self.state = CircuitState\\\n"
" .CLOSED\n"
"\n"
" def record_failure(self):\n"
" self.failures += 1\n"
" self.last_failure = \\\n"
" time.time()\n"
" if self.failures >= \\\n"
" self.failure_threshold:\n"
" self.state = \\\n"
" CircuitState.OPEN"
)
def get_tenacity_example(self) -> str:
"""Tenacity library example."""
return (
"from tenacity import (\n"
" retry, stop_after_attempt,\n"
" wait_exponential_jitter,\n"
" before_sleep_log,\n"
" retry_if_exception_type)\n"
"import logging\n"
"from openai import (\n"
" RateLimitError, APIError)\n"
"\n"
"logger = logging.getLogger(__name__)\n"
"\n"
"@retry(\n"
" stop=stop_after_attempt(5),\n"
" wait=wait_exponential_jitter(\n"
" initial=1, max=60),\n"
" retry=retry_if_exception_type(\n"
" (RateLimitError, APIError)),\n"
" before_sleep=before_sleep_log(\n"
" logger, logging.WARNING),\n"
")\n"
"def call_llm(messages):\n"
" return client.chat\\\n"
" .completions.create(\n"
" model='gpt-4o',\n"
" messages=messages)\n"
"\n"
"# Tenacity handles:\n"
"# - exponential backoff\n"
"# - jitter\n"
"# - retry only on retryable\n"
"# - logging each retry"
)
def get_provider_presets(self) -> dict:
"""Provider-specific retryable error codes."""
return {
"anthropic": [
"rate_limit_error",
"overloaded_error",
"internal_server_error",
],
"openai": [
"server_error",
"rate_limit_exceeded",
"engine_overloaded",
],
"bedrock": [
"ThrottlingException",
"ServiceUnavailable",
"InternalServerException",
],
"gemini": [
"RESOURCE_EXHAUSTED",
"INTERNAL",
"UNAVAILABLE",
],
"http_status": [429, 500, 502, 503, 504],
}
def get_best_practices(self) -> dict:
"""Best practices."""
return {
"three_principles": [
"Cap the blast radius — breakers stop DoSing your vendor",
"Jitter or die — synchronized retries turn 500 into 5,000",
"Honor Retry-After — vendor told you when to come back",
],
"proactive_throttling": [
"Token bucket on your side — block before 429",
"Watch RateLimit-Remaining headers — throttle preemptively",
"Distributed coordinator (Redis) for multi-instance",
],
"error_classification": [
"Retryable (429, 5xx) — exponential backoff",
"Client errors (400, 401, 403) — fix code, no retry",
"Sustained outage — circuit breaker opens, fail fast",
],
"jitter_priority": [
"Full jitter — widest spread, many clients",
"Equal jitter — guaranteed minimum wait",
"Decorrelated — most uniform spread",
"None — testing only, causes thundering herd",
],
"retry_priority": [
"1. Retry-After header (server knows best)",
"2. Reset-header math (nearest reset timestamp)",
"3. Exponential backoff with jitter (last resort)",
],
}
LLM Rate Limiting and Retries Checklist
- [ ] HTTP 429 signals rate limit exceeded — handle with exponential backoff and jitter
- [ ] RPM (Requests Per Minute): API calls in rolling 60-second window
- [ ] TPM (Tokens Per Minute): total input + output tokens per 60 seconds
- [ ] Concurrent request limits: in-flight requests at any instant
- [ ] All three limits apply simultaneously — 429 when ANY is exceeded
- [ ] Anthropic enforces three dimensions: RPM, TPM, concurrent — each with own quota and reset
- [ ] OpenAI: free tier 3 RPM, paid 500-10,000 RPM; free 40K TPM, enterprise 2M TPM
- [ ] Token bucket: capacity = TPM/RPM, refill = TPM/60 per second, empty bucket = 429
- [ ] Exponential backoff: ceiling = min(base_delay * 2^attempt, max_delay)
- [ ] Full jitter: delay = random.uniform(0, ceiling) — AWS-recommended, widest spread
- [ ] Equal jitter: ceiling/2 + random.uniform(0, ceiling/2) — guaranteed minimum wait
- [ ] Decorrelated jitter: random.uniform(base, ceiling) — most uniform spread
- [ ] No jitter: causes thundering herd — synchronized retries turn 500 into 5,000
- [ ] Retry-After header: use server value directly — server knows when capacity frees up
- [ ] Add small random jitter (up to 0.5s) to Retry-After to prevent synchronization
- [ ] Retry priority: 1. Retry-After, 2. reset-header math, 3. exponential backoff with jitter
- [ ] Always honor Retry-After — retrying sooner gets you another 429
- [ ] Categorize errors: retryable (429, 500, 502, 503, 504) need backoff
- [ ] Client errors (400, 401, 403) need code fixes — do not retry
- [ ] Transient 500/502 errors: same backoff as 429, resolve within 2-3 retries
- [ ] Circuit breaker: Closed → Open → Half-Open state machine
- [ ] Circuit breaker Closed: normal operation, requests pass through
- [ ] Circuit breaker Open: fail fast, no retries attempted, return fallback
- [ ] Circuit breaker Half-Open: probe with single request, test recovery
- [ ] Circuit breaker transitions: Closed→Open (failure threshold), Open→Half-Open (recovery timeout)
- [ ] Circuit breaker: Half-Open→Closed (probe success), Half-Open→Open (probe failure)
- [ ] Circuit breaker caps blast radius — stops DoSing your own vendor
- [ ] Exponential backoff recovers transient spikes; circuit breaker handles sustained outages
- [ ] Token bucket for proactive throttling — block requests before 429
- [ ] Watch RateLimit-Remaining headers — throttle preemptively when remaining drops
- [ ] Distributed coordinator (Redis) for multi-instance — shared counter prevents overshoot
- [ ] Per-model isolation — separate rate limits for different models
- [ ] Provider presets: Anthropic (rate_limit_error, overloaded_error), OpenAI (server_error, rate_limit_exceeded)
- [ ] Provider presets: Bedrock (ThrottlingException), Gemini (RESOURCE_EXHAUSTED)
- [ ] Retryable HTTP status codes: 429, 500, 502, 503, 504
- [ ] rate-limit-shield: TokenBucket + CircuitBreaker + RetryPolicy with single .call() API
- [ ] rate-limit-shield: LLMShield for per-model isolation, parses Retry-After, OpenAI/Anthropic/Bedrock
- [ ] llm-retry-py: 6 attempts, 500ms base, 30s cap, full jitter, presets for 4 providers
- [ ] llm-retry-py: no HTTP client — wrap any callable; no circuit breaker — layer on top
- [ ] tenacity: wait_exponential_jitter, retry_if_exception_type, before_sleep_log
- [ ] LiteLLM: built-in retries (num_retries) and fallbacks in proxy config
- [ ] Three principles: cap blast radius, jitter or die, honor Retry-After
- [ ] Read LLM gateway guide for gateway concepts
- [ ] Read LiteLLM tutorial for proxy with retries
- [ ] Read LLM streaming SSE for streaming with rate limits
- [ ] Read switch LLM provider for fallback strategies
- [ ] Test: 429 triggers exponential backoff with full jitter
- [ ] Test: Retry-After header honored with jitter
- [ ] Test: circuit breaker opens after failure threshold
- [ ] Test: circuit breaker half-open probes for recovery
- [ ] Test: token bucket blocks requests before 429
- [ ] Test: client errors (400, 401) not retried
- [ ] Test: distributed rate limiting with Redis across instances
- [ ] Document retry config, circuit breaker thresholds, token bucket capacity, and fallback strategy
FAQ
How do you handle LLM API 429 rate limit errors?
Use exponential backoff with full jitter, honor Retry-After header, and implement a token bucket for proactive throttling. SpeedTestHQ: "Exponential backoff with full jitter: on first 429, wait random(0, base_delay). On next, double upper bound: random(0, base_delay * 2). Capped at max (typically 60s). Random jitter prevents thundering herd. If Retry-After header present, use that value directly — server knows when capacity frees up." ofox.ai: "retry-after header is your best friend. Always honor this value — retrying sooner gets you another 429. Categorize errors: retryable (429, 5xx) need backoff, client errors (400, 401, 403) need code fixes." SitePoint: "Prioritize: use Retry-After when present, fall back to reset-header math, resort to exponential backoff as last fallback. Add small random jitter (up to 0.5s) to prevent synchronization."
What are RPM and TPM rate limits in LLM APIs?
RPM (Requests Per Minute) counts API calls in a 60-second window. TPM (Tokens Per Minute) counts total input plus output tokens. Both apply simultaneously. SpeedTestHQ: "TPM: total input + output tokens in rolling 60-second window. Binding constraint for large prompts. RPM: count of requests regardless of token size. Binding constraint for many small prompts. You hit 429 when EITHER limit is exceeded — both apply simultaneously." ofox.ai: "RPM: OpenAI free tier 3 RPM on GPT-4, paid 500-10,000 RPM. TPM: free 40,000, enterprise 2,000,000. Token bucket: capacity = TPM/RPM, tokens regenerate at TPM/60 per second, each request consumes tokens, empty bucket = 429." SitePoint: "Anthropic enforces three simultaneously: request-rate (RPM), token-rate (TPM), and concurrent-request limits. Each has own quota and reset window."
How do you implement exponential backoff with jitter for LLM APIs?
Use full jitter: random.uniform(0, min(base * 2^attempt, max_delay)). llm-retry-py: "Defaults: 6 attempts, 500ms base delay, 30s cap, full jitter. Backoff for attempt i is min(base * 2^i, max), then jittered. FULL jitter: uniform(0, capped) — AWS-recommended. EQUAL: capped/2 + uniform(0, capped/2). NONE: capped." ofox.ai: "Full jitter (random.uniform(0, exp_delay)) provides widest spread, works best with many clients. Equal jitter guarantees at least half the exponential wait. Decorrelated jitter produces most uniform spread." SpeedTestHQ: "ceiling = min(base_delay * (2 ** retry_count), max_delay). delay = random.uniform(0, ceiling). Same backoff for 500+ server errors." Code: RetryConfig(max_retries=5, base_delay=1.0, max_delay=60.0, jitter_mode='full').
How does a circuit breaker work for LLM API reliability?
Circuit breaker fails fast when retries are consistently futile, preventing cascading failures. SitePoint: "Exponential backoff recovers from transient 429 spikes. It does not handle sustained unavailability. When Anthropic enforces org-level rate limit or experiences incident lasting minutes, retries consume compute, hold threads in sleep, delay fallback. Circuit breaker fails fast once system detects retries are consistently futile." rate-limit-shield: "CircuitBreaker: Closed → Open → Half-Open state machine. Three states, four transitions — breaker fails fast when upstream degrades, then probes for recovery. TokenBucket + CircuitBreaker + RetryPolicy composed with single .call() API. Per-model isolation, parses Retry-After, designed for OpenAI/Anthropic/Bedrock." States: Closed (normal), Open (fail fast, no retries), Half-Open (probe with single request). Transitions: Closed→Open (failure threshold reached), Open→Half-Open (recovery timeout), Half-Open→Closed (probe success), Half-Open→Open (probe failure).
What libraries handle LLM rate limiting and retries in Python?
Several libraries provide production-grade rate limiting and retries for LLM APIs. rate-limit-shield: "Production-grade rate limiting, circuit breaking, and retry shaping. TokenBucket + CircuitBreaker + RetryPolicy with single .call() API. LLMShield for per-model isolation, parses Retry-After, designed for OpenAI/Anthropic/Bedrock. Every team hand-rolls same three primitives — rate-limit-shield ships the 100% version." llm-retry-py: "Exponential backoff with full jitter for LLM API calls. Sync and async share same RetryPolicy. Built-in presets for Anthropic, OpenAI, Bedrock, Gemini. No HTTP client — wrap any callable that raises. No circuit breaker — layer one on top." SitePoint: "tenacity library with custom wait callback: inspect Retry-After and reset headers first, add jitter, fall back to exponential backoff. before_sleep_log hook logs every retry event." Libraries: rate-limit-shield (token bucket + breaker + retry), llm-retry-py (backoff + jitter + presets), tenacity (general retry with custom wait), LiteLLM (built-in retries and fallbacks in proxy).
Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →