LLM Fallbacks and Automatic Switching: Failover Chains for Production AI

Q: What are LLM fallbacks and automatic switching?

LLM fallbacks automatically switch from a primary model to a secondary model when the primary fails. [Maxim AI](https://www.getmaxim.ai/articles/retries-fallbacks-and-circuit-breakers-in-llm-apps-a-production-guide/): 'Retries automatically recover from transient errors like rate limits and network timeouts. Fallbacks seamlessly switch between providers when the primary option fails. Circuit breakers prevent cascading failures by stopping requests to a failing provider.' [BuildMVPFast](https://www.buildmvpfast.com/blog/llm-fallback-strategies-primary-model-secondary-model-2026): 'Define your chain: primary model, same-provider fallback, cross-provider fallback. Three levels minimum. Order by cost. If under 1,000 RPS, LiteLLM. If no infrastructure, OpenRouter.' [OpenRouter Docs](https://openrouter.ai/docs/guides/routing/model-fallbacks): 'Automatic failover between models.' Fallback chain: primary (best quality) → same-provider fallback (same provider, different model) → cross-provider fallback (different provider entirely) → graceful degradation (cached or static response).

Q: How do you configure LLM fallbacks with LiteLLM?

Use fallbacks list in router_settings in config.yaml. [LiteLLM Docs](https://docs.litellm.ai/docs/routing): 'When a deployment in a model group fails, the router moves on to the next entry in fallbacks (a different model group). With enable_weighted_failover, the router first retries inside the same model group by re-picking a different deployment, and only escalates to cross-group fallbacks once every deployment in the group has been tried. Capped by max_fallbacks (default 5). Cooldowns apply: a deployment that crosses allowed_fails is cooled down.' Config: router_settings: fallbacks: [{claude: [gpt-4o, gemini-flash]}]. enable_weighted_failover: true for same-group retry first. allowed_fails: 3, num_retries: 2, request_timeout: 30. Not triggered for ContextWindowExceededError or ContentPolicyViolationError — those have dedicated fallback paths.

Q: How do OpenRouter model fallbacks work?

OpenRouter has two layers: provider-level failover (automatic within a model) and model-level fallback (models array in priority order). [OpenRouter Blog](https://openrouter.ai/blog/insights/reliability-failover/): 'Routes around provider outages in real time using a 30-second health window and published per-model uptime. Worst-case uptime better than any single provider integrated directly. Not zero-risk: August 2025 gateway outage showed routing layer has its own failure mode.' [OpenRouter Docs](https://openrouter.ai/docs/guides/routing/model-fallbacks): 'Automatic failover between models.' Provider-level: if chosen provider returns 5xx or rate-limits, OpenRouter falls through to next provider serving that model. On by default (allow_fallbacks: true). Model-level: pass models array in priority order. Fallbacks trigger on context-length errors, moderation flags, rate-limiting, and downtime. Billed for model that actually runs, not ones that errored.

Q: What is the difference between retries and fallbacks in LLM apps?

Retries repeat the same request to the same provider. Fallbacks switch to a different provider or model. [Maxim AI](https://www.getmaxim.ai/articles/retries-fallbacks-and-circuit-breakers-in-llm-apps-a-production-guide/): 'Retries automatically recover from transient errors like rate limits and network timeouts. Fallbacks seamlessly switch between providers when the primary option fails. Circuit breakers prevent cascading failures by stopping requests to a failing provider.' [BuildMVPFast](https://www.buildmvpfast.com/blog/building-with-unreliable-ai-error-handling-fallback-strategies-2026): 'OpenAI largest outage: June 2025, 34 hours of downtime. Routing nodes hit memory limits. Nodes went dark until not enough capacity left. DesignRush study estimated outage cost.' Retries: same provider, exponential backoff, for transient errors (429, 5xx, timeout). Fallbacks: different provider/model, for sustained failures, provider outages. Circuit breaker: stops all requests to failing provider, fails fast. Use all three: retry first (fast recovery), then fallback (switch provider), then circuit breaker (stop cascading).

Q: How do you build a production LLM fallback chain?

Define a multi-level chain: primary, same-provider fallback, cross-provider fallback, graceful degradation. [BuildMVPFast](https://www.buildmvpfast.com/blog/llm-fallback-strategies-primary-model-secondary-model-2026): 'Primary model, same-provider fallback, cross-provider fallback. Three levels minimum. Order by cost. If under 1,000 RPS, LiteLLM. If enterprise observability, Portkey. If latency is everything, Bifrost. If no infrastructure, OpenRouter.' [Maxim AI](https://www.getmaxim.ai/articles/retries-fallbacks-and-circuit-breakers-in-llm-apps-a-production-guide/): 'Retries recover transient errors. Fallbacks switch providers. Circuit breakers prevent cascading. Build all three layers.' Chain: (1) Primary: best quality model (e.g., Claude 3.5 Sonnet). (2) Same-provider fallback: same provider, different model (e.g., Claude 3 Haiku). (3) Cross-provider fallback: different provider (e.g., GPT-4o). (4) Budget fallback: cheap model (e.g., Gemini Flash). (5) Graceful degradation: cached response or static message. Configure in LiteLLM config.yaml or OpenRouter models array.

By Fadi Updated Jan 14, 2026 12 min read LLM Gateways

TL;DR — LLM fallbacks and automatic switching: failover chains for production. Maxim AI: "Retries recover transient errors. Fallbacks switch providers. Circuit breakers prevent cascading. Build all three layers." BuildMVPFast: "Primary, same-provider fallback, cross-provider fallback. Three levels minimum. Order by cost. LiteLLM for <1K RPS, OpenRouter for no infra." OpenRouter Blog: "30-second health window, per-model uptime. Worst-case uptime better than any single provider. Not zero-risk — August 2025 gateway outage." LiteLLM Docs: "Fallbacks move to next model group. enable_weighted_failover retries within group first. max_fallbacks: 5. Cooldowns for failed deployments." BuildMVPFast: "OpenAI June 2025: 34 hours downtime. Nodes hit memory limits. Build for failure." Learn more with LLM gateway, LiteLLM tutorial, rate limiting, and LiteLLM with OpenRouter.

Maxim AI provides the production framework: "Building production LLM applications requires robust failure handling strategies. Retries automatically recover from transient errors like rate limits and network timeouts. Fallbacks seamlessly switch between providers when the primary option fails. Circuit breakers prevent cascading failures by stopping requests to a failing provider."

BuildMVPFast gives the real-world stakes: "OpenAI's largest outage hit in June 2025: 34 hours of downtime across ChatGPT, the API, and Sora. The root cause? Routing nodes hit memory limits and failed readiness checks. One by one, nodes went dark until there wasn't enough capacity left to serve anyone."

Fallback Architecture

flowchart TD subgraph Request["Incoming Request"] Incoming["User Request
to LLM API"] end subgraph Retry["Layer 1: Retry"] SameProvider["Same Provider Retry
exponential backoff
full jitter
max 3 attempts
for transient errors: 429, 5xx"] end subgraph Fallback["Layer 2: Fallback Chain"] Primary["Primary Model
best quality
e.g., Claude 3.5 Sonnet"] SameFallback["Same-Provider Fallback
same provider, cheaper model
e.g., Claude 3 Haiku"] CrossFallback["Cross-Provider Fallback
different provider
e.g., GPT-4o"] BudgetFallback["Budget Fallback
cheap model
e.g., Gemini Flash"] Degradation["Graceful Degradation
cached response
or static message"] end subgraph Breaker["Layer 3: Circuit Breaker"] Closed["Closed
requests pass through"] Open["Open
fail fast, skip provider
go to next in chain"] HalfOpen["Half-Open
probe with single request
test recovery"] end subgraph Triggers["Fallback Triggers"] RateLimit["Rate Limit (429)"] ServerError["Server Error (5xx)"] Timeout["Request Timeout"] Outage["Provider Outage"] ContextError["Context Window Exceeded"] Moderation["Content Policy Violation"] end subgraph Gateways["Gateway Implementations"] LiteLLM["LiteLLM
fallbacks in config.yaml
enable_weighted_failover
max_fallbacks: 5"] OpenRouter["OpenRouter
models array
provider failover
30s health window"] end Incoming --> SameProvider SameProvider --> Primary Primary --> SameFallback SameFallback --> CrossFallback CrossFallback --> BudgetFallback BudgetFallback --> Degradation Primary --> Closed Closed --> Open Open --> HalfOpen HalfOpen --> Closed RateLimit --> SameProvider ServerError --> SameProvider Timeout --> SameProvider Outage --> Primary ContextError --> CrossFallback Moderation --> CrossFallback LiteLLM --> Fallback OpenRouter --> Fallback

Retry vs Fallback vs Circuit Breaker

Layer	What It Does	When To Use	Recovery
Retry	Repeat same request to same provider	Transient errors (429, 5xx, timeout)	Seconds (exponential backoff)
Fallback	Switch to different provider/model	Sustained failures, provider outage	Immediate (next in chain)
Circuit Breaker	Stop all requests to failing provider	Cascading failures, repeated errors	Minutes (recovery timeout)

Fallback Chain Levels

Level	Type	Example	Cost	Quality
1. Primary	Best quality	Claude 3.5 Sonnet	$$$	Highest
2. Same-provider	Same provider, cheaper	Claude 3 Haiku	$$	High
3. Cross-provider	Different provider	GPT-4o	$$$	High
4. Budget	Cheap model	Gemini 2.0 Flash	$	Medium
5. Degradation	Cached/static	Cached response	$0	N/A

Implementation

from dataclasses import dataclass
from typing import Optional
from enum import Enum
import random
import time

class FallbackTrigger(Enum):
    RATE_LIMIT = "rate_limit"
    SERVER_ERROR = "server_error"
    TIMEOUT = "timeout"
    OUTAGE = "outage"
    CONTEXT_ERROR = "context_error"
    MODERATION = "moderation"

@dataclass
class LLMFallbacksGuide:
    """LLM fallbacks and automatic switching guide."""

    def get_litellm_config(self) -> str:
        """LiteLLM fallbacks config.yaml."""
        return (
            "# config.yaml\n"
            "# LiteLLM fallback chain\n"
            "\n"
            "model_list:\n"
            "  - model_name: claude\n"
            "    litellm_params:\n"
            "      model: anthropic/claude\n"
            "        -3-5-sonnet\n"
            "      api_key: os.environ/\n"
            "        ANTHROPIC_API_KEY\n"
            "      rpm: 50\n"
            "\n"
            "  - model_name: claude-haiku\n"
            "    litellm_params:\n"
            "      model: anthropic/claude\n"
            "        -3-haiku\n"
            "      api_key: os.environ/\n"
            "        ANTHROPIC_API_KEY\n"
            "      rpm: 100\n"
            "\n"
            "  - model_name: gpt-4o\n"
            "    litellm_params:\n"
            "      model: openai/gpt-4o\n"
            "      api_key: os.environ/\n"
            "        OPENAI_API_KEY\n"
            "      rpm: 100\n"
            "\n"
            "  - model_name: gemini-flash\n"
            "    litellm_params:\n"
            "      model: gemini/gemini-2.0\n"
            "        -flash\n"
            "      api_key: os.environ/\n"
            "        GOOGLE_API_KEY\n"
            "      rpm: 200\n"
            "\n"
            "router_settings:\n"
            "  routing_strategy: simple-shuffle\n"
            "  enable_weighted_failover: true\n"
            "  allowed_fails: 3\n"
            "  num_retries: 2\n"
            "  request_timeout: 30\n"
            "  max_fallbacks: 5\n"
            "  fallbacks:\n"
            "    - claude:\n"
            "        - claude-haiku\n"
            "        - gpt-4o\n"
            "        - gemini-flash\n"
            "    - claude-haiku:\n"
            "        - gpt-4o\n"
            "        - gemini-flash\n"
            "    - gpt-4o:\n"
            "        - gemini-flash\n"
            "\n"
            "# enable_weighted_failover:\n"
            "# retry within same model\n"
            "# group first, then cross-\n"
            "# group fallbacks. Capped\n"
            "# by max_fallbacks: 5.\n"
            "# Cooldowns apply for\n"
            "# failed deployments.\n"
            "# Not triggered for\n"
            "# ContextWindowExceededError\n"
            "# or ContentPolicyViolationError."
        )

    def get_openrouter_fallback(self) -> str:
        """OpenRouter model fallback."""
        return (
            "from openai import OpenAI\n"
            "\n"
            "client = OpenAI(\n"
            "    base_url='https://openrouter.ai'\n"
            "        '/api/v1',\n"
            "    api_key='sk-or-v1-xxx',\n"
            ")\n"
            "\n"
            "# Model fallback: try primary,\n"
            "# then each fallback in order\n"
            "response = client.chat\\\n"
            "    .completions.create(\n"
            "    model='anthropic/claude'\n"
            "        '-3.5-sonnet',\n"
            "    messages=[\n"
            "        {'role': 'user',\n"
            "         'content': 'Hello!'}\n"
            "    ],\n"
            "    extra_body={\n"
            "        'models': [\n"
            "            'anthropic/claude'\n"
            "                '-3.5-sonnet',\n"
            "            'openai/gpt-4o',\n"
            "            'google/gemini-2.0'\n"
            "                '-flash',\n"
            "        ],\n"
            "    },\n"
            ")\n"
            "# Billed for model that ran\n"
            "# response.model shows which\n"
            "print(f'Served by: '\n"
            "      f'{response.model}')\n"
            "\n"
            "# Provider failover: automatic\n"
            "# within model (30s health\n"
            "# window, on by default)\n"
            "response = client.chat\\\n"
            "    .completions.create(\n"
            "    model='openai/gpt-4o',\n"
            "    messages=[...],\n"
            "    extra_body={\n"
            "        'provider': {\n"
            "            'allow_fallbacks': True,\n"
            "        }\n"
            "    }\n"
            ")"
        )

    def get_custom_fallback(self) -> str:
        """Custom fallback chain in Python."""
        return (
            "import random\n"
            "import time\n"
            "from openai import OpenAI\n"
            "\n"
            "client = OpenAI()\n"
            "\n"
            "FALLBACK_CHAIN = [\n"
            "    {'model': 'gpt-4o',\n"
            "     'api_key': 'sk-xxx'},\n"
            "    {'model': 'claude-3-5-sonnet',\n"
            "     'base_url': 'https://'\n"
            "       'openrouter.ai/api/v1',\n"
            "     'api_key': 'sk-or-xxx'},\n"
            "    {'model': 'gemini-2.0-flash',\n"
            "     'base_url': 'https://'\n"
            "       'openrouter.ai/api/v1',\n"
            "     'api_key': 'sk-or-xxx'},\n"
            "]\n"
            "\n"
            "def call_with_fallback(\n"
            "    messages: list,\n"
            "    max_retries: int = 3\n"
            "):\n"
            "    '''Call with fallback chain.'''\n"
            "    for i, provider in enumerate(\n"
            "        FALLBACK_CHAIN):\n"
            "        for attempt in range(\n"
            "            max_retries):\n"
            "            try:\n"
            "                c = OpenAI(\n"
            "                    base_url=provider\n"
            "                        .get('base_url',\n"
            "                        'https://api.'\n"
            "                        'openai.com/v1'),\n"
            "                    api_key=provider[\n"
            "                        'api_key'])\n"
            "                return c.chat\\\n"
            "                    .completions\\\n"
            "                    .create(\n"
            "                    model=provider[\n"
            "                        'model'],\n"
            "                    messages=messages)\n"
            "            except Exception as e:\n"
            "                if attempt < \\\n"
            "                    max_retries - 1:\n"
            "                    delay = random\\\n"
            "                        .uniform(\n"
            "                            0, 2 ** attempt)\n"
            "                    time.sleep(delay)\n"
            "                else:\n"
            "                    print(\n"
            "                        f'Provider {i} '\n"
            "                        f'failed: {e}')\n"
            "                    continue\n"
            "    # Graceful degradation\n"
            "    return {\n"
            "        'choices': [{\n"
            "            'message': {\n"
            "                'content': 'Service '\n"
            "                    'temporarily '\n"
            "                    'unavailable.'\n"
            "            }\n"
            "        }]\n"
            "    }"
        )

    def get_graceful_degradation(self) -> dict:
        """Graceful degradation strategies."""
        return {
            "cached_response": (
                "Return cached response from "
                "previous successful call. "
                "Redis cache with TTL."
            ),
            "static_message": (
                "Return static message: "
                "'Service temporarily "
                "unavailable. Please retry.'"
            ),
            "queue_retry": (
                "Queue request for later "
                "processing. Notify user "
                "when response ready."
            ),
            "simplified_response": (
                "Use a simpler/cheaper model "
                "with reduced context for "
                "basic response."
            ),
            "partial_response": (
                "Return partial results "
                "from streaming if "
                "connection interrupted."
            ),
        }

    def get_trigger_types(self) -> dict:
        """Fallback trigger types."""
        return {
            "rate_limit_429": (
                "Provider rate limited. "
                "Retry with backoff first, "
                "then fallback to next "
                "provider."
            ),
            "server_error_5xx": (
                "Provider server error. "
                "Retry 2-3 times, then "
                "fallback to next provider."
            ),
            "timeout": (
                "Request exceeded timeout. "
                "Fallback immediately to "
                "faster provider."
            ),
            "outage": (
                "Provider outage (sustained). "
                "Circuit breaker opens, "
                "all requests fallback."
            ),
            "context_error": (
                "Context window exceeded. "
                "Skip to model with larger "
                "context window. Dedicated "
                "fallback path in LiteLLM."
            ),
            "moderation": (
                "Content policy violation. "
                "Skip to provider with "
                "different moderation policy. "
                "Dedicated fallback path."
            ),
        }

    def get_best_practices(self) -> dict:
        """Best practices."""
        return {
            "three_layers": [
                "Retry: same provider, exponential backoff, transient errors",
                "Fallback: switch provider/model, sustained failures",
                "Circuit breaker: stop cascading, fail fast",
            ],
            "chain_design": [
                "Three levels minimum: primary, same-provider, cross-provider",
                "Order by cost: expensive first, cheap last",
                "Include graceful degradation as final fallback",
                "Test each level independently",
            ],
            "gateway_choice": [
                "LiteLLM: under 1,000 RPS, self-hosted, full control",
                "OpenRouter: no infrastructure, managed failover",
                "Portkey: enterprise observability",
                "Bifrost: latency-critical",
            ],
            "monitoring": [
                "Track which model served each request (response.model)",
                "Log fallback events and trigger reasons",
                "Monitor provider uptime and latency",
                "Alert on fallback rate exceeding threshold",
                "Track cost per fallback level",
            ],
        }

LLM Fallbacks and Automatic Switching Checklist

[ ] LLM fallbacks automatically switch from primary model to secondary when primary fails
[ ] Three layers of failure handling: retry, fallback, circuit breaker
[ ] Retry: same provider, exponential backoff, for transient errors (429, 5xx, timeout)
[ ] Fallback: different provider/model, for sustained failures and provider outages
[ ] Circuit breaker: stop all requests to failing provider, fail fast, prevent cascading
[ ] OpenAI June 2025 outage: 34 hours downtime — build for failure
[ ] Fallback chain: primary (best quality) → same-provider fallback → cross-provider fallback → budget → degradation
[ ] Three levels minimum: primary, same-provider fallback, cross-provider fallback
[ ] Order chain by cost: expensive first, cheap last
[ ] Include graceful degradation as final fallback (cached, static, queued)
[ ] LiteLLM fallbacks: config.yaml router_settings with fallbacks list
[ ] LiteLLM: fallbacks: [{claude: [claude-haiku, gpt-4o, gemini-flash]}]
[ ] LiteLLM: enable_weighted_failover retries within same model group first, then cross-group
[ ] LiteLLM: max_fallbacks: 5 (default cap), cooldowns for failed deployments
[ ] LiteLLM: allowed_fails: 3, num_retries: 2, request_timeout: 30
[ ] LiteLLM: not triggered for ContextWindowExceededError or ContentPolicyViolationError — dedicated paths
[ ] OpenRouter: two layers — provider-level failover (automatic, 30s health window) and model-level fallback (models array)
[ ] OpenRouter: provider failover on by default (allow_fallbacks: true), 30s outage deprioritization
[ ] OpenRouter: model fallback via models array in priority order via extra_body
[ ] OpenRouter: fallbacks trigger on context-length errors, moderation flags, rate-limiting, downtime
[ ] OpenRouter: billed for model that actually runs, not ones that errored — check response.model
[ ] OpenRouter: not zero-risk — August 2025 gateway outage showed routing layer has own failure mode
[ ] Fallback triggers: rate limit (429), server error (5xx), timeout, outage, context error, moderation
[ ] Context window exceeded: skip to model with larger context window (dedicated fallback path)
[ ] Content policy violation: skip to provider with different moderation policy (dedicated fallback path)
[ ] Graceful degradation: cached response, static message, queue retry, simplified response, partial response
[ ] Custom fallback chain: iterate providers, retry each, then graceful degradation
[ ] Gateway choice: LiteLLM (<1K RPS), OpenRouter (no infra), Portkey (enterprise), Bifrost (latency)
[ ] Track which model served each request via response.model
[ ] Log fallback events and trigger reasons for observability
[ ] Monitor provider uptime and latency
[ ] Alert on fallback rate exceeding threshold (indicates primary provider issues)
[ ] Track cost per fallback level — fallbacks may use more expensive models
[ ] Circuit breaker states: Closed → Open (failure threshold) → Half-Open (recovery timeout) → Closed
[ ] Circuit breaker: Open state skips provider entirely, goes to next in chain
[ ] Circuit breaker: Half-Open probes with single request to test recovery
[ ] Cooldowns: deployment crossing allowed_fails is cooled down independently
[ ] enable_weighted_failover: exclusions accumulate across hops, failed deployment never picked again
[ ] Test each fallback level independently
[ ] Test: primary failure triggers same-provider fallback
[ ] Test: all same-provider failures trigger cross-provider fallback
[ ] Test: circuit breaker opens after allowed_fails threshold
[ ] Test: graceful degradation returns cached/static response when all providers fail
[ ] Read LLM gateway guide for gateway concepts
[ ] Read LiteLLM tutorial for proxy with fallbacks
[ ] Read rate limiting for retry strategies
[ ] Read LiteLLM with OpenRouter for combined setup
[ ] Document fallback chain, trigger types, circuit breaker config, and degradation strategy

FAQ

What are LLM fallbacks and automatic switching?

LLM fallbacks automatically switch from a primary model to a secondary model when the primary fails. Maxim AI: "Retries automatically recover from transient errors like rate limits and network timeouts. Fallbacks seamlessly switch between providers when the primary option fails. Circuit breakers prevent cascading failures by stopping requests to a failing provider." BuildMVPFast: "Define your chain: primary model, same-provider fallback, cross-provider fallback. Three levels minimum. Order by cost. If under 1,000 RPS, LiteLLM. If no infrastructure, OpenRouter." OpenRouter Docs: "Automatic failover between models." Fallback chain: primary (best quality) → same-provider fallback (same provider, different model) → cross-provider fallback (different provider entirely) → graceful degradation (cached or static response).

How do you configure LLM fallbacks with LiteLLM?

Use fallbacks list in router_settings in config.yaml. LiteLLM Docs: "When a deployment in a model group fails, the router moves on to the next entry in fallbacks (a different model group). With enable_weighted_failover, the router first retries inside the same model group by re-picking a different deployment, and only escalates to cross-group fallbacks once every deployment in the group has been tried. Capped by max_fallbacks (default 5). Cooldowns apply: a deployment that crosses allowed_fails is cooled down." Config: router_settings: fallbacks: [{claude: [gpt-4o, gemini-flash]}]. enable_weighted_failover: true for same-group retry first. allowed_fails: 3, num_retries: 2, request_timeout: 30. Not triggered for ContextWindowExceededError or ContentPolicyViolationError — those have dedicated fallback paths.

How do OpenRouter model fallbacks work?

OpenRouter has two layers: provider-level failover (automatic within a model) and model-level fallback (models array in priority order). OpenRouter Blog: "Routes around provider outages in real time using a 30-second health window and published per-model uptime. Worst-case uptime better than any single provider integrated directly. Not zero-risk: August 2025 gateway outage showed routing layer has its own failure mode." OpenRouter Docs: "Automatic failover between models." Provider-level: if chosen provider returns 5xx or rate-limits, OpenRouter falls through to next provider serving that model. On by default (allow_fallbacks: true). Model-level: pass models array in priority order. Fallbacks trigger on context-length errors, moderation flags, rate-limiting, and downtime. Billed for model that actually runs, not ones that errored.

What is the difference between retries and fallbacks in LLM apps?

Retries repeat the same request to the same provider. Fallbacks switch to a different provider or model. Maxim AI: "Retries automatically recover from transient errors like rate limits and network timeouts. Fallbacks seamlessly switch between providers when the primary option fails. Circuit breakers prevent cascading failures by stopping requests to a failing provider." BuildMVPFast: "OpenAI largest outage: June 2025, 34 hours of downtime. Routing nodes hit memory limits. Nodes went dark until not enough capacity left. DesignRush study estimated outage cost." Retries: same provider, exponential backoff, for transient errors (429, 5xx, timeout). Fallbacks: different provider/model, for sustained failures, provider outages. Circuit breaker: stops all requests to failing provider, fails fast. Use all three: retry first (fast recovery), then fallback (switch provider), then circuit breaker (stop cascading).

How do you build a production LLM fallback chain?

Define a multi-level chain: primary, same-provider fallback, cross-provider fallback, graceful degradation. BuildMVPFast: "Primary model, same-provider fallback, cross-provider fallback. Three levels minimum. Order by cost. If under 1,000 RPS, LiteLLM. If enterprise observability, Portkey. If latency is everything, Bifrost. If no infrastructure, OpenRouter." Maxim AI: "Retries recover transient errors. Fallbacks switch providers. Circuit breakers prevent cascading. Build all three layers." Chain: (1) Primary: best quality model (e.g., Claude 3.5 Sonnet). (2) Same-provider fallback: same provider, different model (e.g., Claude 3 Haiku). (3) Cross-provider fallback: different provider (e.g., GPT-4o). (4) Budget fallback: cheap model (e.g., Gemini Flash). (5) Graceful degradation: cached response or static message. Configure in LiteLLM config.yaml or OpenRouter models array.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →