LiteLLM with OpenRouter: Combined Gateway for Local Control and Managed Breadth

By Fadi Updated Aug 6, 2025 12 min read LLM Gateways

TL;DR — LiteLLM with OpenRouter: combined gateway. OpenRouter Blog: "Not mutually exclusive. LiteLLM uses OpenRouter as upstream — local RBAC and logging while OpenRouter handles multi-provider failover and model breadth. Switching is base URL and key change." LiteLLM Docs: "Supports all OpenRouter models. model=openrouter/ prefix. api_base defaults to openrouter.ai/api/v1. Pass transforms, models, route." OSINT Team: "LiteLLM routes to OpenRouter. Single OpenAI-compatible endpoint. Handles routing, auth, logging, fallback, budgets, observability." Markaicode: "config.yaml controls routing, cost tracking, access. OpenRouter as primary with fallback to GPT-4o. Redis max_connections 100, cache TTL 300s. Langfuse logging." Learn more with LLM gateway, OpenRouter vs LiteLLM, LiteLLM tutorial, and OpenRouter tutorial.

OpenRouter Blog states the key insight: "The two aren't mutually exclusive. LiteLLM can use OpenRouter as an upstream provider, so you get LiteLLM's local RBAC and logging while OpenRouter handles multi-provider failover and model breadth."

LiteLLM Docs confirms: "LiteLLM supports all the text / chat / vision / embedding models from OpenRouter. Send model=openrouter/ to send it to OpenRouter."

Combined Architecture

flowchart TD subgraph App["Your Application"] SDK["OpenAI SDK
base_url: localhost:4000
api_key: LiteLLM virtual key"] end subgraph LiteLLM["LiteLLM Proxy (Your VPC)"] Config["config.yaml
model_list with openrouter/ prefix
router_settings
litellm_settings"] RBAC["Local Control
virtual keys with budgets
per-team RBAC
SSO/SAML (Enterprise)"] Cache["Redis Caching
TTL: 300s
p95: 193ms on hits
max_connections: 100"] Logging["Observability
Langfuse integration
spend tracking in PostgreSQL
proxy_logging callbacks"] Budget["Budget Enforcement
check_key_budget: true
max_budget per key
Slack alerts"] Router["Router
6 routing modes
fallback chains
allowed_fails, num_retries"] end subgraph OpenRouter["OpenRouter (Managed SaaS)"] ORGateway["OpenRouter Gateway
Cloudflare edge
500+ models, 70+ providers"] ORFailover["Provider Failover
automatic within model
30s outage deprioritization"] ORFree["Free Models
DeepSeek R1, Llama 3.3
Gemma 3"] ORBYOK["BYOK
5% fee with own keys
prioritized/fallback"] end subgraph Direct["Direct Providers (Optional)"] OpenAI["OpenAI direct
gpt-4o"] Anthropic["Anthropic direct
claude-3.5-sonnet"] Local["Local Models
Ollama, vLLM"] end subgraph Infra["Infrastructure"] Postgres["PostgreSQL
spend data, keys
audit logs"] Redis["Redis
caching, rate limits
session state"] Docker["Docker Compose
litellm + db + redis"] end SDK --> Config Config --> RBAC RBAC --> Cache Cache --> Logging Logging --> Budget Budget --> Router Router --> ORGateway Router --> OpenAI Router --> Anthropic Router --> Local ORGateway --> ORFailover ORGateway --> ORFree ORGateway --> ORBYOK Docker --> Postgres Docker --> Redis Docker --> LiteLLM

LiteLLM vs OpenRouter vs Combined

Feature	LiteLLM Only	OpenRouter Only	Combined
Models	100+ providers	500+ models	500+ via OpenRouter + direct
RBAC	Enterprise	No	Yes (LiteLLM)
Budget enforcement	Per-key, per-team	Basic	Full (LiteLLM)
Caching	Redis-backed	No	Yes (LiteLLM Redis)
Observability	Pluggable	Basic	Full (LiteLLM + Langfuse)
Provider failover	Configurable	Automatic	Double safety net
Data residency	Your VPC	OpenRouter infra	Your VPC (LiteLLM)
Free models	No	Yes	Yes (via OpenRouter)
Markup	0%	5-5.5%	5% on OpenRouter models
Infrastructure	Docker + PG + Redis	None	Docker + PG + Redis

Implementation

from dataclasses import dataclass
from typing import Optional
from enum import Enum

class RoutingMode(Enum):
    LATENCY = "latency-based"
    LOWEST_COST = "lowest-cost"
    SIMPLE_SHUFFLE = "simple-shuffle"

@dataclass
class LiteLLMWithOpenRouterGuide:
    """LiteLLM with OpenRouter combined setup guide."""

    def get_config_yaml(self) -> str:
        """config.yaml with OpenRouter as upstream."""
        return (
            "# config.yaml\n"
            "# LiteLLM with OpenRouter upstream\n"
            "\n"
            "model_list:\n"
            "  # OpenRouter models\n"
            "  - model_name: claude\n"
            "    litellm_params:\n"
            "      model: openrouter/\n"
            "        anthropic/claude-3.5\n"
            "        -sonnet\n"
            "      api_key: os.environ/\n"
            "        OPENROUTER_API_KEY\n"
            "      api_base: https://\n"
            "        openrouter.ai/api/v1\n"
            "      rpm: 10\n"
            "      fallback:\n"
            "        - gpt-4o-backup\n"
            "\n"
            "  - model_name: gpt-4o\n"
            "    litellm_params:\n"
            "      model: openrouter/\n"
            "        openai/gpt-4o\n"
            "      api_key: os.environ/\n"
            "        OPENROUTER_API_KEY\n"
            "      api_base: https://\n"
            "        openrouter.ai/api/v1\n"
            "      rpm: 20\n"
            "\n"
            "  - model_name: gpt-4o-backup\n"
            "    litellm_params:\n"
            "      model: openrouter/\n"
            "        openai/gpt-4o\n"
            "      api_key: os.environ/\n"
            "        OPENROUTER_API_KEY\n"
            "      rpm: 20\n"
            "\n"
            "  # Direct provider (bypass OR)\n"
            "  - model_name: direct-gpt4o\n"
            "    litellm_params:\n"
            "      model: openai/gpt-4o\n"
            "      api_key: os.environ/\n"
            "        OPENAI_API_KEY\n"
            "\n"
            "  # Free model via OpenRouter\n"
            "  - model_name: free\n"
            "    litellm_params:\n"
            "      model: openrouter/\n"
            "        google/gemma-3:free\n"
            "      api_key: os.environ/\n"
            "        OPENROUTER_API_KEY\n"
            "\n"
            "  # Local model\n"
            "  - model_name: local\n"
            "    litellm_params:\n"
            "      model: ollama/llama3\n"
            "      api_base: http://\n"
            "        localhost:11434\n"
            "\n"
            "router_settings:\n"
            "  routing_strategy: latency-based\n"
            "  allowed_fails: 3\n"
            "  num_retries: 2\n"
            "  request_timeout: 30\n"
            "  fallbacks:\n"
            "    - claude: [gpt-4o, free]\n"
            "    - gpt-4o: [free]\n"
            "\n"
            "litellm_settings:\n"
            "  cache: true\n"
            "  cache_params:\n"
            "    type: redis\n"
            "    host: redis\n"
            "    port: 6379\n"
            "    max_connections: 100\n"
            "    response_cache_ttl: 300\n"
            "\n"
            "general_settings:\n"
            "  database_url: os.environ/\n"
            "    DATABASE_URL\n"
            "  master_key: sk-1234\n"
            "  alerting: ['slack']\n"
            "\n"
            "litellm_pre_call_actions:\n"
            "  check_key_budget: true\n"
            "\n"
            "proxy_logging:\n"
            "  - callback: langfuse\n"
            "  - callback: newrelic"
        )

    def get_env_vars(self) -> str:
        """Environment variables."""
        return (
            "# .env file\n"
            "OPENROUTER_API_KEY=sk-or-v1-xxx\n"
            "OPENROUTER_API_BASE=https://openrouter.ai\n"
            "  /api/v1\n"
            "OPENAI_API_KEY=sk-xxx  # direct\n"
            "ANTHROPIC_API_KEY=sk-ant-xxx  # direct\n"
            "DATABASE_URL=postgresql://\n"
            "  litellm:pass@db:5432/litellm\n"
            "REDIS_URL=redis://redis:6379\n"
            "LITELLM_MASTER_KEY=sk-1234\n"
            "LITELLM_SALT_KEY=sk-1234\n"
            "SLACK_WEBHOOK_URL=https://\n"
            "  hooks.slack.com/xxx\n"
            "LANGFUSE_PUBLIC_KEY=pk-lf-xxx\n"
            "LANGFUSE_SECRET_KEY=sk-lf-xxx"
        )

    def get_docker_compose(self) -> str:
        """Docker Compose for combined setup."""
        return (
            "# docker-compose.yml\n"
            "version: '3.8'\n"
            "services:\n"
            "  litellm:\n"
            "    image: ghcr.io/berriai/\n"
            "      litellm:main-stable\n"
            "    ports:\n"
            "      - '4000:4000'\n"
            "    volumes:\n"
            "      - ./config.yaml:/app/\n"
            "        config.yaml\n"
            "    env_file: .env\n"
            "    environment:\n"
            "      - DATABASE_URL=${DATABASE_URL}\n"
            "      - STORE_MODEL_IN_DB=True\n"
            "    depends_on:\n"
            "      - db\n"
            "      - redis\n"
            "    command: --config /app/config.yaml\n"
            "    healthcheck:\n"
            "      test: ['CMD-SHELL',\n"
            "        'python3 -c \"import urllib'\n"
            "        '.request; urllib.request'\n"
            "        '.urlopen(\\'http://'\n"
            "        'localhost:4000/health'\n"
            "        '/liveliness\\')\"']\n"
            "      interval: 30s\n"
            "      timeout: 10s\n"
            "      retries: 3\n"
            "      start_period: 40s\n"
            "\n"
            "  db:\n"
            "    image: postgres:16\n"
            "    restart: always\n"
            "    environment:\n"
            "      - POSTGRES_USER=litellm\n"
            "      - POSTGRES_PASSWORD=pass\n"
            "      - POSTGRES_DB=litellm\n"
            "    volumes:\n"
            "      - pg_data:/var/lib/\n"
            "        postgresql/data\n"
            "    healthcheck:\n"
            "      test: ['CMD-SHELL',\n"
            "        'pg_isready -U litellm']\n"
            "      interval: 1s\n"
            "      timeout: 5s\n"
            "      retries: 10\n"
            "\n"
            "  redis:\n"
            "    image: redis:7\n"
            "    restart: always\n"
            "    volumes:\n"
            "      - redis_data:/data\n"
            "\n"
            "volumes:\n"
            "  pg_data:\n"
            "  redis_data:"
        )

    def get_usage(self) -> str:
        """Usage with OpenAI SDK."""
        return (
            "from openai import OpenAI\n"
            "\n"
            "# Point to LiteLLM proxy\n"
            "client = OpenAI(\n"
            "    base_url='http://localhost:'\n"
            "        '4000/v1',\n"
            "    api_key='sk-1234',\n"
            ")\n"
            "\n"
            "# Call OpenRouter model\n"
            "# through LiteLLM\n"
            "response = client.chat\\\n"
            "    .completions.create(\n"
            "    model='claude',\n"
            "    messages=[\n"
            "        {'role': 'user',\n"
            "         'content': 'Hello!'}\n"
            "    ],\n"
            ")\n"
            "# LiteLLM routes to\n"
            "# OpenRouter → Anthropic\n"
            "# with RBAC, caching,\n"
            "# logging, budget check\n"
            "\n"
            "# Free model via OpenRouter\n"
            "response = client.chat\\\n"
            "    .completions.create(\n"
            "    model='free',\n"
            "    messages=[...],\n"
            ")\n"
            "\n"
            "# Direct provider (bypass OR)\n"
            "response = client.chat\\\n"
            "    .completions.create(\n"
            "    model='direct-gpt4o',\n"
            "    messages=[...],\n"
            ")\n"
            "\n"
            "# Local model\n"
            "response = client.chat\\\n"
            "    .completions.create(\n"
            "    model='local',\n"
            "    messages=[...],\n"
            ")"
        )

    def get_sdk_usage(self) -> str:
        """LiteLLM SDK with OpenRouter."""
        return (
            "import litellm\n"
            "import os\n"
            "\n"
            "os.environ['OPENROUTER_API_KEY'] = \\\n"
            "    'sk-or-v1-xxx'\n"
            "\n"
            "# Call OpenRouter via LiteLLM\n"
            "response = litellm.completion(\n"
            "    model='openrouter/\n"
            "        anthropic/claude-3.5'\n"
            "        '-sonnet',\n"
            "    messages=[\n"
            "        {'role': 'user',\n"
            "         'content': 'Hello!'}\n"
            "    ],\n"
            ")\n"
            "\n"
            "# Pass OpenRouter params\n"
            "response = litellm.completion(\n"
            "    model='openrouter/\n"
            "        openai/gpt-4o',\n"
            "    messages=[...],\n"
            "    transforms=[''],\n"
            "    route='',\n"
            ")\n"
            "\n"
            "# Production env config\n"
            "OPENROUTER_API_KEY = os.getenv(\n"
            "    'OPENROUTER_API_KEY')\n"
            "OPENROUTER_BASE_URL = os.getenv(\n"
            "    'OPENROUTER_API_BASE',\n"
            "    'https://openrouter.ai'\n"
            "    '/api/v1')\n"
            "os.environ[\n"
            "    'OPENROUTER_API_KEY'] = \\\n"
            "    OPENROUTER_API_KEY\n"
            "os.environ[\n"
            "    'OPENROUTER_API_BASE'] = \\\n"
            "    OPENROUTER_BASE_URL"
        )

    def get_fallback_strategy(self) -> dict:
        """Double safety net fallback."""
        return {
            "litellm_fallback": (
                "LiteLLM router tries next model "
                "in fallback list when primary "
                "fails. e.g., claude → gpt-4o → free"
            ),
            "openrouter_failover": (
                "OpenRouter automatically falls "
                "back to next provider within "
                "same model. 30s outage "
                "deprioritization. On by default."
            ),
            "combined": (
                "Double safety net: LiteLLM "
                "model-level fallback + "
                "OpenRouter provider-level "
                "failover. If Claude fails on "
                "OpenRouter, OR tries another "
                "provider. If all OR providers "
                "fail, LiteLLM tries gpt-4o."
            ),
            "config": (
                "router_settings: allowed_fails: 3, "
                "num_retries: 2, request_timeout: 30. "
                "Model fallback: claude: [gpt-4o, free]. "
                "enable_weighted_failover: retry within "
                "same model group first, then cross-group."
            ),
        }

    def get_production_tips(self) -> dict:
        """Production tips."""
        return {
            "redis": (
                "max_connections: 100 in config. "
                "Redis timeout: 30s. "
                "tcp-keepalive: 60s. "
                "Monitor: redis-cli CLIENT LIST. "
                "Move to ElastiCache/Upstash "
                "with cluster mode for scale."
            ),
            "scaling": (
                "Multiple LiteLLM instances behind "
                "Nginx/HAProxy load balancer. "
                "OpenRouter 200 RPM per key — "
                "use multiple keys and distribute."
            ),
            "fallback_delay": (
                "When primary returns 401 (budget "
                "exceeded), LiteLLM waits 10s "
                "(request_timeout) before fallback. "
                "Reduce to 5s, set num_retries: 1 "
                "for faster detection."
            ),
            "monitoring": (
                "Langfuse for LLM observability. "
                "New Relic for infrastructure. "
                "Slack for budget alerts. "
                "LiteLLM admin dashboard at /ui."
            ),
        }

LiteLLM with OpenRouter Checklist

[ ] LiteLLM can use OpenRouter as upstream provider — not mutually exclusive
[ ] LiteLLM local RBAC and logging while OpenRouter handles multi-provider failover and model breadth
[ ] LiteLLM supports all OpenRouter models: text, chat, vision, embedding
[ ] Model format: openrouter/provider/model (e.g., openrouter/anthropic/claude-3.5-sonnet)
[ ] api_base: https://openrouter.ai/api/v1 (default, configurable via env)
[ ] Set OPENROUTER_API_KEY environment variable
[ ] config.yaml: model_list with openrouter/ prefix models
[ ] config.yaml: router_settings with routing_strategy, allowed_fails, num_retries, request_timeout
[ ] config.yaml: litellm_settings with cache: true, cache_params (redis, max_connections, ttl)
[ ] config.yaml: general_settings with database_url, master_key, alerting
[ ] config.yaml: litellm_pre_call_actions with check_key_budget: true
[ ] config.yaml: proxy_logging with langfuse, newrelic callbacks
[ ] Environment: OPENROUTER_API_KEY, OPENROUTER_API_BASE, DATABASE_URL, REDIS_URL, LITELLM_MASTER_KEY
[ ] Docker Compose: litellm + PostgreSQL 16 + Redis 7 as three services
[ ] Docker healthcheck: LiteLLM (/health/liveliness), PostgreSQL (pg_isready)
[ ] Docker: config.yaml mounted, env_file for secrets, STORE_MODEL_IN_DB=True
[ ] LiteLLM benefits: virtual keys with budgets, per-team RBAC, SSO/SAML (Enterprise)
[ ] LiteLLM benefits: Redis caching with TTL 300s, p95 193ms on cache hits
[ ] LiteLLM benefits: Langfuse logging, spend tracking in PostgreSQL, Slack alerts
[ ] LiteLLM benefits: budget enforcement (check_key_budget), admin dashboard at /ui
[ ] OpenRouter benefits: 500+ models, 70+ providers through one endpoint
[ ] OpenRouter benefits: automatic provider failover, 30s outage deprioritization
[ ] OpenRouter benefits: free models (DeepSeek R1, Llama 3.3, Gemma 3)
[ ] OpenRouter benefits: BYOK with 5% fee, prioritized/fallback keys
[ ] Combined: double safety net — LiteLLM model fallback + OpenRouter provider failover
[ ] Combined: critical models direct (bypass OpenRouter), rest via OpenRouter
[ ] Combined: hybrid approach for data residency — sensitive data direct, rest via OR
[ ] Fallback config: claude: [gpt-4o, free], gpt-4o: [free]
[ ] Router: routing_strategy latency-based, allowed_fails: 3, num_retries: 2, request_timeout: 30
[ ] enable_weighted_failover: retry within same model group first, then cross-group fallbacks
[ ] max_fallbacks: 5 (default cap), cooldowns apply for failed deployments
[ ] Redis: max_connections: 100, timeout: 30s, tcp-keepalive: 60s
[ ] Redis: monitor with redis-cli CLIENT LIST, move to ElastiCache/Upstash for scale
[ ] Scaling: multiple LiteLLM instances behind Nginx/HAProxy load balancer
[ ] OpenRouter limit: 200 RPM per key — use multiple keys and distribute
[ ] Fallback delay: reduce request_timeout to 5s, num_retries: 1 for faster detection
[ ] Pass OpenRouter params: transforms, models, route as arguments to litellm.completion()
[ ] Production env config: OPENROUTER_API_KEY and OPENROUTER_API_BASE from env vars
[ ] Switching direction: base URL and key change (both speak OpenAI format)
[ ] Both gateways behind one OpenAI SDK client — base_url to LiteLLM proxy
[ ] Read LLM gateway guide for gateway concepts
[ ] Read OpenRouter vs LiteLLM for comparison
[ ] Read LiteLLM tutorial for proxy setup
[ ] Read OpenRouter tutorial for managed setup
[ ] Test: LiteLLM routes to OpenRouter model correctly
[ ] Test: fallback switches from OpenRouter model to backup on failure
[ ] Test: Redis caching returns cached response on repeated requests
[ ] Test: budget enforcement blocks over-spend via virtual keys
[ ] Test: Langfuse logs token usage and cost for OpenRouter models
[ ] Test: direct provider bypass works for critical models
[ ] Test: Docker Compose starts all three services with healthchecks
[ ] Document config.yaml, env vars, fallback strategy, and routing rules

FAQ

Can you use LiteLLM with OpenRouter together?

Yes, LiteLLM can use OpenRouter as an upstream provider, combining local control with managed breadth. OpenRouter Blog: "The two are not mutually exclusive. LiteLLM can use OpenRouter as an upstream provider, so you get LiteLLM local RBAC and logging while OpenRouter handles multi-provider failover and model breadth." LiteLLM Docs: "LiteLLM supports all text/chat/vision/embedding models from OpenRouter. Send model=openrouter/ to send it to OpenRouter. See all OpenRouter models." OSINT Team: "LiteLLM provides a single OpenAI-compatible API endpoint that routes requests to different LLMs. Your application talks to LiteLLM, LiteLLM talks to AI providers, handles routing, authentication, logging, fallback, budgets, and observability."

How do you configure LiteLLM to use OpenRouter as upstream?

Add OpenRouter models to config.yaml with openrouter/ prefix and OpenRouter API key. OpenRouter Blog: "model_list with model_name: or-claude, litellm_params: model: openrouter/anthropic/claude-opus-4.6, api_key: your-openrouter-key, api_base: https://openrouter.ai/api/v1." LiteLLM Docs: "Set OPENROUTER_API_KEY env var. Use model=openrouter/provider/model format. api_base defaults to https://openrouter.ai/api/v1. Pass transforms, models, route as arguments." OSINT Team: "model_name and model keys have openrouter prepended. OPENROUTER_API_BASE=https://openrouter.ai/api/v1. OPENROUTER_API_KEY=sk-or-v1-xxx." Config: model_list with openrouter/ prefix, api_key from env, api_base to openrouter.ai/api/v1, router_settings for routing and fallback.

What are the benefits of combining LiteLLM and OpenRouter?

You get LiteLLM local control (RBAC, budgets, caching, logging) plus OpenRouter managed breadth (500+ models, automatic failover). OpenRouter Blog: "LiteLLM local RBAC and logging while OpenRouter handles multi-provider failover and model breadth. Switching direction is a base URL and key change, since both speak OpenAI format." Markaicode: "LiteLLM is a lightweight proxy with caching, key management, and OpenAI-compatible endpoints. Single config.yaml controls routing, cost tracking, and access without touching code. OpenRouter as primary provider with fallback to GPT-4o if Claude over quota." Benefits: (1) LiteLLM: virtual keys, per-team budgets, RBAC, SSO. (2) LiteLLM: Redis caching, p95 193ms on cache hits. (3) LiteLLM: Langfuse logging, observability. (4) OpenRouter: 500+ models, 70+ providers. (5) OpenRouter: automatic provider failover, 30s outage deprioritization. (6) OpenRouter: free models, BYOK with 5% fee. (7) Both: OpenAI-compatible API.

How do you set up Docker Compose for LiteLLM with OpenRouter?

Use Docker Compose with LiteLLM proxy, PostgreSQL, and Redis, configuring OpenRouter as upstream. OSINT Team: "Docker Compose with litellm image, config.yaml mounted, DATABASE_URL for PostgreSQL, env_file for OpenRouter keys. Healthcheck: python3 -c urllib.request.urlopen health/liveliness. PostgreSQL 16 with healthcheck pg_isready." Markaicode: "Redis: host, port, password, max_connections: 100, response_cache_ttl: 300. litellm_pre_call_actions: check_key_budget: true. proxy_logging: callback langfuse, newrelic." Docker Compose: (1) LiteLLM container with config.yaml. (2) PostgreSQL for spend tracking. (3) Redis for caching and rate limits. (4) Environment: OPENROUTER_API_KEY, DATABASE_URL, REDIS_URL, LITELLM_MASTER_KEY. (5) Healthcheck on LiteLLM and PostgreSQL.

How do you handle fallbacks in LiteLLM with OpenRouter?

Configure fallbacks in config.yaml with LiteLLM router_settings and OpenRouter model fallbacks. Markaicode: "In LiteLLM config, set router_settings: max_retries: 3 with exponential backoff, allowed_fails: 5. Model fallback: openrouter/anthropic/claude-3.5-sonnet with fallback: [openrouter/openai/gpt-4o]. routing_strategy: latency-based. request_timeout: 30. num_retries: 2." LiteLLM Docs: "When a deployment in a model group fails, router moves to next entry in fallbacks. enable_weighted_failover: router first retries inside same model group, then escalates to cross-group fallbacks. Capped by max_fallbacks (default 5). Cooldowns apply: deployment crossing allowed_fails is cooled down." Fallback config: (1) Primary model with fallback list. (2) router_settings: routing_strategy, allowed_fails, num_retries, request_timeout. (3) OpenRouter also has automatic provider failover within model. (4) Double safety net: LiteLLM model fallback + OpenRouter provider failover.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →