LiteLLM Tutorial: Setup Self-Hosted LLM Proxy with Docker and Config

By Fadi Updated Sep 2, 2025 11 min read LLM Gateways

TL;DR — LiteLLM tutorial: self-hosted LLM proxy with Docker. LiteLLM Docs: "Control proxy with config.yaml. model_list with model_name and litellm_params. Start with litellm --config config.yaml." GitHub: "Python SDK, Proxy Server (AI Gateway). Router with retry/fallback. Application-level load balancing and cost tracking. Observability callbacks." LiteLLM Docs: "docker run with DATABASE_URL, config bucket, port 4000." LiteLLM Docs: "Redis Cluster, Sentinel, GCP IAM auth. Cache with TTL." OpenRouter Blog: "Six routing modes, fallback lists, virtual keys, per-team budgets, pluggable observability." Markaicode: "Redis caching p95 193ms, 72% improvement." Learn more with LLM gateway, OpenRouter vs LiteLLM, OpenRouter tutorial, and switch provider.

LiteLLM Docs provides the overview: "Python SDK, Proxy Server (AI Gateway). Direct Python library integration in your codebase, Router with retry/fallback logic across multiple deployments, application-level load balancing and cost tracking, exception handling with OpenAI-compatible errors, observability callbacks."

GitHub describes the project: "Python SDK, Proxy Server (AI Gateway). Router with retry/fallback logic across multiple deployments (e.g. Azure/OpenAI). Application-level load balancing and cost tracking. Exception handling with OpenAI-compatible errors. Observability callbacks (Lunary, MLflow, Langfuse, etc.)."

LiteLLM Proxy Architecture

flowchart TD subgraph App["Your Application"] SDK["OpenAI SDK
base_url: localhost:4000
api_key: virtual key"] end subgraph Proxy["LiteLLM Proxy Server (Docker)"] Config["config.yaml
model_list
router_settings
litellm_settings"] Router["Router
6 routing modes
weighted/latency/rate-limit
least-busy/lowest-cost/custom"] Fallback["Fallback
try next model
on failure"] Cache["Cache
Redis-backed
TTL: 3600s
p95: 193ms"] Auth["Virtual Keys
per-key budgets
per-team budgets
rate limits (rpm/tpm)"] Spend["Spend Tracking
PostgreSQL
per-request costs
admin dashboard"] end subgraph Infra["Infrastructure"] Postgres["PostgreSQL
spend data
virtual keys
audit logs"] Redis["Redis
caching
rate limits
session state"] Docker["Docker Container
litellm proxy
port 4000"] end subgraph Providers["LLM Providers (100+)"] OpenAI["OpenAI
gpt-4o, o1, o3"] Anthropic["Anthropic
claude-3.5-sonnet
claude-4-opus"] Google["Google
gemini-2.0-flash
gemini-pro"] Azure["Azure OpenAI
gpt-4o, gpt-4-turbo"] Local["Local Models
Ollama, vLLM, TGI"] end SDK --> Config Config --> Router Router --> Fallback Fallback --> Cache Cache --> Auth Auth --> Spend Docker --> Postgres Docker --> Redis Docker --> Proxy Spend --> OpenAI Spend --> Anthropic Spend --> Google Spend --> Azure Spend --> Local

LiteLLM Config Structure

Section	Purpose	Key Fields
model_list	Define available models	model_name, litellm_params (model, api_key, api_base)
router_settings	Routing and fallback	routing_strategy, fallbacks
litellm_settings	Cache and general settings	cache, cache_params (type, ttl)
general_settings	Database and auth	database_url, master_key, alerting

Routing Modes

Mode	Description	Best For
simple-shuffle	Random pick from available	Default, simple setups
latency-based-routing	Pick lowest latency	Performance-critical apps
rate-limit-aware-routing	Avoid rate-limited providers	High-volume usage
least-busy-routing	Pick least busy provider	Load distribution
lowest-cost-routing	Pick cheapest model	Cost optimization
custom-router	Custom Python logic	Complex routing rules

Implementation

from dataclasses import dataclass
from typing import Optional
from enum import Enum

class RoutingMode(Enum):
    SIMPLE_SHUFFLE = "simple-shuffle"
    LATENCY_BASED = "latency-based-routing"
    RATE_LIMIT_AWARE = "rate-limit-aware-routing"
    LEAST_BUSY = "least-busy-routing"
    LOWEST_COST = "lowest-cost-routing"
    CUSTOM = "custom-router"

@dataclass
class LiteLLMTutorialGuide:
    """LiteLLM tutorial implementation guide."""

    def get_install(self) -> str:
        """Installation instructions."""
        return (
            "# Option 1: pip install\n"
            "pip install 'litellm[proxy]'\n"
            "\n"
            "# Option 2: Docker\n"
            "docker run --name litellm-proxy \\\n"
            "  -p 4000:4000 \\\n"
            "  -e OPENAI_API_KEY=$OPENAI_API_KEY \\\n"
            "  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \\\n"
            "  -v $(pwd)/config.yaml:/app/config.yaml \\\n"
            "  ghcr.io/berriai/litellm:main-latest \\\n"
            "  --config /app/config.yaml\n"
            "\n"
            "# Option 3: Docker Compose (recommended)\n"
            "# See docker-compose.yml below"
        )

    def get_config_yaml(self) -> str:
        """Basic config.yaml."""
        return (
            "# config.yaml\n"
            "model_list:\n"
            "  - model_name: gpt-4o\n"
            "    litellm_params:\n"
            "      model: openai/gpt-4o\n"
            "      api_key: os.environ/OPENAI_API_KEY\n"
            "  - model_name: gpt-4o\n"
            "    litellm_params:\n"
            "      model: anthropic/claude-3-5-sonnet\n"
            "      api_key: os.environ/ANTHROPIC_API_KEY\n"
            "  - model_name: cheap\n"
            "    litellm_params:\n"
            "      model: google/gemini-2.0-flash\n"
            "      api_key: os.environ/GOOGLE_API_KEY\n"
            "  - model_name: local\n"
            "    litellm_params:\n"
            "      model: ollama/llama3\n"
            "      api_base: http://localhost:11434\n"
            "\n"
            "router_settings:\n"
            "  routing_strategy: simple-shuffle\n"
            "  fallbacks:\n"
            "    - gpt-4o: [cheap]\n"
            "\n"
            "litellm_settings:\n"
            "  cache: true\n"
            "  cache_params:\n"
            "    type: redis\n"
            "    host: redis\n"
            "    port: 6379\n"
            "    ttl: 3600\n"
            "\n"
            "general_settings:\n"
            "  database_url: os.environ/DATABASE_URL\n"
            "  master_key: sk-1234\n"
            "  alerting: ['slack']"
        )

    def get_docker_compose(self) -> str:
        """Docker Compose for production."""
        return (
            "# docker-compose.yml\n"
            "version: '3.8'\n"
            "services:\n"
            "  litellm:\n"
            "    image: ghcr.io/berriai/\n"
            "      litellm:main-latest\n"
            "    ports:\n"
            "      - '4000:4000'\n"
            "    volumes:\n"
            "      - ./config.yaml:/app/\n"
            "        config.yaml\n"
            "    environment:\n"
            "      - OPENAI_API_KEY=${OPENAI_API_KEY}\n"
            "      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}\n"
            "      - GOOGLE_API_KEY=${GOOGLE_API_KEY}\n"
            "      - DATABASE_URL=postgresql://\n"
            "        litellm:litellm@db:5432/litellm\n"
            "      - REDIS_URL=redis://redis:6379\n"
            "    depends_on:\n"
            "      - db\n"
            "      - redis\n"
            "    command: --config /app/config.yaml\n"
            "\n"
            "  db:\n"
            "    image: postgres:16\n"
            "    environment:\n"
            "      - POSTGRES_USER=litellm\n"
            "      - POSTGRES_PASSWORD=litellm\n"
            "      - POSTGRES_DB=litellm\n"
            "    volumes:\n"
            "      - litellm_db:/var/lib/\n"
            "        postgresql/data\n"
            "\n"
            "  redis:\n"
            "    image: redis:7\n"
            "    volumes:\n"
            "      - litellm_redis:/data\n"
            "\n"
            "volumes:\n"
            "  litellm_db:\n"
            "  litellm_redis:"
        )

    def get_usage_code(self) -> str:
        """Usage with OpenAI SDK."""
        return (
            "from openai import OpenAI\n"
            "\n"
            "# Point to LiteLLM proxy\n"
            "client = OpenAI(\n"
            "    base_url='http://localhost:4000/v1',\n"
            "    api_key='sk-1234',  # master key\n"
            ")\n"
            "\n"
            "# Call any model in config\n"
            "response = client.chat.completions\\\n"
            "    .create(\n"
            "    model='gpt-4o',  # routes to OpenAI\n"
            "    messages=[\n"
            "        {'role': 'user',\n"
            "         'content': 'Hello!'}\n"
            "    ],\n"
            ")\n"
            "print(response.choices[0]\n"
            "      .message.content)\n"
            "\n"
            "# Use cheaper model\n"
            "response = client.chat.completions\\\n"
            "    .create(\n"
            "    model='cheap',  # routes to Gemini\n"
            "    messages=[\n"
            "        {'role': 'user',\n"
            "         'content': 'Summarize this'}\n"
            "    ],\n"
            ")"
        )

    def get_virtual_keys(self) -> str:
        """Virtual key creation."""
        return (
            "import requests\n"
            "\n"
            "# Create virtual key with budget\n"
            "response = requests.post(\n"
            "    'http://localhost:4000'\n"
            "    '/key/generate',\n"
            "    headers={\n"
            "        'Authorization': 'Bearer sk-1234'\n"
            "    },\n"
            "    json={\n"
            "        'key_alias': 'team-api-key',\n"
            "        'max_budget': 100.0,  # $100 USD\n"
            "        'budget_duration': 'monthly',\n"
            "        'rpm_limit': 100,  # 100 req/min\n"
            "        'tpm_limit': 100000,  # 100K tok/min\n"
            "        'models': ['gpt-4o', 'cheap'],\n"
            "        'team_id': 'team-123',\n"
            "    }\n"
            ")\n"
            "key = response.json()['key']\n"
            "print(f'Virtual key: {key}')\n"
            "\n"
            "# Use virtual key\n"
            "client = OpenAI(\n"
            "    base_url='http://localhost:4000/v1',\n"
            "    api_key=key,  # virtual key\n"
            ")"
        )

    def get_fallback_config(self) -> str:
        """Fallback configuration."""
        return (
            "# config.yaml - advanced fallback\n"
            "model_list:\n"
            "  - model_name: gpt-4o\n"
            "    litellm_params:\n"
            "      model: openai/gpt-4o\n"
            "      api_key: os.environ/OPENAI_API_KEY\n"
            "  - model_name: gpt-4o\n"
            "    litellm_params:\n"
            "      model: azure/gpt-4o\n"
            "      api_base: os.environ/AZURE_API_BASE\n"
            "      api_key: os.environ/AZURE_API_KEY\n"
            "  - model_name: backup\n"
            "    litellm_params:\n"
            "      model: anthropic/claude-3.5-sonnet\n"
            "      api_key: os.environ/ANTHROPIC_API_KEY\n"
            "  - model_name: cheap-backup\n"
            "    litellm_params:\n"
            "      model: google/gemini-2.0-flash\n"
            "      api_key: os.environ/GOOGLE_API_KEY\n"
            "\n"
            "router_settings:\n"
            "  routing_strategy: lowest-cost-routing\n"
            "  fallbacks:\n"
            "    - gpt-4o: [backup, cheap-backup]\n"
            "    - backup: [cheap-backup]\n"
            "  num_retries: 2\n"
            "  timeout: 30\n"
            "  retry_after: 5"
        )

    def get_observability(self) -> dict:
        """Observability configuration."""
        return {
            "callbacks": [
                "Langfuse — open-source LLM observability",
                "Helicone — request logging and analytics",
                "MLflow — experiment tracking",
                "Lunary — LLM monitoring",
                "OpenTelemetry — distributed tracing",
                "Prometheus — metrics (Enterprise)",
            ],
            "config": (
                "# config.yaml\n"
                "litellm_settings:\n"
                "  success_callback: ['langfuse']\n"
                "  failure_callback: ['langfuse']\n"
                "  langfuse_public_key: os.environ/\n"
                "    LANGFUSE_PUBLIC_KEY\n"
                "  langfuse_secret_key: os.environ/\n"
                "    LANGFUSE_SECRET_KEY"
            ),
            "admin_dashboard": (
                "Access at http://localhost:4000/ui\n"
                "View: spend per key, per team, per model\n"
                "Track: requests, tokens, costs, errors\n"
                "Manage: virtual keys, teams, budgets"
            ),
        }

    def get_production_config(self) -> dict:
        """Production configuration."""
        return {
            "environment_variables": [
                "DATABASE_URL — PostgreSQL for spend tracking",
                "REDIS_URL — Redis for cache and rate limits",
                "OPENAI_API_KEY — OpenAI API key",
                "ANTHROPIC_API_KEY — Anthropic API key",
                "GOOGLE_API_KEY — Google API key",
                "LITELLM_MASTER_KEY — master key for admin",
                "SLACK_WEBHOOK_URL — for alerts",
            ],
            "docker_production": (
                "docker run -d --name litellm \\\n"
                "  -p 4000:4000 \\\n"
                "  -e DATABASE_URL=$DATABASE_URL \\\n"
                "  -e REDIS_URL=$REDIS_URL \\\n"
                "  -e OPENAI_API_KEY=$OPENAI_API_KEY \\\n"
                "  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \\\n"
                "  -v config.yaml:/app/config.yaml \\\n"
                "  --restart unless-stopped \\\n"
                "  ghcr.io/berriai/litellm:main-latest \\\n"
                "  --config /app/config.yaml"
            ),
            "health_check": (
                "curl http://localhost:4000/health\n"
                "curl http://localhost:4000/v1/models\n"
                "curl http://localhost:4000/key/info \\\n"
                "  -H 'Authorization: Bearer sk-1234'"
            ),
        }

LiteLLM Tutorial Checklist

[ ] Install LiteLLM: pip install 'litellm[proxy]' or Docker image
[ ] Create config.yaml with model_list section
[ ] model_list: model_name (alias) + litellm_params (model, api_key, api_base)
[ ] Set environment variables for API keys (OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.)
[ ] Start proxy: litellm --config config.yaml or docker run
[ ] Default endpoint: http://localhost:4000/v1
[ ] Use OpenAI SDK with base_url pointing to localhost:4000
[ ] Docker Compose: litellm + PostgreSQL + Redis as three services
[ ] PostgreSQL: stores spend data, virtual keys, audit logs
[ ] Redis: caching with TTL, rate limits, session state
[ ] DATABASE_URL env var for PostgreSQL connection
[ ] REDIS_URL env var for Redis connection
[ ] router_settings: routing_strategy (simple-shuffle, latency-based, rate-limit-aware, least-busy, lowest-cost, custom)
[ ] router_settings: fallbacks (list of model mappings, e.g., gpt-4o: [backup, cheap-backup])
[ ] router_settings: num_retries, timeout, retry_after
[ ] 6 routing modes: simple-shuffle, latency-based, rate-limit-aware, least-busy, lowest-cost, custom Python
[ ] Fallback chains: try next model when primary fails
[ ] litellm_settings: cache: true with cache_params (type: redis, host, port, ttl)
[ ] Redis-backed caching: TTL 3600s (1 hour default)
[ ] Cache hits: p95 latency 193ms, 72% improvement over external provider
[ ] Cache works for chat/completions and embeddings
[ ] general_settings: database_url, master_key, alerting
[ ] Virtual keys: create via /key/generate endpoint
[ ] Virtual keys: max_budget (dollar limit), budget_duration (monthly)
[ ] Virtual keys: rpm_limit (requests per minute), tpm_limit (tokens per minute)
[ ] Virtual keys: models list to restrict access
[ ] Virtual keys: team_id for per-team budgets
[ ] Budget enforcement: requests blocked when budget exceeded
[ ] Admin dashboard: http://localhost:4000/ui
[ ] Admin dashboard: view spend per key, per team, per model
[ ] Admin dashboard: track requests, tokens, costs, errors
[ ] Admin dashboard: manage virtual keys, teams, budgets
[ ] Observability callbacks: Langfuse, Helicone, MLflow, Lunary, OpenTelemetry
[ ] success_callback and failure_callback in litellm_settings
[ ] Prometheus metrics (Enterprise)
[ ] Local model support: Ollama, vLLM, TGI via api_base
[ ] OpenAI-compatible API — any OpenAI SDK client works
[ ] Exception handling: LiteLLM maps all provider exceptions to OpenAI exceptions
[ ] Health check: curl http://localhost:4000/health
[ ] List models: curl http://localhost:4000/v1/models
[ ] Docker production: --restart unless-stopped, env vars, volume mount
[ ] GCS config bucket: LITELLM_CONFIG_BUCKET_NAME for remote config
[ ] Redis Cluster: redis_startup_nodes for high availability
[ ] Redis Sentinel: service_name and sentinel_nodes for failover
[ ] GCP IAM auth for Redis: gcp_service_account
[ ] Alerting: Slack webhook for budget alerts and errors
[ ] Read LLM gateway guide for gateway concepts
[ ] Read OpenRouter vs LiteLLM for comparison
[ ] Read OpenRouter tutorial for managed alternative
[ ] Read switch LLM provider for provider migration
[ ] Test: proxy starts and responds on port 4000
[ ] Test: model routing works for all configured models
[ ] Test: fallback switches to backup on primary failure
[ ] Test: caching returns cached response on repeated requests
[ ] Test: virtual key budget enforcement blocks over-spend
[ ] Test: admin dashboard shows spend and usage
[ ] Test: Docker Compose starts all three services correctly
[ ] Document config.yaml, Docker setup, virtual keys, and routing strategy

FAQ

How do you install and set up LiteLLM proxy?

Install LiteLLM with pip or Docker, create config.yaml with model_list, and start the proxy on port 4000. LiteLLM Docs: "Control LiteLLM Proxy with a config.yaml file. Create one with your model: model_list with model_name and litellm_params containing model, api_base, api_key. Start with litellm --config config.yaml." GitHub: "Python SDK, Proxy Server (AI Gateway). Direct Python library integration or proxy server. Router with retry/fallback logic across multiple deployments." Steps: (1) pip install 'litellm[proxy]' or docker run. (2) Create config.yaml with model_list. (3) Set environment variables for API keys. (4) Start proxy: litellm --config config.yaml. (5) Endpoint: http://localhost:4000/v1. (6) Use OpenAI SDK pointing to localhost:4000.

How do you configure LiteLLM routing and fallbacks?

Use router_settings in config.yaml with routing_strategy and fallbacks. LiteLLM supports 6 routing modes and configurable fallback chains. LiteLLM Docs: "Retry/fallback logic across multiple deployments. Router with application-level load balancing and cost tracking." OpenRouter Blog: "LiteLLM ships six routing modes: weighted pick, latency-based, rate-limit-aware, least-busy, lowest-cost, and custom Python. Fallback lists let the proxy try the next model when one fails." Config: router_settings with routing_strategy (simple-shuffle, latency-based-routing, rate-limit-aware-routing, least-busy-routing, lowest-cost-routing, custom-router). Fallbacks: list of model mappings (e.g., gpt-4o: [claude-3-5-sonnet, gemini-2.0-flash]). When primary fails, proxy tries fallbacks in order.

How do you set up LiteLLM with Docker and PostgreSQL?

Run LiteLLM proxy as Docker container with PostgreSQL for spend tracking and Redis for caching. LiteLLM Docs: "docker run --name litellm-proxy -e DATABASE_URL= -e LITELLM_CONFIG_BUCKET_NAME= -p 4000:4000." LiteLLM Docs: "Optional Redis Cluster Settings with redis_startup_nodes. Optional Redis Sentinel Settings with service_name and sentinel_nodes. GCP IAM Authentication for Redis with gcp_service_account." Docker setup: (1) PostgreSQL container for DATABASE_URL. (2) Redis container for caching and rate limits. (3) LiteLLM proxy container with config.yaml mounted. (4) docker-compose.yml with all three services. (5) DATABASE_URL env var for spend tracking. (6) REDIS_URL env var for cache and rate limits.

How do you use virtual keys and budget tracking in LiteLLM?

Create virtual keys with per-key, per-team, and per-user budgets via the LiteLLM admin dashboard or API. LiteLLM Docs: "Track spend and set budgets per project. LiteLLM Proxy Server with virtual keys and spend tracking." OpenRouter Blog: "LiteLLM supports virtual keys, per-team budget enforcement, RBAC, SSO, and pluggable observability." Virtual keys: (1) Create master key in config.yaml. (2) Generate virtual keys via /key/generate endpoint. (3) Set budget per key: max_budget parameter. (4) Set rate limits per key: rpm (requests per minute), tpm (tokens per minute). (5) Track spend in PostgreSQL. (6) View spend in admin dashboard at /ui. (7) Budget enforcement: requests blocked when budget exceeded. (8) Per-team budgets: assign keys to teams with team budgets.

How do you enable caching in LiteLLM?

Enable Redis-backed caching in config.yaml with cache settings and TTL. LiteLLM Docs: "Redis settings in litellm_settings with cache: true, cache_params with type: redis, ttl: 3600." Markaicode: "LiteLLM built-in caching and load balancing deliver p95 latencies below 200ms on warm cache hits. 72% improvement over waiting for external provider." Caching setup: (1) litellm_settings: cache: true. (2) cache_params: type: redis, host, port, password. (3) ttl: 3600 (1 hour default). (4) Set REDIS_URL env var. (5) Cache hits return instantly without calling provider. (6) Cache works for chat/completions and embeddings. (7) p95 latency 193ms on warm cache hits. (8) 72% improvement over external provider calls.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →