Provider-Agnostic AI with LiteLLM: One API for 100+ LLMs

TL;DR — LiteLLM is an open-source AI gateway providing a unified, OpenAI-compatible API for 100+ LLM providers. Used as a Python SDK or deployed as a Proxy Server. Key features: unified API (no provider-specific SDKs), drop-in OpenAI compatibility (swap providers without code changes), multi-model routing (simple-shuffle, latency-based, priority-based), automatic fallbacks (if one provider fails, retry on next), cost tracking per key/user/team/org, virtual keys for access control, rate limiting, guardrails, and admin dashboard. 8ms P95 latency at 1,000 RPS. Used by Netflix, Lemonade, 240M+ Docker pulls. Prevents vendor lock-in by abstracting the provider layer — switching from OpenAI to Anthropic to self-hosted Ollama is a config change, not a code rewrite. Enables cost optimization: route 80% of queries to self-hosted Llama 3 ($0.11/M tokens), 15% to mid-tier APIs, 5% to frontier models. Deploy as part of Docker Compose AI platform for complete self-hosted stack.

LiteLLM solves the fundamental problem of multi-provider AI: each LLM provider has different SDKs, auth patterns, request formats, and response schemas. Managing this across OpenAI, Anthropic, Azure, Bedrock, and self-hosted models creates fragile, provider-specific code that's expensive to maintain and impossible to switch.

LiteLLM provides a single OpenAI-compatible interface. Your application code never changes. Switching providers is a configuration change. This is the Terraform of the AI era — a neutral abstraction one layer up that gives you the right to switch and the ability to mix providers in a single deployment.

LiteLLM Architecture

Proxy Server vs Python SDK

Quick Start: Proxy Server

# litellm_config.yaml
model_list:
  # Frontier models (5% of traffic)
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
    model_info:
      mode: chat

  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet
      api_key: os.environ/ANTHROPIC_API_KEY

  # Self-hosted (80% of traffic)
  - model_name: llama-70b
    litellm_params:
      model: ollama/llama3:70b
      api_base: http://ollama:11434

  # Mid-tier (15% of traffic)
  - model_name: mistral
    litellm_params:
      model: ollama/mistral:latest
      api_base: http://ollama:11434

# Routing: fallback from frontier to self-hosted
router_settings:
  routing_strategy: simple-shuffle
  fallbacks:
    - "gpt-4o": ["claude-sonnet", "llama-70b"]
    - "claude-sonnet": ["gpt-4o", "llama-70b"]

# Cost tracking
general_settings:
  master_key: sk-litellm-master-key
  database_url: postgresql://postgres:password@postgres/litellm
  enable_spend_tracking: true

# Start LiteLLM proxy
docker run -p 4000:4000 \
  -v ./litellm_config.yaml:/app/config.yaml \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  ghcr.io/berriai/litellm:main-stable \
  --config /app/config.yaml

# Your application code — never changes regardless of provider
import openai

client = openai.OpenAI(
    api_key="sk-litellm-virtual-key",
    base_url="http://localhost:4000"
)

# Call any model through the gateway
response = client.chat.completions.create(
    model="gpt-4o",  # or "claude-sonnet" or "llama-70b"
    messages=[{"role": "user", "content": "Explain RAG in 3 sentences"}]
)

Quick Start: Python SDK

import litellm

# Call any provider with the same interface
response = litellm.completion(
    model="ollama/llama3:70b",  # Self-hosted
    messages=[{"role": "user", "content": "What is RAG?"}],
    api_base="http://ollama:11434",
)

# Switch provider — only the model string changes
response = litellm.completion(
    model="anthropic/claude-3-5-sonnet",  # Anthropic
    messages=[{"role": "user", "content": "What is RAG?"}],
    api_key=os.environ["ANTHROPIC_API_KEY"],
)

# Same response format regardless of provider
print(response.choices[0].message.content)

Multi-Model Routing with Fallbacks

from litellm import Router

router = Router(model_list=[
    # Primary: self-hosted (cheapest)
    {
        "model_name": "default",
        "litellm_params": {
            "model": "ollama/llama3:70b",
            "api_base": "http://ollama:11434",
        },
    },
    # Fallback 1: Anthropic
    {
        "model_name": "fallback-anthropic",
        "litellm_params": {
            "model": "anthropic/claude-3-5-sonnet",
            "api_key": os.environ["ANTHROPIC_API_KEY"],
        },
    },
    # Fallback 2: OpenAI
    {
        "model_name": "fallback-openai",
        "litellm_params": {
            "model": "openai/gpt-4o",
            "api_key": os.environ["OPENAI_API_KEY"],
        },
    },
],
fallbacks=[{"default": ["fallback-anthropic", "fallback-openai"]}],
)

# If Ollama fails, automatically retries on Anthropic, then OpenAI
response = router.completion(
    model="default",
    messages=[{"role": "user", "content": "Summarize this document"}],
)

Cost Tracking

Supported Providers (100+)

LiteLLM Deployment Checklist

• [ ] Install LiteLLM: pip install litellm (SDK) or docker pull ghcr.io/berriai/litellm:main-stable (Proxy)
• [ ] Create litellm_config.yaml with model list and routing strategy
• [ ] Configure providers: OpenAI, Anthropic, and self-hosted Ollama
• [ ] Set up fallbacks: if primary fails, retry on next provider
• [ ] Generate virtual keys for each team or application
• [ ] Set per-team budgets and rate limits (RPM/TPM)
• [ ] Enable spend tracking with PostgreSQL database
• [ ] Configure cost optimization routing: 80% self-hosted, 15% mid-tier, 5% frontier
• [ ] Test fallback: kill Ollama, verify traffic fails over to Anthropic
• [ ] Use OpenAI-compatible client code (no provider-specific SDKs)
• [ ] Deploy as part of Docker Compose AI platform
• [ ] Configure guardrails: PII masking, content moderation
• [ ] Set up Prometheus metrics for latency, error rate, cost
• [ ] Enable logging to Langfuse or LangSmith for observability
• [ ] Test provider switching: change config, verify no code changes needed
• [ ] Monitor spend dashboard: per-key, per-team, per-model costs
• [ ] Set spend alerts: email when team approaches budget
• [ ] Use semantic answer caching with LiteLLM cache
• [ ] Combine with self-hosted AI for data sovereignty
• [ ] Review vendor lock-in mitigation strategy
• [ ] Evaluate TCO with multi-model routing
• [ ] Consider enterprise features: SSO, audit logs, custom SLAs
• [ ] Document routing rules for the team
• [ ] Set up admin dashboard for platform team visibility

FAQ

What is LiteLLM?

LiteLLM is an open-source AI gateway that provides a single, unified, OpenAI-compatible interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, Ollama, vLLM, and more. You use it as a Python SDK for direct integration or deploy it as a Proxy Server (AI Gateway) for centralized team access. Key features: unified API (no provider-specific SDKs), drop-in OpenAI compatibility (swap providers without rewriting code), cost tracking per key/user/team, load balancing with fallbacks, virtual keys for access control, and an admin dashboard. 8ms P95 latency at 1,000 RPS. Used by Netflix, Lemonade, and 240M+ Docker pulls.

How does LiteLLM prevent vendor lock-in?

LiteLLM prevents vendor lock-in by abstracting the provider layer. Your application code uses the OpenAI format — LiteLLM translates requests to each provider's native format and normalizes responses back. Switching from OpenAI to Anthropic to self-hosted Ollama is a configuration change in LiteLLM's config file, not a code rewrite. This addresses Layer 1 (API dependency) of vendor lock-in. Combined with open-source infrastructure (pgvector, FalkorDB) and portable data formats, LiteLLM makes provider switching a routing configuration, not a migration project.

How does LiteLLM multi-model routing work?

LiteLLM's Router supports multiple routing strategies: simple-shuffle (weighted random, default), latency-based routing (route to lowest-latency deployment), usage-based routing (route by RPM/TPM capacity), and priority-based routing (try order=1 deployments first, then order=2). You can apply different strategies to different model groups. Fallbacks: if one provider fails, LiteLLM automatically retries on the next configured provider. This enables cost optimization (route simple queries to cheap models, complex queries to frontier models) and resilience (if OpenAI rate-limits, failover to Anthropic or self-hosted Ollama).

Can LiteLLM track costs across multiple providers?

Yes. LiteLLM tracks spend per virtual key, user, team, and organization across all providers. It maintains a database of token costs for 100+ models and automatically calculates spend per request. Features: per-key/team/org budgets with hard and soft limits, tag-based spend tracking, spend alerts via email, spend reports via API, and logging to S3/GCS/Azure Blob. This gives platform teams centralized cost visibility across all LLM providers — no more manual cost reconciliation across OpenAI, Anthropic, and Azure bills.

Should I use LiteLLM Proxy Server or Python SDK?

Use the Proxy Server (AI Gateway) if you're a platform team providing LLM access to multiple developers or projects — it centralizes authentication, cost tracking, rate limiting, and logging. Use the Python SDK if you're a developer building a single application — it provides the same unified interface with less infrastructure overhead. Most enterprises start with the SDK for prototyping, then migrate to the Proxy Server for production. The Proxy Server is also useful for self-hosted deployments — it sits in front of Ollama/vLLM and provides the same OpenAI-compatible API as cloud providers.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →