What is an LLM Gateway? Unified API Layer for Multi-Provider AI Apps
TL;DR — LLM gateway: unified API between app and LLM providers. Morph: "Sits between application and LLM providers, handling failover, rate limiting, caching, and observability. Three API integrations, three billing dashboards — gateway fixes this mess." LLM Gateway: "Route, manage, and analyze LLM requests across 35+ providers through one API. App sends one request, we route automatically." Braintrust: "Unified API for OpenAI, Anthropic, Google, AWS Bedrock, Vertex AI, Azure, Mistral. Point existing SDKs to gateway URL." OpenRouter: "LiteLLM: strongest open-source self-hosted gateway. Portkey: enterprise governance. Fallback billed at serving model rate." TECHSY: "9 gateways ranked. LiteLLM for control, OpenRouter for simplicity, Portkey for enterprise." Learn more with OpenRouter vs LiteLLM, LiteLLM tutorial, OpenRouter tutorial, and switch LLM provider.
Morph describes the problem: "Your app calls OpenAI directly. Then you add Anthropic. Then Gemini. Now you have three API integrations, three billing dashboards, three sets of rate limits, and no unified way to route between them. An LLM gateway sits between your application and LLM providers, handling failover, rate limiting, caching, and observability."
TECHSY summarizes the solution: "You're calling OpenAI for your chatbot, Anthropic for your coding assistant, and Gemini for your summarization pipeline. Three API keys, three SDKs, three billing dashboards, three sets of error handling. Now add fallback logic when one provider goes down. That's the mess LLM gateways fix, one unified API."
LLM Gateway Architecture
one API call
one SDK
one API key"] end subgraph Gateway["LLM Gateway"] Unified["Unified API
OpenAI-compatible endpoint
single API key"] Router["Router
route to best/cheapest/fastest
provider based on rules"] Fallback["Fallback
automatic switch
on provider failure"] RateLimit["Rate Limiting
per-provider and global
prevent 429 errors"] Cache["Response Cache
reduce costs and latency
TTL-based caching"] Observability["Observability
logs, metrics, traces
cost tracking per request"] LoadBalance["Load Balancing
distribute across providers
and model instances"] end subgraph Providers["LLM Providers (35+)"] OpenAI["OpenAI
GPT-4o, o1, o3
Azure OpenAI"] Anthropic["Anthropic
Claude 3.5, Claude 4
Sonnet, Opus, Haiku"] Google["Google
Gemini 2.0, Gemini Pro
Vertex AI"] AWS["AWS Bedrock
Claude, Llama, Titan
Mistral"] Mistral["Mistral
Mistral Large, Codestral
Pixtral"] Others["35+ Others
Cohere, AI21, Groq
Together, Anyscale"] end Code --> Unified Unified --> Router Router --> Fallback Fallback --> RateLimit RateLimit --> Cache Cache --> LoadBalance LoadBalance --> Observability LoadBalance --> OpenAI LoadBalance --> Anthropic LoadBalance --> Google LoadBalance --> AWS LoadBalance --> Mistral LoadBalance --> Others Observability --> App
LLM Gateway Comparison
| Gateway | Type | Providers | Key Feature | Best For |
|---|---|---|---|---|
| LiteLLM | Open-source, self-hosted | 100+ | Python proxy, maximum control | Teams with DevOps, data privacy |
| OpenRouter | Managed cloud | 300+ | Pay-per-use, no infrastructure | Teams wanting simplicity |
| Portkey | Enterprise managed | 100+ | Governance, compliance, guardrails | Enterprise, regulated industries |
| Braintrust | Managed | 10+ | Evals integration, observability | Teams focused on AI quality |
| Morph | Managed | 35+ | Unified API, automatic routing | Multi-provider apps |
| LLM Gateway | Managed | 35+ | Routing, analytics, cost tracking | Cost-conscious teams |
Key Features
| Feature | What It Does | Benefit |
|---|---|---|
| Unified API | One endpoint for all providers | Single SDK, single API key |
| Provider Routing | Route to best/cheapest/fastest | Optimize cost and performance |
| Automatic Fallback | Switch providers on failure | High availability |
| Rate Limiting | Per-provider and global limits | Prevent 429 errors |
| Response Caching | Cache identical requests | Reduce costs and latency |
| Observability | Logs, metrics, traces | Debug and monitor |
| Cost Tracking | Per-request, per-user, per-team | Budget management |
| Load Balancing | Distribute across providers | Scale and reliability |
Implementation
from dataclasses import dataclass
from typing import Optional
from enum import Enum
class GatewayType(Enum):
SELF_HOSTED = "self-hosted"
MANAGED = "managed"
ENTERPRISE = "enterprise"
class Provider(Enum):
OPENAI = "openai"
ANTHROPIC = "anthropic"
GOOGLE = "google"
AWS_BEDROCK = "aws-bedrock"
MISTRAL = "mistral"
AZURE = "azure"
@dataclass
class LLMGatewayGuide:
"""LLM gateway implementation guide."""
def get_what_is(self) -> dict:
"""What is an LLM gateway."""
return {
"definition": (
"Unified API layer that sits between "
"your application and LLM providers, "
"handling routing, fallback, rate "
"limiting, caching, observability, "
"and cost tracking"
),
"problem": (
"Without a gateway: three API "
"integrations, three billing "
"dashboards, three sets of rate "
"limits, no unified routing"
),
"solution": (
"One API call, one SDK, one API "
"key. Gateway routes to best "
"provider, handles failures, "
"tracks costs"
),
}
def get_key_features(self) -> dict:
"""Key features of LLM gateways."""
return {
"unified_api": {
"what": (
"One OpenAI-compatible endpoint "
"for all providers"
),
"benefit": (
"Single SDK, single API key, "
"no provider-specific code"
),
},
"provider_routing": {
"what": (
"Route requests to best, "
"cheapest, or fastest provider "
"based on configurable rules"
),
"benefit": (
"Optimize cost and performance "
"automatically"
),
},
"automatic_fallback": {
"what": (
"Switch to backup provider "
"when primary fails"
),
"benefit": (
"High availability — app keeps "
"running during provider outages"
),
"billing": (
"Billed at rate of model that "
"actually served the request"
),
},
"rate_limiting": {
"what": (
"Per-provider and global rate "
"limits to prevent 429 errors"
),
"benefit": (
"Avoid hitting provider limits, "
"distribute load"
),
},
"response_caching": {
"what": (
"Cache identical requests with "
"TTL"
),
"benefit": (
"Reduce costs and latency for "
"repeated queries"
),
},
"observability": {
"what": (
"Unified logs, metrics, traces "
"across all providers"
),
"benefit": (
"Debug issues, monitor "
"performance, track usage"
),
},
"cost_tracking": {
"what": (
"Per-request, per-user, per-team "
"cost tracking"
),
"benefit": (
"Budget management, chargeback, "
"cost optimization"
),
},
"load_balancing": {
"what": (
"Distribute requests across "
"providers and model instances"
),
"benefit": (
"Scale and reliability"
),
},
}
def get_gateway_comparison(self) -> dict:
"""LLM gateway comparison."""
return {
"litellm": {
"type": "open-source, self-hosted",
"providers": "100+",
"key_feature": (
"Python proxy, maximum control, "
"OpenAI-compatible API"
),
"best_for": (
"Teams with DevOps capacity, "
"data privacy requirements, "
"want maximum control"
),
"pros": [
"Maximum control over routing",
"Data stays in your infrastructure",
"No per-request gateway fees",
"100+ provider support",
"Open source (MIT license)",
],
"cons": [
"You handle infrastructure",
"You handle updates and monitoring",
"Operational complexity",
],
},
"openrouter": {
"type": "managed cloud",
"providers": "300+",
"key_feature": (
"Pay-per-use, no infrastructure, "
"instant setup"
),
"best_for": (
"Teams wanting simplicity, "
"no DevOps, fast start"
),
"pros": [
"No infrastructure to manage",
"300+ models available",
"Pay-per-use pricing",
"Instant setup",
"Automatic fallback",
],
"cons": [
"Per-request fees",
"Data goes through third party",
"Less control over routing",
],
},
"portkey": {
"type": "enterprise managed",
"providers": "100+",
"key_feature": (
"Governance, compliance, "
"guardrails, cache"
),
"best_for": (
"Enterprise, regulated industries, "
"compliance requirements"
),
"pros": [
"Enterprise governance",
"Compliance and audit trails",
"Cache and guardrails",
"Data residency controls",
],
"cons": [
"Higher cost",
"Enterprise pricing",
"Less flexibility than self-hosted",
],
},
"braintrust": {
"type": "managed",
"providers": "10+",
"key_feature": (
"Evals integration, observability, "
"unified API"
),
"best_for": (
"Teams focused on AI quality "
"and evaluation"
),
"pros": [
"Built-in evals",
"Observability",
"Unified API for major providers",
"Point existing SDKs to gateway",
],
"cons": [
"Fewer providers than OpenRouter",
"Less routing control than LiteLLM",
],
},
}
def get_decision_framework(self) -> dict:
"""Self-hosted vs managed decision framework."""
return {
"choose_self_hosted": {
"when": [
"Data privacy requirements (HIPAA, SOC2)",
"Team has DevOps capacity",
"Want maximum control over routing",
"Budget for infrastructure",
"Need custom routing logic",
"Compliance requires data in your VPC",
],
"best_option": "LiteLLM",
},
"choose_managed": {
"when": [
"Want to focus on product, not infrastructure",
"No DevOps team",
"Need instant setup",
"Want pay-per-use pricing",
"Need 300+ model access",
"Prefer no operational overhead",
],
"best_option": "OpenRouter",
},
"choose_enterprise": {
"when": [
"Regulated industry (finance, healthcare)",
"Compliance and audit trail requirements",
"Need governance and guardrails",
"Data residency requirements",
"Enterprise support needed",
],
"best_option": "Portkey",
},
}
def get_litellm_example(self) -> str:
"""LiteLLM proxy configuration example."""
return (
"# LiteLLM config.yaml\n"
"model_list:\n"
" - model_name: gpt-4o\n"
" litellm_params:\n"
" model: openai/gpt-4o\n"
" api_key: os.environ/OPENAI_API_KEY\n"
" - model_name: gpt-4o\n"
" litellm_params:\n"
" model: anthropic/claude-3-5-sonnet\n"
" api_key: os.environ/ANTHROPIC_API_KEY\n"
" - model_name: cheap\n"
" litellm_params:\n"
" model: google/gemini-2.0-flash\n"
" api_key: os.environ/GOOGLE_API_KEY\n"
"\n"
"router_settings:\n"
" routing_strategy: simple-shuffle\n"
" fallbacks:\n"
" - gpt-4o: [cheap]\n"
"\n"
"litellm_settings:\n"
" cache: true\n"
" cache_params:\n"
" type: redis\n"
" ttl: 3600\n"
"\n"
"# Run: litellm --config config.yaml\n"
"# Endpoint: http://localhost:4000\n"
"# Use OpenAI SDK pointing to gateway"
)
def get_openai_compatible_call(self) -> str:
"""Call gateway with OpenAI SDK."""
return (
"from openai import OpenAI\n"
"\n"
"# Point to gateway instead of OpenAI\n"
"client = OpenAI(\n"
" base_url='http://localhost:4000'\n"
" '/v1', # LiteLLM\n"
" # base_url='https://'\n"
" # 'openrouter.ai/api/v1', # OR\n"
" api_key='your-gateway-key',\n"
")\n"
"\n"
"response = client.chat.completions\\\n"
" .create(\n"
" model='gpt-4o', # gateway routes\n"
" messages=[\n"
" {'role': 'user',\n"
" 'content': 'Hello!'}\n"
" ],\n"
")\n"
"print(response.choices[0]\n"
" .message.content)"
)
LLM Gateway Checklist
- [ ] LLM gateway sits between application and LLM providers
- [ ] Unified API — one endpoint, one SDK, one API key for all providers
- [ ] Without gateway: three API integrations, three billing dashboards, three rate limit sets
- [ ] With gateway: one API call, gateway routes to best provider automatically
- [ ] Key feature: unified API (OpenAI-compatible endpoint for all providers)
- [ ] Key feature: provider routing (route to best/cheapest/fastest based on rules)
- [ ] Key feature: automatic fallback (switch providers on failure, billed at serving model rate)
- [ ] Key feature: rate limiting (per-provider and global, prevent 429 errors)
- [ ] Key feature: response caching (TTL-based, reduce costs and latency)
- [ ] Key feature: observability (unified logs, metrics, traces across providers)
- [ ] Key feature: cost tracking (per-request, per-user, per-team)
- [ ] Key feature: load balancing (distribute across providers and model instances)
- [ ] LiteLLM — open-source, self-hosted, 100+ providers, Python proxy, maximum control
- [ ] LiteLLM pros: maximum control, data in your infra, no per-request fees, 100+ providers
- [ ] LiteLLM cons: you handle infrastructure, updates, monitoring, operational complexity
- [ ] OpenRouter — managed cloud, 300+ models, pay-per-use, no infrastructure
- [ ] OpenRouter pros: no infrastructure, 300+ models, instant setup, automatic fallback
- [ ] OpenRouter cons: per-request fees, data through third party, less control
- [ ] Portkey — enterprise managed, governance, compliance, guardrails, cache
- [ ] Portkey pros: enterprise governance, compliance, audit trails, data residency
- [ ] Portkey cons: higher cost, enterprise pricing, less flexibility
- [ ] Braintrust — managed, evals integration, observability, unified API
- [ ] Braintrust pros: built-in evals, observability, point existing SDKs to gateway
- [ ] Braintrust cons: fewer providers, less routing control
- [ ] Choose self-hosted (LiteLLM) for: data privacy, DevOps capacity, maximum control, custom routing
- [ ] Choose managed (OpenRouter) for: simplicity, no DevOps, instant setup, pay-per-use
- [ ] Choose enterprise (Portkey) for: regulated industry, compliance, governance, audit trails
- [ ] LiteLLM config: model_list with provider mappings, router_settings with fallbacks, cache with Redis
- [ ] LiteLLM run: litellm --config config.yaml, endpoint http://localhost:4000
- [ ] Use OpenAI SDK pointing to gateway base_url instead of OpenAI directly
- [ ] Gateway handles vendor lock-in avoidance — switch providers without code changes
- [ ] Gateway handles provider outages — automatic fallback keeps app running
- [ ] Gateway handles cost management — route to cheaper models, cache responses
- [ ] Gateway handles observability — unified logs, metrics, traces across all providers
- [ ] Gateway handles simplified integration — one API instead of many provider-specific APIs
- [ ] Gateway handles rate limit management — per-provider and global limits
- [ ] Gateway handles compliance — governance, audit trails, data residency
- [ ] 35+ providers accessible through one API (OpenAI, Anthropic, Google, AWS Bedrock, Mistral, Azure)
- [ ] Read OpenRouter vs LiteLLM for detailed comparison
- [ ] Read LiteLLM tutorial for self-hosted setup
- [ ] Read OpenRouter tutorial for managed setup
- [ ] Read switch LLM provider for provider migration
- [ ] Test: gateway routes to correct provider based on model name
- [ ] Test: fallback switches to backup provider on primary failure
- [ ] Test: rate limiting prevents 429 errors
- [ ] Test: caching returns cached response for identical requests
- [ ] Test: cost tracking records per-request costs
- [ ] Test: OpenAI SDK works with gateway base_url
- [ ] Document gateway choice, provider config, routing rules, and fallback strategy
FAQ
What is an LLM gateway?
An LLM gateway is a unified API layer that sits between your application and LLM providers, handling routing, fallback, rate limiting, caching, observability, and cost tracking across multiple providers. Morph: "Your app calls OpenAI directly. Then you add Anthropic. Then Gemini. Now you have three API integrations, three billing dashboards, three sets of rate limits, and no unified way to route between them. An LLM gateway sits between your application and LLM providers, handling failover, rate limiting, caching, and observability." LLM Gateway: "Route, manage, and analyze your LLM requests across multiple providers with a unified API interface. Access OpenAI, Anthropic, Google, and 35+ providers through one API. Your app sends one request. We route it to the best provider automatically." TECHSY: "You are calling OpenAI for chatbot, Anthropic for coding, Gemini for summarization. Three API keys, three SDKs, three billing dashboards. That is the mess LLM gateways fix, one unified API."
What are the key features of an LLM gateway?
Key features include unified API, provider routing, automatic fallback, rate limiting, response caching, observability, cost tracking, and load balancing. Morph: "LLM gateway handles failover, rate limiting, caching, and observability. Unified API for all providers." Braintrust: "Unified API for routing requests to models from OpenAI, Anthropic, Google, AWS Bedrock, Vertex AI, Azure, Mistral, and other providers. Point existing SDKs to gateway URL and call supported models with a single API key." OpenRouter: "When a fallback fires, you are billed at the rate of the model that actually served the request." Features: (1) Unified API — one endpoint for all providers. (2) Provider routing — route to best/cheapest/fastest. (3) Automatic fallback — switch providers on failure. (4) Rate limiting — per-provider and global limits. (5) Response caching — reduce costs and latency. (6) Observability — logs, metrics, traces. (7) Cost tracking — per-request, per-user, per-team. (8) Load balancing — distribute across providers.
How do LLM gateways compare: LiteLLM, OpenRouter, Portkey, Braintrust?
LiteLLM is open-source self-hosted, OpenRouter is managed cloud, Portkey is enterprise governance, Braintrust is evals-focused. OpenRouter: "LiteLLM: currently one of the strongest open-source, self-hosted gateways for teams that want maximum control and provider flexibility, but you own the infrastructure and operational complexity. Portkey: enterprise-grade with governance features." Braintrust: "Braintrust Gateway provides unified API for routing to OpenAI, Anthropic, Google, AWS Bedrock, Vertex AI, Azure, Mistral. Point existing SDKs to gateway URL." TECHSY: "9 gateways ranked. LiteLLM for self-hosted control. OpenRouter for managed simplicity. Portkey for enterprise governance." Comparison: LiteLLM — open source, self-hosted, Python proxy, 100+ providers, maximum control. OpenRouter — managed cloud, 300+ models, pay-per-use, no infrastructure. Portkey — enterprise, governance, compliance, cache, guardrails. Braintrust — unified API, evals integration, observability.
Why do you need an LLM gateway?
You need an LLM gateway to avoid vendor lock-in, handle provider outages, manage costs, ensure observability, and simplify multi-provider integration. Morph: "Three API integrations, three billing dashboards, three sets of rate limits, and no unified way to route between them. LLM gateway fixes this mess." TECHSY: "Three API keys, three SDKs, three billing dashboards, three sets of error handling. Now add fallback logic when one provider goes down. That is the mess LLM gateways fix." LLM Gateway: "Your app sends one request. We route it to OpenAI, Anthropic, Google, or any of 35+ providers — automatically picking the best path." Reasons: (1) Vendor lock-in avoidance — switch providers without code changes. (2) Provider outages — automatic fallback keeps app running. (3) Cost management — route to cheaper models, cache responses. (4) Observability — unified logs, metrics, traces across providers. (5) Simplified integration — one API instead of many. (6) Rate limit management — per-provider and global. (7) Compliance — governance, audit trails, data residency.
How do you choose between self-hosted and managed LLM gateways?
Choose self-hosted (LiteLLM) for maximum control and data privacy, or managed (OpenRouter, Portkey) for simplicity and no infrastructure. OpenRouter: "LiteLLM: strongest open-source, self-hosted gateway for teams that want maximum control and provider flexibility, but you own the infrastructure and operational complexity." TECHSY: "LiteLLM for self-hosted control. OpenRouter for managed simplicity. Portkey for enterprise governance." Self-hosted (LiteLLM): (1) Maximum control over routing and config. (2) Data stays in your infrastructure. (3) No per-request gateway fees. (4) You handle infrastructure, updates, monitoring. (5) Best for teams with DevOps capacity and data privacy requirements. Managed (OpenRouter, Portkey, Braintrust): (1) No infrastructure to manage. (2) Instant setup. (3) Pay-per-use pricing. (4) Provider management handled for you. (5) Best for teams that want to focus on product, not infrastructure. Choose based on: data privacy requirements, DevOps capacity, budget structure, compliance needs, team size.
Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →