LiteLLM OpenAI Compatible API: One Interface for 100+ LLM Providers

By Fadi Updated Oct 7, 2025 11 min read LLM Gateways

TL;DR — LiteLLM OpenAI compatible API: one interface for 100+ providers. GitHub: "Single unified interface to call 100+ LLM providers using OpenAI format. Endpoints: /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a." LiteLLM Docs: "Open-source library, single unified interface, same completion() for all providers. Translate inputs to provider endpoints. Consistent output — same response format regardless of provider. All exceptions inherit from OpenAI types." LiteLLM Docs: "text-completion-openai/ prefix for /completions. Do NOT add /v1/embedding to base URL." LiteLLM Docs: "Anthropic, OpenAI, Qwen, xAI, Gemini, open-sourced LLMs supported. Selecting openai routes to OpenAI-compatible endpoint." Learn more with LLM gateway, LiteLLM tutorial, OpenRouter vs LiteLLM, and switch provider.

GitHub describes LiteLLM: "LiteLLM is an open source AI Gateway that gives you a single, unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more — using the OpenAI format."

LiteLLM Docs explains the value: "LiteLLM is an open-source library that gives you a single, unified interface to call 100+ LLMs — OpenAI, Anthropic, Vertex AI, Bedrock, and more — using the OpenAI format. Call any provider using the same completion() interface — no re-learning the API for each one."

LiteLLM OpenAI Compatible Architecture

flowchart TD subgraph App["Your Application"] Code["Business Logic
calls litellm.completion()
or OpenAI SDK to proxy"] end subgraph LiteLLM["LiteLLM Layer"] SDK["Python SDK
litellm.completion()
litellm.acompletion()
litellm.embedding()
litellm.image_generation()"] Proxy["Proxy Server
http://localhost:4000/v1
OpenAI-compatible endpoint
config.yaml routing"] Translation["Format Translation
OpenAI format → provider native
provider response → OpenAI format
consistent output"] Exceptions["Exception Handling
all exceptions inherit
OpenAI exception types
error-handling works across providers"] end subgraph Endpoints["Supported Endpoints"] Chat["/chat/completions
chat messages"] Completions["/completions
text completion"] Embeddings["/embeddings
vector embeddings"] Images["/images/generations
image generation"] Audio["/audio/transcriptions
/audio/speech"] Batches["/batches
async batch processing"] Rerank["/rerank
document reranking"] Responses["/responses
OpenAI Responses API"] end subgraph Providers["100+ LLM Providers"] OpenAI["OpenAI
gpt-4o, o1, o3"] Anthropic["Anthropic
Claude 3.5, Claude 4"] Google["Google
Gemini 2.0, Vertex AI"] Bedrock["AWS Bedrock
Claude, Llama, Titan"] Azure["Azure OpenAI
gpt-4o, gpt-4-turbo"] Local["Local Models
Ollama, vLLM, TGI"] Others["80+ Others
Mistral, Cohere, Groq
Together, Replicate, HF"] end Code --> SDK Code --> Proxy SDK --> Translation Proxy --> Translation Translation --> Exceptions Translation --> Chat Translation --> Completions Translation --> Embeddings Translation --> Images Translation --> Audio Translation --> Batches Translation --> Rerank Translation --> Responses Chat --> OpenAI Chat --> Anthropic Chat --> Google Chat --> Bedrock Chat --> Azure Chat --> Local Chat --> Others

Supported Endpoints

Endpoint	Purpose	LiteLLM Function	Example Model
/chat/completions	Chat messages	`litellm.completion()`	`openai/gpt-4o`
/completions	Text completion	`litellm.text_completion()`	`text-completion-openai/gpt-3.5-turbo-instruct`
/embeddings	Vector embeddings	`litellm.embedding()`	`openai/text-embedding-3-small`
/images/generations	Image generation	`litellm.image_generation()`	`openai/dall-e-3`
/audio/transcriptions	Speech-to-text	`litellm.transcription()`	`openai/whisper-1`
/audio/speech	Text-to-speech	`litellm.speech()`	`openai/tts-1`
/batches	Async batch	`litellm.batch_create()`	`openai/gpt-4o`
/rerank	Document reranking	`litellm.rerank()`	`cohere/rerank-english-v3.0`
/responses	Responses API	`litellm.responses()`	`openai/gpt-4o`

Provider Support

Provider	Model Prefix	Example	Notes
OpenAI	`openai/`	`openai/gpt-4o`	Native format
Anthropic	`anthropic/`	`anthropic/claude-3-5-sonnet`	System prompt extraction
Google Gemini	`gemini/`	`gemini/gemini-2.0-flash`	System instruction support
AWS Bedrock	`bedrock/`	`bedrock/anthropic.claude-3-5-sonnet`	Multiple model families
Azure OpenAI	`azure/`	`azure/gpt-4o-deployment`	Deployment name
Mistral	`mistral/`	`mistral/mistral-large-latest`	OpenAI-compatible
Cohere	`cohere/`	`cohere/command-r-plus`	Rerank support
Ollama	`ollama/`	`ollama/llama3`	Local, no API key
vLLM	`vllm/`	`vllm/meta-llama/Llama-3`	Local, OpenAI-compatible
Groq	`groq/`	`groq/llama-3.3-70b`	Fast inference

Implementation

from dataclasses import dataclass
from typing import Optional
from enum import Enum

class Endpoint(Enum):
    CHAT = "/chat/completions"
    COMPLETIONS = "/completions"
    EMBEDDINGS = "/embeddings"
    IMAGES = "/images/generations"
    AUDIO_TT = "/audio/transcriptions"
    AUDIO_TS = "/audio/speech"
    BATCHES = "/batches"
    RERANK = "/rerank"
    RESPONSES = "/responses"

@dataclass
class LiteLLMOpenAICompatibleGuide:
    """LiteLLM OpenAI compatible API guide."""

    def get_install(self) -> str:
        """Installation."""
        return (
            "# Install LiteLLM\n"
            "pip install litellm\n"
            "\n"
            "# With proxy server support\n"
            "pip install 'litellm[proxy]'\n"
            "\n"
            "# Set API keys as env vars\n"
            "export OPENAI_API_KEY='sk-xxx'\n"
            "export ANTHROPIC_API_KEY='sk-ant-xxx'\n"
            "export GOOGLE_API_KEY='AIzaxxx'"
        )

    def get_sdk_usage(self) -> str:
        """LiteLLM SDK usage — same interface for all providers."""
        return (
            "import litellm\n"
            "\n"
            "# OpenAI\n"
            "response = litellm.completion(\n"
            "    model='openai/gpt-4o',\n"
            "    messages=[\n"
            "        {'role': 'user',\n"
            "         'content': 'Hello!'}\n"
            "    ],\n"
            ")\n"
            "print(response.choices[0]\n"
            "      .message.content)\n"
            "\n"
            "# Anthropic — same interface\n"
            "response = litellm.completion(\n"
            "    model='anthropic/claude'\n"
            "        '-3-5-sonnet',\n"
            "    messages=[\n"
            "        {'role': 'user',\n"
            "         'content': 'Hello!'}\n"
            "    ],\n"
            ")\n"
            "print(response.choices[0]\n"
            "      .message.content)\n"
            "\n"
            "# Google Gemini — same interface\n"
            "response = litellm.completion(\n"
            "    model='gemini/gemini-2.0'\n"
            "        '-flash',\n"
            "    messages=[\n"
            "        {'role': 'user',\n"
            "         'content': 'Hello!'}\n"
            "    ],\n"
            ")\n"
            "print(response.choices[0]\n"
            "      .message.content)\n"
            "\n"
            "# All return same format\n"
            "# regardless of provider"
        )

    def get_async_usage(self) -> str:
        """Async usage."""
        return (
            "import litellm\n"
            "\n"
            "# Async completion\n"
            "response = await litellm\\\n"
            "    .acompletion(\n"
            "    model='openai/gpt-4o',\n"
            "    messages=[\n"
            "        {'role': 'user',\n"
            "         'content': 'Hello!'}\n"
            "    ],\n"
            ")\n"
            "\n"
            "# Async streaming\n"
            "response = await litellm\\\n"
            "    .acompletion(\n"
            "    model='anthropic/claude'\n"
            "        '-3-5-sonnet',\n"
            "    messages=[...],\n"
            "    stream=True,\n"
            ")\n"
            "async for chunk in response:\n"
            "    print(chunk.choices[0]\n"
            "          .delta.content or '',\n"
            "          end='', flush=True)"
        )

    def get_embeddings(self) -> str:
        """Embeddings with LiteLLM."""
        return (
            "import litellm\n"
            "\n"
            "# OpenAI embeddings\n"
            "response = litellm.embedding(\n"
            "    model='openai/text-'\n"
            "        'embedding-3-small',\n"
            "    input=['hello world'],\n"
            ")\n"
            "print(response.data[0].embedding)\n"
            "\n"
            "# Cohere embeddings\n"
            "response = litellm.embedding(\n"
            "    model='cohere/embed-'\n"
            "        'english-v3.0',\n"
            "    input=['hello world'],\n"
            ")\n"
            "\n"
            "# Same response format\n"
            "# regardless of provider"
        )

    def get_proxy_usage(self) -> str:
        """Proxy server usage with OpenAI SDK."""
        return (
            "# Start proxy server\n"
            "# litellm --config config.yaml\n"
            "# Endpoint: localhost:4000\n"
            "\n"
            "from openai import OpenAI\n"
            "\n"
            "# Point to LiteLLM proxy\n"
            "client = OpenAI(\n"
            "    base_url='http://localhost:'\n"
            "        '4000/v1',\n"
            "    api_key='sk-1234',\n"
            ")\n"
            "\n"
            "# Call any provider through proxy\n"
            "response = client.chat\\\n"
            "    .completions.create(\n"
            "    model='gpt-4o',\n"
            "    messages=[\n"
            "        {'role': 'user',\n"
            "         'content': 'Hello!'}\n"
            "    ],\n"
            ")\n"
            "# Proxy routes to OpenAI\n"
            "\n"
            "# Switch to Anthropic\n"
            "# by changing model only\n"
            "response = client.chat\\\n"
            "    .completions.create(\n"
            "    model='claude-3-5-sonnet',\n"
            "    messages=[...],\n"
            ")\n"
            "# Proxy routes to Anthropic\n"
            "# Same response format"
        )

    def get_format_translation(self) -> dict:
        """How format translation works."""
        return {
            "input": (
                "Your code sends OpenAI-format "
                "request: messages array with "
                "role:system, role:user, "
                "role:assistant"
            ),
            "translation_to_provider": [
                "OpenAI: pass through (native format)",
                "Anthropic: extract system message to "
                "top-level system parameter",
                "Google: convert to systemInstruction",
                "Bedrock: convert to provider-specific format",
                "Azure: pass through with deployment name",
            ],
            "provider_processes": (
                "Provider processes request in "
                "native format and returns "
                "provider-specific response"
            ),
            "translation_from_provider": [
                "OpenAI: pass through",
                "Anthropic: content[0].text → "
                "choices[0].message.content",
                "Google: result.response.text() → "
                "choices[0].message.content",
                "All: normalize to OpenAI ChatCompletion format",
            ],
            "output": (
                "Your code receives OpenAI-format "
                "response: choices[0].message.content, "
                "usage.total_tokens — same regardless "
                "of which provider handled the request"
            ),
            "exceptions": (
                "All exceptions inherit from OpenAI "
                "exception types. Any error-handling "
                "you write for OpenAI works across "
                "all providers."
            ),
        }

    def get_text_completion(self) -> str:
        """Text completion with prefix."""
        return (
            "import litellm\n"
            "\n"
            "# Text completion requires\n"
            "# text-completion-openai/ prefix\n"
            "response = litellm.text_completion(\n"
            "    model='text-completion-'\n"
            "        'openai/gpt-3.5-turbo-'\n"
            "        'instruct',\n"
            "    prompt='Once upon a time',\n"
            ")\n"
            "print(response.choices[0].text)\n"
            "\n"
            "# For /v1/completions route\n"
            "# with openai/ endpoints,\n"
            "# prefix NOT required"
        )

    def get_error_handling(self) -> str:
        """Error handling — OpenAI exception types."""
        return (
            "import litellm\n"
            "from openai import (\n"
            "    AuthenticationError,\n"
            "    RateLimitError,\n"
            "    APIConnectionError,\n"
            "    Timeout,\n"
            "    APIError,\n"
            ")\n"
            "\n"
            "# Same error handling for\n"
            "# ALL providers — LiteLLM\n"
            "# maps all exceptions to\n"
            "# OpenAI exception types\n"
            "try:\n"
            "    response = litellm.completion(\n"
            "        model='anthropic/claude'\n"
            "            '-3-5-sonnet',\n"
            "        messages=[...],\n"
            "    )\n"
            "except AuthenticationError:\n"
            "    print('Invalid API key')\n"
            "except RateLimitError:\n"
            "    print('Rate limited')\n"
            "except APIConnectionError:\n"
            "    print('Connection error')\n"
            "except Timeout:\n"
            "    print('Request timed out')\n"
            "except APIError as e:\n"
            "    print(f'API error: {e}')\n"
            "\n"
            "# Works for OpenAI, Anthropic,\n"
            "# Google, Bedrock, Azure, all"
        )

LiteLLM OpenAI Compatible API Checklist

[ ] LiteLLM: open-source AI Gateway, single unified interface for 100+ LLM providers
[ ] All providers called using OpenAI format — no re-learning API for each one
[ ] Supported endpoints: /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a
[ ] LiteLLM SDK: litellm.completion(), litellm.acompletion(), litellm.embedding(), litellm.image_generation()
[ ] LiteLLM proxy: http://localhost:4000/v1 — point OpenAI SDK to proxy
[ ] Format translation: OpenAI format input → provider native format → OpenAI format output
[ ] Consistent output: same response format regardless of which provider handles request
[ ] All exceptions inherit from OpenAI exception types — error-handling works across all providers
[ ] Install: pip install litellm or pip install 'litellm[proxy]' for proxy server
[ ] Set API keys as environment variables: OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY
[ ] Model prefix format: provider/model (e.g., openai/gpt-4o, anthropic/claude-3-5-sonnet)
[ ] OpenAI: openai/ prefix, native format, pass through
[ ] Anthropic: anthropic/ prefix, system prompt extraction to top-level parameter
[ ] Google Gemini: gemini/ prefix, system instruction support
[ ] AWS Bedrock: bedrock/ prefix, multiple model families (Claude, Llama, Titan)
[ ] Azure OpenAI: azure/ prefix, uses deployment name
[ ] Mistral: mistral/ prefix, OpenAI-compatible
[ ] Cohere: cohere/ prefix, rerank support
[ ] Ollama: ollama/ prefix, local models, no API key required
[ ] vLLM: vllm/ prefix, local, OpenAI-compatible
[ ] Groq: groq/ prefix, fast inference
[ ] 80+ more providers all accessible via same interface
[ ] /chat/completions: litellm.completion() — chat messages with roles
[ ] /completions: litellm.text_completion() — requires text-completion-openai/ prefix
[ ] /embeddings: litellm.embedding() — vector embeddings, same format for all providers
[ ] /images/generations: litellm.image_generation() — image generation
[ ] /audio/transcriptions: litellm.transcription() — speech-to-text
[ ] /audio/speech: litellm.speech() — text-to-speech
[ ] /batches: litellm.batch_create() — async batch processing
[ ] /rerank: litellm.rerank() — document reranking
[ ] /responses: litellm.responses() — OpenAI Responses API
[ ] Async support: litellm.acompletion(), litellm.aembedding() — async/await
[ ] Streaming: stream=True parameter, iterate over chunks with delta.content
[ ] Proxy server: config.yaml with model_list, start with litellm --config config.yaml
[ ] Proxy endpoint: http://localhost:4000/v1 — OpenAI SDK drop-in replacement
[ ] Proxy: routing, fallback, caching, budget tracking, virtual keys
[ ] Do NOT add /v1/embedding to base URL — LiteLLM handles endpoint routing
[ ] Error handling: AuthenticationError, RateLimitError, APIConnectionError, Timeout, APIError
[ ] Same error handling code works for ALL providers — exceptions mapped to OpenAI types
[ ] Switching providers: change model string only (openai/gpt-4o → anthropic/claude-3-5-sonnet)
[ ] Read LLM gateway guide for gateway concepts
[ ] Read LiteLLM tutorial for proxy setup
[ ] Read OpenRouter vs LiteLLM for comparison
[ ] Read switch LLM provider for provider migration
[ ] Test: same completion() call works for OpenAI, Anthropic, Google
[ ] Test: response format consistent across all providers
[ ] Test: error handling catches OpenAI exception types for all providers
[ ] Test: proxy server routes to correct provider based on model name
[ ] Test: embeddings return same format regardless of provider
[ ] Test: streaming works with stream=True for all providers
[ ] Document model prefixes, endpoints used, proxy config, and error handling

FAQ

What is the LiteLLM OpenAI compatible API?

LiteLLM provides a single unified interface to call 100+ LLM providers using the OpenAI format. GitHub: "LiteLLM is an open source AI Gateway that gives you a single, unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more — using the OpenAI format. All supported endpoints: /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a." LiteLLM Docs: "LiteLLM is an open-source library that gives you a single, unified interface to call 100+ LLMs using the OpenAI format. Call any provider using the same completion() interface — no re-learning the API for each one. Translate inputs to provider endpoints. Consistent output — same response format regardless of which provider you use."

How does LiteLLM translate between provider formats?

LiteLLM translates OpenAI-format requests to each provider's native API format and normalizes responses back to OpenAI format. LiteLLM Docs: "Translate inputs to provider endpoints (/chat/completions, /responses, /embeddings, /images, /audio, /batches, and more). Consistent output — same response format regardless of which provider you use. All exceptions inherit from OpenAI exception types, so any error-handling you write for OpenAI works across all providers." LiteLLM Docs: "For /completions: put text-completion-openai/ in front of model name. Do NOT add anything additional to the base URL e.g. /v1/embedding." Translation: (1) OpenAI format input → LiteLLM translates to provider native format. (2) Provider processes request. (3) LiteLLM translates response back to OpenAI format. (4) Your code sees consistent response regardless of provider.

What endpoints does LiteLLM support?

LiteLLM supports /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, and /a2a endpoints. GitHub: "All supported endpoints: /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a." LiteLLM Docs: "Translate inputs to provider endpoints (/chat/completions, /responses, /embeddings, /images, /audio, /batches, and more)." Endpoints: /chat/completions (chat), /completions (text completion), /embeddings (vector embeddings), /images/generations (image generation), /audio/transcriptions (speech-to-text), /audio/speech (text-to-speech), /batches (async batch processing), /rerank (document reranking), /responses (OpenAI Responses API).

How do you use LiteLLM as a proxy server?

Run LiteLLM proxy server with config.yaml and point your OpenAI SDK to the proxy endpoint. LiteLLM Docs: "Integrate LiteLLM directly into your codebase or use proxy server. Router with retry/fallback logic across multiple deployments." GitHub: "Python SDK, Proxy Server (AI Gateway). Direct Python library integration or proxy server." Proxy setup: (1) Create config.yaml with model_list. (2) Start proxy: litellm --config config.yaml. (3) Endpoint: http://localhost:4000/v1. (4) Point OpenAI SDK: base_url='http://localhost:4000/v1'. (5) All requests go through proxy with routing, fallback, caching, budget tracking. (6) Same OpenAI format for all providers.

Which providers does LiteLLM support?

LiteLLM supports 100+ providers including OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, Mistral, Cohere, and local models. GitHub: "100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more." LiteLLM Docs: "Anthropic, OpenAI, Qwen, xAI, Gemini and most of open-sourced LLMs are supported. Selecting openai as provider routes to OpenAI-compatible endpoint." Providers: OpenAI, Anthropic (Claude), Google (Gemini, Vertex AI), AWS Bedrock, Azure OpenAI, Mistral, Cohere, AI21, Together AI, Anyscale, Replicate, Hugging Face, Ollama, vLLM, TGI, Cerebras, Groq, Clarifai, xAI, Qwen, and 80+ more. All accessible via same completion() interface or proxy endpoint.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →