Switch LLM Provider: Migrate OpenAI to Anthropic or Gemini Without Rewriting
TL;DR — Switch LLM provider without rewriting code. OneAppleFall: "Put a layer between app and providers. Code calls one interface, switching is config change. Gateway like LiteLLM for multiple providers. Cut over with fallback — users never see failure. Plan before you need to." APIScout: "OpenRouter and LiteLLM abstract providers behind OpenAI-compatible interface. Hand-rolled gives more control. Biggest mismatch: system messages (OpenAI role:system, Anthropic top-level system, Google systemInstruction). Adapter layer normalizes — app sees AIMessage[] in, AIResponse out." Markaicode: "Do not call LiteLLM directly — build service layer on top. Circuit breaker reduces cascade failures by 94%. Replace openai calls with litellm calls via find/replace." LLM API Router: "Zero-code switching by modifying configuration. ProviderAdapter converts unified requests to vendor HTTP. Supports OpenAI, Anthropic, Google, DeepSeek, Ollama." Learn more with LLM gateway, OpenRouter vs LiteLLM, LiteLLM tutorial, and OpenRouter tutorial.
OneAppleFall frames the problem: "Relying on a single LLM provider is a quiet risk: one outage, price spike, or deprecated model can break your whole app — and migrating in a panic means rewriting code and re-testing in production. The good news is that switching providers is straightforward if you architect for it."
Markaicode warns against trading one lock-in for another: "If you replace openai.ChatCompletion.create() with litellm.completion(), you've just traded one vendor lock-in for another. The real power comes from building your own abstraction that sits between your business logic and LiteLLM."
Provider Switching Architecture
calls unified interface
AIMessage[] in, AIResponse out"] end subgraph Abstraction["Abstraction Layer"] Interface["Unified Interface
callAI() function
task-based routing"] Adapters["Provider Adapters
OpenAI Adapter
Anthropic Adapter
Google Adapter"] Router["Router
ordered provider list
quality-first / cost-first
task-aware routing"] CircuitBreaker["Circuit Breaker
reduces cascade failures 94%
open/closed/half-open states"] end subgraph Gateway["Gateway Option"] LiteLLM["LiteLLM Proxy
OpenAI-compatible endpoint
6 routing modes
fallback chains"] OpenRouter["OpenRouter
managed SaaS
500+ models
automatic failover"] end subgraph Providers["LLM Providers"] OpenAI["OpenAI
system: role in messages
response: choices[0].message.content"] Anthropic["Anthropic
system: top-level param
response: content[0].text"] Google["Google Gemini
system: systemInstruction
response: result.response.text()"] Local["Local Models
Ollama, vLLM
OpenAI-compatible"] end subgraph Fallback["Fallback Strategy"] Primary["Primary Provider
new provider after switch"] Backup["Backup Provider
old provider as fallback"] AutoSwitch["Automatic Switch
on error or rate-limit
users never see failure"] end Code --> Interface Interface --> Adapters Interface --> Router Router --> CircuitBreaker CircuitBreaker --> LiteLLM CircuitBreaker --> OpenRouter CircuitBreaker --> Adapters LiteLLM --> OpenAI LiteLLM --> Anthropic LiteLLM --> Google LiteLLM --> Local OpenRouter --> OpenAI OpenRouter --> Anthropic OpenRouter --> Google Adapters --> OpenAI Adapters --> Anthropic Adapters --> Google Primary --> AutoSwitch AutoSwitch --> Backup
Provider API Differences
| Feature | OpenAI | Anthropic | Google Gemini |
|---|---|---|---|
| System message | role: "system" in messages array |
Top-level system parameter |
systemInstruction field |
| Response shape | choices[0].message.content |
content[0].text |
result.response.text() |
| SDK | openai Python/TS |
anthropic Python/TS |
google-generativeai Python |
| Streaming | stream=True, delta chunks |
stream=True, content_block_delta |
stream=True, chunk text |
| Function calling | tools parameter |
tools parameter (different format) |
tools parameter (different format) |
| Max tokens | max_tokens |
max_tokens |
max_output_tokens |
Migration Approaches
| Approach | Setup Time | Control | Best For |
|---|---|---|---|
| Gateway (LiteLLM) | ~45 min | Medium | Teams wanting simplicity + control |
| Gateway (OpenRouter) | ~5 min | Low | Teams wanting zero infrastructure |
| Hand-rolled adapters | ~2-3 hours | Maximum | Teams needing custom routing logic |
| Hybrid (gateway + service layer) | ~3-4 hours | High | Production teams needing both |
Implementation
from dataclasses import dataclass
from typing import Optional
from enum import Enum
class Provider(Enum):
OPENAI = "openai"
ANTHROPIC = "anthropic"
GOOGLE = "google"
OLLAMA = "ollama"
class RoutingStrategy(Enum):
QUALITY_FIRST = "quality-first"
COST_FIRST = "cost-first"
TASK_AWARE = "task-aware"
@dataclass
class SwitchLLMProviderGuide:
"""Switch LLM provider implementation guide."""
def get_abstraction_pattern(self) -> str:
"""Abstraction layer pattern."""
return (
"from abc import ABC, abstractmethod\n"
"from dataclasses import dataclass\n"
"from typing import Any\n"
"\n"
"@dataclass\n"
"class AIMessage:\n"
" role: str\n"
" content: str\n"
"\n"
"@dataclass\n"
"class AIResponse:\n"
" content: str\n"
" model: str\n"
" provider: str\n"
" tokens_used: int\n"
" cost: float\n"
"\n"
"class BaseAdapter(ABC):\n"
" '''Unified adapter interface.'''\n"
"\n"
" @abstractmethod\n"
" async def complete(\n"
" self,\n"
" messages: list[AIMessage],\n"
" system: str | None = None,\n"
" temperature: float = 0.7,\n"
" max_tokens: int = 4096,\n"
" ) -> AIResponse:\n"
" pass\n"
"\n"
"class OpenAIAdapter(BaseAdapter):\n"
" '''OpenAI adapter.'''\n"
"\n"
" def __init__(self, api_key: str):\n"
" from openai import AsyncOpenAI\n"
" self.client = AsyncOpenAI(\n"
" api_key=api_key)\n"
"\n"
" async def complete(\n"
" self, messages, system=None,\n"
" temperature=0.7, max_tokens=4096\n"
" ) -> AIResponse:\n"
" msg = [{'role': m.role,\n"
" 'content': m.content}\n"
" for m in messages]\n"
" if system:\n"
" msg.insert(0, {\n"
" 'role': 'system',\n"
" 'content': system})\n"
" r = await self.client\\\n"
" .chat.completions.create(\n"
" model='gpt-4o',\n"
" messages=msg,\n"
" temperature=temperature,\n"
" max_tokens=max_tokens)\n"
" return AIResponse(\n"
" content=r.choices[0]\n"
" .message.content,\n"
" model='gpt-4o',\n"
" provider='openai',\n"
" tokens_used=r.usage\n"
" .total_tokens,\n"
" cost=0.0)\n"
"\n"
"class AnthropicAdapter(\n"
" BaseAdapter):\n"
" '''Anthropic adapter.'''\n"
"\n"
" def __init__(self, api_key: str):\n"
" import anthropic\n"
" self.client = anthropic\\\n"
" .AsyncAnthropic(\n"
" api_key=api_key)\n"
"\n"
" async def complete(\n"
" self, messages, system=None,\n"
" temperature=0.7, max_tokens=4096\n"
" ) -> AIResponse:\n"
" msg = [{'role': m.role,\n"
" 'content': m.content}\n"
" for m in messages]\n"
" r = await self.client\\\n"
" .messages.create(\n"
" model='claude-3-5-sonnet',\n"
" system=system or '',\n"
" messages=msg,\n"
" temperature=temperature,\n"
" max_tokens=max_tokens)\n"
" return AIResponse(\n"
" content=r.content[0].text,\n"
" model='claude-3-5-sonnet',\n"
" provider='anthropic',\n"
" tokens_used=r.usage\n"
" .input_tokens + r.usage\n"
" .output_tokens,\n"
" cost=0.0)\n"
"\n"
"class GoogleAdapter(\n"
" BaseAdapter):\n"
" '''Google Gemini adapter.'''\n"
"\n"
" def __init__(self, api_key: str):\n"
" import google.generativeai as genai\n"
" genai.configure(api_key=api_key)\n"
" self.model = genai\\\n"
" .GenerativeModel(\n"
" 'gemini-2.0-flash')\n"
"\n"
" async def complete(\n"
" self, messages, system=None,\n"
" temperature=0.7, max_tokens=4096\n"
" ) -> AIResponse:\n"
" msg = [{'role': m.role,\n"
" 'parts': [m.content]}\n"
" for m in messages]\n"
" r = await self.model\\\n"
" .generate_content_async(\n"
" msg,\n"
" generation_config={\n"
" 'temperature':\n"
" temperature,\n"
" 'max_output_tokens':\n"
" max_tokens})\n"
" return AIResponse(\n"
" content=r.text,\n"
" model='gemini-2.0-flash',\n"
" provider='google',\n"
" tokens_used=r\n"
" .usage_metadata\n"
" .total_token_count,\n"
" cost=0.0)"
)
def get_router(self) -> str:
"""Router with fallback."""
return (
"class LLMRouter:\n"
" '''Multi-provider router\n"
" with fallback.'''\n"
"\n"
" def __init__(self, adapters:\n"
" dict[str, BaseAdapter]):\n"
" self.adapters = adapters\n"
" self.circuit_breaker = {}\n"
"\n"
" async def call(\n"
" self,\n"
" messages: list[AIMessage],\n"
" system: str | None = None,\n"
" providers: list[str] = None,\n"
" **kwargs\n"
" ) -> AIResponse:\n"
" '''Try providers in order,\n"
" fallback on error.'''\n"
" providers = providers or [\n"
" 'openai', 'anthropic',\n"
" 'google']\n"
" last_error = None\n"
" for p in providers:\n"
" if self.circuit_breaker\\\n"
" .get(p, {})\\\n"
" .get('open', False):\n"
" continue\n"
" try:\n"
" adapter = self\\\n"
" .adapters[p]\n"
" response = await adapter\\\n"
" .complete(\n"
" messages, system,\n"
" **kwargs)\n"
" self._record_success(p)\n"
" return response\n"
" except Exception as e:\n"
" last_error = e\n"
" self._record_failure(p)\n"
" continue\n"
" raise last_error\n"
"\n"
" def _record_success(\n"
" self, provider: str):\n"
" self.circuit_breaker[\n"
" provider] = {\n"
" 'open': False,\n"
" 'failures': 0}\n"
"\n"
" def _record_failure(\n"
" self, provider: str):\n"
" state = self\\\n"
" .circuit_breaker\\\n"
" .setdefault(provider,\n"
" {'open': False,\n"
" 'failures': 0})\n"
" state['failures'] += 1\n"
" if state['failures'] >= 3:\n"
" state['open'] = True"
)
def get_task_routing(self) -> dict:
"""Task-based routing configuration."""
return {
"routing_rules": {
"simple": [
"google", "openai",
"anthropic",
],
"complex": [
"anthropic", "openai",
"google",
],
"long_context": [
"google", "anthropic",
"openai",
],
"cheap": [
"google", "openai",
"anthropic",
],
},
"description": (
"Task type from client signals "
"quality-vs-cost trade-off "
"without knowing which provider "
"is used. Router picks provider "
"order based on task type."
),
}
def get_litellm_migration(self) -> str:
"""LiteLLM migration — find/replace."""
return (
"# Before: direct OpenAI call\n"
"import openai\n"
"response = openai.ChatCompletion\\\n"
" .create(\n"
" model='gpt-4o',\n"
" messages=[...])\n"
"\n"
"# After: LiteLLM call\n"
"# (find/replace)\n"
"import litellm\n"
"response = litellm.completion(\n"
" model='openai/gpt-4o',\n"
" messages=[...])\n"
"\n"
"# Switch to Anthropic:\n"
"# change model string only\n"
"response = litellm.completion(\n"
" model='anthropic/claude'\n"
" '-3-5-sonnet',\n"
" messages=[...])\n"
"\n"
"# Switch to Google:\n"
"response = litellm.completion(\n"
" model='gemini/gemini-2.0'\n"
" '-flash',\n"
" messages=[...])\n"
"\n"
"# WARNING: Do not call LiteLLM\n"
"# directly from business logic.\n"
"# Build service layer on top."
)
def get_openrouter_migration(self) -> str:
"""OpenRouter migration — base_url change."""
return (
"# Before: direct OpenAI\n"
"from openai import OpenAI\n"
"client = OpenAI(\n"
" api_key='sk-xxx')\n"
"\n"
"# After: OpenRouter\n"
"# (change 2 values)\n"
"client = OpenAI(\n"
" base_url='https://'\n"
" 'openrouter.ai/api/v1',\n"
" api_key='sk-or-v1-xxx')\n"
"\n"
"# Switch model by changing\n"
"# model slug only\n"
"response = client.chat\\\n"
" .completions.create(\n"
" model='anthropic/claude'\n"
" '-sonnet-4',\n"
" messages=[...])\n"
"\n"
"# Switching direction is a\n"
"# base URL and key change,\n"
"# since both speak OpenAI format"
)
def get_cutover_strategy(self) -> dict:
"""Safe cutover strategy."""
return {
"steps": [
"Deploy new provider as primary",
"Keep old provider as fallback",
"Monitor error rates and latency",
"Gradually shift traffic",
"Keep fallback for outage protection",
],
"fallback_behavior": (
"If new primary errors or "
"rate-limits, code silently "
"reroutes to old provider. "
"Users never see a failure."
),
"best_practices": [
"Plan for switching before you need to",
"Add abstraction layer from first commit",
"Store keys for at least two providers",
"Design prompts to be model-agnostic",
"Keep escape hatch for provider-native features",
"Run multi-provider by default",
"Route by cost, latency, or capability",
"Fall back automatically during outages",
],
}
Switch LLM Provider Checklist
- [ ] Put an abstraction layer between your app and LLM providers
- [ ] Your code calls one consistent interface — switching is a config change
- [ ] Relying on a single provider is a quiet risk — one outage breaks your app
- [ ] Migrating in a panic means rewriting code and re-testing in production
- [ ] Create adapter interface (BaseAdapter with complete() method)
- [ ] Implement adapters per provider (OpenAI, Anthropic, Google)
- [ ] Use gateway (LiteLLM, OpenRouter) or custom router
- [ ] Do not call LiteLLM directly from business logic — build service layer on top
- [ ] Replacing openai.ChatCompletion.create() with litellm.completion() trades one lock-in for another
- [ ] Build your own abstraction that sits between business logic and LiteLLM
- [ ] Provider differences: system messages (OpenAI role:system, Anthropic top-level system, Google systemInstruction)
- [ ] Provider differences: response shapes (OpenAI choices[0].message.content, Anthropic content[0].text, Google result.response.text())
- [ ] Provider differences: SDK interfaces (openai, anthropic, google-generativeai packages)
- [ ] Provider differences: max tokens param (max_tokens vs max_output_tokens)
- [ ] Adapter layer absorbs all differences — app sees AIMessage[] in, AIResponse out
- [ ] Router: ordered list of providers to try, catch errors, move to next
- [ ] Routing strategies: quality-first, cost-first, task-aware
- [ ] Task-based routing: simple (Google first), complex (Anthropic first), long-context (Google first), cheap (Google first)
- [ ] Circuit breaker: reduces cascade failures by 94% in microservice architectures
- [ ] Circuit breaker: open/closed/half-open states, check before attempting
- [ ] Circuit breaker: record success/failure, open after 3 failures
- [ ] Cut over safely: keep old provider as fallback
- [ ] If new primary errors or rate-limits, code silently reroutes — users never see failure
- [ ] Cutover steps: deploy new as primary, keep old as fallback, monitor, gradually shift, keep fallback
- [ ] LiteLLM migration: find/replace openai.ChatCompletion.create() with litellm.completion()
- [ ] LiteLLM: switch model by changing model string only (openai/gpt-4o → anthropic/claude-3-5-sonnet)
- [ ] OpenRouter migration: change base_url and api_key only — rest of code stays
- [ ] OpenRouter: switching direction is a base URL and key change (both speak OpenAI format)
- [ ] Zero-code switching: modify configuration only, no code changes (LLM API Router pattern)
- [ ] ProviderAdapter converts unified requests into vendor HTTP requests and normalizes responses
- [ ] Map API differences before migration
- [ ] Re-test all prompts with new provider
- [ ] Adjust prompt wording for model differences
- [ ] Test edge cases and error handling
- [ ] Design prompts to be model-agnostic where possible
- [ ] Keep provider-specific prompts as escape hatch for native features
- [ ] Use task-based routing for quality vs cost trade-off
- [ ] Store keys for at least two providers
- [ ] Add abstraction layer from the first commit — costs nothing early, saves rewrite later
- [ ] Many production teams run multi-provider by default
- [ ] Route by cost, latency, or capability
- [ ] Fall back automatically during outages
- [ ] Multi-provider setup is 2-3 hours upfront — prevents single LLM outage from taking down product
- [ ] Gateway (LiteLLM): ~45 min setup, medium control, for simplicity + control
- [ ] Gateway (OpenRouter): ~5 min setup, low control, for zero infrastructure
- [ ] Hand-rolled adapters: ~2-3 hours, maximum control, for custom routing logic
- [ ] Hybrid (gateway + service layer): ~3-4 hours, high control, for production teams
- [ ] Read LLM gateway guide for gateway concepts
- [ ] Read OpenRouter vs LiteLLM for comparison
- [ ] Read LiteLLM tutorial for self-hosted setup
- [ ] Read OpenRouter tutorial for managed setup
- [ ] Test: adapter normalizes system messages correctly for each provider
- [ ] Test: adapter normalizes response shapes to AIResponse
- [ ] Test: router falls back to next provider on failure
- [ ] Test: circuit breaker opens after 3 failures and blocks provider
- [ ] Test: cutover with fallback — users never see failure
- [ ] Document provider config, routing strategy, fallback order, and prompt adjustments
FAQ
How do you switch LLM providers without rewriting code?
Put an abstraction layer between your app and providers. Your code calls one interface; switching is a config change. OneAppleFall: "The core technique: put a layer between your app and the providers. Instead of calling a provider API directly, your code calls one consistent interface that hides the provider behind it. Switching becomes a configuration change because your app interacts only with the abstraction. For more than a couple providers, a gateway like LiteLLM is cleaner than hand-rolled code. It exposes one OpenAI-style endpoint across OpenAI, Claude, Gemini, and self-hosted models." Markaicode: "Do not call LiteLLM directly from application code. Build your own service layer on top. If you replace openai.ChatCompletion.create() with litellm.completion(), you have just traded one vendor lock-in for another." Steps: (1) Create adapter interface. (2) Implement adapters per provider. (3) Use gateway or custom router. (4) Switch via config change. (5) Keep old provider as fallback.
What are the key differences between LLM provider APIs?
Providers differ in system message handling, response shapes, and SDK interfaces. APIScout: "The biggest impedance mismatch is system messages. OpenAI puts system message in messages array with role: system. Anthropic separates it into top-level system parameter. Google uses systemInstruction. Response shapes differ: OpenAI returns choices[0].message.content, Anthropic returns content[0].text, Google returns result.response.text(). The adapter layer absorbs all of this. Your application code only ever sees AIMessage[] in and AIResponse out." LLM API Router: "Zero-code switching by modifying configuration. ProviderAdapter converts unified requests into vendor HTTP requests and normalizes responses. Anthropic handles system prompt extraction automatically. Google Gemini supports system instruction." Differences: (1) System messages: OpenAI (role:system in messages), Anthropic (top-level system param), Google (systemInstruction). (2) Response shapes: OpenAI (choices[0].message.content), Anthropic (content[0].text), Google (result.response.text()). (3) SDK interfaces: different packages and methods.
How do you cut over safely when switching providers?
Keep the old provider as a fallback. If the new primary errors or rate-limits, your code silently reroutes. OneAppleFall: "Cut over safely by keeping the old provider as a fallback. If the new primary errors or rate-limits, your code silently reroutes — users never see a failure. This same pattern also protects against future outages. The best time to plan for switching is before you need to. Add a thin abstraction layer from the first commit. Store keys for at least two providers. Design prompts to be reasonably model-agnostic." Markaicode: "Circuit breaker reduces cascade failures by 94% in microservice architectures. Check circuit breaker before attempting. If open, return fallback completion. Record success/failure for circuit breaker state." Cutover steps: (1) Deploy new provider as primary. (2) Keep old provider as fallback. (3) Monitor error rates and latency. (4) Gradually shift traffic. (5) Keep fallback for outage protection.
Should you use a gateway or hand-rolled abstraction to switch providers?
Use a gateway (LiteLLM, OpenRouter) for simplicity or hand-rolled for maximum control. OneAppleFall: "For more than a couple providers, a gateway like LiteLLM is cleaner than hand-rolled code. It exposes one OpenAI-style endpoint and adds auth, cost tracking, rate limiting, and fallbacks. If your app already speaks the OpenAI SDK, you often just point it at the gateway URL." APIScout: "OpenRouter and LiteLLM can give you multi-provider support with minimal code — they abstract all providers behind a single OpenAI-compatible interface. But a hand-rolled abstraction gives you more control over routing logic, cost attribution, and fallback behavior." Markaicode: "LiteLLM gives a single interface to OpenAI, Anthropic, Google, Azure, Cohere, and dozens of open-source models. But do not call LiteLLM directly — build your own service layer on top." Gateway: faster setup, less control. Hand-rolled: more control, more code. Hybrid: gateway as backend, custom service layer on top.
How do you migrate prompts when switching LLM providers?
Map API differences, re-test prompts with new provider, and design prompts to be model-agnostic. OneAppleFall: "Map the API differences, re-test prompts, and cut over with a fallback. Design your prompts to be reasonably model-agnostic. Many production teams now run multi-provider by default, routing by cost, latency, or capability and falling back automatically." APIScout: "Task-based routing: use the best model for each task. Cost optimization: route to cheapest provider that meets quality needs. The adapter layer normalizes all providers into a single interface." Prompt migration: (1) Map API differences (system messages, response shapes). (2) Re-test all prompts with new provider. (3) Adjust prompt wording for model differences. (4) Test edge cases and error handling. (5) Design prompts to be model-agnostic where possible. (6) Keep provider-specific prompts as escape hatch. (7) Use task-based routing for quality vs cost trade-off.
Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →