LiteLLM, OpenRouter, provider switching, streaming, and OpenAI-compatible APIs. 12 articles, written by the engineers building AI App Lab.
LLM token tracking and cost: calculate per-request spend, OpenAI Anthropic pricing, LiteLLM spend tracking with PostgreSQL, virtual keys with budgets, Langfuse cost analytics, budget alerts, cost attribution, and production checklist for 2026.
LLM streaming with Server-Sent Events: OpenAI stream=True delta chunks, Anthropic content_block_delta, FastAPI StreamingResponse, async generators, backpressure, X-Accel-Buffering, asyncio.Queue, error handling, React client, and production checklist for 2026.
LLM gateway sits between app and LLM providers, handling routing, fallback, rate limiting, caching, observability, and cost tracking. OpenAI, Anthropic, Google, 35+ providers via one API. LiteLLM, OpenRouter, Portkey, Braintrust compared, and production checklist for 2026.
OpenRouter vs LiteLLM: managed SaaS with 500+ models vs open-source self-hosted proxy with 100+ providers. Pricing, routing, fallback, caching, RBAC, data sovereignty, setup time, throughput, and decision framework for 2026.
LLM rate limiting and retries: handle 429 errors with exponential backoff, full jitter, Retry-After header, token bucket, circuit breaker. RPM and TPM limits, OpenAI Anthropic presets, tenacity, rate-limit-shield, and production checklist for 2026.
LLM fallbacks and automatic switching: primary-to-secondary failover chains, LiteLLM fallbacks config, OpenRouter models array, provider failover vs model fallback, circuit breaker, graceful degradation, retry vs fallback, and production checklist for 2026.
AWS Bedrock with LiteLLM: config.yaml with bedrock/ prefix, boto3 authentication, IAM roles, multi-region load balancing, Claude Llama Titan models, routing strategies, fallbacks, converse/invoke endpoints, and production checklist for 2026.
OpenRouter tutorial: get API key, use OpenAI SDK as drop-in replacement, model routing with provider preferences, fallbacks, free models, BYOK, streaming, Auto Router by NotDiamond, credits pricing, and production checklist for 2026.
LiteLLM OpenAI compatible API: call 100+ providers using OpenAI format. Endpoints: /chat/completions, /embeddings, /images, /audio, /batches, /rerank. Format translation, consistent output, proxy server, provider routing, and production checklist for 2026.
LiteLLM tutorial: install proxy server with Docker, configure config.yaml with model_list, routing, fallbacks, caching. PostgreSQL for spend tracking, Redis for cache and rate limits. Virtual keys, per-team budgets, OpenAI-compatible API, observability, and production checklist for 2026.
LiteLLM with OpenRouter as upstream: config.yaml with openrouter/ models, local RBAC and budget enforcement, Redis caching, Langfuse logging, OpenRouter 500+ model access and automatic failover, hybrid setup, Docker Compose, and production checklist for 2026.
Switch LLM provider without rewriting code. Abstraction layer pattern, adapter classes, provider differences (system messages, response shapes), LiteLLM and OpenRouter gateways, fallback cutover, circuit breaker, prompt migration, and production checklist for 2026.