How to Secure LLM API Keys in a Multi-Tenant Environment
TL;DR — Securing LLM API keys in multi-tenant AI requires per-tenant key isolation, encrypted credential vaults, and virtual keys with budgets and rate limits. Never share API keys across tenants — it is a critical security failure. Use Fernet or AES-256-GCM for encryption at the application layer. Implement dual-key rotation for zero-downtime key changes. Virtual API keys sit in front of provider credentials — a leaked virtual key is a five-second revocation, not a five-hour rotation. For high-sensitivity keys, use dynamic secrets from HashiCorp Vault with automatic expiration.
A leaked OpenAI key can rack up thousands of dollars in charges in minutes. A compromised Anthropic key can expose your entire prompt history. In a multi-tenant AI platform, the risk multiplies — one shared key breach exposes every tenant's data and costs (GrizzlyPeak 2026).
Multi-tenant AI agent platforms that share API keys across tenants represent a fundamental failure of the isolation contract that enterprise customers expect and that data protection regulations require (Armalo AI 2026). Per-tenant credential isolation is non-negotiable.
This post covers the architecture for securing LLM API keys in multi-tenant environments: per-tenant isolation, encrypted vaults, virtual keys, rotation patterns, and dynamic secrets. For Fernet encryption of API keys, this post covers the multi-tenant architecture layer.
The Multi-Tenant API Key Problem
| Risk | What happens with shared keys | What happens with per-tenant keys |
|---|---|---|
| Key leak | All tenants exposed | Only one tenant affected |
| Cost overrun | One tenant's runaway loop bills to shared account | Per-tenant budget caps prevent overruns |
| Key revocation | All tenants lose access | Only affected tenant's key revoked |
| Audit trail | Cannot attribute usage to specific tenant | Full per-tenant usage attribution |
| Compliance | Cannot demonstrate tenant isolation | Per-tenant cryptographic isolation |
Per-Tenant Key Isolation Architecture
Each tenant's API keys are stored in a separate encrypted vault namespace. Access is controlled by attribute-based access control (ABAC) — an agent's IAM role tagged with its TenantId can only access secrets tagged with the same TenantId. A bug in the application layer that causes an agent to request another tenant's credentials will be rejected at the IAM level (Armalo AI 2026).
| Isolation layer | What it prevents |
|---|---|
| Separate vault namespaces | Cross-tenant key access at storage level |
| ABAC enforcement | Application bugs cannot access wrong tenant's keys |
| Per-tenant encryption keys | One tenant's key cannot decrypt another's stored keys |
| Scoped service containers | Keys don't persist in memory between tenant requests |
Virtual API Keys: The Gateway Pattern
Virtual API keys are short, scoped, revocable tokens that sit in front of your provider credentials. Instead of distributing provider API keys to every service, you issue virtual keys through a gateway.
| Virtual key attribute | What it controls |
|---|---|
key |
Bearer token — hashed at rest, shown once at creation |
models |
Model allowlist — 403 if key tries unauthorized model |
rpm / tpm |
Rate limits — 429 when exceeded |
monthly_budget_usd |
Hard cost cap — 429 when budget exceeded |
team |
Cost attribution tag |
router |
Per-key routing overrides |
The gateway holds the actual provider credentials in a secret manager. Virtual keys live in your database — scoped, observable, and revocable. A leaked virtual key is a five-second revocation, not a five-hour provider key rotation (Gateway-LLM 2026).
Hierarchical budget model
$10,000/month] --> Team[Team Budget
$2,000/month] Team --> User[User Budget
$500/month] User --> VKey[Virtual Key Budget
$200/month] VKey -->|429 when exceeded| Block[Request Blocked]
The constraint flows downward: a team cannot spend more than its organization allows, a user cannot exceed the team budget, and a virtual key cannot exceed the user's allocation. This decouples "what is the total cost?" from "who spent it and why?" (Tianpan 2026).
Key Encryption and Storage
| Storage method | Security level | Use case |
|---|---|---|
| Environment variables | Minimum | Development only |
| Cloud secrets manager (AWS/GCP/Azure) | Good | Production platform keys |
| HashiCorp Vault | Strong | Multi-tenant, dynamic secrets |
| HSM-backed vault | Strongest | Regulated industries |
| Client-side encrypted vault | Strong | Keys encrypted before reaching server |
For BYOK keys (tenant-provided API keys), use application-layer encryption with Fernet or AES-256-GCM before database storage. The database stores only ciphertext. Decrypt only at the moment of the LLM API call, and discard the plaintext from memory immediately after.
Dual-Key Rotation Pattern
Key rotation is where most teams fail. They know they should rotate but lack a mechanism that avoids downtime. The dual-key pattern solves this:
| Step | Action | What happens |
|---|---|---|
| 1 | Generate new key at provider | Both old and new keys are active |
| 2 | Deploy new key to vault | Application can use either key |
| 3 | Switch traffic to new key | All new requests use new key |
| 4 | Monitor for old key usage | Confirm no in-flight requests use old key |
| 5 | Revoke old key at provider | Old key disabled, rotation complete |
This enables zero-downtime rotation. The overlap window between steps 2 and 5 is typically seconds to minutes (Zylos 2026).
Rotation frequency
| Key type | Recommended rotation | Why |
|---|---|---|
| Platform provider keys | 90 days | Standard security practice |
| High-sensitivity keys | 24 hours or dynamic | Billing-enabled, production |
| Virtual keys | Instant revocation (no rotation) | Just revoke and reissue |
| Tenant BYOK keys | Tenant-controlled | Tenant manages their own rotation |
| Dynamic secrets | Per-request (no rotation) | Auto-expiring, generated on demand |
Dynamic Secrets: The End of Static Keys
HashiCorp Vault's OpenAI dynamic secrets plugin generates fresh LLM API credentials on demand with automatic expiration. Instead of distributing a static API key to every agent instance, Vault generates scoped credentials that expire when the task is done.
| Aspect | Static keys | Dynamic secrets |
|---|---|---|
| Lifetime | Until manually rotated | Minutes to hours, auto-expiring |
| Scope | Full key permissions | Scoped to specific task |
| Rotation | Manual, disruptive | Automatic, transparent |
| Breach impact | Key valid until revoked | Credentials expire automatically |
| SPIFFE integration | N/A | Workload identity, no stored secrets |
For AI agents, SPIFFE/SPIRE solves the "secret zero" problem — agents authenticate using cryptographic workload identity rather than stored secrets. If an agent is terminated, its identity expires with it. Prompt injection attacks that attempt to exfiltrate credentials are ineffective because there are no credentials in the agent's environment to steal (Zylos 2026).
Best Practices Summary
| Practice | Why it matters |
|---|---|
| Never share API keys across tenants | Shared keys = shared breach |
| Use per-tenant encrypted vaults | Cryptographic isolation at storage level |
| Implement virtual keys with budgets | Cost control and blast radius limitation |
| Use dual-key rotation | Zero-downtime key changes |
| Encrypt keys at application layer | Protects from DBA and database breach |
| Never log decrypted key values | Logs are a common breach vector |
| Use separate keys per environment | Dev key leak doesn't affect production |
| Scope keys to minimum permissions | Limit damage if key is compromised |
| Plan for compromise | Have a revocation runbook, practice it |
| Consider dynamic secrets for high-sensitivity | Auto-expiring credentials eliminate rotation |
FAQ
How do you secure LLM API keys in a multi-tenant environment?
Use per-tenant key isolation: each tenant's API keys are encrypted separately and stored in a credential vault. Never share API keys across tenants. Use virtual keys with per-tenant budgets and rate limits. Encrypt keys at the application layer with Fernet or AES-256-GCM.
What are virtual API keys for LLMs?
Virtual API keys are short, scoped, revocable tokens your application uses to call an LLM gateway. They sit in front of your provider credentials. A leaked virtual key is a five-second revocation, not a five-hour provider key rotation. Each carries rate limits, model allowlists, and budget caps.
How often should you rotate LLM API keys?
Platform-level provider keys should rotate every 90 days using a dual-key rotation pattern. High-sensitivity keys (billing-enabled, production) should rotate every 24 hours or be replaced with dynamic secrets from HashiCorp Vault. Virtual keys can be revoked instantly without rotation.
Should you share API keys across tenants in a multi-tenant AI platform?
No. Shared API keys across tenants are a critical security failure. A breach of the shared key exposes all tenants. Use per-tenant credential vaults with cryptographic isolation enforced at the storage layer, not just application-layer policy.
What is the dual-key rotation pattern for LLM API keys?
Maintain two active API keys at the provider. When rotating, deploy the new key while the old key is still valid. Switch traffic to the new key. After confirming all traffic uses the new key, revoke the old key. This enables zero-downtime rotation.
Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →