Multi-Tenant AI Architecture: SaaS Platform Isolation Patterns

TL;DR — Multi-tenant AI architecture serves multiple customers from one AI platform while maintaining strict data isolation. Fastio's design guide notes that "unlike standard SaaS apps where simple database security works, AI agents need deeper separation — they work with files, vector databases, and long-term memory." OWASP Top 10 for LLMs lists improper access control as a core risk. Isolation layers: vector database (per-tenant namespaces or shared with tenant_id filtering), LLM context (no cross-tenant data in prompts), agent memory (per-tenant session isolation), file storage (per-tenant containers), GPU (shared inference with rate limiting). Vector DB isolation must be deterministic — "Relying on LLMs for access control is an architectural anti-pattern." GPU sharing: one H100 serves 100+ tenants at $10.73/tenant/month. Multi-tenant RAG patterns: store-per-tenant (strongest isolation) vs shared with filtering (cost-efficient) vs hybrid. Integrate with company knowledge, zero-trust, platform monitoring, and white-label platform.

AGIX Technologies describes the challenge: "Multi-tenant AI systems serve 1000+ clients per instance with isolated LLM processing and secure data partitioning." Unlike standard SaaS where database-level row security suffices, AI platforms need isolation across vector databases, LLM context windows, agent memory, file storage, and GPU resources.

Truto's architecture guide states the critical principle: "Secure multi-tenant RAG requires deterministic data isolation at the vector database layer using namespaces and JWTs. Relying on LLMs for access control is an architectural anti-pattern."

Multi-Tenant AI Architecture

Isolation Layers

Vector Database Isolation Patterns

Implementation

class MultiTenantAIPlatform:
    """Multi-tenant AI SaaS platform with strict data isolation."""

    def __init__(self, config):
        self.vector_db = config.vector_db  # pgvector with RLS
        self.llm = config.llm  # Shared LLM inference
        self.memory = config.memory  # Per-tenant session memory
        self.storage = config.storage  # Per-tenant file storage
        self.audit = config.audit  # Per-tenant audit log
        self.billing = config.billing  # Per-tenant usage metering

    async def query(self, tenant_id: str, user_id: str, query: str,
                    jwt_token: str) -> dict:
        """Process a tenant-isolated AI query."""

        # 1. Verify JWT and extract tenant identity
        claims = self.verify_jwt(jwt_token)
        if claims["tenant_id"] != tenant_id:
            raise SecurityError("Tenant mismatch")

        # 2. Check rate limits and quotas
        if not await self.check_rate_limit(tenant_id):
            raise RateLimitError("Rate limit exceeded")

        if not await self.check_token_quota(tenant_id):
            raise QuotaError("Token quota exceeded")

        # 3. Retrieve from vector DB with tenant isolation
        # CRITICAL: Never retrieve without tenant_id filter
        context = await self.vector_db.search(
            query=query,
            filter={"tenant_id": tenant_id},  # Mandatory filter
            top_k=10,
        )

        # 4. Build prompt with ONLY tenant's data
        prompt = self.build_prompt(
            query=query,
            context=context,
            tenant_id=tenant_id,  # Ensure no cross-tenant data
        )

        # 5. Get per-tenant session memory
        session = await self.memory.get_session(tenant_id, user_id)

        # 6. Generate response
        response = await self.llm.generate(
            prompt=prompt,
            session_context=session,
        )

        # 7. Meter usage for billing
        await self.billing.record_usage(
            tenant_id=tenant_id,
            tokens_used=response.tokens_used,
            query_type="rag",
        )

        # 8. Audit log with tenant_id
        await self.audit.log(
            tenant_id=tenant_id,
            user_id=user_id,
            query=query,
            tokens_used=response.tokens_used,
            timestamp=datetime.utcnow(),
        )

        return response

PostgreSQL RLS for Tenant Isolation

-- Enable Row-Level Security on all tenant-scoped tables
ALTER TABLE documents ENABLE ROW LEVEL SECURITY;
ALTER TABLE embeddings ENABLE ROW LEVEL SECURITY;
ALTER TABLE agent_memory ENABLE ROW LEVEL SECURITY;
ALTER TABLE audit_logs ENABLE ROW LEVEL SECURITY;

-- Policy: users can only see their tenant's data
CREATE POLICY tenant_isolation ON documents
    USING (tenant_id = current_setting('app.tenant_id')::uuid);

CREATE POLICY tenant_isolation ON embeddings
    USING (tenant_id = current_setting('app.tenant_id')::uuid);

-- Application sets tenant_id per request
SET app.tenant_id = 'tenant-uuid-here';
-- Now all queries automatically filter by tenant_id
SELECT * FROM documents WHERE content LIKE '%policy%';
-- Only returns documents for the current tenant

Cost Model

Multi-Tenant AI Architecture Checklist

• [ ] Implement JWT-based authentication with tenant_id on every request
• [ ] Verify tenant_id at every API layer — never trust client-side tenant claims
• [ ] Enable PostgreSQL Row-Level Security (RLS) on all tenant-scoped tables
• [ ] Set app.tenant_id per request — automatic filtering on all queries
• [ ] Choose vector database isolation: store-per-tenant, shared+filter, or hybrid
• [ ] Implement mandatory tenant_id filter on every vector search query
• [ ] Never rely on LLMs for access control — enforce isolation deterministically
• [ ] Set up per-tenant agent memory — session isolation, no shared context
• [ ] Configure per-tenant file storage — isolated S3/GCS buckets
• [ ] Implement per-tenant rate limiting (requests per minute)
• [ ] Configure per-tenant token quotas (tokens per day)
• [ ] Set up GPU sharing with fair-use policies
• [ ] Implement per-tenant usage metering for billing
• [ ] Track: tokens used, API calls, storage, compute per tenant
• [ ] Set up per-tenant audit logging — tenant_id on every log entry
• [ ] Send audit logs to SIEM with tenant segmentation
• [ ] Implement RBAC with per-tenant roles (Admin, Editor, Viewer)
• [ ] Configure per-tenant model access (which models each tier can use)
• [ ] Set up per-tenant knowledge base isolation in company knowledge
• [ ] Apply zero-trust architecture between tenants
• [ ] Apply OWASP LLM Top 10 access control (ASI03)
• [ ] Implement prompt sanitization — no cross-tenant data in LLM context
• [ ] Test cross-tenant isolation: verify tenant A cannot access tenant B's data
• [ ] Penetration test: attempt cross-tenant access via API, vector DB, file storage
• [ ] Set up platform monitoring per tenant
• [ ] Track per-tenant metrics: latency, error rate, token usage, cost
• [ ] Implement per-tenant governance policies
• [ ] Configure per-tenant action approval rules
• [ ] Set up tenant onboarding: provision resources, configure isolation
• [ ] Set up tenant offboarding: data deletion, resource cleanup
• [ ] Consider white-label platform for resellers
• [ ] Consider BYOK for enterprise tenants
• [ ] Implement per-tenant cost tracking
• [ ] Consider self-hosted AI for data sovereignty
• [ ] Document isolation architecture for compliance audits
• [ ] Quarterly security audit: test all isolation layers
• [ ] Implement noisy neighbor prevention: resource quotas and fair scheduling
• [ ] Consider AI incident response for cross-tenant leaks

FAQ

What is multi-tenant AI architecture?

Multi-tenant AI architecture is a SaaS platform design where a single AI system serves multiple customers (tenants) while maintaining strict data isolation. Unlike standard SaaS where database-level security suffices, AI platforms need deeper separation across vector databases, LLM context, agent memory, and GPU resources. Key isolation layers: (1) Vector database — per-tenant namespaces or separate databases. (2) LLM context — no cross-tenant data in prompts. (3) Agent memory — per-tenant session isolation. (4) File storage — per-tenant containers. (5) GPU — shared inference with per-tenant rate limiting. The OWASP Top 10 for LLM Applications lists improper access control as a core risk — relying on LLMs for access control is an architectural anti-pattern.

How do you isolate vector databases in multi-tenant RAG?

Two main approaches: (1) Store-per-tenant — each tenant gets a dedicated vector database or namespace. Strongest isolation, higher cost, simpler security. Use for regulated industries or large tenants. (2) Multi-tenant store — all tenants share one vector database with tenant_id metadata filtering. Lower cost, more complex security. Use JWT-based filtering at query time to ensure tenant A never retrieves tenant B's vectors. Never rely on LLMs for access control — enforce isolation deterministically at the vector database layer using namespaces and metadata filters. Hybrid approach: shared database for small tenants, dedicated for enterprise tenants.

What is the GPU sharing model for multi-tenant LLM serving?

GPU sharing for multi-tenant LLM serving: (1) Shared inference — all tenants share one GPU cluster with per-tenant rate limiting and token quotas. Most cost-efficient. One H100 can serve 100+ tenants at 100k tokens/day each. (2) Dedicated GPU — enterprise tenants get dedicated GPU instances. Highest performance, highest cost. (3) Hybrid — shared GPU for inference, dedicated for fine-tuning. Cost analysis: $49/month SaaS plan with 100k tokens/day = $1.53 usage cost + $10.73 GPU cost share (at 100 tenants per H100) = $25.83 gross margin per customer. Implement per-tenant token quotas, rate limiting, and fair-use policies to prevent noisy neighbor problems.

How do you prevent cross-tenant data leaks in AI platforms?

Prevent cross-tenant data leaks with: (1) Deterministic vector database isolation — tenant_id metadata filtering on every query, not LLM-based access control. (2) JWT-based authentication — every API request carries tenant identity, verified at every layer. (3) Per-tenant agent memory — session isolation, no shared context between tenants. (4) Per-tenant file storage — isolated containers or buckets. (5) Prompt sanitization — ensure no tenant data leaks into another tenant's LLM context. (6) Audit logging — log every retrieval, generation, and action with tenant_id. (7) Penetration testing — regularly test for cross-tenant access. (8) Row-Level Security in databases — PostgreSQL RLS policies for tenant isolation. Never trust the LLM to enforce access control.

What are the trade-offs of store-per-tenant vs shared vector database?

Store-per-tenant: (Pros) Strongest isolation, simpler security model, independent scaling per tenant, easier compliance for regulated industries. (Cons) Higher cost (database per tenant), operational complexity at scale (1000+ databases), harder to search across tenants. Shared database with tenant filtering: (Pros) Lower cost, simpler operations, easier to manage at scale. (Cons) More complex security (must enforce filtering on every query), risk of filter bypass, noisy neighbor problems. Hybrid approach: shared database for small tenants (cost efficiency), dedicated database for enterprise tenants (isolation guarantee). Choose based on: tenant count, regulatory requirements, cost sensitivity, and per-tenant data volume.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →