Private LLM On-Premises with Ollama: Enterprise Deployment Guide

TL;DR — Ollama is an open-source tool for running LLMs locally — Llama 3, Mistral, Qwen, Phi. Use it on-premises for data sovereignty (GDPR, HIPAA, air-gapped), zero per-token costs, and full control over model behavior. GPU requirements: 8GB VRAM for 7-8B models, 24GB for 33-35B, 48GB for 70B. Apple Silicon M2/M3 with 32GB unified memory runs 70B models. Ollama exposes an OpenAI-compatible API — your application code works with both Ollama and cloud APIs. Ollama has no built-in auth or TLS — secure with Nginx reverse proxy (Bearer token auth, rate limiting, TLS), bind to localhost, Docker isolation, and firewall rules. Cost: ~$2,000-4,000/month infrastructure for unlimited queries vs $2,500/month for 500K GPT-4o queries — break-even at ~500K queries/month. For 5M queries: $4,000/month self-hosted vs $25,000/month cloud (6x cheaper). Hybrid approach: LiteLLM routes 80% to Ollama, 20% to cloud APIs. Deploy with Docker Compose as part of self-hosted AI stack.

Ollama makes running LLMs locally as simple as ollama pull llama3. But production deployment requires security hardening, GPU planning, monitoring, and integration with your RAG pipeline. This guide covers everything from model selection to production deployment.

Ollama Architecture

Model Selection and GPU Requirements

Apple Silicon: M2/M3 with 32GB unified memory runs Llama 3 70B (no NVIDIA GPU needed). M2/M3 with 16GB runs 8B models. Metal acceleration is automatic.

Production Security Hardening

Ollama has no built-in authentication, TLS, or rate limiting. Never expose port 11434 to the internet. Always use a reverse proxy.

Nginx Reverse Proxy with Auth and Rate Limiting

# nginx.conf — Ollama production security
upstream ollama_backend {
    server 127.0.0.1:11434;
    keepalive 32;
}

# Rate limiting: 10 requests/second per IP
limit_req_zone $binary_remote_addr zone=ollama_limit:10m rate=10r/s;

server {
    listen 443 ssl http2;
    server_name llm.internal.company.com;

    # TLS
    ssl_certificate /etc/nginx/certs/fullchain.pem;
    ssl_certificate_key /etc/nginx/certs/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;

    # Auth: Bearer token
    auth_request /auth;

    # Proxy to Ollama (localhost only)
    location / {
        limit_req zone=ollama_limit burst=20 nodelay;

        proxy_pass http://ollama_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_buffering off;  # Streaming responses
        proxy_read_timeout 300s;  # Long generation time
    }

    # Auth endpoint (validate Bearer token)
    location = /auth {
        internal;
        proxy_pass http://127.0.0.1:8001/validate;
        proxy_pass_request_body off;
        proxy_set_header Content-Length "";
        proxy_set_header X-Original-URI $request_uri;
    }
}

Docker Deployment with Isolation

# docker-compose.yml — Ollama production
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "127.0.0.1:11434:11434"  # Localhost only!
    volumes:
      - ollama_data:/root/.ollama
    restart: always
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
        limits:
          memory: 48G
    read_only: true  # Read-only filesystem
    tmpfs:
      - /tmp
    security_opt:
      - no-new-privileges:true
    cap_drop:
      - ALL
    healthcheck:
      test: ["CMD-SHELL", "curl -sf http://localhost:11434/api/tags || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3

  nginx:
    image: nginx:alpine
    ports:
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./certs:/etc/nginx/certs:ro
    depends_on:
      - ollama
    restart: always

volumes:
  ollama_data:

Cost Comparison: Ollama vs Cloud APIs

Ollama costs include server + GPU + electricity. Cloud costs = tokens only. At 500K queries/month, self-hosting breaks even. Above that, savings compound rapidly.

Monitoring

# prometheus.yml — Ollama metrics
scrape_configs:
  - job_name: 'ollama'
    static_configs:
      - targets: ['localhost:11434']
    metrics_path: /metrics
    scrape_interval: 15s

  - job_name: 'nginx'
    static_configs:
      - targets: ['nginx:9113']

Hybrid Deployment with LiteLLM

# Route between Ollama (self-hosted) and cloud APIs
from litellm import Router

router = Router(model_list=[
    # Primary: self-hosted Llama 3 70B (free, data sovereign)
    {
        "model_name": "primary",
        "litellm_params": {
            "model": "ollama/llama3:70b",
            "api_base": "https://llm.internal.company.com",
            "api_key": os.environ["OLLAMA_API_KEY"],
        },
    },
    # Fallback: Claude Sonnet (for complex reasoning)
    {
        "model_name": "fallback",
        "litellm_params": {
            "model": "anthropic/claude-3-5-sonnet",
            "api_key": os.environ["ANTHROPIC_API_KEY"],
        },
    },
],
fallbacks=[{"primary": ["fallback"]}],
)

# 80% of queries go to Ollama (free, on-premises)
# 20% failover to Claude (complex reasoning, cloud)
response = router.completion(
    model="primary",
    messages=[{"role": "user", "content": "Analyze this contract"}],
)

Private LLM Ollama Deployment Checklist

• [ ] Choose model: Llama 3 8B (prototyping) or 70B (production)
• [ ] Verify GPU VRAM: 8GB for 8B, 40GB for 70B, or Apple Silicon 32GB
• [ ] Install Ollama: curl -fsSL https://ollama.com/install.sh | sh
• [ ] Pull model: ollama pull llama3:70b
• [ ] Bind Ollama to localhost: OLLAMA_HOST=127.0.0.1:11434
• [ ] Set up Nginx reverse proxy with TLS (Let's Encrypt or internal CA)
• [ ] Configure Bearer token authentication in Nginx
• [ ] Set rate limiting: 10 req/s per IP, burst 20
• [ ] Configure Docker with read-only filesystem and resource limits
• [ ] Drop all Linux capabilities (cap_drop: ALL)
• [ ] Set no-new-privileges: true in Docker security options
• [ ] Configure firewall: only allow proxy IP to access port 11434
• [ ] Set up Docker Compose deployment with GPU passthrough
• [ ] Configure Prometheus metrics scraping
• [ ] Set up Grafana dashboards: latency, GPU memory, request rate
• [ ] Configure alerts: p99 latency, GPU memory, error rate
• [ ] Pin Ollama version (don't auto-update in production)
• [ ] Set up model storage on persistent volume
• [ ] Use LiteLLM as gateway for hybrid routing
• [ ] Route 80% to Ollama, 20% to cloud APIs for complex queries
• [ ] Integrate with FastAPI + PostgreSQL stack
• [ ] Use pgvector for RAG retrieval
• [ ] Add semantic answer caching to reduce GPU load
• [ ] Implement query rewriting before LLM call
• [ ] Set up cross-encoder reranking for precision
• [ ] Configure hallucination prevention: confidence threshold
• [ ] Evaluate TCO vs cloud APIs
• [ ] Consider self-hosted AI architecture for full stack
• [ ] Test failover: kill Ollama, verify LiteLLM routes to cloud API
• [ ] Set up audit logging for EU AI Act compliance
• [ ] Consider zero-trust security between services
• [ ] Document model versions and quantization settings
• [ ] Plan GPU upgrades: monitor VRAM usage and query latency trends

FAQ

What is Ollama and why use it for on-premises LLM?

Ollama is an open-source tool for running large language models locally. It supports Llama 3, Mistral, Qwen, Phi, and other open-weight models. You use it on-premises when: data cannot leave your network (GDPR, HIPAA, air-gapped), you want to eliminate per-token API costs, you need full control over model behavior, or you want to avoid vendor lock-in. Ollama runs on any machine with 8GB+ RAM (CPU) or a GPU (NVIDIA, Apple Silicon). It exposes an OpenAI-compatible API, so your application code works with both Ollama and cloud APIs without changes.

What GPU do I need for Ollama?

GPU requirements depend on model size. 7-8B models (Llama 3 8B): 8GB VRAM (RTX 4060, T4). 13-14B models: 12GB VRAM (RTX 4070, A10G). 33-35B models: 24GB VRAM (RTX 4090, A10). 70B models: 48GB VRAM (2x A6000, A100 40GB with quantization). Apple Silicon: M2/M3 with 32GB unified memory runs 70B models. For production, use NVIDIA GPUs with CUDA. For prototyping, Apple Silicon or CPU-only works for 7-8B models. Quantization (Q4_K_M default) reduces memory by 4x with minimal quality loss.

How do you secure Ollama for production?

Ollama has no built-in authentication or TLS. Secure it with: (1) Reverse proxy (Nginx/Caddy) for TLS termination and Bearer token authentication, (2) Rate limiting in Nginx to prevent GPU DoS (limit_req_zone with 10 requests/second), (3) Network isolation — bind Ollama to localhost (127.0.0.1) and only expose through proxy, (4) Docker isolation with read-only filesystem and resource limits, (5) API key management with rotating tokens, (6) Audit logging of all requests, (7) Firewall rules restricting access to known IPs. Never expose Ollama's raw API (port 11434) to the internet.

How much does a private LLM with Ollama cost?

Ollama itself is free (MIT license). Costs are infrastructure: a single server with 1x NVIDIA A100 (80GB) costs ~$10,000-12,000 one-time or ~$2,000-4,000/month cloud. This runs Llama 3 70B for unlimited queries. Compare to OpenAI GPT-4o at $2.50/M tokens: at 500K queries/month (avg 2K tokens each), that's $2,500/month. Break-even at ~500K queries/month. For 5M queries/month, self-hosted costs ~$4,000/month vs $25,000/month for GPT-4o — 6x cheaper. Add electricity (~$100-200/month for a GPU server) and a DBA/devops engineer for maintenance.

Can Ollama replace cloud LLM APIs entirely?

For most enterprise use cases, yes — with caveats. Llama 3 70B matches GPT-4o on most benchmarks for RAG, summarization, and Q&A. It falls behind on: complex multi-step reasoning (GPT-5, Claude 3.5 Sonnet), code generation (Claude), and multimodal tasks (GPT-4o vision). Use a hybrid approach: Ollama for 80% of queries (high-volume, simple), cloud APIs for 20% (complex, frontier). Use LiteLLM as a gateway to route between Ollama and cloud APIs. This gives you data sovereignty for most queries while maintaining access to frontier models when needed.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →