Ollama Docker Production: GPU Passthrough, Persistent Volumes, and Multi-Container Deployment

By Fadi Updated Feb 26, 2026 12 min read Local LLMs

TL;DR — Ollama Docker production: GPU, persistence, scaling. OneUptime: "GPU passthrough, persistent model storage, multi-container deployment patterns." MarkAICode: "Reproducible, scalable AI inference with clean dependency management. GPU support, Docker Compose, production best practices." MarkAICode HowTo: "Pin version (not :latest). ollama/ollama:0.5.7 includes CUDA support out of the box." ML Journey: "NVIDIA GPU passthrough with Container Toolkit, AMD ROCm, full Docker Compose stack with Open WebUI, env vars for parallel requests and keep-alive." SitePoint: "Redis cache-aside: SHA-256 hash of prompt+model+temperature. Reduces latency and GPU load." Learn more with Ollama tutorial, VPS setup, REST API, and model selection.

MarkAICode frames the value: "Running Ollama — the leading local LLM runtime — inside Docker containers gives you reproducible, scalable AI inference with clean dependency management."

MarkAICode HowTo adds the production principle: "Pin the version (not :latest) to ensure production environment doesn't change unexpectedly. The ollama/ollama:0.5.7 tag includes CUDA support out of the box — no separate CUDA base image needed."

Docker Production Architecture

flowchart TD subgraph Docker["Docker Container"] Ollama["Ollama Server
port 11434
CUDA support built-in
version pinned"] Models["Model Storage
/root/.ollama
persistent volume
survives rebuilds"] Env["Environment
OLLAMA_HOST
OLLAMA_NUM_PARALLEL
OLLAMA_MAX_LOADED_MODELS
OLLAMA_KEEP_ALIVE"] end subgraph GPU["GPU Passthrough"] NVIDIA["NVIDIA Container Toolkit
--gpus all
driver: nvidia
count: all
capabilities: gpu"] AMD["AMD ROCm
--device /dev/kfd
--device /dev/dri"] CPU["CPU Only
no GPU config
slower inference"] end subgraph Compose["Docker Compose Stack"] OllamaSvc["ollama service
image: ollama/ollama:0.5.7
volumes: ollama_data
restart: unless-stopped
healthcheck"] WebUI["open-webui service
image: ghcr.io/open-webui
OLLAMA_BASE_URL=http://ollama:11434
port 3000
depends_on: ollama"] Redis["redis service
cache-aside pattern
SHA-256 prompt hash
reduces GPU load"] Network["Docker Network
connects services
internal communication"] end subgraph Production["Production Config"] Version["Version Pinning
ollama/ollama:0.5.7
not :latest
predictable environment"] Restart["Restart Policy
unless-stopped
survives host reboots
auto-recovery"] Health["Healthcheck
curl localhost:11434
interval: 30s
timeout: 10s
retries: 3"] Backup["Backup
docker run --rm
tar czf backup
volume data"] end NVIDIA --> Ollama AMD --> Ollama CPU --> Ollama Ollama --> Models Ollama --> Env OllamaSvc --> Ollama WebUI --> OllamaSvc Redis --> OllamaSvc Network --> OllamaSvc Network --> WebUI Version --> OllamaSvc Restart --> OllamaSvc Health --> OllamaSvc Backup --> Models style Docker fill:#4169E1,color:#fff style GPU fill:#39FF14,color:#000 style Compose fill:#2D1B69,color:#fff style Production fill:#FF6B6B,color:#fff

Deployment Configuration Comparison

Config	Best For	GPU	Persistence	Complexity
Basic docker run	Quick testing	Optional	Volume	Low
Docker-compose CPU	CPU production	No	Named volume	Medium
Docker-compose GPU	GPU production	NVIDIA/AMD	Named volume	Medium
Compose + Open WebUI	Full stack	Optional	Shared volumes	Medium
Compose + Redis	High traffic	Recommended	Shared volumes	High
Multi-instance + LB	Scale	Required per instance	Shared/NFS	High

Implementation

from dataclasses import dataclass
from typing import Optional
from enum import Enum

class DockerSetup(Enum):
    BASIC = "basic"
    GPU = "gpu"
    COMPOSE_FULL = "compose_full"
    SCALED = "scaled"

@dataclass
class OllamaDockerProductionGuide:
    """Ollama Docker production implementation guide."""

    def get_basic_docker_run(self) -> str:
        """Basic Docker run command."""
        return (
            "# === BASIC (CPU) ===\n"
            "docker run -d \\\n"
            "  -v ollama:/root/.ollama \\\n"
            "  -p 11434:11434 \\\n"
            "  --name ollama \\\n"
            "  --restart unless-stopped \\\n"
            "  ollama/ollama:0.5.7\n"
            "\n"
            "# === GPU (NVIDIA) ===\n"
            "# Requires nvidia-container-toolkit\n"
            "docker run -d \\\n"
            "  --gpus all \\\n"
            "  -v ollama:/root/.ollama \\\n"
            "  -p 11434:11434 \\\n"
            "  --name ollama \\\n"
            "  --restart unless-stopped \\\n"
            "  ollama/ollama:0.5.7\n"
            "\n"
            "# === PULL MODEL IN CONTAINER ===\n"
            "docker exec -it ollama \\\n"
            "  ollama pull llama3.2\n"
            "\n"
            "# === RUN CHAT IN CONTAINER ===\n"
            "docker exec -it ollama \\\n"
            "  ollama run llama3.2\n"
            "\n"
            "# === CHECK LOGS ===\n"
            "docker logs ollama -f\n"
            "\n"
            "# === HEALTH CHECK ===\n"
            "curl http://localhost:11434/\n"
            "# 'Ollama is running'"
        )

    def get_docker_compose_gpu(self) -> str:
        """Docker Compose with GPU support."""
        return (
            "# docker-compose.yml\n"
            "version: '3.8'\n"
            "services:\n"
            "  ollama:\n"
            "    image: ollama/ollama:0.5.7\n"
            "    container_name: ollama\n"
            "    ports:\n"
            "      - '11434:11434'\n"
            "    volumes:\n"
            "      - ollama_data:/root/.ollama\n"
            "    environment:\n"
            "      - OLLAMA_HOST=0.0.0.0:11434\n"
            "      - OLLAMA_NUM_PARALLEL=4\n"
            "      - OLLAMA_MAX_LOADED_MODELS=2\n"
            "      - OLLAMA_KEEP_ALIVE=10m\n"
            "    restart: unless-stopped\n"
            "    healthcheck:\n"
            "      test: ['CMD-SHELL',\n"
            "        'curl -f '\n"
            "        'http://localhost:11434/ '\n"
            "        '|| exit 1']\n"
            "      interval: 30s\n"
            "      timeout: 10s\n"
            "      retries: 3\n"
            "    deploy:\n"
            "      resources:\n"
            "        reservations:\n"
            "          devices:\n"
            "            - driver: nvidia\n"
            "              count: all\n"
            "              capabilities:\n"
            "                - gpu\n"
            "\n"
            "volumes:\n"
            "  ollama_data:\n"
            "\n"
            "# === RUN ===\n"
            "# docker compose up -d\n"
            "# docker compose logs -f\n"
            "# docker compose down"
        )

    def get_compose_with_webui(self) -> str:
        """Docker Compose with Open WebUI."""
        return (
            "# docker-compose.yml\n"
            "# Full stack: Ollama + Open WebUI\n"
            "version: '3.8'\n"
            "services:\n"
            "  ollama:\n"
            "    image: ollama/ollama:0.5.7\n"
            "    container_name: ollama\n"
            "    volumes:\n"
            "      - ollama_data:/root/.ollama\n"
            "    environment:\n"
            "      - OLLAMA_HOST=0.0.0.0:11434\n"
            "      - OLLAMA_NUM_PARALLEL=4\n"
            "    restart: unless-stopped\n"
            "    healthcheck:\n"
            "      test: ['CMD-SHELL',\n"
            "        'curl -f '\n"
            "        'http://localhost:11434/ '\n"
            "        '|| exit 1']\n"
            "      interval: 30s\n"
            "      timeout: 10s\n"
            "      retries: 3\n"
            "    deploy:\n"
            "      resources:\n"
            "        reservations:\n"
            "          devices:\n"
            "            - driver: nvidia\n"
            "              count: all\n"
            "              capabilities: [gpu]\n"
            "  \n"
            "  open-webui:\n"
            "    image: ghcr.io/open-webui\n"
            "      /open-webui:main\n"
            "    container_name: open-webui\n"
            "    ports:\n"
            "      - '3000:8080'\n"
            "    environment:\n"
            "      - OLLAMA_BASE_URL=\n"
            "        http://ollama:11434\n"
            "    volumes:\n"
            "      - webui_data:/app/backend\n"
            "        /data\n"
            "    depends_on:\n"
            "      ollama:\n"
            "        condition: service_healthy\n"
            "    restart: unless-stopped\n"
            "\n"
            "volumes:\n"
            "  ollama_data:\n"
            "  webui_data:\n"
            "\n"
            "# Access:\n"
            "# Ollama API: localhost:11434\n"
            "# Open WebUI: localhost:3000"
        )

    def get_compose_with_redis(self) -> str:
        """Docker Compose with Redis caching."""
        return (
            "# docker-compose.yml\n"
            "# Production: Ollama + Redis cache\n"
            "version: '3.8'\n"
            "services:\n"
            "  ollama:\n"
            "    image: ollama/ollama:0.5.7\n"
            "    volumes:\n"
            "      - ollama_data:/root/.ollama\n"
            "    environment:\n"
            "      - OLLAMA_NUM_PARALLEL=4\n"
            "      - OLLAMA_KEEP_ALIVE=10m\n"
            "    restart: unless-stopped\n"
            "    deploy:\n"
            "      resources:\n"
            "        reservations:\n"
            "          devices:\n"
            "            - driver: nvidia\n"
            "              count: all\n"
            "              capabilities: [gpu]\n"
            "  \n"
            "  redis:\n"
            "    image: redis:7-alpine\n"
            "    container_name: redis\n"
            "    ports:\n"
            "      - '6379:6379'\n"
            "    volumes:\n"
            "      - redis_data:/data\n"
            "    restart: unless-stopped\n"
            "    command: redis-server\n"
            "      --maxmemory 512mb\n"
            "      --maxmemory-policy allkeys-lru\n"
            "\n"
            "volumes:\n"
            "  ollama_data:\n"
            "  redis_data:\n"
            "\n"
            "# Cache-aside pattern in app:\n"
            "# import redis, hashlib, json\n"
            "# r = redis.Redis(\n"
            "#   host='localhost', port=6379)\n"
            "# \n"
            "# def cached_chat(model,\n"
            "#   messages, temp=0.7):\n"
            "#   key = hashlib.sha256(\n"
            "#     f'{model}:{messages}'\n"
            "#     f':{temp}'.encode()\n"
            "#   ).hexdigest()\n"
            "#   cached = r.get(key)\n"
            "#   if cached:\n"
            "#     return json.loads(cached)\n"
            "#   result = call_ollama(\n"
            "#     model, messages)\n"
            "#   r.setex(key, 3600,\n"
            "#     json.dumps(result))\n"
            "#   return result"
        )

    def get_nvidia_toolkit_setup(self) -> str:
        """NVIDIA Container Toolkit setup."""
        return (
            "# === INSTALL NVIDIA CONTAINER\n"
            "#   TOOLKIT ===\n"
            "# Ubuntu/Debian:\n"
            "curl -fsSL https://nvidia\n"
            "  .github.io/libnvidia-\n"
            "  container/gpgkey | \\\n"
            "  sudo gpg --dearmor -o \\\n"
            "  /usr/share/keyrings/\n"
            "  nvidia-container-toolkit\n"
            "  -keyring.gpg\n"
            "\n"
            "curl -s -L https://nvidia\n"
            "  .github.io/libnvidia-\n"
            "  container/stable/deb/\n"
            "  nvidia-container-toolkit\n"
            "  .list | \\\n"
            "  sed 's#deb://deb [signed-by=/\n"
            "  usr/share/keyrings/\n"
            "  nvidia-container-toolkit\n"
            "  -keyring.gpg]#' | \\\n"
            "  sudo tee /etc/apt/sources\n"
            "  .list.d/nvidia-container\n"
            "  -toolkit.list\n"
            "\n"
            "sudo apt update\n"
            "sudo apt install -y \\\n"
            "  nvidia-container-toolkit\n"
            "\n"
            "# Configure Docker\n"
            "sudo nvidia-ctk runtime \\\n"
            "  configure --runtime=docker\n"
            "sudo systemctl restart docker\n"
            "\n"
            "# Verify GPU access\n"
            "docker run --rm --gpus all \\\n"
            "  nvidia/cuda:12.4-base-\n"
            "  ubuntu22.04 nvidia-smi\n"
            "\n"
            "# Should show GPU info"
        )

    def get_production_tips(self) -> dict:
        """Production deployment tips."""
        return {
            "version_pinning": (
                "Pin image tag (e.g., "
                "ollama/ollama:0.5.7), "
                "not :latest. Ensures "
                "predictable environment. "
                ":latest can change "
                "unexpectedly."
            ),
            "persistent_volumes": (
                "Mount named volume to "
                "/root/.ollama. Models "
                "persist across container "
                "rebuilds. Avoid re-"
                "downloading multi-GB "
                "models."
            ),
            "restart_policy": (
                "restart: unless-stopped "
                "for auto-recovery. "
                "Survives host reboots. "
                "Container restarts on "
                "crash."
            ),
            "healthcheck": (
                "Docker HEALTHCHECK with "
                "curl to localhost:11434. "
                "interval: 30s, timeout: "
                "10s, retries: 3. Docker "
                "marks unhealthy after "
                "failures."
            ),
            "env_vars": (
                "OLLAMA_NUM_PARALLEL: "
                "concurrent requests. "
                "OLLAMA_MAX_LOADED_MODELS: "
                "models in memory. "
                "OLLAMA_KEEP_ALIVE: "
                "retention after use."
            ),
            "redis_cache": (
                "Cache-aside pattern: "
                "check Redis before "
                "Ollama. SHA-256 hash of "
                "prompt+model+temperature. "
                "Reduces latency and GPU "
                "load."
            ),
            "backup": (
                "docker run --rm -v "
                "ollama_data:/data -v "
                "$(pwd):/backup alpine "
                "tar czf /backup/ollama"
                ".tar.gz /data"
            ),
            "logging": (
                "docker logs ollama -f "
                "for real-time. Configure "
                "log driver for external "
                "logging (json-file, "
                "syslog, fluentd)."
            ),
            "networking": (
                "Docker network connects "
                "services. Internal "
                "communication via "
                "service names. Expose "
                "only needed ports."
            ),
        }

Ollama Docker Production Checklist

[ ] Pin image version: ollama/ollama:0.5.7 (not :latest) for predictable environment
[ ] CUDA support built into ollama/ollama image — no separate CUDA base needed
[ ] Persistent volume: -v ollama_data:/root/.ollama for model storage across rebuilds
[ ] Models persist across container restarts and rebuilds — avoid re-downloading multi-GB models
[ ] restart: unless-stopped for auto-recovery and host reboot survival
[ ] Healthcheck: curl -f http://localhost:11434/ || exit 1, interval 30s, timeout 10s, retries 3
[ ] GPU passthrough NVIDIA: --gpus all in docker run, deploy.resources.reservations.devices in compose
[ ] GPU: install nvidia-container-toolkit, configure with nvidia-ctk runtime configure --runtime=docker
[ ] GPU: verify with docker run --rm --gpus all nvidia/cuda:12.4-base-ubuntu22.04 nvidia-smi
[ ] GPU docker-compose: driver: nvidia, count: all, capabilities: [gpu]
[ ] AMD GPU: --device /dev/kfd --device /dev/dri for ROCm
[ ] CPU only: omit GPU config, slower inference but works without GPU
[ ] Environment: OLLAMA_HOST=0.0.0.0:11434 for remote access
[ ] Environment: OLLAMA_NUM_PARALLEL for concurrent request limit
[ ] Environment: OLLAMA_MAX_LOADED_MODELS for concurrent models in memory
[ ] Environment: OLLAMA_KEEP_ALIVE for model retention after last use (default 5m)
[ ] Pull models in container: docker exec -it ollama ollama pull llama3.2
[ ] Run chat in container: docker exec -it ollama ollama run llama3.2
[ ] Open WebUI: separate service with OLLAMA_BASE_URL=http://ollama:11434, port 3000
[ ] Open WebUI: depends_on ollama with condition: service_healthy
[ ] Open WebUI: persistent volume for webui_data:/app/backend/data
[ ] Docker network: connects services, internal communication via service names
[ ] Redis cache: cache-aside pattern, SHA-256 hash of prompt+model+temperature
[ ] Redis: maxmemory 512mb, maxmemory-policy allkeys-lru, setex with TTL
[ ] Redis: reduces latency and GPU load by avoiding redundant inference
[ ] Backup: docker run --rm -v ollama_data:/data -v $(pwd):/backup alpine tar czf /backup/ollama.tar.gz /data
[ ] Logging: docker logs ollama -f, configure log driver for external logging
[ ] Multi-instance: load balancer (nginx/traefik) for multiple Ollama containers
[ ] Monitor: GPU usage (nvidia-smi), memory, tokens/sec, container health
[ ] docker compose up -d to start, docker compose logs -f to monitor, docker compose down to stop
[ ] Read Ollama tutorial for basics
[ ] Read VPS setup for cloud deployment
[ ] Read REST API for endpoint reference
[ ] Read model selection for choosing models
[ ] Test: container starts and serves API on port 11434
[ ] Test: GPU passthrough works (nvidia-smi shows GPU inside container)
[ ] Test: persistent volume retains models across container rebuild
[ ] Test: healthcheck correctly reports container health
[ ] Test: restart policy auto-recovers container after crash
[ ] Test: Open WebUI connects to Ollama via Docker network
[ ] Test: Redis cache returns cached response for repeated prompts
[ ] Test: backup and restore volume data successfully
[ ] Test: version-pinned image produces consistent behavior
[ ] Document image version, volume config, GPU setup, env vars, healthcheck, network topology

FAQ

How do you deploy Ollama in Docker for production?

Use docker-compose with persistent volumes, GPU passthrough, restart policy, and healthcheck. OneUptime: "Comprehensive guide to running Ollama in Docker containers with GPU passthrough, persistent model storage, and multi-container deployment patterns." MarkAICode Integration: "Complete guide to running Ollama in Docker containers with GPU support, Docker Compose, error fixes, and production best practices for LLM inference at scale. Running Ollama inside Docker containers gives you reproducible, scalable AI inference with clean dependency management." MarkAICode HowTo: "Production setup in 4 steps. Pin the version (not :latest) to ensure production environment does not change unexpectedly. ollama/ollama:0.5.7 tag includes CUDA support out of the box." Deploy: (1) docker-compose.yml with ollama service. (2) Persistent volume for /root/.ollama. (3) GPU: deploy.resources.reservations.devices with nvidia driver. (4) restart: unless-stopped. (5) healthcheck. (6) Pin version tag.

How do you configure GPU passthrough for Ollama in Docker?

Use NVIDIA Container Toolkit with --gpus all or docker-compose GPU device config. ML Journey: "Complete guide: CPU-only and NVIDIA GPU passthrough with NVIDIA Container Toolkit, AMD ROCm setup, pulling models into running container, full Docker Compose stack combining Ollama with Open WebUI." MarkAICode Integration: "GPU support, Docker Compose, production best practices for LLM inference at scale." MarkAICode HowTo: "ollama/ollama:0.5.7 tag includes CUDA support out of the box — no separate CUDA base image needed." GPU: (1) Install NVIDIA Container Toolkit: sudo apt install nvidia-container-toolkit. (2) Docker run: docker run --gpus all ollama/ollama. (3) Docker-compose: deploy.resources.reservations.devices with driver: nvidia, count: all, capabilities: [gpu]. (4) Verify: nvidia-smi inside container. (5) AMD: use --device /dev/kfd --device /dev/dri. (6) No GPU: omit GPU config, runs on CPU.

How do you persist Ollama models in Docker?

Mount a Docker volume to /root/.ollama for persistent model storage across container restarts. OneUptime: "Persistent model storage with multi-container deployment patterns." ML Journey: "Full Docker Compose stack with persistent volumes for model storage." MarkAICode Integration: "Reproducible, scalable AI inference with clean dependency management." Persistence: (1) Volume: -v ollama:/root/.ollama in docker run. (2) Docker-compose: volumes: - ollama_data:/root/.ollama. (3) Models persist across container rebuilds and restarts. (4) Avoid re-downloading multi-GB models. (5) Bind mount: -v /path/to/models:/root/.ollama for host directory. (6) Named volume: managed by Docker, survives container deletion. (7) Backup: docker run --rm -v ollama:/data -v $(pwd):/backup alpine tar czf /backup/ollama.tar.gz /data.

How do you set up Ollama with Open WebUI in Docker?

Use docker-compose with both ollama and open-webui services connected via Docker network. ML Journey: "Full Docker Compose stack combining Ollama with Open WebUI. Environment variables for parallel requests and model keep-alive." OneUptime: "Multi-container deployment patterns." Setup: (1) docker-compose.yml with two services: ollama and open-webui. (2) open-webui depends_on: ollama. (3) open-webui environment: OLLAMA_BASE_URL=http://ollama:11434. (4) Docker network connects services. (5) open-webui port: 3000. (6) ollama port: 11434 (internal only). (7) Open WebUI provides chat interface, model management, document upload. (8) Both share persistent volumes.

How do you scale Ollama Docker for production traffic?

Add Redis caching, load balancing, and environment variables for concurrent requests. SitePoint: "Cache-aside pattern: API gateway checks Redis before calling Ollama, writes responses back on cache misses. Cache key is SHA-256 hash of normalized prompt, model name, and temperature. Redis caching reduces latency and GPU load." MarkAICode Integration: "Production best practices for LLM inference at scale." Scaling: (1) Redis cache: cache responses by prompt hash to avoid redundant inference. (2) OLLAMA_NUM_PARALLEL: concurrent request limit (default 1). (3) OLLAMA_MAX_LOADED_MODELS: concurrent models in memory. (4) OLLAMA_KEEP_ALIVE: model retention after last use. (5) Load balancer: nginx or traefik for multiple Ollama instances. (6) Health check: Docker HEALTHCHECK to monitor. (7) Restart: unless-stopped for reliability. (8) Logs: Docker logs or external logging. (9) Monitor: GPU usage, memory, tokens/sec.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →