Ollama Tutorial: Run LLMs Locally with OpenAI-Compatible API and Python SDK

By Fadi Updated Apr 26, 2026 14 min read Local LLMs

TL;DR — Ollama: run LLMs locally for free. AI/TLDR: "Free, open-source. One command: ollama run llama3.2. Downloads model, conversation never touches internet. OpenAI-compatible API at /v1/chat/completions — change one URL, no library changes." Pristren: "Run Llama 3.3, Mistral, Phi-3, Deepseek-R1 on your hardware for free. Install in one command, model downloads in 5 minutes, no API costs." SolutionGigs: "100+ open-source LLMs. 8GB RAM = 7B models, 16GB = 32B. OpenAI SDK compatible — chat, streaming, structured output, tool calling. No API key needed." TechInsider: "REST API at localhost:11434, OpenAI-compatible. Python integration, Docker deployment, performance optimization." Medium: "Apple Silicon: automatic Metal/MLX acceleration, 1.6-2x speedup. Two API surfaces: native /api/ and OpenAI /v1/." Learn more with AI agents, function calling, no framework, and debugging.

AI/TLDR provides the overview: "Ollama is a free, open-source tool that makes running a large language model on your own computer as simple as one terminal command. You type ollama run llama3.2, it downloads the model, and within a minute you are having a conversation that never touches the internet."

Pristren adds: "Ollama is a free, open source runtime that lets you run large language models on your own Mac, Linux, or Windows machine. Installation takes one command, the first model downloads and runs in under 5 minutes, and after that, every query you send stays on your hardware with no API costs."

Ollama Architecture

flowchart TD subgraph Install["Installation"] Download["Download
ollama.com (Mac/Windows)
curl install.sh (Linux)"] Service["Background Service
starts automatically
localhost:11434"] end subgraph Models["Model Management"] Pull["ollama pull llama3.2
download model (~2GB)"] Run["ollama run llama3.2
interactive chat"] List["ollama list
show downloaded models"] PS["ollama ps
show loaded in memory"] Show["ollama show llama3.2
model details"] RM["ollama rm llama3.2
delete model"] end subgraph Server["Ollama Server (localhost:11434)"] Native["Native API
/api/chat
/api/generate
/api/embeddings"] OpenAI["OpenAI-Compatible
/v1/chat/completions
same schema as OpenAI
streaming JSON"] Manage["Model Management
/api/pull
/api/show
/api/delete"] end subgraph Clients["Client Integration"] CLI["CLI
ollama run
interactive chat"] PyOpenAI["Python OpenAI SDK
base_url=localhost:11434/v1
api_key=ollama
all features work"] PyNative["Python Ollama SDK
pip install ollama
ollama.chat()"] Requests["HTTP Requests
POST /api/chat
POST /v1/chat/completions"] Docker["Docker
docker run ollama/ollama
port 11434"] end subgraph Hardware["Hardware"] Apple["Apple Silicon
automatic Metal/MLX
1.6-2x speedup
zero config"] GPU["NVIDIA GPU
CUDA acceleration
automatic detection"] CPU["CPU Only
works without GPU
slower inference"] end Download --> Service Service --> Server Pull --> Server Run --> CLI List --> Server PS --> Server Server --> Native Server --> OpenAI Server --> Manage CLI --> Server PyOpenAI --> OpenAI PyNative --> Native Requests --> Native Requests --> OpenAI Docker --> Server Server --> Hardware Apple --> Server GPU --> Server CPU --> Server style Install fill:#39FF14,color:#000 style Server fill:#4169E1,color:#fff style OpenAI fill:#4169E1,color:#fff

Model Comparison

Model	Size	RAM Needed	Best For	Pull Command
llama3.2:3b	~2GB	8GB	General purpose, lightweight	`ollama pull llama3.2`
mistral:7b	~4.1GB	16GB	Instruction following	`ollama pull mistral`
qwen2.5-coder:7b	~4.7GB	16GB	Coding tasks	`ollama pull qwen2.5-coder`
deepseek-r1:8b	~4.9GB	16GB	Reasoning, chain-of-thought	`ollama pull deepseek-r1`
phi4:14b	~9.1GB	32GB	Microsoft 14B, versatile	`ollama pull phi4`
nomic-embed-text	~274MB	4GB	Embeddings for RAG	`ollama pull nomic-embed-text`

Implementation

from dataclasses import dataclass
from typing import Optional
from enum import Enum

class OllamaAPI(Enum):
    NATIVE = "native"
    OPENAI_COMPATIBLE = "openai_compatible"
    PYTHON_SDK = "python_sdk"

@dataclass
class OllamaTutorialGuide:
    """Ollama tutorial implementation guide."""

    def get_install_and_run(self) -> str:
        """Install Ollama and run first model."""
        return (
            "# === INSTALL ===\n"
            "# Mac/Windows:\n"
            "#   Download from ollama.com\n"
            "# Linux:\n"
            "#   curl -fsSL https://ollama.com/\n"
            "#     install.sh | sh\n"
            "\n"
            "# === PULL MODEL ===\n"
            "# Download without starting chat\n"
            "$ ollama pull llama3.2\n"
            "# Downloading: 100% | 2.0GB\n"
            "\n"
            "# === RUN INTERACTIVE ===\n"
            "$ ollama run llama3.2\n"
            ">>> Hello, what can you do?\n"
            "I can help with writing,\n"
            "coding, analysis, and more.\n"
            ">>> /bye\n"
            "\n"
            "# === MANAGEMENT ===\n"
            "$ ollama list\n"
            "# NAME       SIZE    MODIFIED\n"
            "# llama3.2   2.0GB   2 min ago\n"
            "$ ollama ps\n"
            "# Show loaded in memory\n"
            "$ ollama show llama3.2\n"
            "# Model details: size,\n"
            "# context window, family\n"
            "$ ollama rm llama3.2\n"
            "# Delete to free disk\n"
            "\n"
            "# === SERVER ===\n"
            "# Starts automatically.\n"
            "# Or manually:\n"
            "$ ollama serve\n"
            "# Runs at localhost:11434"
        )

    def get_openai_sdk(self) -> str:
        """Use OpenAI Python SDK with Ollama."""
        return (
            "from openai import OpenAI\n"
            "\n"
            "# Point OpenAI SDK at local\n"
            "# Ollama server — just\n"
            "# change base_url\n"
            "client = OpenAI(\n"
            "    base_url=\n"
            "        'http://localhost'\n"
            "        ':11434/v1',\n"
            "    api_key='ollama'\n"
            "    # Required by SDK,\n"
            "    # value ignored by\n"
            "    # Ollama\n"
            ")\n"
            "\n"
            "# Chat completion —\n"
            "# identical to OpenAI\n"
            "response = client.chat\\\n"
            "    .completions.create(\n"
            "    model='llama3.2',\n"
            "    messages=[\n"
            "        {'role': 'system',\n"
            "         'content': 'You are'\n"
            "          ' a helpful assistant.'},\n"
            "        {'role': 'user',\n"
            "         'content': 'Explain'\n"
            "          ' async/await in'\n"
            "          ' Python'}\n"
            "    ]\n"
            ")\n"
            "print(response.choices[0]\n"
            "      .message.content)\n"
            "\n"
            "# Streaming — same API\n"
            "stream = client.chat\\\n"
            "    .completions.create(\n"
            "    model='llama3.2',\n"
            "    messages=[\n"
            "        {'role': 'user',\n"
            "         'content': 'Write a'\n"
            "          ' haiku about code'}\n"
            "    ],\n"
            "    stream=True\n"
            ")\n"
            "for chunk in stream:\n"
            "    if chunk.choices[0]\n"
            "        .delta.content:\n"
            "        print(chunk.choices[0]\n"
            "            .delta.content,\n"
            "            end='', flush=True)\n"
            "\n"
            "# Structured output —\n"
            "# works with Ollama\n"
            "from pydantic import BaseModel\n"
            "\n"
            "class Summary(BaseModel):\n"
            "    title: str\n"
            "    points: list[str]\n"
            "\n"
            "response = client.chat\\\n"
            "    .completions.create(\n"
            "    model='llama3.2',\n"
            "    messages=[\n"
            "        {'role': 'user',\n"
            "         'content': 'Summarize'\n"
            "          ' Python decorators'}\n"
            "    ],\n"
            "    response_format={\n"
            "        'type': 'json_schema',\n"
            "        'json_schema': {\n"
            "            'name': 'Summary',\n"
            "            'schema': Summary\n"
            "                .model_json_schema()\n"
            "        }\n"
            "    }\n"
            ")\n"
            "\n"
            "# Embeddings for RAG\n"
            "emb = client.embeddings.create(\n"
            "    model='nomic-embed-text',\n"
            "    input='Python is great'\n"
            ")\n"
            "print(emb.data[0].embedding[:5])"
        )

    def get_native_sdk(self) -> str:
        """Native Ollama Python SDK."""
        return (
            "import ollama\n"
            "\n"
            "# pip install ollama\n"
            "# Native SDK — cleaner\n"
            "# for Ollama-specific\n"
            "# features\n"
            "\n"
            "# Chat\n"
            "response = ollama.chat(\n"
            "    model='llama3.2',\n"
            "    messages=[\n"
            "        {'role': 'user',\n"
            "         'content': 'Hello!'}\n"
            "    ]\n"
            ")\n"
            "print(response['message']\n"
            "      ['content'])\n"
            "\n"
            "# Streaming\n"
            "stream = ollama.chat(\n"
            "    model='llama3.2',\n"
            "    messages=[\n"
            "        {'role': 'user',\n"
            "         'content': 'Tell me a'\n"
            "          ' story'}\n"
            "    ],\n"
            "    stream=True\n"
            ")\n"
            "for chunk in stream:\n"
            "    print(chunk['message']\n"
            "          ['content'],\n"
            "          end='', flush=True)\n"
            "\n"
            "# Generate (no chat format)\n"
            "response = ollama.generate(\n"
            "    model='llama3.2',\n"
            "    prompt='Write a function'\n"
            "      ' to reverse a list'\n"
            ")\n"
            "print(response['response'])\n"
            "\n"
            "# Embeddings\n"
            "response = ollama.embeddings(\n"
            "    model='nomic-embed-text',\n"
            "    prompt='Embed this text'\n"
            ")\n"
            "print(response['embedding'][:5])\n"
            "\n"
            "# List models\n"
            "models = ollama.list()\n"
            "for m in models['models']:\n"
            "    print(m['name'], m['size'])\n"
            "\n"
            "# Pull new model\n"
            "ollama.pull('mistral')\n"
            "\n"
            "# Delete model\n"
            "ollama.delete('mistral')"
        )

    def get_curl_api(self) -> str:
        """Direct API calls with curl."""
        return (
            "# Native API: /api/chat\n"
            "curl http://localhost:11434\\\n"
            "  /api/chat -d '{\n"
            "    \"model\": \"llama3.2\",\n"
            "    \"messages\": [\n"
            "      {\"role\": \"user\",\n"
            "       \"content\": \"Hello\"}\n"
            "    ]\n"
            "  }'\n"
            "\n"
            "# OpenAI-compatible:\n"
            "# /v1/chat/completions\n"
            "curl http://localhost:11434\\\n"
            "  /v1/chat/completions \\\n"
            "  -H 'Content-Type:\n"
            "    application/json' \\\n"
            "  -d '{\n"
            "    \"model\": \"llama3.2\",\n"
            "    \"messages\": [\n"
            "      {\"role\": \"user\",\n"
            "       \"content\": \"Hello\"}\n"
            "    ]\n"
            "  }'\n"
            "\n"
            "# Generate (no chat)\n"
            "curl http://localhost:11434\\\n"
            "  /api/generate -d '{\n"
            "    \"model\": \"llama3.2\",\n"
            "    \"prompt\": \"Write code\"\n"
            "  }'\n"
            "\n"
            "# Embeddings\n"
            "curl http://localhost:11434\\\n"
            "  /api/embeddings -d '{\n"
            "    \"model\":\n"
            "      \"nomic-embed-text\",\n"
            "    \"prompt\": \"Embed this\"\n"
            "  }'"
        )

    def get_docker_deployment(self) -> str:
        """Docker deployment for Ollama."""
        return (
            "# Run Ollama in Docker\n"
            "docker run -d \\\n"
            "  -v ollama:/root/.ollama \\\n"
            "  -p 11434:11434 \\\n"
            "  --name ollama \\\n"
            "  ollama/ollama\n"
            "\n"
            "# Pull model in container\n"
            "docker exec -it ollama \\\n"
            "  ollama pull llama3.2\n"
            "\n"
            "# docker-compose.yml\n"
            "version: '3.8'\n"
            "services:\n"
            "  ollama:\n"
            "    image: ollama/ollama\n"
            "    ports:\n"
            "      - '11434:11434'\n"
            "    volumes:\n"
            "      - ollama:/root/.ollama\n"
            "    # For GPU:\n"
            "    # deploy:\n"
            "    #   resources:\n"
            "    #     reservations:\n"
            "    #       devices:\n"
            "    #         - driver: nvidia\n"
            "    #           count: all\n"
            "    #           capabilities:\n"
            "    #             - gpu\n"
            "\n"
            "volumes:\n"
            "  ollama:\n"
            "\n"
            "# Serve on network\n"
            "# OLLAMA_HOST=0.0.0.0\n"
            "# ollama serve\n"
            "# Access from other\n"
            "# machines: \n"
            "# http://host:11434/v1"
        )

    def get_hardware_guide(self) -> dict:
        """Hardware requirements."""
        return {
            "8gb_ram": (
                "3B models: llama3.2:3b, "
                "phi3. Good for basic "
                "tasks, prototyping, "
                "testing."
            ),
            "16gb_ram": (
                "7-8B models: llama3.2, "
                "mistral, qwen2.5, "
                "deepseek-r1:8b. Good "
                "for most coding and "
                "writing tasks."
            ),
            "32gb_ram": (
                "14-32B models: phi4, "
                "qwen2.5:32b. Higher "
                "quality output, "
                "slower inference."
            ),
            "apple_silicon": (
                "Automatic Metal/MLX "
                "acceleration. Zero "
                "config. Ollama 0.19+: "
                "MLX framework, 1.6-2x "
                "speedup. Best local "
                "LLM experience."
            ),
            "nvidia_gpu": (
                "CUDA acceleration. "
                "Automatic detection. "
                "Significant speedup "
                "for 7B+ models. "
                "Docker: add GPU "
                "device config."
            ),
            "cpu_only": (
                "Works without GPU. "
                "Slower inference. "
                "Fine for 3B models "
                "and prototyping. "
                "No additional setup."
            ),
        }

    def get_common_issues(self) -> dict:
        """Common issues and fixes."""
        return {
            "connection_refused": (
                "Ollama server not "
                "running. Start with "
                "'ollama serve' or "
                "ensure systemd service "
                "active on Linux."
            ),
            "404_on_v1": (
                "OpenAI endpoint at "
                "/v1/chat/completions. "
                "Native at /api/chat. "
                "Check URL path."
            ),
            "out_of_memory": (
                "Model too large for "
                "available RAM. Use "
                "smaller model or "
                "quantized version."
            ),
            "slow_inference": (
                "No GPU detected. "
                "Apple Silicon: check "
                "Metal/MLX. NVIDIA: "
                "check CUDA. CPU-only "
                "is slower."
            ),
            "model_not_found": (
                "Pull model first: "
                "ollama pull <name>. "
                "Check ollama list for "
                "downloaded models."
            ),
        }

Ollama Tutorial Checklist

[ ] Ollama: free, open-source runtime for running LLMs locally on Mac, Linux, Windows
[ ] Install: Mac/Windows download from ollama.com, Linux curl install.sh
[ ] Background service starts automatically at localhost:11434
[ ] Pull model: ollama pull llama3.2 (~2GB download)
[ ] Run interactive: ollama run llama3.2 — downloads if needed, opens chat
[ ] Management: ollama list, ollama ps, ollama show, ollama rm
[ ] Server: ollama serve — starts manually if not auto-started
[ ] No internet needed after initial download, no API key, no per-query costs
[ ] Data never leaves your machine — privacy-preserving
[ ] OpenAI-compatible API at /v1/chat/completions — same schema as OpenAI
[ ] Change one URL: base_url='http://localhost:11434/v1', api_key='ollama'
[ ] All OpenAI SDK features work: chat, streaming, structured output, tool calling, embeddings
[ ] Native API: /api/chat, /api/generate, /api/embeddings for full control
[ ] Python OpenAI SDK: from openai import OpenAI; client = OpenAI(base_url=..., api_key='ollama')
[ ] Python native SDK: pip install ollama; import ollama; ollama.chat(), ollama.generate(), ollama.embeddings()
[ ] HTTP requests: POST to /api/chat or /v1/chat/completions with JSON body
[ ] Streaming: stream=True in chat.completions.create() or ollama.chat(stream=True)
[ ] Structured output: response_format with json_schema — works with Ollama
[ ] Embeddings: nomic-embed-text model for RAG pipelines (~274MB)
[ ] Models: llama3.2:3b (2GB, general), mistral:7b (4.1GB, instructions), qwen2.5-coder:7b (4.7GB, coding)
[ ] Models: deepseek-r1:8b (4.9GB, reasoning), phi4:14b (9.1GB, versatile), nomic-embed-text (274MB, RAG)
[ ] 100+ open-source models supported
[ ] Hardware: 8GB RAM = 3B models, 16GB = 7-8B models, 32GB = 14-32B models
[ ] Apple Silicon: automatic Metal/MLX acceleration, zero config, 1.6-2x speedup (Ollama 0.19+)
[ ] NVIDIA GPU: CUDA acceleration, automatic detection, significant speedup
[ ] CPU only: works without GPU, slower inference, fine for prototyping
[ ] Docker: docker run -v ollama:/root/.ollama -p 11434:11434 ollama/ollama
[ ] Docker: docker exec -it ollama ollama pull llama3.2
[ ] Docker GPU: add deploy.resources.reservations.devices with nvidia driver
[ ] Network access: OLLAMA_HOST=0.0.0.0 ollama serve — access from other machines
[ ] Common issue: ConnectionError → server not running, start with ollama serve
[ ] Common issue: 404 on /v1 → check URL path, native is /api/chat, OpenAI is /v1/chat/completions
[ ] Common issue: out of memory → use smaller model or quantized version
[ ] Common issue: slow inference → check GPU/CUDA/Metal, CPU-only is slower
[ ] Common issue: model not found → pull first with ollama pull
[ ] Read AI agents for agent architecture
[ ] Read function calling for tool use with Ollama
[ ] Read no framework for vanilla agent with Ollama
[ ] Read debugging for tracing local agents
[ ] Test: ollama pull and run llama3.2 successfully
[ ] Test: OpenAI SDK connects to localhost:11434/v1 and returns response
[ ] Test: streaming works with both OpenAI SDK and native SDK
[ ] Test: structured output returns valid JSON matching schema
[ ] Test: embeddings generated with nomic-embed-text
[ ] Test: Docker container runs Ollama and serves API on port 11434
[ ] Test: network access works with OLLAMA_HOST=0.0.0.0
[ ] Document model choice, hardware, API endpoint, Docker config, network setup

FAQ

How do you install Ollama and run LLMs locally?

Install Ollama from ollama.com, pull a model with one command, and start chatting. AI/TLDR: "Ollama is a free, open-source tool that makes running a large language model on your own computer as simple as one terminal command. You type ollama run llama3.2, it downloads the model, and within a minute you are having a conversation that never touches the internet." Pristren: "Ollama is a free, open source runtime that lets you run Llama 3.3, Mistral, Phi-3, and Deepseek-R1 on your own hardware for free. Installation takes one command, the first model downloads and runs in under 5 minutes, and every query stays on your hardware with no API costs." SolutionGigs: "Install in 2 minutes, pull a model, call it from Python using the OpenAI SDK (it is API-compatible). 8GB RAM handles 7B models; 16GB handles up to 32B." Install: (1) Mac/Windows: download from ollama.com. (2) Linux: curl -fsSL https://ollama.com/install.sh | sh. (3) Pull model: ollama pull llama3.2. (4) Run: ollama run llama3.2. (5) Server starts automatically at localhost:11434.

How do you use the Ollama OpenAI-compatible API?

Ollama exposes an OpenAI-compatible endpoint at /v1/chat/completions — just change the base URL. AI/TLDR: "Ollama local server speaks the same /v1/chat/completions schema as OpenAI. Any code that calls OpenAI works with Ollama by changing one URL — no library changes needed. Set base_url to http://localhost:11434/v1 and any non-empty string for api_key. Everything else — chat.completions.create(), streaming, system messages — works unchanged." Pristren: "Ollama serves an OpenAI-compatible endpoint at http://localhost:11434/v1. Any tool or library that supports OpenAI API can point to this endpoint and use your local models instead." SolutionGigs: "Use the official OpenAI Python SDK — just change the base URL. All SDK features work — chat, streaming, structured output, tool calling — exactly as with the real OpenAI API." Usage: from openai import OpenAI; client = OpenAI(base_url='http://localhost:11434/v1', api_key='ollama'); response = client.chat.completions.create(model='llama3.2', messages=[...]).

What models can you run with Ollama?

Ollama supports 100+ open-source models including Llama, Mistral, Qwen, DeepSeek, Phi, and Gemma. AI/TLDR: "Models: llama3.2 (3B, ~2GB), deepseek-r1 (7B, ~4.7GB, reasoning/math), qwen2.5 (7B, ~4.7GB, multilingual/coding), nomic-embed-text (~274MB, embeddings for RAG)." Pristren: "Run Llama 3.3, Mistral, Phi-3, Deepseek-R1 on your own hardware. A 7B parameter model on a standard MacBook Pro delivers responses good enough for most real-world coding and writing tasks." SolutionGigs: "100+ open-source LLMs. mistral (4.1GB, instruction following), qwen2.5-coder (4.7GB, coding), phi4 (9.1GB, 14B), deepseek-r1:8b (4.9GB, reasoning), nomic-embed-text (274MB, RAG embeddings)." Models: (1) llama3.2:3b — 2GB, general purpose, fits 8GB RAM. (2) mistral:7b — 4.1GB, instruction following. (3) qwen2.5-coder:7b — 4.7GB, coding tasks. (4) deepseek-r1:8b — 4.9GB, reasoning with chain-of-thought. (5) phi4:14b — 9.1GB, Microsoft's 14B. (6) nomic-embed-text — 274MB, embeddings for RAG.

How do you use Ollama from Python?

Use the OpenAI Python SDK with base_url pointing to localhost:11434/v1, or the native ollama Python package. AI/TLDR: "Point the official openai Python library at your local server with a single URL change — no code rewrites, no API key required." SolutionGigs: "Ollama exposes an OpenAI-compatible REST API. Use the official OpenAI Python SDK — just change the base URL. All SDK features work — chat, streaming, structured output, tool calling." TechInsider: "You can use the OpenAI Python SDK with Ollama by pointing it at the local endpoint. Useful if you have existing code that uses the OpenAI API and want to switch to a local model without rewriting." Python: (1) OpenAI SDK: client = OpenAI(base_url='http://localhost:11434/v1', api_key='ollama'). (2) Native SDK: pip install ollama; import ollama; ollama.chat(model='llama3.2', messages=[...]). (3) Requests: POST to http://localhost:11434/api/chat. (4) Streaming: stream=True in chat.completions.create(). (5) All features work: chat, streaming, structured output, tool calling, embeddings.

What are the hardware requirements for Ollama?

8GB RAM handles 7B models, 16GB handles up to 32B, Apple Silicon gets automatic Metal acceleration. SolutionGigs: "8GB RAM handles 7B models; 16GB handles up to 32B. No API key, no cloud bill, no data leaving your network." Pristren: "A 7B parameter model on a standard MacBook Pro delivers responses good enough for most real-world coding and writing tasks." Medium: "Apple Silicon is the nicest experience. Zero configuration needed. Metal acceleration kicks in automatically. Starting with Ollama 0.19, inference on Apple Silicon moved from llama.cpp to Apple MLX framework, pushing decode speeds up 1.6-2x." Requirements: (1) 8GB RAM: 3B models (llama3.2:3b, phi3). (2) 16GB RAM: 7-8B models (llama3.2, mistral, qwen2.5, deepseek-r1:8b). (3) 32GB RAM: 14-32B models (phi4, qwen2.5:32b). (4) Apple Silicon: automatic Metal/MLX acceleration, no config needed. (5) GPU: NVIDIA CUDA supported, automatic detection. (6) CPU: works without GPU, slower inference. (7) Disk: model size + overhead (2-10GB per model).

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →