LLM Quantization: GGUF, AWQ, GPTQ, and FP8 Comparison with Quality Benchmarks for 2026

TL;DR — LLM quantization 2026: formats, benchmarks, production recommendations. PresencAI: "Four formats: GGUF, MLX, AWQ, GPTQ. Q4 degrades perplexity 1-3% — production sweet spot. Q3 meaningful loss (3-8%). Q5/Q6 minimal improvement, rarely worth it. AWQ outperforms GPTQ by 0.5-1.0%." DataRekha: "Four lanes: FP8 (H100, near-lossless, default 2026), AWQ INT4 (A100/3090, ~1% loss), GPTQ (fallback, 1-3% loss), GGUF (edge, Q4_K_M, ~75% smaller). INT4 breaks on long context (59% drop at 32k), code (syntax errors), math (non-linear degradation)." RunLocalAI: "Q4_K_M / AWQ-INT4 is the knee of the curve. GGUF runs on every backend. AWQ throughput 15-30% faster than GGUF on vLLM." ChaosAndOrder: "AWQ + Marlin: 741 tok/s. GGUF Q4_K_M: 280 tok/s. Production GPU: AWQ. Edge: GGUF." Learn more with model selection, hardware requirements, Ollama tutorial, and open source models.

PresencAI frames the landscape: "Quantization makes local LLMs practical: it shrinks memory, accelerates inference, and trades a small quality cost for large efficiency gains. But the quality cost depends on which quantization format, which bit-width, and which model."

DataRekha identifies the production lanes: "Four families of quantization actually matter in production: weight-only post-training (GPTQ, AWQ), activation-aware (AWQ, SmoothQuant), low-bit native (FP8 on H100/H200, INT4 on consumer), and GGUF for the llama.cpp ecosystem."

Quantization Architecture

Perplexity Benchmarks

Throughput Comparison

Implementation

from dataclasses import dataclass
from typing import Optional
from enum import Enum

class QuantFormat(Enum):
    GGUF = "gguf"
    AWQ = "awq"
    GPTQ = "gptq"
    FP8 = "fp8"
    MLX = "mlx"

@dataclass
class LLMQuantizationGuide:
    """LLM quantization implementation guide."""

    def get_format_comparison(self) -> dict:
        """Format comparison."""
        return {
            "gguf": {
                "full_name": "GPT-Generated Unified Format",
                "ecosystem": "llama.cpp, Ollama, LM Studio, LocalAI",
                "platforms": "CPU, NVIDIA CUDA, AMD ROCm, Apple Metal, Vulkan, Intel SYCL, ARM",
                "strength": "Runs on every backend. Single file with weights + tokenizer + chat template + metadata.",
                "weakness": "Throughput trails AWQ by 15-30% on vLLM. Not for multi-tenant serving.",
                "best_for": "Portability, single-user inference, edge, CPU, Ollama",
                "quants": "K-quants (Q4_K_M, Q5_K_M, Q6_K, Q8_0), i-quants (IQ3_XS, IQ4_XS)",
            },
            "awq": {
                "full_name": "Activation-aware Weight Quantization",
                "ecosystem": "vLLM, SGLang, Hugging Face",
                "platforms": "NVIDIA CUDA (A100, L40, 3090, 4090)",
                "strength": "Outperforms GPTQ by 0.5-1.0% perplexity. Activation-aware scaling. + Marlin kernel = highest throughput.",
                "weakness": "INT4 floor (no 3-bit). NVIDIA only.",
                "best_for": "Production GPU serving, multi-tenant, agents",
                "quants": "INT4, INT3",
            },
            "gptq": {
                "full_name": "Generalized Post-Training Quantization",
                "ecosystem": "vLLM, ExLlamaV2, Hugging Face",
                "platforms": "NVIDIA CUDA, some AMD",
                "strength": "Broadest ecosystem support. 3-bit quantization. EXL2 variant for max single-stream tok/s.",
                "weakness": "1-3% MMLU loss (more than AWQ). Less vLLM optimization attention.",
                "best_for": "Fallback when AWQ not available, ExLlamaV2, 3-bit needs",
                "quants": "INT4, INT3, EXL2 (variable bitrate)",
            },
            "fp8": {
                "full_name": "FP8 (E4M3 format)",
                "ecosystem": "vLLM, TensorRT-LLM, TGI",
                "platforms": "H100, H200, B200 (native tensor cores)",
                "strength": "Near-lossless (~1%). Native tensor core acceleration. Default for new H100 deployments in 2026.",
                "weakness": "Requires H100/H200/B200 hardware. Not for consumer GPUs.",
                "best_for": "H100-class production, near-lossless, long context",
                "quants": "E4M3 (8-bit float)",
            },
            "mlx": {
                "full_name": "MLX",
                "ecosystem": "Apple Silicon native",
                "platforms": "Apple M3/M4/M5 (Pro/Max/Ultra)",
                "strength": "Native Apple Silicon. Unified memory. Metal acceleration. Zero config.",
                "weakness": "Apple Silicon only.",
                "best_for": "Mac deployment, Apple Silicon, unified memory",
                "quants": "4-bit, 8-bit",
            },
        }

    def get_gguf_levels(self) -> dict:
        """GGUF quantization levels."""
        return {
            "q8_0": {
                "bits": 8.0,
                "perplexity_increase": "~0%",
                "vram_savings": "~1.9x",
                "recommendation": "Close to lossless. Quality-critical evaluation.",
            },
            "q6_k": {
                "bits": 6.6,
                "perplexity_increase": "+0.05-0.6%",
                "vram_savings": "~2.5x",
                "recommendation": "Near lossless. Good for math/reasoning.",
            },
            "q5_k_m": {
                "bits": 5.7,
                "perplexity_increase": "+0.1-1.2%",
                "vram_savings": "~2.8x",
                "recommendation": "Quality-focused. Production agents.",
            },
            "q4_k_m": {
                "bits": 4.8,
                "perplexity_increase": "+0.3-1.9%",
                "vram_savings": "~3.3x",
                "recommendation": "General recommended. Production sweet spot.",
            },
            "q3_k_m": {
                "bits": 3.9,
                "perplexity_increase": "+1.0-4.8%",
                "vram_savings": "~4.1x",
                "recommendation": "Memory-constrained. Non-reasoning only.",
            },
            "q2_k": {
                "bits": 3.0,
                "perplexity_increase": "+11-14%",
                "vram_savings": "~5.3x",
                "recommendation": "Extreme memory limits only. Quality degrades badly.",
            },
        }

    def get_production_recommendations(self) -> dict:
        """Production recommendations by use case."""
        return {
            "single_user_chat": "GGUF Q4_K_M — best speed/quality/portability balance",
            "production_agents": "AWQ 4-bit or GGUF Q5_K_M — reasoning quality matters",
            "apple_silicon": "MLX 4-bit if available, fall back to GGUF Q4_K_M",
            "memory_constrained": "Q3_K_M for non-reasoning only, never below Q3",
            "quality_critical": "Q8_0 or FP16 — 2x memory cost is small at this end",
            "h100_production": "FP8 native — near-lossless, default 2026",
            "edge_cpu": "GGUF Q4_K_M with imatrix calibration — nothing else competes",
            "multi_tenant_gpu": "AWQ + Marlin kernel via vLLM — highest throughput",
            "code_generation": "AWQ 4-bit + constrained decoding — not GPTQ",
            "math_reasoning": "Q6_K or higher, or AWQ 4-bit — numerically sensitive",
            "long_context": "FP8 or W8 — INT4 drops 59% at 32k+ context",
            "classification_ner": "INT4 sufficient — minimal accuracy impact",
        }

    def get_breakage_points(self) -> dict:
        """Where INT4 quantization breaks."""
        return {
            "long_context": {
                "issue": "Drops up to 59% at 32k+ context",
                "cause": "Attention precision errors compound across tokens",
                "fix": "Stay at FP8 or W8 for context > 16k",
                "eval": "Test at your actual context length, not model card MMLU",
            },
            "code_generation": {
                "issue": "Syntax errors spike on rarer tokens",
                "cause": "Structured generation sensitive to tail probabilities",
                "fix": "AWQ + constrained decoding, not GPTQ",
                "eval": "HumanEval with rare-token syntax cases",
            },
            "math_logic": {
                "issue": "Multi-step reasoning degrades non-linearly",
                "cause": "Small per-token errors become wrong answers",
                "fix": "FP8 minimum, measure GSM8K not MMLU",
                "eval": "GSM8K, AIME — multi-step math benchmarks",
            },
            "iq3_without_imatrix": {
                "issue": "IQ3 quantizations hallucinate badly",
                "cause": "Importance matrix calibration is load-bearing",
                "fix": "Ensure good imatrix calibration for IQ formats",
                "eval": "Compare with and without imatrix",
            },
        }

    def get_ollama_commands(self) -> str:
        """Ollama quantization commands."""
        return (
            "# === PULL DEFAULT (Q4_K_M) ===\n"
            "$ ollama pull llama3.2\n"
            "# Default quantization is Q4_K_M\n"
            "\n"
            "# === SPECIFY QUANTIZATION ===\n"
            "$ ollama pull llama3.2:8b-q5_K_M\n"
            "$ ollama pull llama3.2:8b-q8_0\n"
            "$ ollama pull llama3.2:8b-q3_K_M\n"
            "\n"
            "# === LIST MODELS ===\n"
            "$ ollama list\n"
            "# NAME           ID    SIZE\n"
            "# llama3.2:latest ...   4.9GB\n"
            "# (Q4_K_M default)\n"
            "\n"
            "# === SHOW MODEL DETAILS ===\n"
            "$ ollama show llama3.2\n"
            "# Shows quantization_level\n"
            "# e.g., Q4_K_M\n"
            "\n"
            "# === CREATE CUSTOM QUANT ===\n"
            "# Create Modelfile:\n"
            "# FROM llama3.2:8b-q5_K_M\n"
            "# SYSTEM You are helpful\n"
            "$ ollama create my-llama \\\n"
            "    -f Modelfile\n"
            "\n"
            "# === BENCHMARK ===\n"
            "$ ollama run llama3.2 \\\n"
            "    'Write a Python function'\n"
            "# Check tokens/sec in stats"
        )

LLM Quantization Checklist

• [ ] Quantization reduces model precision to save memory and speed up inference
• [ ] Four formats matter in 2026: GGUF (llama.cpp), AWQ (activation-aware), GPTQ (Hessian-based), FP8 (H100 native)
• [ ] MLX for Apple Silicon native 4-bit quantization
• [ ] Q4 quantization (4-bit): 1-3% perplexity degradation — practical production sweet spot
• [ ] Q3 quantization: 3-8% perplexity degradation — meaningful quality loss, reasoning regresses
• [ ] Q5/Q6: under 1% improvement over Q4 at meaningful memory cost — rarely worth it
• [ ] Q8_0: ~0% loss, near-lossless — quality-critical evaluation only
• [ ] Q2_K: 11-14% degradation — extreme memory limits only, quality degrades badly
• [ ] Q4_K_M / AWQ-INT4 is the knee of the curve — most VRAM savings for acceptable quality loss
• [ ] GGUF: runs on every backend (CPU, CUDA, ROCm, Metal, Vulkan, SYCL, ARM)
• [ ] GGUF: single file with weights + tokenizer + chat template + metadata
• [ ] GGUF: K-quants (Q4_K_M, Q5_K_M, Q6_K, Q8_0) are modern default
• [ ] GGUF: i-quants (IQ3_XS, IQ4_XS) with imatrix calibration — aggressive low-bit
• [ ] GGUF: throughput trails AWQ by 15-30% on vLLM — not for multi-tenant serving
• [ ] GGUF: best for portability, single-user inference, edge, CPU, Ollama
• [ ] AWQ: outperforms GPTQ by 0.5-1.0% perplexity at same bit-width
• [ ] AWQ: activation-aware scaling, INT4 floor, + Marlin kernel = highest throughput
• [ ] AWQ: best for production GPU serving, multi-tenant, agents
• [ ] AWQ + Marlin: 741 tok/s, 51.8% HumanEval — highest throughput + quality
• [ ] GPTQ: 1-3% MMLU loss (more than AWQ), broadest ecosystem, fallback
• [ ] GPTQ: 3-bit quantization (more aggressive than AWQ's 4-bit floor)
• [ ] GPTQ: EXL2 variant for max single-stream tok/s via ExLlamaV2
• [ ] FP8 (E4M3): near-lossless ~1%, native H100/H200/B200 tensor cores
• [ ] FP8: default for new H100-class deployments in 2026
• [ ] FP8: not for consumer GPUs — requires data center hardware
• [ ] MLX: Apple Silicon native, 4-bit default, Metal acceleration, unified memory
• [ ] Production: single-user chat → GGUF Q4_K_M
• [ ] Production: agents/tool-use → AWQ 4-bit or GGUF Q5_K_M
• [ ] Production: Apple Silicon → MLX 4-bit, fall back to GGUF Q4_K_M
• [ ] Production: edge/CPU → GGUF Q4_K_M with imatrix calibration
• [ ] Production: H100 → FP8 native, near-lossless
• [ ] Production: multi-tenant GPU → AWQ + Marlin via vLLM
• [ ] Production: code generation → AWQ 4-bit + constrained decoding
• [ ] Production: math/reasoning → Q6_K or higher, or AWQ 4-bit
• [ ] Production: long context > 16k → FP8 or W8 (INT4 drops 59% at 32k+)
• [ ] Production: classification/NER → INT4 sufficient
• [ ] Production: mission-critical (medical/legal/financial) → Q5 or Q8
• [ ] INT4 breaks on: long context (59% drop at 32k+), code (syntax errors on rare tokens), math (non-linear degradation)
• [ ] Mitigation: long context → FP8/W8; code → AWQ + constrained; math → FP8, measure GSM8K
• [ ] IQ3 without imatrix → hallucinates badly — ensure good calibration
• [ ] Pick eval that exercises your workload shape, not model card MMLU
• [ ] Ollama default quantization: Q4_K_M — ollama pull model gets Q4 automatically
• [ ] Ollama: specify quantization with tag (e.g., llama3.2:8b-q5_K_M, q8_0, q3_K_M)
• [ ] Ollama: ollama show displays quantization_level
• [ ] For most production use cases: yes, quantized models are fine — 1-3% below noise floor
• [ ] Read model selection for choosing models
• [ ] Read hardware requirements for VRAM tiers
• [ ] Read Ollama tutorial for setup
• [ ] Read open source models for model options
• [ ] Test: quantized model produces acceptable quality for your use case
• [ ] Test: perplexity/benchmark scores match expectations for quantization level
• [ ] Test: throughput meets your latency requirements
• [ ] Test: long context performance at your actual context length
• [ ] Test: code generation accuracy with rare-token syntax cases
• [ ] Test: math/reasoning with GSM8K or AIME benchmarks
• [ ] Document format, quantization level, quality loss, VRAM savings, throughput, use case

FAQ

What is LLM quantization and which format should you use?

Quantization reduces model precision to save memory and speed up inference. PresencAI: "Four formats developers deploy in 2026: GGUF (llama.cpp), MLX (Apple Silicon), AWQ, GPTQ. Q4 quantization degrades perplexity by 1-3 percent versus FP16 — practical production sweet spot. Q3 shows meaningful quality loss (3-8 percent). Q5/Q6 minimal improvement over Q4 (under 1 percent) at meaningful memory cost, rarely worth it." DataRekha: "Four lanes: FP8 native (H100/H200, near-lossless, default 2026), AWQ INT4 (A100/L40/3090, ~1% MMLU loss), GPTQ INT4 (older GPUs, 1-3% MMLU loss), GGUF (CPU+Metal, Q4_K_M sweet spot, ~75% smaller, laptop/edge)." RunLocalAI: "Q4_K_M / AWQ-INT4 is the knee of the curve — most VRAM savings for acceptable quality loss. Below Q4 rarely worth it on coding/reasoning. Above Q5 only when VRAM to burn." Use: GGUF Q4_K_M for local/Ollama, AWQ for production GPU, FP8 for H100, MLX for Apple Silicon.

How do GGUF quantization levels compare in quality?

Q4_K_M is the sweet spot — ~2.5% quality loss for ~3.3x VRAM savings. PresencAI: "Perplexity benchmarks: FP16 baseline, Q8_0 +0.2%, Q6_K +0.6%, Q5_K_M +1.2%, Q4_K_M +1.9%, Q3_K_M +4.8%, Q2_K +13%. Q4 is production sweet spot. Q3 shows meaningful quality loss. Q5/Q6 minimal improvement over Q4." ChaosAndOrder: "Q2_K: +2.5-3.0 perplexity, extreme memory limits. Q3_K_M: +1.0-1.5, memory-constrained. Q4_K_M: +0.3-0.5, general recommended. Q5_K_M: +0.1-0.2, quality-focused. Q6_K: +0.05, near lossless. Q8_0: ~0, close to lossless." RunLocalAI: "Q8_0 ~1% loss ~1.9x savings, Q6_K ~1.5% ~2.5x, Q5_K_M ~2% ~2.8x, Q4_K_M ~2.5% ~3.3x, Q3_K_M ~5-8% ~4.1x, Q2_K ~15%+ ~5.3x." GGUF levels: Q8_0 near-lossless, Q6_K near-lossless, Q5_K_M quality-focused, Q4_K_M recommended, Q3_K_M constrained, Q2_K extreme only.

How do AWQ, GPTQ, and GGUF compare for production?

AWQ outperforms GPTQ on modern models; GGUF best for portability and edge. PresencAI: "AWQ outperforms GPTQ on most modern models (Llama 3+, Qwen 2+) by approximately 0.5-1.0 percent perplexity at same bit-width, due to activation-aware scaling. Production agents: AWQ 4-bit or GGUF Q5_K_M. Single-user chat: GGUF Q4_K_M. Apple Silicon: MLX 4-bit." DataRekha: "AWQ owns older-hardware retrofit, GPTQ remains fallback, GGUF rules the edge. Q4_K_M is production sweet spot — ~75% size reduction, perplexity bump under 5%. GGUF throughput trails AWQ by 15-30% on vLLM." ChaosAndOrder: "AWQ 4bit + Marlin: 741 tok/s, 51.8% HumanEval. GPTQ + Marlin: 712 tok/s, 50.6%. GGUF Q4_K_M: 280 tok/s, 51.8%. Production GPU: AWQ + Marlin. Edge/CPU: GGUF Q4_K_M." AWQ: best quality + throughput on GPU. GPTQ: fallback, broadest ecosystem. GGUF: best portability, edge, CPU.

When does quantization break and how do you mitigate it?

INT4 breaks on long context, code generation, and multi-step math — not visible on MMLU. DataRekha: "Three workloads where INT4 near-lossless claim stops being true: (1) Long context: drops up to 59% at 32k+ context, attention precision errors compound. Fix: stay at FP8/W8 for context > 16k. (2) Code generation: syntax errors spike on rarer tokens, structured generation sensitive to tail probs. Fix: AWQ + constrained decoding, not GPTQ. (3) Math & logic: multi-step reasoning degrades non-linearly, small per-token errors become wrong answers. Fix: FP8 minimum, measure GSM8K not MMLU." PresencAI: "Q3 shows visible regression on reasoning benchmarks (GSM8K, HumanEval). For mission-critical reasoning (medical, legal, financial), Q5 or Q8 is worth the memory cost." RunLocalAI: "Without good importance matrix, IQ3 quantizations hallucinate badly. Pick an eval that exercises your shape, not the model card." Mitigation: (1) Long context > 16k: use FP8/W8. (2) Code: AWQ + constrained decoding. (3) Math: FP8 minimum, measure GSM8K. (4) Mission-critical: Q5 or Q8. (5) IQ3: ensure good imatrix calibration.

Should you use quantized models in production?

Yes for most use cases — Q4_K_M or AWQ INT4 deliver 1-3% degradation below noise floor. PresencAI: "For most production use cases, yes. The 1-3 percent perplexity degradation and 1-2 percent reasoning-benchmark drop is below the noise floor on real-user-facing metrics. For mission-critical reasoning (medical, legal, financial), Q5 or Q8 is worth the memory cost. Q4_K_M or AWQ 4-bit: 1-2 percent perplexity degradation, 3.5-3.8x speedup, roughly 4x memory savings versus FP16." DataRekha: "Q4_K_M is the production sweet spot. Above Q4 gains diminish; below Q4 reasoning degrades meaningfully. FP8 has quietly become the default for new H100-class deployments." ChaosAndOrder: "Production GPU serving: AWQ + Marlin kernel. Edge/CPU: GGUF Q4_K_M. Code generation: AWQ 4bit. Math reasoning: Q6_K or higher. Classification/NER: INT4 sufficient." Production: (1) Most use cases: Q4_K_M or AWQ INT4 — 1-3% loss, 4x savings. (2) Mission-critical: Q5 or Q8. (3) H100: FP8 near-lossless. (4) Edge/CPU: GGUF Q4_K_M. (5) Long context: FP8/W8. (6) Code: AWQ + constrained decoding.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →