How to Handle Context Windows and Token Limits in LLMs: A 2026 Guide

Q: Should you use long context or RAG for your LLM application?

Depends on corpus size and query frequency. [Swfte AI](https://www.swfte.com/blog/llm-context-window-explained): 'Decision tree: corpus 1000/day, use long context with prompt caching. Corpus >1M tokens, use retrieval. Corpus >10M tokens, use hierarchical retrieval. Under ~100 queries against codebase, just stuff it. Over ~1000, build retrieval.' [Google Cloud](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/long-context): 'Larger context windows unlock new use cases. But cost scales linearly — 128K tokens costs 128x more than 1K. For 100 pieces of information at 99% performance, need 100 requests. Context caching reduces cost while keeping performance.' [Tanuj Garg](https://tanujgarg.com/blog/llm-context-window-management-production): 'Cost scales linearly with context length. Latency increases — attention is quadratic. Quality degrades in long contexts.' Decision: (1) Corpus 1000/day: long context + prompt caching. (3) Corpus >1M: RAG retrieval. (4) Corpus >10M: hierarchical retrieval. (5) Under 100 queries: stuff context. (6) Over 1000 queries: build retrieval.

By Fadi Updated Feb 28, 2026 14 min read Dev Fundamentals

TL;DR — Context windows: bigger but not better in the middle. Tanuj Garg: "Cost scales linearly, latency quadratic, quality degrades. Lost-in-the-middle: primacy and recency, middle ignored. Strategies: sliding window, summarization, RAG, dynamic trimming." Zylos: "Models break 30-40% before claimed limit. U-shaped curve: 20 docs drop accuracy 70% to 55%. Chunking: small 100-200, medium 300-500, large 600-1000." Swfte AI: "Corpus <1M + queries <100/day: long context. >1M: RAG. >10M: hierarchical retrieval. Under 100 queries: stuff. Over 1000: build retrieval." Google Cloud: "Gemini 1M+ tokens, >99% retrieval. Context caching reduces cost. Drop old messages, summarize, use RAG, filter text." Learn more with RAG vs fine-tuning, prompt engineering, choose LLM, and cost estimation.

Tanuj Garg frames the production reality: "Context windows are getting larger. GPT-4o supports 128K tokens. Claude supports up to 200K. Gemini has reached 1M+. But cost scales linearly with context length — 128K tokens costs 128x more than 1K. Latency increases — attention computation is quadratic. Quality degrades — the lost-in-the-middle problem is well-documented."

Swfte AI provides the definition: "A context window is not the model's memory. It is the in-context working set for a single inference call, measured in tokens, and bounded by a hard architectural ceiling. In 2026, that ceiling ranges from 64,000 tokens on the smallest open-weight models to 2,000,000 tokens on Gemini 3.1 Pro."

Context Window Management Architecture

flowchart TD subgraph Problem["Context Window Challenges"] Cost["Cost Scaling
linear with context length
128K = 128x cost of 1K
every token costs money"] Latency["Latency
attention is quadratic
long contexts are slow
TTFT increases"] Lost["Lost in Middle
U-shaped performance
primacy + recency
middle ignored
70% to 55% at 4K"] Effective["Effective Limit
models break 30-40%
before claimed limit
200K reliable at ~130K"] end subgraph Strategies["Management Strategies"] Slide["Sliding Window
most recent N tokens
predictable, cheap
discards early context
short chat sessions"] Summarize["Summarization
compress older history
one extra LLM call
retains gist
long-running sessions"] RAG["RAG Retrieval
embed documents
retrieve top-K chunks
vector DB storage
document-heavy workloads"] Trim["Dynamic Trimming
score by relevance
budget per category
trim lowest first
never trim system/user"] end subgraph Chunking["Chunking Strategies"] Fixed["Fixed-Size
512 tokens, 50 overlap
simple, fast, predictable
overlap prevents split meaning"] Semantic["Semantic
split on paragraphs
sections, topic changes
preserves coherent units"] Parent["Parent-Document
retrieve small chunks
expand to parent context
multi-granularity indexing"] Sizes["Chunk Sizes
small 100-200: precision
medium 300-500: balanced
large 600-1000: context, noise"] end subgraph Decision["Long Context vs RAG"] Small["Corpus <1M, Q <100/day
use long context
stuff it in"] Cached["Corpus <1M, Q >1000/day
long context + caching
amortize cost"] Large["Corpus >1M
use RAG retrieval
only relevant chunks"] Huge["Corpus >10M
hierarchical retrieval
multi-level index"] end subgraph Ordering["Context Ordering"] System["System Prompt
always first
static, never trimmed"] Retrieved["Retrieved Context
ranked by relevance
high-value at start/end
avoid middle placement"] History["Conversation History
summarized older
recent turns intact
token budget capped"] Current["Current User Message
always last
never trimmed
most recent input"] end Problem --> Strategies Strategies --> Chunking Chunking --> Decision Decision --> Ordering style Problem fill:#FF6B6B,color:#fff style Strategies fill:#4169E1,color:#fff style Chunking fill:#39FF14,color:#000 style Decision fill:#2D1B69,color:#fff

2026 Context Window Comparison

Model	Context Window	Effective Limit	Output Tokens	Best For
Gemini 3.1 Pro	2M	~1.3M	64K	Long documents, multimodal
Llama 4 Scout	10M	~6.5M	Standard	Industry-leading, codebase
Claude Sonnet 4	1M	~650K	Standard	Long documents, analysis
GPT-5.2	400K	~260K	128K	Large output window
Claude Opus 4	200K	~130K	Standard	Premium reasoning
GPT-4o	128K	~85K	Standard	Mainstream
Llama 3.3	128K	~85K	Standard	Open-source, self-hosted
DeepSeek V4	1M	~650K	Standard	Cost-sensitive long context

Implementation

from dataclasses import dataclass
from typing import Optional
from enum import Enum

class ContextStrategy(Enum):
    SLIDING_WINDOW = "sliding_window"
    SUMMARIZATION = "summarization"
    RAG = "rag"
    DYNAMIC_TRIM = "dynamic_trim"

@dataclass
class ContextWindowGuide:
    """Context window management guide."""

    def get_token_counter(self) -> str:
        """Token counting for context management."""
        return (
            "# === TOKEN COUNTING ===\n"
            "# pip install tiktoken\n"
            "\n"
            "import tiktoken\n"
            "\n"
            "# OpenAI tokenizer\n"
            "enc = tiktoken.encoding_for_model(\n"
            "    'gpt-4o')\n"
            "\n"
            "def count_tokens(text: str) -> int:\n"
            "    '''Count tokens accurately.''' \n"
            "    return len(enc.encode(text))\n"
            "\n"
            "# For Ollama models, use\n"
            "#   API or local tokenizer\n"
            "async def count_ollama_tokens(\n"
            "    text: str,\n"
            "    model: str) -> int:\n"
            "    '''Count via Ollama API.''' \n"
            "    import httpx\n"
            "    async with httpx.AsyncClient(\n"
            "    ) as c:\n"
            "        r = await c.post(\n"
            "            'http://localhost:11434'\n"
            "            '/api/tokenize',\n"
            "            json={'model': model,\n"
            "                  'prompt': text})\n"
            "        return len(r.json()\n"
            "            ['tokens'])\n"
            "\n"
            "# Token budget per category\n"
            "TOKEN_BUDGETS = {\n"
            "    'system': 2000,\n"
            "    'history': 4000,\n"
            "    'retrieved': 4000,\n"
            "    'user': 1000,\n"
            "    'output': 2000,\n"
            "    'total': 13000,\n"
            "}"
        )

    def get_sliding_window(self) -> str:
        """Sliding window conversation management."""
        return (
            "# === SLIDING WINDOW ===\n"
            "from collections import deque\n"
            "\n"
            "class SlidingWindow:\n"
            "    '''Keep most recent N tokens.''' \n"
            "    def __init__(self,\n"
            "                 max_tokens: int = 4000):\n"
            "        self.max_tokens = max_tokens\n"
            "        self.messages = deque()\n"
            "        self.current_tokens = 0\n"
            "    \n"
            "    def add_message(\n"
            "        self, role: str,\n"
            "        content: str,\n"
            "        tokens: int):\n"
            "        '''Add message, trim old.''' \n"
            "        self.messages.append({\n"
            "            'role': role,\n"
            "            'content': content,\n"
            "            'tokens': tokens})\n"
            "        self.current_tokens += tokens\n"
            "        \n"
            "        # Trim oldest until\n"
            "        #   within budget\n"
            "        while (self.current_tokens\n"
            "               > self.max_tokens\n"
            "               and len(\n"
            "                 self.messages) > 1):\n"
            "            old = (\n"
            "                self.messages\n"
            "                .popleft())\n"
            "            self.current_tokens -= (\n"
            "                old['tokens'])\n"
            "    \n"
            "    def get_messages(self):\n"
            "        '''Get context messages.''' \n"
            "        return list(\n"
            "            self.messages)\n"
            "\n"
            "# Usage\n"
            "window = SlidingWindow(\n"
            "    max_tokens=4000)\n"
            "window.add_message(\n"
            "    'user', 'Hello', 2)\n"
            "window.add_message(\n"
            "    'assistant', 'Hi!', 3)"
        )

    def get_dynamic_trim(self) -> str:
        """Dynamic context trimming."""
        return (
            "# === DYNAMIC TRIMMING ===\n"
            "import asyncio\n"
            "\n"
            "async def build_context(\n"
            "    system_prompt: str,\n"
            "    history: list[dict],\n"
            "    retrieved_chunks: list[str],\n"
            "    user_message: str,\n"
            "    max_tokens: int = 12000):\n"
            "    '''Build context with budget.''' \n"
            "    # Never trim system or user\n"
            "    system_tokens = count_tokens(\n"
            "        system_prompt)\n"
            "    user_tokens = count_tokens(\n"
            "        user_message)\n"
            "    \n"
            "    remaining = (\n"
            "        max_tokens\n"
            "        - system_tokens\n"
            "        - user_tokens)\n"
            "    \n"
            "    # Budget split: 50% history,\n"
            "    #   50% retrieved\n"
            "    history_budget = (\n"
            "        remaining // 2)\n"
            "    retrieved_budget = (\n"
            "        remaining // 2)\n"
            "    \n"
            "    # Trim history (keep recent)\n"
            "    trimmed_history = []\n"
            "    hist_tokens = 0\n"
            "    for msg in reversed(history):\n"
            "        msg_tokens = (\n"
            "            count_tokens(\n"
            "                msg['content']))\n"
            "        if (hist_tokens\n"
            "            + msg_tokens\n"
            "            > history_budget):\n"
            "            break\n"
            "        trimmed_history.insert(\n"
            "            0, msg)\n"
            "        hist_tokens += (\n"
            "            msg_tokens)\n"
            "    \n"
            "    # Trim retrieved (keep\n"
            "    #   most relevant)\n"
            "    trimmed_retrieved = []\n"
            "    ret_tokens = 0\n"
            "    for chunk in retrieved_chunks:\n"
            "        chunk_tokens = (\n"
            "            count_tokens(chunk))\n"
            "        if (ret_tokens\n"
            "            + chunk_tokens\n"
            "            > retrieved_budget):\n"
            "            break\n"
            "        trimmed_retrieved.append(\n"
            "            chunk)\n"
            "        ret_tokens += (\n"
            "            chunk_tokens)\n"
            "    \n"
            "    # Assemble in optimal order:\n"
            "    # system (start) -> retrieved\n"
            "    #   -> history -> user (end)\n"
            "    # High-value at start/end,\n"
            "    #   avoid middle placement\n"
            "    messages = [\n"
            "        {'role': 'system',\n"
            "         'content': system_prompt}]\n"
            "    \n"
            "    for chunk in (\n"
            "      trimmed_retrieved):\n"
            "        messages.append({\n"
            "            'role': 'system',\n"
            "            'content': chunk})\n"
            "    \n"
            "    for msg in (\n"
            "      trimmed_history):\n"
            "        messages.append(msg)\n"
            "    \n"
            "    messages.append({\n"
            "        'role': 'user',\n"
            "        'content': user_message})\n"
            "    \n"
            "    return messages"
        )

    def get_production_tips(self) -> dict:
        """Production tips."""
        return {
            "cost_linear": "Cost scales linearly with context length. 128K tokens costs 128x more than 1K. Every token costs money.",
            "latency_quadratic": "Latency increases with context length. Attention computation is quadratic. Long contexts are slow.",
            "lost_middle": "Lost-in-the-middle: U-shaped curve. Place critical info at start or end, not buried in middle.",
            "effective_limit": "Models break 30-40% before claimed limit. 200K model unreliable at ~130K. Test effective limit.",
            "sliding_short": "Sliding window for short chat sessions — predictable, cheap, discards early context.",
            "summarize_long": "Summarization for long-running sessions — compress older history, one extra LLM call, retains gist.",
            "rag_documents": "RAG for document-heavy workloads — embed, retrieve top-K chunks, secondary token budget cap.",
            "compose_strategies": "All three strategies compose: sliding window for recent, summarization for old, RAG for external.",
            "never_trim": "Never trim system prompt or current user message. These are anchoring positions.",
            "instrument_tokens": "Instrument token usage from day one. Token limits are hard walls, not soft guidelines.",
        }

Context Window Checklist

[ ] Understand context window is in-context working set for single inference call, not model memory
[ ] 2026 context windows: Gemini 3.1 Pro 2M, Llama 4 Scout 10M, Claude Sonnet 4 1M, GPT-5.2 400K, GPT-4o 128K
[ ] Models break 30-40% before claimed limit — 200K model unreliable at ~130K, test effective limit
[ ] Cost scales linearly with context length — 128K tokens costs 128x more than 1K tokens
[ ] Latency increases with context length — attention computation is quadratic in context length
[ ] Lost-in-the-middle: U-shaped performance curve — primacy (start) and recency (end) used reliably
[ ] Information in middle of long contexts is frequently ignored or misused
[ ] With 20 retrieved documents (~4K tokens), accuracy drops from 70-75% to 55-60%
[ ] Place most critical information at beginning or end of context, not buried in middle
[ ] Ranked retrieval still matters — concentrates high-value content in slots model uses best
[ ] Use sliding window for short chat sessions — keeps most recent N tokens, predictable, cheap
[ ] Use summarization for long-running sessions — compress older history, one extra LLM call, retains gist
[ ] Use RAG for document-heavy workloads — embed documents, retrieve top-K chunks by semantic similarity
[ ] Use dynamic trimming: score components by relevance, assign token budget per category, trim lowest first
[ ] Never trim system prompt or current user message — these are anchoring positions
[ ] All three strategies compose: sliding window for recent turns, summarization for older history, RAG for external documents
[ ] Token limits are hard walls, not soft guidelines — plan for them upfront
[ ] Count tokens using same tokenizer as the model — tiktoken for OpenAI, check provider docs for others
[ ] For Ollama: use /api/tokenize endpoint or local tokenizer for accurate counts
[ ] Token budget per category: system 2K, history 4K, retrieved 4K, user 1K, output 2K
[ ] Instrument token usage from day one — track input, output, and total tokens per request
[ ] Chunking: fixed-size (512 tokens, 50 overlap) — simple, fast, predictable
[ ] Chunking: semantic (split on paragraphs, sections, topic changes) — preserves coherent units
[ ] Chunking: structure-aware (respect document boundaries) — better for structured documents
[ ] Chunking: parent-document retrieval (retrieve small chunks, expand to parent context)
[ ] Chunk sizes: small (100-200) better precision, medium (300-500) balanced, large (600-1000) context but noise risk
[ ] Top_k is starting point, not ceiling — always apply secondary token budget cap
[ ] Context ordering: system prompt first, retrieved context middle, history, current user message last
[ ] Keep most relevant retrieved content away from middle of context (addressing lost-in-the-middle)
[ ] Long context vs RAG decision: corpus <1M + queries <100/day = long context
[ ] Corpus <1M + queries >1000/day = long context with prompt caching
[ ] Corpus >1M = use RAG retrieval
[ ] Corpus >10M = use hierarchical retrieval with multi-level index
[ ] Under ~100 queries against codebase: stuff it in context
[ ] Over ~1000 queries: build retrieval — RAG amortizes loading costs
[ ] Context caching reduces cost for repeated queries while keeping performance high
[ ] Prompt caching: cached reads cost 80-90% less, saves above ~22% hit rate
[ ] RULER evaluation: below 90% at full context is red flag, above 95% is production floor
[ ] Pick smallest context that holds your task — don't use 1M tokens when 10K suffices
[ ] Multi-granularity indexing: full documents, sections, and paragraphs — retrieve right granularity
[ ] Read RAG vs fine-tuning for knowledge approach
[ ] Read prompt engineering for prompt design
[ ] Read choose LLM for model context windows
[ ] Read cost estimation for token budgeting
[ ] Test: token count matches API tokenizer exactly
[ ] Test: sliding window correctly trims oldest messages when over budget
[ ] Test: dynamic trimming never removes system prompt or current user message
[ ] Test: context ordering places high-value content at start and end
[ ] Test: chunking produces chunks within expected token range
[ ] Test: long context vs RAG decision matches corpus size and query frequency
[ ] Document context strategy, token budgets, chunking approach, ordering rules, effective limits

FAQ

What is the lost-in-the-middle problem in LLM context windows?

Models attend to beginning and end of context but miss information in the middle. Tanuj Garg: "Lost in the middle is well-documented: models disproportionately attend to beginning and end of context, missing information in the middle. Information at beginning (primacy) and end (recency) is used reliably. Information in middle is frequently ignored or misused." Zylos: "U-shaped performance curve. With just 20 retrieved documents (~4K tokens), accuracy drops from 70-75% to 55-60%. Scale amplifies problem — with millions of tokens, middle content becomes statistically insignificant." Swfte AI: "Model accuracy on retrieval and reasoning tasks is highest when relevant information is at start or end of context, falls off in middle. Place most critical information at start or end, not buried in middle." Problem: (1) U-shaped curve: primacy and recency effects. (2) 20 docs (~4K tokens): accuracy drops 70% to 55%. (3) Millions of tokens: middle content statistically insignificant. (4) Solution: place critical info at start or end. (5) Ranked retrieval concentrates high-value content in best slots.

How do you manage context window limits in production LLM applications?

Use sliding window, summarization, RAG retrieval, and dynamic trimming strategies. Tanuj Garg: "Strategies: (1) sliding window keeps most recent N tokens. (2) summarization replaces older content with compressed summary. (3) RAG moves data to vector DB. (4) dynamic trimming: score components by relevance, assign token budgets, trim lowest-scored first, never trim system prompt or current message." DEV Community: "Sliding window for short sessions. Summarization for long-running sessions. Chunked retrieval for document-heavy workloads. All three compose: sliding window for recent turns, summarization for older history, RAG for external documents. Instrument token usage from day one." Google Cloud: "Strategies: drop old messages, summarize previous content, use RAG with semantic search, use filters to remove unnecessary text." Management: (1) Sliding window: most recent N tokens, predictable, cheap. (2) Summarization: compress older history, one extra LLM call. (3) RAG: embed documents, retrieve top-K chunks. (4) Dynamic trimming: score by relevance, budget per category, trim lowest first. (5) Never trim system prompt or current user message.

Should you use long context or RAG for your LLM application?

Depends on corpus size and query frequency. Swfte AI: "Decision tree: corpus <1M tokens and queries <100/day, use long context. Corpus <1M and queries >1000/day, use long context with prompt caching. Corpus >1M tokens, use retrieval. Corpus >10M tokens, use hierarchical retrieval. Under ~100 queries against codebase, just stuff it. Over ~1000, build retrieval." Google Cloud: "Larger context windows unlock new use cases. But cost scales linearly — 128K tokens costs 128x more than 1K. For 100 pieces of information at 99% performance, need 100 requests. Context caching reduces cost while keeping performance." Tanuj Garg: "Cost scales linearly with context length. Latency increases — attention is quadratic. Quality degrades in long contexts." Decision: (1) Corpus <1M, queries <100/day: long context. (2) Corpus <1M, queries >1000/day: long context + prompt caching. (3) Corpus >1M: RAG retrieval. (4) Corpus >10M: hierarchical retrieval. (5) Under 100 queries: stuff context. (6) Over 1000 queries: build retrieval.

What are the best chunking strategies for LLM context management?

Fixed-size, semantic, structure-aware, and parent-document retrieval. Tanuj Garg: "Fixed-size: 512 tokens with 50-token overlap. Simple, fast, predictable. Overlap reduces chance of splitting meaning. Semantic: split on logical boundaries — paragraphs, sections, topic changes. More complex but preserves coherent units. Multi-granularity: index at full documents, sections, and paragraphs. Retrieve right granularity for query." Zylos: "Structure-aware chunking: respect document boundaries. Semantic chunking: break at meaningful boundaries. Parent-document retrieval: retrieve small chunks but expand to parent context. Overlap strategy: maintain continuity. Small chunks (100-200): better precision. Medium (300-500): balanced. Large (600-1000): better context, risk of noise." DEV Community: "Top_k is starting point, not ceiling. Always apply secondary token budget — chunk sizes vary, fixed top_k can blow context limit." Chunking: (1) Fixed: 512 tokens, 50 overlap. (2) Semantic: paragraph/section boundaries. (3) Structure-aware: respect document structure. (4) Parent-document: small chunks, expand to parent. (5) Small (100-200): precision. (6) Medium (300-500): balanced. (7) Large (600-1000): context, noise risk.

How do you count tokens accurately for LLM context management?

Use the same tokenizer as the model. tiktoken for OpenAI, check provider docs for others. DEV Community: "Token limits are hard walls, not soft guidelines. Count tokens the same way the API does. tiktoken handles this for OpenAI-compatible models. For other providers, check their tokenizer documentation." Tanuj Garg: "Implement dynamic trimming: score each context component by relevance, assign token budget to each category, trim greedily from lowest-scored first. Never trim system prompt or current user message." Swfte AI: "Context window is in-context working set for single inference call, measured in tokens, bounded by hard architectural ceiling. Pick smallest context that holds your task." Token counting: (1) Use model-specific tokenizer — tiktoken for OpenAI. (2) For Ollama: count via API or local tokenizer. (3) Token limits are hard walls — plan for them. (4) Count system prompt + history + retrieved context + user message + expected output. (5) Budget per category: system 2K, history 4K, RAG 4K, output 2K. (6) Instrument token usage from day one.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →