RAG Chunk Size: Optimal Token Count for Retrieval Accuracy
TL;DR — RAG chunk size is the single most impactful parameter for retrieval accuracy. Nandigam: "Factoid queries: 256-512 tokens. Analytical queries: 512-1024 tokens. Getting chunk size wrong by one bracket degrades context precision by 15-30%. Practical default: 512 tokens." Vecta: "Fixed 512: 67% accuracy, 0.85 groundedness. Fixed 1024: 61% accuracy but 0.88 doc F1. Larger chunks = more noise, less precision." Inventiple: "512-token chunks + Cohere reranker beats 1024-token without reranking. 256-token + reranking beats 1024-token without." Firecrawl: "Factoid: 256-512 tokens. Analytical: 1024+. NVIDIA page-level won at 0.648 accuracy." LangCopilot: "Align chunk size with embedding model optimal input — typically 256, 512, or 1024 tokens. Sweep to find optimal." Learn more with chunking strategies, what is RAG, build from scratch, and embedding models.
MachineLearningPlus frames the tradeoff: "Optimal chunk size for RAG systems typically ranges from 128-512 tokens, with smaller chunks (128-256 tokens) excelling at precise fact-based queries while larger chunks (256-512 tokens) provide better context for complex reasoning tasks. The key is balancing retrieval precision with context retention."
Nandigam quantifies the stakes: "NVIDIA's research found that factoid queries perform best at 256-512 tokens, while multi-hop analytical queries benefit from 512-1,024 tokens. Getting this wrong by one bracket degrades context precision by 15-30%. The practical default validated across multiple 2025-2026 benchmarks is 512 tokens."
Chunk Size Impact Architecture
(factoid vs analytical)"] D2["Embedding model
optimal input size"] D3["Reranking
(smaller + rerank wins)"] D4["Context window
of generation LLM"] D5["Cost budget
(more chunks = more cost)"] end Decision --> Small Decision --> Medium Decision --> Large Small --> Eval["Sweep + Evaluate
recall@k, precision@k,
accuracy, RAGAS"] Medium --> Eval Large --> Eval Eval --> Optimal["Optimal Chunk Size
for your query mix"]
Chunk Size Benchmark Comparison
| Chunk Size | Accuracy | Doc F1 | Page F1 | Groundedness | Best For | k (top_k) |
|---|---|---|---|---|---|---|
| 128-256 | High precision | Lower | High | Medium | Factoid queries | 5-10 |
| 512 | 67-69% | 0.85-0.86 | 0.88-0.92 | 0.81-0.85 | General purpose | 5 |
| 1024 | 61% | 0.88 | 0.72 | 0.86 | Document retrieval | 3 |
| Page-level | 0.648 (NVIDIA) | — | — | — | PDFs, paginated | 2 |
Query Type to Chunk Size Mapping
| Query Type | Optimal Chunk Size | Rationale | Example |
|---|---|---|---|
| Factoid | 256-512 tokens | Precise facts need focused chunks | "What is the EU AI Act deadline?" |
| Analytical | 512-1024 tokens | Multi-hop reasoning needs context | "Compare RAG vs fine-tuning costs" |
| Summarization | 1024+ tokens | Need broad context per chunk | "Summarize this document" |
| Q&A (specific) | 128-256 tokens | Exact answer in small chunk | "What is the penalty for non-compliance?" |
| Q&A (complex) | 512 tokens | Balance precision and context | "How does GDPR Article 22 apply to AI?" |
| Code search | Function-level | Preserve code boundaries | "Find the authentication function" |
Implementation
import time
from dataclasses import dataclass
from typing import Optional
from enum import Enum
class QueryType(Enum):
FACTOID = "factoid"
ANALYTICAL = "analytical"
SUMMARIZATION = "summarization"
QA_SPECIFIC = "qa_specific"
QA_COMPLEX = "qa_complex"
CODE_SEARCH = "code_search"
@dataclass
class RAGChunkSizeOptimizer:
"""Find optimal chunk size for RAG retrieval accuracy."""
# Benchmark-validated defaults
DEFAULT_CHUNK_SIZE = 512
DEFAULT_OVERLAP = 64 # ~12.5% of 512
# Query type to chunk size mapping (NVIDIA benchmarks)
QUERY_TYPE_SIZES = {
QueryType.FACTOID: 256,
QueryType.ANALYTICAL: 1024,
QueryType.SUMMARIZATION: 1024,
QueryType.QA_SPECIFIC: 128,
QueryType.QA_COMPLEX: 512,
QueryType.CODE_SEARCH: 512, # function-level
}
# Embedding model optimal input sizes
EMBEDDING_MODEL_SIZES = {
"text-embedding-3-small": 512,
"text-embedding-3-large": 512,
"text-embedding-ada-002": 256,
"paraphrase-multilingual-MiniLM-L12-v2": 256,
"bge-large-en-v1.5": 512,
"stella_en_1.5B_v5": 512,
"nomic-embed-text-v1.5": 512,
}
def __init__(self, embedder, vdb, llm, reranker=None):
self.embedder = embedder
self.vdb = vdb
self.llm = llm
self.reranker = reranker
def recommend_chunk_size(self, query_type: QueryType,
embedding_model: str = "text-embedding-3-small",
use_reranking: bool = True) -> dict:
"""Recommend chunk size based on query type and embedding model."""
# Start with query type recommendation
base_size = self.QUERY_TYPE_SIZES.get(
query_type, self.DEFAULT_CHUNK_SIZE)
# Adjust for embedding model optimal input
model_optimal = self.EMBEDDING_MODEL_SIZES.get(
embedding_model, self.DEFAULT_CHUNK_SIZE)
# If reranking is used, smaller chunks are better
# (256 + rerank beats 1024 without rerank)
if use_reranking and base_size > 512:
adjusted_size = 512
adjustment_reason = (
"Reduced to 512 because reranking compensates "
"for smaller chunk size. 256-token + reranker "
"beats 1024-token without reranker."
)
else:
adjusted_size = base_size
adjustment_reason = "No adjustment needed"
# Overlap: 10-20% of chunk size
overlap = max(32, adjusted_size // 8)
return {
"query_type": query_type.value,
"embedding_model": embedding_model,
"use_reranking": use_reranking,
"recommended_chunk_size": adjusted_size,
"recommended_overlap": overlap,
"overlap_pct": round(overlap / adjusted_size * 100, 1),
"base_size_from_query_type": base_size,
"model_optimal_input": model_optimal,
"adjustment_reason": adjustment_reason,
"rationale": self._get_rationale(
query_type, adjusted_size, use_reranking),
}
async def sweep_chunk_sizes(self, documents: list,
test_queries: list,
chunk_sizes: list = None,
overlaps: list = None) -> dict:
"""Sweep chunk sizes and measure retrieval metrics."""
if chunk_sizes is None:
chunk_sizes = [128, 256, 512, 1024]
if overlaps is None:
overlaps = [0, 50, 100]
results = []
for size in chunk_sizes:
for overlap in overlaps:
# Index documents with this chunk size
await self._index_with_size(documents, size, overlap)
# Test retrieval for each query
metrics = {
"chunk_size": size,
"overlap": overlap,
"total_chunks": 0,
"recall_at_5": 0.0,
"precision_at_5": 0.0,
"avg_latency_ms": 0.0,
"avg_context_tokens": 0.0,
}
latencies = []
context_tokens = []
for query_data in test_queries:
query = query_data["query"]
expected = query_data.get("expected_sources", [])
start = time.time()
results_for_query = await self._retrieve(
query, top_k=5)
latency = (time.time() - start) * 1000
latencies.append(latency)
# Calculate recall@5
retrieved_sources = [
r["payload"].get("source", "")
for r in results_for_query
]
if expected:
relevant_retrieved = len(
set(retrieved_sources) & set(expected))
recall = relevant_retrieved / len(expected)
metrics["recall_at_5"] += recall
context_tokens.append(
sum(len(r["payload"]["content"].split())
for r in results_for_query))
num_queries = len(test_queries)
metrics["recall_at_5"] = round(
metrics["recall_at_5"] / num_queries, 3) if num_queries else 0
metrics["avg_latency_ms"] = round(
sum(latencies) / num_queries, 1) if num_queries else 0
metrics["avg_context_tokens"] = round(
sum(context_tokens) / num_queries, 1) if num_queries else 0
results.append(metrics)
# Find best configuration
best = max(results, key=lambda x: x["recall_at_5"])
return {
"sweep_results": results,
"best_config": best,
"recommendation": (
f"Use chunk_size={best['chunk_size']} with "
f"overlap={best['overlap']} for best recall@5 "
f"({best['recall_at_5']})"
),
}
async def _index_with_size(self, documents: list,
chunk_size: int, overlap: int):
"""Index documents with a specific chunk size."""
# Clear existing collection
await self.vdb.recreate_collection("sweep_test")
for doc in documents:
text = " ".join(doc["content"].split())
words = text.split()
chunks = []
start = 0
while start < len(words):
end = min(start + chunk_size, len(words))
chunks.append(" ".join(words[start:end]))
if end == len(words):
break
start = end - overlap
embeddings = await self.embedder.embed_batch(chunks)
points = [
{"id": f"{doc['id']}_{i}", "vector": emb,
"payload": {"content": chunk, "source": doc.get("source", "")}}
for i, (chunk, emb) in enumerate(zip(chunks, embeddings))
]
await self.vdb.upsert("sweep_test", points)
async def _retrieve(self, query: str, top_k: int = 5) -> list:
"""Retrieve top_k results for a query."""
query_embedding = await self.embedder.embed(query)
return await self.vdb.search(
collection_name="sweep_test",
vector=query_embedding,
limit=top_k,
)
def _get_rationale(self, query_type: QueryType,
chunk_size: int, use_reranking: bool) -> str:
rationales = {
QueryType.FACTOID: (
f"Factoid queries need precise chunks. {chunk_size} tokens "
"provides focused retrieval for exact facts."
),
QueryType.ANALYTICAL: (
f"Analytical queries need context. {chunk_size} tokens "
"provides enough context for multi-hop reasoning."
),
QueryType.QA_SPECIFIC: (
f"Specific Q&A needs small chunks. {chunk_size} tokens "
"maximizes precision for exact answers."
),
QueryType.QA_COMPLEX: (
f"Complex Q&A balances precision and context. "
f"{chunk_size} tokens is the production default."
),
}
base = rationales.get(query_type, f"{chunk_size} tokens recommended.")
if use_reranking:
base += " Reranking compensates for smaller chunk size."
return base
RAG Chunk Size Checklist
- [ ] Start with 512 tokens as the production default
- [ ] Set overlap to 10-20% of chunk size (50-100 tokens for 512)
- [ ] Use 256-512 tokens for factoid queries (precise facts)
- [ ] Use 512-1024 tokens for analytical queries (multi-hop reasoning)
- [ ] Use 128-256 tokens for specific Q&A (exact answers)
- [ ] Use 1024+ tokens for summarization (broad context)
- [ ] Use function-level chunking for code search
- [ ] Align chunk size with embedding model optimal input (not max context)
- [ ] text-embedding-3-small: optimal at 512 tokens (max 8191)
- [ ] MiniLM: optimal at 256 tokens (max 256)
- [ ] BGE-large: optimal at 512 tokens (max 512)
- [ ] Getting chunk size wrong by one bracket degrades precision by 15-30%
- [ ] Smaller chunks + reranking beats larger chunks without reranking
- [ ] 256-token + Cohere reranker beats 1024-token without reranker
- [ ] 512-token + reranker is the production sweet spot
- [ ] Larger chunks (1024) = more noise, less precision, but higher doc F1
- [ ] Smaller chunks (128-256) = high precision, less context, more chunks
- [ ] More chunks = higher embedding cost (API or compute)
- [ ] More chunks = more storage in vector database
- [ ] Sweep chunk sizes: 128, 256, 512, 1024 on your actual data
- [ ] Sweep overlap: 0, 50, 100 for each chunk size
- [ ] Measure recall@k for each configuration
- [ ] Measure precision@k for each configuration
- [ ] Measure accuracy for each configuration
- [ ] Measure RAGAS faithfulness (target: 0.8+)
- [ ] Measure avg latency per configuration
- [ ] Measure avg context tokens per configuration
- [ ] Create test set of 20+ questions with expected sources
- [ ] Pick the configuration with best metrics for your query mix
- [ ] Re-evaluate when query patterns change
- [ ] Re-evaluate when document corpus changes
- [ ] Re-evaluate when switching embedding models
- [ ] Consider query type distribution — if 80% factoid, optimize for factoid
- [ ] Consider using different chunk sizes for different document types
- [ ] Consider parent-document retrieval (small chunks for retrieval, large for generation)
- [ ] Consider sentence-window retrieval (retrieve sentence, return surrounding window)
- [ ] NVIDIA page-level chunking: 0.648 accuracy for PDFs
- [ ] Recursive 512: 69% accuracy, 0.86 doc F1, 0.92 page F1
- [ ] Fixed 512: 67% accuracy, 0.85 doc F1, 0.85 groundedness
- [ ] Fixed 1024: 61% accuracy, 0.88 doc F1 (best for doc-level retrieval)
- [ ] Read chunking strategies for splitting methods
- [ ] Read what is RAG for architecture
- [ ] Build RAG from scratch with chunking
- [ ] Choose embedding model to match chunk size
- [ ] Understand embeddings for semantic quality
- [ ] Add hybrid search after chunking
- [ ] Add reranking to compensate for smaller chunks
- [ ] Prevent hallucinations with proper chunking
- [ ] Evaluate with RAG metrics
- [ ] Test: chunk size sweep produces expected recall@k curves
- [ ] Test: smaller chunks + reranking beats larger chunks without
- [ ] Test: overlap reduces boundary retrieval failures
- [ ] Test: embedding model optimal input aligns with chunk size
- [ ] Document optimal chunk size, overlap, and sweep results
FAQ
What is the optimal chunk size for RAG?
The optimal chunk size depends on query type. MachineLearningPlus: "Optimal chunk size typically ranges from 128-512 tokens. Smaller chunks (128-256) excel at precise fact-based queries. Larger chunks (256-512) provide better context for complex reasoning. The key is balancing retrieval precision with context retention." Nandigam: "NVIDIA found factoid queries perform best at 256-512 tokens, while multi-hop analytical queries benefit from 512-1,024 tokens. Getting this wrong by one bracket degrades context precision by 15-30%. The practical default validated across multiple 2025-2026 benchmarks is 512 tokens." Firecrawl: "Query type affected optimal chunk size: factoid queries performed best with 256-512 tokens, analytical queries needed 1024+ tokens." LangCopilot: "Chunk size: 256-512 tokens (sweep to find optimal). Align with your embedding model optimal input size, typically 256, 512, or 1024 tokens."
How does chunk size affect RAG retrieval accuracy?
Chunk size affects retrieval precision and context retention in opposite directions — smaller chunks increase precision but lose context, larger chunks provide context but introduce noise. Vecta: "Fixed 512 with 5 chunks: 67% accuracy, 0.85 doc F1, 0.85 groundedness. Fixed 1024 with 3 chunks: 61% accuracy, 0.88 doc F1, 0.86 groundedness. Larger chunks (1024) had higher document F1 (0.88 vs 0.85) but lower accuracy (61% vs 67%) — more noise, less precision." arxiv: "We experiment with multiple fixed chunk sizes: 64, 128, 256, 512, and 1024 tokens. Each dataset varies in question length, document length, answer locality, and vocabulary diversity, which influences chunking effectiveness." Nandigam: "Getting chunk size wrong by one bracket degrades context precision by 15-30%." Inventiple: "512-token chunks with Cohere reranker outperforms naive 1024-token retrieval consistently. 256-token chunks with reranking consistently outperformed 1024-token chunks without reranking." Smaller chunks + reranking beats larger chunks without reranking.
Should you use 256, 512, or 1024 token chunks?
Use 512 tokens as the default, then sweep 256 and 1024 on your data. MachineLearningPlus: "128-256 tokens for precise fact-based queries, 256-512 tokens for complex reasoning, 512+ for multi-hop analytical queries." Firecrawl: "Factoid queries: 256-512 tokens. Analytical queries: 1024+ tokens. NVIDIA page-level chunking won with 0.648 accuracy." Nandigam: "Factoid: 256-512 tokens. Multi-hop analytical: 512-1024 tokens. Practical default: 512 tokens." Vecta: "Recursive 512 won 69% accuracy. Fixed 512: 67%. Fixed 1024: 61% accuracy but 0.88 doc F1." Inventiple: "512-token chunks + Cohere reranker beats 1024-token without reranking." Decision: start at 512, sweep 256/512/1024, measure recall@k and accuracy, pick the winner for your query mix.
How does embedding model context window affect chunk size?
Chunk size should align with your embedding model's optimal input length. LangCopilot: "A good starting point often aligns with your embedding model optimal input size, typically 256, 512, or 1024 tokens. Too small, and your chunks will not have enough context. Too large, and you introduce noise and increase API costs." arxiv: "We experiment with multiple fixed chunk sizes using stella_en_1.5B_v5 embedding model." Common embedding model context windows: text-embedding-3-small: 8191 tokens, text-embedding-3-large: 8191 tokens, MiniLM: 256 tokens, BGE-large: 512 tokens, stella_en_1.5B_v5: 8192 tokens. While models accept long inputs, optimal embedding quality is typically at 256-512 tokens — longer inputs dilute semantic signal. Match chunk size to the sweet spot of your embedding model, not its maximum context window.
How do you find the optimal chunk size for your RAG system?
Sweep chunk sizes on your actual data and measure retrieval metrics. LangCopilot: "Chunk size: 256-512 tokens (sweep to find optimal). Run a parameter sweep: chunk_size in [128, 256, 512, 1024], overlap in [0, 50, 100]. Measure recall@k and precision@k for each combination." Firecrawl: "Test 2-3 strategies on your actual documents and queries. There is no single best strategy." Vecta: "We benchmarked 7 strategies on real-world data. Most best practice advice was wrong for us." Inventiple: "Evaluate before shipping — RAGAS faithfulness below 0.8 means hallucination risk. Evaluate before and after every significant change." Process: (1) Create test set of 20+ questions. (2) Sweep chunk sizes: 128, 256, 512, 1024. (3) Sweep overlap: 0, 50, 100. (4) Measure recall@k, precision@k, accuracy, RAGAS faithfulness. (5) Pick the configuration with best metrics for your query mix. (6) Re-evaluate when query patterns or documents change.
Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →