What is RAG? A Developer Guide to Retrieval-Augmented Generation

TL;DR — RAG (Retrieval-Augmented Generation) connects LLMs to external knowledge bases so they reference authoritative data before generating responses. IBM: "RAG is an architecture for optimizing AI model performance by connecting it with external knowledge bases." AWS: "RAG references an authoritative knowledge base outside training data before generating a response." Two pipelines: indexing (collect → normalize → chunk → embed → store) and retrieval+generation (query → embed → search → rerank → generate). Glukhov: "Modern RAG includes query rewriting, hybrid search (BM25 + vector), cross-encoder reranking, multi-stage retrieval, and evaluation." Inventiple: "Qdrant is the 2026 production default. 512-token chunks + Cohere reranker outperforms 1024-token without reranking. RAGAS faithfulness below 0.8 means hallucination risk is unacceptable." NerdLevelTech: "RAG lets you connect any LLM to your documents, databases, and knowledge bases instead of expensive fine-tuning." RAG solves: hallucination, stale knowledge, no private data. Learn more with how RAG works, build RAG from scratch, chunking strategies, and embeddings.

AWS defines RAG: "Retrieval-Augmented Generation is the process of optimizing the output of a large language model, so it references an authoritative knowledge base outside of its training data sources before generating a response. Large Language Models are trained on vast volumes of data and use billions of parameters to generate original output for tasks like answering questions, translating languages, and completing sentences. RAG extends the already powerful capabilities of LLMs to specific domains or an organization's internal knowledge base, without the need to retrain the model."

Data Science Collective explains the problem RAG solves: "RAG solves this by grounding probabilistic generation in verifiable external data, effectively mitigating hallucinations."

RAG Architecture

RAG Components Comparison

Vector Database Selection

Implementation

import json
import hashlib
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Optional

@dataclass
class RAGSystem:
    """Retrieval-Augmented Generation system with indexing and retrieval."""

    def __init__(self, embedding_model, vector_db, llm,
                 reranker=None, chunk_size=512, chunk_overlap=64):
        self.embedding_model = embedding_model
        self.vector_db = vector_db
        self.llm = llm
        self.reranker = reranker
        self.chunk_size = chunk_size
        self.chunk_overlap = chunk_overlap

    async def index_documents(self, documents: list,
                              metadata: dict = None) -> dict:
        """Index documents: collect, normalize, chunk, embed, store."""
        total_chunks = 0

        for doc in documents:
            # Step 1-2: Normalize (extract text, clean)
            text = self._normalize_text(doc["content"])

            # Step 3: Chunk
            chunks = self._chunk_text(
                text, self.chunk_size, self.chunk_overlap)

            # Step 4: Embed
            embeddings = await self.embedding_model.embed_batch(chunks)

            # Step 5: Store with metadata
            points = []
            for i, (chunk, embedding) in enumerate(zip(chunks, embeddings)):
                chunk_id = hashlib.sha256(
                    f"{doc['id']}:{i}".encode()).hexdigest()[:16]
                points.append({
                    "id": chunk_id,
                    "vector": embedding,
                    "payload": {
                        "content": chunk,
                        "doc_id": doc["id"],
                        "chunk_index": i,
                        "source": doc.get("source", "unknown"),
                        "title": doc.get("title", ""),
                        "metadata": metadata or {},
                        "indexed_at": datetime.now(timezone.utc).isoformat(),
                    },
                })
                total_chunks += 1

            await self.vector_db.upsert(
                collection_name="knowledge_base",
                points=points,
            )

        return {
            "documents_indexed": len(documents),
            "total_chunks": total_chunks,
            "chunk_size": self.chunk_size,
            "chunk_overlap": self.chunk_overlap,
        }

    async def retrieve(self, query: str, top_k: int = 5,
                       use_hybrid: bool = True,
                       use_reranking: bool = True) -> dict:
        """Retrieve relevant chunks for a query."""
        # Step 1: Embed query
        query_embedding = await self.embedding_model.embed(query)

        # Step 2: Search (hybrid or dense-only)
        if use_hybrid:
            # Hybrid: vector search + BM25 + RRF fusion
            vector_results = await self.vector_db.search(
                collection_name="knowledge_base",
                vector=query_embedding,
                limit=top_k * 3,  # over-fetch for reranking
            )
            bm25_results = await self.vector_db.search(
                collection_name="knowledge_base",
                query=query,  # BM25 keyword search
                limit=top_k * 3,
            )
            results = self._rrf_fusion(
                vector_results, bm25_results, bm25_weight=0.3)
        else:
            results = await self.vector_db.search(
                collection_name="knowledge_base",
                vector=query_embedding,
                limit=top_k * 3,
            )

        # Step 3: Rerank with cross-encoder
        if use_reranking and self.reranker:
            results = await self.reranker.rerank(
                query=query,
                documents=[r["payload"]["content"] for r in results],
                top_k=top_k,
            )
            results = [results[i] for i in range(min(top_k, len(results)))]
        else:
            results = results[:top_k]

        # Step 4: Assemble context
        context = self._assemble_context(results)

        return {
            "query": query,
            "retrieved_chunks": len(results),
            "context": context,
            "sources": [
                {"doc_id": r["payload"]["doc_id"],
                 "chunk_index": r["payload"]["chunk_index"],
                 "source": r["payload"]["source"]}
                for r in results
            ],
        }

    async def generate(self, query: str, top_k: int = 5) -> dict:
        """Retrieve and generate a response."""
        # Retrieve
        retrieval = await self.retrieve(query, top_k=top_k)

        # Generate with grounded prompt
        prompt = self._build_grounded_prompt(
            query, retrieval["context"])
        response = await self.llm.generate(prompt)

        return {
            "query": query,
            "answer": response,
            "sources": retrieval["sources"],
            "retrieved_chunks": retrieval["retrieved_chunks"],
        }

    def _chunk_text(self, text: str, chunk_size: int,
                    overlap: int) -> list:
        """Chunk text into overlapping segments."""
        words = text.split()
        chunks = []
        start = 0

        while start < len(words):
            end = min(start + chunk_size, len(words))
            chunk = " ".join(words[start:end])
            chunks.append(chunk)
            if end == len(words):
                break
            start = end - overlap

        return chunks

    def _normalize_text(self, text: str) -> str:
        """Normalize text: clean boilerplate, deduplicate."""
        # Remove excessive whitespace
        text = " ".join(text.split())
        return text

    def _rrf_fusion(self, vector_results: list,
                    bm25_results: list,
                    bm25_weight: float = 0.3,
                    k: int = 60) -> list:
        """Reciprocal Rank Fusion of vector and BM25 results."""
        scores = {}
        results_map = {}

        for rank, result in enumerate(vector_results):
            doc_id = result["id"]
            scores[doc_id] = scores.get(doc_id, 0) + \
                (1 - bm25_weight) / (k + rank + 1)
            results_map[doc_id] = result

        for rank, result in enumerate(bm25_results):
            doc_id = result["id"]
            scores[doc_id] = scores.get(doc_id, 0) + \
                bm25_weight / (k + rank + 1)
            results_map[doc_id] = result

        sorted_ids = sorted(scores, key=scores.get, reverse=True)
        return [results_map[doc_id] for doc_id in sorted_ids]

    def _assemble_context(self, results: list) -> str:
        """Assemble context from retrieved chunks with citations."""
        context_parts = []
        for i, result in enumerate(results):
            content = result["payload"]["content"]
            source = result["payload"].get("source", "unknown")
            context_parts.append(f"[{i+1}] ({source})\n{content}")
        return "\n\n".join(context_parts)

    def _build_grounded_prompt(self, query: str,
                               context: str) -> str:
        """Build a grounded prompt with context and refusal behavior."""
        return (
            f"Answer the question based on the following context. "
            f"If the context does not contain enough information "
            f"to answer the question, say 'I don't have enough "
            f"information to answer this question.'\n\n"
            f"Context:\n{context}\n\n"
            f"Question: {query}\n\n"
            f"Answer (cite sources using [1], [2], etc.):"
        )

RAG Developer Checklist

• [ ] Understand RAG solves: hallucination, stale knowledge, no private data
• [ ] Choose embedding model (text-embedding-3-small, MiniLM, BGE)
• [ ] Choose vector database (Qdrant for production, Chroma for prototyping)
• [ ] Set chunk size (512 tokens production default) and overlap (64-128)
• [ ] Implement indexing pipeline: collect → normalize → chunk → embed → store
• [ ] Store metadata with each chunk (doc_id, source, title, timestamp)
• [ ] Use versioned embeddings (track which model generated each vector)
• [ ] Implement reindexing strategy when embedding model or chunking changes
• [ ] Implement query embedding using the same model as indexing
• [ ] Start with dense vector search for prototyping
• [ ] Add hybrid search (BM25 + vector + RRF fusion) for production
• [ ] Add metadata filtering (filter by source, date, category)
• [ ] Add cross-encoder reranking (Cohere, ms-marco-MiniLM)
• [ ] Over-fetch candidates (3x top_k) before reranking
• [ ] Implement context assembly with deduplication and citation numbering
• [ ] Build grounded prompt with refusal behavior for insufficient context
• [ ] Log retrieval set, reranked set, final context, latency, and cost
• [ ] Set up RAGAS evaluation: faithfulness (target 0.8+), answer relevancy
• [ ] Track context precision and context recall metrics
• [ ] Track recall@k and precision@k continuously
• [ ] Create test set of 20+ questions for evaluation
• [ ] Evaluate before and after every significant change
• [ ] Add semantic caching for repeated queries
• [ ] Implement query rewriting for complex questions
• [ ] Consider GraphRAG for relationship-heavy knowledge
• [ ] Consider multi-modal RAG for documents with images/tables
• [ ] Add access controls — filter chunks by user permissions
• [ ] Implement streaming responses (SSE) for better UX
• [ ] Add automatic language detection for multilingual corpora
• [ ] Use MMR (Maximum Marginal Relevance) for result diversity
• [ ] Choose self-hosted vector DB for data sovereignty (Qdrant, pgvector)
• [ ] Consider pgvector if already on PostgreSQL (<10M vectors)
• [ ] Consider Pinecone for zero-ops managed production
• [ ] Avoid Chroma for production workloads above 1M vectors
• [ ] Benchmark chunk sizes: 512-token + reranker beats 1024-token without
• [ ] Benchmark hybrid vs dense-only: hybrid catches keyword queries
• [ ] Integrate with how RAG works
• [ ] Build RAG pipeline from scratch
• [ ] Optimize chunking strategies
• [ ] Understand embeddings
• [ ] Compare RAG vs fine-tuning
• [ ] Add hybrid search for production
• [ ] Add reranking for precision
• [ ] Apply zero-trust to RAG access
• [ ] Log all RAG operations in audit logs
• [ ] Test: indexing produces correct chunk count and embeddings
• [ ] Test: hybrid search returns both semantic and keyword matches
• [ ] Test: reranking improves top-k precision
• [ ] Test: grounded prompt refuses when context is insufficient
• [ ] Test: RAGAS faithfulness meets 0.8+ threshold
• [ ] Document RAG architecture, embedding model version, and evaluation results

FAQ

What is RAG (Retrieval-Augmented Generation)?

RAG is an architecture that connects LLMs to external knowledge bases so they can reference authoritative data before generating responses. IBM: "Retrieval augmented generation is an architecture for optimizing the performance of an AI model by connecting it with external knowledge bases. The integration layer is the center of the RAG architecture." AWS: "RAG is the process of optimizing the output of a large language model, so it references an authoritative knowledge base outside of its training data sources before generating a response." Pinecone: "RAG blends the broad capabilities of foundation models with your company authoritative and proprietary knowledge." RAG solves three LLM problems: (1) Hallucination — grounds responses in verified data. (2) Stale knowledge — retrieves current information. (3) No private data — connects to your documents, databases, and APIs.

What are the components of a RAG system?

A RAG system has two pipelines: indexing (offline) and retrieval+generation (runtime). NerdLevelTech: "Every RAG system has two phases: indexing (offline) and retrieval + generation (runtime)." Glukhov: Indexing pipeline: (1) Collect sources — docs, tickets, web pages, PDFs, code. (2) Normalize — extract text, clean, de-duplicate. (3) Chunk — choose strategy, overlap, metadata. (4) Embed — versioned embeddings. (5) Upsert into index — vector store with metadata. Retrieval pipeline: (1) Parse/rewrite query. (2) Retrieve candidates — vector or hybrid search with metadata filtering. (3) Rerank top-K with cross-encoder. (4) Assemble context — dedupe, order by relevance, add citations. (5) Generate with grounded prompt. (6) Log retrieval set, context, latency, cost. (7) Evaluate online/offline. DigitalOcean: "The indexing pipeline prepares the knowledge base. The retrieval pipeline operates during inference — the query is converted to an embedding, the vector database searches for similar chunks, and the LLM generates using context."

How does RAG compare to fine-tuning?

RAG retrieves external knowledge at inference time while fine-tuning bakes knowledge into model weights. RAG is cheaper, more current, and more transparent. Pinecone: "RAG blends the broad capabilities of foundation models with your company knowledge." RAG advantages over fine-tuning: (1) No retraining — update the knowledge base, not the model. (2) Current knowledge — retrieve latest data anytime. (3) Source attribution — cite which document the answer came from. (4) Privacy control — keep data in your vector database, not in model weights. (5) Cost — embedding + retrieval is cheaper than GPU training. (6) Scalability — add millions of documents without retraining. Fine-tuning advantages: (1) Style and tone adaptation. (2) Domain-specific language patterns. (3) Reduced prompt length. Best practice: use RAG for knowledge, fine-tuning for style. They are complementary, not exclusive. See our RAG vs fine-tuning vs prompting guide for detailed comparison.

What vector databases work best for RAG?

Choose a vector database based on scale, deployment model, and features. Inventiple: "For most production RAG systems in 2026, Qdrant is the recommended default: it supports both dense and sparse (BM25) vectors natively, has strong filtering and metadata support, ships as a single binary or managed cloud service, and has an excellent Python client. Pinecone is the easiest managed option with minimal operational overhead. Weaviate suits teams that want a graph-like object model alongside vectors. pgvector works well if you are already on Postgres and your scale is under 10 million vectors. Chroma is fine for development and small-scale deployments but is not recommended for production workloads above 1 million vectors." NerdLevelTech: "Qdrant: self-hosted/cloud, Rust-based, advanced features, recommended for large-scale enterprise. pgvector: PostgreSQL extension, full SQL, good for existing Postgres users." Key criteria: (1) Scale — how many vectors. (2) Hybrid search — native BM25 support. (3) Metadata filtering. (4) Deployment — self-hosted vs managed. (5) Performance — latency and throughput.

How do you evaluate RAG system quality?

Evaluate RAG systems with RAGAS metrics: faithfulness, answer relevancy, context precision, and context recall. Inventiple: "Evaluate before shipping — RAGAS faithfulness below 0.8 means hallucination risk is unacceptable. 512-token chunks with Cohere reranker outperforms naive 1024-token retrieval consistently. Hybrid retrieval (dense + sparse) is the production default — pure dense retrieval misses keyword-specific queries." Glukhov: "Track retrieval metrics — recall@k and precision@k continuously. Mastering RAG architecture, reranking, vector databases, hybrid search, and evaluation will determine whether your AI system remains a demo or becomes production-ready." RAGAS metrics: (1) Faithfulness — is the answer grounded in retrieved context? (target: 0.8+). (2) Answer relevancy — does the answer address the question? (3) Context precision — are retrieved chunks relevant? (4) Context recall — did retrieval find all necessary information? Evaluation process: create a test set of 20+ questions, measure RAGAS scores before and after every change, track metrics in CI/CD pipeline.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →