How to Stop AI Hallucinations with RAG: A Practical Guide

TL;DR — AI hallucinations occur when LLMs generate claims not supported by evidence. RAG reduces hallucination rates from 20-40% to under 5% by grounding responses in retrieved documents. But RAG alone doesn't eliminate hallucinations — poor retrieval, knowledge conflicts, and generation drift can still cause fabricated claims. Seven techniques make RAG hallucination-resistant: (1) hybrid retrieval with RRF fusion, (2) cross-encoder reranking, (3) citation verification, (4) confidence scoring with "I don't know" threshold, (5) faithfulness evaluation, (6) query rewriting, and (7) source reliability estimation. Measure faithfulness (every claim supported by context) and groundedness (no claims beyond context). For enterprise AI, hallucination prevention is not optional — it's the difference between AI that augments decisions and AI that manufactures facts.

LLMs hallucinate because they generate text from learned patterns, not from verified facts. They state falsehoods with complete confidence. Research from ConfRAG shows that base LLMs hallucinate at rates of 20-40% on factual benchmarks. RAG addresses this by changing how models access information — instead of relying on training data, the model synthesizes answers from retrieved source documents.

But RAG is not a hallucination cure. Poor retrieval, knowledge conflicts, and generation drift can still produce fabricated claims. A Stanford study on legal RAG found that even RAG systems with trusted content sources can produce hallucinated answers when retrieval fails or the LLM overrides context with training data.

The goal is not to eliminate hallucinations entirely — that's impossible with current LLM architectures. The goal is to reduce them to an acceptable rate (under 5%) and detect them when they occur.

What Causes Hallucinations in RAG Systems?

The Seven Techniques for Hallucination-Resistant RAG

Technique 1: Query Rewriting

Users don't ask questions in the form that retrieves the best documents. "What's our refund policy?" might match better as "refund policy terms conditions enterprise customers."

class QueryRewriter:
    """Rewrite queries for better retrieval."""

    def rewrite(self, query: str, context: str = "") -> list[str]:
        # Generate multiple query variants
        variants = self.llm.generate(f"""
            Rewrite this query in 3 different ways to improve
            document retrieval. Keep the same intent.

            Original: {query}

            Variants:
        """)

        return [query] + variants  # Include original + variants

    def retrieve_with_rewrites(self, query: str, retriever) -> list:
        """Retrieve using multiple query variants, deduplicate results."""
        variants = self.rewrite(query)
        all_results = []

        for variant in variants:
            results = retriever.search(variant, top_k=20)
            all_results.extend(results)

        # Deduplicate by document ID, keep highest score
        seen = {}
        for r in all_results:
            if r["id"] not in seen or r["score"] > seen[r["id"]]["score"]:
                seen[r["id"]] = r

        return list(seen.values())[:50]

Technique 2: Hybrid Retrieval with RRF

Hybrid retrieval combines BM25 and vector search to cover each method's blind spots. Poor retrieval is the #1 cause of hallucinations — if the system can't find the right documents, the LLM fills gaps with fabricated information.

Technique 3: Cross-Encoder Reranking

After hybrid retrieval, rerank the top candidates with a cross-encoder for higher precision. This ensures the most relevant documents reach the LLM, reducing the chance of generation drift.

Technique 4: Confidence Threshold ("I Don't Know")

The most effective hallucination prevention is the ability to say "I don't know." If retrieval scores are below a threshold, return a refusal instead of generating from poor context.

class ConfidenceAwareRAG:
    """RAG with confidence scoring and 'I don't know' threshold."""

    def __init__(self, retriever, llm, threshold: float = 0.3):
        self.retriever = retriever
        self.llm = llm
        self.threshold = threshold

    def answer(self, query: str) -> dict:
        # Retrieve with hybrid search
        results = self.retriever.retrieve(query, top_k=10)

        # Check confidence: max retrieval score
        max_score = max(r["fused_score"] for r in results) if results else 0

        if max_score < self.threshold:
            return {
                "answer": "I don't have enough information to answer this question.",
                "confidence": max_score,
                "sources": [],
                "hallucination_risk": "high"
            }

        # Generate with retrieved context
        context = self._format_context(results)
        answer = self.llm.generate(
            query=query,
            context=context,
            system_prompt="Answer only using the provided context. "
                         "If the context doesn't contain the answer, say "
                         "'I don't know.' Cite sources for every claim."
        )

        return {
            "answer": answer,
            "confidence": max_score,
            "sources": results[:5],
            "hallucination_risk": "low" if max_score > 0.6 else "medium"
        }

ConfRAG research showed that training LLMs to respond with "I am unsure" when they lack confidence reduced hallucination rates from 20-40% to below 5% — and reduced unnecessary RAG retrievals by over 30%.

Technique 5: Faithfulness Evaluation

Faithfulness measures whether every claim in the answer is supported by the retrieved context. After generation, decompose the answer into individual claims and verify each one.

class FaithfulnessEvaluator:
    """Check if every claim in the answer is grounded in context."""

    def evaluate(self, answer: str, context: list[str]) -> dict:
        # Decompose answer into individual claims
        claims = self._extract_claims(answer)

        # Check each claim against context
        supported = 0
        unsupported_claims = []

        for claim in claims:
            is_supported = self._check_support(claim, context)
            if is_supported:
                supported += 1
            else:
                unsupported_claims.append(claim)

        faithfulness = supported / len(claims) if claims else 0

        return {
            "faithfulness": faithfulness,
            "total_claims": len(claims),
            "supported_claims": supported,
            "unsupported_claims": unsupported_claims,
            "pass": faithfulness >= 0.8  # Threshold
        }

    def _extract_claims(self, text: str) -> list[str]:
        """Use LLM to decompose answer into atomic claims."""
        return self.llm.generate(f"""
            Decompose this text into individual factual claims.
            One claim per line. No commentary.

            Text: {text}

            Claims:
        """).strip().split("\n")

    def _check_support(self, claim: str, context: list[str]) -> bool:
        """Check if a claim is supported by the context."""
        result = self.llm.generate(f"""
            Is this claim supported by the context? Answer yes or no.

            Claim: {claim}
            Context: {' '.join(context)}

            Supported:
        """)
        return "yes" in result.lower()

Technique 6: Citation Verification

Every claim in the answer must include a citation to the source document. AI citations are not just a UX feature — they're a hallucination detection mechanism. If the LLM cannot cite a source for a claim, that claim is likely hallucinated.

class CitationValidator:
    """Verify that all citations in the answer are real."""

    def validate(self, answer: str, retrieved_docs: list) -> dict:
        # Extract citations from answer
        citations = self._extract_citations(answer)

        # Verify each citation exists in retrieved docs
        valid = 0
        fake_citations = []

        for citation in citations:
            doc = self._find_doc(citation, retrieved_docs)
            if doc:
                valid += 1
            else:
                fake_citations.append(citation)

        return {
            "total_citations": len(citations),
            "valid_citations": valid,
            "fake_citations": fake_citations,
            "all_valid": len(fake_citations) == 0
        }

Fake citations — answers that include document titles, quotes, or links that don't exist in the retrieved set — are a particularly dangerous hallucination type. They appear grounded but are fabricated.

Technique 7: Source Reliability Scoring

Not all retrieved documents are equally reliable. An outdated policy document, an unverified Slack message, and an official company wiki page should not carry the same weight.

class SourceReliabilityScorer:
    """Score source reliability to prioritize high-quality context."""

    RELIABILITY_SCORES = {
        "official_policy": 1.0,
        "company_wiki": 0.9,
        "verified_documentation": 0.85,
        "internal_report": 0.7,
        "email_thread": 0.5,
        "slack_message": 0.3,
        "draft_document": 0.2,
    }

    def score(self, document: dict) -> float:
        source_type = document.get("source_type", "unknown")
        base_score = self.RELIABILITY_SCORES.get(source_type, 0.5)

        # Penalize stale documents
        age_days = document.get("age_days", 0)
        freshness_penalty = min(age_days / 365, 0.3)  # Max 30% penalty

        # Penalize documents with low ACL confidence
        acl_confidence = document.get("acl_confidence", 1.0)

        return base_score * (1 - freshness_penalty) * acl_confidence

Measuring Hallucination Rates

Hallucination Prevention Checklist

• [ ] Implement hybrid retrieval (BM25 + vector + RRF)
• [ ] Add cross-encoder reranking for top candidates
• [ ] Set confidence threshold — return "I don't know" when retrieval is poor
• [ ] Require citations for every claim in the answer
• [ ] Implement AI citations with source verification
• [ ] Run faithfulness evaluation on every generated answer
• [ ] Decompose answers into claims and check each against context
• [ ] Detect fake citations (cited docs not in retrieved set)
• [ ] Score source reliability and prioritize high-quality documents
• [ ] Implement query rewriting for better retrieval coverage
• [ ] Use multi-query retrieval (multiple query variants, deduplicate results)
• [ ] Add freshness detection — penalize stale documents
• [ ] Configure system prompt: "Answer only from context, cite sources, say 'I don't know' if unsure"
• [ ] Monitor hallucination rate with automated + human evaluation
• [ ] Log refusal rate (too low = hallucinating, too high = over-cautious)
• [ ] Test with adversarial queries (questions with no answer in the corpus)
• [ ] Enforce document-level ACLs in retrieval
• [ ] Integrate with company brain architecture
• [ ] For high-stakes decisions: add human review for low-confidence answers
• [ ] Measure RAG Triad: context relevance, groundedness, answer relevance

FAQ

How does RAG stop AI hallucinations?

RAG stops hallucinations by grounding LLM responses in retrieved documents. Instead of generating from training data (which may be outdated, incomplete, or fabricated), the LLM synthesizes answers from verified source documents. Every claim in the response must be supported by retrieved context. If no relevant documents are found, the system can say "I don't know" rather than fabricating an answer. RAG reduces hallucination rates from 20-40% to under 5% when properly configured.

What is RAG faithfulness?

RAG faithfulness is a metric that measures whether every claim in the LLM's generated answer is supported by the retrieved context. A faithfulness score of 1.0 means every statement can be traced to the source documents. A score of 0.5 means half the claims are unsupported. Faithfulness is measured by decomposing the answer into individual claims and checking each claim against the retrieved passages. Low faithfulness indicates the LLM is hallucinating beyond its sources.

What causes hallucinations in RAG systems?

Hallucinations in RAG systems have three causes: (1) poor retrieval — the system retrieves irrelevant or low-quality documents, so the LLM fills gaps with fabricated information; (2) knowledge conflicts — the LLM's training data contradicts the retrieved context, and it ignores the context; (3) generation drift — the LLM generates beyond the retrieved context, adding plausible-sounding but unsupported claims. Each cause requires a different mitigation strategy.

Can RAG completely eliminate hallucinations?

No. RAG significantly reduces hallucinations but cannot eliminate them entirely. Even with perfect retrieval, the LLM can still misinterpret context, oversimplify, or generate unsupported claims. The goal is to minimize hallucinations to an acceptable rate (under 5%) and detect them when they occur. Techniques like confidence scoring, citation verification, and faithfulness evaluation help detect remaining hallucinations. For high-stakes applications, add human review for low-confidence answers.

What is the best retrieval strategy to prevent hallucinations?

The best retrieval strategy for hallucination prevention is hybrid retrieval (BM25 + vector search) with Reciprocal Rank Fusion, followed by cross-encoder reranking. This ensures both exact term matching and semantic relevance. Retrieve more candidates than needed (top-50), rerank to top-10, and include a confidence threshold — if retrieval scores are below threshold, return "I don't know" instead of generating from poor context. Query rewriting and multi-query retrieval also improve retrieval quality.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →