RAG Evaluation Metrics: RAGAS, DeepEval, and TruLens for Production

TL;DR — RAG evaluation metrics measure retrieval and generation quality across faithfulness, answer relevancy, context precision, and context recall. arxiv: "RAGAs: reference-free evaluation of RAG pipelines. Three quality aspects: faithfulness (grounded in context), answer relevance (addresses the question), context relevance (focused, minimal noise)." QASkills: "Core suite: context precision + recall for retrieval, faithfulness + answer relevancy + correctness for generation. Faithfulness and context recall are highest-signal. Thresholds: faithfulness ≥0.90, answer relevancy ≥0.85, context precision ≥0.80, context recall ≥0.85. Use RAGAS for offline, DeepEval for CI, TruLens for production." NinadPathak: "Production systems land 0.4-0.7 on faithfulness. Above 0.6 = deployable, above 0.75 = strong. Recall@5 < 0.7: retrieval needs work. Faithfulness < 0.5: hallucinating." Zro2One: "RAGAS is the most popular framework. Continuous evaluation: pre-deployment, A/B testing, production sampling, regression testing. Metrics tell you what is broken before users do." Learn more with prevent hallucinations, citations, reranking, and hybrid search.

arxiv introduces the framework: "We introduce RAGAs (Retrieval Augmented Generation Assessment), a framework for reference-free evaluation of RAG pipelines. Evaluating RAG architectures is challenging because there are several dimensions to consider: the ability of the retrieval system to identify relevant and focused context passages, the ability of the LLM to exploit such passages in a faithful way, or the quality of the generation itself."

QASkills summarizes the production approach: "The core suite is context precision and context recall for retrieval quality, plus faithfulness (groundedness), answer relevancy and answer correctness for generation quality. Faithfulness and context recall are the two highest-signal metrics because they catch the most common failures: hallucination and incomplete retrieval."

RAG Evaluation Architecture

RAG Metrics Threshold Table

Diagnostic Thresholds

Implementation

from dataclasses import dataclass
from typing import Optional
from enum import Enum

class MetricCategory(Enum):
    RETRIEVAL = "retrieval"
    GENERATION = "generation"

@dataclass
class RAGEvaluator:
    """Evaluate RAG pipelines with RAGAS-style metrics."""

    TARGETS = {
        "faithfulness": 0.90,
        "answer_relevancy": 0.85,
        "context_precision": 0.80,
        "context_recall": 0.85,
        "recall_at_k": 0.90,
        "precision_at_k": 0.75,
        "ndcg_at_k": 0.80,
        "groundedness": 0.90,
        "answer_correctness": 0.80,
    }

    def __init__(self, llm_judge=None):
        self.llm_judge = llm_judge

    def evaluate(self, question: str, contexts: list,
                 answer: str, ground_truth: str = None,
                 retrieved_ids: list = None,
                 relevant_ids: list = None) -> dict:
        """Run full RAG evaluation suite."""
        results = {}

        # Retrieval metrics
        if retrieved_ids and relevant_ids:
            results["recall_at_k"] = self._recall_at_k(
                retrieved_ids, relevant_ids)
            results["precision_at_k"] = self._precision_at_k(
                retrieved_ids, relevant_ids)
            results["ndcg_at_k"] = self._ndcg_at_k(
                retrieved_ids, relevant_ids)

        # RAGAS-style metrics
        results["context_precision"] = self._context_precision(
            question, contexts, ground_truth)
        results["context_recall"] = self._context_recall(
            question, contexts, ground_truth)
        results["faithfulness"] = self._faithfulness(
            answer, contexts)
        results["answer_relevancy"] = self._answer_relevancy(
            question, answer)
        results["groundedness"] = self._groundedness(
            answer, contexts)

        if ground_truth:
            results["answer_correctness"] = self._answer_correctness(
                answer, ground_truth)

        # Check targets
        results["meets_targets"] = self._check_targets(results)
        results["overall_score"] = self._overall_score(results)

        return results

    def _recall_at_k(self, retrieved: list,
                     relevant: list) -> float:
        """Recall@K: fraction of relevant items retrieved."""
        if not relevant:
            return 0.0
        retrieved_set = set(retrieved)
        relevant_set = set(relevant)
        return len(retrieved_set & relevant_set) / len(relevant_set)

    def _precision_at_k(self, retrieved: list,
                        relevant: list) -> float:
        """Precision@K: fraction of retrieved that are relevant."""
        if not retrieved:
            return 0.0
        retrieved_set = set(retrieved)
        relevant_set = set(relevant)
        return len(retrieved_set & relevant_set) / len(retrieved_set)

    def _ndcg_at_k(self, retrieved: list,
                   relevant: list) -> float:
        """nDCG@K: rank-weighted relevance."""
        import math
        dcg = 0.0
        for i, doc_id in enumerate(retrieved):
            if doc_id in relevant:
                dcg += 1.0 / math.log2(i + 2)

        # Ideal DCG
        ideal_hits = min(len(relevant), len(retrieved))
        idcg = sum(1.0 / math.log2(i + 2) for i in range(ideal_hits))

        if idcg == 0:
            return 0.0
        return dcg / idcg

    def _faithfulness(self, answer: str,
                      contexts: list) -> float:
        """Faithfulness: fraction of claims supported by context.

        In production: use RAGAS faithfulness with LLM-as-judge.
        Here: simple word-overlap proxy.
        """
        import re
        sentences = re.split(r'(?<=[.!?])\s+', answer)
        if not sentences:
            return 0.0

        context_text = " ".join(contexts).lower()
        supported = 0

        for sentence in sentences:
            clean = re.sub(r'\[Source\s+\d+\]', '', sentence)
            words = set(clean.lower().split())
            context_words = set(context_text.split())
            overlap = len(words & context_words)
            if len(words) > 0 and overlap / len(words) > 0.5:
                supported += 1

        return supported / len(sentences)

    def _answer_relevancy(self, question: str,
                          answer: str) -> float:
        """Answer relevancy: does the answer address the question?

        In production: use RAGAS answer_relevancy with LLM-as-judge.
        Here: word overlap between question and answer.
        """
        q_words = set(question.lower().split())
        a_words = set(answer.lower().split())
        overlap = len(q_words & a_words)
        total = len(q_words)
        if total == 0:
            return 0.0
        return min(1.0, overlap / total * 2)

    def _context_precision(self, question: str,
                           contexts: list,
                           ground_truth: str = None) -> float:
        """Context precision: relevant content ranks higher."""
        if not contexts:
            return 0.0

        q_words = set(question.lower().split())
        scores = []
        for ctx in contexts:
            ctx_words = set(ctx.lower().split())
            overlap = len(q_words & ctx_words)
            scores.append(overlap / max(len(ctx_words), 1))

        # Check if scores are decreasing (good ranking)
        sorted_scores = sorted(scores, reverse=True)
        if scores == sorted_scores:
            return sum(scores) / len(scores)
        else:
            # Penalize for bad ranking
            return sum(scores) / len(scores) * 0.8

    def _context_recall(self, question: str,
                        contexts: list,
                        ground_truth: str = None) -> float:
        """Context recall: all needed info was fetched."""
        if not ground_truth or not contexts:
            return 0.0

        gt_words = set(ground_truth.lower().split())
        context_text = " ".join(contexts).lower()
        context_words = set(context_text.split())

        overlap = len(gt_words & context_words)
        return overlap / max(len(gt_words), 1)

    def _groundedness(self, answer: str,
                      contexts: list) -> float:
        """Groundedness: every claim traceable to source span.

        Similar to faithfulness but stricter — checks specific spans.
        """
        return self._faithfulness(answer, contexts)

    def _answer_correctness(self, answer: str,
                            ground_truth: str) -> float:
        """Answer correctness: matches ground truth."""
        a_words = set(answer.lower().split())
        gt_words = set(ground_truth.lower().split())
        overlap = len(a_words & gt_words)
        total = len(gt_words)
        if total == 0:
            return 0.0
        return overlap / total

    def _check_targets(self, results: dict) -> dict:
        """Check which metrics meet their targets."""
        checks = {}
        for metric, target in self.TARGETS.items():
            if metric in results:
                checks[metric] = {
                    "score": results[metric],
                    "target": target,
                    "meets_target": results[metric] >= target,
                }
        return checks

    def _overall_score(self, results: dict) -> float:
        """Compute overall RAG quality score."""
        key_metrics = ["faithfulness", "answer_relevancy",
                       "context_precision", "context_recall"]
        scores = [results[m] for m in key_metrics if m in results]
        if not scores:
            return 0.0
        return sum(scores) / len(scores)

RAG Evaluation Metrics Checklist

• [ ] Evaluate RAG with faithfulness, answer relevancy, context precision, context recall
• [ ] Faithfulness: answer grounded in context (target ≥ 0.90)
• [ ] Answer Relevancy: answer addresses the question (target ≥ 0.85)
• [ ] Context Precision: relevant chunks well-ranked (target ≥ 0.80)
• [ ] Context Recall: all needed info fetched (target ≥ 0.85)
• [ ] Recall@K: fraction of relevant passages retrieved (target ≥ 0.90)
• [ ] Precision@K: fraction of retrieved that are relevant (target ≥ 0.75)
• [ ] nDCG@K: rank-weighted relevance (target ≥ 0.80)
• [ ] Groundedness: every claim traceable to source span (target ≥ 0.90)
• [ ] Answer Correctness: matches ground truth (target ≥ 0.80)
• [ ] Faithfulness and context recall are the two highest-signal metrics
• [ ] Faithfulness catches hallucination; context recall catches incomplete retrieval
• [ ] RAGAS: reference-free evaluation without ground truth human annotations
• [ ] RAGAS enables faster evaluation cycles of RAG architectures
• [ ] RAGAS is the most popular RAG evaluation framework
• [ ] RAGAS uses LLM-as-judge for automated scoring
• [ ] DeepEval: pytest-style CI gating — failing metric breaks the build
• [ ] TruLens: production observability with RAG Triad dashboard
• [ ] Mature teams run all three: RAGAS for offline, DeepEval for CI, TruLens for production
• [ ] They share the same conceptual metrics — faithfulness regression maps across tools
• [ ] Production systems land 0.4-0.7 on faithfulness
• [ ] Above 0.6 = deployable, above 0.75 = strong, above 0.90 = excellent
• [ ] Diagnostic: Recall@5 < 0.7 → retrieval needs work
• [ ] Diagnostic: Context Precision@5 < 0.5 → too much noise in retrieved passages
• [ ] Diagnostic: Faithfulness < 0.5 → model is hallucinating frequently
• [ ] Diagnostic: Answer Relevancy < 0.6 → not answering the question
• [ ] Diagnostic: High recall, low faithfulness → LLM hallucinating, not retrieval issue
• [ ] Diagnostic: Low recall, high faithfulness → retrieval missing information
• [ ] Diagnostic: High precision, low recall → system is too conservative
• [ ] Pre-deployment: full evaluation on test set before any change
• [ ] A/B testing: compare new retrieval/generation approaches on live traffic
• [ ] Production sampling: evaluate random sample of production queries weekly
• [ ] User feedback: track thumbs up/down, corrections, and escalations
• [ ] Regression testing: run full test suite before every change
• [ ] Metrics tell you what is broken before your users do
• [ ] Create a golden dataset of 20+ questions with ground truth answers
• [ ] Gate CI on faithfulness and recall — these are the highest-signal metrics
• [ ] Evaluate before and after every chunking, retrieval, or prompt change
• [ ] Track metric trends over time — declining faithfulness = degrading system
• [ ] Log evaluation results with metadata: chunk size, embedding model, reranker
• [ ] RAGAS faithfulness below 0.8 means hallucination risk
• [ ] Context relevance penalizes inclusion of redundant information
• [ ] Context that is too long makes LLMs less effective (lost in the middle)
• [ ] nDCG matters most when rank order matters (after reranking)
• [ ] Read prevent hallucinations for faithfulness
• [ ] Read citations for groundedness
• [ ] Read reranking for precision improvement
• [ ] Read hybrid search for recall improvement
• [ ] Read what is RAG for architecture
• [ ] Build RAG from scratch with evaluation
• [ ] Test: faithfulness ≥ 0.90 after all techniques applied
• [ ] Test: context recall ≥ 0.85 with hybrid search
• [ ] Test: CI gate fails when faithfulness drops below threshold
• [ ] Test: evaluation results are logged with metadata for trend tracking
• [ ] Document evaluation framework, thresholds, golden dataset, and CI integration

FAQ

What are the key RAG evaluation metrics?

The core RAG evaluation metrics are faithfulness, answer relevancy, context precision, and context recall. arxiv: "RAGAs introduces a framework for reference-free evaluation of RAG pipelines. Three quality aspects: (1) Faithfulness — the answer should be grounded in the given context to avoid hallucinations. (2) Answer Relevance — the generated answer should address the actual question. (3) Context Relevance — the retrieved context should be focused, containing as little irrelevant information as possible." QASkills: "The core suite is context precision and context recall for retrieval quality, plus faithfulness (groundedness), answer relevancy and answer correctness for generation quality. Faithfulness and context recall are the two highest-signal metrics because they catch the most common failures: hallucination and incomplete retrieval." Zro2One: "Metric thresholds: Faithfulness minimum 0.85, good 0.93, excellent 0.97+. Answer Relevance minimum 0.75, good 0.85, excellent 0.92+. Context Precision minimum 0.60, good 0.75, excellent 0.85+. Recall@10 minimum 0.80, good 0.90, excellent 0.95+."

How does RAGAS faithfulness work?

RAGAS faithfulness measures whether the generated answer is grounded in the retrieved context — every claim in the answer must be supported by the context. arxiv: "Faithfulness refers to the idea that the answer should be grounded in the given context. This is important to avoid hallucinations, and to ensure that the retrieved context can act as a justification for the generated answer. RAG systems are often used in applications where the factual consistency of the generated text w.r.t. the grounded sources is highly important, e.g. in domains such as law." NinadPathak: "Faithfulness measures whether the generated answer stays consistent with the retrieved context. A faithful answer does not invent a date or a figure that appears nowhere in the passages it was given. Production systems tend to land between 0.4 and 0.7 on faithfulness. Scores above 0.6 typically indicate a system worth deploying, and above 0.75 is strong." Inventiple: "RAGAS faithfulness below 0.8 means hallucination risk. Evaluate before and after every significant change." Faithfulness = supported claims / total claims. Target: ≥ 0.90.

What is the difference between context precision and context recall?

Context precision measures whether relevant content ranks higher within retrieved passages, while context recall measures whether retrieval fetched all the information needed for the answer. QASkills: "Context Precision: are retrieved chunks relevant and well-ranked? Threshold ≥ 0.80. Context Recall: did retrieval fetch all info needed for the answer? Threshold ≥ 0.85." NinadPathak: "Context Precision measures whether relevant content ranks higher within a passage. A retrieved passage might be 80% relevant and 20% noise, and context precision penalizes the ones where the useful sentence sits buried under boilerplate. Recall@K measures what fraction of all relevant passages appear in the top-K results. If the relevant passage is not retrieved, the LLM cannot use it. Recall is the most important retrieval metric for RAG." Zro2One: "High recall, low faithfulness: retrieval found the right passages but the LLM is hallucinating. Low recall, high faithfulness: the model is accurate but missing information. High precision, low recall: the system is conservative."

Which RAG evaluation framework should you use?

Use RAGAS for offline experiments, DeepEval for CI gating, and TruLens for production observability. QASkills: "Ragas, DeepEval and TruLens are the three leaders. Ragas is best for offline dataset scoring when tuning retrievers, DeepEval is best for pytest-style CI gating because a failing metric breaks the build, and TruLens is best for live production observability with its RAG Triad dashboard. They share the same conceptual metrics, so most mature teams run all three at different stages of the pipeline." Zro2One: "RAGAS is the most popular RAG evaluation framework. Provides automated metrics using LLM-as-judge. from ragas import evaluate; from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall; results = evaluate(dataset=eval_dataset, metrics=[...]). Results: faithfulness: 0.92, answer_relevancy: 0.88, context_precision: 0.85, context_recall: 0.91." arxiv: "RAGAS provides reference-free evaluation without ground truth human annotations, enabling faster evaluation cycles."

How do you set up continuous RAG evaluation?

Set up continuous evaluation across pre-deployment, A/B testing, production sampling, and regression testing. Zro2One: "Continuous evaluation: (1) Pre-deployment: full evaluation on test set before any change. (2) A/B testing: compare new retrieval/generation approaches on live traffic. (3) Production sampling: evaluate a random sample of production queries weekly. (4) User feedback: track thumbs up/down, corrections, and escalations. (5) Regression testing: run the full test suite before every change. The RAG systems that work well in production are the ones that are measured continuously. Metrics tell you what is broken before your users do." QASkills: "Mature teams run all three: Ragas for offline experiments when tuning chunking and embeddings, DeepEval as the CI gate on a curated golden set, and TruLens for production observability. They share the same conceptual metrics, so a faithfulness regression caught in DeepEval maps cleanly to a groundedness dip in TruLens." NinadPathak: "Diagnostic thresholds: Recall@5 < 0.7: retrieval needs work. Context Precision@5 < 0.5: too much noise. Faithfulness < 0.5: model is hallucinating. Answer Relevancy < 0.6: not answering the question."

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →