RAG Cross-Encoder Reranking: Precision Retrieval for Enterprise AI

TL;DR — Cross-encoder reranking is a two-stage retrieval pattern: bi-encoder (vector search) retrieves top-50 candidates fast, then cross-encoder reranks them for precision. Bi-encoders encode query and document separately (fast, O(1) lookup, imprecise). Cross-encoders encode query and document together in one transformer pass (slow, O(n), precise). The standard RAG pipeline: hybrid retrieval with RRF fusion → cross-encoder reranking → LLM generation. Best open-source reranker: BGE-Reranker-v2-m3 (568M params, multilingual, Apache 2.0). Best managed API: Cohere Rerank 3. Reranking adds 100-200ms latency but improves NDCG@10 by 10-20%. Don't use reranking when latency budget is under 200ms or first-stage retrieval is already high-precision. Combine with query rewriting for maximum accuracy.

Cross-encoder reranking is the precision layer in a RAG pipeline. After hybrid retrieval produces top-50 candidates, a cross-encoder reranks them to surface the most relevant documents for the LLM.

The problem it solves: bi-encoder (vector) retrieval is fast but imprecise. It encodes the query and document independently, missing fine-grained interactions. A cross-encoder encodes them together, capturing token-level relationships — but it's too slow for searching millions of documents. The two-stage pattern gets the best of both: fast first-stage retrieval, precise second-stage reranking.

Bi-Encoder vs Cross-Encoder

How Cross-Encoder Reranking Works

class CrossEncoderReranker:
    """Rerank retrieved documents using a cross-encoder model."""

    def __init__(self, model_name: str = "BAAI/bge-reranker-v2-m3"):
        from sentence_transformers import CrossEncoder
        self.model = CrossEncoder(model_name)

    def rerank(self, query: str, documents: list, top_k: int = 10) -> list:
        """Rerank documents by cross-encoder relevance score."""
        # Create query-document pairs
        pairs = [[query, doc["content"]] for doc in documents]

        # Score each pair with cross-encoder
        scores = self.model.predict(pairs)

        # Attach scores and sort
        for doc, score in zip(documents, scores):
            doc["rerank_score"] = float(score)

        # Sort by rerank score descending
        reranked = sorted(documents, key=lambda d: -d["rerank_score"])

        return reranked[:top_k]

Full Pipeline with Hybrid Retrieval + Reranking

class RAGPipelineWithReranking:
    """Full RAG pipeline: hybrid retrieval → RRF fusion → cross-encoder reranking."""

    def __init__(self, bm25_store, vector_store, reranker, llm, k_rrf=60):
        self.bm25_store = bm25_store
        self.vector_store = vector_store
        self.reranker = reranker
        self.llm = llm
        self.k_rrf = k_rrf

    def retrieve(self, query: str, user_id: str, top_k: int = 10) -> list:
        # Stage 1: Hybrid retrieval (BM25 + vector)
        bm25_results = self.bm25_store.search(query, top_k=50)
        vector_results = self.vector_store.search(
            query_embedding=self._embed(query),
            top_k=50,
            filter_acl=self._get_user_acls(user_id)
        )

        # Stage 1b: RRF fusion
        fused = self._rrf_fuse(bm25_results, vector_results, k=self.k_rrf)

        # Stage 2: Cross-encoder reranking (top-50 → top-k)
        reranked = self.reranker.rerank(query, fused[:50], top_k=top_k)

        return reranked

    def answer(self, query: str, user_id: str) -> dict:
        # Retrieve with reranking
        documents = self.retrieve(query, user_id, top_k=10)

        # Check confidence
        if not documents or documents[0]["rerank_score"] < 0.1:
            return {"answer": "I don't have enough information.", "sources": []}

        # Generate with citations
        context = self._format_context(documents)
        answer = self.llm.generate(query=query, context=context)

        return {
            "answer": answer,
            "sources": documents[:5],
            "rerank_scores": [d["rerank_score"] for d in documents[:5]]
        }

Reranker Model Comparison

Latency Breakdown

Reranking is 10% of total latency but improves NDCG@10 by 10-20%. The ROI is clear: a 150ms investment for a 15% accuracy gain.

When to Use Reranking

Fine-Tuning Rerankers

For domain-specific accuracy, fine-tune a reranker on your query-document pairs:

class RerankerFineTuner:
    """Fine-tune a cross-encoder on domain-specific data."""

    def prepare_training_data(self, queries, relevant_docs, irrelevant_docs):
        """Create training pairs: (query, doc, label)."""
        train_data = []

        for query, rel_docs, irr_docs in zip(queries, relevant_docs, irrelevant_docs):
            # Positive pairs (relevant docs)
            for doc in rel_docs:
                train_data.append((query, doc, 1.0))

            # Negative pairs (irrelevant docs)
            for doc in irr_docs:
                train_data.append((query, doc, 0.0))

        return train_data

    def fine_tune(self, model_name, train_data, epochs=3):
        from sentence_transformers import CrossEncoder, InputExample
        from torch.utils.data import DataLoader

        model = CrossEncoder(model_name)

        train_examples = [
            InputExample(texts=[q, d], label=label)
            for q, d, label in train_data
        ]

        train_dataloader = DataLoader(train_examples, batch_size=16, shuffle=True)

        model.fit(
            train_dataloader=train_dataloader,
            epochs=epochs,
            warmup_steps=100,
            output_path="./fine-tuned-reranker"
        )

RAG Cross-Encoder Reranking Checklist

• [ ] Start with hybrid retrieval (BM25 + vector + RRF) as first stage
• [ ] Retrieve top-50 candidates from first stage (3-5x final result count)
• [ ] Choose reranker: BGE-Reranker-v2-m3 (self-hosted) or Cohere Rerank 3 (API)
• [ ] Rerank top-50 down to top-10 for LLM context
• [ ] Measure NDCG@10 before and after reranking to validate improvement
• [ ] Set rerank score threshold — filter documents below 0.1 score
• [ ] Profile latency: reranking should be under 200ms
• [ ] For self-hosted: use GPU for BGE-Reranker (100ms vs 300ms on CPU)
• [ ] For ultra-low latency: use FlashRank (20ms on CPU, 8M params)
• [ ] Consider fine-tuning on domain-specific query-document pairs
• [ ] Use query rewriting before retrieval for better candidates
• [ ] Apply RRF fusion before reranking
• [ ] Enforce document-level ACLs in first-stage retrieval
• [ ] Monitor reranking latency p99 (target: <200ms)
• [ ] A/B test reranked vs non-reranked results on user satisfaction
• [ ] Consider ColBERT as a middle ground (late interaction, faster than cross-encoder)
• [ ] For hallucination prevention: high rerank scores = high confidence
• [ ] Integrate with AI knowledge base pipeline
• [ ] Cache reranking results for repeated queries
• [ ] Log rerank scores for audit trail and quality monitoring

FAQ

What is cross-encoder reranking in RAG?

Cross-encoder reranking is a two-stage retrieval pattern where a fast bi-encoder (vector search) retrieves top-50 candidates, then a cross-encoder reranks them for precision. A bi-encoder encodes the query and document separately and computes cosine similarity — fast but imprecise. A cross-encoder encodes the query and document together in a single transformer pass, capturing fine-grained interactions — precise but slow. The cross-encoder is too slow for searching millions of documents, so it's used only to rerank the top candidates from the first stage.

What is the difference between bi-encoder and cross-encoder?

A bi-encoder encodes the query and document independently into vectors, then computes similarity (cosine, dot product). This is fast (O(1) lookup) but misses query-document interactions. A cross-encoder concatenates the query and document as a single input to a transformer, which attends to all token interactions. This is more accurate but O(n) per document — too slow for large-scale retrieval. The standard pattern: bi-encoder for first-stage retrieval (top-50), cross-encoder for second-stage reranking (top-10).

Which cross-encoder reranker should I use?

For self-hosted: BGE-Reranker-v2-m3 (BAAI) is the best open-source option — multilingual, 568M parameters, Apache 2.0 license. For managed API: Cohere Rerank 3 is production-ready with low latency. For lightweight deployment: FlashRank or mxbai-rerank are fast and small. For enterprise: Voyage rerank-2 offers the highest accuracy but requires API access. Choose based on your self-hosted vs managed preference, latency budget, and language requirements.

How much latency does cross-encoder reranking add?

Cross-encoder reranking adds 50-200ms depending on the model and number of candidates. BGE-Reranker-v2-m3 reranking 50 candidates takes ~100ms on GPU, ~300ms on CPU. Cohere Rerank 3 API adds ~100-150ms network latency. The total RAG pipeline with reranking is typically: retrieval (20-50ms) + reranking (100-200ms) + generation (500-2000ms). Reranking is 5-15% of total latency but improves NDCG@10 by 10-20%.

When should you NOT use cross-encoder reranking?

Don't use cross-encoder reranking when: (1) latency budget is under 200ms total (reranking alone takes 100-200ms), (2) your first-stage retrieval already has high precision (NDCG@10 > 0.9), (3) you're doing simple factual lookups where BM25 is sufficient, (4) your candidate pool is small (<10 documents), (5) cost is a concern and you're using a paid API reranker at high volume. For these cases, hybrid retrieval with RRF fusion is sufficient.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →