RAG Query Rewriting: Multi-Query, HyDE, and Sub-Query Decomposition

TL;DR — RAG query rewriting transforms user queries before retrieval to improve accuracy. Four techniques: (1) Multi-query retrieval generates 3-5 query reformulations and combines results for higher recall, (2) HyDE generates a hypothetical answer and uses it as the search query, (3) Step-back prompting extracts a broader principle before retrieving, (4) Sub-query decomposition breaks complex questions into simpler parts. Query rewriting bridges the gap between how users ask questions and how documents are written. Use it when queries are short, conversational, complex, or reference previous context. Don't use it for simple factual lookups — it adds latency (one LLM call) without benefit. Combine with hybrid retrieval and cross-encoder reranking for maximum accuracy.

Users don't ask questions the way documents are written. A user asks "How do I handle a breach?" but the document says "Incident Response Procedures for Security Compromises." The semantic intent is the same, but the lexical and vector similarity may be low.

Query rewriting bridges this gap by transforming the user's query before retrieval. Instead of searching with the original question, the system generates one or more modified queries that are more likely to match relevant documents.

The Query Rewriting Problem

Four Query Rewriting Techniques

Technique 1: Multi-Query Retrieval

Generate multiple reformulations of the query, retrieve for each, and combine results:

class MultiQueryRetriever:
    """Generate multiple query reformulations and combine results."""

    REWRITE_PROMPT = """Rewrite this query in {n} different ways to improve 
    document retrieval. Keep the same intent but use different phrasing, 
    synonyms, and perspectives.

    Original query: {query}

    Rewrites (one per line):
    """

    def retrieve(self, query: str, retriever, n_rewrites: int = 3) -> list:
        # Generate rewrites
        rewrites_text = self.llm.generate(
            self.REWRITE_PROMPT.format(query=query, n=n_rewrites)
        )
        rewrites = [query] + rewrites_text.strip().split('\n')

        # Retrieve for each query
        all_results = []
        for rewritten_query in rewrites:
            results = retriever.search(rewritten_query, top_k=20)
            all_results.extend(results)

        # Deduplicate by document ID, keep highest score
        seen = {}
        for result in all_results:
            doc_id = result["id"]
            if doc_id not in seen or result["score"] > seen[doc_id]["score"]:
                seen[doc_id] = result

        return list(seen.values())[:20]

When to use: Broad queries where recall matters more than precision. When the original query might not match the document's vocabulary.

Cost: One LLM call for rewriting + n_rewrites × retrieval calls.

Technique 2: HyDE (Hypothetical Document Embeddings)

HyDE generates a hypothetical answer and uses it as the search query:

class HyDERetriever:
    """Use hypothetical answer as search query."""

    HYDE_PROMPT = """Generate a brief, factual answer to this question. 
    Don't worry about accuracy — this will be used for document retrieval, 
    not as a final answer.

    Question: {query}

    Hypothetical answer:
    """

    def retrieve(self, query: str, vector_store, top_k: int = 10) -> list:
        # Generate hypothetical answer
        hypothetical_answer = self.llm.generate(
            self.HYDE_PROMPT.format(query=query)
        )

        # Use hypothetical answer as search query
        # (it's closer in semantic space to the real answer document)
        query_embedding = self.embedder.embed(hypothetical_answer)

        results = vector_store.search(
            query_embedding=query_embedding,
            top_k=top_k
        )

        return results

When to use: Short, conversational queries. When the question is far in semantic space from the answer document.

Cost: One LLM call for hypothetical answer generation + one embedding + one retrieval.

Why it works: A hypothetical answer like "Our refund policy allows enterprise customers to request refunds within 90 days of purchase..." is semantically closer to the actual refund policy document than the question "How do I get my money back?"

Technique 3: Sub-Query Decomposition

Break complex questions into simpler sub-questions:

class SubQueryDecomposer:
    """Decompose complex queries into sub-questions."""

    DECOMPOSE_PROMPT = """Break this complex question into {n} simpler 
    sub-questions that can each be answered independently. Each sub-question 
    should cover one aspect of the original question.

    Original question: {query}

    Sub-questions (one per line):
    """

    def retrieve_and_answer(self, query: str, retriever, llm) -> dict:
        # Decompose into sub-queries
        sub_queries_text = llm.generate(
            self.DECOMPOSE_PROMPT.format(query=query, n=3)
        )
        sub_queries = sub_queries_text.strip().split('\n')

        # Retrieve and answer each sub-query
        sub_answers = []
        for sq in sub_queries:
            results = retriever.search(sq, top_k=5)
            context = self._format_context(results)
            answer = llm.generate(
                f"Answer this question using the context:\n"
                f"Question: {sq}\nContext: {context}\nAnswer:"
            )
            sub_answers.append({
                "sub_query": sq,
                "answer": answer,
                "sources": results
            })

        # Synthesize final answer from sub-answers
        final_answer = llm.generate(
            f"Synthesize a complete answer from these sub-answers:\n"
            f"Original question: {query}\n"
            f"Sub-answers: {json.dumps(sub_answers, indent=2)}\n"
            f"Final answer:"
        )

        return {
            "answer": final_answer,
            "sub_queries": sub_answers,
            "decomposition_used": True
        }

When to use: Multi-hop questions, comparison questions, questions requiring information from multiple documents.

Cost: One LLM call for decomposition + n × (retrieval + LLM call) + one LLM call for synthesis.

Technique 4: Step-Back Prompting

Extract a broader principle or concept before retrieving:

class StepBackRetriever:
    """Extract broader principle before retrieval."""

    STEPBACK_PROMPT = """Given this specific question, identify a broader, 
    more general principle or concept that would help answer it.

    Specific question: {query}

    Broader principle:
    """

    def retrieve(self, query: str, retriever, top_k: int = 10) -> list:
        # Extract broader principle
        principle = self.llm.generate(
            self.STEPBACK_PROMPT.format(query=query)
        )

        # Retrieve using both original and step-back query
        original_results = retriever.search(query, top_k=top_k)
        stepback_results = retriever.search(principle, top_k=top_k)

        # Combine and deduplicate
        combined = self._deduplicate(original_results + stepback_results)
        return combined[:top_k]

When to use: Questions that require domain principles or general rules. When the specific query is too narrow to retrieve relevant background.

Comparison of Techniques

When NOT to Use Query Rewriting

• Simple factual lookups — "What is our PTO policy?" already matches the document
• Exact term matching — "ISO 27001" is better served by BM25 than rewriting
• Latency-sensitive applications — rewriting adds 200-500ms per query
• High-volume, low-complexity queries — cost of rewriting outweighs benefit

Combining Query Rewriting with Other Techniques

class AdvancedRAGPipeline:
    """Full RAG pipeline with query rewriting, hybrid retrieval, and reranking."""

    def answer(self, query: str, user_id: str) -> dict:
        # Step 1: Classify query — does it need rewriting?
        query_type = self._classify_query(query)

        # Step 2: Apply appropriate rewriting
        if query_type == "simple":
            search_queries = [query]
        elif query_type == "conversational":
            search_queries = [self.hyde.generate(query)]
        elif query_type == "complex":
            search_queries = self.decomposer.decompose(query)
        else:
            search_queries = self.multi_query.rewrite(query, n=3)

        # Step 3: Hybrid retrieval for each query
        all_results = []
        for sq in search_queries:
            results = self.hybrid_retriever.retrieve(sq, user_id, top_k=20)
            all_results.extend(results)

        # Step 4: Deduplicate and rerank
        deduped = self._deduplicate(all_results)
        reranked = self.reranker.rerank(query, deduped[:50])

        # Step 5: Generate with citations
        answer = self.generator.generate(query, reranked[:10])

        return answer

RAG Query Rewriting Checklist

• [ ] Classify queries: simple, conversational, complex, or abstract
• [ ] Don't rewrite simple queries — it adds latency without benefit
• [ ] Implement multi-query retrieval for broad, low-recall queries (3-5 rewrites)
• [ ] Implement HyDE for short, conversational queries
• [ ] Implement sub-query decomposition for complex, multi-hop questions
• [ ] Implement step-back prompting for abstract, principle-based questions
• [ ] Combine with hybrid retrieval (BM25 + vector)
• [ ] Apply RRF fusion when combining multi-query results
• [ ] Add cross-encoder reranking after retrieval
• [ ] Deduplicate results when using multi-query or sub-query techniques
• [ ] Measure recall improvement: compare rewritten vs non-rewritten retrieval
• [ ] Monitor latency: query rewriting adds 200-500ms per query
• [ ] Cache rewritten queries for repeated or similar questions
• [ ] Use conversation-aware rewriting for follow-up questions
• [ ] Enforce document-level ACLs in all retrieval calls
• [ ] Test with real enterprise queries (short, ambiguous, complex)
• [ ] Consider semantic answer caching for query-level dedup
• [ ] Integrate with AI knowledge base pipeline
• [ ] Measure hallucination rates with and without rewriting
• [ ] A/B test rewriting techniques on your actual query distribution

FAQ

What is RAG query rewriting?

RAG query rewriting is the process of transforming a user's original query into one or more modified queries before retrieval. Users often ask questions in conversational, ambiguous, or incomplete ways that don't match the language in source documents. Query rewriting bridges this gap by making queries more specific, decomposing complex questions into sub-queries, or generating hypothetical answers to use as search queries. Techniques include multi-query rewriting, HyDE, step-back prompting, and sub-query decomposition.

What is HyDE in RAG?

HyDE (Hypothetical Document Embeddings) is a query rewriting technique where the LLM first generates a hypothetical answer to the user's question, then uses that hypothetical answer as the search query instead of the original question. The idea is that a hypothetical answer is closer in semantic space to the real answer document than the question is. HyDE improves retrieval when queries are short or conversational but adds one LLM call of latency before retrieval.

What is multi-query retrieval?

Multi-query retrieval generates multiple reformulations of the user's query, retrieves documents for each reformulation, and combines (deduplicates) the results. This improves recall by casting a wider net — different reformulations may match different relevant documents. The original query plus 3-5 reformulations is typical. Results are fused using Reciprocal Rank Fusion or simple deduplication with score averaging.

What is sub-query decomposition?

Sub-query decomposition breaks a complex question into simpler sub-questions that can each be answered independently. For example, "What's the risk profile of this customer given their recent acquisitions?" decomposes into "What are the customer's recent acquisitions?" and "What is the risk profile of companies with those acquisition patterns?" Each sub-query is retrieved separately, and results are combined for the final answer.

When should you use query rewriting in RAG?

Use query rewriting when: (1) queries are short or conversational and don't match document language, (2) queries are complex and require multiple pieces of information, (3) retrieval recall is low despite having relevant documents, (4) users ask follow-up questions that reference previous context. Don't use query rewriting for simple factual lookups where the original query already matches document language well — it adds latency without benefit.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →