RAG Conversation Memory: Multi-Turn Dialogue with Contextual Retrieval

By Fadi Updated Jan 8, 2026 12 min read RAG Fundamentals

TL;DR — RAG conversation memory enables multi-turn dialogue with contextual retrieval. ZenVanRiel: "Retrieval becomes context-dependent. Conversational RAG requires memory, reformulation, and contextual awareness. Transform contextual queries into standalone queries before retrieval." AILog: "Persistent memory system enabling contextual conversations across multiple sessions." Medium: "Sliding Window: last N messages. Summary-based: compress older messages. Memory Fusion: combine and compress." ChatNexus: "Treating each user query in isolation produces disjointed or repetitive answers." Oracle: "From RAG to AI Memory Systems: building stateful architectures." Learn more with RAG evaluation, prevent hallucinations, what is RAG, and build from scratch.

ZenVanRiel frames the challenge: "Retrieval becomes context-dependent. 'Show me the pricing' means different things depending on what product the conversation established. Conversational RAG requires mechanisms that single-turn systems don't need: memory, reformulation, and contextual awareness."

ChatNexus explains the problem: "Maintaining coherent conversations across multiple turns is a fundamental challenge for chatbots powered by RAG. RAG systems excel at grounding responses in external knowledge, but treating each user query in isolation often produces disjointed or repetitive answers."

Conversation Memory Architecture

flowchart TD subgraph Input["User Input"] Query["Contextual Query
'What about the Pro plan?'"] end subgraph Rewrite["Query Rewriting"] History["Conversation History
last N turns"] RewriteLLM["LLM Rewrite
resolve references"] Standalone["Standalone Query
'What are features and pricing
of Pro plan for Acme CRM?'"] end subgraph Retrieve["Contextual Retrieval"] Search["Hybrid Search
with standalone query"] Rerank["Cross-Encoder Rerank"] Context["Retrieved Context"] end subgraph Memory["Memory Management"] Window["Sliding Window
last 5 turns verbatim"] Summary["Summary Compression
older turns summarized"] Entities["Entity Memory
track products, people"] LongTerm["Long-Term Store
persist across sessions"] end subgraph Generate["Generation"] Assemble["Assemble Prompt
summary + recent + context"] GenLLM["LLM Generate
with conversation awareness"] Answer["Contextual Answer"] end Query --> History --> RewriteLLM --> Standalone Standalone --> Search --> Rerank --> Context Window --> Assemble Summary --> Assemble Entities --> Assemble Context --> Assemble Assemble --> GenLLM --> Answer Answer --> Window Answer --> Summary Answer --> Entities Answer --> LongTerm

Memory Strategies Comparison

Strategy	Token Budget	Coherence	Complexity	Best For
Sliding Window	Fixed (last N turns)	Good for recent	Low	Short conversations
Summary Compression	Variable (summary + recent)	Good for long	Medium	Long conversations
Token Buffer	Dynamic (until limit)	Good	Low	Medium conversations
Entity Memory	Small (entities only)	Targeted	Medium	Q&A with entities
Long-Term Persistent	Small (loaded on demand)	Cross-session	High	Returning users

Token Budget Allocation

Component	Tokens	Purpose
System prompt	200	Instructions, rules
Conversation summary	200	Compressed older context
Recent turns (sliding window)	500	Verbatim last 5 turns
Retrieved context	2000	RAG retrieval results
Current query	100	User's current question
Generation budget	1000	LLM answer generation
Total	4000	Within context window

Implementation

from dataclasses import dataclass, field
from typing import Optional
from enum import Enum
import time

class MemoryStrategy(Enum):
    SLIDING_WINDOW = "sliding_window"
    SUMMARY_COMPRESSION = "summary_compression"
    TOKEN_BUFFER = "token_buffer"
    ENTITY_MEMORY = "entity_memory"
    LONG_TERM = "long_term"

@dataclass
class ConversationTurn:
    role: str  # "user" or "assistant"
    content: str
    timestamp: float = field(default_factory=time.time)
    metadata: dict = field(default_factory=dict)

@dataclass
class RAGConversationMemory:
    """Manage conversation memory for multi-turn RAG."""

    def __init__(self, strategy: MemoryStrategy = MemoryStrategy.SUMMARY_COMPRESSION,
                 max_turns: int = 5,
                 max_summary_tokens: int = 200,
                 llm=None,
                 retriever=None,
                 reranker=None):
        self.strategy = strategy
        self.max_turns = max_turns
        self.max_summary_tokens = max_summary_tokens
        self.llm = llm
        self.retriever = retriever
        self.reranker = reranker

        self.turns: list[ConversationTurn] = []
        self.summary: str = ""
        self.entities: dict = {}
        self.session_id: str = ""

    async def chat(self, user_query: str,
                   session_id: str = None) -> dict:
        """Process a conversational query with memory."""
        if session_id:
            self.session_id = session_id

        # Step 1: Rewrite contextual query to standalone
        standalone_query = await self._rewrite_query(user_query)

        # Step 2: Retrieve with standalone query
        candidates = await self.retriever.search(
            standalone_query, top_k=15, mode="hybrid")

        # Step 3: Rerank if available
        if self.reranker:
            candidates = await self.reranker.rerank(
                standalone_query, candidates, top_k=5)
        else:
            candidates = candidates[:5]

        # Step 4: Assemble prompt with memory
        prompt = self._build_conversational_prompt(
            user_query, standalone_query, candidates)

        # Step 5: Generate answer
        answer = await self.llm.generate(prompt)

        # Step 6: Update memory
        self._add_turn("user", user_query)
        self._add_turn("assistant", answer)
        self._update_memory()

        return {
            "answer": answer,
            "standalone_query": standalone_query,
            "sources": [
                {"content": c["payload"]["content"][:200],
                 "source": c["payload"].get("source", "")}
                for c in candidates[:5]
            ],
            "memory_info": {
                "strategy": self.strategy.value,
                "turns_in_memory": len(self.turns),
                "summary_length": len(self.summary),
                "entities_tracked": len(self.entities),
            },
        }

    async def _rewrite_query(self, query: str) -> str:
        """Rewrite contextual query to standalone using LLM."""
        if not self.turns or not self.llm:
            return query

        # Build conversation context for rewriting
        recent = self._get_recent_turns(3)
        history_text = "\n".join(
            f"{t.role}: {t.content}" for t in recent)

        rewrite_prompt = f"""Given the conversation history and the user's follow-up question, 
rewrite the follow-up as a standalone question that can be understood without 
the conversation context.

Conversation history:
{history_text}

Follow-up question: {query}

Standalone question:"""

        rewritten = await self.llm.generate(rewrite_prompt)
        return rewritten.strip()

    def _build_conversational_prompt(self, user_query: str,
                                     standalone_query: str,
                                     candidates: list) -> str:
        """Build prompt with conversation memory and retrieved context."""
        # Get memory components based on strategy
        recent_turns = self._get_recent_turns(self.max_turns)
        history_text = "\n".join(
            f"{t.role.capitalize()}: {t.content}"
            for t in recent_turns)

        # Build context from retrieved chunks
        context_parts = []
        for i, c in enumerate(candidates):
            source = c["payload"].get("source", f"Source {i+1}")
            context_parts.append(
                f"[Source {i+1}: {source}]\n{c['payload']['content']}\n"
            )
        context_text = "\n".join(context_parts)

        # Assemble full prompt
        prompt = f"""You are a helpful assistant in a conversation. Answer based on the retrieved context and conversation history.

RULES:
1. Answer using the retrieved context and conversation history.
2. Cite sources using [Source N] format.
3. If the context does not contain the answer, say "I don't have enough information."
4. Maintain continuity with previous conversation turns.
5. Reference previous topics when relevant.

"""
        if self.summary:
            prompt += f"CONVERSATION SUMMARY:\n{self.summary}\n\n"

        if history_text:
            prompt += f"RECENT CONVERSATION:\n{history_text}\n\n"

        prompt += f"""RETRIEVED CONTEXT:
{context_text}

CURRENT QUESTION: {user_query}

ANSWER (with citations, maintaining conversation continuity):"""

        return prompt

    def _add_turn(self, role: str, content: str):
        """Add a conversation turn."""
        self.turns.append(ConversationTurn(
            role=role, content=content))

    def _get_recent_turns(self, n: int) -> list:
        """Get the last N conversation turns."""
        return self.turns[-n*2:] if self.turns else []

    def _update_memory(self):
        """Update memory based on strategy."""
        if self.strategy == MemoryStrategy.SLIDING_WINDOW:
            # Keep only last N turns
            max_turns = self.max_turns * 2
            if len(self.turns) > max_turns:
                self.turns = self.turns[-max_turns:]

        elif self.strategy == MemoryStrategy.SUMMARY_COMPRESSION:
            # Compress older turns into summary
            max_turns = self.max_turns * 2
            if len(self.turns) > max_turns:
                old_turns = self.turns[:-max_turns]
                self.turns = self.turns[-max_turns:]
                self._compress_turns(old_turns)

        elif self.strategy == MemoryStrategy.ENTITY_MEMORY:
            # Extract and track entities
            self._extract_entities()

    def _compress_turns(self, old_turns: list):
        """Compress old turns into a summary.

        In production: use LLM to generate summary.
        Here: simple concatenation.
        """
        if not old_turns:
            return

        turn_text = "\n".join(
            f"{t.role}: {t.content}" for t in old_turns)

        if self.llm:
            # Use LLM to summarize
            # In production: await self.llm.generate(summary_prompt)
            pass

        # Simple: append to existing summary
        new_summary = f"Previous context: {turn_text[:500]}..."
        if self.summary:
            self.summary = f"{self.summary}\n{new_summary}"
        else:
            self.summary = new_summary

        # Keep summary within token budget
        words = self.summary.split()
        if len(words) > self.max_summary_tokens:
            self.summary = " ".join(words[-self.max_summary_tokens:])

    def _extract_entities(self):
        """Extract entities from conversation.

        In production: use NER model or LLM.
        Here: simple keyword extraction.
        """
        for turn in self.turns[-2:]:
            words = turn.content.split()
            for word in words:
                if word[0].isupper() and len(word) > 3:
                    self.entities[word] = self.entities.get(word, 0) + 1

    def get_memory_state(self) -> dict:
        """Get current memory state for debugging."""
        return {
            "strategy": self.strategy.value,
            "total_turns": len(self.turns),
            "recent_turns": len(self._get_recent_turns(self.max_turns)),
            "summary_length": len(self.summary),
            "summary_preview": self.summary[:200] if self.summary else "",
            "entities": dict(list(self.entities.items())[:10]),
            "session_id": self.session_id,
        }

    def clear(self):
        """Clear conversation memory."""
        self.turns = []
        self.summary = ""
        self.entities = {}

RAG Conversation Memory Checklist

[ ] Implement query rewriting to transform contextual queries into standalone
[ ] "What about the Pro plan?" → "What are features and pricing of Pro plan for Acme CRM?"
[ ] Use LLM to rewrite queries based on conversation history
[ ] Store conversation history per session
[ ] Include conversation summary in generation context
[ ] Retrieve with the rewritten standalone query, not the raw contextual query
[ ] Sliding Window: keep last N turns (e.g., 5 turns) in memory verbatim
[ ] Sliding Window pros: simple, predictable token budget
[ ] Sliding Window cons: loses older context
[ ] Summary Compression: compress older turns into a summary
[ ] Summary Compression pros: retains key information from long conversations
[ ] Summary Compression cons: lossy compression
[ ] Token Buffer: keep turns until token limit, then compress
[ ] Entity Memory: track entities (products, people, decisions) mentioned
[ ] Long-Term Persistent: store summaries across sessions in database
[ ] Long-Term enables assistant to remember user preferences across sessions
[ ] Token budget allocation: system prompt 200, summary 200, recent 500, context 2000, query 100, generation 1000
[ ] Total token budget: ~4000 tokens within context window
[ ] When conversation exceeds budget: compress oldest turns into summary
[ ] Use LLM to generate summary: "Summarize the key facts and decisions"
[ ] At session end: generate conversation summary and extract key entities
[ ] At new session start: load previous summaries as context
[ ] Retrieve relevant past conversations using semantic search on summaries
[ ] Update user profile with new information from each session
[ ] Conversational RAG requires memory, reformulation, and contextual awareness
[ ] Single-turn RAG treats each query in isolation — produces disjointed answers
[ ] Retrieval becomes context-dependent in multi-turn conversations
[ ] "Show me the pricing" means different things depending on conversation context
[ ] Maintain continuity with previous conversation turns
[ ] Reference previous topics when relevant in answers
[ ] Memory fusion: combine and compress older messages — balanced approach
[ ] From RAG to AI Memory Systems: building stateful architectures
[ ] EU AI Act requires higher bar on data governance, audit, and human oversight
[ ] Persistent memory enables contextual conversations across multiple sessions
[ ] Track entities: products mentioned, people referenced, decisions made
[ ] Store entity memory separately for targeted context injection
[ ] Use semantic search on conversation summaries for long-term retrieval
[ ] Implement session management: session_id, user_id, conversation_id
[ ] Clear memory between unrelated conversations
[ ] Provide memory state debugging for development
[ ] Read RAG evaluation for multi-turn metrics
[ ] Read prevent hallucinations for grounding
[ ] Read what is RAG for architecture
[ ] Build RAG from scratch with memory
[ ] Add hybrid search with rewritten queries
[ ] Add reranking after contextual retrieval
[ ] Apply chunking strategies for retrieval
[ ] Evaluate with RAG metrics including multi-turn
[ ] Test: query rewriting resolves contextual references correctly
[ ] Test: sliding window maintains last N turns
[ ] Test: summary compression preserves key facts
[ ] Test: long-term memory persists across sessions
[ ] Test: token budget stays within context window limit
[ ] Document memory strategy, token budget, and session management

FAQ

How do you implement conversation memory in RAG?

Implement RAG conversation memory with query rewriting, dialogue history management, and contextual retrieval. ZenVanRiel: "Retrieval becomes context-dependent. Show me the pricing means different things depending on what product the conversation established. Conversational RAG requires mechanisms that single-turn systems do not need: memory, reformulation, and contextual awareness. Transform contextual queries into standalone queries before retrieval." AILog: "A classic RAG system processes each query independently. But users expect continuous conversations where the assistant remembers previous exchanges. This guide explains how to implement a persistent memory system enabling contextual conversations across multiple sessions." ChatNexus: "Maintaining coherent conversations across multiple turns is a fundamental challenge. RAG systems excel at grounding responses in external knowledge, but treating each user query in isolation often produces disjointed or repetitive answers." Steps: (1) Store conversation history per session. (2) Rewrite contextual queries into standalone queries using LLM. (3) Retrieve with the rewritten query. (4) Include conversation summary in generation context. (5) Manage memory with sliding window or summary compression.

What is query rewriting in conversational RAG?

Query rewriting transforms contextual user queries into standalone queries that can be used for retrieval without conversation history. ZenVanRiel: "Transform contextual queries into standalone queries before retrieval. The user asks Show me the pricing — the system needs to resolve what product they mean based on conversation history, producing a standalone query like Show me the pricing for Acme CRM Enterprise plan." Medium: "Query rewriting: use LLM to reformulate the user question based on conversation history. Sliding window: only keeps the last N messages in memory. Memory fusion: combine and compress older messages into a summary." ChatNexus: "Contextual RAG maintains conversation context by rewriting queries to include relevant context from previous turns." Example: User asks 'What about the Pro plan?' → Rewrite to 'What are the features and pricing of the Pro plan for Acme CRM?' → Retrieve with standalone query.

What are the different RAG memory strategies?

RAG memory strategies range from simple sliding windows to sophisticated summary-based compression and long-term persistent memory. Medium: "Sliding Window: only keeps the last N messages (e.g., 5 turns) in memory. Pros: simple, predictable token budget. Cons: loses older context. Summary-based: compress older messages into a summary. Pros: retains key information. Cons: lossy compression. Memory Fusion: combine and compress older messages. Pros: balanced. Cons: more complex." AILog: "Persistent memory system enabling contextual conversations across multiple sessions." Oracle: "From RAG to AI Memory Systems: building stateful architectures." Strategies: (1) Sliding Window — last N turns. (2) Summary Compression — summarize older turns. (3) Token Buffer — keep until token limit. (4) Entity Memory — track entities mentioned. (5) Long-term Persistent — store across sessions in database.

How do you manage token budget in conversational RAG?

Manage token budget by combining sliding window for recent context with summary compression for older context. Medium: "Sliding Window keeps the last N messages. Pros: simple, predictable token budget. Cons: loses older context. Summary-based compression: pros: retains key information. Cons: lossy." ZenVanRiel: "Conversational RAG requires mechanisms that single-turn systems do not need: memory, reformulation, and contextual awareness." Token budget allocation: (1) System prompt: ~200 tokens. (2) Conversation summary: ~200 tokens. (3) Recent turns (sliding window): ~500 tokens. (4) Retrieved context: ~2000 tokens. (5) Current query: ~100 tokens. (6) Generation budget: ~1000 tokens. Total: ~4000 tokens. When conversation exceeds budget: compress oldest turns into summary, keep recent turns verbatim. Use LLM to generate summary: 'Summarize the key facts and decisions from this conversation.'

How do you implement long-term memory across sessions?

Implement long-term memory by persisting conversation summaries and key entities in a database, then loading them for future sessions. AILog: "Complete guide to implementing a persistent memory system enabling contextual conversations across multiple sessions." Oracle: "From RAG to AI Memory Systems: building stateful architectures. The EU AI Act high-risk obligations require a higher bar on data governance, audit, and human oversight." Implementation: (1) At session end, generate a conversation summary with LLM. (2) Extract key entities (products, people, decisions). (3) Store summary + entities in user profile database. (4) At new session start, load previous summaries. (5) Include previous summaries in system prompt as context. (6) Retrieve relevant past conversations using semantic search on summaries. (7) Update user profile with new information. This enables the assistant to remember user preferences, past decisions, and ongoing projects across sessions.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →