Multimodal RAG: Retrieval Over Images, Charts, Tables, and Text in 2026

TL;DR — Multimodal RAG retrieves over images, charts, tables, and text using vision-language models. HelainZimmermann: "Dual-encoder pattern: separate text and visual encoders, shared vector store, cross-modal score fusion. Images, charts, and visual elements as first-class retrievable units." BigDataBoutique: "ViDoRe V1 saturated at 90+ nDCG@5. V2 is harder, multilingual. ColPali multi-vector vs single-vector (Cohere Embed 4, Voyage multimodal) gap narrowing." Medium: "ColPali: index visually-rich documents, run efficient approximate search. Visually rich docs with tables, images, formulas, charts make multimodal search essential." DevTo: "Build RAG pipeline with vision-based ColPali model. Integrate VLMs into agentic RAG." EmergentMind: "ColQwen2: 100K+ maps, <1s/25K images, multimodal summarization via Llama 3.2." Learn more with document indexing, embeddings, embedding models, and what is RAG.

HelainZimmermann frames the evolution: "Multimodal RAG extends text-only pipelines by treating images, charts, and visual elements as first-class retrievable units alongside text chunks. The architecture follows a dual-encoder pattern: separate text and visual encoders feeding into a shared vector store with cross-modal score fusion at query time."

Medium identifies the problem: "Many documents are visually rich, containing tables, images, formulas, charts, and so on, which makes multimodal search essential. In real-world scenarios, the bottleneck in a RAG system is often the retrieval quality on visually-rich documents."

Multimodal RAG Architecture

Multimodal RAG Approaches Comparison

ViDoRe Benchmark Results

Implementation

from dataclasses import dataclass, field
from typing import Optional
from enum import Enum
import asyncio

class MultimodalApproach(Enum):
    DUAL_ENCODER = "dual_encoder"
    MULTI_VECTOR = "multi_vector"
    SINGLE_VECTOR = "single_vector"
    TEXT_DESCRIPTION = "text_description"

@dataclass
class MultimodalRAG:
    """Multimodal RAG: retrieval over images, charts, tables, and text."""

    def __init__(self, approach: MultimodalApproach = MultimodalApproach.MULTI_VECTOR,
                 vlm_model: str = "colqwen2",
                 generation_vlm: str = "llama-3.2-vision",
                 vector_store=None,
                 embedder=None):
        self.approach = approach
        self.vlm_model = vlm_model
        self.generation_vlm = generation_vlm
        self.vector_store = vector_store
        self.embedder = embedder

    async def index_document(self, file_path: str,
                             page_numbers: list = None) -> dict:
        """Index a document for multimodal retrieval."""
        # Step 1: Render pages as images
        page_images = await self._render_pages(file_path, page_numbers)

        # Step 2: Generate embeddings based on approach
        if self.approach == MultimodalApproach.MULTI_VECTOR:
            embeddings = await self._colpali_embed(page_images)
        elif self.approach == MultimodalApproach.DUAL_ENCODER:
            embeddings = await self._clip_embed(page_images)
        elif self.approach == MultimodalApproach.SINGLE_VECTOR:
            embeddings = await self._unified_embed(page_images)
        elif self.approach == MultimodalApproach.TEXT_DESCRIPTION:
            embeddings = await self._text_description_embed(page_images)

        # Step 3: Store in vector database
        stored = []
        for i, (page_img, embedding) in enumerate(
                zip(page_images, embeddings)):
            point = {
                "id": f"{file_path}_page_{i}",
                "vector": embedding,
                "payload": {
                    "file_path": file_path,
                    "page_number": i,
                    "page_image": page_img,
                    "approach": self.approach.value,
                    "vlm_model": self.vlm_model,
                },
            }
            if self.vector_store:
                await self.vector_store.upsert([point])
            stored.append(point)

        return {
            "file_path": file_path,
            "pages_indexed": len(stored),
            "approach": self.approach.value,
            "vlm_model": self.vlm_model,
        }

    async def search(self, query: str,
                     top_k: int = 5,
                     query_image: str = None) -> dict:
        """Search multimodal index with text or image query."""
        # Generate query embedding
        if query_image:
            query_embedding = await self._embed_image(query_image)
        else:
            query_embedding = await self._embed_query(query)

        # Search vector store
        if self.approach == MultimodalApproach.MULTI_VECTOR:
            results = await self._multi_vector_search(
                query_embedding, top_k)
        else:
            results = await self.vector_store.search(
                vector=query_embedding, limit=top_k)

        return {
            "query": query,
            "query_type": "image" if query_image else "text",
            "results": results[:top_k],
            "approach": self.approach.value,
        }

    async def answer(self, query: str,
                     top_k: int = 5) -> dict:
        """Generate answer using multimodal RAG."""
        # Retrieve relevant pages
        search_results = await self.search(query, top_k)

        # Build multimodal context (page images)
        page_images = [
            r["payload"]["page_image"]
            for r in search_results["results"]
        ]

        # Generate with VLM
        prompt = self._build_multimodal_prompt(
            query, page_images)
        answer = await self._vlm_generate(prompt, page_images)

        return {
            "answer": answer,
            "query": query,
            "sources": [
                {
                    "file_path": r["payload"]["file_path"],
                    "page_number": r["payload"]["page_number"],
                    "score": r.get("score", 0),
                }
                for r in search_results["results"]
            ],
            "approach": self.approach.value,
            "generation_vlm": self.generation_vlm,
        }

    async def _render_pages(self, file_path: str,
                            page_numbers: list = None) -> list:
        """Render document pages as images.

        In production: use pdf2image or PyMuPDF (fitz)
        import fitz
        doc = fitz.open(file_path)
        pages = []
        for page in doc:
            pix = page.get_pixmap()
            pages.append(pix.tobytes("png"))
        """
        # Simulated: return placeholder page images
        return [f"page_image_{i}" for i in range(3)]

    async def _colpali_embed(self, page_images: list) -> list:
        """Generate ColPali multi-vector embeddings.

        In production: load ColQwen2 from HuggingFace
        from colpali_engine.models import ColQwen2, ColQwen2Processor
        model = ColQwen2.from_pretrained("vidore/colqwen2-v1.0")
        processor = ColQwen2Processor.from_pretrained(...)
        # Generate multi-vector embeddings for each page
        """
        return [[0.1] * 128 for _ in page_images]

    async def _clip_embed(self, page_images: list) -> list:
        """Generate CLIP dual-encoder embeddings.

        In production: use open_clip or transformers CLIPModel
        from transformers import CLIPModel, CLIPProcessor
        model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
        """
        return [[0.1] * 768 for _ in page_images]

    async def _unified_embed(self, page_images: list) -> list:
        """Generate single-vector unified embeddings.

        In production: use Cohere Embed 4 or Voyage multimodal-3
        import cohere
        client = cohere.Client(api_key)
        response = client.embed(
            texts=[], images=page_images,
            model="embed-multimodal-v4.0"
        )
        """
        return [[0.1] * 1024 for _ in page_images]

    async def _text_description_embed(self,
                                      page_images: list) -> list:
        """Generate text descriptions and embed as text.

        In production: use VLM to describe each image
        description = await vlm.describe(page_image)
        embedding = await text_embedder.embed(description)
        """
        return [[0.1] * 768 for _ in page_images]

    async def _embed_query(self, query: str) -> list:
        """Embed text query for cross-modal search."""
        return [0.1] * 128

    async def _embed_image(self, image_path: str) -> list:
        """Embed image query for cross-modal search."""
        return [0.1] * 128

    async def _multi_vector_search(self,
                                   query_embedding: list,
                                   top_k: int) -> list:
        """Multi-vector search with MaxSim late interaction.

        In production: use Vespa or Qdrant multi-vector support
        Score = sum over query tokens of max over doc tokens
        of dot product.
        """
        return [
            {
                "id": "doc_page_0",
                "score": 0.95,
                "payload": {
                    "file_path": "doc.pdf",
                    "page_number": 0,
                    "page_image": "page_image_0",
                },
            }
        ][:top_k]

    def _build_multimodal_prompt(self, query: str,
                                 page_images: list) -> str:
        """Build prompt for VLM with page images as context."""
        return f"""You are a document assistant. Answer the question based on the provided page images.

RULES:
1. Answer based ONLY on the visual and text content in the provided pages.
2. If the pages do not contain the answer, say "I don't have enough information."
3. Reference the page number for each claim (e.g., [Page 1]).
4. Describe charts, tables, and visual elements when relevant.

QUESTION: {query}

Answer based on the {len(page_images)} page images provided."""

    async def _vlm_generate(self, prompt: str,
                            page_images: list) -> str:
        """Generate answer using vision-language model.

        In production: use Llama 3.2 Vision or GPT-4o
        response = await vlm.chat(
            messages=[{"role": "user", "content": [
                {"type": "text", "text": prompt},
                *[{"type": "image", "image": img}
                  for img in page_images]
            ]}]
        )
        """
        return f"[Answer based on {len(page_images)} page images]"

Multimodal RAG Checklist

• [ ] Use multimodal RAG for documents with images, charts, tables, formulas
• [ ] Text-only RAG loses visual information — multimodal preserves it
• [ ] Visually-rich documents require multimodal search for accurate retrieval
• [ ] The bottleneck in RAG for visual docs is often retrieval quality
• [ ] ColPali: multi-vector retrieval using vision-language model
• [ ] ColPali renders pages as images and embeds with VLM (ColQwen2)
• [ ] ColPali uses late-interaction (MaxSim) for query-page similarity
• [ ] ColQwen2: production-proven, 100K+ documents, <1s/25K images
• [ ] CLIP: dual-encoder approach, separate text and image encoders
• [ ] CLIP: shared embedding space, cosine similarity for cross-modal
• [ ] CLIP: simpler but loses fine-grained visual details (~65 nDCG@5)
• [ ] Cohere Embed 4: single-vector unified, narrowing gap with ColPali
• [ ] Voyage multimodal-3: single-vector, strong on ViDoRe V2
• [ ] Text-description: generate text description of image with VLM, embed as text
• [ ] Text-description: simplest but lossy — loses visual nuances
• [ ] Dual-encoder pattern: separate text and visual encoders, shared vector store
• [ ] Cross-modal score fusion at query time
• [ ] Images, charts, and visual elements as first-class retrievable units
• [ ] ViDoRe V1 saturated: ColPali >90 nDCG@5, CLIP ~65 nDCG@5
• [ ] ViDoRe V2 (May 2025): harder, more diverse, multilingual
• [ ] Gap between ColPali multi-vector and single-vector models narrowing
• [ ] Render PDF pages as images using pdf2image or PyMuPDF
• [ ] Store page images in vector database payload for generation
• [ ] Use VLM (Llama 3.2 Vision, GPT-4o) for multimodal answer generation
• [ ] Pass page images + query to VLM for answer generation
• [ ] VLM can read charts, tables, formulas, and visual layouts
• [ ] Reference page numbers in answers for citation grounding
• [ ] Support both text and image queries for cross-modal search
• [ ] ColPali pipeline: render → VLM embed → store → query embed → MaxSim → retrieve
• [ ] MaxSim: sum over query tokens of max over doc tokens of dot product
• [ ] Use Vespa or Qdrant for multi-vector storage and search
• [ ] ColQwen2 available on HuggingFace for self-hosted deployment
• [ ] Map-RAS: ColQwen2 for historic map collections, multimodal summarization
• [ ] Production: 100K+ documents with sub-second search latency
• [ ] Choose approach based on accuracy needs vs deployment simplicity
• [ ] Multi-vector (ColPali): best accuracy, higher storage, complex similarity
• [ ] Single-vector (Cohere/Voyage): simpler deployment, narrowing accuracy gap
• [ ] Dual-encoder (CLIP): general purpose, lower accuracy on complex docs
• [ ] Text-description: quickest prototype, lossy for production
• [ ] Read document indexing for ingestion
• [ ] Read embeddings for fundamentals
• [ ] Read embedding models for selection
• [ ] Read what is RAG for architecture
• [ ] Build RAG from scratch with multimodal
• [ ] Add reranking after multimodal retrieval
• [ ] Evaluate with RAG metrics
• [ ] Test: multimodal retrieval outperforms text-only on visual documents
• [ ] Test: ColPali MaxSim produces correct page rankings
• [ ] Test: VLM generates accurate answers from page images
• [ ] Test: both text and image queries work in cross-modal search
• [ ] Test: production latency <1s for 25K images
• [ ] Document approach, VLM model, vector store, and benchmark results

FAQ

What is multimodal RAG and how does it differ from text-only RAG?

Multimodal RAG extends text-only pipelines by treating images, charts, tables, and visual elements as first-class retrievable units alongside text chunks. HelainZimmermann: "Multimodal RAG extends text-only pipelines by treating images, charts, and visual elements as first-class retrievable units alongside text chunks. The architecture follows a dual-encoder pattern: separate text and visual encoders feeding into a shared vector store with cross-modal score fusion at query time." BigDataBoutique: "Multimodal RAG in 2026: retrieval over images, PDFs, and text. ViDoRe V1 is now saturated with several models exceeding 90 nDCG@5. ViDoRe V2 (May 2025) is harder, more diverse, and multilingual." Medium: "Many documents are visually rich, containing tables, images, formulas, charts, which makes multimodal search essential. In real-world scenarios, the bottleneck in a RAG system is often the retrieval quality on visually-rich documents." Key difference: text-only RAG loses visual information (tables, charts, layouts); multimodal RAG preserves and retrieves it.

What is ColPali and how does it work for multimodal RAG?

ColPali is a multi-vector retrieval model that embeds document pages as images using a vision-language model, enabling retrieval without text extraction. Medium: "ColPali: index visually-rich documents and run efficient approximate search. Many documents are visually rich, containing tables, images, formulas, charts, which makes multimodal search essential. ColPali embeds document pages as images using a vision-language model." DevTo: "Build a RAG pipeline using vision-based model based on ColPali. Integrating vision-language models into agentic RAG systems." EmergentMind: "ColPali methodologies underpin diverse production search and RAG systems: Map-RAS for historic map collections embeds 100K+ Library of Congress maps with ColQwen2; enables text/image queries, search latency <1s/25K images, multimodal summarization via Llama 3.2." ColPali: (1) Renders each page as an image. (2) Uses VLM (Qwen2-VL) to generate multi-vector embeddings (token-level). (3) Stores multi-vectors in vector database. (4) At query time, embeds text query and computes late-interaction similarity. (5) Returns page images as retrieved context.

What are the approaches to multimodal RAG?

There are three main approaches: dual-encoder (CLIP), multi-vector (ColPali), and single-vector unified (Cohere Embed 4, Voyage multimodal). HelainZimmermann: "Dual-encoder pattern: separate text and visual encoders feeding into a shared vector store with cross-modal score fusion at query time." BigDataBoutique: "The gap between ColPali-style multi-vector models and well-tuned single-vector models (Cohere Embed 4, voyage-multimodal-3.x) is narrowing on ViDoRe V2." Approaches: (1) Dual-encoder (CLIP): separate text and image encoders, shared embedding space, cosine similarity for cross-modal. (2) Multi-vector (ColPali/ColQwen2): page-as-image, token-level multi-vector embeddings, late interaction similarity. (3) Single-vector unified (Cohere Embed 4, Voyage multimodal-3): single model embeds both text and images into same vector space. (4) Text-description: generate text description of image with VLM, embed as text (simpler but lossy).

How do you implement multimodal RAG with ColPali?

Implement ColPali by rendering pages as images, embedding with ColQwen2 VLM, storing multi-vectors, and querying with late-interaction similarity. Medium: "Step-by-step code examples to index visually-rich documents and run efficient approximate search with ColPali." DevTo: "Build a RAG pipeline using vision-based model based on ColPali. Integrating VLMs into agentic RAG systems." EmergentMind: "Map-RAS embeds 100K+ maps with ColQwen2; search latency <1s/25K images." Steps: (1) Render each PDF page as an image (using pdf2image or PyMuPDF). (2) Load ColQwen2 model (from HuggingFace). (3) Generate multi-vector embeddings for each page image. (4) Store multi-vectors in vector database (Qdrant, Vespa, or custom). (5) At query time, embed text query with same model. (6) Compute late-interaction similarity (MaxSim) between query tokens and page tokens. (7) Return top-k page images as retrieved context. (8) Pass page images + query to VLM (Llama 3.2 Vision) for answer generation.

What are the benchmarks for multimodal RAG?

ViDoRe (Visual Document Retrieval) is the standard benchmark, with V1 saturated and V2 providing harder, multilingual evaluation. BigDataBoutique: "ViDoRe V1 is now saturated, with several models exceeding 90 nDCG@5. The newer ViDoRe V2 (May 2025) is harder, more diverse, and multilingual. The gap between ColPali-style multi-vector models and well-tuned single-vector models (Cohere Embed 4, voyage-multimodal-3.x) is narrowing." EmergentMind: "ColPali production systems: Map-RAS with 100K+ maps, search latency <1s/25K images. ColQwen2 for historic map collections." Medium: "In real-world scenarios, the bottleneck in a RAG system is often the retrieval quality on visually-rich documents." Benchmark results: (1) ViDoRe V1: ColPali >90 nDCG@5, CLIP ~65 nDCG@5. (2) ViDoRe V2: ColPali and single-vector models converging. (3) Production: ColQwen2 handles 100K+ images with <1s latency. (4) Text-only RAG on visually-rich docs: significant accuracy loss vs multimodal.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →