Semantic Answer Caching: Cut RAG Costs by 50% with Embedding Similarity

TL;DR — Semantic answer caching stores LLM responses indexed by query embeddings, not exact text. When a new query arrives, the system computes its embedding and searches for similar cached queries. If cosine similarity is above 0.92-0.97, the cached response is returned without calling the LLM. This cuts API costs by 20-50%, reduces latency from 500-3000ms to 25-60ms for cache hits, and achieves 25-50% hit rates (3-5x higher than exact-match caching). GPTCache is the leading open-source solution. Redis Stack, LangChain SemanticCache, and pgvector also support it. Start with threshold 0.95, tune based on your query distribution. Combine with provider prompt caching (OpenAI, Anthropic) for maximum savings. For enterprise RAG with 500K queries/month, semantic caching saves ~$1,000/month in API costs. Don't cache: personalized queries, time-sensitive answers, or queries with low similarity confidence.

Semantic caching is the single most underutilized cost optimization technique for RAG. Teams implementing it typically reduce AI API costs by 20-50%, with some high-traffic applications seeing savings above 60%.

The idea is simple: many users ask similar questions. "What is our refund policy?" and "How do I get a refund?" are the same question in different words. Exact-match caching misses this. Semantic caching recognizes the semantic equivalence and returns the cached answer without calling the LLM.

Exact-Match vs Semantic Caching

How Semantic Caching Works

Implementation with GPTCache

from gptcache import cache
from gptcache.adapter import openai
from gptcache.embedding import Onnx as Embedding
from gptcache.similarity_evaluation import SearchDistanceEvaluation

# Initialize semantic cache
cache.init(
    embedding_func=Embedding(),
    similarity_evaluation=SearchDistanceEvaluation(),
    data_dir="semantic_cache",
)

# Use cached OpenAI — automatically checks cache before calling API
response = openai.ChatCompletion.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is our refund policy?"}],
)
# If "How do I get a refund?" was previously cached, this returns instantly

Implementation with pgvector

class SemanticCache:
    """Semantic cache using pgvector."""

    def __init__(self, conn, embedder, threshold=0.95):
        self.conn = conn
        self.embedder = embedder
        self.threshold = threshold

    def get_or_generate(self, query: str, generate_fn) -> str:
        # Compute query embedding
        query_embedding = self.embedder.embed(query)

        # Search for similar cached queries
        result = self.conn.execute("""
            SELECT answer, 1 - (embedding <=> %s) AS similarity
            FROM semantic_cache
            WHERE 1 - (embedding <=> %s) > %s
            ORDER BY embedding <=> %s
            LIMIT 1
        """, (query_embedding, query_embedding, self.threshold, query_embedding))

        row = result.fetchone()

        if row and row[1] >= self.threshold:
            # Cache hit — return cached answer
            return row[0]

        # Cache miss — generate new answer
        answer = generate_fn(query)

        # Store in cache
        self.conn.execute("""
            INSERT INTO semantic_cache (query, answer, embedding, created_at)
            VALUES (%s, %s, %s, NOW())
        """, (query, answer, query_embedding))

        return answer

Implementation with Redis

import redis
import numpy as np

class RedisSemanticCache:
    """Semantic cache using Redis Stack with vector search."""

    def __init__(self, redis_url, embedder, threshold=0.95):
        self.r = redis.from_url(redis_url)
        self.embedder = embedder
        self.threshold = threshold

    def get_or_generate(self, query: str, generate_fn) -> str:
        query_embedding = self.embedder.embed(query)

        # Search Redis for similar cached queries
        results = self.r.ft("semantic_cache").search(
            Query("*=>[KNN 1 @embedding $vec AS similarity]")
            .add_param("vec", np.array(query_embedding, dtype=np.float32).tobytes())
            .return_fields("answer", "similarity")
        )

        if results and results.docs:
            similarity = 1 - float(results.docs[0].similarity)
            if similarity >= self.threshold:
                return results.docs[0].answer

        # Cache miss — generate and store
        answer = generate_fn(query)
        self.r.hset(f"cache:{hash(query)}", mapping={
            "query": query,
            "answer": answer,
            "embedding": np.array(query_embedding, dtype=np.float32).tobytes(),
        })

        return answer

Cache Hit Rates by Use Case

Cost Savings Calculation

Assumes $0.00425/query for GPT-4o, $0.0012/query for Claude Sonnet, infrastructure-only for self-hosted.

When NOT to Cache

Combining with Provider Prompt Caching

Provider prompt caching and semantic caching are complementary. Use both.

Semantic Caching Checklist

• [ ] Choose cache backend: GPTCache (Python), Redis Stack, pgvector, or LangChain SemanticCache
• [ ] Set similarity threshold: start at 0.95, tune based on false positive rate
• [ ] Use 0.92-0.95 for factual Q&A (higher hit rate, acceptable false positive risk)
• [ ] Use 0.95-0.97 for nuanced queries (lower false positive risk)
• [ ] Compute query embedding before cache search (~10ms overhead)
• [ ] Store query embedding + answer + timestamp in cache
• [ ] Set TTL for cache entries (24-72 hours for most use cases)
• [ ] Implement cache invalidation when source documents change
• [ ] Monitor cache hit rate (target: 25-50%)
• [ ] Monitor false positive rate (users reporting wrong cached answers)
• [ ] Don't cache personalized queries (or cache per-user with user ID)
• [ ] Don't cache time-sensitive queries (or use short TTL)
• [ ] Combine with provider prompt caching (OpenAI, Anthropic) for maximum savings
• [ ] Log cache hits/misses for cost analysis
• [ ] Measure cost savings: compare API costs before and after caching
• [ ] For secure ingestion: validate inputs before caching
• [ ] Enforce document-level ACLs — cache per ACL group
• [ ] Integrate with AI knowledge base pipeline
• [ ] Consider content hash deduplication for ingestion-level dedup
• [ ] Use query rewriting to normalize queries before cache lookup
• [ ] Test with adversarial queries to ensure cache doesn't serve wrong answers
• [ ] For hallucination prevention: cached answers skip faithfulness check

FAQ

What is semantic answer caching in RAG?

Semantic answer caching stores LLM responses indexed by query embeddings instead of exact text. When a new query arrives, the system computes its embedding and searches for similar cached queries. If a cached query has cosine similarity above a threshold (typically 0.92-0.97), the cached response is returned without calling the LLM. This reduces API costs by 20-50% and latency from 500-3000ms to 25-60ms for cache hits. Hit rates are 25-50% (3-5x higher than exact-match caching's 5-15%).

How is semantic caching different from exact-match caching?

Exact-match caching only returns cached results when the query text is identical to a previous query (5-15% hit rate). Semantic caching compares query embeddings and returns cached results for semantically similar queries (25-50% hit rate). "What is our refund policy?" and "How do I get a refund?" are different text but the same semantic intent — semantic caching recognizes this and returns the cached answer. The tradeoff is a small embedding computation per query (~10ms) vs. a full LLM call (~500-3000ms).

What similarity threshold should I use for semantic caching?

Use 0.92-0.97 cosine similarity as the cache hit threshold. Below 0.92, you risk returning answers to different questions (false positives). Above 0.97, hit rates drop to near exact-match levels. Start at 0.95 and tune based on your query distribution. For factual Q&A (support, policies), use 0.92-0.95 for higher hit rates. For nuanced or creative queries, use 0.95-0.97 to avoid false cache hits. Monitor cache hit quality — if users report wrong answers, raise the threshold.

How much does semantic caching save?

Semantic caching reduces LLM API costs by 20-50% with 25-50% hit rates. For a 500K queries/month workload on GPT-4o ($2,125/month), semantic caching with 48% hit rate saves $1,027/month. Response time drops from 500-3000ms to 25-60ms for cache hits. The embedding computation cost (~$0.0001/query) is negligible compared to LLM generation cost (~$0.004/query). GPTCache reports 10x cost reduction and 100x latency improvement for cache-heavy workloads.

Which tools support semantic caching for RAG?

GPTCache (open-source, Python, integrates with LangChain and LlamaIndex) is the most mature purpose-built solution. Redis Stack with vector search is ideal for teams already using Redis. LangChain has a built-in SemanticCache class. For PostgreSQL users, pgvector can implement semantic caching with a simple SQL query. Momento offers a managed semantic cache. For self-hosted enterprise: GPTCache with pgvector or Redis Stack. Combine with provider prompt caching (OpenAI, Anthropic) for maximum savings.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →