What Are Embeddings? Vector Representations for Semantic AI

TL;DR — Embeddings are dense vectors of floating-point numbers where proximity equals semantic similarity. AI/TLDR: "An embedding model accepts a sentence and returns a list of numbers — a precise location in a high-dimensional space of meaning. The only question that matters is: how close are two addresses? Close means similar; far means unrelated." ML Journey: "Embeddings convert text into dense vectors where similar meanings end up close together. This geometric relationship powers semantic search, RAG, clustering, classification, deduplication, and recommendations." Let's Data Science: "Text embeddings convert language into vectors where proximity equals similarity. The field moved from 300-dim Word2Vec in 2013 to 4,096-dim instruction-tuned models in 2026 handling 100+ languages." Explainx: "Embeddings are the single most important low-level primitive in applied AI. Three stages: tokenization, transformer encoding, pooling. L2 normalization enables faster dot product similarity." Learn more with choose embedding model, cosine vs dot product, what is RAG, and hybrid search.

Explainx frames the importance: "Embeddings are the single most important low-level primitive in applied AI. Yet most tutorials skip past them in a paragraph on the way to showing you a LangChain tutorial. This guide goes the other direction — starting from first principles, building up the math, and arriving at production-ready practices."

Let's Data Science gives the core intuition: "A customer types 'my laptop keeps freezing after updates' into your tech support search bar. Your knowledge base has an article titled 'System hangs following OS patch installation.' Zero words overlap, yet these describe the exact same problem. Text embeddings are the reason modern search engines connect them. They convert text into numerical vectors where meaning, not spelling, determines proximity."

Embedding Architecture

Embedding Model Dimensions Comparison

Sparse vs Dense Embeddings

Implementation

import math
from dataclasses import dataclass
from typing import Optional
from enum import Enum

class SimilarityMetric(Enum):
    COSINE = "cosine"
    DOT_PRODUCT = "dot_product"
    EUCLIDEAN = "euclidean"

@dataclass
class EmbeddingExplainer:
    """Demonstrate how embeddings work: creation, similarity, search."""

    def __init__(self, embedding_model=None):
        self.model = embedding_model

    def explain_embedding(self, text: str) -> dict:
        """Explain the embedding process for a given text."""
        # In production: use actual embedding model
        # Here: demonstrate the pipeline stages

        # Stage 1: Tokenization
        tokens = self._tokenize(text)

        # Stage 2: Transformer encoding (simulated)
        # In production: self.model.encode(text)
        raw_vector = self._simulate_embedding(text, dimensions=1536)

        # Stage 3: L2 Normalization
        normalized = self._l2_normalize(raw_vector)

        return {
            "input_text": text,
            "tokens": tokens,
            "token_count": len(tokens),
            "raw_vector_dim": len(raw_vector),
            "raw_vector_sample": raw_vector[:5],
            "normalized_vector_sample": normalized[:5],
            "vector_magnitude_before_norm": round(
                math.sqrt(sum(x*x for x in raw_vector)), 4),
            "vector_magnitude_after_norm": round(
                math.sqrt(sum(x*x for x in normalized)), 4),
            "is_normalized": abs(
                math.sqrt(sum(x*x for x in normalized)) - 1.0) < 0.001,
        }

    def compute_similarity(self, text1: str, text2: str,
                           metric: SimilarityMetric = SimilarityMetric.COSINE
                           ) -> dict:
        """Compute similarity between two texts."""
        vec1 = self._simulate_embedding(text1, 1536)
        vec2 = self._simulate_embedding(text2, 1536)

        # Normalize for cosine and dot product
        norm1 = self._l2_normalize(vec1)
        norm2 = self._l2_normalize(vec2)

        cosine_sim = self._cosine_similarity(vec1, vec2)
        dot_product = sum(a * b for a, b in zip(norm1, norm2))
        euclidean_dist = math.sqrt(
            sum((a - b) ** 2 for a, b in zip(norm1, norm2)))

        return {
            "text1": text1,
            "text2": text2,
            "cosine_similarity": round(cosine_sim, 4),
            "dot_product_normalized": round(dot_product, 4),
            "euclidean_distance": round(euclidean_dist, 4),
            "are_identical_rankings": abs(cosine_sim - dot_product) < 0.001,
            "interpretation": self._interpret_similarity(cosine_sim),
        }

    def semantic_search(self, query: str, documents: list,
                        top_k: int = 5) -> list:
        """Perform semantic search using embeddings."""
        query_vec = self._l2_normalize(
            self._simulate_embedding(query, 1536))

        results = []
        for doc in documents:
            doc_vec = self._l2_normalize(
                self._simulate_embedding(doc["content"], 1536))
            similarity = sum(a * b for a, b in zip(query_vec, doc_vec))
            results.append({
                "content": doc["content"],
                "source": doc.get("source", ""),
                "similarity": round(similarity, 4),
            })

        # Sort by similarity descending
        results.sort(key=lambda x: x["similarity"], reverse=True)
        return results[:top_k]

    def _tokenize(self, text: str) -> list:
        """Simulate subword tokenization."""
        # In production: use tokenizer from embedding model
        words = text.lower().split()
        tokens = []
        for word in words:
            # Simulate subword splitting for long words
            if len(word) > 8:
                tokens.append(word[:len(word)//2])
                tokens.append(word[len(word)//2:])
            else:
                tokens.append(word)
        return tokens

    def _simulate_embedding(self, text: str,
                            dimensions: int = 1536) -> list:
        """Simulate embedding generation (deterministic from text)."""
        # In production: use actual embedding model
        # Here: generate deterministic pseudo-embedding from text hash
        import hashlib
        import struct

        result = []
        text_bytes = text.encode('utf-8')
        for i in range(dimensions):
            # Generate deterministic float from text + dimension index
            seed = hashlib.md5(
                text_bytes + str(i).encode()).digest()
            value = struct.unpack('f', seed[:4])[0]
            # Scale to reasonable range
            value = (value % 2.0) - 1.0
            result.append(round(value, 6))

        return result

    def _l2_normalize(self, vector: list) -> list:
        """L2 normalize a vector to unit length."""
        magnitude = math.sqrt(sum(x * x for x in vector))
        if magnitude == 0:
            return vector
        return [x / magnitude for x in vector]

    def _cosine_similarity(self, v1: list, v2: list) -> float:
        """Compute cosine similarity between two vectors."""
        dot = sum(a * b for a, b in zip(v1, v2))
        mag1 = math.sqrt(sum(a * a for a in v1))
        mag2 = math.sqrt(sum(b * b for b in v2))
        if mag1 == 0 or mag2 == 0:
            return 0.0
        return dot / (mag1 * mag2)

    def _interpret_similarity(self, sim: float) -> str:
        if sim > 0.9:
            return "Very similar meaning"
        elif sim > 0.7:
            return "Similar meaning"
        elif sim > 0.5:
            return "Related but different"
        elif sim > 0.3:
            return "Weakly related"
        elif sim > 0.1:
            return "Mostly unrelated"
        else:
            return "Unrelated"

What Are Embeddings Checklist

• [ ] Understand embeddings are dense vectors where proximity equals semantic similarity
• [ ] Embeddings convert text to numbers that capture meaning, not just keywords
• [ ] "The kitten slept" and "The cat napped" produce similar vectors
• [ ] "Quarterly earnings forecast" produces a very different vector
• [ ] Three pipeline stages: tokenization → transformer encoding → pooling
• [ ] Tokenization: split raw text into integer IDs (subword tokens)
• [ ] Transformer: converts token IDs to contextualized vectors via self-attention
• [ ] Pooling: collapse per-token vectors into one fixed-length vector
• [ ] L2 normalization: scale vector to unit length for faster similarity search
• [ ] Normalized vectors allow dot product instead of cosine (same ranking, faster)
• [ ] Sparse embeddings (BM25, TF-IDF): one dimension per word, mostly zeros
• [ ] Dense embeddings: 384-3072 continuous dimensions, all carry information
• [ ] Dense captures synonyms and semantic similarity that sparse misses
• [ ] Production RAG uses hybrid (dense + sparse) for best of both worlds
• [ ] Embeddings are trained with contrastive learning (similar pairs close, dissimilar far)
• [ ] Training uses Multiple Negatives Ranking Loss or InfoNCE loss
• [ ] Word2Vec (2013): 300 dimensions, static word vectors
• [ ] BERT (2018): 768 dimensions, contextualized representations
• [ ] 2026 models: 1536-4096 dimensions, instruction-tuned, 100+ languages
• [ ] Dimensionality tradeoff: more dimensions = finer distinctions but more storage
• [ ] 384d: 1.5KB/vector, good for prototyping and local deployment
• [ ] 768d: 3KB/vector, balanced for production
• [ ] 1536d: 6KB/vector, production default (text-embedding-3-small)
• [ ] 3072d: 12KB/vector, maximum distinction (text-embedding-3-large)
• [ ] 1M vectors at 1536d = 6GB storage
• [ ] Cosine similarity: measures angle between vectors (-1 to 1)
• [ ] Dot product: same as cosine for normalized vectors, but faster
• [ ] Euclidean distance: suffers from curse of dimensionality at 768+ dims
• [ ] Production default: cosine similarity (or dot product on normalized vectors)
• [ ] Embeddings power: semantic search, RAG, clustering, classification, dedup, recommendations
• [ ] Embedding models are distinct from generative LLMs — they produce vectors, not text
• [ ] Transformer encoder only (no decoder needed for embeddings)
• [ ] Self-attention: each token learns from all other tokens in the input
• [ ] "Bank" in "river bank" and "central bank" start identical, end different after transformer
• [ ] Choose embedding model based on use case
• [ ] Understand cosine vs dot product for search
• [ ] Use embeddings in RAG pipelines
• [ ] Build RAG from scratch with embeddings
• [ ] Add hybrid search (dense + sparse)
• [ ] Optimize chunk size for embedding quality
• [ ] Apply chunking strategies before embedding
• [ ] Add reranking after embedding-based retrieval
• [ ] Prevent hallucinations with grounded retrieval
• [ ] Evaluate with RAG metrics
• [ ] Test: similar texts produce high cosine similarity
• [ ] Test: dissimilar texts produce low cosine similarity
• [ ] Test: L2 normalization produces unit-length vectors
• [ ] Test: dot product and cosine give identical rankings on normalized vectors
• [ ] Test: semantic search finds documents with zero keyword overlap
• [ ] Document embedding model, dimensions, normalization, and similarity metric

FAQ

What are embeddings in AI?

Embeddings are dense vectors of floating-point numbers that encode the semantic meaning of text. AI/TLDR: "An embedding model is a machine that accepts a sentence and returns a list of numbers. That list — the embedding vector — is a precise location in a high-dimensional space of meaning. Feed in 'The kitten slept' and you get [0.12, -0.87, 0.41, ...] across 768 or 1,024 slots. Feed in 'The cat napped' and you get a very similar list, because the two sentences mean almost the same thing." ML Journey: "An embedding model converts text into a dense vector of floating-point numbers. Texts with similar meanings end up close together in that space, while texts with different meanings end up far apart. This geometric relationship is what makes embeddings useful: measuring the distance or angle between two vectors tells you how semantically similar the underlying texts are, without any explicit keyword matching." Explainx: "Embeddings are the single most important low-level primitive in applied AI. An embedding replaces sparse, meaningless one-hot encoding with a dense, learned vector — typically 768 to 3072 floating-point numbers — where the position encodes semantic meaning."

How do text embeddings work under the hood?

Text embeddings work through three stages: tokenization, transformer encoding, and pooling. AI/TLDR: "Every text embedding model processes input through three stages: (1) Tokenization — splits raw text into integer IDs. (2) Transformer layers — converts token IDs into static vectors via embedding matrix, then self-attention updates each token vector with context from all other tokens. Bank in river bank and bank in central bank start identical but end up different after the transformer. (3) Pooling — collapses the matrix of per-token vectors into one fixed-length vector. After pooling, L2 normalization scales the vector so its length equals exactly 1, allowing dot product instead of cosine for faster similarity search." Let's Data Science: "Dense embeddings compress meaning into 768 to 4,096 continuous dimensions learned from massive text corpora. A well-trained model places laptop freezing after update and system slow following patch close together because it learned from billions of examples that these phrases appear in similar contexts." Explainx: "The dominant architecture in 2026 is the transformer encoder — the E in BERT. Unlike a full language model, an embedding model needs only the encoder half because its job is to read text and produce a representation, not generate new text."

What is the difference between sparse and dense embeddings?

Sparse embeddings have one dimension per vocabulary word (mostly zeros), while dense embeddings compress meaning into 384-3072 continuous dimensions where every value carries information. Let's Data Science: "Sparse representations like Bag-of-Words and TF-IDF create vectors with one dimension per vocabulary word, resulting in mostly zeros. Dense embeddings compress meaning into 768 to 4,096 continuous dimensions where every value carries information. Dense representations capture synonyms and semantic similarity that sparse methods miss entirely, which is why modern search and retrieval systems have largely adopted them." Explainx: "The earliest approach was one-hot encoding: create a vector with one dimension per word in the vocabulary, and mark a single dimension as 1 while all others are 0. An embedding replaces this sparse, meaningless representation with a dense, learned vector." AI/TLDR: "The embedding space has hundreds of dimensions. The only question that matters is: how close are two addresses? Close means similar in meaning; far means unrelated." Sparse: BM25, TF-IDF. Dense: text-embedding-3-small, BGE, MiniLM. Production RAG uses hybrid (dense + sparse).

How are embeddings trained?

Embeddings are trained using contrastive learning on large text corpora. ML Journey: "Most modern embedding models are transformer-based encoders trained on large text corpora using contrastive learning objectives. The training process presents the model with pairs of texts — some semantically similar (a question and its correct answer, two paraphrases) and some dissimilar — and trains the model to produce vectors that are close for similar pairs and distant for dissimilar ones. This is often done using Multiple Negatives Ranking Loss or InfoNCE loss. The result is a model whose output vectors capture semantic relationships with remarkable fidelity. Synonyms map to nearby points. Sentences about the same topic cluster together. Questions and their answers end up in similar regions of the space, even though they are phrased very differently." ACL: "Embedding techniques initially focused on words (Word2Vec, GloVe) then moved to contextualized representations (ELMo, BERT). Transformers have proven very effective for encoding contextualized knowledge thanks to their self-attention mechanism." Let's Data Science: "The field has moved from 300-dimensional Word2Vec vectors in 2013 to 4,096-dimensional, instruction-tuned models in 2026 that handle 100+ languages."

What dimensionality should embeddings have?

Embedding dimensionality ranges from 384 to 3072, with tradeoffs between richness and cost. ML Journey: "The dimensionality of the output vector — typically 384, 768, 1536, or 3072 dimensions depending on the model — determines the richness of the representation: more dimensions can encode finer distinctions, at the cost of more storage and computation." AI/TLDR: "More dimensions allow the model to capture finer distinctions between meanings, but every dimension adds storage cost and slows similarity search." Explainx: "Each dimension is a float32 (4 bytes), so a 1536-dimensional embedding takes 6KB per vector. One million vectors = 6GB of storage." Common models: MiniLM (384d, 1.5KB/vector), BGE-base (768d, 3KB), text-embedding-3-small (1536d, 6KB), text-embedding-3-large (3072d, 12KB). Choose based on: (1) Storage budget — more dimensions = more storage. (2) Search speed — more dimensions = slower similarity search. (3) Distinction needed — complex domains need more dimensions. (4) Cost — API pricing often scales with dimensions. Production default: 1536d (text-embedding-3-small) for best balance.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →