Generate Embeddings in Python: OpenAI, Sentence-Transformers, BGE, and Cohere

By Fadi Updated Aug 18, 2025 11 min read Vector Databases

TL;DR — Generate embeddings in Python with OpenAI API, sentence-transformers, BGE, Cohere, and HuggingFace TEI. LangChain: "OpenAI text-embedding-3-* supports 8191 tokens and MRL dimension control." HuggingFace: "Select pre-trained model, embed documents, compare queries to embedded dataset." LlamaIndex: "Easiest local model: HuggingFaceEmbedding. Default batch size 10 — may be too small for many documents." OneUptime: "Text embeddings transform words into dense numerical vectors capturing semantic meaning. In RAG, embeddings are the foundation for finding relevant context." HuggingFace TEI: "Blazing fast text embedding service with Docker GPU inference." Learn more with embeddings, embedding models, dimensions, and semantic search.

OneUptime explains the foundation: "Text embeddings transform words and sentences into dense numerical vectors that capture semantic meaning. In RAG (Retrieval Augmented Generation) systems, embeddings are the foundation that enables finding relevant context to feed into your LLM."

HuggingFace provides the workflow: "The first step is selecting an existing pre-trained model for creating embeddings. We can choose, embed documents using the Inference API, upload the embedded questions to the Hub for free hosting, and compare a customer's query to the embedded dataset to identify which is the most similar."

Embedding Generation Architecture

flowchart TD subgraph Sources["Embedding Sources"] OpenAI["OpenAI API
text-embedding-3-small/large
$0.02-0.13/1M tokens
8191 token limit"] ST["sentence-transformers
all-MiniLM-L6-v2 (384d)
BGE-large-en-v1.5 (1024d)
Free, local"] Cohere["Cohere API
embed-english-v4
1024 dims
Multilingual"] TEI["HuggingFace TEI
Docker GPU server
Production inference
Self-hosted"] end subgraph Pipeline["Embedding Pipeline"] Text["Input Text
chunked documents"] Batch["Batch Embed
100-1000 per batch"] MRL["MRL Dimension Control
truncate to 384/512/768"] Vectors["Embedding Vectors
float32 or halfvec"] Store["Store in pgvector
or vector database"] end subgraph Production["Production Concerns"] Rate["Rate Limiting
delays between batches"] Async["Async Embedding
parallel API calls"] Cache["Embedding Cache
avoid re-embedding"] Monitor["Monitor
latency, cost, errors"] end OpenAI --> Batch ST --> Batch Cohere --> Batch TEI --> Batch Text --> Batch --> MRL --> Vectors --> Store Rate --> Batch Async --> Batch Cache --> Vectors Monitor --> Pipeline

Embedding Provider Comparison

Provider	Model	Dimensions	Cost	Max Tokens	Deployment	Best For
OpenAI	text-embedding-3-small	1536 (MRL)	$0.02/1M	8191	API	Production, general purpose
OpenAI	text-embedding-3-large	3072 (MRL)	$0.13/1M	8191	API	High precision, technical
sentence-transformers	all-MiniLM-L6-v2	384	Free	256	Local	Prototyping, edge
sentence-transformers	BGE-large-en-v1.5	1024	Free	512	Local	Best open-source quality
sentence-transformers	BGE-M3	1024	Free	8192	Local	Long context, multilingual
Cohere	embed-english-v4	1024	Paid	512	API	Multilingual, enterprise
HuggingFace TEI	Any HF model	Varies	Free (self-hosted)	Varies	Docker GPU	Production, self-hosted
Voyage AI	voyage-4-large	1024	Paid	32000	API	State-of-art retrieval
Google	text-embedding-004	768	Free tier	2048	API	Google Cloud users

Implementation

import os
import time
import asyncio
from dataclasses import dataclass
from typing import Optional
from enum import Enum

class EmbeddingProvider(Enum):
    OPENAI_SMALL = "openai_small"
    OPENAI_LARGE = "openai_large"
    MINILM = "minilm"
    BGE_LARGE = "bge_large"
    BGE_M3 = "bge_m3"
    COHERE = "cohere"
    TEI = "tei"

@dataclass
class EmbeddingGenerator:
    """Generate embeddings with multiple providers in Python."""

    def __init__(self, provider: EmbeddingProvider =
                 EmbeddingProvider.OPENAI_SMALL,
                 dimensions: int = None):
        self.provider = provider
        self.dimensions = dimensions
        self._client = None
        self._model = None

    def _ensure_client(self):
        """Lazy initialization of embedding client."""
        if self._client is not None:
            return

        if self.provider in (EmbeddingProvider.OPENAI_SMALL,
                             EmbeddingProvider.OPENAI_LARGE):
            from openai import OpenAI
            self._client = OpenAI()
        elif self.provider == EmbeddingProvider.COHERE:
            import cohere
            self._client = cohere.Client()
        elif self.provider in (EmbeddingProvider.MINILM,
                               EmbeddingProvider.BGE_LARGE,
                               EmbeddingProvider.BGE_M3):
            from sentence_transformers import SentenceTransformer
            model_map = {
                EmbeddingProvider.MINILM: 'all-MiniLM-L6-v2',
                EmbeddingProvider.BGE_LARGE: 'BAAI/bge-large-en-v1.5',
                EmbeddingProvider.BGE_M3: 'BAAI/bge-m3',
            }
            self._model = SentenceTransformer(
                model_map[self.provider])

    def embed_single(self, text: str) -> list:
        """Embed a single text."""
        self._ensure_client()

        if self.provider in (EmbeddingProvider.OPENAI_SMALL,
                             EmbeddingProvider.OPENAI_LARGE):
            model_name = (
                "text-embedding-3-small"
                if self.provider == EmbeddingProvider.OPENAI_SMALL
                else "text-embedding-3-large"
            )
            kwargs = {"input": text, "model": model_name}
            if self.dimensions:
                kwargs["dimensions"] = self.dimensions
            response = self._client.embeddings.create(**kwargs)
            return response.data[0].embedding

        elif self.provider == EmbeddingProvider.COHERE:
            response = self._client.embed(
                texts=[text],
                model='embed-english-v4',
                input_type='search_document'
            )
            return response.embeddings[0]

        else:
            embedding = self._model.encode(text)
            if self.dimensions:
                embedding = embedding[:self.dimensions]
            return embedding.tolist()

    def embed_batch(self, texts: list,
                    batch_size: int = 100) -> list:
        """Embed a batch of texts efficiently."""
        self._ensure_client()
        all_embeddings = []

        for i in range(0, len(texts), batch_size):
            batch = texts[i:i + batch_size]

            if self.provider in (EmbeddingProvider.OPENAI_SMALL,
                                 EmbeddingProvider.OPENAI_LARGE):
                model_name = (
                    "text-embedding-3-small"
                    if self.provider == EmbeddingProvider.OPENAI_SMALL
                    else "text-embedding-3-large"
                )
                kwargs = {"input": batch, "model": model_name}
                if self.dimensions:
                    kwargs["dimensions"] = self.dimensions
                response = self._client.embeddings.create(**kwargs)
                batch_embeddings = [
                    d.embedding for d in response.data
                ]
                time.sleep(0.1)  # rate limit

            elif self.provider == EmbeddingProvider.COHERE:
                response = self._client.embed(
                    texts=batch,
                    model='embed-english-v4',
                    input_type='search_document'
                )
                batch_embeddings = response.embeddings
                time.sleep(0.1)

            else:
                batch_embeddings = (
                    self._model.encode(batch).tolist()
                )

            all_embeddings.extend(batch_embeddings)

        return all_embeddings

    async def embed_batch_async(self, texts: list,
                                batch_size: int = 100) -> list:
        """Async batch embedding for parallel API calls."""
        import aiohttp

        async def embed_chunk(chunk):
            # For OpenAI async client
            from openai import AsyncOpenAI
            client = AsyncOpenAI()
            model_name = (
                "text-embedding-3-small"
                if self.provider == EmbeddingProvider.OPENAI_SMALL
                else "text-embedding-3-large"
            )
            kwargs = {"input": chunk, "model": model_name}
            if self.dimensions:
                kwargs["dimensions"] = self.dimensions
            response = await client.embeddings.create(**kwargs)
            return [d.embedding for d in response.data]

        tasks = [
            embed_chunk(texts[i:i + batch_size])
            for i in range(0, len(texts), batch_size)
        ]
        results = await asyncio.gather(*tasks)

        all_embeddings = []
        for batch_result in results:
            all_embeddings.extend(batch_result)
        return all_embeddings

    def get_tei_example(self) -> str:
        """HuggingFace TEI Docker setup example."""
        return (
            "# Start TEI server with GPU\n"
            "docker run --gpus all -p 8080:80 \\\n"
            "  -v $PWD/data:/data \\\n"
            "  ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 \\\n"
            "  --model-id BAAI/bge-large-en-v1.5\n"
            "\n"
            "# Embed via API\n"
            "import requests\n"
            "\n"
            "response = requests.post(\n"
            "    'http://localhost:8080/embed',\n"
            "    json={'inputs': ['text1', 'text2']}\n"
            ")\n"
            "embeddings = response.json()"
        )

    def get_mrl_example(self) -> str:
        """Matryoshka dimension control example."""
        return (
            "# OpenAI with MRL dimension control\n"
            "from openai import OpenAI\n"
            "client = OpenAI()\n"
            "\n"
            "# Request 512 dims instead of default 1536\n"
            "response = client.embeddings.create(\n"
            "    input='text to embed',\n"
            "    model='text-embedding-3-small',\n"
            "    dimensions=512  # MRL truncation\n"
            ")\n"
            "embedding = response.data[0].embedding\n"
            "print(len(embedding))  # 512\n"
            "\n"
            "# sentence-transformers with manual truncation\n"
            "from sentence_transformers import SentenceTransformer\n"
            "model = SentenceTransformer('nomic-ai/nomic-embed-text-v1.5')\n"
            "embedding = model.encode('text')\n"
            "\n"
            "# Truncate to 384 dims (98.1% quality retained)\n"
            "truncated = embedding[:384]"
        )

    def get_cost_estimate(self, token_count: int) -> dict:
        """Estimate embedding generation cost."""
        costs = {
            EmbeddingProvider.OPENAI_SMALL: 0.02,
            EmbeddingProvider.OPENAI_LARGE: 0.13,
            EmbeddingProvider.MINILM: 0,
            EmbeddingProvider.BGE_LARGE: 0,
            EmbeddingProvider.BGE_M3: 0,
            EmbeddingProvider.COHERE: 0.10,
        }

        rate = costs.get(self.provider, 0)
        total_cost = (token_count / 1_000_000) * rate

        return {
            "provider": self.provider.value,
            "tokens": token_count,
            "rate_per_1m": f"${rate}",
            "total_cost": f"${total_cost:.2f}",
            "notes": (
                "Free models (sentence-transformers, BGE) "
                "require local compute but no API cost. "
                "TEI requires GPU server but no per-token cost."
            ),
        }

Generate Embeddings Checklist

[ ] Choose embedding provider: OpenAI for production, sentence-transformers for local/free
[ ] OpenAI text-embedding-3-small: 1536 dims, $0.02/1M tokens, 8191 token limit
[ ] OpenAI text-embedding-3-large: 3072 dims, $0.13/1M tokens, 8191 token limit
[ ] sentence-transformers all-MiniLM-L6-v2: 384 dims, free, 256 token limit
[ ] sentence-transformers BGE-large-en-v1.5: 1024 dims, free, 512 token limit
[ ] sentence-transformers BGE-M3: 1024 dims, free, 8192 token limit, multilingual
[ ] Cohere embed-english-v4: 1024 dims, paid, multilingual
[ ] HuggingFace TEI: Docker GPU server for production self-hosted inference
[ ] Voyage AI voyage-4-large: 1024 dims, paid, 32000 token limit, state-of-art retrieval
[ ] Install: pip install openai for OpenAI, pip install sentence-transformers for local
[ ] OpenAI: from openai import OpenAI; client = OpenAI(); response = client.embeddings.create(input='text', model='text-embedding-3-small')
[ ] sentence-transformers: from sentence_transformers import SentenceTransformer; model = SentenceTransformer('all-MiniLM-L6-v2'); embeddings = model.encode(['text1', 'text2'])
[ ] Batch embedding: send multiple texts in single API call or model inference
[ ] OpenAI batch: up to 2048 texts per request
[ ] sentence-transformers: model.encode(texts) handles batching automatically
[ ] Default LlamaIndex batch size is 10 — may be too small for many documents
[ ] Rate limiting: add delays between batches to avoid API limits
[ ] Async embedding: use AsyncOpenAI with asyncio.gather for parallel calls
[ ] HuggingFace TEI: docker run --gpus all -p 8080:80 ghcr.io/huggingface/text-embeddings-inference --model-id BAAI/bge-large-en-v1.5
[ ] TEI: POST to /embed endpoint with JSON inputs
[ ] MRL dimension control: OpenAI dimensions parameter to request fewer dims
[ ] MRL: client.embeddings.create(input='text', model='text-embedding-3-small', dimensions=512)
[ ] MRL: manual truncation for sentence-transformers: embedding[:512]
[ ] MRL models: OpenAI 3-small, OpenAI 3-large, nomic-embed-text, Cohere v4, Voyage-4
[ ] MRL truncation quality: 768 (100%), 512 (99.2%), 384 (98.1%), 256 (96.5%)
[ ] Always use same dimensions for indexing and querying — mismatched dims produce garbage
[ ] Use same embedding model for indexing and querying
[ ] Text embeddings transform words into dense numerical vectors capturing semantic meaning
[ ] In RAG, embeddings are the foundation for finding relevant context to feed into LLM
[ ] Select pre-trained model based on: language, domain, token limit, cost, quality
[ ] For long chunks: prefer long-context models (BGE-M3 8192, OpenAI 8191, Voyage 32000)
[ ] For multilingual: pick model trained on your languages (BGE-M3, Cohere, Voyage)
[ ] For cost-sensitive: use free local models (sentence-transformers, BGE)
[ ] For production scale: use TEI Docker server with GPU
[ ] Cache embeddings to avoid re-embedding unchanged documents
[ ] Monitor: embedding latency, API cost, error rates, token usage
[ ] HuggingFace Inference API: client.feature_extraction('text', model='BAAI/bge-large-en-v1.5')
[ ] Upload embedded dataset to HuggingFace Hub for free hosting
[ ] Compare query to embedded dataset to identify most similar documents
[ ] Cost estimate: 10M tokens with OpenAI 3-small = $0.20, with 3-large = $1.30
[ ] Free models require local compute but no API cost
[ ] TEI requires GPU server but no per-token cost
[ ] Read embeddings for fundamentals
[ ] Read embedding models for model selection
[ ] Read embedding dimensions for dimension optimization
[ ] Read semantic search for search implementation
[ ] Read pgvector tutorial for storage
[ ] Read what is RAG for RAG architecture
[ ] Test: OpenAI embedding returns correct dimension count
[ ] Test: sentence-transformers produces embeddings locally without API
[ ] Test: batch embedding handles 100+ texts correctly
[ ] Test: MRL truncation produces shorter embeddings with minimal quality loss
[ ] Test: async embedding parallelizes API calls correctly
[ ] Test: same model and dimensions used for indexing and querying
[ ] Document provider choice, dimensions, batch size, and cost estimate

FAQ

How do you generate embeddings with OpenAI in Python?

Use the OpenAI Python SDK to call the embeddings API with text-embedding-3-small or text-embedding-3-large. LangChain: "OpenAI text-embedding-3-* supports 8191 tokens. If your chunks are long, prefer long-context models." OneUptime: "Text embeddings transform words and sentences into dense numerical vectors that capture semantic meaning. In RAG systems, embeddings are the foundation that enables finding relevant context." Code: from openai import OpenAI; client = OpenAI(); response = client.embeddings.create(input='text', model='text-embedding-3-small'); embedding = response.data[0].embedding. For batch: client.embeddings.create(input=['text1', 'text2'], model='text-embedding-3-small'). Cost: $0.02/1M tokens for 3-small, $0.13/1M for 3-large. Supports MRL dimension control with dimensions parameter.

How do you generate embeddings locally with sentence-transformers?

Install sentence-transformers and load a model like all-MiniLM-L6-v2 or BGE-large-en-v1.5 for free local embeddings. HuggingFace: "Select an existing pre-trained model for creating embeddings. Compare a query to the embedded dataset to identify the most similar." LlamaIndex: "The easiest way to use a local model is by using HuggingFaceEmbedding from llama-index-embeddings-huggingface." Code: from sentence_transformers import SentenceTransformer; model = SentenceTransformer('all-MiniLM-L6-v2'); embeddings = model.encode(['text1', 'text2']). For BGE: model = SentenceTransformer('BAAI/bge-large-en-v1.5'). Free, local, no API calls. all-MiniLM-L6-v2: 384 dims, fast. BGE-large-en-v1.5: 1024 dims, high quality.

How do you batch generate embeddings efficiently in Python?

Batch embedding by sending multiple texts in a single API call or model inference, with batches of 100-1000 for API and 32-256 for local models. LlamaIndex: "By default, embeddings requests are sent to OpenAI in batches of 10. For some users, this may incur a rate limit. For other users embedding many documents, this batch size may be too small." HuggingFace TEI: "A blazing fast text embedding service. Docker run with GPU for production inference." Batch strategies: (1) OpenAI API: batch input list of texts, up to 2048 per request. (2) sentence-transformers: model.encode(texts) handles batching automatically. (3) HuggingFace TEI: Docker-based server for production batch inference. (4) Async: use asyncio with aiohttp for parallel API calls. (5) Rate limiting: add delays between batches to avoid API limits.

How do you generate embeddings with HuggingFace Inference API?

Use the HuggingFace Inference API or Text Embeddings Inference (TEI) Docker server for hosted or self-hosted embedding generation. HuggingFace: "Embed documents using the Inference API. Upload embedded questions to the Hub for free hosting. Compare queries to embedded dataset." HuggingFace TEI: "Docker run --gpus all -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id BAAI/bge-reranker-large." Code: from huggingface_hub import InferenceClient; client = InferenceClient(); embeddings = client.feature_extraction('text', model='BAAI/bge-large-en-v1.5'). For self-hosted TEI: docker run with model-id, then POST to /embed endpoint. TEI is optimized for production with GPU acceleration and batching.

How do you control embedding dimensions with MRL in Python?

Use the dimensions parameter in OpenAI API or truncate embeddings manually for Matryoshka Representation Learning models. LangChain: "OpenAI text-embedding-3-* supports MRL dimension control." OpenAI: response = client.embeddings.create(input='text', model='text-embedding-3-small', dimensions=512). This returns 512-dim embeddings instead of default 1536. For sentence-transformers with MRL models: truncate the embedding vector to desired dimensions: embedding = model.encode('text')[:512]. MRL models: OpenAI 3-small (1536 to 256+), OpenAI 3-large (3072 to 256+), nomic-embed-text (768 to 256+), Cohere v4 (1024 to 256+). Truncation quality: 768 (100%), 512 (99.2%), 384 (98.1%), 256 (96.5%). Always use the same dimensions for indexing and querying.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →