pgvector, Pinecone, Weaviate, similarity search, HNSW, and indexing. 14 articles, written by the engineers building AI App Lab.
Semantic search with pgvector and Python: embed text with sentence-transformers or OpenAI, store in PostgreSQL, query with cosine distance. FastAPI search endpoint, batch indexing, filtered search, hybrid BM25+vector, and production checklist for building a semantic search engine in 2026.
HNSW explained: multi-layer graph for approximate nearest neighbor search. Introduced by Malkov and Yashunin in 2016. Greedy routing from coarse upper layers to fine lower layers. Parameters m, ef_construction, ef_search. O(log n) search complexity. 95%+ recall. Production checklist for 2026.
pgvector performance tuning: 5 parameters drive 90% of performance. maintenance_work_mem 8-16GB for HNSW builds, ef_search 10-200 per query, shared_buffers 25-40% RAM, EXPLAIN ANALYZE for seq scan detection, iterative scans for filtering, halfvec for storage, and production checklist for 2026.
Embedding dimensions compared: 384 for prototyping, 768-1024 as production default, 1536 for OpenAI, 3072 for max quality. MTEB scores, storage costs, latency, Matryoshka truncation, pgvector halfvec, and dimension selection guide with benchmarks and production checklist for 2026.
Vector databases store high-dimensional embeddings and answer nearest-neighbor queries with HNSW, IVF, and product quantization. Compare Pinecone, Weaviate, Qdrant, Milvus, pgvector, and Chroma. ANN index types, filtering, quantization, and when to use each database for production RAG.
PostgreSQL hybrid search: combine BM25 keyword search with pgvector semantic search using Reciprocal Rank Fusion (RRF). ParadeDB BM25, tsvector ts_rank_cd, cosine similarity, RRF formula, Python HybridSearch implementation, and production checklist for 2026.
pgvector vs Weaviate vs Qdrant: pgvector for Postgres simplicity, Weaviate for hybrid BM25+vector, Qdrant for fastest filtered search in Rust. Compare performance, filtering, hybrid search, multi-vector, quantization, scale, and deployment. Decision matrix with 8 criteria and production checklist.
Vector database incremental updates: CDC pipelines for embedding versioning, HNSW absorbs inserts without rebuild, IVFFlat needs REINDEX after 30% new rows. Compare upsert strategies, embedding versioning, stale vector detection, reindexing pipeline, and production checklist for 2026.
Generate embeddings in Python with OpenAI API, sentence-transformers, BGE-M3, Cohere, and HuggingFace Inference. Batch embedding, local vs hosted models, dimension control with MRL, async embedding, and production checklist for building embedding pipelines in 2026.
pgvector vs Pinecone: pgvector matches Pinecone at 1M scale with HNSW, queries in 5-20ms at 95%+ recall. Pinecone offers managed serverless with zero ops. Compare performance, cost, filtering, scale, hybrid search, and vendor lock-in. Decision matrix with 7 criteria and production checklist.
pgvector tutorial: install, create extension, store embeddings, build HNSW index, and query with cosine distance. Python examples with psycopg2 and asyncpg. Distance operators, index tuning, filtered search, upserts, and production checklist for PostgreSQL vector search in 2026.
HNSW vs IVFFlat in pgvector: HNSW is the default for most workloads with 95%+ recall and no rebuild needed. IVFFlat uses 2-5x less memory for large static datasets. Compare parameters (m, ef_construction, ef_search vs lists, probes), recall, latency, memory, and when to choose each index in 2026.
pgvector Docker setup: use pgvector/pgvector:0.8.0-pg16 image, docker-compose with persistent volumes, init.sql for auto-extension, production config (shared_buffers, maintenance_work_mem), HNSW index tuning, resource limits, and production checklist for 2026.
Build a production vector search API with FastAPI and pgvector: asyncpg connection pooling, upsert endpoints, cosine similarity search, hybrid search with reranking, streaming responses, Docker deployment, PgBouncer, read replicas, and production checklist for 2026.