A RAG assistant must search ten million document chunks in under one hundred milliseconds per query. Storing embeddings in a flat file and computing cosine similarity against every vector does not scale. Vector database explained for builders: specialized storage and indexing for dense vectors plus metadata, optimized for approximate nearest neighbor (ANN) search. Enterprise AI chatbot stacks depend on vector stores for retrieval; AI video pipelines embed scene descriptions and transcript segments into the same indexes for multimodal search.
Role of Vector Databases in RAG
Vector databases hold embedding vectors with identifiers and metadata, returning the top-k nearest neighbors to a query embedding for injection into LLM prompts. RAG ingestion embeds chunks, upserts vectors with source URLs and ACL tags, and queries at answer time. The vector database is not the LLM; it is the retrieval tier. Quality still depends on chunking, embedding model choice, and hybrid keyword search. Without filtered metadata, semantic search may return another tenant's documents or outdated policy versions.
Some teams start with pgvector inside PostgreSQL when vector count stays below a few million and QPS is moderate. Dedicated vector databases add ANN algorithms, sharding, and replication tuned for billion-scale similarity search. The boundary shifts as products grow; plan migration triggers (latency SLO breach, index rebuild time) before you need them.
ANN Index Types: HNSW, IVF, and Flat
Approximate indexes trade perfect recall for speed: HNSW (graph-based), IVF (cluster-based), and flat (exact brute force) are the common choices. HNSW (Hierarchical Navigable Small World) builds layered graphs for fast greedy search; strong default for many workloads with tunable efConstruction and efSearch parameters. IVF partitions vectors into clusters and searches only nearby clusters, reducing memory at some recall cost. Flat indexes compute exact distances; viable only for small collections or as a quality baseline during evaluation.
| Index type | Strength | Tuning note |
|---|---|---|
| HNSW | Low latency, high recall | Higher memory; rebuild on bulk ingest |
| IVF | Memory efficient at scale | Choose nlist/nprobe for recall |
| Flat | Exact nearest neighbors | Only for small N or benchmarks |
Recall-latency tradeoffs
ANN parameters control how many graph nodes or clusters are explored. Higher efSearch improves recall but increases query latency. Measure recall@k against a flat index on a sample of queries when tuning. Production SLOs often target p95 latency under fifty to one hundred milliseconds with recall above 0.95 relative to exact search on your embedding model.
Metadata Filters and Hybrid Search
Metadata filters restrict ANN search to subsets (tenant, product, date, language) before or during similarity scoring; hybrid search combines dense vectors with sparse keyword indexes. Pre-filtering enforces multi-tenancy: only vectors tagged with the user's org_id are eligible. Post-filtering retrieves extra neighbors then drops mismatches, which can empty results if filters are tight. Hybrid BM25 plus vector fusion (RRF or weighted sum) catches SKU codes and statutes that pure embeddings miss.
| Pattern | Use case | Risk |
|---|---|---|
| Pre-filter by ACL | Enterprise RAG | Overly narrow filters reduce recall |
| Hybrid BM25 + vector | Support tickets, catalogs | Fusion weights need tuning |
| Time decay boost | News, changelogs | May hide evergreen docs |
Operations, Scaling, and Reliability
Operating vector databases means planning shard strategy, replication, backup of vectors and metadata, index rebuild windows, and monitoring query latency and recall drift. Write-heavy ingestion can lag behind ANN index updates; batch upserts during maintenance windows or use streaming index updates if supported. Replicating read replicas spreads query load. Snapshot backups before embedding model migrations. Alert on insert failure rates and orphan metadata without vectors.
Capacity planning
Memory scales with vector count times dimension times four bytes (float32) plus index overhead (HNSW often 1.5x to 2x raw vector size). A million 1536-dim vectors is roughly six GB raw plus index. Disk-backed options trade latency for cost. Quantization (scalar or product) compresses vectors with small recall loss when configured carefully.
Build vs Buy Vector Storage
Buy managed vector databases (Pinecone, Weaviate Cloud, Zilliz, etc.) for faster time to market; build on pgvector, OpenSearch k-NN, or Milvus self-hosted when you need data control or unified ops. Managed services handle sharding, upgrades, and SLAs. Self-hosted fits strict data residency, existing Kubernetes skills, or tight coupling with PostgreSQL transactions. Search plugins in Elasticsearch or OpenSearch suit teams already running hybrid search clusters.
| Option | Best when | Watch out for |
|---|---|---|
| Managed vector DB | Small team, fast launch | Egress and storage pricing |
| pgvector | Under ~5M vectors, SQL joins | ANN tuning less mature than specialists |
| OpenSearch / Elasticsearch | Existing search team | Vector recall tuning learning curve |
Vector Database vs Traditional Database
Relational databases excel at transactional CRUD and joins; vector databases excel at similarity search over millions of embeddings with ANN indexes. You can store vectors in PostgreSQL blobs without an ANN index, but query time grows linearly. Use the right tool per access pattern: metadata and permissions in SQL, similarity in a vector index, synchronized by document ID. Some products unify both in one engine; others maintain dual writes with event pipelines.
Ingestion Patterns and Idempotency
Vector ingestion pipelines must be idempotent: the same document version upserts to the same vector ID so re-runs do not duplicate neighbors in search results. Use content hashes or source system version IDs as primary keys. Stream events from Kafka or webhooks into embed workers with dead-letter queues for failed chunks. Partial failures leave indexes inconsistent unless you track per-document ingest status in a side table. Full reindex jobs run on blue-green collections: build new index, flip read alias, delete old collection after smoke tests pass.
Batch vs streaming upsert
Batch jobs suit nightly wiki syncs; streaming suits live chat transcript indexing. ANN indexes may lag seconds to minutes behind writes depending on engine; document that lag in SLAs so support knows search may miss the last five minutes of edits. Throttle burst ingest to avoid starving query nodes.
Observability for Vector Search
Instrument vector search with query latency histograms, recall proxies (click-through on top result), filter cardinality, and empty-result rate. Log query embedding model version alongside collection name. When users thumbs-down RAG answers, store retrieved vector IDs for replay. Compare ANN recall weekly against a sampled flat search on staging. Alert when p95 latency doubles or empty-result rate spikes after deploys.
| Signal | Healthy range | Investigate if |
|---|---|---|
| p95 query latency | Under 100 ms | Sustained 2x baseline |
| Empty results | Under 2% | Jumps after filter change |
| Ingest lag | Within SLA window | Missed two sync cycles |
Security and Multi-Tenancy
Enforce tenant isolation with metadata filters on every query, separate collections per tenant for strict isolation, or encryption at rest with access-controlled API keys. Never rely on application-side filtering alone after retrieving global top-k. Audit logs should record collection, filter predicates, and returned IDs without logging full document text in insecure sinks. Rotate API keys per environment.
Frequently Asked Questions
Do I always need a dedicated vector database?
No; pgvector or embedded stores suffice for prototypes and moderate scale. Adopt a dedicated vector database when p95 latency, recall, or shard complexity exceed what your current stack handles comfortably.
How are updates and deletes handled?
Upserts replace vectors by ID; deletes must remove both vector and metadata to prevent ghost retrieval. Some indexes require periodic compaction after heavy deletes. Test incremental sync from source systems.
Can one vector database store text and image embeddings?
Yes, if dimensions and models align or you use separate collections per modality with a fusion layer. Multimodal models may emit joint embeddings; mismatched models need separate indexes and merged ranking.
How do I migrate between vector databases?
Export vectors with IDs and metadata, re-import into the target, rebuild indexes, and run recall benchmarks before cutover. Re-embedding from source text is safer than binary transfer when dimension or model changes.
What drives vector database cost?
Storage (vector count times dimension), query QPS, replica count, and managed service tier. Quantization and dimension reduction lower storage. Batch queries amortize overhead for offline jobs.
Disaster Recovery and Index Rebuild Playbooks
Vector indexes are rebuildable from source text plus embedding jobs, but rebuild time at billion scale can exceed RTO if not rehearsed. Keep source documents in durable object storage independent of the vector tier. Document step-by-step rebuild: export metadata, re-embed with pinned model version, load into fresh collection, validate recall sample, swap alias. Test quarterly on staging with production-scale subset. Backup strategies vary: some engines support snapshot export; others require full re-embed from corpus. Finance should budget re-embed cost when embedding APIs price per token.
When a Search Plugin Beats a Dedicated Vector Database
If your team already operates Elasticsearch with hybrid kNN and BM25, adding a vector field may beat introducing a second database with dual-write complexity. Unified search simplifies ACL, highlighting, and ops runbooks. Dedicated vector DBs win when ANN performance, filtering expressiveness, or managed scaling exceed plugin limits. Prototype both on your QPS and recall targets before architectural commitment. Migration cost from plugin to specialist (or reverse) is non-trivial; choose with eighteen-month horizon in mind.
Collection Naming and Environment Strategy
Use separate collections or namespaces per environment (dev, staging, prod) and per major embedding
model version to prevent accidental cross-contamination. Naming conventions like
prod_wiki_v3_e5_large make runbooks obvious during incidents. Never point staging apps at
production vector collections for convenience; test data pollutes analytics and may leak in demos. Automate
TTL on dev collections to control cost. Document which collection alias production reads follow during
blue-green index migrations.
Latency SLO Examples by Use Case
Interactive chat RAG often targets p95 under eighty milliseconds for retrieval alone; batch analytics may tolerate seconds per query. Video scene search during editing sessions needs sub-second response across million-frame indexes. Customer-facing site search marketing "instant AI" cannot hide two-second vector round trips without hurting conversion. Define SLOs per surface and size collections accordingly: shard hot tenants, replicate read-heavy collections, and pre-warm indexes after deploy. Load test with query distributions matching production (short head queries plus long tail paraphrases), not uniform random vectors.
RFP Questions for Vector Vendors
Ask vendors about maximum vectors per collection, filter expressiveness, hybrid search maturity, export portability, multi-region replication, and pricing at your projected eighteen-month scale. Request reference architectures for your embedding dimension and QPS. Run a load test during POC, not slide-deck promises. Clarify whether professional services are required for production HA or included in enterprise tier. Insist on a written recall benchmark using your embedding model before signing multi-year contracts.
Conclusion
Vector databases power semantic retrieval in RAG and multimodal search by indexing embeddings with ANN structures like HNSW and IVF, metadata filters, and optional hybrid keyword fusion. Choose build vs buy based on scale, ops skills, and compliance. Tune recall against latency, operate with backups and tenant isolation, and treat the vector tier as critical infrastructure alongside your embedding and generation stacks.
