Home / Insights / Generative AI and LLMs
Generative AI and LLMs
Choosing a Vector Database for a Permission-Aware RAG Product
Most vector database comparisons focus on speed. In a business RAG product the harder questions are whether retrieval respects user permissions, how tenants are isolated and what it costs to run. This guide gives selection criteria, a scorecard and a benchmark protocol to test candidates on your own data.
TopicsGenerative AI and LLMsAI and Machine LearningArchitecture and Modernization
Key takeaways
- In a business RAG product, retrieval must respect who may see each document. Filtering after retrieval can return too few results or leak information through answers.
- Choose between a vector index inside a database you already run and a dedicated vector database based on scale, filtering needs, operations and team skills, not on headline benchmarks.
- Decide your multi-tenancy model early: shared index with tenant filters, separate namespaces or collections, or separate deployments.
- Evaluate with your own documents, queries and permission rules, measuring recall with filters, latency and cost.
- Retrieval quality depends more on chunking, metadata and hybrid search than on the database alone.
On this page
Retrieval-augmented generation (RAG) gives a language model access to your documents at question time. The vector database stores numerical representations of document chunks and finds those most similar to a question. Many comparisons rank vector databases by queries per second on public datasets. Those numbers rarely decide a business product. More often the decision turns on three questions: can retrieval respect each user’s permissions, how are customers isolated from each other and what will it cost to operate?
This guide is for teams building RAG products for business users: internal knowledge assistants, customer support tools or SaaS features that answer questions over customer data. If you are still deciding whether RAG is the right approach, start with our guide to RAG or fine-tuning.
Why permissions change the choice
Imagine an internal assistant for a services company. Sales staff should see pricing documents; technicians should see service manuals; only managers should see HR policies. The assistant retrieves the five most similar chunks and passes them to the model.
| Approach | What happens | Risk |
|---|---|---|
| — | — | — |
| No permission filtering | Retrieves HR chunks for a technician’s question | Data leak in the answer |
| Filter after retrieval | Retrieves top 5, then removes chunks the user cannot see | Often too few or no relevant chunks remain; quality drops |
| Filter during retrieval | Searches only chunks the user may see | Requires the store to combine vector search with metadata filters efficiently |
| Separate indexes per audience | Each role group searches its own index | Simple for coarse groups; unmanageable for document-level permissions |
Filtering during retrieval is usually the right default, which makes filter support a key selection criterion.
Filtered vector search is harder than it looks. Approximate nearest neighbour indexes are built for unfiltered similarity; with restrictive filters, some engines return fewer results than requested or slow down sharply. Different products handle this differently. For example, pgvector added iterative index scans in version 0.8.0, which continue searching when filters remove too many candidates. Test this behaviour with your real permission rules, not with unfiltered benchmarks.
Keep permissions current
Permissions change: people move roles, documents are reclassified. Decide how permission metadata reaches the index (sync from the source system, events, scheduled jobs), how quickly changes must apply and how you will test that a revoked user no longer retrieves a document.
Two broad options
A vector index in a database you already run
Extensions and features in general-purpose databases, such as pgvector for PostgreSQL or vector search in document databases, let you store embeddings alongside your existing data. Advantages: one system to operate, transactions, familiar access control and joins with business data. Limits: scaling very large indexes, specialised retrieval features and isolating vector workloads from transactional ones.
A dedicated vector database
Purpose-built engines offer specialised index types, built-in hybrid search, sharding for large collections and managed cloud services. Advantages: scale and retrieval features. Limits: another system to operate and secure, data synchronisation and often a different access control model.
| Situation | Leaning |
|---|---|
| — | — |
| Up to a few million chunks, already on PostgreSQL, strong relational permissions | Vector index in existing database |
| Tens or hundreds of millions of chunks, heavy query volume | Dedicated vector database |
| Many tenants with strict isolation requirements | Either, depending on namespace or collection support and your deployment model |
| Team without capacity to run another data system | Existing database or a managed service |
| Need for advanced hybrid search and reranking out of the box | Dedicated engine or a search engine with vector support |
Indicative guidance. Test with your own data before deciding.
Multi-tenancy models
For SaaS products, decide how tenants are separated:
| Model | Description | Trade-offs |
|---|---|---|
| — | — | — |
| Shared index with tenant filter | All tenants in one collection; every query filtered by tenant ID | Efficient; a missing filter is a data leak, so enforce it in one place |
| Namespace or collection per tenant | Logical separation inside one deployment | Clear isolation and easy deletion; many tenants can strain limits |
| Deployment per tenant | Separate instance for large or regulated tenants | Strongest isolation; highest cost and operations |
Many products combine models: shared for small tenants, dedicated for large or regulated ones.
Whatever you choose, enforce the tenant filter in a single retrieval service rather than trusting each caller to add it, and test it: a query from tenant A must never return tenant B’s chunks.
Selection criteria
| Criterion | Questions |
|---|---|
| — | — |
| Filtered retrieval | Does recall hold with restrictive filters? How does latency change? |
| Permission and tenant model | Namespaces, collections, row-level security, per-tenant deletion? |
| Hybrid search | Can it combine keyword and vector search? Reranking support? |
| Scale | Expected chunks in 12 to 24 months; index build and update times |
| Freshness | How quickly do inserts, updates and deletes become searchable? |
| Operations | Managed option, backups, monitoring, upgrades, disaster recovery |
| Security and compliance | Encryption, audit logs, data residency, certifications |
| Integration | Client libraries, frameworks, embedding pipelines |
| Cost | Storage, compute, queries, replicas and the people to run it |
| Portability | Export format, ease of moving embeddings and metadata |
Weight these by your product’s needs; filtered retrieval and tenancy usually rank highest for business RAG.
A benchmark protocol for your own data
Vendor and public benchmarks use datasets, hardware and settings unlike yours. Run a small, fair test instead:
- Prepare a representative corpus: a realistic sample of your documents, chunked as you plan to in production, with real permission and tenant metadata.
- Build a query set: 100 to 300 real or realistic questions, each labelled with the chunks that should be retrieved and the user or role asking.
- Define measurements: recall at k with permission filters applied, latency at p50 and p95, index build time, update visibility time and resource use.
- Equalise settings: same embedding model, same chunks, comparable hardware or service tiers, documented index parameters.
- Test restrictive filters: include users who may see only a small fraction of documents, where filtered search is hardest.
- Test change: revoke a permission or delete a document and measure how long until it stops being retrieved.
- Estimate cost at current and 12-month projected volume, including operations effort.
- Record everything so the test can be rerun when products or your data change.
We have not published benchmark numbers here because results depend heavily on data, filters and configuration. The protocol matters more than any single published result.
A scorecard
| Criterion | Weight | Candidate A | Candidate B | Candidate C |
|---|---|---|---|---|
| — | — | — | — | — |
| Filtered recall and latency | 25% | |||
| Permission and tenant model | 20% | |||
| Operations and team fit | 15% | |||
| Hybrid search and retrieval features | 10% | |||
| Scale and freshness | 10% | |||
| Security and compliance | 10% | |||
| Cost over 24 months | 10% |
Fill in scores from the benchmark protocol and documented checks, not from marketing pages.
Retrieval quality is more than the database
Teams often blame the database for poor answers that come from elsewhere:
- Chunking: chunks too large dilute relevance; too small lose context.
- Metadata: titles, sections, dates and document types improve filtering and ranking.
- Hybrid search: keyword search catches exact terms such as product codes that embeddings may miss.
- Reranking: a second-stage model can reorder candidates for relevance.
- Evaluation: measure retrieval separately from answer quality so you know which part to fix. Our guide to evaluating AI chatbots explains how.
Frequently asked questions
Do we need a dedicated vector database to start?
Often not. If you already run PostgreSQL or a database with vector support and your scale is modest, starting there reduces operations. Plan a migration path if growth is likely.
How do we stop the assistant revealing documents a user cannot see?
Filter by permission during retrieval, enforce it in one retrieval service and test it with users who have narrow access.
Can we switch vector databases later?
Yes, if you keep source documents, chunks and metadata outside the vector store and can regenerate embeddings. Plan for re-indexing time and cost.
Which embedding model should we use?
Choose by testing retrieval quality on your own queries. Changing the embedding model later requires re-embedding all content.
Sources and methodology
Facts in this article were checked against the sources below. Directory data (ratings, review counts, rate bands and team sizes) changes over time and is shown as published on the date accessed. Illustrative examples are labelled as such and are not quotes.
- PostgreSQL News: pgvector 0.8.0 released — Iterative index scans for filtered queries; announced November 2024
- AltexSoft: How to Choose the Right Vector Database — Reviewed as a baseline; compares eight products by general criteria, with little on permission-aware retrieval
Related reading
Continue your research
Building a RAG product for business users?
Share your documents, users and permission model. We will help you choose a retrieval store and test it on your own data before you commit.
