Home / Insights / Generative AI and LLMs

Generative AI and LLMs

Choosing a Vector Database for a Permission-Aware RAG Product

Most vector database comparisons focus on speed. In a business RAG product the harder questions are whether retrieval respects user permissions, how tenants are isolated and what it costs to run. This guide gives selection criteria, a scorecard and a benchmark protocol to test candidates on your own data.

TopicsGenerative AI and LLMsAI and Machine LearningArchitecture and Modernization

Key takeaways

  • In a business RAG product, retrieval must respect who may see each document. Filtering after retrieval can return too few results or leak information through answers.
  • Choose between a vector index inside a database you already run and a dedicated vector database based on scale, filtering needs, operations and team skills, not on headline benchmarks.
  • Decide your multi-tenancy model early: shared index with tenant filters, separate namespaces or collections, or separate deployments.
  • Evaluate with your own documents, queries and permission rules, measuring recall with filters, latency and cost.
  • Retrieval quality depends more on chunking, metadata and hybrid search than on the database alone.

Retrieval-augmented generation (RAG) gives a language model access to your documents at question time. The vector database stores numerical representations of document chunks and finds those most similar to a question. Many comparisons rank vector databases by queries per second on public datasets. Those numbers rarely decide a business product. More often the decision turns on three questions: can retrieval respect each user’s permissions, how are customers isolated from each other and what will it cost to operate?

This guide is for teams building RAG products for business users: internal knowledge assistants, customer support tools or SaaS features that answer questions over customer data. If you are still deciding whether RAG is the right approach, start with our guide to RAG or fine-tuning.

Why permissions change the choice

Imagine an internal assistant for a services company. Sales staff should see pricing documents; technicians should see service manuals; only managers should see HR policies. The assistant retrieves the five most similar chunks and passes them to the model.

ApproachWhat happensRisk
———
No permission filteringRetrieves HR chunks for a technician’s questionData leak in the answer
Filter after retrievalRetrieves top 5, then removes chunks the user cannot seeOften too few or no relevant chunks remain; quality drops
Filter during retrievalSearches only chunks the user may seeRequires the store to combine vector search with metadata filters efficiently
Separate indexes per audienceEach role group searches its own indexSimple for coarse groups; unmanageable for document-level permissions

Filtering during retrieval is usually the right default, which makes filter support a key selection criterion.

Filtered vector search is harder than it looks. Approximate nearest neighbour indexes are built for unfiltered similarity; with restrictive filters, some engines return fewer results than requested or slow down sharply. Different products handle this differently. For example, pgvector added iterative index scans in version 0.8.0, which continue searching when filters remove too many candidates. Test this behaviour with your real permission rules, not with unfiltered benchmarks.

Keep permissions current

Permissions change: people move roles, documents are reclassified. Decide how permission metadata reaches the index (sync from the source system, events, scheduled jobs), how quickly changes must apply and how you will test that a revoked user no longer retrieves a document.

Two broad options

A vector index in a database you already run

Extensions and features in general-purpose databases, such as pgvector for PostgreSQL or vector search in document databases, let you store embeddings alongside your existing data. Advantages: one system to operate, transactions, familiar access control and joins with business data. Limits: scaling very large indexes, specialised retrieval features and isolating vector workloads from transactional ones.

A dedicated vector database

Purpose-built engines offer specialised index types, built-in hybrid search, sharding for large collections and managed cloud services. Advantages: scale and retrieval features. Limits: another system to operate and secure, data synchronisation and often a different access control model.

SituationLeaning
——
Up to a few million chunks, already on PostgreSQL, strong relational permissionsVector index in existing database
Tens or hundreds of millions of chunks, heavy query volumeDedicated vector database
Many tenants with strict isolation requirementsEither, depending on namespace or collection support and your deployment model
Team without capacity to run another data systemExisting database or a managed service
Need for advanced hybrid search and reranking out of the boxDedicated engine or a search engine with vector support

Indicative guidance. Test with your own data before deciding.

Multi-tenancy models

For SaaS products, decide how tenants are separated:

ModelDescriptionTrade-offs
———
Shared index with tenant filterAll tenants in one collection; every query filtered by tenant IDEfficient; a missing filter is a data leak, so enforce it in one place
Namespace or collection per tenantLogical separation inside one deploymentClear isolation and easy deletion; many tenants can strain limits
Deployment per tenantSeparate instance for large or regulated tenantsStrongest isolation; highest cost and operations

Many products combine models: shared for small tenants, dedicated for large or regulated ones.

Whatever you choose, enforce the tenant filter in a single retrieval service rather than trusting each caller to add it, and test it: a query from tenant A must never return tenant B’s chunks.

Selection criteria

CriterionQuestions
——
Filtered retrievalDoes recall hold with restrictive filters? How does latency change?
Permission and tenant modelNamespaces, collections, row-level security, per-tenant deletion?
Hybrid searchCan it combine keyword and vector search? Reranking support?
ScaleExpected chunks in 12 to 24 months; index build and update times
FreshnessHow quickly do inserts, updates and deletes become searchable?
OperationsManaged option, backups, monitoring, upgrades, disaster recovery
Security and complianceEncryption, audit logs, data residency, certifications
IntegrationClient libraries, frameworks, embedding pipelines
CostStorage, compute, queries, replicas and the people to run it
PortabilityExport format, ease of moving embeddings and metadata

Weight these by your product’s needs; filtered retrieval and tenancy usually rank highest for business RAG.

A benchmark protocol for your own data

Vendor and public benchmarks use datasets, hardware and settings unlike yours. Run a small, fair test instead:

  1. Prepare a representative corpus: a realistic sample of your documents, chunked as you plan to in production, with real permission and tenant metadata.
  2. Build a query set: 100 to 300 real or realistic questions, each labelled with the chunks that should be retrieved and the user or role asking.
  3. Define measurements: recall at k with permission filters applied, latency at p50 and p95, index build time, update visibility time and resource use.
  4. Equalise settings: same embedding model, same chunks, comparable hardware or service tiers, documented index parameters.
  5. Test restrictive filters: include users who may see only a small fraction of documents, where filtered search is hardest.
  6. Test change: revoke a permission or delete a document and measure how long until it stops being retrieved.
  7. Estimate cost at current and 12-month projected volume, including operations effort.
  8. Record everything so the test can be rerun when products or your data change.

We have not published benchmark numbers here because results depend heavily on data, filters and configuration. The protocol matters more than any single published result.

A scorecard

CriterionWeightCandidate ACandidate BCandidate C
—————
Filtered recall and latency25%
Permission and tenant model20%
Operations and team fit15%
Hybrid search and retrieval features10%
Scale and freshness10%
Security and compliance10%
Cost over 24 months10%

Fill in scores from the benchmark protocol and documented checks, not from marketing pages.

Retrieval quality is more than the database

Teams often blame the database for poor answers that come from elsewhere:

  • Chunking: chunks too large dilute relevance; too small lose context.
  • Metadata: titles, sections, dates and document types improve filtering and ranking.
  • Hybrid search: keyword search catches exact terms such as product codes that embeddings may miss.
  • Reranking: a second-stage model can reorder candidates for relevance.
  • Evaluation: measure retrieval separately from answer quality so you know which part to fix. Our guide to evaluating AI chatbots explains how.

Frequently asked questions

Do we need a dedicated vector database to start?

Often not. If you already run PostgreSQL or a database with vector support and your scale is modest, starting there reduces operations. Plan a migration path if growth is likely.

How do we stop the assistant revealing documents a user cannot see?

Filter by permission during retrieval, enforce it in one retrieval service and test it with users who have narrow access.

Can we switch vector databases later?

Yes, if you keep source documents, chunks and metadata outside the vector store and can regenerate embeddings. Plan for re-indexing time and cost.

Which embedding model should we use?

Choose by testing retrieval quality on your own queries. Changing the embedding model later requires re-embedding all content.

Sources and methodology

Facts in this article were checked against the sources below. Directory data (ratings, review counts, rate bands and team sizes) changes over time and is shown as published on the date accessed. Illustrative examples are labelled as such and are not quotes.

  1. PostgreSQL News: pgvector 0.8.0 released — Iterative index scans for filtered queries; announced November 2024
  2. AltexSoft: How to Choose the Right Vector Database — Reviewed as a baseline; compares eight products by general criteria, with little on permission-aware retrieval

Building a RAG product for business users?

Share your documents, users and permission model. We will help you choose a retrieval store and test it on your own data before you commit.