Cruxly › AI engineering › Retrieval and RAG

Multi-Tenant, Multilingual RAG at Scale

row-level security, language routing, and graph-aware hops

Prompt: take the single-tenant, English-only RAG pipeline and make it serve multiple business customers, each with private documents, and users querying in several languages — without ever leaking one tenant's data to another, and without doubling latency for non-English users.

40 minutes first. Both requirements are correctness requirements, not performance tuning, even though they show up in a "scale" conversation.

Tenant isolation: a security property, not a WHERE clause

Row-level security enforced by the database is the only version that fails closed. An application-layer filter (WHERE tenant_id = ? added in your query-building code) works exactly as long as every single code path that touches the vector store remembers to add it. A new admin tool, a batch export script, a debugging endpoint added under deadline pressure — each is a chance to forget the filter, and each forgotten filter is a tenant's private documents returned to a different tenant's query. Database- enforced row-level security (RLS) makes the omission structurally impossible: the database itself refuses to return rows outside the connection's authorized tenant, regardless of what the application code does or forgets to do.

Index structure has a real cost here, and it's worth naming. A single shared vector index across all tenants, filtered by tenant ID at query time, means large tenants' documents dilute the approximate-nearest-neighbor search for small tenants (the index has to consider vastly more candidates before filtering). Per-tenant indexes avoid this at the cost of operational overhead — many indexes to manage, rebuild, and monitor instead of one. For a modest tenant count, RLS on a shared index is the pragmatic default; past some scale, per-tenant (or per-tenant-cohort) indexes become worth the operational cost, and knowing which regime you're in is the actual skill.

Watch for the filter that eats your results. With pgvector, an HNSW index search finds the nearest candidates first and RLS or WHERE tenant_id filters them afterwards. A small tenant's documents may make up almost none of the nearest candidates, so a top-10 query can return 2 results — or none — even though relevant documents exist. Fixes: pgvector 0.8's iterative index scans (keep scanning until enough rows pass the filter), partial or partitioned indexes per large tenant, or a vector database's native namespaces/partitions, which filter before the search instead of after.

Tenant isolation must extend to evaluation and logging, not just serving. A RAGAS evaluation run, a debug trace, an error log — every one of these can just as easily leak cross-tenant data as the serving path itself if it wasn't built with the same isolation discipline. The serving path is not the only path.

HNSW versus IVF: the index choice that stops being a reputation contest at scale

HNSW and IVF are not interchangeable defaults, and picking one on reputation instead of measuring both against your own corpus is a real, common mistake once a corpus grows past a comfortable single-tenant demo. They trade recall, memory, and latency differently, and which one wins depends on your actual data, your actual write pattern, and your actual memory budget — not on which one showed up first in a tutorial.

HNSW (Hierarchical Navigable Small World) builds a navigable graph with many edges per node. It gives strong recall and low latency for interactive queries — which is why it's the default reached for first.

Two build-time parameters, one query-time knob. M (edges per node) and ef_construction (build-time search width) are fixed once the graph is built. ef_search is a live, per-query recall/latency dial — raise it for a hard query, lower it under load. HNSW is not a "set once" index.
Its real weakness is writes and deletes. Inserting a node rewires its neighbours' edge lists, which gets lock-heavy under sustained concurrent writes; deletes usually leave tombstones that degrade the graph until a rebuild.
Graph overhead is smaller than its reputation. The edges cost roughly M × 2 × 4 bytes per vector — a few hundred bytes — next to 3–6 KB for a 768–1536-dim float32 vector. The vectors themselves are the memory problem, not the graph.

IVF (Inverted File Index) clusters the vector space into partitions and probes only a subset per query. nprobe is its per-query knob: more partitions probed means more recall and more latency.

IVF has its own write problem, just a slower one. Centroids are trained on the data present at build time. As the corpus drifts, new vectors pile into a few partitions, recall drops, and you eventually retrain centroids and rebuild — a scheduled cost, not "holds up better under writes."

Property	HNSW	IVF
Structure	Navigable graph, many edges/node	Clustered partitions, probe a subset
Per-query knob	`ef_search`	`nprobe`
Fixed at build	`M`, `ef_construction`	Number of lists, trained centroids
Write behaviour	Insert contention; deletes leave tombstones	Partitions unbalance as data drifts; periodic retrain
Typical fit	Low-latency interactive search, mostly-in-memory	Very large corpora, usually combined with quantization

The real memory levers are quantization and disk, not the HNSW-vs-IVF choice itself.

Vector quantization: scalar (int8 — about 4× smaller), product quantization (IVF-PQ — an order of magnitude or more), binary (32× smaller, usually with a full-precision rescoring pass on the top candidates).
Matryoshka embeddings: models trained so the first N dimensions are usable on their own — truncate 1536 → 256 dims for a 6× cut, with measured recall loss.
Disk-based indexes (DiskANN-style): keep compressed vectors in RAM and full vectors on SSD, for corpora too large to hold in memory at all.

Every one of these trades recall for memory, so measure the recall cost on your own queries rather than assuming it's acceptable.

Hybrid search in production: dense plus sparse, with reranking on a bounded window

Hybrid search combining dense vector similarity with sparse keyword search (BM25) is the production default, not an optional enhancement — and the reason is a specific, recurring failure mode: exact-term matches (product codes, names, acronyms, an order ID) are something embeddings alone are unreliable at, because a specific alphanumeric code carries little semantic weight in embedding space even though it's exactly what the query needs matched precisely. Dense and sparse retrieval are run in parallel by default, not reached for only when an exact-match failure is noticed in production (the order-ID worked example is in the hybrid and graph-aware section below).

Results from both retrievers are commonly combined via reciprocal-rank fusion (RRF) — rank-based merging, detailed in the hybrid and graph-aware section below, which sidesteps the "0.82 cosine similarity versus a BM25 score of 14.3, which one actually wins" problem that comparing raw scores across dense and sparse retrieval runs into.

An optional reranking pass — exact or cross-encoder — runs over a small, bounded candidate window, never the full retrieved set, because reranking is too expensive to run broadly. This is the same two-stage shape (cheap broad retrieval, expensive narrow reranking) from the from-scratch RAG lesson, now stated as the production default specifically for a hybrid dense-plus-sparse pipeline: fuse the dense and sparse candidate lists first, take a bounded window off the top of the fused ranking (tens of candidates, not thousands), and only rerank that window. Running a cross-encoder over the full fused candidate set defeats the purpose of having a cheap first stage at all.

Multilingual retrieval without the translate-then-search tax

Start with a cross-lingual embedding model, and you mostly don't need language routing at all. Models such as BGE-M3, multilingual-e5, or Cohere's multilingual embeddings map a Hindi query and an English document about the same thing to nearby vectors. One index, one search path, no detection step, no translation round-trip.

Why this beats detect-then-route for Indian traffic: a large share of real queries are code-mixed ("mera order kab aayega, tracking ID ORD-88213"). Language detection is unreliable on Hinglish and on short queries, so a routing architecture fails exactly on the biggest segment. The NLP lesson on multilingual classification makes the same point.
Measure per language anyway. Cross-lingual retrieval quality is uneven — strong for Hindi, weaker for lower-resource languages. Build eval queries per language (and code-mixed) and check recall@k for each, not one blended number.

Translation is the fallback, for languages the embedding model handles poorly. If a language's recall stays low after measuring, translate the query into the corpus's dominant language — never translate the entire corpus into every language ahead of time, which multiplies storage and staleness by the number of languages supported.

The translation step needs its own evals. A mistranslated query produces a wrong-but-fluent retrieval that looks identical to a correct one. The failure that matters is domain terms — product names, technical terms, regional phrasing — that general-purpose translation handles worst.
Exact identifiers bypass all of this. Order IDs, SKUs and error codes are language-independent; the BM25 leg of hybrid search catches them regardless of what language surrounds them.

Hybrid and graph-aware retrieval: knowing which tool the query needs

Vector search answers "what's semantically similar." Keyword search answers "what contains this exact term." A query like "what's the status of order ORD-88213" needs an exact match on an order ID far more than semantic similarity — a vector search might rank a document about order statuses in general above the one document containing that specific order ID, because the specific ID carries little semantic weight in embedding space. Running both a vector search and a keyword (BM25) search in parallel and merging results (reciprocal rank fusion is the standard merge function) catches both failure modes that either search alone misses.

Graph-aware retrieval earns its complexity only when the domain has real multi-hop relationships to exploit. "Which vendors supply parts used in products that customer X purchased" requires traversing customer → orders → products → parts → vendors — a chain no single vector search or keyword search can answer, because the answer isn't semantically or lexically present in any one document; it's assembled from relationships across several. Building a graph layer for a domain that doesn't have this kind of multi-hop question is pure overhead with no payoff — name the actual multi-hop questions your product needs to answer before reaching for this, rather than adding it because it sounds sophisticated.

Merging three retrieval sources (vector, keyword, graph) needs an explicit fusion strategy, not string concatenation of results. Reciprocal rank fusion — scoring each candidate by the sum of 1/(rank + k) across each source's ranking, rather than raw scores, which aren't comparable across different retrieval methods — is the standard approach precisely because it sidesteps the "vector similarity of 0.82 vs BM25 score of 14.3, which one wins" problem entirely.

What breaks at scale, and the honest edge cases

A tenant with a much larger corpus dominates shared infrastructure costs and can degrade shared-index query latency for everyone else unless usage is metered and, past a threshold, migrated to isolated infrastructure. Naming this before a customer complaint forces the migration under pressure is the senior move.

Re-indexing after an embedding model upgrade is a full-corpus operation across every tenant, and it cannot happen instantaneously — there is necessarily a window where old and new vectors coexist. Version-tag every vector and either maintain dual indexes during migration (serve from whichever version a given document has, until migration completes) or accept a maintenance window; silently mixing versions mid-migration is the failure mode from the "no framework" lesson, now happening at the worst possible scale.

A language with very little corpus coverage will retrieve poorly no matter how good the retrieval architecture is, because there's insufficient signal to match against — this is a content gap, not an engineering gap, and the honest response is surfacing it as a known limitation rather than tuning retrieval parameters indefinitely against a problem retrieval can't solve.

Cross-tenant evaluation aggregates can hide a single tenant's regression. An overall RAGAS faithfulness score that looks stable can mask one tenant's score dropping sharply if their query volume is small relative to the total. Evaluate per-tenant, not only in aggregate, or a real regression for one customer goes unnoticed until they complain.

How to say it in the interview

"Tenant isolation has to be enforced at the database layer, not in application code, because an app-layer filter that's missed in one code path is a data breach — and it will be missed eventually, in some code path, by someone. And multilingual can't mean translate-then-search; that loses precision and doubles cost on every non-English query."

Checklist

Caught the post-filter problem: ANN search then tenant filter can return fewer than K rows
Enforced tenant isolation with database-level row security, not an app-layer filter
Named the shared-index-vs-per-tenant-index trade explicitly
Extended isolation discipline to evaluation runs and logs, not just the serving path
Used a cross-lingual embedding model instead of a detect-then-route pipeline, and measured recall per language
Evaluated translation quality on real domain queries, not vendor benchmarks
Combined vector and keyword search with a real fusion strategy
Justified graph-aware retrieval by naming actual multi-hop questions, not by default
Evaluated faithfulness per-tenant, not only in aggregate
Measured HNSW versus IVF against the actual corpus rather than picking one by reputation
Named the per-query knobs: ef_search for HNSW, nprobe for IVF
Treated quantization (scalar, PQ, binary, Matryoshka truncation) or disk-based indexes as the memory lever, and measured the recall cost
Ran dense and sparse (BM25) retrieval in parallel by default, not only after noticing an exact-match failure
Bounded the reranking pass to a small candidate window after fusion, not the full retrieved set

Practice this interactively