Cruxly › AI engineering › Retrieval and RAG

Query Understanding and Agentic Retrieval: Rewriting, Routing, Multi-Hop and GraphRAG

fix the query before the index, then pay for a loop only when the question needs three documents

Prompt: "Our HR assistant handles 'What about for NRIs?' as a standalone query and returns nonsense. Another question needs a policy, an exception list and a form, in three different documents. Leadership asks whether we need an agentic RAG loop or GraphRAG. How do you handle follow-up questions, when do you loop, what is GraphRAG good for, and how do you retrieve three documents for one question?"

40 minutes first. Price a single-shot answer against a three-hop loop in tokens and rupees, and say what you would measure to decide. The trap is reaching for the most elaborate architecture for a failure that a one-line rewrite fixes.

Why queries fail before the index does

Users do not type retrieval queries; they type fragments of a conversation. Before you add a loop, sort your failed queries into four bins, because each bin has a different fix.

• Ambiguity and missing context. "What about for NRIs?" after a question on leave policy has no subject. Embedded alone it matches the NRI taxation documents. The fix is condensation (next section).
• Multi-intent questions. "What is the notice period and can I take my pending leave during it?" is two questions with two answer chunks. One query vector sits between both topics and may retrieve neither well.
• Vocabulary mismatch. The user says "salary slip", the policy says "payroll statement". The dense leg often bridges this and BM25 does not; Hybrid Search, Rerankers and Late Interaction is the right place to look first.
• Sequential dependence. "Which form do I submit for the exception that applies to my grade?" cannot be answered until the exception is found, because the form depends on it. Only here does a loop earn its place.

Measure the bins. Take 100 failures from your logs (the Evaluation Foundations: Golden Sets, Error Analysis and Reading Benchmarks habit), label each with its bin, and count. If 55 are missing-context follow-ups, 25 vocabulary, 15 multi-intent, and 5 sequential, a loop addresses 5 percent of your problem and a condensation step addresses more than half. The numbers here are an example; what matters is that you get yours before choosing an architecture.

Chat makes it worse. In a multi-turn product, the conversation state is what the model sees, not the user's last message (Multi-Turn Conversation State and Context-Window Limits). Retrieval needs a standalone query, so retrieval is the first place where history handling bites. Log the exact string you sent to the retriever for every turn: debugging "why did it answer about taxation" is impossible otherwise.

The rewrite family: four tools, each with a cost and a failure

Each rewrite technique adds an LLM call before retrieval, so each must beat its own latency and its own failure mode.

Conversational condensation. One call with the history plus the new message, producing a standalone question: "What about for NRIs?" becomes "What is the maternity leave policy for NRI employees?". Use a small, fast model; this call is 200 to 400 ms of latency. Failure: the rewrite drops a constraint ("for contract staff") or invents one. Log the rewrite next to the original, include multi-turn cases in your golden set, and retrieve with both the original and the rewrite if cheap.

Multi-query expansion. Generate three to five paraphrases, retrieve for each, fuse by RRF (Hybrid Search, Rerankers and Late Interaction). It improves recall on vocabulary mismatch. Cost: one LLM call plus 3 to 5x the retrieval work and extra reranker input. Failure: paraphrases drift off topic, and more variants mean more candidates for the reranker to read. Cap it at 3 and test it.

HyDE (Hypothetical Document Embeddings). Gao et al. (2022) have an instruction-following model write a hypothetical answer to the query, embed that, and retrieve real documents near it; their abstract reports gains over the unsupervised Contriever retriever and results comparable to fine-tuned retrievers in a zero-shot setting with no relevance labels. The intuition: an answer looks more like an answer passage than a question does. Failure: if the model confidently invents the wrong answer ("the notice period is 90 days" when it is 60), the embedding is pulled toward passages about 90-day periods in other contexts. It also adds a full generation of latency. Try it when queries are short and the corpus is prose; skip it for identifier and numeric lookups.

Decomposition. Break a multi-intent question into sub-questions and retrieve for each: "notice period" and "leave during notice". Independent sub-questions can run in parallel; dependent ones need the loop in section 4. Failure: over-splitting a simple question into five pieces multiplies cost for no gain, and the final synthesis must reconcile partial answers.

Pay only when needed. A rewrite that fires on every query taxes the 70 percent that did not need it. Trigger condensation only when the message has pronouns or no named subject, and trigger decomposition only when a classifier or a heuristic ("and", two question marks) says multi-intent.

Routing: which index, which tool, or no retrieval at all

A router decides what happens to a query before any expensive step, and it is usually a small classifier, not an agent. Three decisions:

1. Retrieve or not. Greetings, arithmetic, "rewrite this paragraph", and questions about the conversation itself need no retrieval. If 30 percent of traffic is like that and a normal retrieval adds 10 chunks of 600 tokens, skipping it saves 0.3 x 6,000 = 1,800 prompt tokens per query on average, plus the retrieval latency.
2. Which source. HR policies, product docs, a SQL table, a ticket system. A router picks one, or a few, rather than searching everything and polluting the context with irrelevant chunks. The Function Calling Basics: The Loop, Parallel Calls, Errors and Forced Choice mechanism is the natural implementation: each source is a tool with a description.
3. Which filters. "Invoices from March 2025 above 50,000 rupees" becomes {doc_type: "invoice", date: "2025-03", amount_gt: 50000} applied as metadata filters, so the vector search runs over the right slice. This is much more precise than hoping the embedding encodes a date.

How to build the router cheaply. Start with rules (regex for IDs and dates, keyword lists), then a small classifier or a small LLM with a fixed label set. Every router stage adds latency (20 ms for rules, 200 to 400 ms for an LLM call), so measure what it costs against what it saves.

How routers fail, and the asymmetry that sets the policy. A router can make two errors, and they are not equal. Sending a query that needed documents down the no-retrieval path produces a fluent hallucination; sending a chit-chat query to retrieval wastes tokens. Bias toward retrieving: require recall of at least 0.99 on the needs-retrieval class on a labelled set, and accept lower precision. For filter extraction, the failure is a hallucinated filter: the model infers department = "Finance" that the user never said, and the query returns zero results. Validate the filter against the schema and the known values (schema-constrained structured output helps with the shape, but not with whether the value was ever said), and add a fallback: if a filtered search returns zero results, retry without the inferred filters and mark the answer as unfiltered.

Permissions are not a routing decision. Never let a router, a rewriter or an agent decide who may see what; access control is enforced in the database, as Multi-Tenant, Multilingual RAG at Scale says. A rewrite step that can influence a filter must not be able to widen the user's access.

Agentic retrieval: search as a tool in a loop, and when it pays

An agentic retrieval loop gives the model a search tool and lets it decide the next query from what it has already read. It is the ReAct pattern (Yao et al., 2022, reasoning and acting interleaved) applied to a corpus; the loop mechanics are in Function Calling Basics: The Loop, Parallel Calls, Errors and Forced Choice. It beats single-shot retrieval on exactly one kind of question: where the next query depends on the previous answer.

The numbers, from a deliberately simple model. A question needs three evidence documents, one per hop. Suppose a single retrieval finds a given document with probability 0.70.

• Single-shot must find all three from the one original query: 0.70^3 = 0.343. (And since hop 2's wording depends on hop 1's answer, the true figure is likely lower.)
• Loop rewrites the query at each hop using what was found. Suppose that lifts each hop to 0.92: 0.92^3 = 0.779.

Those per-hop probabilities are assumptions I chose to illustrate the structure, not measurements; replace them with a run on a public multi-hop set such as HotpotQA or MuSiQue before you quote an improvement to anyone.

Now the cost, with stated token counts. System prompt and question: 1,500 tokens. Each retrieval returns 5 chunks of 600 tokens (3,000). Each model step emits about 100 tokens. The single-shot call retrieves 10 chunks: 1,500 + 10 x 600 = 7,500 tokens. The loop makes three search calls and one final answer call, and every call re-reads the growing history. Input sizes: call 1 reads 1,500; call 2 reads 1,500 + 100 + 3,000 = 4,600; call 3 reads 4,600 + 3,100 = 7,700; the final call reads 7,700 + 3,100 = 10,800. That is a total of 24,600 input tokens (ignoring the few hundred output tokens), which is 3.3x the single shot. At an assumed $2 per million input tokens, that is $0.015 against $0.049 per question. Latency scales with the number of sequential model calls: if one call takes about 1.2 s and a search 0.3 s, four calls plus three searches is about 5.7 s against 1.5 s for one call and one search. Every figure in this paragraph is an assumption chosen to show the arithmetic; measure yours.

Budgets and stop rules are not optional. A loop without limits is the standard way to produce a surprise bill.

• Hop cap (3 to 4) and token cap (for example 40k per question).
• Stop when the model states it can answer with citations, or when two consecutive searches add no new document.
• Deduplicate queries: an identical search string twice means the loop is stuck.
• Compact the history: keep the found passages' key lines, not every full chunk, as Multi-Turn Conversation State and Context-Window Limits describes.
• Fall back to the best single-shot answer when the budget runs out, with a flag that the evidence may be incomplete.

When it only multiplies cost. Single-fact lookups, FAQ-style questions, and anything the router can answer with one filtered search. Put the loop behind the router: roughly 5 percent of queries in our earlier example are truly sequential; paying 3.3x tokens on all of them instead of on those few would raise average cost by a factor of 3.3 across the board, where gating it raises it by 0.95 + 0.05 x 3.3 = 1.11 (same assumed figures).

GraphRAG: global questions, indexing cost, and when a loop is enough

GraphRAG is built for questions about a whole corpus, not for finding three linked facts. The Microsoft paper "From Local to Global" (Edge et al., 2024) describes the method: an LLM extracts an entity-relationship graph from the documents, the graph is partitioned into communities of closely related entities, and a summary is pre-generated for each community. At query time each community summary produces a partial answer, and the partial answers are summarised into a final one. The authors target global sensemaking questions ("What are the main themes across these 3,000 documents?") that a top-k retriever cannot answer, since no handful of chunks represents the corpus. They report substantial improvements over a conventional RAG baseline in the comprehensiveness and diversity of answers on corpora around one million tokens; the abstract gives no further figures, and it is the authors' own evaluation, so treat it as such.

What it costs. The index is built by an LLM reading the entire corpus at least once for extraction, then writing summaries on top. Even at one pass, a 100M-token corpus read by a model at an assumed $0.50 per million input tokens is 100 x $0.50 = $50, and real pipelines make several passes (extraction, repeated extraction to find missed entities, community summaries), so treat that as a floor and measure on a sample. Then add staleness: when documents change, entities, edges and the summaries of affected communities must be recomputed, and a stale summary answers confidently from a world that moved on.

When a loop (or a join) is enough. "Which vendors supply parts used in products customer X bought?" has a structure: customer, orders, products, parts, vendors. That is a multi-hop lookup, answerable by a SQL join if the data is relational, or by a 4-hop agentic loop over documents. A community summary of all vendors adds nothing to it. This is the same advice Multi-Tenant, Multilingual RAG at Scale gives about graph-aware retrieval: name the actual multi-hop questions before building a graph layer.

Before paying for GraphRAG, ask whether the global questions are common (one in twenty, or one in two?), and whether a cheaper map-reduce over document clusters answers them well enough in an evaluation.

Evaluating the whole thing: accuracy, recall per hop, tokens and latency

An agentic retrieval system needs four numbers, not one, because accuracy alone cannot tell you if you got it for a reasonable price. Evaluate on a multi-hop golden set (build one from your own logs; HotpotQA or MuSiQue are public places to rehearse the method).

1. End-to-end answer accuracy, graded by labelled reference answers or a calibrated judge (LLM-as-a-Judge: Evaluation That Scales Without Lying to You).
2. Retrieval recall per hop. For each question, label the evidence documents in order. Did hop 1 find document 1? Given document 1, did hop 2's rewritten query find document 2? A system that scores 0.78 end to end because hop 2 is fine and hop 3 is at 0.60 needs a different fix than one with a uniform 0.92.
3. Tokens per question, the sum over all calls, with the p95 (the expensive tail is where loops go wrong).
4. Latency per question, p50 and p95, with the number of hops.

Report cost per correct answer. In our assumed model the single-shot costs $0.015 and succeeds 0.343 of the time, which is $0.015 / 0.343 = $0.044 per success. The loop costs $0.049 and succeeds 0.779 of the time: $0.049 / 0.779 = $0.063 per success. The loop is 2.3 times better at getting the right answer and about 1.4x more expensive per correct answer (all inputs are the assumptions above, not measurements). Whether that is acceptable depends on what a wrong answer costs. For an internal HR bot with a human fallback, perhaps not; for a compliance assistant where a missing exception is a legal exposure, almost certainly yes. Say that out loud in the interview.

Evaluate the pieces separately too. A condensation step has its own test (the rewritten question matches a human rewrite on 50 multi-turn cases); a router has its precision and recall; a loop has hop-count and budget-exhaustion rates. If 12 percent of loops hit the budget cap, that is a finding: either the cap is too low, or those questions are unanswerable and should be flagged. And as always, run it per segment and per tenant (Evaluation Gates That Actually Block a Bad Deploy), because a regression on one document type disappears in an average.

How to say it in the interview

"Most retrieval failures are query failures, so I look at the failed queries first and sort them: pronouns in a follow-up, several intents in one sentence, vocabulary that does not match the corpus, or a question that genuinely needs documents in sequence. The first three are fixed cheaply by condensing the question with the chat history, expanding it into a few variants, or routing it. Only the last needs a loop, and in my worked model (assumed numbers, not a measurement) a loop is about three times the input tokens and nearly four times the latency, so I would add it behind a router, with a hop and token budget, and prove it on a multi-hop eval set. GraphRAG is for global questions about a whole corpus, not for multi-hop lookups, and its indexing cost is at least one LLM read of everything."

Checklist

• Labelled 100 real failures into missing-context, vocabulary, multi-intent and sequential bins before choosing a fix
• Condense follow-ups into standalone questions and log original and rewritten query side by side
• Added multi-turn and multi-intent cases to the golden set
• Triggered each rewrite (condense, expand, decompose, HyDE) only on queries that show its symptom
• Capped multi-query at 3 variants and fused with RRF
• Built the router as rules then a small classifier, biased to retrieve, with recall of 0.99 on the needs-retrieval class
• Validated inferred metadata filters and retried unfiltered when they return zero results
• Kept access control in the database, not in routers, rewriters or agents
• Set hop, token and duplicate-query limits on the agentic loop with a flagged best-effort fallback
• Gated the loop behind a router so only sequential questions pay the loop's extra tokens (about 3.3x in the lesson's assumed model)
• Measured recall per hop, tokens and latency (p95), and cost per correct answer
• Priced GraphRAG as an LLM pass over the whole corpus plus re-summarising on change, and asked whether global questions are common

Related lessons

• RAG From Scratch, No Framework
• Chunking and Document Parsing: Structure-Aware Splits, Parent-Child, Late Chunking and Contextual Retrieval
• Choosing, Evaluating and Fine-Tuning an Embedding Model
• ANN Indexes in Depth: HNSW, IVF-PQ, DiskANN and the Recall/Latency/Memory Triangle
• Hybrid Search, Rerankers and Late Interaction
• Multi-Tenant, Multilingual RAG at Scale