Table of Contents
By Khimananda Oli | Last reviewed: October 2026
Agentic RAG treats retrieval as a tool the model can call multiple times within one turn, rather than a single step that runs once before generation. Classic RAG answers "what is X" well because one retrieval covers it; it answers "how does X relate to Y, and what changed between them" badly, because that needs the result of one search to shape the next. Agentic RAG closes that gap with a retrieve-reason-retrieve loop — at the cost of unpredictable latency and a real risk of looping forever if you don't cap it in code.
What's actually wrong with running retrieval once?
If you've built a standard RAG pipeline — see RAG explained if you haven't — you already know the shape: embed the query, search a vector store, stuff the top-k chunks into the prompt, generate one answer. That architecture has a structural ceiling, not an implementation bug you can tune away. A single retrieval call encodes the original question into a search; it cannot encode a question that only becomes askable after seeing the first batch of results.
"Which engineer worked on both the payments migration and the incident that followed it" is unanswerable in one retrieval pass if those two facts live in different documents written months apart — the query embedding for the combined question doesn't resemble either document closely enough to retrieve both reliably. A human researcher wouldn't try to search for that in one query either: they'd search for the migration, read who was on it, then search again for incidents naming those people. That second search only exists because of what the first one returned. That is precisely the capability classic RAG's single-pass architecture cannot express, and precisely what agentic RAG adds.
What changes when retrieval becomes a tool instead of a pipeline step?
The architectural shift is smaller than it sounds and more consequential than it looks. Instead of calling your retriever as a fixed step before the model ever runs, you register it as a tool — the same tool-calling mechanism covered in function calling and tool use with LLMs — and let the model decide whether, when, and how many times to call it, in exactly the same way it would decide to call a calculator or a web search function.
retrieval_tool = {
"name": "search_documents",
"description": "Search the knowledge base for passages relevant "
"to a query. Call again with a refined or follow-up "
"query if the first results are incomplete.",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string"},
"filters": {"type": "object",
"description": "optional metadata filters"},
},
"required": ["query"],
},
}Nothing about the retriever itself changes — same embeddings, same vector store, same chunking. What changes is who decides when to call it and with what query, and that the model sees the result of each call before deciding on the next one. This is the entire mechanism: classic RAG retrieves then generates, once; agentic RAG lets the model retrieve, read, decide it needs more, retrieve again with a refined query informed by what it just read, and repeat until it has enough to answer — or until a cap you set in code stops it.
When does the extra complexity actually pay for itself?
Not every query benefits, and sending every query through a multi-hop loop is a predictable way to triple your latency and cost for no quality gain. The dividing line is whether the question can be answered from one retrieval's worth of evidence or genuinely needs the first retrieval to inform the second.
| Query shape | Example | Fits |
|---|---|---|
| Single-fact lookup | "What's our refund policy for digital goods?" | Classic RAG — one retrieval has everything needed |
| Comparison across known entities | "How does plan A's rate limit compare to plan B's?" | Classic RAG, if both live in one or two retrievable chunks |
| Multi-hop, entities discovered mid-search | "Who worked on both the payments migration and the incident after it?" | Agentic RAG — the second search depends on the first's results |
| Open-ended investigation | "Why did latency regress last quarter?" | Agentic RAG — the right documents aren't known until partway through |
| Anything with a hard deadline on response time | A live chat widget's first response | Classic RAG, or a capped single hop — agentic RAG's variable latency is a poor fit |
A practical pattern many teams converge on is classifying the query first and routing accordingly — simple-looking questions go straight to a single-pass retrieval, and only questions that look like they need cross-referencing or investigation go to the agentic path. That classification step is the same decision-making problem covered in LLM model routing and cascading, applied to a choice between architectures instead of between models — and it exists for the same reason: most traffic doesn't need the expensive path, so don't default everything onto it.
What stops the loop from running forever?
This is the part that actually matters in production, and it is where most agentic RAG implementations that work in a demo fail under real traffic. A model given an open-ended "call this tool as many times as you need" instruction will, often enough, keep calling it — chasing a slightly better answer, re-querying a near-duplicate of its last search, or genuinely getting stuck reasoning in circles. None of this is a model bug to wait out; it's an architecture that has no edges, and edges have to come from your code, not the prompt.
def agentic_retrieve(query, retriever, llm, max_hops=4, max_results_seen=20):
evidence = []
seen_queries = set()
for hop in range(max_hops):
decision = llm.decide_next_action(query, evidence)
if decision.action == "answer":
return llm.generate_answer(query, evidence)
search_query = decision.search_query
if search_query in seen_queries:
# model re-issued a query it already tried — force a stop
break
seen_queries.add(search_query)
results = retriever.search(search_query, k=5)
evidence.extend(results)
if len(evidence) >= max_results_seen:
break # enough raw material; force a final answer attempt
return llm.generate_answer(query, evidence, forced=True)Three caps do most of the work: a hard limit on the number of hops (max_hops), a hard limit on total evidence gathered regardless of hop count, and a duplicate-query check that treats repeating an already-tried search as a signal to stop rather than loop. A practitioner discipline worth stating plainly: cap the loop in code, not only in the prompt. A system prompt that says "retrieve at most 3 times" is a suggestion the model can and will occasionally ignore under the right (or wrong) conditions; a for loop with a fixed range cannot.
The fourth cap worth adding once you've watched this run in production: stop when a new retrieval returns nothing meaningfully different from what you already have. A hop that surfaces the same three chunks the previous hop already surfaced is not making progress, and letting the loop continue past that point burns latency and tokens without improving the answer.
What does a real agentic RAG trace actually look like?
Walking through the payments-migration example end to end makes the loop concrete rather than abstract. The question: "Who worked on both the payments migration and the incident that followed it?"
Hop 1 — query: "payments migration project team"
→ returns: migration doc naming 4 engineers, including
one "Priya N." and one "Marcus T."
Hop 2 — query: "incident payments after migration Priya Marcus"
→ informed by hop 1's names; returns: an incident
postmortem naming "Marcus T." as the responder,
no mention of the other 3 engineers
Hop 3 — model decision: enough evidence (one name appears
in both documents; the others don't overlap) → generate
Answer: "Marcus T. worked on both the migration and the
incident response that followed it."Note what the second hop's query contains that the first hop's query could not have: the actual names "Priya" and "Marcus," which only existed after hop one returned the migration document. A single-pass retrieval could never have constructed that second query, because the original question doesn't mention either name — it only becomes answerable once the first search's results are available to build the next search from. This is the entire value of agentic RAG in one trace, and it's also exactly the kind of transcript worth logging verbatim for the evaluation discussed next: three hops, each query, and a clear decision point where the model judged it had enough to stop.
How do you evaluate whether the extra hops actually helped?
Agentic RAG adds exactly the kind of multi-step behaviour that output-only evaluation misses — the same gap covered generally in evaluating AI agents: trajectory, tool calls, task completion. A correct final answer reached in four unnecessary hops is a worse outcome than the same answer reached in two, even though a pass/fail eval scores them identically; log the hop count, the queries issued at each hop, and whether each hop actually introduced evidence the previous one didn't have. That log is what tells you whether your caps from the previous section are too tight, too loose, or about right — not a hunch after watching it run a few times.
The retrieval quality metrics from RAG evaluation metrics: diagnose retrieval before you fine-tune still apply at each individual hop — recall@k and precision@k are just as meaningful on hop three as on a single-pass retrieval. What's new is a hop-level question those metrics don't answer on their own: did this specific hop's query, informed by the previous hop's results, retrieve something the previous hop's query could not have found? If the answer is consistently no across your evaluation set, the extra hops are adding latency without adding capability, and the query classifier deciding which questions get the agentic path needs tightening.
What does this cost compared to classic RAG?
Multiply by the hop count, roughly. A two-hop agentic RAG answer costs approximately twice a single-pass answer in both retrieval calls and the generation tokens spent reasoning between hops; a four-hop answer costs roughly four times as much, plus the latency of four sequential round trips rather than one. This is exactly why the routing decision in the earlier table matters commercially, not just architecturally — every query that reaches the agentic path and could have been answered in one hop is pure waste, measured the same way LLM cost optimization for production apps measures any other avoidable spend. If you're also instrumenting these calls, OpenTelemetry's GenAI semantic conventions give you gen_ai.execute_tool.duration per retrieval call already, which is the natural place to track hop count and cost per completed agentic RAG query without inventing a separate metric for it.
Start narrow: route only the query shapes from the earlier table that clearly need multi-hop reasoning onto the agentic path, cap the loop in code from day one rather than after an incident, and measure hop-level contribution before trusting it on the bulk of your traffic. If you want agentic RAG built against your own corpus and query patterns — with the caps, the routing logic, and the evaluation to prove the extra hops are earning their cost — my DevOps and cloud consulting services cover exactly this.
