Agentic RAG Explained: Build a Self-Correcting RAG Agent

Imagine you ask the clerk at a photocopy shop near your college for "notes on GST registration". The clerk searches the drawer for the word "GST", photocopies the first three pages that match and hands them over. Two pages are about GST rates, not registration. The clerk does not care. The job was to photocopy.

Now imagine a senior student helping you instead. She reads your request, thinks "this is about registration, not rates", checks the drawer, sees the pages are off-topic, rephrases the search as "GST registration steps and documents", and tries the government circular folder when the drawer has nothing useful. If nothing turns up, she tells you so instead of bluffing.

What is agentic RAG?

The photocopy clerk is classic RAG: one fixed retrieval step, then generation. The senior student is agentic RAG: she routes the question to a source, grades what she found, rewrites the query when results are weak and stops honestly when nothing fits.

Agentic RAG is a retrieval-augmented generation design in which an AI agent controls retrieval, deciding when to search, which source to use, whether the retrieved passages are relevant, and whether to retry with a better query before answering.

If plain RAG is new to you, start with what RAG is and come back. This lesson builds on the multi-agent system patterns and function calling from earlier in the series. All code targets Python 3.10+ and runs offline with a simulated model, the same convention used throughout this course.

The grader is the heart of agentic RAG: it decides between answering, retrying and giving up.

How is agentic RAG different from classic RAG?

Classic RAG is a straight pipeline. The question is embedded, the top passages are fetched, and the model writes an answer from whatever came back. It is cheap and predictable, and it fails silently when the retriever returns the wrong passages.

Agentic RAG wraps that pipeline in decisions. The table shows where they differ.

Same retriever, same model. The difference is who checks the passages before the answer is written.

The four decisions inside an agentic RAG loop

Every agentic RAG system, whatever framework it uses, makes some version of these four decisions.

1. Route. Does this question need retrieval at all, and from which source? "Hi" needs none. "What is our refund window?" needs the policy documents. "What is today's repo rate?" needs a fresh source.
2. Retrieve. Run the search with the current query.
3. Grade. Are the returned passages actually about the question? A grader can be a small model call, a reranker score or a keyword check.
4. Rewrite or answer. If the grade is poor, rewrite the query and retry, up to a limit. If it is good, generate the answer from the passing passages only. If retries run out, say "I could not find this".

The retry limit matters. Without it, a question that has no answer in your documents would loop forever, costing money on every pass.

Build an agentic RAG agent in Python

You will build it in three small pieces: a retriever, a grader with a router, and the full loop. The retriever uses word overlap instead of embeddings so the code runs anywhere, but the agent logic is identical to a production system. For the real retrieval layer, see vector databases and hybrid search with reranking.

Step 1: A tiny retriever with scores

DOCS = {
    "policy-1": "Refunds are allowed within 7 days of purchase if the seal is unbroken.",
    "policy-2": "Shipping inside Indore takes 1 day; other cities in Madhya Pradesh take 3 days.",
    "policy-3": "GST invoices are emailed within 24 hours of payment.",
    "faq-1": "Store timings are 10 am to 9 pm, closed on Tuesdays.",
}

def words(text):
    return {w.strip(".,?;:").lower() for w in text.split() if len(w) > 2}

def retrieve(query, k=2):
    q = words(query)
    scored = [(len(q & words(t)) / len(q), doc_id, t) for doc_id, t in DOCS.items()]
    return sorted(scored, reverse=True)[:k]

for score, doc_id, text in retrieve("Are refunds allowed after 7 days?"):
    print(f"{score:.2f}  {doc_id}: {text}")

The score is the share of query words found in the passage. A real system would use cosine similarity on embeddings, but either way you get a number the grader can reason about.

Step 2: A router and a relevance grader

In production, both of these are short LLM calls with a strict output format. Here they are rules that behave the same way, so you can see the decisions clearly.

def route(question):
    q = question.lower()
    if q.strip(" !?.") in {"hi", "hello", "thanks", "thank you"}:
        return "none"                       # small talk: skip retrieval
    if any(w in q for w in ["today", "latest", "current", "news"]):
        return "web"                        # needs fresh data
    return "docs"                           # default: company documents

def grade(question, passage, score):
    # Real systems ask a model: "Does this passage answer the question? yes/no"
    return "relevant" if score >= 0.4 else "irrelevant"

for q in ["Hi!", "What is the latest repo rate?", "When are GST invoices sent?"]:
    print(f"{q!r:35} -> route: {route(q)}")
print(grade("refund days", "Refunds are allowed within 7 days...", 0.5))
print(grade("refund days", "Store timings are 10 am to 9 pm...", 0.1))

Step 3: The full loop with query rewriting

Now combine them. The fake_llm_rewrite function plays the role of a model that rephrases a weak query, using a small synonym table. The loop stops after two rewrites.

DOCS = {
    "policy-1": "Refunds are allowed within 7 days of purchase if the seal is unbroken.",
    "policy-2": "Shipping inside Indore takes 1 day; other cities take 3 days.",
    "faq-1": "Store timings are 10 am to 9 pm, closed on Tuesdays.",
}
words = lambda t: {w.strip(".,?;:").lower() for w in t.split() if len(w) > 2}

def retrieve(query):
    q = words(query)
    return max((len(q & words(t)) / len(q), d, t) for d, t in DOCS.items())

def fake_llm_rewrite(query):               # a real model would rephrase freely
    swaps = {"money back": "refunds allowed", "open": "store timings", "delivery": "shipping takes"}
    for old, new in swaps.items():
        query = query.replace(old, new)
    return query

def agentic_rag(question, max_rewrites=2):
    query = question.lower()
    for attempt in range(max_rewrites + 1):
        score, doc_id, text = retrieve(query)
        print(f"  try {attempt}: {query!r} -> {doc_id} ({score:.2f})")
        if score >= 0.4:
            return f"{text} [source: {doc_id}]"
        new_query = fake_llm_rewrite(query)
        if new_query == query:              # nothing left to try
            break
        query = new_query
    return "I could not find this in the documents."

for q in ["Can I get my money back?", "When is the shop open?", "Do you sell gift cards?"]:
    print(q); print("  ->", agentic_rag(q))

Read the trace for each question. The first two fail on the original wording, get rewritten into the documents' vocabulary and succeed. The gift-card question has no answer anywhere.

When the rewrite produces the same query again, the agent stops early and says so instead of inventing an answer. That honest "not found" is the single most valuable behaviour agentic RAG adds.

Agentic RAG with a real LLM and vector store

In a production build, retrieval becomes a tool the model can call, and the model's own reasoning does the routing and rewriting. The example below uses the OpenAI Agents SDK (version 0.24 at the time of writing) with a hosted vector store. Check the Agents SDK tools documentation before you copy it, because hosted tool options change between releases.

from agents import Agent, Runner, FileSearchTool, function_tool

@function_tool
def search_orders_db(order_id: str) -> str:
    """Look up the live status of one order by its id, e.g. ORD-1042."""
    return lookup_order_status(order_id)       # your own database query

rag_agent = Agent(
    name="Support RAG agent",
    instructions=(
        "Answer only from tool results. Use file_search for policy questions and "
        "search_orders_db for order status. If the passages do not answer the "
        "question, rephrase the query and search once more. If it is still missing, "
        "say you could not find it. Cite the file name of every passage you use."
    ),
    tools=[
        FileSearchTool(vector_store_ids=["YOUR_VECTOR_STORE_ID"], max_num_results=4),
        search_orders_db,
    ],
)

result = Runner.run_sync(rag_agent, "Can I return a kettle I bought 5 days ago?", max_turns=6)
print(result.final_output)

Notice that the grading and rewriting rules moved into the instructions. That is simpler to write, but harder to test. Many teams keep an explicit grader step in code, like Step 2, because a separate yes or no check is easier to measure than free-form reasoning.

When is agentic RAG worth the extra cost?

Each decision is an extra model call, so a question that costs one call in classic RAG can cost four or five here. Before switching, check what your failures actually look like.

• Use agentic RAG when questions are varied, your knowledge lives in several sources, users ask multi-part questions, or a wrong answer is costly (policies, finance, medical information).
• Stay with classic RAG when questions are narrow and repetitive, one source covers everything, and latency must stay under about two seconds.
• Try cheaper fixes first: better chunking (see RAG chunking strategies), a reranker, or HyDE. These often close most of the gap without an agent loop.

A practical middle path is "corrective RAG": keep the single retrieval, add only the grader, and fall back to "not found" when the grade fails. That one extra call removes most confident wrong answers.

Common mistakes with agentic RAG

• No retry limit. Unanswerable questions then loop until the step cap or the budget runs out.
• Grading against the rewritten query. Always grade passages against the user's original question, or a drifting rewrite can "pass" off-topic text.
• Letting the model answer from memory. Instructions must say to answer only from retrieved passages, and an output check should confirm a source is cited.
• Too many sources at once. Five search tools with overlapping descriptions confuse the router. Start with two and add more only when a real question needs them.
• Measuring only the final answer. Track how often the grader fires, how many rewrites happen and how often the agent says "not found". The agent evaluation lesson shows how.

Key takeaways

• Agentic RAG lets an agent control retrieval: route, retrieve, grade, rewrite, answer.
• The grader and the honest "not found" fallback remove most confident wrong answers.
• Always cap retries and grade against the original question.
• It costs two to five model calls per question, so try better chunking and reranking first.
• In production, retrieval becomes a tool, and explicit grading code keeps the system testable.