AI Agent vs LLM vs RAG: Key Differences with Examples

Picture the enquiry desk at a busy coaching centre in Indore during admission season. Three people sit behind it.

The first is a senior student who has been around for years. Ask him anything and he answers instantly from memory. He is fluent, but his information is a year old and he has never seen this year's prospectus.

The second is the office assistant. Before answering, she pulls out the current prospectus, finds the right page and reads the answer off it. She is slower, but accurate, and can show you where the answer came from.

The third is the admissions coordinator. You tell her "I want to join the weekend batch." She checks the prospectus, opens the seat register, sees three seats left, books you a demo slot and sends you a confirmation. She did not just answer. She did the job.

What is the difference between an LLM, RAG and an AI agent?

The senior student is a plain LLM: everything he knows was learned before the day you met him. The assistant with the prospectus is RAG, retrieval-augmented generation: look up first, then answer. The coordinator is an AI agent: she decides which steps to take, uses tools such as the seat register, and keeps going until the goal is met.

An LLM generates an answer from its training data, a RAG system generates an answer from retrieved documents, and an AI agent uses an LLM in a loop to plan steps, call tools and act until a goal is reached.

All three use the same kind of engine underneath, a large language model. What changes is what gets wrapped around it. If you are new to agents, start with the overview of how AI agents work and come back here.

Same model inside, three different setups around it: answer, look up then answer, or loop and act.

How LLMs, RAG and agents each work

How does a plain LLM answer?

A plain LLM takes your prompt and predicts a reply one token at a time. Its knowledge comes from text it saw during training, which stopped on a fixed date called the knowledge cutoff.

That makes it excellent at language tasks that need no fresh facts: rewriting an email, explaining a concept or translating a paragraph into Hindi.

It also explains its two classic weaknesses. It does not know anything that happened after the cutoff, and it has never seen your private data, such as your company's leave policy or this week's batch timings. When asked about those, it may produce a fluent guess, which is called a hallucination.

How does RAG change the answer?

RAG adds a retrieval step before generation. Your question is used to search a collection of documents, usually with embeddings and a vector database. The best matching passages are pasted into the prompt, and the model answers from them.

The model is unchanged. You have simply handed it the right page before asking. That is why RAG is the standard fix for "the model doesn't know our data". You update the documents, not the model.

RAG is still a single pass, though. It retrieves once, answers once and stops. It cannot notice that the first search found the wrong document and try again, and it cannot book, send or update anything. The full mechanics are covered in what RAG is and how it works.

What does an AI agent add?

An agent puts the LLM in a loop. At each step the model looks at the goal and everything seen so far, then chooses: call a tool, or give the final answer. Your code runs the tool and feeds the result back in.

This gives an agent two abilities the other two lack. It can take several steps and change course based on what it finds. And it can act on the world through tools: query a database, call an API, write a file or send a message.

Retrieval often becomes one of those tools. An agent might search the documents, find nothing useful, rephrase the query and search again. That combination has its own name, agentic RAG.

See all three in Python

The simulation below sends one question to all three setups. No real model is called: small Python functions stand in for each system so you can see how the outputs differ. The "LLM" only has an old fact, the "RAG" function retrieves a document by word overlap, and the "agent" checks a seat register and books a slot.

# Made-up data for a fictional institute, "Example Academy". Not real timings.
OLD_TRAINING_FACT = "Example Academy GenAI batch: weekdays at 10 am (last year's timing)."
DOCS = [
    "Example Academy GenAI weekend batch: Saturday and Sunday, 11 am to 1 pm.",
    "Example Academy Data Analytics weekday batch: Monday to Friday, 6 pm.",
    "A free demo class is available before admission.",
]
seats = {"genai weekend": 3}

def retrieve(question, docs):
    words = set(question.lower().replace("?", "").split())
    return max(docs, key=lambda d: len(words & set(d.lower().replace(":", "").split())))

def llm_only(q): return OLD_TRAINING_FACT     # no lookup, no tools
def rag(q):      return "From the prospectus: " + retrieve(q, DOCS)

def agent(q):
    steps, timing = ["search_docs", "check_seats"], retrieve(q, DOCS)
    if seats["genai weekend"] == 0:             # the agent reacts to what it sees
        return "Batch is full, added you to the waitlist.", steps
    seats["genai weekend"] -= 1
    steps.append("book_demo")
    return f"{timing} Demo booked, {seats['genai weekend']} seats left.", steps

q = "When is the GenAI weekend batch and can I book a demo?"
print("LLM  :", llm_only(q))
print("RAG  :", rag(q))
answer, steps = agent(q)
print("Agent:", answer)
print("Agent used tools:", steps)

Set seats to {"genai weekend": 0} and run it again. The LLM and RAG answers do not change, but the agent takes a different path. Reacting to an observation is the defining trait of an agent.

AI agent vs LLM vs RAG: side-by-side comparison

The row that matters most is the action row. The moment a system can change something in the real world, you need permissions, logging and stopping rules, which is why guardrails for AI agents get a whole article in this series.

Why agents cost more per request

Every pass through an agent's loop is another model call, and each call re-reads the growing conversation. The next block estimates tokens for each approach, using made-up but realistic sizes.

# Rough token estimates per request (illustrative numbers)
question, answer = 40, 120
retrieved_chunks, chunk_size = 3, 200
agent_steps, tool_result = 4, 150

llm_tokens = question + answer
rag_tokens = question + retrieved_chunks * chunk_size + answer

agent_tokens, history = 0, question
for step in range(agent_steps):
    agent_tokens += history + 60          # re-read history, emit a tool call
    history += 60 + tool_result           # call plus its result join the history
agent_tokens += history + answer          # final answer

for name, t in [("LLM", llm_tokens), ("RAG", rag_tokens), ("Agent", agent_tokens)]:
    print(f"{name:6} {t:6} tokens  ~{t / llm_tokens:4.1f}x the plain LLM")

With these numbers the agent uses about 16 times the tokens of a single answer. The exact figure varies, but the shape is always the same: the history grows and each step pays to re-read it.

How to choose between an LLM, RAG and an agent

Walk through these questions in order and stop at the first "yes".

1. Does the task need actions or several dependent steps? For example, "find the invoice, check whether it is paid, and email the client if it isn't". If yes, build an agent.
2. Does the answer depend on private or recent information? For example, your product manual or today's prices. If yes, use RAG.
3. Otherwise, a well-written prompt to a plain LLM is enough, and it will be the cheapest and fastest option.

Start at the top and take the first branch that applies. Most real products mix all three.

The same rules fit in a few lines of code, handy for reviewing feature ideas with your team.

def choose(needs_action, needs_steps, needs_private_or_fresh_data):
    if needs_action or needs_steps:
        return "AI agent"
    if needs_private_or_fresh_data:
        return "RAG"
    return "Plain LLM"

ideas = {
    "Summarise a meeting transcript":         (False, False, False),
    "Answer HR policy questions":             (False, False, True),
    "Reschedule a class and notify students": (True, True, True),
    "Translate a notice into Hindi":          (False, False, False),
}
for idea, flags in ideas.items():
    print(f"{idea:42} -> {choose(*flags)}")

The options are not exclusive. A real student-support bot could use a plain LLM for small talk, RAG for policy questions and an agent only when the student asks to reschedule.

Common mistakes when comparing them

• Calling every chatbot an agent. If it answers once and never calls a tool, it is an LLM or a RAG app, however clever it sounds.
• Using an agent where RAG would do. If the user only needs an answer from documents, the loop adds cost and failure points without adding value.
• Expecting RAG to fix reasoning. Retrieval supplies facts. It does not make the model better at multi-step logic or calculations.
• Forgetting that agents inherit LLM weaknesses. An agent can still misread a tool result or hallucinate an argument, so its tools must validate inputs.
• Comparing on a single demo. Test each option on 20 to 50 real questions and compare accuracy, latency and cost.

Key takeaways

• An LLM answers from training data, RAG answers from retrieved documents, and an agent loops, uses tools and acts.
• RAG solves missing knowledge. Agents solve missing actions and multi-step work.
• Each step up costs more tokens, adds latency and needs more safety checks.
• Pick the simplest option that works, and mix them inside one product when needed.
• The next article breaks an agent into its building blocks, so you can design one deliberately.