Skip to content

11. Agentic RAG

1. Introduction - Agentic RAG in the Agentic AI Ecosystem

Retrieval-Augmented Generation (RAG) grounds language-model responses in external knowledge. Agentic RAG extends this interaction by allowing an LLM-powered agent to decide which capability to invoke, which knowledge source is appropriate, whether external information is required, and how to respond when the available evidence is insufficient.

Part V already explains Agentic RAG from the retrieval architecture perspective: retrieval planning, strategy selection, result evaluation, and iterative retrieval. This chapter focuses on the agent-engineering perspective covered by the Agentic AI course: structured tool use, external feedback, reasoning-and-action patterns, specialized agent roles, and LangGraph orchestration.

For retrieval planning and retrieval strategy design, see Part V --- Agentic RAG.

  • Part V: How the system plans, executes, and improves retrieval.
  • Part VII: How agents make decisions, invoke capabilities, exchange results, and coordinate the workflow.

2. From Generation to Tool-Augmented Responses

In a conventional RAG interaction, the application retrieves context and supplies it to the LLM to generate an answer. In an agentic workflow, the model can participate in deciding what to do next. It may select a knowledge source, invoke a search tool, inspect the returned information, and determine whether another action is needed.

The course illustrates source selection with two knowledge bases:

  • Internal knowledge: company policies, procedures, and guidelines.
  • General knowledge: industry standards, best practices, and public resources.

For example, "What is the company's policy on remote work during holidays?" should be directed to internal documentation. "What are common industry practices for remote work?" is better suited to the general knowledge source.

These examples illustrate agent decision-making and routing. The underlying retrieval algorithms and retrieval-strategy choices remain in Part V.

3. Structured Tool Calling

An agent needs a well-defined interface to interact with retrieval systems and external services. Tool calling allows the model to select a tool and provide arguments; the application validates those arguments, executes the operation, and returns the result.

A Pydantic schema can define the input contract for a knowledge-search capability:

from pydantic import BaseModel, Field
from typing import Literal

class KnowledgeSearchInput(BaseModel):
    query: str = Field(description="Question or search query")
    source: Literal["internal", "general"]

The course demonstrates Pydantic-style structured schemas and LangChain tool binding. A model can be given a schema so that its output is machine-readable rather than free-form text:

from pydantic import BaseModel, Field

class AnswerQuestion(BaseModel):
    answer: str = Field(description="Current answer")
    reflection: str = Field(description="What is missing or incorrect")
    search_query: str = Field(description="Focused query for more evidence")

structured_llm = llm.with_structured_output(AnswerQuestion)
draft = structured_llm.invoke(question)

4. Reflection and Reflexion with External Evidence

Reflection uses internal critique to improve a model-generated response. A generator produces an answer, and a reflector reviews it for missing details, relevance, or clarity. The workflow may repeat until a stopping condition is met. Because the feedback is generated by the model itself, reflection can improve coherence but does not independently verify factual accuracy.

Reflexion, as demonstrated in the course, adds external feedback. A responder creates an initial answer and identifies missing information or a focused search query. A search tool such as Tavily retrieves external results. A revisor then uses the original question, draft, reflection, and retrieved evidence to improve the answer and include references.

User Question
     |
     v
Responder
(answer + reflection + search query)
     |
     v
Search Tool (e.g. Tavily)
     |
     v
External Evidence
     |
     v
Revisor
(revised answer + references)
     |
     +----> Continue if evidence is still insufficient
     |
     v
Final Response

A simplified LangGraph structure makes the control flow explicit:

from typing import TypedDict
from langgraph.graph import StateGraph, START, END

class ReflexionState(TypedDict):
    question: str
    answer: str
    reflection: str
    search_query: str
    evidence: list
    iterations: int

def responder(state: ReflexionState):
    # Return structured answer, reflection, and search query.
    ...

def search(state: ReflexionState):
    # Execute search; retain title, URL, and content.
    ...

def revisor(state: ReflexionState):
    # Revise using the question and retrieved evidence.
    ...

builder = StateGraph(ReflexionState)
builder.add_node("responder", responder)
builder.add_node("search", search)
builder.add_node("revisor", revisor)
builder.add_edge(START, "responder")
builder.add_edge("responder", "search")
builder.add_edge("search", "revisor")
# Add conditional routing from revisor:
# continue to responder, or finish at END, with a maximum iteration count.
graph = builder.compile()

This is an orchestration skeleton, not a complete search application: node bodies, Tavily configuration, prompts, and the stopping function must be supplied by the application. The important pattern is the responder--tool--revisor feedback loop. A bounded iteration count prevents repeated searches from continuing indefinitely.

5. ReAct --- Reasoning Before Acting

ReAct combines reasoning with tool execution. Rather than answering in one pass, the agent identifies what information is needed, selects an action, receives an observation, and uses that observation to determine the next step.

The course presents the cycle as:

  1. Thought: identify the information or action required.
  2. Action: select a tool.
  3. Action Input: provide the tool arguments.
  4. Observation: receive the tool result.
  5. Final Answer: respond using the information gathered.

For Agentic RAG, an action might invoke an internal knowledge-search tool. Its observation is the returned evidence. The agent can then decide whether another permitted action is needed or whether it can answer with the available information.

LangGraph's prebuilt ReAct agent can provide this model/tool loop:

from langgraph.prebuilt import create_react_agent

# These tools are application-defined and enforce their own access rules.
agent = create_react_agent(
    model=llm,
    tools=[internal_search, general_search]
)

result = agent.invoke({
    "messages": [
        ("user", "What is our company's remote-work policy?")
    ]
})

ReAct is useful when the next action depends on the previous observation. Reflexion has a different emphasis: it revises a draft using critique and external evidence. Both patterns can use tools, but they address different workflow needs.

6. Multi-Agent Agentic RAG --- Retrieval, Reasoning, and Verification

A multi-agent workflow divides a complex task among agents with focused responsibilities. The course describes three complementary roles:

  • Retrieval agent: obtains relevant information from permitted knowledge sources.
  • Reasoning agent: interprets retrieved information and performs task-specific inference.
  • Verification agent: checks accuracy, consistency, and whether the candidate answer is supported by the evidence.

A coordinator can assign work and decide whether the result is ready or needs another step.

                 User Question
                       |
                       v
                Coordinator Agent
                       |
                       v
                 Retrieval Agent
                       |
                       v
                Retrieved Evidence
                       |
                       v
                 Reasoning Agent
                       |
                       v
                Candidate Answer
                       |
                       v
                Verification Agent
                       |
              Supported and complete?
                   /          \
                 No            Yes
                 |              |
          Return for work    Final Answer

The following compact LangGraph sketch represents the roles as separate nodes:

from typing import TypedDict
from langgraph.graph import StateGraph, START, END

class RAGState(TypedDict):
    question: str
    evidence: list
    candidate_answer: str
    verified: bool

def retrieve(state: RAGState): ...
def reason(state: RAGState): ...
def verify(state: RAGState): ...

builder = StateGraph(RAGState)
builder.add_node("retrieve", retrieve)
builder.add_node("reason", reason)
builder.add_node("verify", verify)
builder.add_edge(START, "retrieve")
builder.add_edge("retrieve", "reason")
builder.add_edge("reason", "verify")

def verification_route(state: RAGState):
    return "end" if state["verified"] else "reason"

builder.add_conditional_edges(
    "verify", verification_route,
    {"reason": "reason", "end": END}
)
multi_agent_rag = builder.compile()

This is an illustrative graph skeleton: the retrieval, reasoning, and verification functions and their model/tool configurations are application-specific. It demonstrates role separation and verification-controlled routing, not a complete production implementation.

7. Orchestrating Agentic Workflows with LangGraph

LangGraph represents an agent workflow as a graph:

  • State holds shared information such as the question, messages, retrieved evidence, draft answer, and iteration count.
  • Nodes perform work, such as responding, executing a tool, reasoning, or verifying.
  • Edges define the next step.
  • Conditional edges choose a path based on the current state, such as whether a tool call is required or verification passed.

A reliable workflow must define its stopping conditions. It should finish when the response is sufficiently supported, when the request is outside the available knowledge scope, or when a configured execution limit is reached. Reaching a limit should produce a controlled outcome rather than an endless loop or an unsupported answer.

8. Key Takeaways

  • Agentic RAG adds agent decisions and tool use to retrieval-grounded generation.
  • Agents can select between internal and general knowledge sources and route out-of-scope questions to a fallback.
  • Pydantic schemas and structured tool calls make model outputs and tool interfaces more predictable.
  • Reflection uses internal critique; Reflexion adds external evidence; ReAct alternates reasoning, tool actions, and observations.
  • Multi-agent workflows can separate retrieval, reasoning, and verification responsibilities.
  • LangGraph coordinates these patterns through shared state, nodes, edges, conditional routing, and bounded loops.
  • Part V remains the reference for retrieval planning, retrieval strategies, and the retrieval decision loop.

9. What's Next?

Chapter 12 --- Agent Communication & Integration Protocols --- moves from an individual agent's tool interactions to how agents and systems exchange messages, capabilities, and task context.

References


🧭 Chapter Navigation

⬅️ Previous: 10-Agentic Workflow Evaluation & Reliability

📚 Part: Agentic AI: Multi-Agent Systems Architecture, Orchestration, Collaboration

➡️ Next: 12-Agent Communication & Integration Protocols


Enterprise AI Engineering Handbook
Building Production-Grade Enterprise AI Systems — One Chapter at a Time.