06. Agentic RAG¶
Category: Advanced RAG Architecture
Module: Part V — Advanced Retrieval-Augmented Generation
Difficulty: Advanced
📖 Overview¶
Traditional RAG follows a relatively fixed pipeline:
This architecture works well for straightforward questions, but enterprise queries are often more complex.
A single question may require:
Multiple retrieval strategies
Multiple search iterations
Query rewriting
SQL queries
Knowledge Graph traversal
Vector search
Document inspection
Validation
Additional retrieval
For example:
"Which customers were affected by the payment
incident, which services were involved, and what
was the root cause?"
A fixed RAG pipeline may not know in advance whether it needs:
Agentic RAG introduces an orchestration layer capable of reasoning about the retrieval process, selecting tools, executing retrieval steps, evaluating results, and deciding whether additional retrieval is necessary.
The architecture becomes:
User Query
↓
Agent / Planner
↓
Decide What Information Is Needed
↓
Select Retrieval Tool
↓
Retrieve
↓
Evaluate Evidence
↓
Need More Information?
│
┌┴───────┐
Yes No
│ │
▼ ▼
Retrieve Generate
Again Answer
The important idea is:
Agentic RAG makes retrieval an adaptive process rather than a single fixed retrieval step.
🎯 Learning Objectives¶
After completing this chapter, you will be able to:
- Understand Agentic RAG
- Understand the difference between traditional RAG and Agentic RAG
- Understand agentic retrieval
- Understand retrieval planning
- Understand query decomposition
- Understand iterative retrieval
- Understand tool-based retrieval
- Understand retrieval routing
- Build SQL retrieval tools
- Build Vector retrieval tools
- Build Knowledge Graph retrieval tools
- Combine multiple retrieval mechanisms
- Understand agent state
- Understand retrieval loops
- Implement bounded agent execution
- Understand reflection and self-evaluation
- Understand evidence verification
- Understand adaptive query rewriting
- Understand multi-hop retrieval
- Design Agentic RAG workflows
- Understand deterministic vs agentic orchestration
- Implement guardrails
- Control agent cost and latency
- Implement observability
- Evaluate Agentic RAG systems
- Design production-grade Agentic RAG architectures
🧠 1. What Is Agentic RAG?¶
Agentic RAG combines:
The agent can decide:
What should I retrieve?
Which retriever should I use?
Do I need another query?
Is the evidence sufficient?
Should I verify the answer?
Instead of:
we have:
🔎 2. Traditional RAG vs Agentic RAG¶
Traditional RAG¶
The retrieval path is mostly predetermined.
Agentic RAG¶
Question
↓
Agent
↓
Plan
↓
Select Tool
↓
Retrieve
↓
Evaluate
↓
Re-plan
↓
Retrieve Again
↓
Validate
↓
Answer
The retrieval path is adaptive.
📊 3. Comparison¶
| Capability | Traditional RAG | Agentic RAG |
|---|---|---|
| Single retrieval | ✅ | ✅ |
| Query rewriting | Optional | Dynamic |
| Multiple retrieval steps | Limited | ✅ |
| Tool selection | Fixed | Dynamic |
| SQL retrieval | Possible | Dynamic |
| Graph retrieval | Possible | Dynamic |
| Vector retrieval | ✅ | ✅ |
| Query decomposition | Limited | ✅ |
| Iterative retrieval | Limited | ✅ |
| Evidence evaluation | Basic | Dynamic |
| Planning | Limited | ✅ |
| Adaptive routing | Limited | ✅ |
| Complex multi-hop questions | Limited | Strong |
| Cost predictability | Higher | Lower |
| Operational complexity | Lower | Higher |
🧩 4. Why Agentic RAG?¶
Consider:
"Which customers affected by the payment incident
were using the services that experienced the highest
error rate, and what remediation was implemented?"
This may require:
Step 1:
Find payment incident.
Step 2:
Find affected services.
Step 3:
Query transaction database.
Step 4:
Identify affected customers.
Step 5:
Traverse service dependencies.
Step 6:
Retrieve remediation documentation.
Step 7:
Combine evidence.
A fixed retrieval pipeline may not know this sequence in advance.
An agent can construct it dynamically.
🧠 5. Agentic RAG Mental Model¶
USER QUERY
│
▼
AGENT
│
PLAN / DECIDE
│
┌───────────────────┼───────────────────┐
▼ ▼ ▼
Vector Search SQL Graph
│ │ │
▼ ▼ ▼
Evidence Evidence Evidence
│ │ │
└───────────────────┼───────────────────┘
▼
EVIDENCE EVALUATION
│
┌─────────┴─────────┐
▼ ▼
Sufficient Insufficient
│ │
▼ ▼
Answer Re-plan
│
▼
More Retrieval
🏗️ 6. Basic Agentic RAG Architecture¶
flowchart TD
A["User Query"] --> B["Agent"]
B --> C["Planner"]
C --> D{"Select Tool"}
D --> E["Vector Retriever"]
D --> F["SQL Retriever"]
D --> G["Graph Retriever"]
E --> H["Evidence"]
F --> H
G --> H
H --> I["Evidence Evaluator"]
I --> J{"Enough Evidence?"}
J -->|No| C
J -->|Yes| K["Answer Generator"]
K --> L["Response"]🧠 7. Agent vs RAG Pipeline¶
A traditional RAG pipeline might be:
def rag(query):
documents = retriever.retrieve(query)
context = build_context(documents)
return llm.generate(context, query)
An Agentic RAG system is conceptually closer to:
def agentic_rag(query):
state = initialize_state(query)
while not state.completed:
action = planner.decide(state)
result = execute(action)
state = update_state(state, result)
return generate_answer(state)
The agent controls the retrieval process.
🧩 8. Agentic Retrieval¶
Agentic retrieval means the system can dynamically decide:
Example:
🔀 9. Retrieval Tools¶
An Agentic RAG system may expose:
Example:
The agent chooses which tool to use.
🧠 10. Tool-Based Retrieval¶
Each tool should have a clear contract.
Example:
SQL:
Graph:
class GraphSearchTool:
def search(
self,
entity: str,
relationship: str
):
raise NotImplementedError
🏛️ 11. Capability-Based Retrieval Tools¶
A production architecture should expose capabilities instead of infrastructure-specific implementations.
Agent
│
├── DocumentSearch
├── VectorSearch
├── SQLQuery
├── GraphQuery
├── ImageSearch
└── MetadataSearch
The underlying implementation can change independently.
🧠 12. Tool Descriptions¶
The agent needs to understand when a tool should be used.
Example:
Vector Search:
Search semantic enterprise documents.
SQL:
Retrieve exact structured business data.
Knowledge Graph:
Traverse entity relationships.
Image Search:
Retrieve visual enterprise evidence.
Clear tool descriptions improve tool selection.
🔎 13. Query Classification¶
Before acting, the agent can classify the question.
Possible categories:
🧠 14. Query Decomposition¶
Complex queries can be decomposed into sub-questions.
Example:
Decompose:
🔄 15. Query Decomposition Pipeline¶
flowchart LR
A["Complex Query"] --> B["Query Decomposer"]
B --> C["Sub-question 1"]
B --> D["Sub-question 2"]
B --> E["Sub-question 3"]
C --> F["Retriever"]
D --> G["Retriever"]
E --> H["Retriever"]
F --> I["Evidence"]
G --> I
H --> I
I --> J["Synthesis"]🧩 16. Parallel Retrieval¶
Independent sub-questions can be executed in parallel.
Complex Query
│
┌─────────┼─────────┐
▼ ▼ ▼
Q1 Q2 Q3
│ │ │
▼ ▼ ▼
Vector SQL Graph
│ │ │
└─────────┼─────────┘
▼
Synthesis
This can reduce latency.
⚡ 17. Parallel vs Sequential Retrieval¶
Not all tasks can be parallelized.
Example:
Q2 depends on Q1.
Therefore:
Sequential execution is required.
🧠 18. Dependency-Aware Planning¶
The agent can construct a graph:
This is effectively a retrieval execution plan.
🏗️ 19. Retrieval Plan¶
Example:
{
"steps": [
{
"id": "step-1",
"tool": "vector_search",
"query": "payment incident"
},
{
"id": "step-2",
"tool": "sql",
"depends_on": ["step-1"]
},
{
"id": "step-3",
"tool": "graph",
"depends_on": ["step-1"]
}
]
}
This gives the orchestration layer explicit control.
🧠 20. Agent State¶
Agentic systems need state.
Example:
{
"original_query": "...",
"sub_questions": [],
"retrieved_documents": [],
"sql_results": [],
"graph_results": [],
"observations": [],
"tool_calls": [],
"confidence": 0.0,
"iteration": 0,
"status": "running"
}
The state becomes the working memory of the retrieval workflow.
🔄 21. Agent State Machine¶
stateDiagram-v2
[*] --> Analyze
Analyze --> Plan
Plan --> Retrieve
Retrieve --> Evaluate
Evaluate --> Plan: More Evidence Needed
Evaluate --> Generate: Evidence Sufficient
Generate --> Validate
Validate --> Retrieve: Validation Failed
Validate --> Complete: Validation Passed
Complete --> [*]🧠 22. Observation → Decision → Action¶
A useful Agentic RAG model is:
Example:
Observation:
Search results mention incident INC-123.
Decision:
Need affected customers.
Action:
Query SQL database.
Observation:
10,423 customers affected.
Decision:
Need remediation documentation.
Action:
Vector search.
Observation:
Runbook found.
Decision:
Evidence sufficient.
Action:
Generate answer.
🔎 23. Iterative Retrieval¶
Agentic RAG can perform multiple retrieval rounds.
Round 1
↓
Initial Evidence
Round 2
↓
Refined Query
Round 3
↓
Additional Evidence
Round 4
↓
Validation
Answer
This can improve complex-query performance.
⚠️ 24. Retrieval Loops Need Limits¶
An agent should never be allowed to retrieve indefinitely.
Use:
Example:
🛡️ 25. Bounded Agent Execution¶
If evidence remains insufficient:
rather than continuing indefinitely.
🧠 26. Evidence Evaluation¶
The agent should evaluate:
Example:
🔎 27. Evidence Sufficiency¶
A simple decision model:
Evidence
│
├── Relevant?
│
├── Complete?
│
├── Authoritative?
│
├── Current?
│
└── Consistent?
│
▼
Sufficient?
If not:
🧠 28. Reflection¶
Reflection means the system evaluates its own intermediate result.
Example:
Reflection should be treated as a controlled verification mechanism rather than unrestricted self-reasoning.
🔄 29. Retrieval Reflection Loop¶
flowchart TD
A["Query"] --> B["Retrieve"]
B --> C["Evidence"]
C --> D["Draft Answer"]
D --> E["Evaluate"]
E --> F{"Grounded?"}
F -->|Yes| G["Final Answer"]
F -->|No| H["Identify Missing Evidence"]
H --> I["Refine Query"]
I --> B🧩 30. Query Rewriting¶
The agent can rewrite a query based on retrieved evidence.
Example:
Original:
"Payment issue"
Retrieved:
Incident INC-1042
Rewritten:
"INC-1042 root cause and remediation"
This can improve subsequent retrieval.
🧠 31. Adaptive Query Expansion¶
Initial query:
Retrieved entities:
Next query:
The agent uses retrieved evidence to improve the next search.
🔎 32. Multi-Hop Retrieval¶
Multi-hop retrieval requires several connected retrieval steps.
Example:
Question:
This is difficult for a single retrieval step.
🧠 33. Multi-Hop RAG¶
flowchart LR
A["Question"] --> B["Customer"]
B --> C["Product"]
C --> D["Service"]
D --> E["Incident"]
E --> F["Evidence"]
F --> G["Answer"]Agentic orchestration can dynamically perform each hop.
🏢 34. Enterprise Multi-Hop Example¶
Question:
"Which banking customers were impacted by
the authentication incident and what was
the business impact?"
Potential workflow:
Step 1:
Vector → Find authentication incident.
Step 2:
Graph → Find affected authentication service.
Step 3:
SQL → Find affected customers.
Step 4:
SQL → Calculate transaction impact.
Step 5:
Vector → Find incident postmortem.
Step 6:
Synthesize.
🔀 35. SQL + Graph + Vector Agent¶
flowchart TD
A["User Query"] --> B["Agent Planner"]
B --> C["Vector Search"]
B --> D["SQL Query"]
B --> E["Graph Query"]
C --> F["Document Evidence"]
D --> G["Structured Evidence"]
E --> H["Relationship Evidence"]
F --> I["Evidence Store"]
G --> I
H --> I
I --> J["Agent Evaluator"]
J --> K{"Sufficient?"}
K -->|No| B
K -->|Yes| L["Answer Generator"]
L --> M["Validation"]
M --> N["Response"]🧠 36. Agentic RAG and Multimodal RAG¶
Agents can also select visual tools.
Example:
Question:
"Which service is shown communicating
with PostgreSQL in the architecture diagram?"
Agent:
↓
Image Search
↓
Architecture Diagram
↓
Vision Analysis
↓
Knowledge Graph Verification
↓
Answer
🧩 37. Agentic RAG Tool Registry¶
A production system may maintain:
class ToolRegistry:
def register(self, tool):
...
def get(self, name):
...
def list_tools(self):
...
Example:
The agent can access only the tools assigned to its role.
🔐 38. Tool Authorization¶
Not every agent should access every tool.
Example:
Research Agent
├── Vector Search
├── Graph Search
└── Document Search
Analytics Agent
├── SQL
└── Metadata Search
Enterprise Agent
├── Vector
├── SQL
├── Graph
└── Document
Tool access should be governed by authorization.
🛡️ 39. Tool-Level Security¶
Each tool should enforce:
Do not rely on the LLM to enforce security policies.
🚨 40. Agent Prompt Injection¶
Retrieved documents can contain malicious instructions.
Example:
This is a retrieval-time prompt injection risk.
The agent must distinguish:
from:
🧠 41. Retrieval Content Is Data, Not Authority¶
A critical security principle:
not:
The agent should never automatically execute instructions found inside retrieved content.
🛡️ 42. Tool Output Validation¶
Tool output should be treated as untrusted data.
This prevents malformed or unexpected tool output from directly influencing execution.
🧩 43. Agent Guardrails¶
Guardrails can enforce:
Allowed Tools
Allowed Domains
Allowed Databases
Allowed Tables
Maximum Iterations
Maximum Query Cost
Maximum Result Size
Maximum Token Usage
🧠 44. Deterministic vs Agentic Orchestration¶
Not every workflow needs an agent.
Deterministic¶
Advantages:
Agentic¶
Advantages:
📊 45. When to Use Agentic RAG¶
Use Agentic RAG when:
Queries require multiple retrieval steps
Queries require tool selection
Queries require multi-hop reasoning
Evidence is incomplete
Multiple knowledge sources are required
The retrieval path cannot be predetermined
🚫 46. When Not to Use Agentic RAG¶
Avoid agents for simple queries such as:
If:
works reliably, adding an agent may only increase:
🧠 47. Agentic RAG Decision Framework¶
Query
│
▼
Is it straightforward?
/ \
Yes No
│ │
▼ ▼
Traditional RAG Need multiple steps?
/ \
No Yes
│ │
▼ ▼
Advanced RAG Agentic RAG
The objective is not to use agents everywhere.
The objective is to use agents where adaptive orchestration provides measurable value.
🏗️ 48. Agentic RAG Architecture Layers¶
┌─────────────────────────────────────┐
│ User Interface │
├─────────────────────────────────────┤
│ Query Understanding │
├─────────────────────────────────────┤
│ Agent / Planner │
├─────────────────────────────────────┤
│ Tool Orchestration │
├─────────────────────────────────────┤
│ Retrieval Capabilities │
├─────────────────────────────────────┤
│ Vector │ SQL │ Graph │ Multimodal │
├─────────────────────────────────────┤
│ Evidence / State Store │
├─────────────────────────────────────┤
│ Validation / Guardrails │
├─────────────────────────────────────┤
│ Observability / Audit │
└─────────────────────────────────────┘
🧠 49. Agent State Model¶
A production state might look like:
from dataclasses import dataclass, field
@dataclass
class AgentState:
query: str
plan: list = field(default_factory=list)
observations: list = field(default_factory=list)
evidence: list = field(default_factory=list)
tool_calls: list = field(default_factory=list)
iteration: int = 0
status: str = "RUNNING"
confidence: float = 0.0
🔄 50. Agent Execution Loop¶
Conceptually:
def run_agent(state):
while state.status == "RUNNING":
if state.iteration >= MAX_ITERATIONS:
state.status = "STOPPED"
break
action = planner.plan(state)
observation = execute(action)
state.observations.append(observation)
state = evaluator.update(state)
state.iteration += 1
return state
Production implementations require stronger error handling, authorization, timeout, and state management.
🧩 51. Agent Actions¶
Possible actions:
SEARCH_DOCUMENTS
SEARCH_IMAGES
QUERY_SQL
QUERY_GRAPH
REWRITE_QUERY
VERIFY_EVIDENCE
GENERATE_ANSWER
ASK_CLARIFICATION
STOP
Representing actions explicitly makes orchestration easier to observe and test.
🧠 52. Action Model¶
{
"action": "QUERY_SQL",
"reason": "Need exact transaction count",
"parameters": {
"metric": "failed_transactions",
"date_range": "2026-08-10"
}
}
A structured action can be validated before execution.
🔐 53. Structured Tool Calls¶
Prefer:
over allowing the model to produce arbitrary executable commands.
The orchestration layer can translate the structured request into a controlled operation.
🧠 54. Agent Memory¶
Agentic RAG may maintain temporary state such as:
Current Query
Retrieved Evidence
Previous Searches
Intermediate Findings
Tool Results
Open Questions
This is different from long-term user memory.
For most RAG tasks, short-lived execution state is sufficient.
🗃️ 55. Evidence Store¶
A useful internal evidence store might contain:
{
"evidence_id": "ev-102",
"source_type": "vector",
"source_id": "doc-912",
"content": "...",
"relevance": 0.91,
"timestamp": "2026-08-11T10:32:00"
}
Other evidence:
{
"evidence_id": "ev-103",
"source_type": "sql",
"source_id": "analytics-db",
"query_id": "q-819",
"result": {
"count": 12450
}
}
🧠 56. Evidence Graph¶
Evidence can also be represented as relationships:
Question
│
├── Evidence A
│ │
│ └── supports → Claim 1
│
├── Evidence B
│ │
│ └── supports → Claim 2
│
└── Evidence C
│
└── contradicts → Claim 3
This can support sophisticated response validation.
🔎 57. Evidence Deduplication¶
Multiple tools may retrieve the same evidence.
Example:
Deduplicate evidence before final context construction.
This reduces:
🧠 58. Evidence Ranking¶
Rank evidence using:
Example:
The exact scoring strategy should be evaluated empirically.
🔄 59. Evidence Fusion¶
flowchart TD
A["Vector Evidence"] --> D["Evidence Store"]
B["SQL Evidence"] --> D
C["Graph Evidence"] --> D
E["Image Evidence"] --> D
D --> F["Deduplication"]
F --> G["Ranking"]
G --> H["Conflict Detection"]
H --> I["Evidence Fusion"]
I --> J["Context Builder"]⚠️ 60. Conflicting Evidence¶
Example:
Vector:
Payment Service uses PostgreSQL.
Graph:
Payment Service uses PostgreSQL.
SQL:
Current database = MySQL.
The agent should not blindly merge these facts.
It should consider:
and potentially retrieve more evidence.
🧠 61. Confidence-Aware Retrieval¶
A useful approach:
High Confidence
↓
Answer
Medium Confidence
↓
Verify
Low Confidence
↓
Retrieve More
No Evidence
↓
Ask Clarification / Abstain
🛑 62. Agentic Abstention¶
A production agent should be able to say:
This is preferable to:
🧠 63. Clarification¶
Some queries cannot be safely resolved.
Example:
The system may need:
The agent can ask a clarification question instead of making assumptions.
🔄 64. Clarification Loop¶
flowchart TD
A["User Query"] --> B["Agent"]
B --> C{"Ambiguous?"}
C -->|Yes| D["Ask Clarification"]
D --> E["User"]
E --> B
C -->|No| F["Plan Retrieval"]
F --> G["Execute"]
G --> H["Answer"]🧩 65. Agentic RAG with SQL¶
Example question:
Workflow:
Vector Search
↓
Find Incident ID
↓
Extract Incident Time Range
↓
SQL
↓
Count Failed Payments
↓
Validate
↓
Answer
🧩 66. Agentic RAG with Graph¶
Question:
Workflow:
Graph
↓
Find Authentication Service
↓
Traverse DEPENDS_ON
↓
Find Affected Services
↓
Vector Search
↓
Find Outage Evidence
↓
Answer
🧩 67. Agentic RAG with Multimodal Retrieval¶
Question:
Workflow:
Image Search
↓
Architecture Diagram
↓
Vision Analysis
↓
Identify Service + Database
↓
Graph Verification
↓
Answer
🏢 68. Enterprise Agentic RAG¶
A mature enterprise agent may have access to:
Architecture:
flowchart TD
A["Enterprise User"] --> B["AI Gateway"]
B --> C["Agent Orchestrator"]
C --> D["Planner"]
D --> E["Vector"]
D --> F["SQL"]
D --> G["Graph"]
D --> H["Multimodal"]
D --> I["Enterprise APIs"]
E --> J["Evidence"]
F --> J
G --> J
H --> J
I --> J
J --> K["Evidence Evaluator"]
K --> L{"Sufficient?"}
L -->|No| D
L -->|Yes| M["Response Generator"]
M --> N["Validation"]
N --> O["Citation"]
O --> P["Enterprise Response"]🧠 69. Agentic RAG vs Agentic AI¶
These terms are related but not identical.
Agentic RAG¶
The agent primarily focuses on:
General Agentic AI¶
An agent may:
Agentic RAG should generally maintain a narrower scope when the goal is knowledge retrieval.
🛡️ 70. Read vs Write Tools¶
A critical production distinction:
versus:
Agentic RAG systems should normally prioritize read-only tools.
Write capabilities require significantly stronger authorization and approval mechanisms.
🔐 71. Human Approval¶
For high-impact operations:
Even though Agentic RAG is primarily retrieval-oriented, enterprise systems may eventually connect agents to operational tools.
🧠 72. Agent Planning Strategies¶
Possible approaches include:
Single-Step Planning
ReAct-Style Loop
Plan-and-Execute
Query Decomposition
Graph-Based Planning
Workflow-Based Planning
The right approach depends on:
🔄 73. ReAct-Style Agentic Retrieval¶
Conceptually:
In production systems, internal reasoning should not be exposed to users.
The observable interface should focus on:
🧠 74. Plan-and-Execute¶
Another approach:
Example:
1. Find incident.
2. Find affected services.
3. Query transaction impact.
4. Find remediation.
5. Synthesize.
This can provide more predictable execution than completely free-form agent loops.
⚖️ 75. Plan-and-Execute vs ReAct¶
| Characteristic | ReAct-Style | Plan-and-Execute |
|---|---|---|
| Dynamic adaptation | High | Medium |
| Predictability | Medium | High |
| Planning upfront | Low | High |
| Tool flexibility | High | High |
| Debugging | Medium | Easier |
| Cost control | Harder | Easier |
| Complex dependencies | Strong | Strong |
A hybrid architecture can combine both.
🏗️ 76. Hybrid Agent Architecture¶
This balances:
🧠 77. Agent Failure Modes¶
Common failures:
Wrong Tool Selection
Wrong Query
Retrieval Loop
Tool Overuse
Missing Evidence
Conflicting Evidence
Incorrect Planning
Hallucinated Tool Results
Prompt Injection
Unauthorized Tool Access
Excessive Cost
Excessive Latency
Incorrect Final Synthesis
🚨 78. Tool Selection Failure¶
Question:
The agent chooses:
instead of:
This produces a poor answer.
Tool selection should therefore be evaluated independently.
🚨 79. Retrieval Loop Failure¶
Example:
Use:
🚨 80. Tool Overuse¶
A simple query:
should not trigger:
if one document retrieval is enough.
Over-orchestration increases cost and latency.
🧠 81. Agent Cost Model¶
Agentic RAG cost can include:
A simple conceptual model:
⚡ 82. Latency Model¶
Similarly:
Parallel execution can reduce latency when dependencies allow it.
🧩 83. Agent Budget¶
A production agent can have:
When the budget is exhausted:
or:
depending on the confidence level.
🧠 84. Agent Observability¶
Every agent execution should be traceable.
Track:
Trace ID
Session ID
User
Tenant
Original Query
Plan
Tool Calls
Tool Inputs
Tool Results
Iterations
Latency
Token Usage
Cost
Errors
Evidence
Final Answer
Sensitive information should be redacted according to organizational policy.
📊 85. Agent Trace¶
Example:
Trace: agent-8291
00ms Query received
120ms Planner
→ vector_search
350ms Vector result
→ Incident INC-1042
410ms Planner
→ sql_query
720ms SQL result
→ 12,430 affected customers
780ms Planner
→ vector_search
1040ms Retrieved remediation document
1100ms Evidence validation
1300ms Answer generated
This trace makes production debugging significantly easier.
🧠 86. Agent Evaluation¶
Agentic RAG should be evaluated at multiple levels.
Retrieval¶
Tool Selection¶
Planning¶
Execution¶
Grounding¶
Operations¶
📊 87. Agent Evaluation Matrix¶
| Metric | Purpose |
|---|---|
| Retrieval Recall | Evidence discovery |
| Tool Selection Accuracy | Correct capability selection |
| Plan Success Rate | Planning quality |
| Task Success Rate | End-to-end success |
| Evidence Sufficiency | Retrieval completeness |
| Groundedness | Answer support |
| Citation Accuracy | Source correctness |
| Iterations | Agent efficiency |
| Tool Calls | Orchestration efficiency |
| Latency | User experience |
| Cost | Operational efficiency |
🧪 88. Agentic RAG Evaluation Dataset¶
Include:
Simple Queries
Multi-Hop Queries
Multi-Source Queries
Ambiguous Queries
SQL Questions
Graph Questions
Visual Questions
Incomplete Evidence
Conflicting Evidence
Security Attacks
Prompt Injection
Tool Failures
🔎 89. Regression Testing¶
Every production change should be evaluated against a regression set.
Example:
Example:
{
"query": "What caused the payment outage?",
"expected_tools": [
"vector_search"
],
"max_tool_calls": 3
}
🧠 90. Deterministic Evaluation¶
Where possible, evaluate deterministic artifacts:
rather than relying only on an LLM judge.
LLM-based evaluation can supplement deterministic evaluation but should not be the only measurement mechanism.
🧩 91. Agentic RAG Security Architecture¶
flowchart TD
A["User"] --> B["Authentication"]
B --> C["Authorization"]
C --> D["Agent Gateway"]
D --> E["Policy Engine"]
E --> F["Agent"]
F --> G["Tool Authorization"]
G --> H["Tool"]
H --> I["Tool Validation"]
I --> J["Execution"]
J --> K["Audit Log"]Security should surround the agent rather than depend on the agent.
🛡️ 92. Guardrail Layers¶
A robust architecture can apply guardrails at multiple points:
Examples:
Input:
Prompt Injection Detection
Agent:
Allowed Tool Set
Tool:
Authorization
Data:
Tenant Isolation
Output:
Grounding / PII Validation
🧠 93. Production Agentic RAG Architecture¶
flowchart TD
A["User"] --> B["AI Gateway"]
B --> C["Authentication"]
C --> D["Authorization"]
D --> E["Agent Orchestrator"]
E --> F["Planner"]
F --> G["Policy Engine"]
G --> H["Tool Router"]
H --> I["Vector Search"]
H --> J["SQL"]
H --> K["Knowledge Graph"]
H --> L["Multimodal Search"]
I --> M["Evidence Store"]
J --> M
K --> M
L --> M
M --> N["Evidence Evaluator"]
N --> O{"Sufficient?"}
O -->|No| F
O -->|Yes| P["Context Builder"]
P --> Q["Foundation Model"]
Q --> R["Response Validator"]
R --> S["Citation"]
S --> T["Enterprise Response"]
E --> U["Observability"]
H --> U
N --> U
Q --> U🧠 94. Enterprise Agentic RAG Reference Architecture¶
┌───────────────┐
│ USER │
└───────┬───────┘
│
▼
┌───────────────┐
│ AI GATEWAY │
└───────┬───────┘
│
▼
┌───────────────┐
│ AUTHORIZATION │
└───────┬───────┘
│
▼
┌───────────────────┐
│ AGENT ORCHESTRATOR│
└─────────┬─────────┘
│
┌──────────┴──────────┐
▼ ▼
┌─────────┐ ┌──────────┐
│ PLANNER │ │ STATE │
└────┬────┘ └────┬─────┘
│ │
└─────────┬──────────┘
▼
┌────────────────┐
│ TOOL ROUTER │
└───────┬────────┘
│
┌──────────────────────┼────────────────────────┐
▼ ▼ ▼
┌─────────────┐ ┌────────────┐ ┌─────────────┐
│ Vector RAG │ │ SQL / Data │ │ Graph RAG │
└──────┬──────┘ └─────┬──────┘ └──────┬──────┘
│ │ │
└──────────────────────┼────────────────────────┘
▼
┌─────────────────┐
│ EVIDENCE STORE │
└────────┬────────┘
│
▼
┌─────────────────┐
│ EVIDENCE │
│ EVALUATOR │
└────────┬────────┘
│
┌──────┴──────┐
│ │
More Enough
│ │
▼ ▼
Re-plan Context
│
▼
┌──────────────┐
│ FOUNDATION │
│ MODEL │
└──────┬───────┘
│
▼
┌──────────────┐
│ VALIDATION │
└──────┬───────┘
│
▼
┌──────────────┐
│ CITATIONS │
└──────┬───────┘
│
▼
┌──────────────┐
│ RESPONSE │
└──────────────┘
┌───────────────────────────────────────────┐
│ OBSERVABILITY / AUDIT / COST │
└───────────────────────────────────────────┘
🧪 95. Practical Project¶
Build an Agentic RAG assistant for an enterprise knowledge base.
Data Sources¶
Architecture Documents
Incident Reports
Product Documentation
Customer Data
Service Dependency Graph
Tools¶
Example Query¶
"Which customers were affected by the latest
payment incident, what services were involved,
and what remediation was implemented?"
🔄 96. Expected Agent Workflow¶
User Query
↓
Planner
↓
Vector Search
↓
Incident Found
↓
Graph Search
↓
Affected Services
↓
SQL
↓
Affected Customers
↓
Vector Search
↓
Remediation
↓
Evidence Evaluation
↓
Context Assembly
↓
LLM
↓
Validation
↓
Citations
↓
Response
🧪 97. Advanced Project Extensions¶
Add:
Query Decomposition
Parallel Retrieval
Adaptive Query Rewriting
Evidence Ranking
Conflict Detection
Multimodal Retrieval
SQL Validation
Graph Traversal
Agent Memory
Human Approval
Then measure:
🚨 98. Common Mistakes¶
Mistake 1 — Using Agents Everywhere¶
Simple RAG queries do not need an agent.
Mistake 2 — Unlimited Loops¶
Always enforce execution budgets.
Mistake 3 — Giving the Agent Every Tool¶
Use capability-based authorization.
Mistake 4 — Allowing Arbitrary SQL¶
Use structured query plans and SQL validation.
Mistake 5 — Trusting Retrieved Instructions¶
Retrieved content is data, not authority.
Mistake 6 — No Evidence Validation¶
The agent should verify that retrieved information actually supports the answer.
Mistake 7 — No Abstention¶
Agents must be able to stop when evidence is insufficient.
Mistake 8 — Ignoring Cost¶
Multiple model and tool calls can make Agentic RAG expensive.
Mistake 9 — Ignoring Observability¶
Without traces, debugging agent behavior becomes difficult.
Mistake 10 — Exposing Internal Reasoning¶
Production systems should expose useful execution metadata rather than private chain-of-thought.
🧠 99. Design Principles¶
Principle 1 — Use Agents for Complexity¶
Principle 2 — Retrieval Should Be Adaptive¶
The agent should be able to change retrieval strategy based on evidence.
Principle 3 — Tools Should Be Capability-Based¶
Expose:
rather than infrastructure-specific implementation details.
Principle 4 — Keep Tools Controlled¶
Every tool should have:
Principle 5 — Bound Everything¶
Control:
Principle 6 — Evidence Before Answer¶
Principle 7 — Support Abstention¶
If evidence is insufficient:
Principle 8 — Preserve Provenance¶
Every claim should be traceable to evidence.
Principle 9 — Keep Security Outside the LLM¶
The model should never be the final authorization layer.
Principle 10 — Measure the Agent¶
Monitor:
📋 100. Production Checklist¶
☐ Identify Agentic RAG use cases
☐ Identify queries that require adaptive retrieval
☐ Identify required tools
☐ Define tool contracts
☐ Define tool authorization
☐ Implement query understanding
☐ Implement query decomposition
☐ Implement planning
☐ Implement retrieval routing
☐ Implement agent state
☐ Implement tool execution
☐ Implement Vector Search
☐ Implement SQL Retrieval
☐ Implement Graph Retrieval
☐ Implement Multimodal Retrieval
☐ Implement Document Search
☐ Implement evidence collection
☐ Implement evidence deduplication
☐ Implement evidence ranking
☐ Implement evidence evaluation
☐ Implement conflict detection
☐ Implement query rewriting
☐ Implement iterative retrieval
☐ Implement multi-hop retrieval
☐ Implement parallel retrieval
☐ Implement dependency-aware execution
☐ Implement maximum iterations
☐ Implement maximum tool calls
☐ Implement latency budget
☐ Implement token budget
☐ Implement cost budget
☐ Implement authentication
☐ Implement authorization
☐ Implement tenant isolation
☐ Implement tool-level security
☐ Implement SQL security
☐ Implement prompt-injection defenses
☐ Treat retrieved content as untrusted data
☐ Validate tool inputs
☐ Validate tool outputs
☐ Restrict tool capabilities
☐ Implement answer grounding
☐ Implement response validation
☐ Implement citation
☐ Implement abstention
☐ Implement clarification
☐ Implement agent tracing
☐ Track tool calls
☐ Track iterations
☐ Track latency
☐ Track tokens
☐ Track cost
☐ Track errors
☐ Build evaluation dataset
☐ Test tool selection
☐ Test planning
☐ Test multi-hop retrieval
☐ Test conflicting evidence
☐ Test prompt injection
☐ Test unauthorized tool access
☐ Compare Agentic RAG with traditional RAG
☐ Measure accuracy improvement
☐ Measure latency overhead
☐ Measure cost overhead
☐ Establish production SLOs
📚 101. Key Takeaways¶
- Agentic RAG makes retrieval adaptive.
- Traditional RAG generally follows a predetermined retrieval path.
- Agentic RAG can dynamically select tools and retrieval strategies.
- Agents are useful for complex, multi-hop, multi-source questions.
- Agentic retrieval can combine Vector Search, SQL, Knowledge Graphs, and Multimodal Retrieval.
- Query decomposition can break complex questions into manageable sub-questions.
- Dependent sub-questions should be executed sequentially.
- Independent sub-questions can often be executed in parallel.
- Agent state tracks the current retrieval process.
- Evidence evaluation helps determine whether additional retrieval is necessary.
- Query rewriting can improve subsequent retrieval rounds.
- Multi-hop retrieval allows the system to follow relationships across multiple sources.
- Agent loops must always be bounded.
- Tool calls should be controlled and authorized.
- Retrieved documents must be treated as data, not executable instructions.
- Tool outputs should be validated before influencing subsequent execution.
- SQL access should remain read-only for typical RAG workloads.
- Agentic RAG should support abstention when evidence is insufficient.
- Agents should not replace deterministic workflows where deterministic workflows are sufficient.
- Agentic RAG introduces additional latency and cost.
- Observability is essential for understanding agent behavior.
- Agent evaluation should include retrieval, tool selection, planning, grounding, task success, latency, and cost.
- Security controls must exist outside the LLM.
- Production Agentic RAG is an orchestration problem as much as an AI problem.
- The goal is not maximum autonomy.
- The goal is controlled, observable, evidence-driven adaptability.
🧠 Final Mental Model¶
USER QUESTION
│
▼
QUERY UNDERSTANDING
│
▼
AGENT
│
▼
PLANNER
│
▼
TOOL SELECTION
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
VECTOR SQL GRAPH
│ │ │
└─────────────────┼─────────────────┘
│
▼
TOOL RESULTS
│
▼
EVIDENCE STORE
│
▼
EVIDENCE EVALUATOR
│
┌──────────┴──────────┐
▼ ▼
Insufficient Sufficient
│ │
▼ ▼
QUERY REWRITE CONTEXT BUILD
│ │
▼ ▼
RETRIEVE LLM
│ │
└──────────┐ ▼
│ VALIDATION
│ │
│ ▼
└──────► CITATION
│
▼
ENTERPRISE RESPONSE
The central principle is:
Agentic RAG turns retrieval from a fixed pipeline into a controlled decision-making loop that can select tools, gather evidence, evaluate results, and adapt the retrieval strategy before generating an answer.
A production-grade Agentic RAG architecture therefore combines:
Agentic Orchestration
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
Planning Tooling State
│ │ │
└─────────────────┼─────────────────┘
▼
Retrieval Fabric
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
Vector SQL Graph
│ │ │
└─────────────────┼─────────────────┘
│
▼
Evidence Engineering
│
┌──────────┼──────────┐
▼ ▼ ▼
Ranking Validation Conflict
Detection
│ │ │
└──────────┼──────────┘
▼
Context Engineering
│
▼
Foundation Model
│
▼
Response Validation
│
▼
Citation Layer
│
▼
Enterprise Response
The strongest enterprise design is not:
It is:
User
↓
Policy
↓
Agent
↓
Controlled Tools
↓
Trusted Retrieval
↓
Evidence Evaluation
↓
Context Engineering
↓
LLM
↓
Validation
↓
Citation
↓
Enterprise Response
That distinction is what separates an experimental AI agent from a production-grade enterprise Agentic RAG system.
🧭 Chapter Navigation¶
Part V — Advanced Retrieval-Augmented Generation¶
Previous:
05. Multimodal RAG
Next:
01. Prompt Assembly
Section:
05 — Advanced RAG Architecture
Advanced RAG Architecture Path¶
01 Advanced RAG Architecture
↓
02 Graph RAG
↓
03 Knowledge Graphs for RAG
↓
04 SQL RAG
↓
05 Multimodal RAG
↓
06 Agentic RAG
↓
06 Production RAG Engineering
Enterprise AI Engineering Handbook
Building Production-Grade Enterprise AI Systems — One Chapter at a Time.