07. RAG Observability¶
Category: Production RAG Engineering
Module: Part V — Advanced Retrieval-Augmented Generation
Difficulty: Advanced
📖 Overview¶
A production RAG system is not observable simply because application logs exist.
Enterprise RAG introduces a multi-stage execution pipeline:
User Query
↓
Query Processing
↓
Query Rewriting
↓
Embedding
↓
Retrieval
↓
Filtering
↓
Reranking
↓
Context Selection
↓
Prompt Assembly
↓
LLM Generation
↓
Response Validation
↓
Citation
↓
Enterprise Response
When a user receives a poor answer, engineers need to determine:
Was the query rewritten incorrectly?
Did embedding fail?
Did retrieval return the wrong documents?
Did metadata filtering remove valid evidence?
Did reranking select poor results?
Was useful context discarded?
Was the prompt assembled incorrectly?
Did the LLM hallucinate?
Did response validation fail?
Were citations incorrect?
Was the request slow because of retrieval or generation?
Why did token usage increase?
Why did cost increase?
Traditional application logs are often insufficient to answer these questions.
Production RAG therefore requires end-to-end observability across the complete retrieval, reasoning, generation, and response pipeline.
The objective is not simply to collect more logs.
The objective is to create a system where every important RAG decision can be:
Production RAG observability connects system execution with answer quality, performance, cost, security, and user experience.
🎯 Learning Objectives¶
After completing this chapter, you will be able to:
- Understand observability in RAG systems
- Understand RAG observability vs traditional application observability
- Design end-to-end RAG traces
- Design RAG spans
- Trace retrieval pipelines
- Trace reranking
- Trace query rewriting
- Trace context selection
- Trace prompt assembly
- Trace LLM calls
- Trace response validation
- Trace citation generation
- Capture token usage
- Monitor latency
- Monitor throughput
- Monitor errors
- Monitor retries
- Monitor fallbacks
- Monitor retrieval quality signals
- Monitor context quality signals
- Monitor generation quality signals
- Monitor citation quality signals
- Monitor cost
- Implement structured logging
- Implement distributed tracing
- Design RAG-specific metrics
- Design RAG dashboards
- Build production alerts
- Perform trace-based debugging
- Correlate quality with infrastructure metrics
- Detect RAG regressions
- Detect retrieval failures
- Detect model failures
- Detect prompt failures
- Detect cost anomalies
- Detect latency regressions
- Implement tenant-aware observability
- Design privacy-aware observability
- Build enterprise-grade RAG observability architecture
🧠 1. What Is RAG Observability?¶
RAG observability is the ability to understand:
What happened?
Why did it happen?
Where did it happen?
How long did it take?
How much did it cost?
What evidence was used?
What answer was generated?
Was the answer trustworthy?
A useful model is:
RAG Observability
│
├── Logs
├── Metrics
├── Traces
├── Events
├── Quality Signals
├── Cost Signals
└── Security Signals
🧠 2. Traditional Observability vs RAG Observability¶
Traditional backend observability often focuses on:
RAG requires those metrics plus AI-specific signals:
Retrieved Documents
Retrieval Scores
Reranker Scores
Context Size
Prompt Size
LLM Tokens
Model
Temperature
Citations
Grounding
Faithfulness
Answer Quality
Cost
🧠 3. The Three Pillars¶
The classic observability model is:
For RAG, extend this with:
RAG OBSERVABILITY
│
┌───────────────┼────────────────┐
▼ ▼ ▼
Logs Metrics Traces
│ │ │
└───────────────┼────────────────┘
▼
AI QUALITY
│
┌────────────┼────────────┐
▼ ▼ ▼
Retrieval Generation Citation
Quality Quality Quality
🧠 4. Logs¶
Logs answer:
What happened?
Example:
Logs are useful for:
🧠 5. Metrics¶
Metrics answer:
How often and how much?
Examples:
RAG requests / second
p95 latency
retrieval failure rate
average top-K
average context tokens
LLM tokens
cost / request
citation coverage
Metrics are ideal for:
🧠 6. Traces¶
Traces answer:
What happened during this particular request?
Example:
Trace
│
├── Query Processing 12 ms
├── Embedding 25 ms
├── Vector Search 42 ms
├── BM25 Search 31 ms
├── Reranking 88 ms
├── Context Selection 14 ms
├── LLM 920 ms
├── Validation 18 ms
└── Citation 11 ms
🧠 7. Why Traces Matter¶
Suppose:
Without tracing:
With tracing:
The bottleneck becomes obvious.
🧠 8. End-to-End RAG Trace¶
flowchart TD
A["User Request"] --> B["Query Processing"]
B --> C["Query Rewriting"]
C --> D["Embedding"]
D --> E["Retrieval"]
E --> F["Filtering"]
F --> G["Reranking"]
G --> H["Context Selection"]
H --> I["Prompt Assembly"]
I --> J["LLM"]
J --> K["Response Validation"]
K --> L["Citation"]
L --> M["Final Response"]
A -.-> N["Trace"]
B -.-> N
C -.-> N
D -.-> N
E -.-> N
F -.-> N
G -.-> N
H -.-> N
I -.-> N
J -.-> N
K -.-> N
L -.-> N🧠 9. Trace and Span¶
A trace represents the complete request.
A span represents one operation.
Trace
│
├── Query Processing Span
├── Embedding Span
├── Retrieval Span
├── Reranking Span
├── Context Span
├── Prompt Span
├── LLM Span
├── Validation Span
└── Citation Span
🧠 10. Parent-Child Spans¶
RAG Request
│
├── Retrieval
│ ├── Vector Search
│ └── BM25 Search
│
├── Reranking
│
├── Generation
│ └── LLM Call
│
└── Validation
This allows engineers to see both:
🧠 11. RAG Trace Context¶
A trace should propagate through services:
API Gateway
↓
RAG Service
↓
Retrieval Service
↓
Vector DB
↓
Reranker Service
↓
LLM Gateway
↓
Validation Service
The same trace context should be maintained where supported.
🧠 12. Trace ID¶
Every request should have a unique trace identifier.
Use it to correlate:
🧠 13. Request ID vs Trace ID¶
They solve different problems.
A single user request may cross multiple microservices.
🧠 14. Correlation IDs¶
Useful identifiers include:
Be careful with:
These should not be blindly logged.
🧠 15. RAG Trace Data¶
A RAG trace can capture:
{
"trace_id": "abc-123",
"request_id": "req-901",
"model": "enterprise-llm",
"retriever": "hybrid",
"top_k": 10,
"context_tokens": 4200,
"input_tokens": 5100,
"output_tokens": 340,
"latency_ms": 1820
}
🧠 16. Query Processing Observability¶
Track:
Original Query
Query Type
Query Length
Language
Query Rewrite
Number of Generated Queries
Classification
Routing Decision
Example:
Original:
"What DB does payment use?"
Rewritten:
"payment service database technology"
Queries Generated:
3
🧠 17. Query Rewrite Observability¶
For advanced query rewriting:
Capture:
🧠 18. Multi-Query Observability¶
Example:
Track:
Generated Queries
Successful Queries
Duplicate Queries
Retrieval Results per Query
Final Merged Results
🧠 19. Embedding Observability¶
Track:
Example:
🧠 20. Retrieval Observability¶
Track:
Example:
🧠 21. Retrieval Score Distribution¶
Scores can help detect retrieval problems.
A sudden drop in score distribution may indicate:
🧠 22. Retrieval Score Observability¶
Track:
Example:
These are diagnostic signals, not universal quality guarantees.
🧠 23. Retrieval Result Observability¶
Track:
Example:
{
"document_id": "DOC-1024",
"chunk_id": "C-17",
"rank": 1,
"score": 0.91,
"source": "architecture-guide"
}
🧠 24. Metadata Filtering Observability¶
Track:
Example:
If this suddenly becomes:
it may indicate a filtering problem.
🧠 25. Tenant-Aware Retrieval Observability¶
For enterprise systems:
must remain isolated.
Observe:
Never expose sensitive tenant data through unrestricted logs.
🧠 26. Reranking Observability¶
Track:
Example:
🧠 27. Reranking Score Changes¶
Useful diagnostic:
This can help diagnose ranking behavior.
🧠 28. Context Selection Observability¶
Track:
Example:
🧠 29. Context Compression Ratio¶
A useful operational signal:
Example:
🧠 30. Prompt Observability¶
Track:
Prompt Version
Prompt Template
System Prompt Version
Context Tokens
Instruction Tokens
Total Input Tokens
Avoid storing sensitive raw prompts unless required and appropriately protected.
🧠 31. Prompt Versioning¶
Example:
When answer quality changes:
the prompt becomes a candidate cause.
🧠 32. LLM Observability¶
Track:
Provider
Model
Model Version
Temperature
Max Tokens
Input Tokens
Output Tokens
Total Tokens
Latency
Time to First Token
Finish Reason
Errors
Retries
🧠 33. LLM Latency¶
Separate:
from:
This helps distinguish:
🧠 34. Token Observability¶
Track:
Example:
🧠 35. Token Growth Detection¶
Monitor:
over time.
Example:
Possible causes:
🧠 36. Cost Observability¶
Track:
Embedding Cost
Retrieval Cost
Reranker Cost
LLM Input Cost
LLM Output Cost
Evaluation Cost
Infrastructure Cost
🧠 37. Cost Per Request¶
Example:
Embedding $0.0002
Reranking $0.0015
LLM Input $0.0120
LLM Output $0.0030
Infrastructure $0.0010
-----------------------
Total $0.0177
🧠 38. Cost by Tenant¶
Enterprise systems may require:
This helps with:
🧠 39. Cost by Model¶
Compare:
rather than cost alone.
🧠 40. RAG Error Observability¶
Classify failures:
QUERY_ERROR
EMBEDDING_ERROR
RETRIEVAL_ERROR
FILTER_ERROR
RERANKING_ERROR
CONTEXT_ERROR
PROMPT_ERROR
LLM_ERROR
VALIDATION_ERROR
CITATION_ERROR
TIMEOUT
RATE_LIMIT
SECURITY_ERROR
🧠 41. Error Rate¶
Track:
But also classify errors.
An overall:
does not tell you whether the failures are:
or:
🧠 42. Retry Observability¶
Track:
Example:
Retries can improve reliability but increase:
🧠 43. Fallback Observability¶
Example:
Track:
🧠 44. Circuit Breaker Observability¶
For external services:
track:
🧠 45. RAG SLOs¶
Production RAG can define SLOs such as:
Availability >= 99.9%
p95 Latency <= 2 seconds
Error Rate <= 0.5%
Citation Accuracy >= 95%
Faithfulness >= 90%
Quality thresholds should be defined according to the application and risk level.
🧠 46. Quality SLOs¶
Unlike traditional services, RAG can have AI quality SLOs:
This creates:
🧠 47. Error Budget¶
Traditional:
RAG can also track:
Example:
🧠 48. RAG Health Score¶
A dashboard may combine:
into a health view.
However:
Do not hide critical failures behind a single aggregate health score.
🧠 49. RAG Observability Dashboard¶
┌─────────────────────────────────────────────┐
│ RAG OVERVIEW │
├─────────────────────────────────────────────┤
│ Requests/sec 142 │
│ p95 Latency 1.82s │
│ Error Rate 0.31% │
│ │
│ Retrieval Recall 93.1% │
│ Faithfulness 95.7% │
│ Citation Accuracy 97.2% │
│ │
│ Avg Context Tokens 4,210 │
│ Avg Total Tokens 5,020 │
│ Cost / Request $0.018 │
│ │
│ Fallback Rate 0.8% │
│ Abstention Rate 3.2% │
└─────────────────────────────────────────────┘
Values are illustrative.
🧠 50. Retrieval Dashboard¶
┌─────────────────────────────────────┐
│ RETRIEVAL OBSERVABILITY │
├─────────────────────────────────────┤
│ Recall@5 88.4% │
│ Recall@10 94.1% │
│ Hit Rate 96.2% │
│ MRR 81.7% │
│ NDCG@5 79.8% │
│ │
│ Avg Top-1 Score 0.91 │
│ Avg Top-K Score 0.74 │
│ Avg Results 10 │
│ Retrieval p95 96 ms │
└─────────────────────────────────────┘
🧠 51. Generation Dashboard¶
┌─────────────────────────────────────┐
│ GENERATION QUALITY │
├─────────────────────────────────────┤
│ Faithfulness 95.7% │
│ Answer Relevance 93.8% │
│ Completeness 91.2% │
│ Groundedness 96.1% │
│ Citation Accuracy 97.2% │
│ Citation Coverage 95.9% │
└─────────────────────────────────────┘
🧠 52. Latency Dashboard¶
Total p95 Latency
│
├── Query Processing 20ms
├── Embedding 35ms
├── Retrieval 70ms
├── Reranking 120ms
├── Context 15ms
├── LLM 1,420ms
├── Validation 25ms
└── Citation 12ms
🧠 53. Cost Dashboard¶
Monthly Cost
│
├── LLM 68%
├── Embeddings 12%
├── Reranking 8%
├── Vector DB 7%
└── Observability 5%
This helps prioritize optimization.
🧠 54. Trace Visualization¶
A production trace might look like:
TRACE: 4c9d8c7e
0ms ───────────────────────────────────────── 1800ms
Query ███
Embedding ████
Retrieval ███████
Reranking ███████████
Context ███
Prompt ███
LLM █████████████████
Validation ██
Citation ██
This quickly exposes latency bottlenecks.
🧠 55. Trace Waterfall¶
sequenceDiagram
participant U as User
participant API as RAG API
participant R as Retriever
participant RR as Reranker
participant L as LLM
participant V as Validator
participant C as Citation
U->>API: Query
API->>R: Retrieve
R-->>API: Documents
API->>RR: Rerank
RR-->>API: Ranked Context
API->>L: Generate
L-->>API: Answer
API->>V: Validate
V-->>API: Valid
API->>C: Attribute Sources
C-->>API: Citations
API-->>U: Response🧠 56. Span Attributes¶
A retrieval span may include:
retriever.type
retriever.top_k
retriever.index
retriever.collection
retriever.filter
retriever.result_count
retriever.latency_ms
A generation span:
llm.provider
llm.model
llm.temperature
llm.input_tokens
llm.output_tokens
llm.total_tokens
llm.latency_ms
llm.finish_reason
🧠 57. Span Events¶
Events can record important moments:
query_rewritten
retrieval_completed
reranking_completed
context_compressed
llm_retry
validation_failed
citation_added
fallback_triggered
🧠 58. Structured Logging¶
Prefer structured logs:
{
"timestamp": "2026-08-11T10:15:30Z",
"level": "INFO",
"service": "rag-service",
"trace_id": "abc123",
"event": "retrieval_completed",
"retriever": "hybrid",
"top_k": 10,
"result_count": 10,
"latency_ms": 72
}
Structured logs are easier to query and aggregate.
🧠 59. Bad Logging¶
Avoid:
This is difficult to search or aggregate.
Prefer:
🧠 60. Sensitive Data Logging¶
Never blindly log:
RAG systems may process highly sensitive enterprise information.
🧠 61. Prompt Logging Strategy¶
Possible levels:
Level 0
No prompt content
Level 1
Metadata only
Level 2
Redacted prompt
Level 3
Encrypted prompt storage
Level 4
Full prompt with strict access controls
Choose based on:
🧠 62. Document Logging Strategy¶
Avoid storing complete sensitive documents inside traces.
Instead capture:
Example:
🧠 63. Trace Sampling¶
Tracing every request can be expensive.
Possible strategies:
100% Errors
100% High-Latency Requests
100% Security Events
10% Normal Requests
1% Low-Value Requests
Actual sampling should depend on system requirements.
🧠 64. Tail-Based Sampling¶
Instead of deciding sampling before seeing the result:
Keep traces with:
🧠 65. Quality-Based Sampling¶
A powerful RAG-specific strategy:
Possible triggers:
🧠 66. User Feedback Observability¶
Capture:
User behavior can become an important quality signal.
🧠 67. Feedback Correlation¶
Correlate:
Example:
🧠 68. Citation Click Observability¶
If citations are interactive:
Track:
This can help understand whether users trust and use citations.
🧠 69. Conversation Observability¶
For conversational RAG:
Track:
🧠 70. Conversation Context Growth¶
Long conversations can create:
Monitor:
🧠 71. Memory Observability¶
For memory-enabled RAG:
🧠 72. Agentic RAG Observability¶
Agentic RAG requires additional tracing:
🧩 73. Agentic RAG Trace¶
flowchart TD
A["User Query"] --> B["Planner"]
B --> C["Tool Selection"]
C --> D["Retriever"]
D --> E["Observation"]
E --> F{"Enough Evidence?"}
F -->|No| B
F -->|Yes| G["Generation"]
G --> H["Validation"]
H --> I["Final Answer"]Track:
🧠 74. Graph RAG Observability¶
Track:
Entities Retrieved
Relationships Retrieved
Graph Traversal Depth
Nodes Visited
Edges Visited
Subgraph Size
Graph Query Latency
Example:
🧠 75. SQL RAG Observability¶
Track:
Generated SQL
Schema Selected
Tables
Columns
Execution Time
Rows Returned
Query Success
Validation Result
Do not expose sensitive SQL or data without appropriate controls.
🧠 76. SQL Safety Observability¶
Track whether generated SQL attempted:
Production SQL RAG should have explicit read/write policies.
🧠 77. Multimodal RAG Observability¶
Track:
Image ID
Document ID
OCR
Vision Model
Image Embedding
Text Embedding
Cross-Modal Retrieval
Visual Context
🧠 78. Evaluation + Observability¶
These systems complement each other.
Together:
🧠 79. Quality-Trace Correlation¶
Suppose:
Trace analysis may show:
which may show:
The chain becomes:
Model Change
↓
Embedding Change
↓
Retrieval Degradation
↓
Context Degradation
↓
Faithfulness Degradation
This is the real value of RAG observability.
🧠 80. Root Cause Analysis¶
A production debugging workflow:
User Complaint
↓
Find Trace
↓
Inspect Latency
↓
Inspect Retrieval
↓
Inspect Ranking
↓
Inspect Context
↓
Inspect Prompt
↓
Inspect LLM
↓
Inspect Validation
↓
Inspect Citation
🧠 81. Retrieval Failure Debugging¶
Potential causes:
🧠 82. Generation Failure Debugging¶
Possible causes:
🧠 83. Citation Failure Debugging¶
Potential causes:
🧠 84. Latency Failure Debugging¶
or:
The optimization target becomes obvious.
🧠 85. Cost Failure Debugging¶
Possible root cause:
🧠 86. Observability Data Model¶
A useful conceptual model:
Trace
│
├── Request
│
├── Query
│
├── Retrieval
│ ├── Documents
│ └── Scores
│
├── Context
│
├── Prompt
│
├── LLM
│
├── Validation
│
├── Citation
│
├── Metrics
│
└── Feedback
🧠 87. Event Model¶
Example:
{
"event": "reranking_completed",
"trace_id": "abc123",
"candidate_count": 50,
"selected_count": 5,
"latency_ms": 94
}
🧠 88. Metric Types¶
Use:
Examples:
Counter¶
Gauge¶
Histogram¶
Distribution¶
🧠 89. RAG Counters¶
Useful counters:
rag_requests_total
rag_errors_total
retrieval_requests_total
retrieval_failures_total
llm_requests_total
llm_failures_total
fallbacks_total
validation_failures_total
citation_failures_total
🧠 90. RAG Histograms¶
Useful histograms:
rag_latency
retrieval_latency
reranking_latency
llm_latency
context_tokens
input_tokens
output_tokens
cost_per_request
🧠 91. Prometheus-Style Metrics¶
Example:
Latency:
Retrieval:
🧠 92. Metric Cardinality¶
Be careful with labels.
Bad:
as high-cardinality metric labels.
This can create huge metric stores.
Prefer:
and keep high-cardinality identifiers in traces/logs.
🧠 93. Logs vs Metrics vs Traces¶
| Data | Best For |
|---|---|
| Logs | Detailed events |
| Metrics | Trends and alerts |
| Traces | Request-level debugging |
| Evaluation | Quality measurement |
| Feedback | User experience |
🧠 94. Observability Architecture¶
flowchart LR
A["RAG Application"] --> B["Telemetry SDK"]
B --> C["Logs"]
B --> D["Metrics"]
B --> E["Traces"]
C --> F["Log Backend"]
D --> G["Metrics Backend"]
E --> H["Trace Backend"]
F --> I["Observability Platform"]
G --> I
H --> I
J["RAG Evaluation"] --> I
I --> K["Dashboards"]
I --> L["Alerts"]
I --> M["Root Cause Analysis"]🧠 95. OpenTelemetry Concept¶
An enterprise RAG architecture can use an open telemetry standard for:
The application instruments:
and exports telemetry to the organization's observability platform.
🧠 96. Instrumentation Strategy¶
Instrument at these boundaries:
and RAG-specific boundaries:
🧠 97. Custom RAG Spans¶
Examples:
rag.query
rag.rewrite
rag.embedding
rag.retrieve
rag.rerank
rag.context
rag.prompt
rag.generate
rag.validate
rag.citation
These provide a consistent vocabulary.
🧠 98. Trace Naming¶
Good:
Avoid inconsistent names such as:
A consistent naming convention improves observability across teams.
🧠 99. Production RAG Trace¶
TRACE
│
├── rag.query
│ └── query classification
│
├── rag.rewrite
│ ├── query-1
│ ├── query-2
│ └── query-3
│
├── rag.embedding
│
├── rag.retrieve
│ ├── dense
│ └── sparse
│
├── rag.rerank
│
├── rag.context
│
├── rag.prompt
│
├── rag.generate
│
├── rag.validate
│
└── rag.citation
🧠 100. Distributed RAG¶
In a microservice architecture:
Client
↓
API Gateway
↓
RAG Orchestrator
↓
Retrieval Service
↓
Vector Service
↓
Reranker
↓
LLM Gateway
↓
Validation Service
Trace context should propagate across the service boundaries.
🧠 101. Service Dependency Map¶
flowchart TD
A["RAG API"] --> B["Query Service"]
A --> C["Retrieval Service"]
C --> D["Vector DB"]
C --> E["Search Engine"]
A --> F["Reranker"]
A --> G["LLM Gateway"]
A --> H["Validation Service"]
A --> I["Citation Service"]
A --> J["Telemetry"]
B --> J
C --> J
F --> J
G --> J
H --> J
I --> J🧠 102. Dependency Observability¶
Monitor:
for:
🧠 103. External LLM Provider Monitoring¶
Track:
🧠 104. Model Routing Observability¶
For multi-model RAG:
Track:
🧠 105. Multi-Model RAG¶
Example:
Observability must explain:
🧠 106. Router Observability¶
Track:
🧠 107. Cache Observability¶
RAG systems often use:
Track:
🧠 108. Cache Hit Rate¶
Example:
🧠 109. Cache Correctness¶
A cache hit is useful only if the cached result remains valid.
Track:
🧠 110. Knowledge Base Observability¶
RAG quality depends on the knowledge base.
Monitor:
Documents
Chunks
Embeddings
Index Size
Ingestion Failures
Duplicate Documents
Stale Documents
Deleted Documents
🧠 111. Ingestion Observability¶
Track each stage.
🧠 112. Ingestion Trace¶
🧠 113. Data Freshness¶
Monitor:
Measure:
Example:
🧠 114. Stale Knowledge Detection¶
Track:
This is particularly important for:
🧠 115. Knowledge Graph Observability¶
Track:
Nodes
Edges
Entity Extraction
Relationship Extraction
Graph Updates
Graph Query Latency
Graph Traversal
🧠 116. Graph Update Monitoring¶
Failures at any stage can affect Graph RAG.
🧠 117. SQL RAG Observability¶
Monitor:
This creates a full SQL RAG trace.
🧠 118. Security Observability¶
Enterprise RAG must observe:
Authorization Decisions
Tenant Filters
Access Denials
Prompt Injection Detection
Sensitive Data Detection
Policy Violations
🧠 119. Prompt Injection Events¶
Example:
Security events should be highly visible.
🧠 120. Data Leakage Monitoring¶
Monitor whether responses expose:
🧠 121. Auditability¶
For regulated enterprise systems, maintain appropriate audit information:
Who requested
When requested
What system version
What sources were used
What policies applied
What response was produced
Do not retain more sensitive content than necessary.
🧠 122. Observability Retention¶
Not every telemetry type needs the same retention.
Example:
Retention should follow:
🧠 123. Observability Cost¶
Observability itself can become expensive.
Costs include:
Avoid logging everything blindly.
🧠 124. High-Value Telemetry¶
Prioritize:
Errors
Slow Requests
Low-Quality Requests
Security Events
Fallbacks
Cost Anomalies
Retrieval Failures
🧠 125. Low-Value Telemetry¶
Avoid excessive:
🧠 126. Observability Governance¶
Define:
What is logged?
What is traced?
What is sampled?
What is retained?
Who can access it?
How is it redacted?
🧠 127. PII Redaction¶
Before telemetry storage:
Example:
Use an appropriate enterprise redaction mechanism.
🧠 128. Secrets Redaction¶
Never expose:
in:
🧠 129. Access Control¶
Observability systems themselves contain sensitive information.
Use:
🧠 130. Enterprise Observability Architecture¶
USERS
│
▼
┌─────────────┐
│ RAG API │
└──────┬──────┘
│
▼
┌─────────────────┐
│ RAG ORCHESTRATOR│
└────────┬────────┘
│
┌──────────────────┼──────────────────┐
▼ ▼ ▼
Retrieval Reranking LLM
│ │ │
└──────────────────┼──────────────────┘
▼
Response Validation
│
▼
Citation
│
▼
RESPONSE
│
▼
┌─────────────────┐
│ TELEMETRY LAYER │
└────────┬────────┘
│
┌───────────────┼───────────────┐
▼ ▼ ▼
Logs Metrics Traces
│ │ │
└───────────────┼───────────────┘
▼
┌──────────────────┐
│ OBSERVABILITY │
│ PLATFORM │
└────────┬─────────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
Dashboard Alerts Analytics
│
▼
Root Cause
│
▼
System Improvement
🧠 131. RAG Observability Data Flow¶
flowchart LR
A["User Query"] --> B["RAG Pipeline"]
B --> C["Telemetry"]
C --> D["Logs"]
C --> E["Metrics"]
C --> F["Traces"]
B --> G["Evaluation"]
G --> H["Quality Signals"]
D --> I["Observability Platform"]
E --> I
F --> I
H --> I
I --> J["Dashboard"]
I --> K["Alerts"]
I --> L["Root Cause Analysis"]🧠 132. RAG Observability Golden Signals¶
Traditional services often monitor:
RAG should extend this to:
Latency
Traffic
Errors
Saturation
+
Retrieval Quality
Context Quality
Generation Quality
Citation Quality
Token Usage
Cost
🧠 133. RAG Golden Signals¶
RAG GOLDEN SIGNALS
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
System AI Quality Economics
│ │ │
Latency Retrieval Tokens
Traffic Grounding Cost
Errors Citation Model Usage
Saturation Relevance
Completeness
🧠 134. RAG Health Model¶
System Health
│
├── Availability
├── Latency
├── Errors
├── Retrieval
├── Generation
├── Grounding
├── Citation
├── Security
└── Cost
🧠 135. Alerting¶
Alerts should be actionable.
Bad:
Good:
ALERT:
RAG p95 latency exceeded SLO.
Current:
3.2 seconds
Target:
2.0 seconds
Primary contributor:
Reranker latency
Trace samples:
Available
🧠 136. Quality Alert¶
ALERT:
Faithfulness degradation detected.
Current:
88.7%
Baseline:
95.1%
Change:
-6.4 percentage points
Potential correlation:
Embedding model changed 2 hours ago.
🧠 137. Cost Alert¶
ALERT:
Average RAG cost increased by 42%.
Previous:
$0.018/request
Current:
$0.025/request
Primary signal:
Context tokens +61%
🧠 138. Retrieval Alert¶
ALERT:
Recall proxy degradation detected.
Top-1 retrieval score:
0.91 → 0.63
Affected retriever:
hybrid-v3
Affected tenant:
tenant-group-a
🧠 139. Alert Severity¶
Use severity levels:
Example:
🧠 140. Alert Fatigue¶
Too many alerts create:
Only alert when:
🧠 141. Trace-Based Debugging Workflow¶
1. Find affected request
↓
2. Open trace
↓
3. Check total latency
↓
4. Inspect retrieval
↓
5. Inspect ranking
↓
6. Inspect context
↓
7. Inspect prompt
↓
8. Inspect LLM
↓
9. Inspect validation
↓
10. Inspect citations
↓
11. Compare evaluation score
↓
12. Identify root cause
🧠 142. Incident Example¶
User reports:
Trace:
Query
↓
Retrieval
↓
Old document ranked #1
↓
New document ranked #8
↓
Context selection selected #1
↓
LLM generated answer
Root cause:
not necessarily:
🧠 143. Another Incident¶
User reports:
Trace:
Root cause:
🧠 144. Another Incident¶
User reports:
Trace:
Root cause:
🧠 145. Observability and RAG Evaluation¶
RAG REQUEST
│
┌─────────────┴─────────────┐
▼ ▼
OBSERVABILITY EVALUATION
│ │
What happened? Was it good?
│ │
Trace / Logs Quality Metrics
Metrics Judge
│ Human Review
└─────────────┬─────────────┘
▼
ROOT CAUSE
│
▼
SYSTEM IMPROVEMENT
🧠 146. Production Feedback Loop¶
flowchart TD
A["Production Request"] --> B["RAG Execution"]
B --> C["Telemetry"]
B --> D["User Feedback"]
C --> E["Observability"]
D --> F["Evaluation"]
E --> G["Failure Detection"]
F --> G
G --> H["Root Cause Analysis"]
H --> I["Engineering Change"]
I --> J["Regression Evaluation"]
J --> K{"Quality Gate"}
K -->|Pass| L["Deploy"]
K -->|Fail| I
L --> A🧠 147. Production RAG Observability Checklist¶
☐ Trace every important RAG stage
☐ Propagate trace context across services
☐ Use structured logging
☐ Define RAG-specific metrics
☐ Monitor p50/p95/p99 latency
☐ Monitor throughput
☐ Monitor error rates
☐ Monitor retries
☐ Monitor fallbacks
☐ Monitor circuit breakers
☐ Monitor query rewriting
☐ Monitor embedding
☐ Monitor retrieval
☐ Monitor retrieval scores
☐ Monitor metadata filtering
☐ Monitor reranking
☐ Monitor context selection
☐ Monitor context size
☐ Monitor prompt version
☐ Monitor LLM calls
☐ Monitor token usage
☐ Monitor validation
☐ Monitor citations
☐ Monitor retrieval quality
☐ Monitor grounding
☐ Monitor faithfulness
☐ Monitor answer relevance
☐ Monitor citation accuracy
☐ Monitor citation coverage
☐ Monitor cost
☐ Monitor cost by model
☐ Monitor cost by tenant
☐ Monitor cost anomalies
☐ Monitor knowledge freshness
☐ Monitor ingestion failures
☐ Monitor index health
☐ Monitor stale documents
☐ Monitor prompt injection
☐ Monitor authorization
☐ Monitor tenant isolation
☐ Monitor sensitive data leakage
☐ Implement dashboards
☐ Implement alerts
☐ Implement trace sampling
☐ Implement quality-based sampling
☐ Implement PII redaction
☐ Implement secrets redaction
☐ Implement RBAC
☐ Implement retention policies
☐ Correlate traces with evaluations
☐ Correlate traces with user feedback
☐ Implement failure taxonomy
☐ Implement root-cause analysis
☐ Implement continuous improvement
🧪 148. Practical Project¶
Build a Production RAG Observability Platform.
The platform should capture:
and expose:
🧪 149. Suggested Project Structure¶
rag-observability/
│
├── instrumentation/
│ ├── query.py
│ ├── retrieval.py
│ ├── reranking.py
│ ├── context.py
│ ├── generation.py
│ ├── validation.py
│ └── citation.py
│
├── telemetry/
│ ├── logging/
│ ├── metrics/
│ └── tracing/
│
├── evaluation/
│ ├── quality/
│ ├── grounding/
│ └── citation/
│
├── dashboards/
│
├── alerts/
│
├── security/
│ ├── redaction/
│ └── access-control/
│
├── storage/
│
└── configuration/
🧪 150. Example Instrumentation¶
class RAGRetriever:
def retrieve(self, query):
with tracer.start_as_current_span(
"rag.retrieve"
) as span:
span.set_attribute(
"retriever.type",
"hybrid"
)
span.set_attribute(
"retriever.top_k",
10
)
results = self.search(query)
span.set_attribute(
"retriever.result_count",
len(results)
)
return results
🧪 151. LLM Instrumentation¶
class LLMService:
def generate(self, prompt):
with tracer.start_as_current_span(
"rag.generate"
) as span:
response = self.llm.generate(
prompt
)
span.set_attribute(
"llm.model",
self.model_name
)
span.set_attribute(
"llm.input_tokens",
response.input_tokens
)
span.set_attribute(
"llm.output_tokens",
response.output_tokens
)
return response
🧪 152. RAG Trace Record¶
{
"trace_id": "trace-001",
"query": {
"length": 42,
"language": "en"
},
"retrieval": {
"type": "hybrid",
"top_k": 10,
"latency_ms": 72
},
"reranking": {
"enabled": true,
"candidates": 50,
"selected": 5,
"latency_ms": 94
},
"context": {
"chunks": 5,
"tokens": 4200
},
"generation": {
"model": "enterprise-llm",
"input_tokens": 5100,
"output_tokens": 340,
"latency_ms": 1420
},
"citation": {
"count": 3
}
}
🧪 153. Example Metrics¶
rag_requests = Counter(
"rag_requests_total",
"Total RAG requests"
)
rag_latency = Histogram(
"rag_request_duration_seconds",
"RAG request latency"
)
retrieval_latency = Histogram(
"rag_retrieval_duration_seconds",
"Retrieval latency"
)
llm_tokens = Counter(
"rag_llm_tokens_total",
"Total LLM tokens"
)
🧪 154. Observability Test¶
Create a test query:
Expected trace:
rag.query
↓
rag.embedding
↓
rag.retrieve
↓
rag.rerank
↓
rag.context
↓
rag.prompt
↓
rag.generate
↓
rag.validate
↓
rag.citation
🧪 155. Observability Acceptance Criteria¶
The project should be able to answer:
What happened?
How long did it take?
Which documents were retrieved?
What scores did they receive?
Which documents were selected?
How many context tokens were used?
Which model generated the answer?
How many tokens were consumed?
How much did the request cost?
Were citations generated?
Was validation successful?
Did the request use a fallback?
Did the user provide negative feedback?
🧠 156. Advanced Production Exercise¶
Extend the platform to support:
☐ Distributed tracing
☐ OpenTelemetry instrumentation
☐ Trace sampling
☐ Tail-based sampling
☐ Quality-based sampling
☐ Structured logs
☐ Prometheus metrics
☐ RAG dashboards
☐ Quality dashboards
☐ Cost dashboards
☐ Tenant dashboards
☐ Alerting
☐ Error budgets
☐ RAG SLOs
☐ User feedback correlation
☐ Evaluation correlation
☐ Failure taxonomy
☐ Root cause analysis
☐ Knowledge freshness
☐ Cache observability
☐ Multi-model routing
☐ Agentic RAG tracing
☐ Graph RAG tracing
☐ SQL RAG tracing
☐ Multimodal RAG tracing
☐ PII redaction
☐ Secret redaction
☐ RBAC
☐ Auditability
🧠 157. Production RAG Observability Maturity¶
Level 1 — Application Logs¶
Level 2 — Metrics¶
Level 3 — Distributed Tracing¶
Level 4 — RAG-Aware Observability¶
Level 5 — Quality Observability¶
Level 6 — Enterprise AI Observability¶
🧠 158. Observability Maturity Model¶
Enterprise AI
▲
│
Quality + Governance
│
RAG-Aware Telemetry
│
Distributed Tracing
│
Metrics
│
Logs
│
└──────────────►
🧠 159. Final Production Architecture¶
USER
│
▼
┌────────────┐
│ RAG API │
└─────┬──────┘
│
▼
┌─────────────────┐
│ RAG ORCHESTRATOR│
└────────┬────────┘
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
Query Retrieval Model
│ │ │
▼ ▼ ▼
Rewrite Reranker LLM
│ │
└────────┬────────┘
▼
Validation
│
▼
Citation
│
▼
Response
│
▼
┌──────────────────┐
│ TELEMETRY │
└────────┬─────────┘
│
┌────────────────┼────────────────┐
▼ ▼ ▼
Logs Metrics Traces
│ │ │
└────────────────┼────────────────┘
▼
Observability Platform
│
┌────────────────────┼────────────────────┐
▼ ▼ ▼
Dashboard Alerts Analysis
│ │ │
└────────────────────┼────────────────────┘
▼
RAG Evaluation
│
▼
Root Cause Analysis
│
▼
System Improvement
│
▼
Regression Testing
│
▼
Deployment
🧠 160. Final Mental Model¶
RAG OBSERVABILITY
│
┌───────────────────────┼───────────────────────┐
▼ ▼ ▼
SYSTEM AI QUALITY ECONOMICS
│ │ │
Latency Retrieval Tokens
Traffic Grounding Cost
Errors Faithfulness Model Usage
Saturation Citation Cache
│ Relevance
│ Completeness
└───────────────────────┼───────────────────────┘
│
▼
SECURITY
│
┌──────────┼──────────┐
▼ ▼ ▼
Access Leakage Injection
│
▼
USER EXPERIENCE
│
┌──────────┼──────────┐
▼ ▼ ▼
Feedback Citations Escalation
│
▼
ROOT CAUSE ANALYSIS
│
▼
CONTINUOUS IMPROVEMENT
The fundamental production loop is:
Production RAG observability is not simply monitoring infrastructure. It is the engineering discipline that connects every retrieval, context, generation, citation, quality, cost, security, and user-experience signal into one explainable system.
📚 161. Key Takeaways¶
- Logs tell you what happened.
- Metrics tell you what is happening at scale.
- Traces tell you how an individual request executed.
- RAG requires AI-specific observability in addition to infrastructure telemetry.
- Every major RAG stage should have appropriate telemetry.
- Query rewriting should be observable.
- Embedding generation should be observable.
- Retrieval should expose diagnostic metadata.
- Reranking should expose candidate and ranking information.
- Context selection should expose context size and selection behavior.
- Prompt versions should be tracked.
- LLM model, token, latency, and finish information should be captured.
- Response validation should generate observable events.
- Citation generation should be traceable.
- Retrieval quality should be correlated with final response quality.
- Token usage is both a performance and cost signal.
- Context growth can cause latency and cost regressions.
- RAG observability should include quality signals such as groundedness and faithfulness.
- User feedback can be correlated with traces to identify failure patterns.
- Production RAG requires observability for agentic workflows.
- Graph RAG requires graph-specific telemetry.
- SQL RAG requires SQL generation and execution telemetry.
- Multimodal RAG requires cross-modal telemetry.
- Enterprise systems require tenant-aware observability.
- Sensitive prompts and documents should not be blindly logged.
- PII and secrets must be appropriately redacted.
- Observability platforms themselves require access control.
- Trace sampling can reduce observability costs.
- Tail-based and quality-based sampling can preserve high-value traces.
- Knowledge-base freshness should be observable.
- Ingestion pipelines should be observable.
- Cache behavior should be observable.
- Model routing should be observable.
- Quality SLOs can complement traditional infrastructure SLOs.
- RAG error budgets can help manage AI quality degradation.
- Alerts should be actionable rather than noisy.
- Trace-based debugging enables root-cause analysis.
- Evaluation tells you whether the result was good.
- Observability tells you what happened.
- Combining both enables continuous RAG improvement.
🧭 162. Chapter Navigation¶
Part V — Advanced Retrieval-Augmented Generation¶
Previous:
06. RAG Evaluation & Benchmarking
Next:
08. RAG Performance Optimization
Section:
06 — Production RAG Engineering
Production RAG Engineering Path¶
01 Prompt Assembly
↓
02 Context Selection & Context Engineering
↓
03 Response Validation
↓
04 Citation & Source Attribution
↓
05 Enterprise Response
↓
06 RAG Evaluation & Benchmarking
↓
07 RAG Observability
↓
08 RAG Performance Optimization
↓
09 RAG Cost Optimization
↓
10 Production Retrieval Architecture
↓
11 Building Production RAG Systems
Enterprise AI Engineering Handbook
Building Production-Grade Enterprise AI Systems — One Chapter at a Time.