08. RAG Performance Optimization¶
Category: Production RAG Engineering
Module: Part V — Advanced Retrieval-Augmented Generation
Difficulty: Advanced
📖 Overview¶
A RAG system can be functionally correct and still fail in production because it is:
- Too slow
- Too expensive
- Difficult to scale
- Inefficient with context
- Over-fetching documents
- Performing unnecessary model calls
- Saturating vector databases
- Generating excessive tokens
- Performing redundant retrieval
- Using expensive models for simple requests
Production RAG performance optimization is therefore not a single optimization technique.
It is a systematic engineering discipline covering the complete pipeline:
User Query
↓
Query Processing
↓
Query Rewriting
↓
Embedding
↓
Retrieval
↓
Filtering
↓
Reranking
↓
Context Selection
↓
Prompt Assembly
↓
LLM Generation
↓
Validation
↓
Citation
↓
Response
The goal is to optimize the complete system across:
Latency
Throughput
Accuracy
Context Efficiency
Token Usage
Cost
Scalability
Reliability
Resource Utilization
The fastest RAG system is not necessarily the one with the fastest individual component. It is the system that minimizes unnecessary work while preserving answer quality.
🎯 Learning Objectives¶
After completing this chapter, you will be able to:
- Understand RAG performance bottlenecks
- Decompose end-to-end RAG latency
- Optimize retrieval latency
- Optimize embedding generation
- Optimize vector search
- Optimize hybrid retrieval
- Optimize reranking
- Optimize context selection
- Reduce unnecessary context
- Optimize prompt construction
- Reduce LLM latency
- Optimize token usage
- Implement caching
- Implement parallel retrieval
- Implement asynchronous processing
- Optimize batching
- Optimize model selection
- Implement query routing
- Optimize top-K
- Optimize reranking candidates
- Optimize context windows
- Optimize vector indexes
- Optimize database connections
- Improve throughput
- Control concurrency
- Optimize resource utilization
- Design performance SLOs
- Perform latency profiling
- Perform capacity planning
- Build production performance dashboards
- Balance latency, quality, and cost
🧠 1. What Is RAG Performance Optimization?¶
RAG performance optimization means improving the system's:
while maintaining acceptable:
A useful objective is:
🧠 2. Performance Is a Multi-Dimensional Problem¶
Do not define performance as latency alone.
RAG PERFORMANCE
│
┌───────────────┼────────────────┐
▼ ▼ ▼
Latency Throughput Cost
│ │ │
▼ ▼ ▼
p50/p95/p99 req/sec $/request
│
▼
Scalability
│
▼
Quality
🧠 3. End-to-End Latency¶
A simplified model:
T_total =
T_query
+ T_embedding
+ T_retrieval
+ T_reranking
+ T_context
+ T_prompt
+ T_generation
+ T_validation
+ T_citation
The first optimization step is therefore:
Measure before optimizing.
🧠 4. Latency Waterfall¶
Query Processing ███ 20 ms
Embedding █████ 35 ms
Retrieval ███████ 70 ms
Reranking ███████████ 120 ms
Context Selection ██ 15 ms
Prompt Assembly ██ 10 ms
LLM Generation █████████████████████ 1420 ms
Validation ███ 25 ms
Citation ██ 12 ms
────────────────────────────────────────────────────
Total 1727 ms
The largest component should usually receive the most attention.
🧠 5. Latency Percentiles¶
Average latency is not enough.
Monitor:
Example:
A good average can hide poor tail latency.
🧠 6. Why p95 and p99 Matter¶
Suppose:
The average may still appear reasonable.
But that one slow request represents a serious tail-latency problem.
Production systems therefore optimize:
for predictable user experience.
🧠 7. Performance Optimization Framework¶
Use this loop:
Measure
↓
Profile
↓
Identify Bottleneck
↓
Form Hypothesis
↓
Optimize
↓
Benchmark
↓
Evaluate Quality
↓
Deploy
↓
Monitor
Never optimize blindly.
🧠 8. RAG Performance Bottlenecks¶
Common bottlenecks include:
Slow Embeddings
Slow Vector Search
Large Top-K
Expensive Reranking
Large Context
Large Prompt
Slow LLM
Repeated LLM Calls
Sequential Retrieval
Database Saturation
Network Latency
Excessive Serialization
Poor Caching
High Concurrency
🧠 9. Retrieval Pipeline¶
flowchart LR
A["Query"] --> B["Embedding"]
B --> C["Vector Search"]
A --> D["Keyword Search"]
C --> E["Candidate Merge"]
D --> E
E --> F["Filtering"]
F --> G["Reranking"]
G --> H["Context Selection"]
H --> I["LLM"]Every stage can become a bottleneck.
🧠 10. The Optimization Principle¶
A powerful production rule:
Do Less Work
↓
Do Necessary Work in Parallel
↓
Use the Cheapest Suitable Component
↓
Cache Reusable Results
↓
Measure Quality
🧠 11. Optimize the Critical Path¶
The critical path is the sequence of operations that determines response time.
Example:
If independent operations exist:
do not unnecessarily execute them sequentially.
🧠 12. Sequential vs Parallel Retrieval¶
Sequential¶
Latency:
Parallel¶
Latency becomes approximately:
🧠 13. Parallel Retrieval¶
from concurrent.futures import ThreadPoolExecutor
def retrieve(query):
with ThreadPoolExecutor(max_workers=2) as executor:
dense_future = executor.submit(
dense_retriever.retrieve,
query
)
sparse_future = executor.submit(
sparse_retriever.retrieve,
query
)
dense_results = dense_future.result()
sparse_results = sparse_future.result()
return merge_results(
dense_results,
sparse_results
)
Use concurrency carefully according to the client, database, and service behavior.
🧠 14. Async Retrieval¶
For I/O-heavy systems:
import asyncio
async def retrieve(query):
dense_task = asyncio.create_task(
dense_retriever.retrieve(query)
)
sparse_task = asyncio.create_task(
sparse_retriever.retrieve(query)
)
dense, sparse = await asyncio.gather(
dense_task,
sparse_task
)
return merge_results(dense, sparse)
🧠 15. Parallelism Trade-Off¶
Parallelism can reduce latency but increase:
Therefore:
Concurrency must be bounded.
🧠 16. Concurrency Control¶
Use:
Example:
import asyncio
semaphore = asyncio.Semaphore(20)
async def safe_retrieve(query):
async with semaphore:
return await retriever.retrieve(query)
🧠 17. Top-K Optimization¶
A larger K is not always better.
Increasing K can improve recall but increases:
🧠 18. Retrieval Top-K Trade-Off¶
The improvement often diminishes after a certain point.
🧠 19. Candidate K vs Final K¶
Separate:
from:
Example:
This is often more effective than sending all 50 documents to the LLM.
🧠 20. Adaptive Top-K¶
Instead of always using:
adapt K based on query characteristics.
🧠 21. Score-Based Retrieval Cutoff¶
Instead of selecting only by fixed K:
Example:
Thresholds must be calibrated for the retrieval system.
🧠 22. Dynamic Retrieval¶
A more advanced pipeline:
Query
↓
Initial Retrieval
↓
Are results sufficient?
│
├── Yes → Continue
│
└── No → Expand Retrieval
This avoids expensive retrieval for easy queries.
🧠 23. Early Exit¶
Example:
This is a powerful optimization.
🧠 24. Query Classification¶
Before expensive processing:
Simple requests can bypass unnecessary stages.
🧠 25. Router-Based Optimization¶
flowchart TD
A["User Query"] --> B["Query Router"]
B --> C["Simple Retrieval"]
B --> D["Hybrid Retrieval"]
B --> E["Graph RAG"]
B --> F["SQL RAG"]
B --> G["Agentic RAG"]
C --> H["Response"]
D --> H
E --> H
F --> H
G --> HThe objective:
Use the simplest pipeline that can reliably answer the query.
🧠 26. Query Rewriting Cost¶
Query rewriting improves retrieval but adds latency.
If rewriting costs:
for every request, it may become a major bottleneck.
🧠 27. Conditional Query Rewriting¶
This avoids unnecessary LLM calls.
🧠 28. Multi-Query Optimization¶
Multi-query retrieval:
can improve recall but increases:
Use it selectively.
🧠 29. Multi-Query Parallelization¶
┌── Query A ── Search ──┐
├── Query B ── Search ──┤
Original Query ──┼── Query C ── Search ──┼── Merge
└── Query D ── Search ──┘
Parallelize independent searches.
🧠 30. Embedding Optimization¶
Embedding latency can be reduced using:
🧠 31. Embedding Batching¶
Instead of:
batch:
This is particularly important during ingestion.
🧠 32. Query Embedding Cache¶
Queries can sometimes repeat.
🧠 33. Embedding Cache Example¶
def get_embedding(query):
key = hash_query(query)
cached = cache.get(key)
if cached:
return cached
embedding = embedding_model.embed(query)
cache.set(key, embedding)
return embedding
Use appropriate invalidation/versioning.
🧠 34. Embedding Model Selection¶
Larger models may provide better embeddings but can increase:
Evaluate:
together.
🧠 35. Vector Search Optimization¶
Vector search performance depends on:
🧠 36. ANN Search¶
Approximate Nearest Neighbor search trades exactness for speed.
Exact Search
↓
High Recall
High Cost
ANN Search
↓
Lower Search Cost
Very Fast
Potential Recall Trade-Off
🧠 37. Index Selection¶
Common structures include:
Selection depends on:
🧠 38. Flat Search¶
Complexity grows with dataset size.
Good for:
🧠 39. HNSW¶
HNSW creates a graph-based search structure.
Search navigates the graph rather than scanning every vector.
🧠 40. HNSW Search Parameters¶
Common parameters include:
Increasing search effort can improve recall but increase latency.
🧠 41. efSearch Trade-Off¶
Tune against a benchmark rather than choosing arbitrary values.
🧠 42. Vector Dimension¶
Higher dimensions can increase:
But reducing dimensions can affect retrieval quality.
Therefore:
🧠 43. Quantization¶
Quantization reduces representation size.
Potential benefits:
Potential trade-off:
Benchmark before production adoption.
🧠 44. Vector Database Optimization¶
Optimize:
🧠 45. Connection Pooling¶
Bad:
Better:
Reuse connections.
🧠 46. Retrieval Payload Optimization¶
Avoid returning unnecessary data.
Bad:
if only:
is needed.
🧠 47. Hybrid Search Optimization¶
Hybrid retrieval:
can improve quality but adds work.
Optimize using:
🧠 48. Hybrid Retrieval Pipeline¶
Query
│
┌─────────┴─────────┐
▼ ▼
Dense Search Sparse Search
│ │
└─────────┬─────────┘
▼
Merge
↓
Rerank
↓
Top-N Context
🧠 49. Reranking Cost¶
Reranking is often more expensive than initial retrieval.
Example:
If you rerank:
the cost can become significant.
🧠 50. Candidate Reduction¶
Instead of:
use:
This reduces reranking work.
🧠 51. Two-Stage Retrieval¶
This is a common production architecture.
🧠 52. Multi-Stage Retrieval¶
flowchart LR
A["Query"] --> B["Cheap Retrieval"]
B --> C["Candidate Set"]
C --> D["Filtering"]
D --> E["Reranking"]
E --> F["Context Compression"]
F --> G["Final Context"]
G --> H["LLM"]Each stage reduces the amount of data passed to the next expensive stage.
🧠 53. Reranker Batching¶
When reranking multiple candidates:
batch them when supported.
This can improve accelerator utilization and reduce per-request overhead.
🧠 54. Reranker Selection¶
Possible options:
Use the least expensive mechanism that meets the quality target.
🧠 55. Context Optimization¶
One of the most important RAG optimizations is:
Do not send irrelevant context to the LLM.
Large context can cause:
🧠 56. Context Compression¶
Example:
🧠 57. Context Selection¶
Use:
to select final context.
🧠 58. Context Token Budget¶
Define:
Example:
Selection should respect the budget.
🧠 59. Token Budgeting¶
A useful conceptual budget:
Model Context Window
│
├── System Prompt
├── User Query
├── Retrieved Context
├── Conversation Memory
└── Output Budget
If context grows excessively:
🧠 60. Context Ordering¶
Ordering can affect generation quality.
Possible strategy:
or an empirically tested ordering strategy.
Do not assume one ordering works universally.
🧠 61. Duplicate Context Removal¶
Retrieval systems may return:
Deduplicate before generation.
def deduplicate(chunks):
seen = set()
result = []
for chunk in chunks:
if chunk.id not in seen:
seen.add(chunk.id)
result.append(chunk)
return result
🧠 62. Parent-Child Retrieval Optimization¶
Parent-child retrieval can provide:
Optimization:
rather than sending entire documents.
🧠 63. MMR Optimization¶
MMR can reduce redundant context.
Instead of:
you may select:
This can improve context efficiency.
🧠 64. Context Diversity¶
Example:
D1 → Payment architecture
D2 → Payment architecture
D3 → Payment architecture
D4 → Database configuration
D5 → Error handling
A diversity-aware selection can provide broader evidence.
🧠 65. Prompt Optimization¶
Prompt construction has two goals:
Avoid:
🧠 66. Prompt Template Optimization¶
Bad:
Better:
🧠 67. Prompt Caching¶
If supported by the model provider:
Potential benefits:
🧠 68. Static vs Dynamic Prompt Content¶
Separate:
Static
├── System Instructions
├── Response Schema
└── Policy
Dynamic
├── Query
├── Context
└── Conversation
This makes caching and prompt management easier.
🧠 69. LLM Latency Optimization¶
LLM latency can be influenced by:
🧠 70. Model Routing¶
Use different models for different workloads:
The objective is:
🧠 71. Model Cascade¶
This can reduce average cost and latency.
🧠 72. Streaming¶
Without streaming:
With streaming:
Streaming improves perceived latency even when total generation time remains similar.
🧠 73. Time to First Token¶
Track:
separately from:
Example:
A system may feel responsive despite a longer total generation time.
🧠 74. Output Token Optimization¶
Large answers increase:
Use:
where appropriate.
🧠 75. Structured Output¶
If the application requires a fixed response:
Structured output can reduce unnecessary generation and downstream parsing work.
🧠 76. Validation Optimization¶
Validation itself can add latency.
Avoid:
for every low-risk query unless justified.
Possible strategies:
🧠 77. Selective Validation¶
Response
↓
Risk Classifier
│
├── Low Risk → Lightweight Validation
│
└── High Risk → Deep Validation
🧠 78. Citation Optimization¶
Citation generation can be optimized by maintaining source IDs throughout the pipeline.
Instead of reconstructing citations:
carry:
through the pipeline.
🧠 79. End-to-End Source Tracking¶
The source identity should remain attached to the context object.
🧠 80. Caching¶
Caching is one of the most effective RAG optimizations.
Possible cache layers:
🧠 81. Cache Architecture¶
flowchart TD
A["User Query"] --> B["Cache"]
B -->|Hit| C["Cached Result"]
B -->|Miss| D["RAG Pipeline"]
D --> E["Store Result"]
E --> C🧠 82. Retrieval Cache¶
Cache:
Example key:
🧠 83. Cache Invalidation¶
The biggest cache problem:
Invalidate when:
🧠 84. Cache Versioning¶
Use:
This makes invalidation more deterministic.
🧠 85. Semantic Cache¶
A semantic cache attempts to reuse results for semantically similar queries.
These may be semantically similar.
But semantic caching must consider:
🧠 86. Semantic Cache Risk¶
Two queries may be similar but require different answers.
Therefore:
Use conservative thresholds and evaluate carefully.
🧠 87. Cache by Tenant¶
Never allow:
to reuse unauthorized data.
Cache keys should include appropriate isolation dimensions.
🧠 88. Batch Processing¶
Batching can improve:
Example:
The optimal batch size depends on infrastructure and model behavior.
🧠 89. Ingestion Performance¶
Production ingestion:
Optimize using:
🧠 90. Incremental Indexing¶
Avoid rebuilding the complete index when only a few documents changed.
instead of:
🧠 91. Incremental Embedding¶
Track document versions:
Only regenerate embeddings when relevant content changes.
🧠 92. Change Detection¶
Use:
Example:
If the hash is unchanged:
🧠 93. Index Build Optimization¶
For large ingestion jobs:
Avoid one-document-at-a-time indexing.
🧠 94. Bulk Indexing¶
Instead of:
use:
This reduces network and transaction overhead.
🧠 95. Database Optimization¶
Monitor:
🧠 96. Vector DB Saturation¶
Symptoms:
Possible solutions:
🧠 97. Horizontal Scaling¶
Load Balancer
│
┌──────────┼──────────┐
▼ ▼ ▼
RAG-1 RAG-2 RAG-3
│ │ │
└──────────┼──────────┘
▼
Vector DB
Stateless RAG services scale more easily.
🧠 98. Autoscaling¶
Scale based on:
For AI systems also consider:
🧠 99. Backpressure¶
When downstream services cannot keep up:
Without backpressure:
🧠 100. Load Shedding¶
Under extreme load:
while protecting:
🧠 101. Rate Limiting¶
Apply limits at:
Example:
🧠 102. Resource Isolation¶
Use separate resource pools for:
This prevents batch workloads from degrading interactive traffic.
🧠 103. Priority Queues¶
Priority 1
Production User Query
Priority 2
Internal Query
Priority 3
Evaluation
Priority 4
Batch Processing
🧠 104. Network Optimization¶
RAG often crosses:
Network latency can accumulate.
Reduce it using:
🧠 105. Region Selection¶
If the application runs in:
but the LLM endpoint is far away:
latency increases.
Use an appropriate region/provider architecture while respecting:
🧠 106. Serialization Optimization¶
Avoid transferring unnecessary:
between services.
Prefer compact payloads.
🧠 107. Compression¶
Compression can reduce:
but adds:
Benchmark the trade-off.
🧠 108. Retrieval Result Payload¶
Prefer:
instead of transferring the entire source document when it is not needed.
🧠 109. Observability Overhead¶
Instrumentation itself consumes:
Optimize telemetry using:
🧠 110. Performance vs Quality¶
The key trade-off:
The goal is not:
but:
🧠 111. Quality-Latency-Cost Triangle¶
Optimizing one dimension can affect the others.
🧠 112. Example Trade-Off¶
Option A:
Option B:
The correct choice depends on the product SLO and risk profile.
🧠 113. Performance Budget¶
Define a latency budget:
Example:
Query Processing 50 ms
Embedding 100 ms
Retrieval 200 ms
Reranking 250 ms
Context 50 ms
LLM 1200 ms
Validation 100 ms
Citation 50 ms
--------------------------------
Total 2000 ms
🧠 114. Budget Violation¶
If:
then other stages cannot consume unlimited latency.
The budget forces architectural discipline.
🧠 115. Throughput¶
Throughput is often measured as:
For LLM systems also monitor:
🧠 116. Retrieval Throughput¶
Example:
The LLM becomes the bottleneck.
🧠 117. Bottleneck Identification¶
flowchart LR
A["Query"] --> B["Embedding"]
B --> C["Retrieval"]
C --> D["Reranking"]
D --> E["LLM"]
E --> F["Response"]
D -. "Potential Bottleneck" .-> G["Profile"]
E -. "Potential Bottleneck" .-> GAlways profile the actual workload.
🧠 118. Queueing Effects¶
As utilization approaches capacity:
A component operating near saturation can cause dramatic tail latency.
🧠 119. Capacity Planning¶
Estimate:
Then size:
🧠 120. Load Testing¶
Test:
and observe:
🧠 121. Stress Testing¶
Push beyond expected capacity.
Find:
🧠 122. Performance Regression Testing¶
Every major change should be benchmarked.
Example:
🧠 123. Benchmark Table¶
| Configuration | p95 Latency | Recall@10 | Tokens | Cost |
|---|---|---|---|---|
| Dense | 620 ms | 88% | 4,800 | Low |
| Hybrid | 710 ms | 93% | 5,000 | Medium |
| Hybrid + Reranker | 920 ms | 96% | 4,300 | Higher |
| Hybrid + Reranker + Compression | 980 ms | 96% | 3,100 | Medium |
Illustrative values only.
🧠 124. A/B Performance Testing¶
Compare:
against:
using:
🧠 125. Canary Deployment¶
Monitor:
Increase traffic only if healthy.
🧠 126. Performance Optimization Experiment¶
Example:
Hypothesis:
Reducing reranker candidates
from 100 → 40
will reduce latency
without significant recall loss.
Experiment:
Measure:
🧠 127. Performance Optimization Notebook¶
experiments = [
{
"name": "reranker_candidates_100",
"candidates": 100
},
{
"name": "reranker_candidates_40",
"candidates": 40
}
]
Run the same evaluation set against both.
🧠 128. Optimization Scorecard¶
| Dimension | Baseline | Optimized | Change |
|---|---|---|---|
| p95 Latency | 2.1s | 1.4s | -33% |
| Recall@10 | 94% | 93% | -1 pp |
| Faithfulness | 95% | 95% | 0 |
| Tokens | 5,400 | 3,600 | -33% |
| Cost | $0.025 | $0.017 | -32% |
The objective is not necessarily to maximize every metric independently.
🧠 129. Common Optimization Mistakes¶
Mistake 1¶
without measuring context quality.
Mistake 2¶
for every query.
Mistake 3¶
without considering latency.
Mistake 4¶
for every request.
Mistake 5¶
without candidate reduction.
Mistake 6¶
without invalidation.
Mistake 7¶
without measuring quality.
🧠 130. Over-Optimization¶
A system can become:
but:
Example:
Latency may be excellent while answer quality collapses.
🧠 131. Under-Optimization¶
The opposite:
Top-K:
100
Reranker:
100
Context:
20,000 tokens
LLM:
Largest Model
Validation:
2 additional LLM calls
Quality may improve slightly while:
explode.
🧠 132. Optimization Priority¶
A practical order:
1. Measure
2. Remove unnecessary work
3. Parallelize independent work
4. Reduce candidate volume
5. Optimize context
6. Cache reusable work
7. Optimize model selection
8. Optimize infrastructure
9. Fine-tune low-level components
🧠 133. Remove Unnecessary Work¶
Ask:
Do we need query rewriting?
Do we need multi-query?
Do we need reranking?
Do we need a second validation model?
Do we need 20 documents?
Do we need 8,000 context tokens?
Do we need the largest model?
The cheapest optimization is often:
Not performing the operation at all.
🧠 134. Performance Architecture¶
flowchart TD
A["User Query"] --> B["Query Router"]
B --> C{"Simple?"}
C -->|Yes| D["Fast Retrieval"]
C -->|No| E["Advanced Retrieval"]
E --> F["Parallel Dense + Sparse"]
F --> G["Candidate Merge"]
G --> H["Metadata Filtering"]
H --> I["Reranking"]
I --> J["Context Compression"]
D --> K["Context Selection"]
J --> K
K --> L["Prompt Assembly"]
L --> M["Model Router"]
M --> N["Small Model"]
M --> O["Large Model"]
N --> P["Validation"]
O --> P
P --> Q["Citation"]
Q --> R["Response"]🧠 135. Production Performance Architecture¶
The key architectural principles are:
Fast Path
+
Adaptive Path
+
Parallel Retrieval
+
Candidate Reduction
+
Context Budget
+
Caching
+
Model Routing
+
Bounded Concurrency
+
Observability
🧪 136. Practical Project¶
Build a:
Production RAG Performance Optimization Lab
Start with:
then progressively optimize it.
🧪 137. Baseline Architecture¶
Measure:
🧪 138. Optimization Stage 1¶
Add:
Measure:
🧪 139. Optimization Stage 2¶
Add:
Measure:
🧪 140. Optimization Stage 3¶
Add:
Measure:
🧪 141. Optimization Stage 4¶
Add:
Measure:
🧪 142. Optimization Stage 5¶
Add:
Measure:
🧪 143. Optimization Stage 6¶
Add:
Architecture:
🧪 144. Optimization Experiment Matrix¶
| Experiment | Latency | Recall | Tokens | Cost | Quality |
|---|---|---|---|---|---|
| Baseline | — | — | — | — | — |
| Cache | — | — | — | — | — |
| Parallel Retrieval | — | — | — | — | — |
| Reranking | — | — | — | — | — |
| Compression | — | — | — | — | — |
| Model Routing | — | — | — | — | — |
| Adaptive Retrieval | — | — | — | — | — |
Populate using actual benchmark results.
🧪 145. Performance Test Dataset¶
Create representative query categories:
Simple FAQ
Technical Query
Multi-hop Query
Ambiguous Query
Long Query
Short Query
SQL Query
Graph Query
No-Answer Query
High-Context Query
Do not benchmark using only easy questions.
🧪 146. Performance Test Harness¶
def benchmark(rag, queries):
results = []
for query in queries:
result = rag.answer(query)
results.append({
"query": query,
"latency_ms": result.latency_ms,
"input_tokens": result.input_tokens,
"output_tokens": result.output_tokens,
"cost": result.cost
})
return results
🧪 147. Performance Metrics¶
Calculate:
p50 latency
p95 latency
p99 latency
Average retrieval latency
Average reranking latency
Average generation latency
Average context tokens
Average input tokens
Average output tokens
Cost/request
Requests/second
🧪 148. Quality Metrics¶
Do not optimize without measuring:
🧪 149. Final Benchmark¶
The optimized system should answer:
Did latency improve?
Did throughput improve?
Did token usage decrease?
Did cost decrease?
Did retrieval quality remain acceptable?
Did answer quality remain acceptable?
Did citation quality remain acceptable?
Did infrastructure utilization improve?
🧠 150. Production Performance Checklist¶
☐ Measure end-to-end latency
☐ Measure p50
☐ Measure p95
☐ Measure p99
☐ Profile every RAG stage
☐ Identify critical path
☐ Remove unnecessary operations
☐ Parallelize independent retrieval
☐ Use bounded concurrency
☐ Optimize embedding
☐ Batch embeddings
☐ Cache embeddings
☐ Optimize vector indexes
☐ Tune ANN parameters
☐ Optimize vector DB connections
☐ Reduce retrieval payloads
☐ Tune Top-K
☐ Separate candidate K from final K
☐ Use adaptive retrieval
☐ Use score thresholds where appropriate
☐ Reduce reranker candidates
☐ Batch reranking
☐ Optimize hybrid retrieval
☐ Deduplicate context
☐ Compress context
☐ Set context budgets
☐ Optimize context ordering
☐ Remove irrelevant chunks
☐ Track context tokens
☐ Optimize prompts
☐ Version prompts
☐ Use prompt caching where appropriate
☐ Reduce repeated instructions
☐ Optimize LLM selection
☐ Use model routing
☐ Use model cascades where appropriate
☐ Stream responses
☐ Track TTFT
☐ Limit output tokens
☐ Cache retrieval results
☐ Cache reusable computations
☐ Version caches
☐ Implement invalidation
☐ Preserve tenant isolation
☐ Optimize ingestion
☐ Batch indexing
☐ Incrementally update indexes
☐ Avoid unnecessary re-embedding
☐ Use content hashes
☐ Control database connections
☐ Control concurrency
☐ Implement backpressure
☐ Implement rate limits
☐ Implement load shedding
☐ Separate workloads
☐ Configure autoscaling
☐ Load test
☐ Stress test
☐ Capacity test
☐ Regression test
☐ Canary performance changes
☐ Monitor latency
☐ Monitor throughput
☐ Monitor cost
☐ Monitor token usage
☐ Monitor retrieval quality
☐ Monitor answer quality
☐ Compare optimization experiments
☐ Preserve quality SLOs
☐ Document performance budgets
☐ Monitor production regressions
🧠 151. Performance Optimization Mental Model¶
RAG PERFORMANCE
│
┌──────────────────┼──────────────────┐
▼ ▼ ▼
LATENCY THROUGHPUT COST
│ │ │
▼ ▼ ▼
Critical Path Concurrency Tokens
Parallelism Batching Models
Caching Scaling Caching
│ │ │
└──────────────────┼──────────────────┘
▼
QUALITY
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Retrieval Generation Citation
Quality Quality Quality
│
▼
RELIABILITY
│
▼
PRODUCTION SLOs
🧠 152. Final Mental Model¶
The complete optimization loop is:
PRODUCTION RAG
│
▼
MEASURE
│
▼
PROFILE
│
▼
FIND BOTTLENECK
│
▼
REMOVE UNNECESSARY WORK
│
▼
PARALLELIZE WORK
│
▼
REDUCE DATA FLOW
│
▼
CACHE
│
▼
OPTIMIZE COMPONENTS
│
▼
OPTIMIZE MODEL ROUTING
│
▼
BENCHMARK QUALITY
│
▼
LOAD TEST
│
▼
DEPLOY
│
▼
OBSERVE
│
└───────────────►
The fundamental production principle is:
Do the minimum amount of computation necessary to produce the required quality within the required latency and cost budget.
📚 153. Key Takeaways¶
- RAG performance is broader than latency.
- Optimize latency, throughput, cost, scalability, and resource utilization together.
- Always measure before optimizing.
- Use distributed tracing to identify the real bottleneck.
- Monitor p50, p95, and p99 latency.
- Optimize the critical path.
- Parallelize independent retrieval operations.
- Bound concurrency to protect downstream services.
- Do not blindly increase Top-K.
- Separate retrieval candidates from final context.
- Use adaptive retrieval when appropriate.
- Use early exits when confidence is sufficient.
- Query rewriting should be conditional when possible.
- Multi-query retrieval should be used selectively.
- Batch embedding operations.
- Cache repeated query embeddings.
- Optimize vector indexes according to recall and latency requirements.
- ANN indexes trade some exactness for performance.
- HNSW search parameters require workload-specific tuning.
- Quantization can reduce memory and latency but requires quality benchmarking.
- Reduce retrieval payload sizes.
- Use connection pooling.
- Parallelize dense and sparse retrieval.
- Reduce reranking candidates before expensive reranking.
- Batch reranking when supported.
- Context optimization is one of the most important RAG performance techniques.
- Remove duplicate and irrelevant context.
- Use context budgets.
- Context compression can reduce token usage and latency.
- Optimize prompt size.
- Track prompt versions.
- Use prompt caching where appropriate.
- Track LLM time to first token separately from total generation time.
- Use model routing when different queries have different complexity.
- Streaming improves perceived responsiveness.
- Limit unnecessary output tokens.
- Use selective validation for appropriate workloads.
- Preserve source identity throughout the pipeline for efficient citation.
- Caching can significantly reduce repeated work.
- Cache invalidation and versioning are essential.
- Semantic caching requires careful handling of freshness and authorization.
- Batch ingestion and indexing.
- Use incremental indexing instead of rebuilding everything.
- Avoid unnecessary re-embedding.
- Use content hashes for change detection.
- Control vector database connections and concurrency.
- Use backpressure during overload.
- Separate interactive, batch, ingestion, and evaluation workloads.
- Use autoscaling based on actual workload characteristics.
- Network locality can materially affect RAG latency.
- Observability itself has a performance and cost footprint.
- Performance optimization must preserve retrieval and answer quality.
- Quality, latency, and cost form a continuous engineering trade-off.
- Benchmark every significant optimization.
- Use canary deployment for high-impact performance changes.
- Performance regression testing should become part of the production lifecycle.
- The best optimization is often eliminating unnecessary work.
🧭 154. Chapter Navigation¶
Part V — Advanced Retrieval-Augmented Generation¶
Previous:
07. RAG Observability
Next:
09. RAG Cost Optimization
Section:
06 — Production RAG Engineering
Production RAG Engineering Path¶
01 Prompt Assembly
↓
02 Context Selection & Context Engineering
↓
03 Response Validation
↓
04 Citation & Source Attribution
↓
05 Enterprise Response
↓
06 RAG Evaluation & Benchmarking
↓
07 RAG Observability
↓
08 RAG Performance Optimization
↓
09 RAG Cost Optimization
↓
10 Production Retrieval Architecture
↓
11 Building Production RAG Systems
Enterprise AI Engineering Handbook
Building Production-Grade Enterprise AI Systems — One Chapter at a Time.