09. RAG Cost Optimization¶
Category: Production RAG Engineering
Module: Part V — Advanced Retrieval-Augmented Generation
Difficulty: Advanced
📖 Overview¶
A production RAG system must be optimized not only for accuracy and latency, but also for economic efficiency.
A system that produces excellent answers but costs:
may not be viable when serving:
Similarly, a system that is inexpensive but produces poor answers creates operational and business risk.
Production RAG cost optimization therefore focuses on the complete cost chain:
User Query
↓
Query Processing
↓
Embedding
↓
Retrieval
↓
Reranking
↓
Context Processing
↓
Prompt Assembly
↓
LLM Generation
↓
Validation
↓
Citation
↓
Observability
The objective is:
A useful production principle is:
Do not minimize cost blindly. Minimize the cost of achieving the required quality, latency, and reliability.
🎯 Learning Objectives¶
After completing this chapter, you will be able to:
- Understand RAG cost architecture
- Identify major RAG cost drivers
- Calculate cost per request
- Calculate cost per user
- Calculate cost per tenant
- Calculate cost per workflow
- Understand LLM token economics
- Optimize input token usage
- Optimize output token usage
- Optimize retrieval costs
- Optimize embedding costs
- Optimize reranking costs
- Optimize validation costs
- Optimize agentic RAG costs
- Optimize Graph RAG costs
- Optimize SQL RAG costs
- Optimize multimodal RAG costs
- Implement caching strategies
- Implement model routing
- Implement model cascading
- Implement adaptive retrieval
- Reduce unnecessary LLM calls
- Optimize context size
- Optimize prompt size
- Optimize infrastructure costs
- Optimize vector database costs
- Optimize observability costs
- Implement cost budgets
- Implement cost guardrails
- Implement tenant-level cost controls
- Build cost dashboards
- Detect cost anomalies
- Perform cost attribution
- Perform cost forecasting
- Design cost-aware RAG architectures
🧠 1. What Is RAG Cost Optimization?¶
RAG cost optimization is the process of reducing the resources required to serve RAG requests while preserving acceptable:
A simplified objective is:
🧠 2. RAG Cost Is More Than LLM Cost¶
A common mistake is:
In reality:
Total RAG Cost
│
├── LLM
├── Embeddings
├── Reranking
├── Vector Database
├── Search Infrastructure
├── Compute
├── Storage
├── Network
├── Observability
├── Evaluation
└── Background Processing
🧠 3. Cost Architecture¶
flowchart TD
A["RAG Request"] --> B["Query Processing"]
B --> C["Embedding"]
C --> D["Retrieval"]
D --> E["Reranking"]
E --> F["Context Processing"]
F --> G["LLM"]
G --> H["Validation"]
H --> I["Citation"]
C --> J["Embedding Cost"]
D --> K["Vector/Search Cost"]
E --> L["Reranker Cost"]
G --> M["LLM Cost"]
H --> N["Validation Cost"]
I --> O["Processing Cost"]
P["Infrastructure"] --> Q["Compute"]
P --> R["Storage"]
P --> S["Network"]
P --> T["Observability"]🧠 4. Cost Categories¶
A practical classification:
Variable Cost
↓
Cost changes with requests/tokens
Fixed Cost
↓
Infrastructure that exists regardless of request volume
Semi-Variable Cost
↓
Resources that scale with workload
Examples:
Variable¶
Fixed¶
Semi-Variable¶
🧠 5. Cost Per Request¶
A simplified model:
C_request =
C_embedding
+ C_retrieval
+ C_reranking
+ C_context
+ C_generation
+ C_validation
+ C_observability
+ C_infrastructure
🧠 6. LLM Cost¶
LLM cost is commonly driven by:
Conceptually:
Actual pricing varies by provider, model, region, and pricing program.
🧠 7. RAG Token Composition¶
A request may contain:
Therefore:
🧠 8. Context Is Often the Largest Optimization Opportunity¶
Example:
System Prompt 1,000 tokens
User Query 100 tokens
Conversation 900 tokens
Retrieved Context 8,000 tokens
───────────────────────────────
Input 10,000 tokens
If the context is reduced:
the input token cost can fall significantly.
🧠 9. Context Cost¶
The goal is not:
but:
🧠 10. Context Efficiency¶
A useful conceptual metric:
Higher is generally better.
This is not a universal standardized metric; use it as an engineering diagnostic.
🧠 11. Context Waste¶
Example:
The system is paying for:
while deriving useful information from approximately:
🧠 12. Context Optimization¶
Use:
Top-K Tuning
+
Reranking
+
MMR
+
Context Compression
+
Deduplication
+
Metadata Filtering
+
Token Budgeting
🧠 13. Token Budget¶
Define:
Example:
Then:
🧠 14. Token Budget Architecture¶
flowchart LR
A["Model Context Window"] --> B["System Instructions"]
A --> C["User Query"]
A --> D["Conversation"]
A --> E["Retrieved Context"]
A --> F["Output Budget"]
E --> G["Context Budget"]
G --> H["Relevant Evidence"]🧠 15. Dynamic Context Budget¶
Not every query needs the same amount of context.
Use adaptive budgets where appropriate.
🧠 16. Cost of Top-K¶
Increasing K can increase:
Example:
The quality improvement may not justify the additional cost.
🧠 17. Cost-Aware Top-K¶
Instead of:
use:
🧠 18. Adaptive Retrieval¶
Query
↓
Retrieve Small Candidate Set
↓
Confidence Check
│
├── High Confidence → Generate
│
└── Low Confidence → Expand Retrieval
This reduces expensive work for easy queries.
🧠 19. Early Exit¶
Example:
Initial Retrieval
↓
Top Result Score = 0.94
↓
Evidence Sufficient?
│
├── Yes → Generate
└── No → Expand
Use calibrated signals rather than arbitrary score thresholds.
🧠 20. Query Rewriting Cost¶
Query rewriting may require an LLM.
If rewriting costs:
and is executed:
the rewriting layer alone contributes:
before the main generation cost.
🧠 21. Conditional Query Rewriting¶
This can significantly reduce unnecessary model calls.
🧠 22. Multi-Query Cost¶
Multi-query:
may multiply:
Use it when the quality improvement justifies the additional cost.
🧠 23. Multi-Query Cost Optimization¶
🧠 24. Embedding Cost¶
Embedding cost can come from:
Query embeddings happen frequently.
Document embeddings can become expensive during large ingestion operations.
🧠 25. Embedding Cost Optimization¶
Use:
where quality requirements permit.
🧠 26. Avoid Re-Embedding Unchanged Documents¶
Use content hashes:
Pipeline:
🧠 27. Incremental Indexing¶
Bad:
Better:
🧠 28. Batch Embedding¶
Batching can reduce per-request overhead and improve accelerator utilization.
🧠 29. Embedding Model Economics¶
Compare:
Model A
Quality = 91%
Cost = Low
Model B
Quality = 94%
Cost = Medium
Model C
Quality = 96%
Cost = High
Choose based on:
🧠 30. Local vs Hosted Embeddings¶
Hosted¶
Costs may include:
Local¶
Costs shift toward:
Neither is universally cheaper.
🧠 31. Reranking Cost¶
Rerankers can be expensive because they may evaluate many query-document pairs.
Conceptually:
🧠 32. Candidate Reduction¶
Instead of:
use:
🧠 33. Reranker Selection¶
Possible options:
Use the least expensive approach that meets the quality requirement.
🧠 34. Selective Reranking¶
This can reduce average cost.
🧠 35. LLM Cost¶
LLM cost often dominates production RAG economics.
RAG Request
│
▼
┌──────────────────────────┐
│ LLM COST │
├──────────────────────────┤
│ Input Tokens │
│ Output Tokens │
│ Number of Calls │
│ Model Selection │
│ Retries │
│ Validation Calls │
│ Agent Tool Calls │
└──────────────────────────┘
🧠 36. Reduce Number of LLM Calls¶
One of the strongest cost optimizations:
Example:
Query
↓
Classifier
│
├── FAQ → Cached Answer
├── Search → Retrieval + Small LLM
├── Complex → Large LLM
└── SQL → SQL Pipeline
🧠 37. Model Routing¶
flowchart TD
A["Query"] --> B["Model Router"]
B --> C["Small Model"]
B --> D["Medium Model"]
B --> E["Large Model"]
C --> F["Response"]
D --> F
E --> FThe router should consider:
🧠 38. Model Cascade¶
Average cost can decrease if most queries are successfully handled by the smaller model.
🧠 39. Model Routing Example¶
🧠 40. Model Routing Economics¶
Suppose:
instead of:
the average cost can be substantially lower, assuming quality remains acceptable.
🧠 41. Output Token Optimization¶
Output tokens directly affect:
Use:
🧠 42. Output Budget¶
Do not allocate the maximum output budget to every request.
🧠 43. Response Contract¶
Example:
This reduces unnecessary output.
🧠 44. Prompt Optimization¶
Prompt cost includes:
Reduce:
🧠 45. Prompt Versioning¶
Use:
Measure:
A shorter prompt is not automatically better if quality drops.
🧠 46. Prompt Caching¶
If supported by the model/provider:
Potentially reduces:
🧠 47. Conversation Cost¶
Conversational RAG can become expensive because history grows.
Turn 1 → 500 tokens
Turn 2 → 1,200 tokens
Turn 3 → 2,500 tokens
Turn 4 → 4,500 tokens
Turn 5 → 7,000 tokens
🧠 48. Conversation Compression¶
Instead of passing all history:
🧠 49. Memory Budget¶
Set:
Use:
rather than blindly sending the complete conversation.
🧠 50. Semantic Conversation Selection¶
Retrieve only relevant historical messages:
This reduces token usage.
🧠 51. Retrieval Cache¶
Cache retrieval results for repeated requests.
Key should account for:
🧠 52. Cache Economics¶
If:
and:
then:
may avoid some downstream computation.
Actual savings depend on what the cache bypasses.
🧠 53. Semantic Cache¶
Example:
Potentially reuse the result.
But semantic caching must validate:
🧠 54. Cache Invalidation¶
Invalidate when:
Document Changes
Index Changes
Embedding Changes
Prompt Changes
Retriever Changes
Authorization Changes
🧠 55. Cache Versioning¶
Example:
This prevents stale results from older pipeline versions.
🧠 56. Tenant Cost Attribution¶
Enterprise RAG should answer:
How much does Tenant A cost?
How much does Tenant B cost?
Which tenant consumes the most tokens?
Which tenant generates the most expensive requests?
🧠 57. Cost Per Tenant¶
Example:
This enables:
🧠 58. Cost Per User¶
Track:
Avoid exposing user-level data broadly; apply appropriate privacy and access controls.
🧠 59. Cost by Application¶
Enterprise platforms may host:
Track cost separately.
🧠 60. Cost by Workflow¶
A single application may have:
Each can have a different cost profile.
🧠 61. Cost by Model¶
Track:
with:
🧠 62. Cost by Provider¶
For multi-cloud enterprise systems:
compare:
🧠 63. Reranking Cost by Query¶
Some queries may require:
others:
and complex queries:
Use query-aware policies.
🧠 64. Agentic RAG Cost¶
Agentic RAG can multiply cost.
Potentially:
🧠 65. Agentic RAG Cost Guardrails¶
Define:
Example:
🧠 66. Agentic Early Termination¶
Avoid unnecessary planning loops.
🧠 67. Agentic Loop Detection¶
Potential problem:
Use:
🧠 68. Graph RAG Cost¶
Graph RAG may require:
Cost can increase with traversal depth.
🧠 69. Graph Traversal Budget¶
Instead of:
use:
🧠 70. SQL RAG Cost¶
SQL RAG can generate expensive queries.
Potential risks:
Use:
🧠 71. SQL Result Budget¶
Instead of:
use:
Only return data required for reasoning.
🧠 72. Multimodal RAG Cost¶
Multimodal workloads may involve:
Optimize using:
🧠 73. Image Processing Cost¶
Avoid processing every image at maximum resolution.
🧠 74. Validation Cost¶
If every answer invokes:
the cost can multiply.
Possible alternatives:
🧠 75. Risk-Based Validation¶
flowchart TD
A["Generated Response"] --> B["Risk Classifier"]
B --> C["Low Risk"]
B --> D["Medium Risk"]
B --> E["High Risk"]
C --> F["Rule Validation"]
D --> G["Lightweight Validation"]
E --> H["Deep Validation"]
F --> I["Final Response"]
G --> I
H --> I🧠 76. Cost-Aware Citation¶
Citations should ideally be produced using source metadata already carried through the pipeline.
Avoid unnecessary additional LLM calls just to reconstruct sources.
🧠 77. Infrastructure Cost¶
Infrastructure includes:
RAG Compute
Vector DB
Search Engine
Object Storage
Databases
GPU
Load Balancers
Network
Observability
🧠 78. Compute Optimization¶
Optimize:
🧠 79. CPU vs GPU¶
Use GPU when:
CPU may be more appropriate for:
🧠 80. GPU Utilization¶
Poor:
while paying for a large GPU.
Potential solutions:
🧠 81. Autoscaling¶
Scale based on:
🧠 82. Scale-to-Zero¶
For workloads that are:
scale-to-zero or scheduled compute can reduce infrastructure cost where supported.
🧠 83. Vector Database Cost¶
Vector DB cost depends on:
🧠 84. Vector Storage Optimization¶
Reduce:
where quality and operational requirements permit.
🧠 85. Data Lifecycle¶
Implement:
Not every document needs identical storage characteristics.
🧠 86. Document Retention¶
Enterprise knowledge bases may contain:
Remove expired data from the active retrieval path when policy allows.
🧠 87. Storage Tiering¶
Retrieve cold data only when required.
🧠 88. Network Cost¶
Network costs may come from:
Reduce unnecessary payloads and cross-region communication.
🧠 89. Region Optimization¶
Choose infrastructure locations based on:
Do not optimize cost by violating data residency or regulatory requirements.
🧠 90. Observability Cost¶
Observability can become expensive when capturing:
Use:
🧠 91. Evaluation Cost¶
RAG evaluation can itself consume LLM calls.
Large evaluation suites can become expensive.
🧠 92. Evaluation Sampling¶
Instead of evaluating:
consider:
The right sampling policy depends on the application's risk.
🧠 93. Continuous Evaluation Cost¶
Use:
rather than evaluating every request with expensive models.
🧠 94. Cost-Aware Evaluation¶
Prioritize:
🧠 95. Cost Anomaly Detection¶
Monitor:
🧠 96. Cost Spike Example¶
Investigate:
🧠 97. Cost Dashboard¶
┌──────────────────────────────────────────────┐
│ RAG COST DASHBOARD │
├──────────────────────────────────────────────┤
│ Requests/day 120,000 │
│ Avg Cost/request $0.018 │
│ Daily Cost $2,160 │
│ Monthly Forecast $64,800 │
│ │
│ LLM 69% │
│ Embeddings 11% │
│ Reranking 7% │
│ Vector DB 6% │
│ Infrastructure 4% │
│ Observability 3% │
│ │
│ Cache Hit Rate 31% │
│ Avg Context Tokens 3,900 │
│ Avg Output Tokens 420 │
└──────────────────────────────────────────────┘
Values are illustrative.
🧠 98. Cost Breakdown¶
TOTAL COST
│
┌────────────────┼────────────────┐
▼ ▼ ▼
AI DATA PLATFORM
│ │ │
LLM Tokens Vector DB Compute
Embeddings Storage Network
Reranker Search Observability
Evaluation Object Storage
🧠 99. Cost per Request Formula¶
If:
then:
🧠 100. Monthly Cost Forecast¶
A simple estimate:
Example:
This is a simple forecast and does not account for traffic growth or tiered pricing.
🧠 101. Cost Forecasting¶
Forecast using:
Example:
But cost may not exactly double if:
🧠 102. Cost Elasticity¶
Measure:
A highly elastic architecture may scale cost almost linearly.
An optimized architecture can sometimes achieve:
through:
🧠 103. Cost per Successful Answer¶
A more useful business metric than raw cost:
This captures quality.
🧠 104. Cost per High-Quality Answer¶
Example:
Then:
🧠 105. Quality-Adjusted Cost¶
Conceptually:
This can help compare architectures.
However, quality scores must be defined consistently.
🧠 106. Cost vs Quality¶
There is often a point of diminishing returns.
🧠 107. Diminishing Returns¶
Example:
The additional:
may not justify:
depending on the use case.
🧠 108. Cost Optimization Frontier¶
The goal is to operate near an efficient frontier rather than blindly choosing the cheapest or most accurate configuration.
🧠 109. Cost Guardrails¶
Production systems should have limits:
Max Cost / Request
Max Tokens / Request
Max Agent Iterations
Max Tool Calls
Max Retrieval Candidates
Max Context Tokens
🧠 110. Request Budget¶
Example:
Pipeline:
🧠 111. Budget-Aware Routing¶
If the request has already consumed:
a cheaper path may be selected.
🧠 112. Agent Cost Budget¶
Agent
│
├── Step 1 → $0.01
├── Step 2 → $0.02
├── Step 3 → $0.02
└── Step 4 → $0.03
│
▼
$0.08
│
▼
Budget = $0.10
Next expensive action may be blocked.
🧠 113. Tenant Budgets¶
Example:
Use:
according to business policy.
🧠 114. Soft vs Hard Budgets¶
Soft Budget¶
Hard Budget¶
🧠 115. Graceful Degradation¶
When budget is constrained:
or:
or:
🧠 116. Cost-Aware Architecture¶
flowchart TD
A["User Query"] --> B["Budget Manager"]
B --> C["Query Router"]
C --> D["Fast Path"]
C --> E["Standard RAG"]
C --> F["Advanced RAG"]
E --> G["Retrieval"]
F --> G
G --> H["Reranking"]
H --> I["Context Budget"]
I --> J["Model Router"]
J --> K["Small Model"]
J --> L["Large Model"]
K --> M["Validation"]
L --> M
M --> N["Response"]
B --> O["Cost Tracking"]
O --> P["Budget Enforcement"]🧠 117. Cost-Aware Retrieval¶
A production retriever should consider:
not only:
🧠 118. Cost-Aware Query Routing¶
Use the expensive route only when required.
🧠 119. Cost-Aware Validation¶
🧠 120. Cost-Aware Agentic RAG¶
Agent should understand:
Example:
🧠 121. Cost-Aware Graph RAG¶
Use:
to prevent runaway graph exploration.
🧠 122. Cost-Aware SQL RAG¶
Use:
where supported.
🧠 123. Cost-Aware Multimodal RAG¶
Use:
🧠 124. Cost-Aware Evaluation¶
Prioritize evaluation of:
🧠 125. Cost Optimization by Layer¶
Layer Optimization
Query Routing / Rewrite selectively
Embedding Cache / Batch
Retrieval Top-K / ANN
Hybrid Parallel / Candidate reduction
Reranking Smaller candidate set
Context Compression / Deduplication
Prompt Shorter / Cache
LLM Model routing
Validation Risk-based
Citation Metadata propagation
Agent Step budgets
Graph Traversal limits
SQL Query limits
Multimodal Resolution / routing
Infrastructure Autoscaling
Observability Sampling
🧠 126. Cost Optimization Priority¶
A practical sequence:
1. Measure total cost
2. Identify largest cost component
3. Reduce unnecessary work
4. Reduce token volume
5. Reduce number of model calls
6. Add caching
7. Introduce model routing
8. Optimize retrieval
9. Optimize infrastructure
10. Add budget guardrails
11. Continuously benchmark
🧠 127. Cost Optimization Example¶
Baseline:
Optimized:
Top-K = Adaptive
Reranker = Selective
Context = 4,000 tokens
LLM = Routed
Validation = Risk-Based
Cache = Enabled
🧠 128. Example Cost Comparison¶
| Configuration | Tokens | LLM Calls | Avg Cost | p95 Latency |
|---|---|---|---|---|
| Baseline | 8,500 | 3 | $0.052 | 2.8s |
| Context Optimized | 5,000 | 3 | $0.035 | 2.2s |
| Model Routing | 5,000 | 2 | $0.021 | 1.7s |
| Cached + Routed | 5,000 | 1.4 avg | $0.015 | 1.3s |
Values are illustrative.
🧠 129. Cost Optimization Experiment¶
Hypothesis:
Measure:
🧠 130. Cost Experiment Matrix¶
| Experiment | Cost | Latency | Quality | Decision |
|---|---|---|---|---|
| Baseline | — | — | — | — |
| Context Reduction | — | — | — | — |
| Smaller Model | — | — | — | — |
| Caching | — | — | — | — |
| Selective Reranking | — | — | — | — |
| Model Routing | — | — | — | — |
| Adaptive Retrieval | — | — | — | — |
Populate using real benchmarks.
🧪 131. Practical Project¶
Build a:
Production RAG Cost Optimization Lab
Start with a baseline RAG system and progressively optimize:
🧪 132. Baseline Project¶
Measure:
🧪 133. Optimization Stage 1 — Context¶
Change:
to:
Measure:
🧪 134. Optimization Stage 2 — Reranking¶
Change:
to:
Measure:
🧪 135. Optimization Stage 3 — Model Routing¶
Measure:
🧪 136. Optimization Stage 4 — Caching¶
Add:
Measure:
🧪 137. Optimization Stage 5 — Adaptive Retrieval¶
Measure:
🧪 138. Optimization Stage 6 — Budget Guardrails¶
Implement:
Test:
🧪 139. Cost Test Dataset¶
Include:
Simple Queries
Complex Queries
Long Queries
Multi-Hop Queries
No-Answer Queries
Repeated Queries
Ambiguous Queries
SQL Queries
Graph Queries
Multimodal Queries
Agentic Queries
🧪 140. Cost Benchmark Harness¶
def benchmark_cost(rag, queries):
results = []
for query in queries:
result = rag.answer(query)
results.append({
"query": query,
"cost": result.cost,
"latency_ms": result.latency_ms,
"input_tokens": result.input_tokens,
"output_tokens": result.output_tokens,
"llm_calls": result.llm_calls
})
return results
🧪 141. Cost Metrics¶
Track:
Cost / Request
Cost / Successful Answer
Cost / High-Quality Answer
Cost / Tenant
Cost / User
Cost / Workflow
Cost / Model
Cost / Provider
🧪 142. Token Metrics¶
Track:
🧪 143. Model Metrics¶
Track:
🧪 144. Cache Metrics¶
Track:
🧪 145. Budget Metrics¶
Track:
🧪 146. Cost Dashboard¶
A production dashboard should show:
┌────────────────────────────────────────────┐
│ COST OVERVIEW │
├────────────────────────────────────────────┤
│ Daily Cost $2,160 │
│ Monthly Forecast $64,800 │
│ Cost / Request $0.018 │
│ Cost / Success $0.021 │
│ │
│ Input Tokens 4,100 │
│ Output Tokens 390 │
│ Cache Hit Rate 31% │
│ │
│ LLM 69% │
│ Embedding 11% │
│ Reranking 7% │
│ Vector DB 6% │
│ Infra 4% │
│ Observability 3% │
└────────────────────────────────────────────┘
Illustrative values.
🧪 147. Tenant Cost Dashboard¶
This enables chargeback and optimization.
🧪 148. Cost Anomaly Detection¶
Alert when:
or:
or:
🧪 149. Example Cost Alert¶
ALERT: RAG cost anomaly
Average cost/request:
$0.018
Current:
$0.029
Increase:
61%
Primary signal:
Context tokens +72%
Possible cause:
Top-K configuration changed.
🧠 150. Cost Governance¶
Enterprise RAG should define:
🧠 151. Model Governance¶
Define:
🧠 152. Cost Governance by Risk¶
Different workloads can have different budgets:
🧠 153. FinOps for RAG¶
RAG can adopt FinOps principles:
🧠 154. RAG FinOps Architecture¶
flowchart TD
A["RAG Usage"] --> B["Telemetry"]
B --> C["Cost Attribution"]
C --> D["Tenant"]
C --> E["Application"]
C --> F["Model"]
C --> G["Workflow"]
D --> H["Budgets"]
E --> H
F --> H
G --> H
H --> I["Optimization"]
I --> J["Routing"]
I --> K["Caching"]
I --> L["Context Optimization"]
I --> M["Infrastructure Optimization"]
J --> N["Lower Cost"]
K --> N
L --> N
M --> N🧠 155. Production Cost Optimization Architecture¶
USER
│
▼
┌─────────────┐
│ RAG API │
└──────┬──────┘
│
▼
┌─────────────────┐
│ Budget Manager │
└────────┬────────┘
│
▼
Query Router
│
┌────────────┼────────────┐
▼ ▼ ▼
Fast Path Standard Advanced
│ │ │
└────────────┼────────────┘
▼
Retrieval
│
▼
Reranking
│
▼
Context Budget
│
▼
Model Router
│
┌────────┴────────┐
▼ ▼
Small LLM Large LLM
│ │
└────────┬────────┘
▼
Validation
│
▼
Citation
│
▼
Response
│
▼
Cost Telemetry
│
┌────────────┼────────────┐
▼ ▼ ▼
Tenant Model Workflow
Cost Cost Cost
│ │ │
└────────────┼────────────┘
▼
Cost Dashboard
│
▼
Optimization
🧠 156. Cost Optimization Maturity¶
Level 1 — Basic Visibility¶
Level 2 — Token Visibility¶
Level 3 — Component Cost¶
Level 4 — Cost Attribution¶
Level 5 — Cost Controls¶
Level 6 — Cost-Aware RAG¶
Level 7 — Autonomous Optimization¶
🧠 157. Cost Optimization Mental Model¶
RAG COST
│
┌────────────────────┼────────────────────┐
▼ ▼ ▼
AI DATA PLATFORM
│ │ │
LLM Vector DB Compute
Embedding Storage Network
Reranker Search Observability
Evaluation
│ │ │
└────────────────────┼────────────────────┘
▼
COST CONTROL
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Reduce Reuse Route
Work Work Work
│ │ │
└─────────────┼─────────────┘
▼
COST / QUALITY
🧠 158. Final Cost Optimization Loop¶
Measure
↓
Attribute
↓
Find Largest Cost Driver
↓
Remove Unnecessary Work
↓
Reduce Tokens
↓
Reduce Model Calls
↓
Cache
↓
Route
↓
Optimize Infrastructure
↓
Apply Budgets
↓
Benchmark Quality
↓
Deploy
↓
Monitor
↓
Repeat
🧠 159. Production Principles¶
Principle 1¶
The cheapest request is the request you do not need to execute.
Use:
Principle 2¶
The second-cheapest request is the one executed with the smallest suitable model.
Principle 3¶
Context is a cost center.
Do not treat retrieved tokens as free.
Principle 4¶
Every additional RAG stage has an economic cost.
Before adding:
measure the quality improvement.
Principle 5¶
Optimize cost per successful answer, not cost per request alone.
🧠 160. Production RAG Cost Checklist¶
☐ Calculate cost/request
☐ Calculate cost/successful answer
☐ Calculate cost/high-quality answer
☐ Track LLM input tokens
☐ Track LLM output tokens
☐ Track context tokens
☐ Track conversation tokens
☐ Track embedding cost
☐ Track reranking cost
☐ Track retrieval infrastructure
☐ Track vector DB cost
☐ Track compute cost
☐ Track network cost
☐ Track observability cost
☐ Track evaluation cost
☐ Optimize context size
☐ Optimize Top-K
☐ Optimize reranking candidates
☐ Optimize query rewriting
☐ Optimize multi-query
☐ Batch embeddings
☐ Cache embeddings
☐ Incrementally embed documents
☐ Use content hashes
☐ Optimize vector indexes
☐ Cache retrieval
☐ Implement semantic cache where appropriate
☐ Version caches
☐ Implement cache invalidation
☐ Preserve tenant isolation
☐ Implement model routing
☐ Implement model cascades
☐ Use smaller models where appropriate
☐ Limit output tokens
☐ Optimize prompts
☐ Use prompt caching where appropriate
☐ Implement selective validation
☐ Optimize citation processing
☐ Limit agent iterations
☐ Limit agent tool calls
☐ Limit agent token budgets
☐ Limit Graph RAG traversal
☐ Limit SQL result sets
☐ Optimize multimodal processing
☐ Optimize compute
☐ Optimize GPU utilization
☐ Optimize vector DB
☐ Optimize storage
☐ Optimize network
☐ Implement autoscaling
☐ Separate workloads
☐ Implement backpressure
☐ Implement tenant budgets
☐ Implement application budgets
☐ Implement workflow budgets
☐ Implement request budgets
☐ Implement soft limits
☐ Implement hard limits
☐ Implement graceful degradation
☐ Build cost dashboards
☐ Build tenant dashboards
☐ Build model dashboards
☐ Build workflow dashboards
☐ Detect anomalies
☐ Forecast costs
☐ Benchmark optimizations
☐ Monitor quality regressions
☐ Review cost continuously
📚 161. Key Takeaways¶
- RAG cost is broader than LLM API cost.
- Cost should be measured across the entire architecture.
- LLM tokens are often a major cost driver.
- Context size is one of the most important optimization opportunities.
- Reduce unnecessary context before reducing model quality.
- Do not blindly increase Top-K.
- Use adaptive retrieval where appropriate.
- Query rewriting should be conditional when possible.
- Multi-query retrieval should justify its additional cost.
- Batch embedding operations.
- Cache repeated embeddings.
- Avoid re-embedding unchanged documents.
- Use incremental indexing.
- Use content hashes for change detection.
- Reranking cost grows with candidate volume.
- Reduce candidates before expensive reranking.
- Selective reranking can reduce cost.
- Model routing can significantly reduce average LLM cost.
- Model cascades can use expensive models only when required.
- Output token limits reduce both cost and latency.
- Prompt optimization reduces unnecessary input tokens.
- Conversation compression controls growing history cost.
- Retrieval caching can avoid repeated downstream computation.
- Semantic caching requires careful freshness and authorization controls.
- Cache versioning is essential.
- Tenant isolation must apply to caches.
- Agentic RAG requires explicit cost budgets.
- Agent loops must have limits.
- Graph traversal should have bounded depth and result size.
- SQL RAG requires execution and result-size controls.
- Multimodal RAG requires image and vision cost controls.
- Validation should be risk-based where appropriate.
- Citation processing should reuse source metadata.
- Infrastructure cost matters alongside model cost.
- Vector DB cost depends on storage, compute, replication, and workload.
- Autoscaling can reduce idle infrastructure costs.
- Storage tiering can reduce long-term knowledge-base costs.
- Observability can itself become a significant cost center.
- Evaluation should be sampled intelligently where appropriate.
- Cost should be attributable by tenant, application, model, and workflow.
- Cost budgets provide predictable governance.
- Cost guardrails protect against runaway agentic or high-token requests.
- Graceful degradation allows systems to remain useful under budget constraints.
- RAG FinOps combines visibility, allocation, optimization, and governance.
- Cost optimization must preserve quality and reliability.
- The correct target is not minimum cost.
- The correct target is minimum cost for the required production quality and service level.
🧭 162. Chapter Navigation¶
Part V — Advanced Retrieval-Augmented Generation¶
Previous:
08. RAG Performance Optimization
Next:
10. Production Retrieval Architecture
Section:
06 — Production RAG Engineering
Production RAG Engineering Path¶
01 Prompt Assembly
↓
02 Context Selection & Context Engineering
↓
03 Response Validation
↓
04 Citation & Source Attribution
↓
05 Enterprise Response
↓
06 RAG Evaluation & Benchmarking
↓
07 RAG Observability
↓
08 RAG Performance Optimization
↓
09 RAG Cost Optimization
↓
10 Production Retrieval Architecture
↓
11 Building Production RAG Systems
Enterprise AI Engineering Handbook
Building Production-Grade Enterprise AI Systems — One Chapter at a Time.