10. Production Retrieval Architecture¶
Category: Production RAG Engineering
Module: Part V — Advanced Retrieval-Augmented Generation
Difficulty: Advanced
📖 Overview¶
Production Retrieval-Augmented Generation systems are fundamentally different from prototype RAG applications.
A prototype may look like:
A production retrieval architecture must address:
Accuracy
Scalability
Latency
Cost
Security
Multi-Tenancy
Freshness
Reliability
Observability
Failure Recovery
Versioning
Evaluation
Governance
The retrieval layer therefore becomes a dedicated engineering subsystem rather than a single vector database call.
A mature production architecture typically separates:
Query Understanding
↓
Retrieval Orchestration
↓
Candidate Generation
↓
Filtering
↓
Ranking
↓
Context Selection
↓
Evidence Management
↓
Generation
The goal is to build a retrieval platform that can evolve independently from the LLM and application layers.
Production RAG is not "LLM + Vector Database". It is a distributed knowledge retrieval system with explicit contracts, policies, observability, and operational controls.
🎯 Learning Objectives¶
After completing this chapter, you will be able to:
- Design production-grade retrieval architectures
- Separate retrieval from generation
- Design retrieval service boundaries
- Design retrieval orchestration layers
- Build pluggable retriever architectures
- Design dense retrieval pipelines
- Design sparse retrieval pipelines
- Design hybrid retrieval architectures
- Design multi-stage retrieval
- Integrate reranking
- Implement metadata-aware retrieval
- Implement tenant-aware retrieval
- Design authorization-aware retrieval
- Design retrieval caching
- Design retrieval fallback strategies
- Design retrieval timeouts
- Design retrieval circuit breakers
- Design retrieval observability
- Design retrieval SLOs
- Design scalable vector search infrastructure
- Design retrieval APIs
- Version indexes and retrieval pipelines
- Handle knowledge-base updates
- Support incremental indexing
- Design disaster recovery
- Design multi-region retrieval
- Design high-availability retrieval systems
- Build cloud-native retrieval architectures
- Design retrieval for enterprise security
- Design retrieval quality gates
- Build production-ready RAG retrieval platforms
🧠 1. Prototype vs Production RAG¶
Prototype¶
Production¶
User
↓
API Gateway
↓
Authentication
↓
Tenant Context
↓
Query Understanding
↓
Retrieval Router
↓
Parallel Retrieval
↓
Metadata / ACL Filtering
↓
Candidate Fusion
↓
Reranking
↓
Context Selection
↓
Evidence Packaging
↓
LLM
↓
Validation
↓
Citation
↓
Response
🧠 2. Production Retrieval as a Platform¶
Instead of embedding retrieval logic directly inside every application:
The retrieval platform provides reusable capabilities:
🧠 3. High-Level Architecture¶
flowchart TD
A["Client"] --> B["API Gateway"]
B --> C["RAG Application"]
C --> D["Query Understanding"]
D --> E["Retrieval Orchestrator"]
E --> F["Dense Retriever"]
E --> G["Sparse Retriever"]
E --> H["SQL Retriever"]
E --> I["Graph Retriever"]
F --> J["Candidate Fusion"]
G --> J
H --> J
I --> J
J --> K["Metadata / ACL Filtering"]
K --> L["Reranker"]
L --> M["Context Selector"]
M --> N["Evidence Package"]
N --> O["LLM"]
O --> P["Response Validation"]
P --> Q["Citation Builder"]
Q --> R["Response"]🧠 4. Retrieval Architecture Layers¶
A production system can be divided into:
1. Access Layer
2. Query Layer
3. Retrieval Layer
4. Ranking Layer
5. Context Layer
6. Generation Layer
7. Governance Layer
8. Observability Layer
9. Data / Indexing Layer
10. Infrastructure Layer
🧠 5. Access Layer¶
Responsibilities:
Example:
🧠 6. Tenant Context¶
A production request should carry tenant information:
{
"tenant_id": "enterprise-a",
"user_id": "user-123",
"roles": [
"engineering"
],
"request_id": "req-789"
}
This context must be propagated into retrieval.
🧠 7. Why Tenant Context Matters¶
Without tenant isolation:
This creates a critical information-isolation failure.
Tenant context must influence:
🧠 8. Query Layer¶
Responsibilities include:
Query Normalization
Intent Classification
Query Rewriting
Query Expansion
Query Routing
Language Detection
Security Checks
🧠 9. Query Understanding¶
flowchart LR
A["User Query"] --> B["Normalization"]
B --> C["Intent Detection"]
C --> D["Query Classification"]
D --> E["Retrieval Strategy"]🧠 10. Query Classification¶
Possible categories:
🧠 11. Retrieval Orchestrator¶
The orchestrator decides:
Which retrievers?
How many candidates?
Which filters?
Which ranking strategy?
Which reranker?
Which context budget?
Which fallback?
Example:
🧠 12. Retrieval Contract¶
Every retriever should expose a consistent contract.
from dataclasses import dataclass
from typing import Any
@dataclass
class RetrievalRequest:
query: str
top_k: int
tenant_id: str
filters: dict[str, Any]
@dataclass
class RetrievalResult:
document_id: str
chunk_id: str
text: str
score: float
metadata: dict[str, Any]
🧠 13. Retriever Interface¶
from abc import ABC, abstractmethod
class Retriever(ABC):
@abstractmethod
async def retrieve(
self,
request: RetrievalRequest
) -> list[RetrievalResult]:
pass
This enables:
to share the same application-level contract.
🧠 14. Capability-Based Retrieval¶
A production architecture can expose capabilities rather than coupling applications to specific vendors.
Retriever
├── SemanticSearch
├── KeywordSearch
├── HybridSearch
├── MetadataFiltering
├── Reranking
└── ContextSelection
This makes infrastructure replacement easier.
🧠 15. Retrieval Adapters¶
Application
↓
Retriever Interface
↓
Provider Adapter
├── FAISS
├── Chroma
├── Milvus
├── Elasticsearch
└── Cloud Search
The application should not depend directly on vendor-specific APIs.
🧠 16. Ports and Adapters¶
flowchart LR
A["RAG Application"] --> B["Retriever Port"]
B --> C["FAISS Adapter"]
B --> D["Milvus Adapter"]
B --> E["Elasticsearch Adapter"]
B --> F["Cloud Search Adapter"]This keeps retrieval infrastructure replaceable.
🧠 17. Candidate Generation¶
The first retrieval stage should favor:
rather than perfect ranking.
Example:
🧠 18. Candidate Fusion¶
Possible strategies:
🧠 19. Hybrid Retrieval¶
flowchart TD
A["Query"] --> B["Dense Retrieval"]
A --> C["Sparse Retrieval"]
B --> D["Dense Candidates"]
C --> E["Sparse Candidates"]
D --> F["Fusion"]
E --> F
F --> G["Deduplication"]
G --> H["Reranking"]🧠 20. Metadata Filtering¶
Metadata filtering should happen as early as practical.
Example:
{
"tenant_id": "tenant-a",
"department": "finance",
"classification": "internal",
"region": "eu",
"document_type": "policy"
}
🧠 21. Filter Before Expensive Processing¶
Prefer:
rather than:
when the underlying retrieval infrastructure and security model allow safe early filtering.
🧠 22. Authorization-Aware Retrieval¶
Security filtering is not simply:
A safer architecture is:
🧠 23. ACL-Aware Retrieval¶
Documents may have:
The retrieval system must enforce the effective access policy.
🧠 24. Retrieval Security Boundary¶
Never rely on the LLM to enforce document access.
🧠 25. Ranking Layer¶
Ranking improves candidate ordering.
🧠 26. Multi-Stage Retrieval¶
flowchart LR
A["Query"] --> B["Fast Retrieval"]
B --> C["Top 50"]
C --> D["Metadata Filter"]
D --> E["Top 30"]
E --> F["Reranker"]
F --> G["Top 10"]
G --> H["Context Selector"]
H --> I["Top 5"]🧠 27. Why Multi-Stage Retrieval?¶
Expensive operations should operate on smaller candidate sets.
🧠 28. Context Selection¶
The context layer converts:
into:
Selection considers:
🧠 29. Context Budget¶
Model Context
├── System Prompt
├── User Query
├── Conversation
├── Retrieved Evidence
└── Output Budget
Define an explicit retrieval context budget.
🧠 30. Evidence Package¶
Do not pass raw chunks directly to the LLM.
Create a structured evidence object.
from dataclasses import dataclass
@dataclass
class Evidence:
document_id: str
chunk_id: str
text: str
source: str
score: float
metadata: dict
🧠 31. Evidence Provenance¶
Maintain:
This enables:
🧠 32. Source Attribution¶
The source identity should not be reconstructed after generation if it can be preserved throughout the pipeline.
🧠 33. Retrieval Cache¶
A production retrieval service can use multiple caches:
🧠 34. Cache Key¶
A safe retrieval cache key may include:
Example:
def cache_key(request):
return hash((
request.tenant_id,
request.query,
request.top_k,
request.filters,
RETRIEVER_VERSION,
INDEX_VERSION
))
🧠 35. Retrieval Cache Risk¶
Never allow:
Cache isolation is part of the security architecture.
🧠 36. Indexing Architecture¶
Retrieval quality depends on the indexing pipeline.
Documents
↓
Ingestion
↓
Parsing
↓
Normalization
↓
Chunking
↓
Metadata Enrichment
↓
Embedding
↓
Indexing
🧠 37. Production Ingestion Pipeline¶
flowchart LR
A["Source Systems"] --> B["Ingestion"]
B --> C["Parsing"]
C --> D["Chunking"]
D --> E["Metadata"]
E --> F["Embedding"]
F --> G["Index Builder"]
G --> H["Vector Index"]
E --> I["Keyword Index"]
E --> J["Metadata Index"]🧠 38. Source Systems¶
Enterprise knowledge may originate from:
Document Management
SharePoint
Confluence
Git
Databases
Object Storage
CRM
Ticketing Systems
Email
APIs
🧠 39. Change Data Flow¶
Production knowledge is constantly changing.
🧠 40. Event-Driven Indexing¶
flowchart TD
A["Source Document"] --> B["Change Event"]
B --> C["Message Queue"]
C --> D["Ingestion Worker"]
D --> E["Chunking"]
E --> F["Embedding"]
F --> G["Index Update"]🧠 41. Why Event-Driven Indexing?¶
It provides:
🧠 42. Index Versioning¶
A production system should know:
Example:
🧠 43. Retrieval Reproducibility¶
A production answer should be traceable to:
Query
Prompt Version
Retriever Version
Index Version
Embedding Version
Reranker Version
Model Version
This enables debugging.
🧠 44. Blue-Green Index Deployment¶
This reduces deployment risk.
🧠 45. Canary Index Deployment¶
Compare:
🧠 46. Rollback¶
If the new index causes:
rollback:
Index versioning makes this possible.
🧠 47. High Availability¶
Production retrieval should avoid:
where availability requirements justify redundancy.
🧠 48. Retrieval Service Scaling¶
flowchart TD
A["Load Balancer"] --> B["Retriever 1"]
A --> C["Retriever 2"]
A --> D["Retriever 3"]
B --> E["Vector DB"]
C --> E
D --> ERetrieval services should ideally be stateless.
🧠 49. Stateless Retrieval Services¶
Keep request state outside application instances.
This allows:
🧠 50. Vector Database Scaling¶
Depending on the technology:
may be used.
🧠 51. Partitioning¶
Partition by dimensions such as:
But avoid over-partitioning.
🧠 52. Tenant-Aware Index Architecture¶
Possible models:
Shared Index¶
with strong metadata isolation.
Dedicated Index¶
Hybrid¶
🧠 53. Shared vs Dedicated Index¶
| Strategy | Isolation | Cost | Operational Complexity |
|---|---|---|---|
| Shared | Medium/High with strong filtering | Lower | Lower |
| Dedicated | High | Higher | Higher |
| Hybrid | Configurable | Medium | Medium |
Architecture must be based on:
🧠 54. Multi-Region Retrieval¶
Global Router
│
┌──────────┴──────────┐
▼ ▼
Region A Region B
│ │
Retriever Retriever
│ │
Vector DB Vector DB
🧠 55. Region Selection¶
Route based on:
🧠 56. Cross-Region Replication¶
Trade-offs include:
🧠 57. Consistency Model¶
Retrieval systems need a defined freshness expectation.
Possible models:
For many enterprise knowledge systems:
may be sufficient, but requirements are workload-specific.
🧠 58. Freshness¶
A production retrieval system should track:
Define:
🧠 59. Freshness SLO¶
Example:
The exact target depends on the application.
🧠 60. Retrieval Reliability¶
Retrieval can fail because of:
Vector DB Timeout
Network Failure
Provider Failure
Index Corruption
Connection Exhaustion
Rate Limit
🧠 61. Timeout Strategy¶
Every external dependency should have a timeout.
Never allow a downstream service to block the complete request indefinitely.
🧠 62. Fallback Retrieval¶
Example:
The system may continue with degraded capability if policy permits.
🧠 63. Retrieval Fallback Hierarchy¶
Primary Hybrid
↓
Dense Search
↓
Sparse Search
↓
Keyword Search
↓
Cached Result
↓
Graceful "No Evidence"
Fallback should never bypass authorization.
🧠 64. Circuit Breaker¶
This protects the system from repeatedly calling a failing dependency.
🧠 65. Retry Strategy¶
Do not blindly retry every retrieval failure.
Use:
🧠 66. Retry Storm¶
Bad:
This can overload an already failing service.
🧠 67. Bulkhead Isolation¶
Separate resource pools:
Failure in one workload should not consume all resources.
🧠 68. Backpressure¶
Use bounded queues to prevent memory exhaustion.
🧠 69. Rate Limiting¶
Apply limits to:
🧠 70. Retrieval SLOs¶
Define measurable objectives:
Example:
Values are illustrative.
🧠 71. Retrieval SLI¶
Examples:
retrieval_latency_ms
retrieval_success_rate
retrieval_error_rate
retrieval_recall
index_freshness_seconds
🧠 72. Observability Architecture¶
flowchart TD
A["RAG Request"] --> B["Retrieval Orchestrator"]
B --> C["Metrics"]
B --> D["Logs"]
B --> E["Traces"]
C --> F["Monitoring"]
D --> G["Log Platform"]
E --> H["Tracing Platform"]
F --> I["Alerts"]
G --> I
H --> I🧠 73. Distributed Trace¶
A single RAG request should expose:
request
├── query_processing
├── embedding
├── dense_retrieval
├── sparse_retrieval
├── fusion
├── filtering
├── reranking
├── context_selection
├── llm
└── validation
🧠 74. Retrieval Metrics¶
Track:
🧠 75. Operational Metrics¶
Track:
🧠 76. Cost Metrics¶
Track:
🧠 77. Retrieval Trace Example¶
{
"request_id": "req-123",
"tenant_id": "tenant-a",
"retriever": "hybrid",
"index_version": "v12",
"candidate_count": 50,
"reranked_count": 30,
"final_context_count": 6,
"latency_ms": 420
}
Production telemetry should respect privacy and security policies.
🧠 78. Security Architecture¶
A production retrieval system should address:
Authentication
Authorization
Tenant Isolation
Data Classification
Encryption
Audit Logging
Secrets
Network Security
Prompt Injection
Data Exfiltration
🧠 79. Retrieval-Level Authorization¶
Do not rely only on application-level authorization.
The retrieval layer should enforce the allowed knowledge boundary.
🧠 80. Metadata Security¶
Do not leak sensitive metadata such as:
unless the client is authorized to receive them.
🧠 81. Encryption¶
Use encryption:
for:
🧠 82. Prompt Injection in Retrieved Content¶
Retrieved documents may contain malicious instructions.
Example:
The retrieval architecture must treat retrieved content as:
rather than trusted instructions.
🧠 83. Evidence vs Instruction¶
The application should maintain explicit trust boundaries.
🧠 84. Retrieval Security Pipeline¶
flowchart TD
A["User"] --> B["Authentication"]
B --> C["Authorization"]
C --> D["Query"]
D --> E["Retrieval"]
E --> F["ACL Filtering"]
F --> G["Content Security Checks"]
G --> H["Evidence"]
H --> I["LLM"]🧠 85. Data Classification¶
Documents may be classified as:
Retrieval policy should respect these classifications.
🧠 86. Auditability¶
Record sufficient information to answer:
Do not store sensitive content unnecessarily.
🧠 87. Retrieval API¶
A production API might expose:
Request:
{
"query": "What is the payment retry policy?",
"top_k": 10,
"filters": {
"document_type": "policy"
}
}
🧠 88. Retrieval Response¶
{
"results": [
{
"document_id": "doc-123",
"chunk_id": "chunk-7",
"score": 0.92,
"text": "Payment retries...",
"metadata": {
"source": "payment-policy.pdf"
}
}
],
"index_version": "v12",
"retriever_version": "v8"
}
🧠 89. API Contract Principles¶
A production retrieval API should define:
Request Schema
Response Schema
Error Model
Timeout Behavior
Pagination
Filtering
Authentication
Authorization
Versioning
🧠 90. API Versioning¶
Use:
when breaking changes are required.
🧠 91. Error Contract¶
Example:
{
"error": {
"code": "RETRIEVAL_TIMEOUT",
"message": "Retrieval service timed out",
"request_id": "req-123"
}
}
Do not expose internal infrastructure details to external clients.
🧠 92. Retrieval Orchestrator Code¶
class RetrievalOrchestrator:
def __init__(
self,
dense,
sparse,
reranker,
context_selector
):
self.dense = dense
self.sparse = sparse
self.reranker = reranker
self.context_selector = context_selector
async def retrieve(self, request):
dense_results, sparse_results = await gather(
self.dense.retrieve(request),
self.sparse.retrieve(request)
)
candidates = fuse(
dense_results,
sparse_results
)
candidates = apply_authorization(
candidates,
request
)
ranked = await self.reranker.rank(
request.query,
candidates
)
return self.context_selector.select(
ranked
)
🧠 93. Production Retrieval Pipeline¶
Request
↓
Authenticate
↓
Resolve Tenant
↓
Authorize
↓
Normalize Query
↓
Classify Query
↓
Select Retrieval Strategy
↓
Parallel Candidate Retrieval
↓
Filter
↓
Fuse
↓
Rerank
↓
Select Context
↓
Build Evidence Package
↓
Return
🧠 94. Retrieval and Generation Separation¶
A strong architecture separates:
from:
Example:
flowchart LR
A["User"] --> B["RAG Application"]
B --> C["Retrieval Service"]
C --> D["Evidence"]
D --> E["Generation Service"]
E --> F["Response"]Benefits:
Independent Scaling
Independent Testing
Independent Deployment
Reusable Retrieval
Provider Flexibility
🧠 95. Why Separate Retrieval?¶
You may want:
serving:
🧠 96. Retrieval as a Shared Capability¶
Retrieval Platform
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
Chat Copilot Agent
│ │ │
└─────────────────┼─────────────────┘
▼
Evidence
🧠 97. Retrieval Service Scaling¶
Retrieval workload may scale differently from generation.
Separating them enables independent scaling.
🧠 98. Retrieval vs Generation SLO¶
Possible:
This makes bottlenecks easier to identify.
🧠 99. Retrieval Gateway¶
A retrieval gateway can centralize:
🧠 100. Retrieval Provider Strategy¶
Retriever Interface
│
├── FAISS
├── Chroma
├── Milvus
├── Elasticsearch
├── PostgreSQL
├── Graph DB
└── Cloud Search
Applications should depend on interfaces rather than infrastructure providers.
🧠 101. Cloud-Native Retrieval¶
A cloud-native architecture may use:
API Gateway
Container Platform
Vector Database
Object Storage
Message Queue
Cache
Observability
Secrets Manager
Identity
🧠 102. AWS Example¶
Supporting services may include:
🧠 103. Azure Example¶
Supporting services may include:
🧠 104. GCP Example¶
Supporting services may include:
🧠 105. Multi-Cloud Retrieval¶
flowchart TD
A["Enterprise RAG Platform"] --> B["Retrieval Abstraction"]
B --> C["AWS Adapter"]
B --> D["Azure Adapter"]
B --> E["GCP Adapter"]
C --> F["AWS Search"]
D --> G["Azure Search"]
E --> H["GCP Search"]The application remains independent of the cloud-specific implementation.
🧠 106. Retrieval Configuration¶
Keep operational parameters externalized:
retrieval:
top_k: 20
rerank_k: 10
context_k: 5
hybrid:
enabled: true
cache:
enabled: true
timeout_ms: 500
🧠 107. Configuration Versioning¶
Track:
🧠 108. Feature Flags¶
Use feature flags for controlled rollout:
🧠 109. Retrieval Experimentation¶
Feature flags enable:
🧠 110. Retrieval Evaluation Gate¶
Before production deployment:
New Retriever
↓
Offline Evaluation
↓
Quality Threshold?
│
├── No → Reject
└── Yes
↓
Performance Test
↓
Security Test
↓
Canary
🧠 111. Retrieval Quality Gate¶
Example:
Illustrative thresholds.
🧠 112. Retrieval Regression¶
A new retriever can improve:
while degrading:
or improve:
while increasing:
Track all dimensions.
🧠 113. Production Benchmark¶
| Architecture | Recall@10 | p95 | Cost | Complexity |
|---|---|---|---|---|
| Dense | 88% | 120 ms | Low | Low |
| Hybrid | 93% | 180 ms | Medium | Medium |
| Hybrid + Reranker | 96% | 300 ms | Higher | High |
Illustrative values.
🧠 114. Architecture Decision Matrix¶
Choose retrieval architecture based on:
Dataset Size
Query Volume
Recall Target
Latency Target
Freshness
Tenant Count
Security
Budget
Operational Capability
🧠 115. Production Retrieval Patterns¶
Common patterns include:
Single Retriever
Hybrid Retriever
Router Retriever
Multi-Stage Retriever
Parent-Child
Multi-Vector
Graph Retrieval
SQL Retrieval
Agentic Retrieval
🧠 116. Pattern Selection¶
Simple Semantic Search
↓
Dense Retriever
Keyword + Semantic
↓
Hybrid Retriever
Complex Knowledge
↓
Multi-Stage Retrieval
Structured Data
↓
SQL Retriever
Relationship-Heavy
↓
Graph Retriever
🧠 117. Retrieval Decision Tree¶
flowchart TD
A["Query"] --> B{"Structured Data?"}
B -->|Yes| C["SQL Retrieval"]
B -->|No| D{"Relationship Heavy?"}
D -->|Yes| E["Graph Retrieval"]
D -->|No| F{"Keyword + Semantic?"}
F -->|Yes| G["Hybrid Retrieval"]
F -->|No| H["Dense Retrieval"]
G --> I["Optional Reranking"]
H --> I
C --> J["Evidence"]
E --> J
I --> J🧠 118. Production Retrieval Anti-Patterns¶
Anti-Pattern 1¶
This tightly couples application logic to infrastructure.
Anti-Pattern 2¶
This can create unnecessary latency and cost.
Anti-Pattern 3¶
Authorization must happen before protected content reaches generation.
Anti-Pattern 4¶
This makes debugging and rollback difficult.
Anti-Pattern 5¶
This creates severe security risk.
🧠 119. Another Anti-Pattern¶
More stages do not automatically mean better production performance.
🧠 120. Simplicity Principle¶
Use:
that satisfies:
🧠 121. Production Retrieval Maturity¶
Level 1¶
Level 2¶
Level 3¶
Level 4¶
Level 5¶
Level 6¶
Level 7¶
Level 8¶
Level 9¶
Level 10¶
🧪 122. Practical Project¶
Build a:
Production Retrieval Platform
The project should expose:
and support:
Dense Search
Sparse Search
Hybrid Search
Metadata Filtering
Reranking
Caching
Tenant Isolation
Observability
Versioning
Fallbacks
🧪 123. Project Architecture¶
Client
│
▼
API Gateway
│
▼
Retrieval Gateway
│
▼
Query Orchestrator
│
┌───────────┼───────────┐
▼ ▼ ▼
Dense Sparse SQL
Search Search Search
│ │ │
└───────────┼───────────┘
▼
Fusion
│
▼
Filtering
│
▼
Reranking
│
▼
Context Selection
│
▼
Evidence API
🧪 124. Project Components¶
Implement:
retrieval-api
retrieval-core
retrieval-orchestrator
retrieval-adapters
ranking
context-engine
security
cache
observability
evaluation
configuration
🧪 125. Suggested Project Structure¶
production-retrieval-platform/
│
├── api/
│ ├── retrieval_controller.py
│ └── schemas.py
│
├── core/
│ ├── retriever.py
│ ├── retrieval_request.py
│ ├── retrieval_result.py
│ └── evidence.py
│
├── orchestrator/
│ └── retrieval_orchestrator.py
│
├── retrievers/
│ ├── dense.py
│ ├── sparse.py
│ ├── hybrid.py
│ ├── sql.py
│ └── graph.py
│
├── ranking/
│ ├── reranker.py
│ └── fusion.py
│
├── context/
│ ├── selector.py
│ ├── compressor.py
│ └── budget.py
│
├── security/
│ ├── authorization.py
│ ├── tenant.py
│ └── policies.py
│
├── cache/
│ └── retrieval_cache.py
│
├── observability/
│ ├── metrics.py
│ ├── tracing.py
│ └── logging.py
│
├── evaluation/
│ └── benchmark.py
│
└── config/
└── retrieval.yaml
🧪 126. Core Retrieval Interface¶
from abc import ABC, abstractmethod
class Retriever(ABC):
@abstractmethod
async def retrieve(
self,
request
):
raise NotImplementedError
🧪 127. Dense Adapter¶
class DenseRetriever(Retriever):
def __init__(self, vector_store):
self.vector_store = vector_store
async def retrieve(self, request):
return await self.vector_store.similarity_search(
query=request.query,
top_k=request.top_k,
filters=request.filters
)
🧪 128. Hybrid Retriever¶
class HybridRetriever(Retriever):
def __init__(
self,
dense,
sparse,
fusion
):
self.dense = dense
self.sparse = sparse
self.fusion = fusion
async def retrieve(self, request):
dense_results, sparse_results = await gather(
self.dense.retrieve(request),
self.sparse.retrieve(request)
)
return self.fusion.merge(
dense_results,
sparse_results
)
🧪 129. Production Request Flow¶
POST /v1/retrieve
↓
Authentication
↓
Tenant Resolution
↓
Authorization
↓
Query Validation
↓
Query Router
↓
Retriever
↓
Filtering
↓
Ranking
↓
Context Selection
↓
Evidence
🧪 130. Testing Strategy¶
Test:
Unit Tests
Integration Tests
Contract Tests
Security Tests
Load Tests
Chaos Tests
Regression Tests
Retrieval Quality Tests
🧪 131. Unit Testing¶
Test:
🧪 132. Integration Testing¶
Test:
Retriever ↔ Vector DB
Retriever ↔ Search Engine
Retriever ↔ Cache
Retriever ↔ Authorization
Retriever ↔ Observability
🧪 133. Security Testing¶
Test:
Tenant Isolation
ACL Enforcement
Unauthorized Documents
Cache Isolation
Metadata Leakage
Prompt Injection
Data Exfiltration
🧪 134. Load Testing¶
Test:
Measure:
🧪 135. Failure Testing¶
Simulate:
Vector DB Down
Search Timeout
Cache Down
Reranker Down
Network Failure
Index Unavailable
LLM Unavailable
Verify graceful degradation.
🧪 136. Chaos Scenario¶
The application should continue where policy permits.
🧪 137. Production Readiness Checklist¶
☐ Retrieval API defined
☐ Retriever interface defined
☐ Provider adapters implemented
☐ Dense retrieval implemented
☐ Sparse retrieval implemented
☐ Hybrid retrieval implemented
☐ Candidate fusion implemented
☐ Metadata filtering implemented
☐ ACL filtering implemented
☐ Tenant isolation implemented
☐ Reranking implemented
☐ Context selection implemented
☐ Evidence model implemented
☐ Retrieval cache implemented
☐ Cache isolation implemented
☐ Cache versioning implemented
☐ Index versioning implemented
☐ Configuration versioning implemented
☐ Feature flags implemented
☐ Authentication implemented
☐ Authorization implemented
☐ Encryption implemented
☐ Audit logging implemented
☐ Data classification supported
☐ Timeouts implemented
☐ Retries implemented
☐ Circuit breakers implemented
☐ Backpressure implemented
☐ Rate limiting implemented
☐ Bulkhead isolation implemented
☐ Fallback retrieval implemented
☐ Horizontal scaling supported
☐ Autoscaling supported
☐ High availability supported
☐ Backup strategy defined
☐ Disaster recovery defined
☐ Multi-region strategy defined
☐ Retrieval SLO defined
☐ Latency metrics defined
☐ Retrieval quality metrics defined
☐ Cost metrics defined
☐ Distributed tracing implemented
☐ Alerts implemented
☐ Offline evaluation implemented
☐ Regression testing implemented
☐ Load testing implemented
☐ Security testing implemented
☐ Chaos testing implemented
☐ Blue-green deployment supported
☐ Canary deployment supported
☐ Rollback supported
☐ Index rollback supported
☐ Tenant cost attribution implemented
☐ Performance budgets defined
☐ Cost budgets defined
☐ Quality gates defined
🧠 138. Production Architecture Mental Model¶
USER
│
▼
API / SECURITY
│
▼
QUERY LAYER
│
▼
RETRIEVAL ORCHESTRATOR
│
┌──────────────┼──────────────┐
▼ ▼ ▼
DENSE SPARSE SQL
│ │ │
└──────────────┼──────────────┘
▼
FUSION
│
▼
AUTH / FILTERING
│
▼
RERANKING
│
▼
CONTEXT SELECTION
│
▼
EVIDENCE MODEL
│
▼
LLM
│
▼
VALIDATION
│
▼
CITATION
│
▼
RESPONSE
┌────────────────────────────────────┐
│ CROSS-CUTTING SERVICES │
│ │
│ Cache | Observability | Cost │
│ Security | Config | Evaluation │
│ Versioning | Resilience │
└────────────────────────────────────┘
🧠 139. Final Mental Model¶
Production retrieval engineering is about building a controlled pipeline:
PRODUCTION RETRIEVAL
│
▼
DISCOVER
│
▼
FILTER
│
▼
RETRIEVE
│
▼
FUSE
│
▼
RERANK
│
▼
COMPRESS
│
▼
SELECT
│
▼
CITE
│
▼
OBSERVE
│
▼
EVALUATE
│
▼
OPTIMIZE
Around that pipeline:
must operate continuously.
📚 140. Key Takeaways¶
- Production retrieval is a platform capability, not merely a vector database call.
- Separate retrieval from generation where independent scaling and reuse are valuable.
- Use stable retrieval interfaces and provider adapters.
- Keep applications independent of vendor-specific search infrastructure.
- Use capability-based abstractions.
- Centralize retrieval orchestration.
- Use query classification to select appropriate retrieval strategies.
- Use dense retrieval for semantic similarity.
- Use sparse retrieval for lexical matching.
- Use hybrid retrieval when both signals are valuable.
- Use multi-stage retrieval to control expensive ranking operations.
- Apply authorization before protected content reaches generation.
- Tenant context must influence retrieval, filtering, caching, and cost attribution.
- Treat retrieved documents as untrusted evidence.
- Preserve source provenance throughout the pipeline.
- Use structured evidence objects.
- Carry document and chunk identifiers through generation.
- Use metadata filtering to reduce irrelevant candidates.
- Use reranking selectively.
- Keep context budgets explicit.
- Use caching carefully.
- Include tenant and authorization scope in cache isolation.
- Version indexes.
- Version retrievers.
- Version embedding models.
- Version configurations.
- Use blue-green or canary index deployment for safer changes.
- Support rollback.
- Build incremental indexing pipelines.
- Use event-driven ingestion for scalable knowledge updates.
- Define freshness expectations.
- Design for high availability.
- Use stateless retrieval services for horizontal scaling.
- Consider tenant-aware index strategies.
- Use multi-region architecture when availability, latency, or residency requirements justify it.
- Define retrieval SLOs.
- Track retrieval latency and quality separately.
- Use distributed tracing across the retrieval pipeline.
- Implement timeouts.
- Implement bounded retries.
- Use circuit breakers.
- Use backpressure.
- Use bulkheads.
- Implement graceful retrieval fallbacks.
- Never allow fallback paths to bypass authorization.
- Separate interactive and background workloads.
- Build retrieval quality gates before production deployment.
- Benchmark recall, latency, cost, and reliability together.
- Use feature flags for controlled retrieval experimentation.
- Design APIs with explicit contracts and versioning.
- Test retrieval systems for security, load, resilience, and quality.
- Cloud-native retrieval should use managed infrastructure where it improves operational efficiency.
- Multi-cloud retrieval can be supported through provider adapters rather than application-level cloud coupling.
- The production retrieval layer should become independently observable, scalable, testable, and deployable.
- The ultimate goal is a retrieval platform that provides relevant, authorized, fresh, explainable, and operationally reliable evidence to downstream AI systems.
🧭 141. Chapter Navigation¶
Part V — Advanced Retrieval-Augmented Generation¶
Previous:
09. RAG Cost Optimization
Next:
11. Building Production RAG Systems
Section:
06 — Production RAG Engineering
Production RAG Engineering Path¶
01 Prompt Assembly
↓
02 Context Selection & Context Engineering
↓
03 Response Validation
↓
04 Citation & Source Attribution
↓
05 Enterprise Response
↓
06 RAG Evaluation & Benchmarking
↓
07 RAG Observability
↓
08 RAG Performance Optimization
↓
09 RAG Cost Optimization
↓
10 Production Retrieval Architecture
↓
11 Building Production RAG Systems
Enterprise AI Engineering Handbook
Building Production-Grade Enterprise AI Systems — One Chapter at a Time.