16 — Retrieval and Generation Pipeline¶
Understand how a RAG application transforms a user query into retrieved evidence, assembles grounded context, invokes an LLM, and produces a reliable final response.
📖 Overview¶
A Retrieval-Augmented Generation system can be divided into two major runtime stages:
The complete runtime pipeline is:
User Query
↓
Query Processing
↓
Query Embedding
↓
Retrieval
↓
Candidate Results
↓
Filtering / Ranking
↓
Context Assembly
↓
Prompt Construction
↓
LLM Generation
↓
Response Validation
↓
Answer + Citations
The key principle is:
Retrieval determines what evidence the LLM sees; generation determines how that evidence is transformed into the final response.
1. Retrieval vs Generation¶
A RAG pipeline contains two distinct responsibilities.
Retrieval¶
Retrieval answers:
Generation¶
Generation answers:
Conceptually:
RAG Pipeline
┌─────────────────────┐
│ Retrieval │
│ │
Query ─┤ Find Relevant Data │
└──────────┬──────────┘
↓
Retrieved Context
↓
┌─────────────────────┐
│ Generation │
│ │
│ Context + Query │
│ ↓ │
│ LLM │
└──────────┬──────────┘
↓
Answer
Keeping these responsibilities separate makes RAG systems easier to design, evaluate, and debug.
2. End-to-End Runtime Pipeline¶
flowchart TD
A["User Query"] --> B["Query Processing"]
B --> C["Query Embedding"]
C --> D["Retriever"]
E["Vector Store"] --> D
F["Authorization Context"] --> D
D --> G["Candidate Results"]
G --> H["Filtering / Ranking"]
H --> I["Context Assembly"]
I --> J["Prompt Construction"]
J --> K["LLM"]
K --> L["Response Validation"]
L --> M["Answer + Citations"]Every stage has a different responsibility.
3. Query Processing¶
The runtime pipeline begins with the user's question.
Example:
Before retrieval, the system may perform:
Not every application requires all of these operations.
4. Query Object¶
A production system should avoid passing a raw string through every layer.
A structured query object can contain:
{
"query": "What is the annual leave policy?",
"tenant_id": "tenant-a",
"user_id": "user-123",
"language": "en",
"filters": {
"country": "IN",
"department": "HR"
}
}
This provides the downstream pipeline with the context required for retrieval.
5. Query Validation¶
Basic validation may include:
Query is not empty
Query length is acceptable
Input is within allowed limits
Required tenant context exists
Required authorization context exists
Example:
def validate_query(query: str):
if not query or not query.strip():
raise ValueError("Query cannot be empty")
if len(query) > 5000:
raise ValueError("Query is too long")
return query.strip()
Production validation should be aligned with application requirements.
6. Query Normalization¶
Some applications normalize queries before embedding.
For example:
Original:
"What is our company's annual leave policy???"
Normalized:
"What is our company's annual leave policy?"
Possible operations include:
Whitespace normalization
Encoding normalization
Removing accidental formatting
Language normalization
Be careful not to remove information that changes the meaning of the query.
7. Query Understanding¶
A more advanced system may identify:
Example:
Query:
"What is the 2026 leave policy for employees in India?"
Detected:
Intent → Policy lookup
Year → 2026
Country → India
Topic → Leave
These signals can improve retrieval without changing the fundamental RAG architecture.
8. Query Embedding¶
The query is converted into a vector.
Example:
The resulting vector is used for semantic retrieval.
9. Query and Document Embedding Compatibility¶
Documents may have been indexed using:
The query should normally use the same compatible embedding model.
Avoid accidentally mixing incompatible vector spaces.
10. Retrieval¶
The retriever receives:
and returns candidate documents.
Example:
11. Retrieval Inputs¶
A production retrieval operation may look conceptually like:
{
"query": "What is the annual leave policy?",
"top_k": 10,
"similarity_threshold": 0.72,
"filters": {
"tenant_id": "tenant-a",
"country": "IN",
"department": "HR"
}
}
The exact implementation depends on the vector store and retrieval architecture.
12. Retrieval Output¶
The retriever should return structured results.
Example:
[
{
"chunk_id": "hr-policy-007",
"content": "Employees receive 25 days of annual leave.",
"score": 0.91,
"metadata": {
"document": "employee-handbook",
"page": 42,
"section": "Annual Leave"
}
}
]
The result contains both:
and:
13. Candidate Retrieval¶
The first retrieval stage may intentionally return more candidates than the final context requires.
For example:
The system can then process those candidates:
This separates:
from:
14. Candidate Generation¶
The candidate generation stage should prioritize:
The goal is:
Find enough potentially relevant information for the downstream ranking and context stages.
A candidate that is never retrieved cannot be selected later.
15. Filtering¶
Retrieved candidates may need additional filtering.
Examples:
Conceptually:
16. Authorization-Aware Retrieval¶
Authorization must be applied before content reaches the LLM.
Incorrect:
Correct:
The LLM is not a security boundary.
17. Ranking¶
After retrieval and filtering, candidates may be ranked.
Ranking may consider:
The exact strategy depends on the application.
18. Deduplication¶
Multiple chunks may contain the same information.
Example:
Chunk A:
"Employees receive 25 days of annual leave."
Chunk B:
"Employees receive 25 days of annual leave."
A context assembly stage should avoid wasting context on duplicates.
19. Context Selection¶
The final context should contain the most useful evidence.
The objective is:
20. Context Assembly¶
Retrieved chunks must be converted into a coherent context.
Example:
[Source: Employee Handbook]
[Section: Annual Leave]
[Page: 42]
Employees are entitled to 25 days
of annual paid leave.
---
[Source: Leave Policy]
[Section: Carry Forward]
[Page: 43]
Unused leave may be carried forward
according to company policy.
The context should preserve useful source information.
21. Context Ordering¶
Possible ordering strategies include:
For policy documents, document order may sometimes preserve important relationships.
For semantic retrieval, score-based ordering may be useful.
The correct strategy should be evaluated for the target workload.
22. Context Window Management¶
The LLM has a finite context window.
Conceptually:
If too much information is retrieved:
Therefore context selection is a critical RAG component.
23. Token Budgeting¶
A context builder can enforce a token budget.
def build_context(documents, max_tokens):
context = []
token_count = 0
for document in documents:
tokens = estimate_tokens(
document.content
)
if token_count + tokens > max_tokens:
break
context.append(document)
token_count += tokens
return context
In production, token counting should use a tokenizer appropriate for the target model.
24. Context Compression¶
If retrieved content is too large, a system may reduce it before generation.
Conceptually:
Compression can reduce:
Advanced context compression techniques are covered later in the retrieval section.
25. Prompt Construction¶
The retrieved context is inserted into a prompt.
A simple structure is:
Example:
System:
You are an enterprise knowledge assistant.
Answer using only the supplied context.
Context:
Employees are entitled to 25 days
of annual paid leave.
Question:
How many annual leave days do employees receive?
26. Grounding Instructions¶
A production prompt should define what the model should do when evidence is insufficient.
Example:
You are an enterprise knowledge assistant.
Use only the provided context to answer.
If the context does not contain sufficient
information to answer the question, clearly
state that the information is not available.
Do not invent facts.
Context:
{context}
Question:
{question}
The exact prompt should be designed for the application.
27. Prompt Template¶
A reusable prompt template can be implemented as:
RAG_PROMPT = """
You are an enterprise knowledge assistant.
Answer the user's question using only
the provided context.
If the context does not contain enough
information, say so clearly.
Context:
{context}
Question:
{question}
Answer:
"""
Then:
28. Prompt Injection Consideration¶
Retrieved documents are untrusted input.
A document might contain text such as:
The system should treat retrieved content as:
rather than:
A grounded prompt should clearly separate:
from:
29. LLM Generation¶
The prompt is sent to the selected LLM.
Example:
The application should keep the LLM provider behind an abstraction where portability is important.
30. LLM Provider Interface¶
A Java-first architecture can define:
Possible implementations include:
The RAG service should not depend directly on a vendor SDK.
31. Generation Parameters¶
Common generation parameters include:
For enterprise knowledge applications, deterministic or low-variance generation is often desirable.
For example:
The appropriate values depend on the model and application.
32. Generation Does Not Equal Grounding¶
The LLM may still generate unsupported information even when context is provided.
Therefore:
RAG improves grounding but does not mathematically guarantee correctness.
33. Response Validation¶
A production system may validate the generated response.
Possible checks:
For structured applications:
If validation fails:
34. Citation Generation¶
The retrieved chunks should carry source metadata.
Example:
The final response can then provide:
Citations should be derived from retrieved evidence rather than invented by the model.
35. Citation Pipeline¶
flowchart LR
A["Source Document"] --> B["Chunk"]
B --> C["Metadata"]
C --> D["Vector Store"]
D --> E["Retriever"]
E --> F["Retrieved Evidence"]
F --> G["Context Builder"]
F --> H["Citation Builder"]
G --> I["LLM"]
I --> J["Answer"]
H --> K["Source References"]
J --> L["Final Response"]
K --> L36. Complete Runtime Pipeline¶
User
↓
API
↓
Authentication
↓
Query Validation
↓
Query Processing
↓
Query Embedding
↓
Retriever
↓
Candidate Retrieval
↓
Security Filtering
↓
Metadata Filtering
↓
Ranking
↓
Deduplication
↓
Context Selection
↓
Token Budgeting
↓
Prompt Construction
↓
LLM
↓
Response Validation
↓
Citation Assembly
↓
Final Answer
37. Runtime Sequence Diagram¶
sequenceDiagram
participant U as User
participant API as API
participant R as Retriever
participant E as Embedding Provider
participant V as Vector Store
participant C as Context Builder
participant P as Prompt Builder
participant L as LLM
participant S as Source/Citation Layer
U->>API: Ask question
API->>API: Validate request
API->>E: Embed query
E-->>API: Query vector
API->>R: Retrieve(query, vector, filters)
R->>V: Similarity search
V-->>R: Candidate chunks
R-->>API: Ranked results
API->>C: Build context
C-->>API: Context
API->>P: Build grounded prompt
P-->>API: Prompt
API->>L: Generate response
L-->>API: Generated answer
API->>S: Build citations
S-->>API: Source references
API-->>U: Answer + citations38. Retrieval and Generation Latency¶
Total response latency can be thought of as:
Total Latency ≈
Query Processing
+
Embedding
+
Retrieval
+
Context Processing
+
LLM Generation
+
Validation
For example:
Query Processing 5 ms
Embedding 30 ms
Retrieval 40 ms
Context 10 ms
LLM 800 ms
Validation 20 ms
-------------------------
Total 905 ms
These numbers are illustrative.
The important principle is:
Optimize the complete request path rather than only the vector search.
39. Latency Breakdown¶
flowchart LR
A["Query"] --> B["Processing"]
B --> C["Embedding"]
C --> D["Retrieval"]
D --> E["Context"]
E --> F["LLM"]
F --> G["Validation"]
G --> H["Response"]Each stage should be measurable independently.
40. Parallelization Opportunities¶
Some operations may be parallelized.
For example:
Query Processing
↓
┌───────────────┐
│ │
Embedding Security Context
│ │
└───────┬───────┘
↓
Retrieval
Parallelization can reduce end-to-end latency where dependencies allow it.
41. Streaming Generation¶
For interactive applications, the LLM response can be streamed.
Instead of:
the system can provide:
Conceptually:
Streaming improves perceived responsiveness but does not necessarily reduce total generation time.
42. Streaming Architecture¶
flowchart LR
A["User"] --> B["RAG API"]
B --> C["Retrieval"]
C --> D["Prompt"]
D --> E["LLM Streaming"]
E --> F["Token 1"]
E --> G["Token 2"]
E --> H["Token 3"]
E --> I["..."]
F --> J["Client"]
G --> J
H --> J
I --> J43. Failure Handling¶
Every stage can fail.
A production system should handle failures explicitly.
44. Retrieval Failure¶
If retrieval fails:
Possible responses:
Do not silently fabricate an answer.
45. LLM Failure¶
If the LLM fails:
the application may use:
depending on business requirements.
46. Timeout Budget¶
A production RAG request should have an overall timeout.
Conceptually:
Request Timeout
↓
┌──────────────────────────────┐
│ Query + Retrieval + LLM │
└──────────────────────────────┘
Individual components should also have reasonable timeouts.
For example:
These values are illustrative and should be determined through benchmarking.
47. Retry Strategy¶
Retries should be used carefully.
For transient errors:
Avoid retrying indefinitely.
A production policy might use:
48. Fallback Models¶
Some applications may support:
Example:
Fallback should be evaluated for:
49. Fallback Retrieval¶
A retrieval system may also have fallback mechanisms.
For example:
This can improve robustness for exact identifiers and unusual terminology.
50. Empty Retrieval¶
A key production case is:
The system should not automatically answer from model memory.
Possible policy:
Example:
"I couldn't find sufficient information
in the available enterprise knowledge base
to answer this question."
51. Low-Confidence Retrieval¶
A system may also detect:
and respond differently.
This can reduce unsupported answers.
52. Query Clarification¶
Some questions are ambiguous.
Example:
Possible interpretations:
The application may ask:
instead of retrieving unrelated documents.
53. Query Rewrite¶
A system may transform:
into:
using conversation context.
This should be done carefully so the rewritten query preserves the user's intent.
54. Conversational RAG¶
For multi-turn conversations:
User:
"What is the annual leave policy?"
Assistant:
"Employees receive 25 days."
User:
"Can it be carried forward?"
The second query depends on the first.
The system may construct:
before retrieval.
55. Conversational RAG Architecture¶
flowchart TD
A["Conversation History"] --> B["Query Understanding"]
C["Current User Query"] --> B
B --> D["Standalone Retrieval Query"]
D --> E["Query Embedding"]
E --> F["Retriever"]
F --> G["Context"]
G --> H["LLM"]
H --> I["Answer"]Conversation history should not automatically be sent wholesale to every retrieval operation.
56. RAG Context vs Conversation Context¶
There are two different types of context:
Conversation Context¶
Knowledge Context¶
A production prompt may combine them:
The system should distinguish these sources explicitly.
57. Generation Context Architecture¶
flowchart TD
A["System Instructions"] --> E["Prompt"]
B["Conversation Context"] --> E
C["Retrieved Knowledge"] --> E
D["Current User Query"] --> E
E --> F["LLM"]
F --> G["Response"]58. Context Priority¶
A grounded application should define how conflicting information is handled.
For example:
System Instructions
↓
Security Policies
↓
Retrieved Enterprise Knowledge
↓
Conversation Context
↓
User Request
The exact priority depends on the application's instruction hierarchy.
59. Conflicting Documents¶
Suppose retrieval returns:
The system should not simply combine them.
Metadata such as:
should help determine which document is authoritative.
60. Freshness¶
For frequently changing enterprise knowledge:
The retrieval pipeline should be designed to minimize stale knowledge.
61. Retrieval Freshness Architecture¶
flowchart LR
A["Source Document Update"] --> B["Change Detection"]
B --> C["Reprocessing"]
C --> D["Embedding"]
D --> E["Vector Store Update"]
F["User Query"] --> G["Retriever"]
E --> G
G --> H["Current Evidence"]62. RAG and Source of Truth¶
The vector store generally represents derived data.
If the index is lost:
should ideally be possible.
63. RAG Orchestrator¶
A RAG application service can coordinate the complete runtime pipeline.
class RagService:
def __init__(
self,
retriever,
context_builder,
prompt_builder,
llm,
validator
):
self.retriever = retriever
self.context_builder = context_builder
self.prompt_builder = prompt_builder
self.llm = llm
self.validator = validator
def answer(self, query):
documents = self.retriever.retrieve(
query
)
context = self.context_builder.build(
documents
)
prompt = self.prompt_builder.build(
query,
context
)
response = self.llm.generate(
prompt
)
return self.validator.validate(
response
)
The implementation can be Python, Java, or another language.
64. Java RAG Service¶
A Java-first implementation could look like:
public class RagService {
private final Retriever retriever;
private final ContextBuilder contextBuilder;
private final PromptBuilder promptBuilder;
private final LLMProvider llmProvider;
private final ResponseValidator validator;
public RagService(
Retriever retriever,
ContextBuilder contextBuilder,
PromptBuilder promptBuilder,
LLMProvider llmProvider,
ResponseValidator validator
) {
this.retriever = retriever;
this.contextBuilder = contextBuilder;
this.promptBuilder = promptBuilder;
this.llmProvider = llmProvider;
this.validator = validator;
}
public Answer answer(Query query) {
var documents =
retriever.retrieve(
query,
RetrievalOptions.defaults()
);
var context =
contextBuilder.build(documents);
var prompt =
promptBuilder.build(query, context);
var result =
llmProvider.generate(prompt);
return validator.validate(result);
}
}
This keeps orchestration separate from infrastructure adapters.
65. Retriever Interface¶
public interface Retriever {
List<RetrievedDocument> retrieve(
Query query,
RetrievalOptions options
);
}
The implementation can use:
66. Context Builder Interface¶
Responsibilities may include:
67. Prompt Builder Interface¶
This keeps prompt construction independent from retrieval.
68. Response Validator¶
The validator can perform:
where required.
69. RAG Capability Architecture¶
flowchart TD
A["RAG Application"] --> B["Retriever"]
A --> C["ContextBuilder"]
A --> D["PromptBuilder"]
A --> E["LLMProvider"]
A --> F["ResponseValidator"]
B --> G["EmbeddingProvider"]
B --> H["VectorStore"]
G --> I["Embedding Adapter"]
H --> J["Vector DB Adapter"]
E --> K["LLM Adapter"]This follows a capability-oriented architecture.
70. LangChain RAG Example¶
A simplified LangChain-style implementation:
from langchain_core.prompts import ChatPromptTemplate
prompt = ChatPromptTemplate.from_template(
"""
Answer using only the provided context.
Context:
{context}
Question:
{question}
Answer:
"""
)
question = "What is the annual leave policy?"
documents = retriever.invoke(question)
context = "\n\n".join(
doc.page_content
for doc in documents
)
response = llm.invoke(
prompt.format_messages(
context=context,
question=question
)
)
print(response.content)
The framework simplifies implementation, but the architecture remains:
71. LlamaIndex RAG Example¶
A simplified LlamaIndex workflow:
from llama_index.core import VectorStoreIndex
index = VectorStoreIndex.from_documents(
documents
)
query_engine = index.as_query_engine(
similarity_top_k=5
)
response = query_engine.query(
"What is the annual leave policy?"
)
print(response)
The framework combines retrieval and generation behind the query engine abstraction.
72. Framework vs Production Architecture¶
A framework may expose:
but internally the system may perform:
Understanding these underlying stages is essential for production troubleshooting.
73. RAG Evaluation¶
RAG quality should be evaluated at multiple stages.
Retrieval Evaluation¶
Generation Evaluation¶
System Evaluation¶
These metrics should not be collapsed into one number without understanding what each measures.
74. Retrieval Evaluation¶
The first question is:
Did the retriever find the correct evidence?
Example:
The correct evidence was retrieved.
75. Generation Evaluation¶
The second question is:
Did the LLM correctly use the retrieved evidence?
Example:
Retrieval succeeded.
Generation failed.
This distinction is critical when debugging RAG systems.
76. End-to-End Evaluation¶
A complete evaluation asks:
Question
↓
Was the right evidence retrieved?
↓
Was the evidence correctly assembled?
↓
Did the LLM use it correctly?
↓
Were citations correct?
↓
Was the answer useful?
77. RAG Observability¶
A production RAG request should ideally produce a trace.
trace_id
↓
Query
↓
Embedding
↓
Retrieval
↓
Retrieved IDs
↓
Scores
↓
Context Size
↓
Prompt
↓
LLM
↓
Output Tokens
↓
Validation
↓
Final Response
Sensitive data should be handled according to the application's privacy and security requirements.
78. RAG Trace Example¶
{
"trace_id": "rag-10042",
"retrieval": {
"top_k": 10,
"results": 5,
"latency_ms": 42
},
"context": {
"chunks": 5,
"estimated_tokens": 1850
},
"generation": {
"model": "enterprise-llm",
"latency_ms": 820,
"output_tokens": 180
}
}
This kind of telemetry helps identify bottlenecks without necessarily logging sensitive content.
79. Cost Breakdown¶
RAG cost can come from:
Query Embedding
+
Vector Search Infrastructure
+
Reranking
+
LLM Input Tokens
+
LLM Output Tokens
+
Observability
The biggest cost in many applications is LLM inference.
Reducing unnecessary context can therefore reduce cost.
80. Cost Optimization¶
A simplified flow:
Optimization may involve:
Each optimization should be evaluated for its effect on answer quality.
81. Caching¶
Some RAG applications can cache:
For example:
Caching should consider:
82. Cache Safety¶
Never use a shared response cache without considering authorization.
Dangerous:
Cache keys may need to include:
83. RAG Pipeline Resilience¶
Production systems should consider:
These are general distributed-system patterns applied to AI workloads.
84. Backpressure¶
If the LLM provider slows down:
Without controls:
Production AI systems should therefore consider request limits and backpressure.
85. RAG Pipeline Scaling¶
Different components scale differently.
API Layer
↓
Horizontal Scaling
Embedding Service
↓
Batching / Scaling
Vector Store
↓
Index / Sharding / Replication
LLM
↓
Provider Scaling / Model Infrastructure
The architecture should avoid assuming that one scaling strategy fits the entire pipeline.
86. Production RAG Deployment¶
flowchart TD
A["Clients"] --> B["API Gateway"]
B --> C["RAG Service"]
C --> D["Retriever Service"]
C --> E["LLM Service"]
D --> F["Embedding Service"]
D --> G["Vector Database"]
E --> H["LLM Provider"]
C --> I["Cache"]
C --> J["Observability"]
K["Document Pipeline"] --> GThis is a conceptual deployment architecture.
87. RAG Request Lifecycle¶
T0 User sends query
↓
T1 Authentication
↓
T2 Query validation
↓
T3 Query embedding
↓
T4 Vector retrieval
↓
T5 Filtering
↓
T6 Ranking
↓
T7 Context assembly
↓
T8 Prompt creation
↓
T9 LLM invocation
↓
T10 Response validation
↓
T11 Citation assembly
↓
T12 Response returned
This timeline can be instrumented for latency analysis.
88. Debugging the Pipeline¶
When the final answer is incorrect, inspect the stages in order:
1. Was the user query understood correctly?
2. Was the query embedded correctly?
3. Were the right documents retrieved?
4. Were security filters correct?
5. Were relevant chunks discarded?
6. Was context assembled correctly?
7. Was the prompt correct?
8. Did the LLM follow the grounding instructions?
9. Was the output validated?
10. Were citations generated correctly?
This is much more effective than simply changing the LLM.
89. Retrieval Failure Example¶
Question:
"What is the 2026 leave policy?"
Retrieved:
2021 Leave Policy
2019 Leave Policy
2020 Leave Policy
Problem:
Changing the prompt alone will not solve the problem.
90. Generation Failure Example¶
Question:
"What is the annual leave entitlement?"
Retrieved:
"Employees receive 25 days of annual leave."
LLM:
"Employees receive 30 days."
Problem:
The retriever found the correct evidence.
91. Context Failure Example¶
Retrieved:
Chunk 1:
"Employees receive 25 days..."
Chunk 2:
"Exceptions apply to contractors."
Context builder accidentally removes Chunk 2.
The generation stage now lacks important information.
Problem:
92. Prompt Failure Example¶
Suppose the prompt says:
instead of:
The LLM may rely more heavily on its pretrained knowledge.
Problem:
93. RAG Debugging Matrix¶
| Symptom | Possible Stage |
|---|---|
| Wrong documents | Retrieval |
| No documents | Retrieval / Filters |
| Old information | Indexing / Metadata |
| Correct context, wrong answer | Generation |
| Missing evidence | Context Assembly |
| Wrong source | Citation Layer |
| High latency | Any stage |
| High cost | Context / Generation |
| Unauthorized information | Security / Filtering |
| Empty results | Retrieval / Query Processing |
94. Production Design Principles¶
Separation of Concerns
↓
Explicit Data Contracts
↓
Capability Interfaces
↓
Provider Independence
↓
Observability
↓
Security
↓
Evaluation
Each component should be independently testable.
95. Recommended Component Boundaries¶
QueryProcessor
↓
EmbeddingProvider
↓
Retriever
↓
ContextBuilder
↓
PromptBuilder
↓
LLMProvider
↓
ResponseValidator
↓
CitationBuilder
This structure provides clean boundaries between retrieval and generation.
96. Component Responsibility Matrix¶
| Component | Responsibility |
|---|---|
| Query Processor | Validate and prepare user query |
| Embedding Provider | Create query vector |
| Retriever | Find candidate evidence |
| Filter | Apply metadata/security constraints |
| Ranker | Order candidates |
| Context Builder | Select and format evidence |
| Prompt Builder | Build grounded prompt |
| LLM Provider | Generate response |
| Validator | Validate generated output |
| Citation Builder | Attach source references |
| Observability | Capture telemetry |
97. Production Workflow¶
1. Receive user query.
2. Authenticate the user.
3. Resolve tenant and authorization context.
4. Validate the query.
5. Normalize or transform the query if required.
6. Generate query embedding.
7. Execute candidate retrieval.
8. Apply authorization filters.
9. Apply metadata filters.
10. Rank candidates.
11. Deduplicate results.
12. Select context within token budget.
13. Preserve source metadata.
14. Build grounded prompt.
15. Invoke LLM.
16. Validate response.
17. Build citations.
18. Return final answer.
19. Record telemetry.
20. Evaluate system quality continuously.
98. Complete Production Architecture¶
flowchart TD
A["User"] --> B["API Gateway"]
B --> C["Authentication"]
C --> D["RAG Application"]
D --> E["Query Processor"]
E --> F["Query Embedding"]
D --> G["Authorization Context"]
F --> H["Retriever"]
G --> H
H --> I["Vector Store"]
I --> J["Vector Database"]
H --> K["Candidate Results"]
K --> L["Filtering"]
L --> M["Ranking"]
M --> N["Deduplication"]
N --> O["Context Builder"]
O --> P["Prompt Builder"]
P --> Q["LLM Provider"]
Q --> R["LLM"]
R --> S["Response Validator"]
S --> T["Citation Builder"]
T --> U["Final Answer"]
D --> V["Observability"]
H --> V
Q --> V
S --> V99. Framework Mapping¶
The architecture can be implemented using different frameworks.
| Capability | Example Technologies |
|---|---|
| Query Processing | Custom Java / Python |
| Embeddings | OpenAI / Hugging Face / WatsonX |
| Vector Store | Chroma / Qdrant / pgvector / FAISS |
| Retrieval | Custom / LangChain / LlamaIndex |
| Context Assembly | Custom / Framework |
| Prompting | Custom / LangChain / LlamaIndex |
| LLM | OpenAI / Anthropic / WatsonX / Google |
| Orchestration | Spring Boot / Python / LangGraph |
| Observability | OpenTelemetry / Prometheus / Grafana |
The framework should implement the architecture rather than replace the architectural concepts.
100. Why Retrieval and Generation Must Be Separated¶
Separating retrieval and generation allows engineers to answer:
or:
Without this separation:
becomes difficult to debug.
With clear boundaries:
each stage can be tested independently.
101. Key Takeaways¶
- A RAG runtime pipeline has two major stages:
- Retrieval
- Generation
- Retrieval determines which evidence the LLM receives.
- Generation determines how the LLM uses that evidence.
- Query processing prepares the user request.
- Query embedding converts the request into a searchable vector.
- Retrieval generates candidate evidence.
- Metadata and authorization filters restrict what can be retrieved.
- Ranking determines which candidates are most useful.
- Deduplication removes redundant evidence.
- Context assembly creates the final knowledge context.
- Token budgeting prevents excessive context from reaching the LLM.
- Prompt construction combines instructions, evidence, and the user question.
- Retrieved documents should be treated as data, not trusted instructions.
- LLM generation transforms retrieved evidence into a response.
- Response validation can enforce schema, business, and grounding requirements.
- Citation metadata should be preserved throughout the pipeline.
- Empty retrieval should not result in fabricated answers.
- Retrieval failures and generation failures must be diagnosed separately.
- Conversation context and retrieved knowledge are different types of context.
- Freshness and document versioning are important for enterprise RAG.
- Streaming improves perceived responsiveness.
- Caching can reduce latency and cost but must respect authorization and data freshness.
- Production RAG systems require timeouts, retries, rate limits, and failure handling.
- Every stage should be observable.
- Capability-based interfaces make the architecture provider-independent.
- LangChain and LlamaIndex can simplify implementation, but the underlying architecture remains the same.
- Retrieval quality and generation quality should be evaluated independently.
- A production RAG system should optimize:
- Relevance
- Recall
- Latency
- Cost
- Security
- Reliability
- Groundedness
The central principle is:
Retrieval finds the evidence, context engineering selects and organizes it, and generation turns that evidence into a useful response.
102. Chapter Navigation¶
Part IV — Prompt Engineering & RAG Fundamentals¶
Previous Chapter: 15. RAG Pipeline Components
Current Chapter: 16 — Retrieval and Generation Pipeline
Next Chapter: 17. Vector Databases in RAG
Part IV Chapters¶
- 01. Introduction to Prompt Engineering
- 02. Prompt Engineering Fundamentals
- 03. Advanced Prompt Engineering
- 04. Prompt Design Patterns
- 05. Zero-shot, One-shot & Few-shot Prompting
- 06. Chain-of-Thought Prompting
- 07. ReAct Prompting
- 08. Structured Outputs & Output Parsing
- 09. Function Calling & Tool Calling
- 10. Embeddings in Practice
- 11. Document Processing & Vectorization
- 12. Document Chunking Strategies
- 13. Vector Database Fundamentals
- 14. Similarity Search Techniques
- 15. RAG Pipeline Components
- 16. Retrieval and Generation Pipeline
- 17. Vector Databases in RAG
- 18. Building Your First RAG Pipeline
- 19. RAG Evaluation Fundamentals
- 20. Enterprise Generative AI Application Architecture
- 21. Deploying AI Applications with Gradio
References¶
- Retrieval-Augmented Generation architecture documentation
- LangChain documentation
- LlamaIndex documentation
- Hugging Face documentation
- Vector database documentation
- Embedding model documentation
- Enterprise search architecture documentation
- LLM application architecture documentation
- RAG evaluation and retrieval documentation
- OpenTelemetry documentation
- Enterprise AI observability and reliability documentation
Enterprise AI Engineering Handbook
Building Production-Grade Enterprise AI Systems — One Chapter at a Time.