15 — RAG Pipeline Components¶
Understand the core building blocks of a Retrieval-Augmented Generation (RAG) system and how document processing, chunking, embeddings, vector stores, retrieval, context assembly, prompting, and generation work together to build reliable enterprise AI applications.
📖 Overview¶
Retrieval-Augmented Generation (RAG) combines:
Instead of asking an LLM to answer a question using only its pretrained knowledge, a RAG system retrieves relevant information from an external knowledge source and provides that information as context to the model.
The basic architecture is:
Enterprise Documents
↓
Document Processing
↓
Chunking
↓
Embedding
↓
Vector Store
↓
Retrieval
↓
Context Assembly
↓
Prompt
↓
LLM
↓
Generated Answer
A production RAG system is therefore not a single model or database.
It is a pipeline of cooperating components.
1. What Is RAG?¶
Retrieval-Augmented Generation is an architecture where an LLM generates an answer using information retrieved from an external knowledge source.
Without RAG:
With RAG:
The key difference is the introduction of an external retrieval step.
2. Why RAG Is Required¶
LLMs have several limitations.
They may:
Not know private enterprise information
Have outdated knowledge
Generate unsupported answers
Lack access to internal documents
Need current information
Need document-level grounding
RAG addresses these limitations by retrieving information at inference time.
For example:
Question:
"What is our company's parental leave policy?"
↓
Retrieve:
Employee Handbook
HR Policy
Parental Leave Section
↓
LLM
↓
Grounded Answer
3. RAG vs Fine-Tuning¶
RAG and fine-tuning solve different problems.
RAG¶
Useful for:
Private Knowledge
Frequently Changing Information
Enterprise Documents
Search
Citations
Knowledge Grounding
Fine-Tuning¶
Useful for:
A simplified distinction:
4. RAG Pipeline Overview¶
flowchart TD
A["Enterprise Documents"] --> B["Document Processing"]
B --> C["Chunking"]
C --> D["Embedding Model"]
D --> E["Vector Store"]
F["User Query"] --> G["Query Processing"]
G --> H["Query Embedding"]
H --> E
E --> I["Retrieval"]
I --> J["Context Assembly"]
J --> K["Prompt Construction"]
K --> L["LLM"]
L --> M["Generated Answer"]This architecture contains two major paths:
and:
5. Indexing Path¶
The indexing path prepares knowledge for retrieval.
This generally happens:
and may run:
6. Query Path¶
The query path executes when a user asks a question.
User Query
↓
Query Processing
↓
Query Embedding
↓
Retrieval
↓
Context Assembly
↓
Prompt
↓
LLM
↓
Answer
The query path is usually latency-sensitive.
7. Two-Path RAG Architecture¶
flowchart TD
subgraph Indexing["Indexing Pipeline"]
A["Documents"] --> B["Processing"]
B --> C["Chunking"]
C --> D["Embedding"]
D --> E["Vector Store"]
end
subgraph Query["Query Pipeline"]
F["User Query"] --> G["Query Processing"]
G --> H["Query Embedding"]
H --> E
E --> I["Retrieval"]
I --> J["Context Assembly"]
J --> K["Prompt"]
K --> L["LLM"]
L --> M["Answer"]
endUnderstanding this separation is fundamental to designing production RAG systems.
8. Component 1 — Document Sources¶
RAG begins with knowledge sources.
Common enterprise sources include:
PDF
DOCX
PPTX
HTML
Markdown
CSV
JSON
Databases
Knowledge Bases
SharePoint
Confluence
Object Storage
Internal APIs
Web Content
The source is the authoritative location from which knowledge is extracted.
9. Component 2 — Document Processing¶
Raw documents are rarely ready for direct embedding.
A document may contain:
Document processing converts these sources into structured content.
10. Document Processing Example¶
A PDF might contain:
The processing layer should preserve useful metadata:
{
"text": "Employees are entitled to 25 days...",
"page": 42,
"document_id": "employee-handbook",
"section": "Annual Leave"
}
11. Component 3 — Document Chunking¶
Large documents should generally be divided into smaller retrieval units.
For example:
Chunking is critical because retrieval usually operates on chunks rather than entire documents.
12. Why Chunking Matters¶
Poor chunking can result in:
Good chunking should attempt to preserve meaningful semantic units.
Examples:
13. Component 4 — Metadata¶
Each chunk should carry useful metadata.
Example:
{
"chunk_id": "hr-policy-2026-007",
"document_id": "hr-policy-2026",
"page": 7,
"section": "Annual Leave",
"department": "HR",
"country": "IN",
"version": "2026",
"source": "sharepoint"
}
Metadata supports:
14. Component 5 — Embedding Model¶
An embedding model converts text into vectors.
Example:
The resulting vector represents the semantic characteristics of the content.
15. Query Embedding¶
The same embedding space is used for the user query.
Then:
The query and document vectors must be compatible.
16. Component 6 — Vector Store¶
The vector store stores:
Conceptually:
It provides the infrastructure required to search the embedded knowledge base.
17. Component 7 — Retriever¶
The retriever is responsible for finding relevant information.
Conceptually:
The retriever may internally perform:
18. Retriever Responsibilities¶
A retriever may handle:
Query Transformation
Embedding
Vector Search
Keyword Search
Metadata Filtering
Top-K
Thresholds
Candidate Selection
Advanced retrievers may also support:
These advanced techniques are covered later in the handbook.
19. Component 8 — Retrieval¶
Retrieval converts a user question into evidence.
Example:
Question:
"What is the annual leave entitlement?"
Retrieved:
Chunk 17
"Employees receive 25 days..."
Chunk 32
"Leave may be carried forward..."
Chunk 48
"Employees must request leave..."
20. Top-K Retrieval¶
The retriever normally returns a limited number of results.
For:
the system returns:
Choosing K is a trade-off.
21. Component 9 — Metadata Filtering¶
Enterprise retrieval often requires structured constraints.
Example:
Query:
"What is the leave policy?"
Filters:
tenant_id = tenant-a
country = IN
department = HR
version = current
The architecture becomes:
22. Security-Aware Retrieval¶
Security must not depend on the LLM.
Incorrect:
Correct:
The LLM should receive only information the user is authorized to access.
23. Component 10 — Context Assembly¶
Retrieval may return multiple chunks.
These need to be converted into a coherent context.
Example:
[Source: HR Policy, Page 7]
Employees are entitled to 25 days
of annual paid leave.
[Source: HR Policy, Page 8]
Unused leave may be carried forward
according to company policy.
24. Context Ordering¶
The order of retrieved information can matter.
Possible strategies include:
The correct strategy depends on the application.
25. Context Deduplication¶
Different chunks may contain overlapping information.
For example:
Sending both may waste context.
A context assembly stage can:
26. Component 11 — Prompt Construction¶
The retrieved context must be incorporated into the LLM prompt.
Conceptually:
Example:
System:
Answer using the provided context.
Context:
Employees receive 25 days of annual leave.
Question:
How many annual leave days do employees receive?
27. Grounded Prompt¶
A simple grounded prompt could be:
You are an enterprise knowledge assistant.
Answer the question using only the
provided context.
If the context does not contain enough
information, say that the information
could not be found.
Context:
{retrieved_context}
Question:
{user_query}
This provides an explicit grounding policy.
28. Component 12 — LLM¶
The LLM receives:
and generates:
The LLM's responsibility is primarily:
It should not be treated as the source of enterprise truth.
29. Component 13 — Generation¶
The generation stage produces the final response.
For example:
30. Component 14 — Citations¶
Enterprise RAG applications often need source attribution.
The response may include:
Citation metadata should originate from the retrieved documents rather than being invented by the model.
31. Citation Flow¶
flowchart LR
A["Source Document"] --> B["Chunk"]
B --> C["Metadata"]
C --> D["Vector Store"]
D --> E["Retriever"]
E --> F["Context"]
F --> G["LLM"]
E --> H["Source Metadata"]
H --> I["Citation Builder"]
G --> J["Answer"]
I --> JThis allows answers to be connected back to their source material.
32. Component 15 — Response Validation¶
Production systems may validate the generated response.
Possible checks include:
Structured Output Validation
Citation Validation
Policy Compliance
Content Safety
Grounding Checks
Schema Validation
The exact checks depend on the application.
33. Complete RAG Pipeline¶
flowchart TD
A["Source Systems"] --> B["Document Ingestion"]
B --> C["Document Processing"]
C --> D["Chunking"]
D --> E["Metadata Enrichment"]
E --> F["Embedding"]
F --> G["Vector Store"]
H["User"] --> I["Query"]
I --> J["Query Processing"]
J --> K["Query Embedding"]
K --> L["Retriever"]
G --> L
M["Authorization Context"] --> L
L --> N["Retrieved Chunks"]
N --> O["Context Assembly"]
O --> P["Prompt Construction"]
P --> Q["LLM"]
Q --> R["Response Validation"]
R --> S["Answer + Citations"]34. Indexing vs Retrieval Components¶
| Indexing Stage | Query Stage |
|---|---|
| Document Source | User Query |
| Document Processing | Query Processing |
| Chunking | Query Embedding |
| Metadata Enrichment | Retrieval |
| Embedding | Filtering |
| Vector Store | Context Assembly |
| Index Creation | Prompt Construction |
| Index Updates | LLM Generation |
This separation is important when designing production systems.
35. Component Dependency Graph¶
Document Source
↓
Document Processor
↓
Chunker
↓
Metadata Enricher
↓
Embedding Provider
↓
Vector Store
↓
Retriever
↓
Context Builder
↓
Prompt Builder
↓
LLM Provider
↓
Response Validator
↓
Final Response
Each component should have a clear responsibility.
36. Enterprise Capability Interfaces¶
A Java-first enterprise architecture can expose capabilities through interfaces.
public interface VectorStore {
void upsert(
List<VectorRecord> records
);
List<RetrievedDocument> search(
QueryVector query,
SearchOptions options
);
}
37. Retriever Interface¶
public interface Retriever {
List<RetrievedDocument> retrieve(
Query query,
RetrievalOptions options
);
}
The retriever can coordinate:
without exposing implementation details to the application.
38. Context Builder¶
Responsibilities may include:
39. Prompt Builder¶
This keeps prompt construction separate from retrieval.
40. LLM Provider¶
This makes the application independent of a specific model provider.
41. RAG Orchestrator¶
The overall workflow can be coordinated by an application service.
public class RagService {
private final Retriever retriever;
private final ContextBuilder contextBuilder;
private final PromptBuilder promptBuilder;
private final LLMProvider llmProvider;
public Answer answer(Query query) {
var documents =
retriever.retrieve(query, RetrievalOptions.defaults());
var context =
contextBuilder.build(documents);
var prompt =
promptBuilder.build(query, context);
var result =
llmProvider.generate(prompt);
return new Answer(result);
}
}
This separates orchestration from individual capabilities.
42. RAG Application Architecture¶
flowchart TD
A["API / UI"] --> B["RAG Application Service"]
B --> C["Retriever"]
B --> D["Context Builder"]
B --> E["Prompt Builder"]
B --> F["LLM Provider"]
C --> G["Embedding Provider"]
C --> H["Vector Store"]
G --> I["Embedding Model"]
H --> J["Vector Database"]
F --> K["LLM"]43. Ports & Adapters Architecture¶
A production RAG application can use:
Application Core
RAG Service
│
┌─────────────┼─────────────┐
↓ ↓ ↓
Retriever LLMProvider ContextBuilder
│
VectorStore
│
↓
Adapters
Concrete implementations may include:
OpenAIEmbeddingProvider
HuggingFaceEmbeddingProvider
WatsonXEmbeddingProvider
ChromaVectorStore
QdrantVectorStore
PgVectorStore
OpenAILLMProvider
WatsonXLLMProvider
AnthropicLLMProvider
This keeps the architecture provider-neutral.
44. Framework-Based RAG¶
Frameworks can simplify implementation.
Common frameworks include:
A framework may provide abstractions for:
Document Loaders
Text Splitters
Embeddings
Vector Stores
Retrievers
Prompt Templates
LLMs
RAG Chains
Agents
However, understanding the underlying components remains important.
45. LangChain Conceptual RAG¶
A simplified LangChain-style pipeline:
from langchain_core.prompts import ChatPromptTemplate
prompt = ChatPromptTemplate.from_template(
"""
Answer using only the following context.
Context:
{context}
Question:
{question}
"""
)
documents = retriever.invoke(
"What is the annual leave policy?"
)
context = "\n\n".join(
document.page_content
for document in documents
)
response = llm.invoke(
prompt.format(
context=context,
question="What is the annual leave policy?"
)
)
The framework simplifies orchestration, but the underlying stages remain:
46. LlamaIndex Conceptual RAG¶
A simplified LlamaIndex-style workflow:
from llama_index.core import VectorStoreIndex
index = VectorStoreIndex.from_documents(
documents
)
query_engine = index.as_query_engine(
similarity_top_k=5
)
response = query_engine.query(
"What is the annual leave policy?"
)
print(response)
The framework combines several RAG components behind the query interface.
47. Framework vs Architecture¶
A useful distinction:
while:
For example:
could be implemented using:
48. RAG Pipeline Data Flow¶
The data flowing through the system changes representation at each stage.
Raw Document
↓
Processed Document
↓
Document Chunks
↓
Embedding Vectors
↓
Indexed Records
↓
Retrieved Documents
↓
Context
↓
Prompt
↓
LLM Tokens
↓
Generated Response
Understanding these transformations makes RAG debugging much easier.
49. RAG Data Contracts¶
Each stage should ideally have a defined contract.
For example:
This is especially useful in enterprise architectures.
50. Document Chunk Contract¶
Metadata should preserve enough lineage to identify the original source.
51. Retrieval Result Contract¶
public record RetrievedDocument(
String chunkId,
String content,
double score,
Map<String, Object> metadata
) {
}
The score can support:
52. Context Contract¶
This provides a clear boundary between retrieval and prompt construction.
53. RAG Configuration¶
A production RAG pipeline may need configuration for:
rag:
retrieval:
top-k: 10
similarity-threshold: 0.72
context:
max-chunks: 8
deduplicate: true
generation:
temperature: 0.1
max-tokens: 1000
Configuration should be:
54. Token Budget¶
Retrieved context consumes LLM context-window capacity.
Therefore:
More context does not automatically produce better answers.
55. Context Window Management¶
A production context builder may need to:
Rank chunks
Remove duplicates
Trim irrelevant content
Prioritize high-value chunks
Respect token budget
Preserve citations
Conceptually:
flowchart TD
A["Retrieved Chunks"] --> B["Rank"]
B --> C["Deduplicate"]
C --> D["Token Budget"]
D --> E["Context Selection"]
E --> F["Prompt"]56. Empty Retrieval Handling¶
A robust RAG pipeline must handle:
Possible behavior:
For example:
The system should not fabricate an answer simply because retrieval failed.
57. Retrieval Failure vs Generation Failure¶
These are different problems.
Retrieval Failure¶
The correct information was not retrieved.
Generation Failure¶
The correct information was retrieved but the LLM produced a poor response.
Debugging must identify which stage failed.
58. RAG Observability¶
Monitor every major stage.
Document Processing
↓
Chunking Metrics
↓
Embedding Metrics
↓
Indexing Metrics
↓
Retrieval Metrics
↓
Context Metrics
↓
LLM Metrics
↓
Response Metrics
Useful metrics include:
Latency
Token Usage
Error Rate
Retrieval Recall
Empty Retrieval Rate
Similarity Scores
Context Size
LLM Cost
59. RAG Trace¶
A production request should ideally have a trace:
with stages:
This allows engineers to understand where latency and failures originate.
60. Production RAG Workflow¶
1. Receive user request.
2. Authenticate user.
3. Resolve tenant and authorization context.
4. Validate the query.
5. Generate query embedding.
6. Execute retrieval.
7. Apply metadata and security filters.
8. Select Top-K candidates.
9. Apply similarity threshold where appropriate.
10. Optionally rerank candidates.
11. Remove duplicate content.
12. Build context.
13. Enforce token budget.
14. Construct grounded prompt.
15. Invoke LLM.
16. Validate generated output.
17. Attach citations.
18. Return response.
19. Record telemetry.
20. Evaluate retrieval and generation quality.
61. Production RAG Architecture¶
flowchart TD
A["Client"] --> B["API Gateway"]
B --> C["Authentication / Authorization"]
C --> D["RAG Application"]
D --> E["Query Processor"]
E --> F["Embedding Provider"]
F --> G["Vector Store"]
G --> H["Vector Database"]
D --> I["Retriever"]
I --> G
J["Tenant / Security Context"] --> I
I --> K["Candidate Results"]
K --> L["Ranking / Reranking"]
L --> M["Context Builder"]
M --> N["Prompt Builder"]
N --> O["LLM Provider"]
O --> P["LLM"]
P --> Q["Response Validation"]
Q --> R["Citation Builder"]
R --> S["Final Response"]
D --> T["Observability"]
I --> T
O --> T
Q --> T62. Common RAG Mistakes¶
62.1 Treating RAG as Just Vector Search¶
RAG includes:
62.2 Poor Document Processing¶
If the source content is extracted incorrectly, retrieval quality will suffer.
62.3 Poor Chunking¶
Bad chunk boundaries can remove important context.
62.4 Missing Metadata¶
Without metadata, enterprise filtering and citations become difficult.
62.5 Retrieving Too Much¶
Large context can increase:
62.6 Retrieving Too Little¶
A small Top-K can miss necessary evidence.
62.7 No Security Filtering¶
Sensitive documents must never reach unauthorized users.
62.8 Treating the LLM as the Source of Truth¶
The retrieved enterprise knowledge should be the primary source for grounded answers.
62.9 No Empty-Result Handling¶
The system must explicitly handle situations where evidence cannot be found.
62.10 No Observability¶
Without traces and metrics, debugging RAG quality becomes difficult.
63. Best Practices¶
1. Separate indexing and query pipelines.
2. Keep source documents authoritative.
3. Preserve document lineage.
4. Use semantic chunking where appropriate.
5. Attach meaningful metadata.
6. Use compatible embedding models.
7. Keep vector-store access behind an abstraction.
8. Enforce tenant isolation.
9. Apply authorization-aware retrieval.
10. Tune Top-K using evaluation data.
11. Calibrate similarity thresholds.
12. Use reranking when required.
13. Deduplicate retrieved context.
14. Respect LLM token budgets.
15. Keep prompts grounded.
16. Handle empty retrieval explicitly.
17. Preserve citation metadata.
18. Validate generated responses where required.
19. Monitor retrieval and generation independently.
20. Trace the complete RAG request.
21. Measure both retrieval quality and system performance.
22. Version retrieval configuration.
23. Re-evaluate after embedding or chunking changes.
24. Keep framework code separate from business capabilities.
25. Design for provider and vector-store portability.
64. RAG Component Checklist¶
[ ] Document Sources
[ ] Document Processor
[ ] Document Chunker
[ ] Metadata Enrichment
[ ] Embedding Provider
[ ] Vector Store
[ ] Vector Index
[ ] Query Processor
[ ] Retriever
[ ] Metadata Filtering
[ ] Authorization Filtering
[ ] Top-K Selection
[ ] Similarity Threshold
[ ] Optional Reranker
[ ] Context Builder
[ ] Token Budget Management
[ ] Prompt Builder
[ ] LLM Provider
[ ] Response Validation
[ ] Citation Builder
[ ] Observability
[ ] Evaluation
65. RAG Component Responsibilities¶
| Component | Primary Responsibility |
|---|---|
| Document Source | Provide authoritative knowledge |
| Document Processor | Extract and normalize content |
| Chunker | Create retrieval units |
| Metadata Enricher | Attach searchable context |
| Embedding Provider | Convert text into vectors |
| Vector Store | Store and search vectors |
| Retriever | Find relevant information |
| Filter | Restrict eligible results |
| Reranker | Improve candidate ordering |
| Context Builder | Assemble usable context |
| Prompt Builder | Create grounded instructions |
| LLM Provider | Generate response |
| Validator | Check output |
| Citation Builder | Preserve source attribution |
| Observability | Measure system behavior |
66. Framework Mapping¶
The same conceptual architecture can be implemented using different tools.
| Capability | Possible Implementation |
|---|---|
| Document Loading | Custom / LangChain / LlamaIndex |
| Chunking | Custom / LangChain / LlamaIndex |
| Embeddings | OpenAI / Hugging Face / WatsonX / Custom |
| Vector Store | Chroma / FAISS / Qdrant / pgvector / others |
| Retrieval | Custom / LangChain / LlamaIndex |
| Prompting | Custom / LangChain / LlamaIndex |
| LLM | OpenAI / Anthropic / WatsonX / others |
| Orchestration | Spring Boot / Python / LangChain / LlamaIndex / LangGraph |
The architecture should remain understandable even when the implementation framework changes.
67. RAG as a Composable Architecture¶
The major lesson is that RAG should be treated as a set of capabilities:
Document Processing
+
Chunking
+
Embedding
+
Vector Search
+
Retrieval
+
Context Management
+
Prompting
+
Generation
+
Validation
This allows individual components to evolve independently.
68. Final Architecture¶
flowchart LR
A["Enterprise Knowledge"] --> B["Ingestion"]
B --> C["Processing"]
C --> D["Chunking"]
D --> E["Embedding"]
E --> F["Vector Index"]
G["User"] --> H["Query"]
H --> I["Query Embedding"]
I --> J["Retriever"]
F --> J
K["Authorization"] --> J
J --> L["Relevant Context"]
L --> M["Context Builder"]
M --> N["Prompt"]
N --> O["LLM"]
O --> P["Validation"]
P --> Q["Answer + Citations"]69. Key Takeaways¶
- RAG combines information retrieval with LLM generation.
- A RAG system consists of multiple independent components.
- The indexing pipeline prepares enterprise knowledge for retrieval.
- The query pipeline retrieves and uses that knowledge at runtime.
- Document processing converts raw sources into usable content.
- Chunking creates retrieval units.
- Metadata enables filtering, security, versioning, and citations.
- Embedding models convert text into vectors.
- Vector stores provide searchable storage for embeddings.
- Retrievers locate relevant information.
- Metadata filters restrict the search space.
- Authorization must be enforced before sensitive content reaches the LLM.
- Context assembly converts retrieved chunks into usable LLM context.
- Prompt construction combines instructions, context, and the user question.
- The LLM generates the final response.
- Citation metadata should be preserved from the source.
- Response validation can enforce application-specific requirements.
- Token budgets must be managed carefully.
- Too much retrieved context can introduce noise and increase cost.
- Too little context can cause missing evidence.
- Empty retrieval should be handled explicitly.
- Retrieval failures and generation failures should be diagnosed separately.
- Production RAG systems require observability across the entire pipeline.
- Frameworks such as LangChain and LlamaIndex simplify implementation but do not replace architectural understanding.
- A Java-first enterprise implementation can use capability-based interfaces such as:
DocumentProcessorDocumentChunkerEmbeddingProviderVectorStoreRetrieverContextBuilderPromptBuilderLLMProvider- Ports & Adapters architecture allows infrastructure providers to change without rewriting application logic.
- RAG should be treated as a composable enterprise architecture rather than a single framework feature.
The central principle is:
A production RAG system is a coordinated pipeline that transforms enterprise knowledge into retrievable evidence and uses that evidence to ground LLM generation.
70. Chapter Navigation¶
Part IV — Prompt Engineering & RAG Fundamentals¶
Previous Chapter: 14. Similarity Search Techniques
Current Chapter: 15 — RAG Pipeline Components
Next Chapter: 16. Retrieval and Generation Pipeline
Part IV Chapters¶
- 01. Introduction to Prompt Engineering
- 02. Prompt Engineering Fundamentals
- 03. Advanced Prompt Engineering
- 04. Prompt Design Patterns
- 05. Zero-shot, One-shot & Few-shot Prompting
- 06. Chain-of-Thought Prompting
- 07. ReAct Prompting
- 08. Structured Outputs & Output Parsing
- 09. Function Calling & Tool Calling
- 10. Embeddings in Practice
- 11. Document Processing & Vectorization
- 12. Document Chunking Strategies
- 13. Vector Database Fundamentals
- 14. Similarity Search Techniques
- 15. RAG Pipeline Components
- 16. Retrieval and Generation Pipeline
- 17. Vector Databases in RAG
- 18. Building Your First RAG Pipeline
- 19. RAG Evaluation Fundamentals
- 20. Enterprise Generative AI Application Architecture
- 21. Deploying AI Applications with Gradio
References¶
- Retrieval-Augmented Generation architecture documentation
- LangChain documentation
- LlamaIndex documentation
- Hugging Face documentation
- Vector database documentation
- Embedding model documentation
- Enterprise search architecture documentation
- LLM application architecture documentation
- RAG evaluation and retrieval documentation
Enterprise AI Engineering Handbook
Building Production-Grade Enterprise AI Systems — One Chapter at a Time.