05. Multimodal RAG¶
Category: Advanced RAG Architecture
Module: Part V — Advanced Retrieval-Augmented Generation
Difficulty: Advanced
📖 Overview¶
Traditional Retrieval-Augmented Generation (RAG) systems are primarily designed around text:
Enterprise knowledge, however, is rarely text-only.
Important information can exist in:
Text
Images
Tables
Charts
Diagrams
Screenshots
Scanned Documents
Audio
Video
Presentations
Technical Drawings
Forms
Invoices
Contracts
A Multimodal RAG system extends RAG so that retrieval and generation can work across multiple information modalities.
Instead of treating every document as plain text, Multimodal RAG preserves and retrieves the different forms of evidence contained in enterprise knowledge.
The architecture therefore becomes:
Enterprise Knowledge
│
┌────────────┬───────┼────────┬───────────┐
▼ ▼ ▼ ▼ ▼
Text Images Tables Audio Video
│ │ │ │ │
└────────────┴───────┼────────┴───────────┘
▼
Multimodal Retrieval
│
▼
Context Assembly
│
▼
Multimodal Foundation Model
│
▼
Enterprise Response
The goal is not simply to "put images into a prompt."
The goal is to build a production-grade retrieval architecture that understands, indexes, retrieves, grounds, validates, and cites multimodal enterprise knowledge.
🎯 Learning Objectives¶
After completing this chapter, you will be able to:
-
Understand Multimodal RAG
-
Understand why text-only RAG is insufficient for many enterprise workloads
-
Understand multimodal documents
-
Understand multimodal ingestion pipelines
-
Extract text, images, tables, and layout information
-
Understand OCR in Multimodal RAG
-
Understand image embeddings
-
Understand multimodal embeddings
-
Understand cross-modal retrieval
-
Understand image-to-text retrieval
-
Understand text-to-image retrieval
-
Understand image-to-image retrieval
-
Understand multimodal chunking
-
Understand document layout preservation
-
Understand table-aware retrieval
-
Understand chart and diagram retrieval
-
Understand visual document retrieval
-
Design multimodal vector stores
-
Combine text and image retrieval
-
Build multimodal context
-
Use multimodal foundation models
-
Handle scanned documents
-
Handle PDFs containing images and tables
-
Build multimodal enterprise RAG pipelines
-
Combine Multimodal RAG with Graph RAG and SQL RAG
-
Design multimodal citation and provenance
-
Secure multimodal data
-
Evaluate multimodal retrieval and generation
-
Optimize multimodal RAG latency and cost
-
Design production Multimodal RAG systems
🧠 1. What Is Multimodal RAG?¶
Multimodal RAG is a RAG architecture that retrieves and uses information from multiple modalities.
For example:
A user may ask:
A text-only RAG system may fail if the relationship exists only inside the diagram.
A Multimodal RAG system can retrieve:
and provide both to the model.
🔎 2. Traditional RAG vs Multimodal RAG¶
Traditional Text RAG¶
Multimodal RAG¶
Document
↓
Multimodal Parsing
↓
┌─────────────┬─────────────┬─────────────┐
▼ ▼ ▼
Text Images Tables
│ │ │
▼ ▼ ▼
Embeddings Embeddings Structured Index
└─────────────┴─────────────┘
│
▼
Multimodal Retrieval
│
▼
Multimodal Context
│
▼
Multimodal Model
🧩 3. What Makes Enterprise Data Multimodal?¶
Consider a technical architecture document.
It may contain:
Title
Paragraphs
Architecture Diagram
Tables
Code Snippets
Screenshots
Sequence Diagram
Deployment Diagram
A text extractor may capture:
but completely miss the meaning of:
Multimodal RAG attempts to preserve these relationships.
📚 4. Common Enterprise Multimodal Sources¶
Typical sources include:
PDF
PowerPoint
Word Documents
Scanned Documents
Images
Architecture Diagrams
Technical Drawings
Invoices
Receipts
Contracts
Forms
Dashboards
Screenshots
Product Catalogs
Medical Images
Satellite Images
Video Recordings
Audio Recordings
🧠 5. Multimodal RAG Mental Model¶
A useful mental model is:
USER QUERY
│
▼
QUERY UNDERSTANDING
│
▼
MODALITY ANALYSIS
│
┌───────────┼───────────┐
▼ ▼ ▼
TEXT IMAGE TABLE
│ │ │
└───────────┼───────────┘
▼
MULTIMODAL RETRIEVAL
│
▼
EVIDENCE FUSION
│
▼
CONTEXT ENGINEERING
│
▼
MULTIMODAL MODEL
│
▼
VALIDATED RESPONSE
🧩 6. Multimodal Data Model¶
A multimodal document should not necessarily be represented as one large blob.
Instead:
Each component can have its own representation.
🏗️ 7. Multimodal Document Representation¶
Example:
{
"document_id": "architecture-001",
"page": 12,
"elements": [
{
"type": "text",
"content": "Payment architecture"
},
{
"type": "image",
"asset_id": "img-128"
},
{
"type": "table",
"asset_id": "table-22"
}
]
}
This preserves the document structure.
📐 8. Layout Matters¶
Consider:
The text alone:
does not preserve the relationships represented spatially.
Layout-aware processing is therefore important for many enterprise documents.
🧠 9. Document Layout¶
A document parser may identify:
The ingestion system can preserve:
🏗️ 10. Multimodal Ingestion Pipeline¶
flowchart TD
A["Enterprise Document"] --> B["Document Parser"]
B --> C["Layout Analysis"]
C --> D["Text Extraction"]
C --> E["Image Extraction"]
C --> F["Table Extraction"]
C --> G["Chart / Diagram Extraction"]
D --> H["Text Processing"]
E --> I["Visual Processing"]
F --> J["Table Processing"]
G --> K["Visual Processing"]
H --> L["Multimodal Index"]
I --> L
J --> L
K --> L
🔍 11. OCR¶
OCR stands for:
OCR converts text contained in images into machine-readable text.
Example:
OCR is particularly important for:
🧠 12. OCR Is Not Document Understanding¶
OCR provides:
but does not necessarily understand:
For example:
may be extracted correctly, but the system may still need layout information to determine which value belongs to which field in a complex form.
🧩 13. OCR + Layout Understanding¶
A stronger pipeline is:
Example:
🖼️ 14. Image Embeddings¶
Images can be represented as vectors.
Conceptually:
Example:
This allows image similarity search.
🔗 15. Multimodal Embeddings¶
A multimodal embedding model can map different modalities into a compatible semantic space.
Conceptually:
Then:
can potentially retrieve:
even though the query and retrieved object have different modalities.
🧠 16. Cross-Modal Retrieval¶
Cross-modal retrieval means the query and result can use different modalities.
Examples:
Example:
🔎 17. Text-to-Image Retrieval¶
This is useful for:
🖼️ 18. Image-to-Text Retrieval¶
Example:
User uploads:
Architecture Diagram
↓
Visual Embedding
↓
Retrieve:
Architecture Documentation
Runbook
API Documentation
This allows visual information to become a retrieval signal.
🖼️ 19. Image-to-Image Retrieval¶
Example:
This can support:
📊 20. Table Retrieval¶
Tables require special treatment.
Consider:
\| Product | Revenue | Growth |
\|---------|---------|--------|
\| A | 10M | 12% |
\| B | 8M | 17% |
Naively embedding the table as plain text may lose:
A better architecture can preserve both:
🧩 21. Table-Aware Representation¶
Example:
{
"type": "table",
"columns": [
"Product",
"Revenue",
"Growth"
],
"rows": [
["A", "10M", "12%"],
["B", "8M", "17%"]
]
}
The system can additionally generate a textual representation for semantic retrieval.
📈 22. Charts¶
Charts contain information that may not exist directly as text.
Example:
Text extraction might only capture:
The actual trend must be understood visually or reconstructed from structured chart data.
🧠 23. Chart Understanding¶
A multimodal pipeline may extract:
The representation could be:
{
"chart_type": "line",
"title": "Monthly Sales",
"x_axis": ["Jan", "Feb", "Mar"],
"trend": "increasing"
}
🖼️ 24. Diagram Understanding¶
Enterprise diagrams may represent:
A multimodal model can analyze:
🏗️ 25. Architecture Diagram Example¶
┌─────────────┐
│ Client │
└──────┬──────┘
│
▼
┌─────────────┐
│ API Gateway │
└──────┬──────┘
│
┌───────┴───────┐
▼ ▼
┌──────────┐ ┌──────────┐
│ Payment │ │ Auth │
│ Service │ │ Service │
└────┬─────┘ └──────────┘
│
▼
┌──────────┐
│PostgreSQL│
└──────────┘
The graph of relationships may be:
A multimodal system can retrieve the diagram and derive these relationships.
🔗 26. Multimodal RAG and Knowledge Graphs¶
Visual relationships can be transformed into graph knowledge.
Architecture Diagram
↓
Vision Understanding
↓
Entity Extraction
↓
Relationship Extraction
↓
Knowledge Graph
Example:
This connects Multimodal RAG with Knowledge Graph RAG.
🧠 27. Multimodal RAG and SQL¶
Tables and charts may contain structured information.
For example:
A production system may transform the information into:
and store it in SQL.
Then:
can enable exact analytical queries.
🔀 28. Multimodal + SQL + Graph + Vector¶
A mature enterprise system may combine:
Enterprise Query
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Vector SQL Graph
│ │ │
▼ ▼ ▼
Text Tables Relationships
│ │ │
└─────────────┼─────────────┘
│
▼
Multimodal
Evidence
│
▼
LLM
This creates a broader enterprise knowledge fabric.
🧩 29. Multimodal Chunking¶
Traditional chunking:
Multimodal chunking may preserve:
Example:
{
"chunk_id": "chunk-102",
"text": "Payment architecture...",
"images": [
"architecture-12.png"
],
"tables": [
"dependency-table-12"
],
"page": 12,
"section": "Payment Architecture"
}
🧠 30. Parent-Child Multimodal Retrieval¶
A child chunk may represent:
while the parent context contains:
Retrieval can find the child and return the parent multimodal context.
🔎 31. Caption-Based Image Retrieval¶
Images can be enriched with generated captions.
Example:
Caption:
"Microservices architecture showing an API Gateway
connected to Payment and Authentication services."
Then:
can support image retrieval.
🧠 32. Image Metadata¶
Useful image metadata includes:
Document ID
Page
Section
Image Type
Caption
Bounding Box
Creation Date
Source
Access Policy
Related Text
Related Entities
🧩 33. Multimodal Index¶
A production index may contain:
or a unified multimodal vector index.
Conceptually:
Multimodal Index
│
┌────────────────┼────────────────┐
▼ ▼ ▼
Text Vectors Image Vectors Metadata
│ │ │
└────────────────┼────────────────┘
▼
Retrieval Layer
🏗️ 34. Unified vs Separate Indexes¶
Unified Index¶
Advantages:
Separate Indexes¶
Advantages:
A hybrid approach is often useful.
🔀 35. Hybrid Multimodal Retrieval¶
flowchart TD
A["User Query"] --> B["Query Analysis"]
B --> C["Text Retrieval"]
B --> D["Image Retrieval"]
B --> E["Table Retrieval"]
C --> F["Text Candidates"]
D --> G["Image Candidates"]
E --> H["Table Candidates"]
F --> I["Candidate Fusion"]
G --> I
H --> I
I --> J["Cross-Modal Re-ranking"]
J --> K["Multimodal Context"]
🧠 36. Modality-Aware Query Routing¶
Not every query needs every modality.
Example:
"What does the refund policy say?"
↓
Text
"Show the architecture diagram for payments."
↓
Image
"What was revenue in Q2?"
↓
SQL / Table
"Which service connects to the payment gateway?"
↓
Graph + Image + Text
The query planner should determine which modalities are relevant.
🔎 37. Modality Router¶
class ModalityRouter:
def route(self, query):
"""
Return required retrieval modalities.
"""
raise NotImplementedError
Possible result:
🏗️ 38. Multimodal Retriever Interface¶
A capability-oriented architecture can expose:
class MultimodalRetriever:
def retrieve(
self,
query,
modalities=None,
filters=None,
top_k=10
):
raise NotImplementedError
This keeps the application independent of a specific retrieval implementation.
🏛️ 39. Ports & Adapters Architecture¶
flowchart LR
A["RAG Application"] --> B["Multimodal Retrieval Port"]
B --> C["Text Adapter"]
B --> D["Image Adapter"]
B --> E["Table Adapter"]
C --> F["Text Index"]
D --> G["Image Index"]
E --> H["Structured Store"]
The application layer should depend on capabilities rather than infrastructure SDKs.
🧠 40. Multimodal Foundation Models¶
A multimodal model can process multiple input types.
Conceptually:
For RAG, retrieved evidence may therefore include:
in a single model request.
🔗 41. Multimodal Context¶
Instead of:
a prompt may contain:
TEXT:
Payment service documentation...
IMAGE:
Architecture diagram...
TABLE:
Service dependencies...
The model can reason over the combined evidence.
🧠 42. Context Ordering¶
Context ordering matters.
A possible structure:
SYSTEM INSTRUCTIONS
USER QUESTION
RELEVANT TEXT
RELEVANT TABLES
RELEVANT IMAGES
SOURCE METADATA
RESPONSE REQUIREMENTS
The exact structure depends on the model and workload.
🧩 43. Visual Context Selection¶
Do not blindly pass every retrieved image.
Use:
Example:
🔍 44. Image Re-ranking¶
Candidate images can be ranked using:
Example:
Only the strongest candidates need to reach the generation stage.
🧠 45. Multimodal Evidence Fusion¶
Different modalities can provide different evidence.
Example:
Text:
"Payment Service uses PostgreSQL."
Image:
Architecture diagram showing
Payment Service → PostgreSQL.
Graph:
PaymentService --USES--> PostgreSQL.
The system can combine:
to improve grounding.
🛡️ 46. Evidence Agreement¶
A production system can detect whether different evidence sources agree.
Text:
Payment Service → PostgreSQL
Image:
Payment Service → PostgreSQL
Graph:
Payment Service → PostgreSQL
Agreement:
But:
requires additional validation.
⚠️ 47. Conflicting Multimodal Evidence¶
Enterprise documents may contain contradictory information.
Example:
A production system should consider:
rather than simply combining both.
🕒 48. Multimodal Freshness¶
Visual documents can become stale.
Example:
may no longer represent:
Metadata should include:
📚 49. Document Versioning¶
A useful model:
Retrieval should normally prefer:
unless the user explicitly requests historical information.
🧠 50. Multimodal Provenance¶
Every visual evidence item should retain:
Example:
{
"asset_id": "image-928",
"document_id": "architecture-v4",
"page": 12,
"section": "Payment Architecture",
"bbox": [120, 240, 980, 740],
"version": "4.0"
}
🔗 51. Multimodal Citations¶
Text citations:
Image citations:
Table citations:
The final response should preserve the connection between the generated claim and its evidence.
🧾 52. Citation Example¶
Response:
Evidence:
This is stronger than simply citing:
🧠 53. Multimodal RAG Grounding¶
Grounding means that generated claims should be supported by retrieved evidence.
For example:
Claim:
Payment Service uses PostgreSQL.
Evidence:
Architecture Diagram
\+
Architecture Documentation
The generation layer should avoid adding unsupported details.
🚨 54. Multimodal Hallucination¶
Multimodal models can hallucinate:
For example:
This is a visual hallucination.
🛡️ 55. Multimodal Response Validation¶
Validation can compare generated claims against:
Conceptually:
Response
↓
Claim Extraction
↓
Evidence Matching
↓
Unsupported Claim Detection
↓
Correction / Rejection
🧠 56. Table Grounding¶
If a model answers:
the system should be able to identify:
as evidence.
📈 57. Chart Grounding¶
If a model says:
the evidence should point to:
rather than only the surrounding text.
🧩 58. Multimodal Response Contract¶
A structured internal response can contain:
{
"answer": "...",
"claims": [
{
"text": "Payment Service uses PostgreSQL.",
"evidence": [
{
"type": "image",
"asset_id": "architecture-12"
},
{
"type": "text",
"chunk_id": "chunk-812"
}
]
}
]
}
This supports downstream validation and citation.
🔐 59. Multimodal Security¶
Images and documents may contain sensitive information.
Examples:
Screenshots
Credentials
Customer Data
Internal Architecture
Financial Information
Personal Information
Security Diagrams
Security controls should apply to:
🛡️ 60. Image-Level Authorization¶
A user may be allowed to access:
but not:
Therefore authorization should be evaluated at the asset level.
👥 61. Multi-Tenant Multimodal RAG¶
Tenant boundaries should be preserved across:
Example:
Tenant A
├── Text
├── Images
├── Tables
└── Graph Data
Tenant B
├── Text
├── Images
├── Tables
└── Graph Data
Cross-tenant retrieval must be prevented.
🧠 62. Multimodal PII¶
PII may exist inside images even when metadata contains no obvious PII.
For example:
OCR can expose this information.
Therefore security and privacy scanning should consider both:
🧪 63. Multimodal Evaluation¶
Evaluation should cover multiple layers:
Document Parsing
OCR
Image Retrieval
Text Retrieval
Table Retrieval
Cross-Modal Retrieval
Evidence Fusion
Answer Generation
Citation
📊 64. Retrieval Metrics¶
For retrieval:
For image retrieval:
🧠 65. Visual Question Answering Evaluation¶
For visual questions:
Example:
Question:
Which service is connected to PostgreSQL?
Image:
Architecture Diagram
Expected:
Payment Service
📊 66. Multimodal Groundedness¶
Evaluate:
Is the answer supported by the image?
Is the answer supported by the table?
Is the answer supported by the text?
Are citations pointing to the correct evidence?
🧪 67. Multimodal Evaluation Dataset¶
Example:
{
"question": "Which service uses PostgreSQL?",
"image": "architecture-12.png",
"expected_answer": "Payment Service",
"evidence": {
"page": 12,
"figure": 3
}
}
A production evaluation set should include:
Easy Visual Questions
Complex Diagrams
Tables
Charts
OCR Cases
Cross-Modal Questions
Conflicting Evidence
Low-Quality Images
🚨 68. Multimodal Failure Modes¶
Common failures include:
OCR Errors
Image Retrieval Failure
Incorrect Image Interpretation
Wrong Table Extraction
Layout Loss
Chart Misinterpretation
Incorrect Cross-Modal Linking
Stale Visual Evidence
Missing Provenance
Visual Hallucination
Context Overflow
High Inference Cost
🧩 69. Low-Resolution Images¶
Poor image quality can cause:
A preprocessing pipeline may use:
Only apply expensive processing where required.
🖼️ 70. Image Cropping¶
Instead of passing a full page:
retrieve only:
This can reduce:
🔍 71. Region-Level Retrieval¶
A document image can be divided into regions:
Each region can have independent metadata and embeddings.
🧠 72. Visual Chunking¶
Visual chunking is analogous to text chunking.
The retrieval layer can then operate at the appropriate granularity.
🧩 73. Multimodal Parent-Child Retrieval¶
A query may retrieve:
but return:
This provides richer context.
⚡ 74. Performance Optimization¶
Multimodal processing can be expensive.
Optimization areas include:
OCR
Image Embeddings
Visual Embeddings
Vision Model Calls
Image Storage
Network Transfer
Context Size
💰 75. Cost Optimization¶
Avoid sending every image to a multimodal model.
Use a funnel:
Broad Retrieval
↓
Metadata Filtering
↓
Text / Caption Retrieval
↓
Visual Re-ranking
↓
Top Images
↓
Vision Model
This reduces expensive inference.
⚡ 76. Two-Stage Multimodal Retrieval¶
flowchart TD
A["User Query"] --> B["Cheap Candidate Retrieval"]
B --> C["Top 50 Candidates"]
C --> D["Metadata / Text Filtering"]
D --> E["Top 10 Candidates"]
E --> F["Visual Re-ranking"]
F --> G["Top 3"]
G --> H["Multimodal LLM"]
This is usually more scalable than sending dozens of images directly to the model.
🧠 77. Modality-Specific Caching¶
Cache:
For immutable documents:
🗄️ 78. Asset Storage¶
Original assets should generally be stored separately from embeddings.
Object Storage
│
├── Original PDF
├── Images
├── Cropped Regions
└── Extracted Assets
Vector Store
│
└── Embeddings + Metadata
The vector store should not necessarily become the primary binary asset store.
🧩 79. Multimodal Metadata¶
Useful metadata:
asset_id
document_id
page
section
modality
mime_type
caption
entities
created_at
updated_at
version
tenant_id
access_policy
source_uri
🏗️ 80. Production Multimodal RAG Architecture¶
flowchart TD
A["Enterprise Sources"] --> B["Ingestion Layer"]
B --> C["Document Parsing"]
C --> D["Text"]
C --> E["Images"]
C --> F["Tables"]
C --> G["Audio / Video"]
D --> H["Text Embeddings"]
E --> I["Visual Embeddings"]
F --> J["Structured Representation"]
G --> K["Transcription / Visual Processing"]
H --> L["Multimodal Retrieval Layer"]
I --> L
J --> L
K --> L
L --> M["Metadata Filtering"]
M --> N["Cross-Modal Re-ranking"]
N --> O["Evidence Fusion"]
O --> P["Context Engineering"]
P --> Q["Multimodal Foundation Model"]
Q --> R["Response Validation"]
R --> S["Citation / Attribution"]
S --> T["Enterprise Response"]
🔄 81. End-to-End Multimodal RAG¶
ENTERPRISE SOURCES
│
┌───────────────┼───────────────┐
▼ ▼ ▼
Text Images Tables
│ │ │
▼ ▼ ▼
Text Parsing Vision/OCR Table Parsing
│ │ │
└───────────────┼───────────────┘
▼
Multimodal Indexing
│
▼
Query Analysis
│
┌──────────────┼──────────────┐
▼ ▼ ▼
Text Image Table
Retrieval Retrieval Retrieval
│ │ │
└──────────────┼──────────────┘
▼
Candidate Fusion
│
▼
Re-ranking
│
▼
Evidence Selection
│
▼
Multimodal Context
│
▼
Multimodal LLM
│
┌────────┴────────┐
▼ ▼
Validation Citation
│ │
└────────┬────────┘
▼
Enterprise Response
🧠 82. Multimodal RAG + Agentic RAG¶
Agents can use modality-specific tools.
Agent
│
├── Text Search
│
├── Image Search
│
├── Table / SQL Tool
│
├── Knowledge Graph
│
├── OCR
│
└── Vision Analysis
Example:
User:
"Analyze the architecture diagram and
identify the database used by Payment Service."
Agent
↓
Image Retrieval
↓
Vision Analysis
↓
Knowledge Graph Verification
↓
Response
🔗 83. Multimodal RAG + Knowledge Graph¶
A diagram can become graph evidence:
The graph can then be queried independently.
This creates a pipeline:
🔀 84. Multimodal RAG + SQL¶
Tables can become structured data:
This allows questions such as:
to be answered using exact structured computation rather than visual approximation.
🏢 85. Enterprise Multimodal Knowledge Fabric¶
flowchart TD
A["Enterprise Knowledge"] --> B["Knowledge Fabric"]
B --> C["Documents"]
B --> D["Vector Store"]
B --> E["Knowledge Graph"]
B --> F["SQL"]
B --> G["Image / Media Store"]
C --> H["RAG Orchestrator"]
D --> H
E --> H
F --> H
G --> H
H --> I["Multimodal Foundation Model"]
I --> J["Validation"]
J --> K["Enterprise Response"]
🧠 86. Multimodal Query Router¶
A production router may classify queries into:
Example:
class QueryModality:
TEXT_ONLY = "text"
IMAGE = "image"
TABLE = "table"
GRAPH = "graph"
SQL = "sql"
MULTIMODAL = "multimodal"
🧩 87. Capability-Based Architecture¶
Rather than coupling the application to one multimodal provider:
class VisionProvider:
def analyze_image(self, image, prompt):
raise NotImplementedError
class OCRProvider:
def extract_text(self, image):
raise NotImplementedError
class EmbeddingProvider:
def embed_text(self, text):
raise NotImplementedError
def embed_image(self, image):
raise NotImplementedError
Cloud- or model-specific implementations can sit behind these interfaces.
🏛️ 88. Ports & Adapters¶
Application
│
┌──────────┼──────────┐
▼ ▼ ▼
VisionPort OCRPort EmbeddingPort
│ │ │
▼ ▼ ▼
Adapter A Adapter B Adapter C
│ │ │
└──────────┼──────────┘
▼
AI / Cloud Services
This keeps the application architecture portable.
🧪 89. Practical Exercise¶
Build a multimodal document collection:
Extract:
Then index them.
🔎 90. Practice Queries¶
Test:
1\. Find the architecture diagram for the payment platform.
2\. Which service connects to PostgreSQL?
3\. What does the architecture diagram show?
4\. What was the highest-revenue region?
5\. Find the invoice containing customer Acme.
6\. What amount appears on the invoice?
7\. Which database is shown in the deployment diagram?
8\. Compare the architecture in version 2 and version 3.
🧪 91. Compare Retrieval Strategies¶
Implement:
A. Text-only RAG
B. Image-only Retrieval
C. Text + Image Retrieval
D. Text + Image + Table Retrieval
E. Text + Image + Table + Graph
Measure:
📊 92. Example Evaluation Matrix¶
| Architecture | Retrieval | Grounding | Latency | Cost |
|---|---|---|---|---|
| Text RAG | Text | Medium | Low | Low |
| Image Retrieval | Visual | Medium | Medium | Medium |
| Text + Image | High | High | Medium | Medium |
| Text + Image + Table | High | High | Medium | Medium |
| Full Multimodal + Graph + SQL | Very High | Very High | High | High |
The exact results depend on the dataset and implementation.
🚨 93. Common Mistakes¶
Mistake 1 — Converting Everything to Text¶
This can destroy visual relationships.
Mistake 2 — Sending Every Image to the LLM¶
This increases:
Mistake 3 — Ignoring Layout¶
Spatial relationships can contain important meaning.
Mistake 4 — Treating OCR as Complete Understanding¶
OCR provides text, not complete visual semantics.
Mistake 5 — Ignoring Tables¶
Tables should often be represented structurally.
Mistake 6 — Ignoring Provenance¶
Visual evidence should be traceable to:
Mistake 7 — Ignoring Versioning¶
Old diagrams can produce incorrect answers.
Mistake 8 — Trusting Vision Models Blindly¶
Visual models can hallucinate objects and relationships.
Mistake 9 — Ignoring Security¶
Images can contain highly sensitive information.
Mistake 10 — Building Multimodal Infrastructure Without a Use Case¶
Multimodal processing can be expensive.
Start with a measurable business requirement.
🧠 94. Design Principles¶
Principle 1 — Preserve Modality¶
Do not convert everything to text if visual structure carries meaning.
Principle 2 — Retrieve Before Reasoning¶
Use retrieval to reduce the amount of visual and textual information sent to the model.
Principle 3 — Use the Right Representation¶
Text → Text Embedding
Image → Visual Embedding
Table → Structured Representation
Graph → Relationship Representation
Principle 4 — Preserve Layout¶
Page and region relationships can be critical.
Principle 5 — Preserve Provenance¶
Every multimodal asset should be traceable.
Principle 6 — Route by Modality¶
Do not perform expensive visual reasoning when text retrieval is sufficient.
Principle 7 — Validate Visual Claims¶
Multimodal models can hallucinate.
Principle 8 — Combine Modalities¶
Different modalities often provide complementary evidence.
Principle 9 — Secure Every Representation¶
Protect:
Principle 10 — Optimize the Expensive Path¶
Use:
before expensive multimodal inference.
📋 95. Production Checklist¶
☐ Identify multimodal use cases
☐ Identify supported modalities
☐ Identify source systems
☐ Identify document types
☐ Implement document parsing
☐ Implement layout extraction
☐ Implement OCR
☐ Implement image extraction
☐ Implement table extraction
☐ Implement chart extraction
☐ Implement diagram extraction
☐ Preserve page information
☐ Preserve bounding boxes
☐ Preserve section relationships
☐ Preserve captions
☐ Preserve document versions
☐ Generate text embeddings
☐ Generate image embeddings
☐ Define multimodal embedding strategy
☐ Build multimodal indexes
☐ Build metadata indexes
☐ Implement text retrieval
☐ Implement image retrieval
☐ Implement table retrieval
☐ Implement cross-modal retrieval
☐ Implement modality routing
☐ Implement candidate fusion
☐ Implement re-ranking
☐ Implement multimodal context assembly
☐ Implement visual context selection
☐ Implement context compression
☐ Implement vision model integration
☐ Implement response validation
☐ Implement citation resolution
☐ Implement provenance tracking
☐ Implement authentication
☐ Implement authorization
☐ Implement tenant isolation
☐ Protect sensitive images
☐ Protect OCR output
☐ Protect embeddings
☐ Implement asset versioning
☐ Implement freshness tracking
☐ Implement deletion handling
☐ Evaluate OCR quality
☐ Evaluate image retrieval
☐ Evaluate text retrieval
☐ Evaluate table retrieval
☐ Evaluate cross-modal retrieval
☐ Evaluate groundedness
☐ Evaluate citation accuracy
☐ Monitor retrieval latency
☐ Monitor vision inference latency
☐ Monitor OCR latency
☐ Monitor model cost
☐ Monitor context size
☐ Monitor failed retrievals
☐ Cache embeddings
☐ Cache OCR
☐ Cache image analysis
☐ Use candidate filtering
☐ Use visual re-ranking
☐ Control multimodal model calls
☐ Build regression datasets
☐ Test conflicting evidence
☐ Test stale documents
☐ Test low-quality images
☐ Test security boundaries
📚 96. Key Takeaways¶
-
Multimodal RAG extends RAG beyond text-only knowledge.
-
Enterprise knowledge often exists in images, tables, charts, diagrams, audio, and video.
-
OCR is important for scanned documents but does not replace visual understanding.
-
Layout information can be critical for preserving document meaning.
-
Images can be represented using visual embeddings.
-
Multimodal embeddings can enable cross-modal retrieval.
-
Text-to-image retrieval allows natural-language queries to find visual evidence.
-
Image-to-text retrieval allows visual inputs to retrieve textual documentation.
-
Image-to-image retrieval supports visual similarity use cases.
-
Tables should often retain their structural representation.
-
Charts and diagrams can contain information that plain text extraction loses.
-
Multimodal chunking should preserve relationships between text, images, tables, captions, and sections.
-
Parent-child retrieval can return richer multimodal context.
-
Caption generation can make visual assets easier to retrieve.
-
Separate or unified multimodal indexes can be used depending on requirements.
-
Modality-aware routing can reduce unnecessary inference cost.
-
Cross-modal re-ranking can improve retrieval quality.
-
Multimodal evidence should preserve provenance.
-
Visual evidence should be linked to document, page, figure, or region.
-
Multimodal models can hallucinate visual facts and therefore require grounding and validation.
-
Conflicting visual and textual evidence should be resolved using source authority, version, and temporal metadata.
-
Knowledge Graphs can represent relationships extracted from diagrams and visual documents.
-
SQL can provide exact structured computation for information extracted from tables.
-
Multimodal RAG can therefore work together with Vector RAG, Graph RAG, and SQL RAG.
-
Security must apply to original assets, OCR, embeddings, metadata, and retrieved context.
-
Multimodal RAG can be significantly more expensive than text-only RAG.
-
Candidate filtering, caching, cropping, and re-ranking help control cost.
-
Production Multimodal RAG requires evaluation, observability, provenance, security, and governance.
🧠 Final Mental Model¶
ENTERPRISE KNOWLEDGE
│
┌──────────────┬───────────┼───────────┬──────────────┐
▼ ▼ ▼ ▼ ▼
TEXT IMAGES TABLES AUDIO VIDEO
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
Parsing OCR Extraction Transcription Processing
│ │ │ │ │
└──────────────┴───────────┼───────────┴──────────────┘
▼
MULTIMODAL INDEXING
│
┌────────────┼────────────┐
▼ ▼ ▼
Text Index Image Index Structured Index
│ │ │
└────────────┼────────────┘
▼
QUERY ANALYSIS
│
▼
MODALITY ROUTING
│
┌──────────────────┼──────────────────┐
▼ ▼ ▼
Text Image Table
Retrieval Retrieval Retrieval
│ │ │
└──────────────────┼──────────────────┘
▼
CANDIDATE FUSION
│
▼
CROSS-MODAL RANKING
│
▼
EVIDENCE SELECTION
│
▼
CONTEXT ENGINEERING
│
▼
MULTIMODAL FOUNDATION MODEL
│
┌──────────┴──────────┐
▼ ▼
Validation Citation
│ │
└──────────┬──────────┘
▼
ENTERPRISE RESPONSE
The central idea is:
Multimodal RAG preserves and retrieves knowledge in the form in which it actually exists, rather than forcing every enterprise artifact into plain text.
A mature enterprise architecture therefore becomes:
Enterprise Knowledge
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
Documents Images Tables
│ │ │
▼ ▼ ▼
Vector Search Visual Search SQL
│ │ │
└─────────────────┼─────────────────┘
│
┌────────────┴────────────┐
▼ ▼
Knowledge Graph Other Sources
│ │
└────────────┬────────────┘
▼
Evidence Fusion
│
▼
Context Engineering
│
▼
Multimodal Foundation Model
│
▼
Response Validation
│
▼
Citation / Attribution
│
▼
Enterprise Response
The important architectural principle is:
RIGHT KNOWLEDGE
│
▼
RIGHT MODALITY
│
▼
RIGHT RETRIEVER
│
▼
RIGHT EVIDENCE
│
▼
MULTIMODAL REASONING
│
▼
VALIDATED RESPONSE
Multimodal RAG is therefore not simply:
It is a broader retrieval architecture that combines:
Multimodal Ingestion
\+
Layout Understanding
\+
Modality-Specific Indexing
\+
Cross-Modal Retrieval
\+
Evidence Fusion
\+
Multimodal Reasoning
\+
Provenance
\+
Validation
\+
Security
\+
Observability
This makes Multimodal RAG particularly valuable for enterprise applications involving technical documentation, architecture repositories, financial reports, invoices, contracts, product catalogs, dashboards, engineering documents, visual knowledge bases, and other domains where critical information exists outside plain text.
🧠 84. Cross-Modal Embeddings and Contrastive Learning¶
A Multimodal RAG system needs a representation that lets its retriever compare a query with evidence from the relevant modalities. One approach is to use a model trained to align related items—such as an image and a text description—in a shared embedding space.
Contrastive learning trains representations so that related pairs become more similar and unrelated pairs become less similar, according to the training objective. For example, an image of a cat and the phrase “a domestic feline” may be mapped to nearby representations. This alignment can enable a text query to retrieve an image, or an image query to retrieve related text.
Paired training examples
(image, matching description)
│
▼
Modality encoders
│
▼
Contrastive objective
│
▼
Aligned embedding space
│
▼
Cross-modal retrieval
For an application using a pretrained model, the engineering task is usually to select a suitable aligned encoder, apply the correct preprocessing, create compatible embeddings for indexed items and queries, and evaluate retrieval quality. Training or fine-tuning the embedding model is a separate option—not a mandatory step for every Multimodal RAG system.
Important distinction: embeddings are useful for similarity search, but they are not the original media and do not guarantee that two items are identical. Retrieval scores must be evaluated against representative queries and labelled evidence.
🧩 85. Practical Pattern: Visual Product Retrieval with Style Finder¶
A useful example is a visual shopping assistant. A user uploads a photograph of a complete outfit; the application searches an external product dataset and returns relevant clothing details and purchase links. The system combines visual similarity retrieval with a vision-language model (VLM), rather than asking the model to invent catalogue facts.
flowchart TD
A["Uploaded outfit image"] --> B["Validate and preprocess image"]
B --> C["ResNet50 feature extraction"]
C --> D["Query feature vector"]
D --> E["Cosine-similarity ranking"]
E --> F["Best matching outfit record"]
F --> G["Retrieve associated catalogue items"]
G --> H["Product names, prices and URLs"]
B --> I["Encode image for model transport"]
I --> J["Vision-language model"]
H --> J
J --> K["Validate and format response"]A practical flow is:
- Prepare the image. Validate the file and apply the same preprocessing used for catalogue images.
- Extract visual features. A pretrained ResNet50 model from
torchvisioncan produce a feature vector for similarity matching. - Rank candidates. Compare the query vector with precomputed dataset vectors using cosine similarity.
- Resolve the catalogue record. Use the best match to retrieve the related outfit items and their authoritative metadata.
- Augment the model request. Supply the uploaded image and relevant catalogue records to a VLM, with a prompt asking it to describe visible characteristics and organize the retrieved information.
- Validate the response. Check structure and ensure product names, prices, and URLs are grounded in the catalogue rather than generated by the model.
Feature vectors and Base64 are not interchangeable¶
In this pattern, two representations serve different purposes:
- Feature vector: a numerical representation used to compare images and rank retrieval candidates.
- Base64-encoded image: a transport representation that allows image bytes to be included in a request format that accepts Base64 data.
Base64 is not an image embedding, does not encode semantic similarity, and should not be placed in a vector index as though it were a feature vector. Keep the original media, retrieval features, and request payload conceptually separate.
Cosine similarity is a ranking signal—not proof of exact product identity. If the best match is weak or ambiguous, the application should return a qualified result, show alternatives, or ask for clarification. Product facts and purchase links must come from the trusted catalogue.
🎧 86. Audio and Video as Retrievable Evidence¶
The chapter already treats audio and video as supported modalities. In production, they need modality-specific preprocessing and metadata so that retrieval can return the correct passage, speaker, frame, or time interval—not merely an entire recording.
Audio indexing¶
A practical audio ingestion path may include:
Audio asset
↓
Format and quality validation
↓
Speech recognition / transcription
↓
Transcript segments + timestamps
↓
Optional speaker and audio metadata
↓
Text indexing + retained audio reference
A transcript supports text search over spoken content, while the original audio may be needed for voice characteristics, sound events, or verification of an uncertain transcription. Keep timestamps and source references so a retrieved transcript segment can be traced back to the relevant audio interval. If speaker diarization is used, retain speaker labels as model-produced metadata and evaluate their accuracy rather than treating them as guaranteed identities.
Video indexing¶
Video retrieval should preserve temporal structure. A pipeline can sample representative frames, extract or transcribe speech, and associate both with timestamps:
Video
├── Sampled frames → visual descriptions / image embeddings
├── Audio track → transcription / audio analysis
└── Timestamps and segment metadata
↓
Linked multimodal index
Frame sampling frequency is a quality–cost trade-off: sparse sampling reduces processing expense but can miss brief events or transitions. A retrieved frame or transcript segment should retain its timestamp and link to the source video. For questions about a sequence of events, retrieve and assemble multiple relevant intervals rather than assuming one frame captures the whole event.
Audio and video support depends on the selected parser, embedding model, retrieval design, and generation endpoint. Do not assume that one vector index or one model supports every modality or every cross-modal query.
⚖️ 87. Choosing an MM-RAG Design: Quality, Latency and Cost¶
Multimodal retrieval adds capabilities, but also introduces more processing stages and more opportunities for error. Choose the simplest architecture that satisfies the retrieval and evidence requirements.
| Design choice | Useful when | Trade-off to evaluate |
|---|---|---|
| Shared embedding space | Text and media must retrieve one another | Alignment quality depends on the encoder and domain |
| Separate modality indexes | Each modality needs specialized search or scaling | Candidate fusion and score calibration become important |
| Generated captions for image search | Text search is useful and image-native retrieval is limited | Captions may omit details or introduce errors |
| OCR and layout extraction | Small text, forms, tables, or document structure matter | OCR and layout errors can propagate into retrieval |
| Visual reranking | Initial retrieval returns many plausible images | Better ranking may add model latency and cost |
| Original media in model context | The model must inspect visual or acoustic details | Media tokens, context limits, and inference cost can rise |
| Structured extraction into SQL or a graph | Questions require exact values or explicit relationships | Extraction must be validated and provenance retained |
Use staged retrieval when possible: apply inexpensive metadata and text filters first, then use visual or multimodal reranking on a smaller candidate set. Cache reusable extraction results and embeddings for unchanged assets, while invalidating them when source content, preprocessing, or encoder versions change.
Do not treat a larger number of modalities as an automatic quality improvement. Measure whether each modality contributes useful evidence to the target task.
🧪 88. MM-RAG Evaluation: Test the Retrieval Path End to End¶
A useful evaluation set should cover the complete path from source processing to grounded answer. Include examples where the evidence is visual, textual, tabular, cross-modal, noisy, stale, or contradictory.
| Layer | Example checks |
|---|---|
| Parsing and extraction | OCR accuracy, table-cell extraction, layout and region association |
| Embedding and retrieval | Recall@K, Precision@K, MRR or NDCG for each supported query–result modality pair |
| Candidate fusion and reranking | Whether the best evidence survives fusion and reaches the final context |
| Generation | Answer correctness, completeness, uncertainty handling, and unsupported-claim rate |
| Grounding and citations | Whether each material claim points to the correct document, page, figure, table, region, or time interval |
| Operations | End-to-end and per-stage latency, failure rate, model usage, and cost per successful request |
| Security | Authorization filtering, tenant isolation, sensitive-content handling, and resistance to instructions embedded in media |
Build labelled examples with an expected answer and an evidence pointer—for example, document ID, page, figure, bounding box, or video timestamp. Include difficult cases such as low-resolution diagrams, similar-looking products, incorrect OCR, missing frames, ambiguous speech, and conflicting document versions.
Evaluate both retrieval and generation. A correct answer produced from the wrong evidence is still a grounding failure; a relevant retrieved item that the model misinterprets is a generation or interpretation failure. Track these failure types separately so remediation targets the right component.
🧭 89. Design Checklist for a Production MM-RAG Pipeline¶
Before deployment, verify that the system:
- [ ] Preserves source IDs, page/region references, timestamps, versions, and access policies.
- [ ] Uses compatible preprocessing and embedding models for indexed items and queries.
- [ ] Keeps source media, embeddings, metadata, and model transport payloads separate.
- [ ] Supports modality-aware routing and avoids invoking every retriever for every query.
- [ ] Fuses candidates with a defined ranking strategy and selects evidence under a context budget.
- [ ] Returns the correct parent context when a small region or child chunk is retrieved.
- [ ] Grounds generated claims in retrieved evidence and cites the appropriate source location.
- [ ] Handles low-confidence matches, missing evidence, and conflicting sources explicitly.
- [ ] Enforces authorization before evidence enters model context, including at image and region level.
- [ ] Evaluates retrieval quality, answer quality, citation accuracy, latency, and cost separately.
- [ ] Caches reusable processing results and tracks encoder, parser, and source versions.
- [ ] Logs traceable operational metadata without unnecessarily storing raw media or sensitive payloads.
The implementation should remain modular: parsing, modality-specific processing, embedding, retrieval, reranking, evidence assembly, generation, validation, and citation should have explicit responsibilities. This makes it possible to improve one stage without silently changing the assumptions of the whole pipeline.
🧭 Chapter Navigation¶
Part V — Advanced Retrieval-Augmented Generation¶
Previous:
[04. SQL RAG](04-sql-rag.md)
Next:
[06. Agentic RAG](06-agentic-rag.md)
Section:
05 — Advanced RAG Architecture
Advanced RAG Architecture Path¶
01 Advanced RAG Architecture
↓
02 Graph RAG
↓
03 Knowledge Graphs for RAG
↓
04 SQL RAG
↓
05 Multimodal RAG
↓
06 Agentic RAG
Enterprise AI Engineering Handbook
Building Production-Grade Enterprise AI Systems — One Chapter at a Time.