Metadata-Aware Retrieval¶
📖 Overview¶
Metadata-aware retrieval enhances traditional retrieval by using structured information associated with documents, chunks, users, tenants, and business entities.
Instead of relying only on semantic similarity:
metadata-aware retrieval adds additional signals and constraints:
Metadata can describe:
Document
├── source
├── title
├── author
├── department
├── document_type
├── created_at
├── updated_at
├── language
├── version
├── tenant_id
├── access_level
├── product
├── region
└── status
In enterprise RAG, metadata is not merely descriptive information.
It can become an important part of:
🎯 Learning Objectives¶
After completing this chapter, you will be able to:
- Understand metadata-aware retrieval
- Understand document and chunk metadata
- Design useful metadata schemas
- Apply metadata filters during retrieval
- Understand pre-filtering vs post-filtering
- Implement tenant-aware retrieval
- Apply date and time filters
- Use metadata for document type filtering
- Understand metadata-based routing
- Combine metadata with semantic similarity
- Implement metadata-aware ranking
- Understand authorization vs metadata filtering
- Design hierarchical metadata
- Handle metadata inheritance
- Understand metadata normalization
- Avoid common metadata design problems
- Build production-grade metadata-aware retrieval pipelines
- Monitor and evaluate metadata-driven retrieval
1. Why Metadata Matters in RAG¶
Semantic similarity answers:
"Which documents are conceptually similar to this query?"
Metadata can answer:
"Which documents are allowed, relevant, current, authoritative, or applicable?"
Consider:
Semantic search might return:
2022 Payment Policy
2023 Payment Policy
2024 Payment Policy
2025 Payment Policy
Current Payment Policy
Metadata can narrow the candidate space:
The resulting search space becomes much more precise.
2. Metadata vs Document Content¶
A document contains:
Metadata describes:
Example:
{
"content": "Payment authentication requires OAuth...",
"metadata": {
"document_id": "DOC-1024",
"document_type": "security_policy",
"department": "security",
"version": "4.2",
"status": "approved",
"updated_at": "2026-07-15"
}
}
The content answers:
Metadata answers:
What is this document?
When was it updated?
Who owns it?
Who can access it?
What category does it belong to?
3. Metadata-Aware Retrieval Architecture¶
flowchart TD
A["User Query"] --> B["Query Processing"]
B --> C["Metadata Extraction"]
C --> D["Security / Tenant Context"]
D --> E["Metadata Filters"]
E --> F["Vector / Keyword Retrieval"]
F --> G["Candidate Documents"]
G --> H["Re-ranking"]
H --> I["Context Selection"]
I --> J["LLM"]Metadata can therefore influence retrieval before, during, and after semantic search.
4. Types of Metadata¶
Common enterprise metadata categories include:
Example:
{
"document_id": "DOC-1001",
"tenant_id": "tenant-a",
"department": "finance",
"document_type": "policy",
"language": "en",
"region": "EU",
"created_at": "2026-01-10",
"updated_at": "2026-07-20",
"status": "approved",
"version": "3.1"
}
5. Identity Metadata¶
Identity metadata identifies the source object.
Examples:
Example:
Identity metadata is critical for:
6. Source Metadata¶
Source metadata describes where the content originated.
Examples:
source = confluence
source = sharepoint
source = s3
source = database
source = github
source = uploaded_file
Example:
Source information becomes useful when users ask:
7. Document Type Metadata¶
Documents can be classified:
Example:
This can support queries such as:
The system can prioritize:
8. Temporal Metadata¶
Time metadata is particularly important for enterprise knowledge.
Examples:
Example:
This enables retrieval based on:
9. Current vs Historical Knowledge¶
Consider:
All three may be semantically similar.
Metadata can identify:
For a current-policy question:
can help select the correct version.
10. Organizational Metadata¶
Enterprise documents often belong to organizational structures:
Example:
This supports scoped retrieval.
11. Business Metadata¶
Business metadata can describe domain-specific entities.
Examples:
Example:
This can be used to narrow enterprise search.
12. Geographic Metadata¶
Useful fields include:
Example:
A legal or compliance query may require jurisdiction-specific retrieval.
13. Language Metadata¶
For multilingual systems:
can be used to restrict retrieval.
Example:
However, language filtering should be used carefully if cross-lingual retrieval is supported.
14. Security Metadata¶
Security metadata may include:
Example:
{
"tenant_id": "tenant-a",
"classification": "internal",
"allowed_roles": [
"finance-admin",
"finance-user"
]
}
Security metadata must be treated differently from ordinary ranking metadata.
15. Authorization vs Metadata Filtering¶
This distinction is critical.
Metadata Filtering¶
Example:
This determines relevance.
Authorization¶
Example:
This determines whether the document can be returned at all.
Therefore:
Never use ranking to compensate for missing authorization controls.
16. Multi-Tenant Retrieval¶
Enterprise SaaS systems frequently use:
Documents should contain:
At query time:
The retrieval system must ensure:
17. Tenant Isolation Architecture¶
flowchart TD
A["User Request"] --> B["Identity / Tenant Context"]
B --> C["Authorization Policy"]
C --> D["Tenant Filter"]
D --> E["Metadata Filters"]
E --> F["Retrieval"]
F --> G["Ranking"]
G --> H["LLM"]Tenant isolation should be enforced at the retrieval boundary.
18. Why Post-Filtering Can Be Dangerous¶
Consider:
Suppose:
After filtering:
Relevant Tenant A documents may have never entered the Top-10.
More importantly, depending on implementation, unauthorized data may have been exposed to an intermediate system.
A safer architecture is:
19. Pre-Filtering¶
Pre-filtering means:
Example:
Then vector search operates within that filtered space.
This can improve:
when supported by the vector store.
20. Post-Filtering¶
Post-filtering means:
This may be problematic when:
contains many documents that will later be removed.
The final result set may become too small.
21. Pre-Filtering vs Post-Filtering¶
| Approach | Filtering Point | Advantages | Risks |
|---|---|---|---|
| Pre-filtering | Before retrieval | Better isolation and candidate quality | Requires database support |
| Post-filtering | After retrieval | Simple implementation | Can lose relevant results |
| Hybrid | Multiple stages | Flexible | More complexity |
For authorization, enforce filtering as early and as strongly as the architecture permits.
22. Basic Metadata Filter¶
Conceptually:
The exact filter syntax depends on the vector database.
23. Multiple Metadata Conditions¶
Example:
Conceptually:
24. OR Conditions¶
Some systems support:
Conceptually:
Exact syntax varies by database.
25. Range Filters¶
Metadata can support numeric and temporal ranges.
Example:
or:
Conceptually:
The exact query language depends on the vector database.
26. Metadata Extraction from the Query¶
Metadata-aware retrieval becomes more powerful when metadata constraints can be inferred from natural language.
Query:
The system can derive:
Then:
can be executed together.
27. Query-to-Filter Architecture¶
flowchart TD
A["Natural Language Query"] --> B["Query Understanding"]
B --> C["Semantic Query"]
B --> D["Metadata Constraints"]
C --> E["Retriever"]
D --> E
E --> F["Filtered Candidate Pool"]
F --> G["Re-ranking"]
G --> H["Context"]This is a powerful enterprise retrieval pattern.
28. Self-Query Retrieval¶
A self-query retriever can translate natural-language requests into:
Example:
The system produces:
This connects metadata-aware retrieval with self-query retrieval.
29. Metadata Schema¶
A metadata schema should be designed intentionally.
Example:
{
"document_id": "DOC-123",
"parent_id": "DOC-123",
"tenant_id": "tenant-a",
"source": "confluence",
"document_type": "architecture",
"department": "engineering",
"team": "payments",
"product": "payment-gateway",
"language": "en",
"region": "EU",
"status": "approved",
"version": "4.2",
"created_at": "2026-01-12",
"updated_at": "2026-07-20"
}
30. Metadata Should Be Structured¶
Avoid putting everything into one field:
Prefer:
{
"department": "engineering",
"team": "payments",
"region": "EU",
"status": "approved",
"year": 2026
}
Structured metadata enables:
31. Metadata Normalization¶
Inconsistent metadata reduces retrieval quality.
Bad:
Better:
Normalize metadata during ingestion.
32. Metadata Taxonomy¶
Define controlled values.
Example:
Instead of allowing arbitrary values.
This prevents:
from representing the same category.
33. Metadata Validation¶
Validate metadata during ingestion.
REQUIRED_FIELDS = [
"document_id",
"tenant_id",
"document_type",
"status"
]
def validate_metadata(metadata):
missing = [
field
for field in REQUIRED_FIELDS
if field not in metadata
]
if missing:
raise ValueError(
f"Missing metadata: {missing}"
)
This prevents incomplete documents from entering the retrieval system.
34. Metadata at Document Level¶
Example:
When chunked:
document-level metadata often needs to be inherited by every chunk.
35. Metadata Inheritance¶
document_metadata = {
"document_id": "DOC-123",
"department": "engineering",
"status": "approved"
}
chunk_metadata = {
**document_metadata,
"chunk_id": "DOC-123-C01"
}
This allows every chunk to be independently filtered and traced.
36. Chunk-Level Metadata¶
Some metadata belongs specifically to chunks.
Examples:
Example:
37. Document-Level vs Chunk-Level Metadata¶
| Metadata | Level |
|---|---|
| document_id | Document |
| title | Document |
| author | Document |
| department | Document |
| version | Document |
| chunk_id | Chunk |
| page_number | Chunk |
| section | Chunk |
| paragraph_index | Chunk |
| table_id | Chunk |
Some metadata may exist at both levels.
38. Hierarchical Metadata¶
Enterprise knowledge can have hierarchical structure:
Metadata can preserve this hierarchy.
Example:
{
"department": "engineering",
"team": "payments",
"project": "payment-platform",
"document_id": "DOC-123",
"section": "authentication",
"chunk_id": "DOC-123-C08"
}
39. Hierarchical Filtering¶
A query might specify:
The system can search:
This reduces the search space.
40. Metadata and Parent-Child Retrieval¶
Metadata can help map:
Example:
This enables retrieval at multiple levels.
41. Metadata and Versioning¶
Enterprise documentation frequently evolves.
Example:
Metadata should identify:
This allows the system to distinguish:
42. Version-Aware Retrieval¶
Example:
Alternatively:
The correct implementation depends on how lifecycle metadata is modeled.
43. Temporal Retrieval¶
Queries may explicitly include time:
Metadata extraction:
Then retrieval can target:
rather than current documents.
44. Relative Time Queries¶
Users may say:
The query processor may need to resolve these expressions into structured metadata constraints.
For example:
becomes:
The exact interpretation should be based on the request date and application semantics.
45. Metadata and Freshness¶
Metadata can support freshness-aware retrieval:
Example:
Then:
Freshness should be treated as a ranking preference unless the business rule requires a hard time constraint.
46. Metadata-Aware Ranking¶
Metadata can influence ranking.
Conceptually:
Example:
final_score = (
0.70 * semantic_score
+ 0.15 * authority_score
+ 0.10 * freshness_score
+ 0.05 * business_priority
)
The weights are illustrative.
They should be calibrated using an evaluation dataset.
47. Metadata vs Re-ranking¶
Metadata filtering:
Re-ranking:
Example:
These are separate concerns.
48. Metadata + Re-ranking + MMR¶
A mature pipeline may look like:
User Query
↓
Authorization
↓
Metadata Filtering
↓
Hybrid Retrieval
↓
Candidate Pool
↓
Re-ranking
↓
MMR
↓
Context Selection
↓
LLM
Each stage contributes a different capability.
49. Metadata-Aware Hybrid Retrieval¶
flowchart TD
A["Query"] --> B["Metadata Extraction"]
B --> C["Security Filter"]
C --> D["Business Filters"]
D --> E["Dense Retrieval"]
D --> F["BM25"]
E --> G["Candidate Fusion"]
F --> G
G --> H["Re-ranking"]
H --> I["MMR"]
I --> J["Context"]Metadata can therefore constrain multiple retrieval strategies consistently.
50. Metadata Routing¶
Metadata can also determine which retriever should be used.
Example:
document_type = api_reference
↓
Technical Retriever
document_type = policy
↓
Policy Retriever
document_type = incident
↓
Incident Retriever
This creates:
51. Metadata Routing Architecture¶
flowchart TD
A["Query"] --> B["Query Understanding"]
B --> C["Metadata / Intent"]
C --> D{"Document Type"}
D -->|API| E["API Retriever"]
D -->|Policy| F["Policy Retriever"]
D -->|Incident| G["Incident Retriever"]
D -->|General| H["General Retriever"]
E --> I["Candidate Results"]
F --> I
G --> I
H --> I
I --> J["Re-ranking"]This is useful when different knowledge domains require different retrieval strategies.
52. Metadata and Query Routing¶
Suppose:
The query processor may infer:
The router can then choose:
rather than searching the entire enterprise corpus.
53. Metadata as a Retrieval Contract¶
A strong architecture defines metadata fields as part of the retrieval contract.
Example:
Required:
tenant_id
document_id
document_type
Recommended:
source
department
status
updated_at
Optional:
region
product
language
priority
This makes retrieval behavior predictable.
54. Metadata Schema Evolution¶
Metadata schemas evolve.
Version 1:
Version 2:
Version 3:
The ingestion and retrieval systems should support schema evolution carefully.
55. Metadata Versioning¶
Example:
This helps identify:
It is particularly useful in long-lived enterprise RAG platforms.
56. Metadata Backfilling¶
If a new metadata field is introduced:
existing documents may not contain it.
Options include:
Avoid silently treating missing metadata as equivalent to a valid value.
57. Missing Metadata¶
Suppose:
is required for a policy query.
But some documents have:
The retrieval system should not automatically interpret:
Instead:
should remain distinct.
58. Metadata Quality¶
Metadata quality can be measured.
Example:
Other useful metrics:
Poor metadata can degrade retrieval even when embeddings are excellent.
59. Metadata Observability¶
Track:
Missing Metadata
Invalid Values
Unknown Categories
Filter Usage
Filter Rejection Rate
No-Result Queries
Metadata Extraction Errors
Example:
{
"metadata_filter": {
"document_type": "policy",
"status": "approved"
},
"candidate_count": 142,
"filtered_count": 38
}
60. No-Result Queries¶
Metadata filtering can become too restrictive.
Example:
Result:
The system needs a controlled strategy.
Possible responses:
Never relax authorization filters automatically.
61. Hard vs Soft Metadata Constraints¶
Hard Constraint¶
Must not be relaxed.
Soft Constraint¶
May be relaxed when appropriate.
This distinction is essential for safe retrieval design.
62. Filter Relaxation¶
Example:
If no results:
while preserving:
This is an advanced retrieval strategy.
63. Safe Filter Relaxation¶
flowchart TD
A["Query"] --> B["Required Filters"]
A --> C["Optional Filters"]
B --> D["Filtered Retrieval"]
C --> D
D --> E{"Results?"}
E -->|Yes| F["Continue"]
E -->|No| G["Relax Optional Filter"]
G --> D
B -. Never Relax .-> DRequired security constraints must remain enforced.
64. Metadata Extraction with LLM¶
An LLM can convert natural language into structured filters.
Example prompt:
Extract retrieval filters from the query.
Return JSON:
{
"document_type": "...",
"department": "...",
"region": "...",
"date_from": "...",
"date_to": "..."
}
Input:
Output:
{
"document_type": "architecture",
"department": "payments",
"region": "EU",
"status": "approved",
"date_from": "2026-01-01"
}
65. Structured Filter Validation¶
Never blindly execute LLM-generated filters.
Validate:
Example:
Reject unexpected fields.
66. Filter Injection¶
Natural-language filter generation introduces a potential attack surface.
A malicious query could attempt:
The application must never allow the LLM to override:
The security context must come from trusted application state.
67. Trusted vs Untrusted Metadata¶
Trusted Metadata¶
Created by:
Untrusted Metadata¶
Extracted from:
Trusted metadata should dominate security decisions.
68. Metadata Injection from Documents¶
A document may contain text such as:
That does not make it authoritative security metadata.
Security metadata should come from:
rather than arbitrary document content.
69. Metadata Security Architecture¶
flowchart TD
A["Identity Provider"] --> B["Trusted Security Context"]
C["Document"] --> D["Content Metadata Extraction"]
B --> E["Authorization Layer"]
D --> F["Search Metadata"]
E --> G["Allowed Candidate Set"]
F --> G
G --> H["Retrieval"]
H --> I["Ranking"]This separates security metadata from descriptive metadata.
70. Metadata and Citations¶
Metadata should survive retrieval.
Example:
{
"document_id": "DOC-100",
"chunk_id": "DOC-100-C04",
"title": "Payment Security Policy",
"source": "confluence",
"page": 12,
"updated_at": "2026-07-15"
}
This information can support:
71. Metadata and Enterprise Response¶
A final response may include:
Example:
Metadata helps construct trustworthy enterprise responses.
72. Metadata and Auditability¶
Enterprise systems may need to answer:
Which documents were retrieved?
Why were they selected?
Which filters were applied?
Which tenant was active?
Which model ranked them?
Metadata enables much of this traceability.
73. Retrieval Trace¶
Example:
{
"query": "current payment policy",
"tenant_id": "tenant-a",
"filters": {
"document_type": "policy",
"status": "approved"
},
"candidate_count": 74,
"reranked_count": 20,
"final_count": 8
}
This is valuable for debugging and compliance.
74. Metadata and RAG Observability¶
A production trace can capture:
Query
↓
Extracted Metadata
↓
Applied Filters
↓
Candidate Count
↓
Rejected Count
↓
Re-ranking
↓
MMR
↓
Final Context
This allows engineers to understand retrieval failures.
75. Metadata Retrieval Failure Modes¶
75.1 Missing Metadata¶
Result:
75.2 Incorrect Metadata¶
when the document actually belongs to engineering.
75.3 Inconsistent Metadata¶
representing the same region.
75.4 Overly Restrictive Filters¶
produce:
75.5 Stale Metadata¶
The document changes but:
remain outdated.
76. Metadata Drift¶
Metadata can drift over time.
Example:
but metadata remains:
This can cause retrieval errors.
Metadata should therefore be updated as part of document lifecycle management.
77. Metadata Synchronization¶
Enterprise sources may change independently.
Example:
A metadata synchronization process may be required.
78. Metadata Enrichment¶
Metadata can be enriched during ingestion.
Example:
Raw Document
↓
Document Classification
↓
Entity Extraction
↓
Topic Classification
↓
Metadata Enrichment
↓
Chunking
↓
Embedding
↓
Vector Store
This can produce richer retrieval capabilities.
79. Metadata Enrichment Example¶
Input:
Enrichment:
{
"document_type": "architecture",
"product": "payment-service",
"domain": "payments",
"team": "platform",
"technology": [
"Kafka",
"Spring Boot"
]
}
These fields can later support targeted retrieval.
80. Metadata Extraction Pipeline¶
flowchart TD
A["Raw Document"] --> B["Parser"]
B --> C["Text Extraction"]
C --> D["Metadata Extraction"]
D --> E["Metadata Validation"]
E --> F["Metadata Normalization"]
F --> G["Chunking"]
G --> H["Embedding"]
H --> I["Vector Store"]Metadata should be treated as a first-class ingestion artifact.
81. Metadata and Chunking¶
Chunking can create metadata:
Example:
This makes retrieval and citation more precise.
82. Section-Aware Retrieval¶
Suppose the query is:
Metadata may identify:
The system can use section metadata to improve retrieval.
83. Metadata and Document Hierarchy¶
A useful hierarchy:
Metadata can preserve:
This supports:
84. Metadata Filtering with Vector Stores¶
Different vector databases support different metadata capabilities.
Typical concepts include:
Examples:
The exact implementation depends on the selected vector database.
85. Chroma-Style Example¶
Conceptually:
results = collection.query(
query_embeddings=[query_embedding],
n_results=10,
where={
"department": "engineering"
}
)
The exact syntax should be verified against the deployed Chroma version.
86. Metadata Filtering with Multiple Conditions¶
Conceptually:
This expresses:
Again, filter syntax is vector-store-specific.
87. Metadata and FAISS¶
FAISS primarily provides vector similarity search.
It does not itself provide the same metadata filtering capabilities as many full vector databases.
A common architecture is:
Example:
However, care must be taken to avoid authorization leakage and excessive post-filter loss.
88. External Metadata Store¶
An enterprise architecture may separate:
from:
Example:
flowchart LR
A["Query"] --> B["Vector Index"]
B --> C["Document IDs"]
C --> D["Metadata Store"]
D --> E["Authorized Metadata"]
E --> F["Context"]This can provide flexibility but introduces consistency challenges.
89. Metadata Consistency¶
If:
is updated but:
is not, retrieval can become inconsistent.
Therefore:
should be coordinated.
90. Metadata as a First-Class Retrieval Layer¶
A mature architecture treats metadata as its own layer:
┌───────────────────────────────┐
│ Query Processing │
├───────────────────────────────┤
│ Security / Metadata │
├───────────────────────────────┤
│ Candidate Retrieval │
├───────────────────────────────┤
│ Re-ranking │
├───────────────────────────────┤
│ MMR / Diversity │
├───────────────────────────────┤
│ Context Selection │
└───────────────────────────────┘
This separation improves maintainability.
91. Capability-Based Metadata Filtering¶
A production architecture can define:
Implementations might include:
This keeps filtering responsibilities modular.
92. Filter Composition¶
Filters can be composed:
Then:
This provides a clean architecture for enterprise retrieval.
93. Trusted Security Filter¶
Security should be separate:
from:
This prevents business-level filtering logic from accidentally weakening authorization.
94. Metadata Query Contract¶
A structured internal representation can look like:
{
"semantic_query": "payment policy",
"required_filters": {
"tenant_id": "tenant-a"
},
"optional_filters": {
"document_type": "policy",
"status": "approved",
"region": "EU"
}
}
This is useful for complex retrieval pipelines.
95. Required vs Optional Filters¶
Example:
{
"required_filters": {
"tenant_id": "tenant-a"
},
"optional_filters": {
"region": "EU",
"language": "en"
}
}
If no documents match:
must never be relaxed.
But:
might be relaxed if the application supports multilingual retrieval.
96. Metadata-Aware Query Planning¶
A query planner can determine:
Which filters are hard?
Which are optional?
Which retriever should run?
Which ranking strategy should be used?
Example:
Query
↓
Query Planner
├── Security Filters
├── Metadata Filters
├── Retriever
├── Re-ranker
└── Diversity Strategy
This begins to resemble an enterprise retrieval execution engine.
97. Metadata and Agentic Retrieval¶
An agent can decide:
Example:
Agent reasoning may identify:
Then execute retrieval.
However, the agent should not control trusted security constraints.
98. Metadata and Graph RAG¶
Metadata can connect documents to entities:
Graph RAG can use these relationships.
Metadata-aware retrieval can therefore act as a bridge between:
and:
99. Metadata and SQL RAG¶
SQL RAG can use metadata to select:
Example:
Metadata may route the query toward:
This is especially useful in enterprise environments with many databases.
100. Metadata and Multimodal RAG¶
Multimodal documents can have:
Example:
Retrieval can then target appropriate representations.
101. Metadata and Multi-Modal Routing¶
flowchart TD
A["Query"] --> B["Query Understanding"]
B --> C{"Required Modality"}
C -->|Text| D["Text Retriever"]
C -->|Table| E["Table Retriever"]
C -->|Image| F["Vision Retriever"]
C -->|Mixed| G["Multimodal Retriever"]
D --> H["Candidate Pool"]
E --> H
F --> H
G --> H
H --> I["Ranking"]Metadata helps route retrieval to the appropriate representation.
102. Metadata and Cost Optimization¶
Metadata can reduce unnecessary retrieval.
Example:
Instead of searching:
search:
This can reduce:
103. Metadata and Latency¶
A smaller search space can improve latency:
However, metadata filtering itself has an implementation cost.
Benchmark the complete pipeline.
104. Metadata and Retrieval Precision¶
A useful conceptual relationship is:
But overly restrictive filters can reduce recall.
Therefore:
can occur if metadata filters are too aggressive.
105. Metadata Filter Trade-Off¶
No Filters
↓
High Recall
Lower Precision
Balanced Filters
↓
Good Recall
Good Precision
Too Many Filters
↓
Low Recall
Potentially High Precision
The goal is a balanced retrieval strategy.
106. Metadata-Aware Evaluation¶
Evaluate:
against:
Measure:
Also measure:
for enterprise systems.
107. Filter Recall Testing¶
Create test cases:
Example:
{
"query": "current payment policy",
"filters": {
"document_type": "policy",
"status": "approved"
},
"expected_documents": [
"POLICY-2026"
]
}
This allows automated regression testing.
108. Security Retrieval Testing¶
Test explicitly:
must never return:
Test cases should include:
Security retrieval tests should be automated.
109. Metadata Test Matrix¶
| Scenario | Expected |
|---|---|
| Valid tenant | Tenant documents only |
| Invalid tenant | No documents |
| Approved policy | Approved policies |
| Expired policy | Excluded when current requested |
| EU query | EU documents |
| Missing metadata | Controlled behavior |
| Unknown filter | Rejected |
| Unauthorized document | Never returned |
110. Metadata Observability Dashboard¶
A production dashboard might show:
Metadata Filter Usage
──────────────────────────────
Tenant Filters 98%
Document Type Filters 64%
Date Filters 41%
Region Filters 23%
No-Result Rate 4%
Metadata Errors 0.3%
Average Candidate Count 87
P95 Retrieval Latency 180 ms
These metrics can reveal retrieval issues.
111. Common Anti-Patterns¶
Anti-Pattern 1 — Metadata as Free-Text¶
This makes filtering difficult.
Anti-Pattern 2 — Uncontrolled Metadata Values¶
Anti-Pattern 3 — Security as Ranking¶
Incorrect.
Unauthorized documents should be excluded.
Anti-Pattern 4 — Post-Filtering Everything¶
This can destroy recall.
Anti-Pattern 5 — Blind LLM Filter Execution¶
Never allow LLM output to override trusted security context.
112. Common Anti-Patterns — Continued¶
Anti-Pattern 6 — Excessive Filters¶
may result in:
Anti-Pattern 7 — Stale Metadata¶
Documents change but metadata does not.
Anti-Pattern 8 — No Metadata Validation¶
Invalid metadata enters the index.
Anti-Pattern 9 — Losing Metadata During Chunking¶
Chunks become impossible to trace to their source.
Anti-Pattern 10 — No Metadata Observability¶
Teams cannot understand why retrieval failed.
113. Recommended Enterprise Metadata Model¶
A practical starting structure:
Identity
├── document_id
├── parent_id
└── chunk_id
Security
├── tenant_id
├── classification
└── access_policy
Source
├── source_system
├── source_id
└── source_url
Content
├── document_type
├── language
├── topic
└── modality
Organization
├── department
├── team
└── business_unit
Business
├── product
├── service
└── region
Lifecycle
├── version
├── status
├── created_at
├── updated_at
├── effective_from
└── effective_until
114. Production Retrieval Flow¶
flowchart TD
A["User"] --> B["Query API"]
B --> C["Identity Context"]
C --> D["Query Planner"]
D --> E["Semantic Query"]
D --> F["Metadata Constraints"]
C --> G["Trusted Security Filters"]
F --> H["Filter Planner"]
G --> H
H --> I["Vector / Hybrid Retrieval"]
E --> I
I --> J["Candidate Pool"]
J --> K["Re-ranking"]
K --> L["MMR / Diversity"]
L --> M["Context Selection"]
M --> N["Prompt Assembly"]
N --> O["LLM"]
O --> P["Response Validation"]
P --> Q["Citation"]
Q --> R["Enterprise Response"]115. Production Metadata Checklist¶
☐ Define metadata taxonomy
☐ Define required fields
☐ Normalize values
☐ Validate metadata during ingestion
☐ Preserve metadata during chunking
☐ Enforce tenant isolation
☐ Separate security from ranking
☐ Support date filtering
☐ Support document-type filtering
☐ Support business filtering
☐ Track metadata versions
☐ Handle missing metadata
☐ Monitor metadata quality
☐ Test filter behavior
☐ Test authorization behavior
☐ Preserve provenance
☐ Monitor no-result queries
☐ Measure filter impact on recall
☐ Implement safe filter relaxation
116. Practical Design Example¶
Consider an enterprise payment knowledge base.
Metadata:
{
"tenant_id": "bank-a",
"department": "payments",
"team": "platform",
"product": "payment-gateway",
"document_type": "architecture",
"region": "EU",
"status": "approved",
"version": "5.1",
"updated_at": "2026-07-20"
}
User asks:
Query processing may produce:
Semantic Query:
payment gateway authentication
Filters:
tenant_id = bank-a
product = payment-gateway
region = EU
status = approved
Then:
117. Practical Python Metadata Model¶
A typed model helps maintain consistency.
from dataclasses import dataclass
from datetime import datetime
@dataclass
class DocumentMetadata:
document_id: str
tenant_id: str
document_type: str
status: str
department: str | None = None
team: str | None = None
product: str | None = None
region: str | None = None
language: str | None = None
created_at: datetime | None = None
updated_at: datetime | None = None
This provides a clear metadata contract.
118. Metadata Filter Object¶
A structured filter object can separate query intent from database syntax.
from dataclasses import dataclass
@dataclass
class RetrievalFilter:
tenant_id: str | None = None
document_type: str | None = None
department: str | None = None
status: str | None = None
region: str | None = None
The vector-store adapter can translate this into its database-specific filter language.
119. Adapter Architecture¶
This prevents application code from becoming tightly coupled to:
120. Capability-Based Retrieval Architecture¶
class MetadataAwareRetriever:
def retrieve(
self,
query: str,
filters: RetrievalFilter,
top_k: int
):
raise NotImplementedError
Implementations can include:
The application depends on the capability rather than the database.
121. Enterprise Retrieval Pipeline¶
┌────────────────────┐
│ User Query │
└─────────┬──────────┘
↓
┌────────────────────┐
│ Query Understanding│
└─────────┬──────────┘
↓
┌────────────┴────────────┐
↓ ↓
┌────────────────┐ ┌─────────────────┐
│ Semantic Query │ │ Metadata Filters│
└────────┬───────┘ └────────┬────────┘
│ │
└───────────┬────────────┘
↓
┌─────────────────┐
│ Security Filter │
└────────┬────────┘
↓
┌─────────────────┐
│ Candidate Search│
└────────┬────────┘
↓
┌─────────────────┐
│ Re-ranking │
└────────┬────────┘
↓
┌─────────────────┐
│ MMR │
└────────┬────────┘
↓
┌─────────────────┐
│ Context Builder │
└────────┬────────┘
↓
LLM
122. Key Takeaways¶
- Metadata-aware retrieval combines semantic retrieval with structured document information.
- Metadata can improve precision, security, routing, freshness, and context selection.
- Metadata should be structured rather than stored as uncontrolled text.
- Document-level and chunk-level metadata serve different purposes.
- Metadata should be inherited appropriately during chunking.
- Tenant and authorization metadata are security-critical.
- Authorization must never be implemented as a ranking preference.
- Security filters should be applied before documents enter the retrieval pipeline.
- Pre-filtering generally provides stronger isolation and better candidate quality when supported.
- Post-filtering can reduce recall if too many retrieved candidates are discarded.
- Metadata can represent document type, department, team, product, region, language, version, status, and lifecycle.
- Metadata can support current, historical, and time-bounded retrieval.
- Natural-language queries can be transformed into semantic queries plus structured metadata filters.
- Self-query retrieval is a natural extension of metadata-aware retrieval.
- LLM-generated filters must be validated before execution.
- Trusted security context must come from the application rather than the LLM.
- Metadata normalization is essential for consistent filtering.
- Metadata schemas should evolve deliberately and be versioned.
- Metadata quality should be monitored like any other production data quality dimension.
- Metadata can be used for routing to specialized retrievers.
- Metadata can reduce retrieval, re-ranking, and generation costs.
- Metadata can support source attribution, citation, and auditability.
- Metadata filtering can increase precision but may reduce recall if overly restrictive.
- Required and optional filters should be explicitly distinguished.
- Optional filters may sometimes be safely relaxed when no results are found.
- Security filters must never be automatically relaxed.
- Metadata works particularly well with hybrid retrieval, re-ranking, MMR, Graph RAG, SQL RAG, and multimodal retrieval.
- Metadata should remain available throughout the complete RAG pipeline.
- A production metadata layer should be observable, testable, versioned, and security-aware.
The central pattern is:
Semantic Understanding
+
Trusted Metadata
+
Security Context
↓
Scoped Candidate Retrieval
↓
Precise Ranking
↓
Diversity-Aware Selection
↓
Grounded Context
↓
Enterprise Response
Or:
Semantic Search tells you:
"What is relevant?"
Metadata tells you:
"Which relevant information applies here?"
🧭 Chapter Navigation¶
Part V — Advanced Retrieval-Augmented Generation¶
Previous:
11. MMR and Diversity-Aware Retrieval
Next:
13. Advanced Query Rewriting
Section:
02 — Enterprise Retrieval Engineering
Enterprise Retrieval Engineering Path¶
01 Contextual Compression Retriever
↓
02 Ensemble Retriever
↓
03 Multi-Vector Retriever
↓
04 Time-Weighted Retriever
↓
05 Hybrid Search Retriever
↓
06 HyDE Retriever
↓
07 Router Retriever
↓
08 Multi-Stage Retrieval
↓
09 Agentic Retrieval
↓
10 Re-ranking Techniques
↓
11 MMR & Diversity-Aware Retrieval
↓
12 Metadata-Aware Retrieval
↓
13 Advanced Query Rewriting
Enterprise AI Engineering Handbook
Building Production-Grade Enterprise AI Systems — One Chapter at a Time.