Contextual Compression Retriever¶
📖 Overview¶
A Contextual Compression Retriever improves RAG retrieval by reducing irrelevant information from documents returned by a retriever.
A traditional retriever may successfully identify relevant documents, but those documents can still contain a large amount of information that is unrelated to the user's question.
For example:
The retriever may return:
Production API Security Guide
├── Authentication
├── Authorization
├── Network Security
├── Rate Limiting
├── Logging
├── Monitoring
├── Deployment
└── Incident Response
The document is relevant, but the LLM may only need:
Contextual compression introduces an additional processing stage:
The goal is not simply to retrieve fewer documents.
The goal is to provide the LLM with more relevant context with less unnecessary information.
🎯 Learning Objectives¶
After completing this chapter, you will be able to:
- Understand contextual compression in RAG
- Understand why retrieved documents may contain irrelevant information
- Differentiate retrieval from context compression
- Understand the role of a base retriever
- Understand document compressors
- Implement a contextual compression pipeline
- Understand embedding-based compression
- Understand LLM-based compression
- Understand reranking-based compression
- Combine compression with advanced retrievers
- Evaluate compression quality
- Identify common compression failure modes
- Design production-ready contextual compression architectures
1. The Context Problem in RAG¶
A basic RAG pipeline typically looks like:
Suppose the retriever returns:
Document A → 1,800 tokens
Document B → 1,500 tokens
Document C → 1,700 tokens
Document D → 1,200 tokens
Total:
But only:
may actually be useful for answering the question.
The LLM still receives:
unless the application introduces another mechanism to reduce the context.
This can lead to:
- Higher token usage
- Higher cost
- Higher latency
- More irrelevant context
- More context competition
- Lower answer quality
2. What Is Contextual Compression?¶
Contextual compression takes retrieved documents and keeps only the information relevant to the current query.
The basic architecture is:
User Query
↓
Base Retriever
↓
Candidate Documents
↓
Contextual Compressor
↓
Compressed Context
↓
Prompt Assembly
↓
LLM
↓
Response
The important distinction is:
Retriever
↓
Find potentially relevant information
Compressor
↓
Remove information that is not useful for this query
3. Retrieval vs Compression¶
Retrieval asks:
"Which documents might contain the answer?"
Compression asks:
"Which parts of those documents are useful for answering this specific query?"
Therefore:
This creates a two-stage retrieval architecture:
4. Why Do We Need Contextual Compression?¶
Consider an enterprise policy document:
Remote Work Policy
1. Introduction
2. Eligibility
3. Working Arrangements
4. Approval Process
5. Exceptions
6. Security Requirements
7. Manager Responsibilities
8. Compliance
9. Reporting
User query:
A retriever may return the entire policy.
But the answer may primarily exist in:
Contextual compression can reduce:
to:
This allows the generation layer to work with a smaller and more focused context.
5. Core Architecture¶
flowchart LR
A["User Query"] --> B["Base Retriever"]
B --> C["Candidate Documents"]
C --> D["Contextual Compressor"]
D --> E["Relevant Passages"]
E --> F["Context Selector"]
F --> G["Prompt Assembly"]
G --> H["LLM"]
H --> I["Response"]The architecture separates:
from:
This separation is important in production RAG systems because retrieval and context optimization have different responsibilities.
6. Contextual Compression Components¶
A typical implementation contains two primary components:
Base Retriever¶
Responsible for:
Base Compressor¶
Responsible for:
This allows compression to be added without completely redesigning the existing retrieval layer.
7. Basic LangChain Example¶
LangChain provides a ContextualCompressionRetriever abstraction.
A simplified example:
from langchain.retrievers import ContextualCompressionRetriever
compression_retriever = ContextualCompressionRetriever(
base_compressor=compressor,
base_retriever=retriever
)
documents = compression_retriever.invoke(
"What authentication mechanism is required?"
)
for document in documents:
print(document.page_content)
The important architecture is:
The application can continue using the retriever interface while the compression stage remains encapsulated.
8. Base Retriever¶
For example:
The pipeline becomes:
The advantage is that retrieval can remain relatively broad.
The compressor then controls what is actually passed forward.
9. Why Retrieve More Before Compressing?¶
Suppose:
The correct evidence may not appear in those three documents.
Increasing retrieval to:
can improve recall.
However, sending all 10 documents to the LLM may increase context size significantly.
Contextual compression provides a middle ground:
This gives the system an opportunity to optimize:
10. Extractive Compression¶
The simplest form of compression is extractive compression.
The system keeps relevant sentences or passages from the original document.
Example:
Original Document
The platform supports OAuth 2.0 authentication.
Applications must register with the identity service.
API requests require valid access tokens.
Logs are retained for 90 days.
Monitoring is enabled across production environments.
Query:
Compressed context:
No new information is generated.
The compressor simply selects relevant evidence.
11. Why Extractive Compression Is Valuable¶
Extractive compression has an important advantage:
rather than:
This reduces the risk of introducing new information.
For enterprise systems, this can be especially useful for:
where preserving the exact source meaning is important.
12. Embedding-Based Compression¶
Another approach is to calculate semantic similarity between the query and smaller pieces of retrieved documents.
The pipeline becomes:
flowchart TD
A["Retrieved Document"] --> B["Passage Splitter"]
B --> C["Individual Passages"]
C --> D["Passage Embeddings"]
E["Query Embedding"] --> F["Similarity Calculation"]
D --> F
F --> G["Similarity Scores"]
G --> H["Relevant Passages"]Conceptually:
for passage in passages:
score = similarity(
query_embedding,
passage_embedding
)
if score >= threshold:
selected.append(passage)
13. Embedding Compression Example¶
Suppose the query is:
Retrieved document contains:
Similarity scores:
If:
then:
is retained.
The compression flow becomes:
14. Similarity Thresholds¶
Embedding-based compression requires threshold tuning.
For example:
may remove too much information.
While:
may keep too much information.
There is no universal threshold.
It depends on:
- Embedding model
- Similarity metric
- Passage size
- Document type
- Query distribution
- Evaluation dataset
Therefore, threshold values should be established through evaluation rather than copied blindly between systems.
15. LLM-Based Compression¶
An LLM can also extract query-relevant information.
Conceptually:
A simple prompt might be:
compression_prompt = """
Extract only the information relevant to the question.
Question:
{query}
Document:
{document}
Rules:
- Do not introduce new information.
- Do not infer missing facts.
- Preserve important conditions and exceptions.
- Preserve numbers and dates.
"""
The compressor is not supposed to answer the question.
It is supposed to extract evidence.
16. Compression LLM vs Answer LLM¶
It is important to separate these responsibilities.
Compression LLM¶
Answer LLM¶
Architecture:
flowchart LR
A["Retrieved Documents"] --> B["Compression LLM"]
B --> C["Relevant Evidence"]
C --> D["Prompt Assembly"]
D --> E["Answer LLM"]
E --> F["Final Response"]This separation can make the pipeline easier to reason about and evaluate.
17. Reranking-Based Compression¶
A reranker can score passages based on their relevance to the query.
For example:
Pipeline:
flowchart LR
A["Query"] --> B["Retriever"]
B --> C["Candidate Documents"]
C --> D["Passage Extraction"]
D --> E["Reranker"]
E --> F["Top Relevant Passages"]
F --> G["LLM"]Reranking is often used before compression when the candidate set is large.
18. Reranking vs Compression¶
These concepts are related but different.
Reranking¶
Changes the ordering:
Compression¶
Reduces the content:
Therefore:
They can be used together.
19. Combined Reranking and Compression¶
A production pipeline could be:
Query
↓
Retriever
↓
Candidate Documents
↓
Reranker
↓
Top Candidates
↓
Contextual Compressor
↓
Relevant Evidence
↓
LLM
Architecture:
flowchart TD
A["User Query"] --> B["Retriever"]
B --> C["Candidate Documents"]
C --> D["Reranker"]
D --> E["Top Candidates"]
E --> F["Contextual Compressor"]
F --> G["Relevant Evidence"]
G --> H["Prompt Assembly"]
H --> I["LLM"]This combination becomes increasingly useful as retrieval systems scale.
20. Contextual Compression with Multi-Query¶
Contextual compression can also be combined with Multi-Query Retrieval.
User Query
↓
Multi-Query Generation
↓
Query A / B / C
↓
Multiple Retrieval Paths
↓
Candidate Pool
↓
Compression
↓
Final Context
Architecture:
flowchart TD
A["User Query"] --> B["Multi-Query Generator"]
B --> C["Query A"]
B --> D["Query B"]
B --> E["Query C"]
C --> F["Retriever"]
D --> F
E --> F
F --> G["Candidate Pool"]
G --> H["Deduplication"]
H --> I["Contextual Compression"]
I --> J["Context"]
J --> K["LLM"]This is useful when retrieval recall is important but the resulting candidate context is too large.
21. Contextual Compression with Self-Query¶
Self-Query Retrieval can be used before compression.
For example:
Self-Query may produce:
Then:
Architecture:
flowchart LR
A["Natural Language Query"] --> B["Self-Query"]
B --> C["Semantic Query"]
B --> D["Metadata Filters"]
C --> E["Vector Search"]
D --> E
E --> F["Retrieved Documents"]
F --> G["Compression"]
G --> H["LLM"]This can reduce the amount of irrelevant content entering the compression stage.
22. Contextual Compression with Parent-Document Retrieval¶
Parent-Document Retrieval can also be combined with compression.
Query
↓
Child Retrieval
↓
Parent Resolution
↓
Parent Documents
↓
Compression
↓
Relevant Parent Sections
↓
LLM
Architecture:
flowchart TD
A["Query"] --> B["Child Retriever"]
B --> C["Child Matches"]
C --> D["Parent IDs"]
D --> E["Parent Store"]
E --> F["Parent Documents"]
F --> G["Contextual Compressor"]
G --> H["Relevant Sections"]
H --> I["LLM"]This can provide:
23. Compression Ordering¶
There are multiple possible pipeline designs.
Option A¶
Option B¶
Option C¶
There is no universal ordering.
The correct architecture depends on:
- Candidate Count
- Compression Cost
- Reranker Cost
- Latency Requirements
- Context Size
For example, if a retriever returns thousands of candidates, an inexpensive filtering stage may be necessary before applying an expensive LLM compressor.
24. Contextual Compression and Chunking¶
Chunking and compression happen at different stages.
Chunking¶
Usually occurs during ingestion:
Compression¶
Occurs during retrieval:
Therefore:
25. Contextual Compression vs Summarization¶
These are also different.
Summarization¶
The objective is to represent the overall document.
Contextual Compression¶
The objective is to preserve information relevant to the current query.
Therefore:
Contextual compression is query-aware, while ordinary summarization does not necessarily depend on the user's question.
26. Compression and Information Loss¶
Compression introduces an important risk:
Consider:
An aggressive compressor might return:
The condition:
has been removed.
The meaning has changed.
Therefore, compression must preserve:
- Conditions
- Exceptions
- Restrictions
- Numbers
- Dates
- Thresholds
- Qualifications
- Definitions
27. Compression and Faithfulness¶
The compressor should not invent information.
Bad:
Source:
Employees may work remotely up to three days.
Compressed:
Employees are entitled to three remote-work days.
The word:
introduces a stronger interpretation.
Better:
The compression stage should therefore behave as an evidence transformation, not an answer-generation stage.
28. High-Stakes Enterprise Applications¶
Contextual compression requires additional care for:
A safer architecture may prefer:
rather than unrestricted generative summarization.
The objective should be:
29. Preserving Citations¶
Compression must preserve source metadata.
For example:
{
"content": "Employees may work remotely up to three days per week.",
"source": "remote-work-policy.pdf",
"page": 14,
"section": "Working Arrangements"
}
The compressed context can retain:
Source:
remote-work-policy.pdf
Page:
14
Section:
Working Arrangements
Content:
Employees may work remotely up to three days per week.
This allows downstream components to generate reliable citations.
30. Compression with Citation Metadata¶
Architecture:
flowchart TD
A["Retrieved Document"] --> B["Compressor"]
B --> C["Relevant Evidence"]
A --> D["Source Metadata"]
C --> E["Evidence + Metadata"]
D --> E
E --> F["Prompt Assembly"]
F --> G["LLM"]
G --> H["Citation Validation"]
H --> I["Enterprise Response"]The key rule is:
Compression should never break source traceability.
31. Structured Compression Output¶
For production systems, structured output can be useful.
Example:
{
"relevant": true,
"content": "Employees may work remotely up to three days per week, subject to manager approval.",
"source": "remote-work-policy.pdf",
"page": 14,
"section": "Working Arrangements"
}
This gives downstream systems access to:
rather than only an unstructured string.
32. Framework-Agnostic Compressor Interface¶
An enterprise AI platform can define a generic compressor interface.
from abc import ABC, abstractmethod
class DocumentCompressor(ABC):
@abstractmethod
def compress(
self,
query: str,
documents: list
) -> list:
pass
Possible implementations:
class EmbeddingCompressor(DocumentCompressor):
...
class RerankingCompressor(DocumentCompressor):
...
class LLMCompressor(DocumentCompressor):
...
class ExtractiveCompressor(DocumentCompressor):
...
This keeps the application independent from a particular AI framework.
33. Composable Retrieval Architecture¶
The retrieval architecture can now be expressed as capabilities:
Architecture:
flowchart LR
A["Query"] --> B["Retriever"]
B --> C["Reranker"]
C --> D["Compressor"]
D --> E["Context Selector"]
E --> F["Prompt Builder"]
F --> G["LLM"]Each component can be independently replaced.
This is a useful design for enterprise AI platforms.
34. Context Compression Metrics¶
Compression should not be evaluated only by how many tokens it removes.
Important metrics include:
Compression Ratio¶
Conceptually:
For example:
But:
If important evidence is removed, answer quality can decrease.
35. Quality vs Compression¶
The relationship can be visualized conceptually:
Answer Quality
↑
│
│ ●
│ ●
│ ●
│ ●
│ ●
│ ●
│ ●
└────────────────────────→
Compression Aggressiveness
Initially:
Eventually:
Therefore, the goal is:
Maximum useful context reduction without losing answer-critical evidence.
36. Common Failure Modes¶
36.1 Over-Compression¶
36.2 Under-Compression¶
36.3 Qualification Loss¶
Original:
Compressed:
The qualification was lost.
36.4 Citation Loss¶
The compressor removes:
making citation generation difficult.
36.5 Hallucinated Compression¶
The LLM compressor adds facts that were not present in the source.
This is especially dangerous for high-stakes applications.
37. Guardrails for LLM Compression¶
A production compression prompt should contain explicit constraints.
Example:
You are a retrieval context compressor.
Given a user query and retrieved document:
1. Extract only information relevant to the query.
2. Do not introduce new facts.
3. Do not infer missing information.
4. Preserve numbers, dates, conditions,
exceptions, and qualifications.
5. Preserve source metadata.
6. Do not answer the user's question.
7. If no relevant information exists,
return no evidence.
The compressor should therefore be treated as:
rather than:
38. Production RAG Architecture¶
A mature enterprise architecture may look like:
flowchart TD
A["Client"] --> B["RAG API"]
B --> C["Authentication"]
C --> D["Authorization"]
D --> E["Query Processing"]
E --> F["Retriever"]
F --> G["Candidate Documents"]
G --> H["Deduplication"]
H --> I["Reranker"]
I --> J["Contextual Compressor"]
J --> K["Context Selector"]
K --> L["Prompt Assembly"]
L --> M["LLM"]
M --> N["Response Validation"]
N --> O["Citation"]
O --> P["Enterprise Response"]
F --> Q["Observability"]
I --> Q
J --> Q
M --> QObservability should span the complete pipeline:
39. Decision Flow¶
flowchart TD
A["Retrieved Documents"] --> B{"Too Much Irrelevant Context?"}
B -->|No| C["Use Retrieved Context"]
B -->|Yes| D{"Need Better Candidate Ordering?"}
D -->|Yes| E["Add Re-ranking"]
D -->|No| F{"Need Passage-Level Filtering?"}
F -->|Yes| G["Add Contextual Compression"]
F -->|No| H["Improve Retrieval"]
E --> I{"Still Too Much Context?"}
I -->|Yes| G
I -->|No| J["Context Selection"]This highlights an important engineering principle:
Compression should solve a measured context problem rather than being added simply because it is an advanced technique.
40. Production Checklist¶
Before deploying contextual compression:
☐ Compression preserves source metadata
☐ Compression does not invent facts
☐ Conditions and exceptions are preserved
☐ Numeric values are preserved
☐ Dates are preserved
☐ Citation mapping is maintained
☐ Compression latency is measured
☐ Compression cost is measured
☐ Information loss is evaluated
☐ Empty compression results are handled
☐ High-stakes use cases have stronger validation
☐ Similarity thresholds are evaluated
☐ Retrieval baseline is available
☐ Answer quality is compared before and after compression
41. When to Use Contextual Compression¶
Contextual compression is especially useful when:
- Retrieved documents are large
- Retrieved chunks contain significant irrelevant information
- The application has strict context limits
- Token costs are important
- Retrieval recall needs to remain high
- Enterprise documents contain multiple unrelated sections
- The LLM performs poorly with noisy context
Typical use cases include:
Enterprise Knowledge Assistants
Technical Documentation
Legal Document Search
Compliance Systems
Research Assistants
Financial Knowledge Systems
Security Knowledge Bases
Enterprise Copilots
42. When It May Not Be Necessary¶
Contextual compression may provide little benefit when:
or:
or:
or:
For example, a simple FAQ:
may already provide highly focused retrieval context.
43. Recommended Enterprise Pattern¶
A strong production starting architecture is:
flowchart LR
A["Query"] --> B["Retriever"]
B --> C["Top-K Candidates"]
C --> D["Re-ranker"]
D --> E["Contextual Compressor"]
E --> F["Context Selector"]
F --> G["Prompt Assembly"]
G --> H["LLM"]
H --> I["Response Validation"]
I --> J["Enterprise Response"]The important point is that each layer has a clear responsibility:
Retriever
→ Find candidates
Re-ranker
→ Order candidates
Compressor
→ Remove irrelevant content
Context Selector
→ Control final context
LLM
→ Generate answer
44. Key Takeaways¶
- Contextual Compression reduces irrelevant information from retrieved documents.
- Retrieval and compression solve different problems.
- A retriever finds potentially relevant documents.
- A compressor extracts or filters useful content from those documents.
- Extractive compression selects existing evidence without generating new facts.
- Embedding-based compression can filter passages using semantic similarity.
- LLM-based compression provides strong semantic understanding but adds cost and latency.
- Reranking and compression are complementary.
- Compression can be combined with Multi-Query Retrieval.
- Compression can be combined with Self-Query Retrieval.
- Compression can be combined with Parent-Document Retrieval.
- Compression should preserve conditions, exceptions, numbers, dates, and qualifications.
- Source metadata must survive compression for reliable citations.
- Compression should not be confused with chunking or summarization.
- Excessive compression can remove answer-critical evidence.
- Insufficient compression may provide little benefit.
- High-stakes systems should use conservative compression strategies and stronger validation.
- Production systems should measure compression ratio, context relevance, information retention, answer quality, latency, and cost.
- The objective is not maximum compression.
- The objective is maximum useful context reduction without losing important evidence.
The central pattern is:
Retrieve Broadly
↓
Rank Candidates
↓
Compress Context
↓
Select Evidence
↓
Build Prompt
↓
Generate Answer
Or simply:
🧭 Chapter Navigation¶
Part V — Advanced Retrieval-Augmented Generation¶
Previous:
05. Retriever Comparison
Next:
02. Ensemble Retriever
Section:
02 — Enterprise Retrieval Engineering
Enterprise Retrieval Engineering Path¶
01 Contextual Compression Retriever
↓
02 Hybrid Search
↓
03 Metadata Filtering
↓
04 Parent-Child Retrieval
↓
05 Multi-Vector Retrieval
↓
06 Re-ranking
Enterprise AI Engineering Handbook
Building Production-Grade Enterprise AI Systems — One Chapter at a Time.