16 — LLM Evaluation¶
A practical, production-oriented guide to LLM Evaluation, covering evaluation fundamentals, why LLM evaluation is different from traditional ML evaluation, offline evaluation, online evaluation, benchmark datasets, golden datasets, reference-based metrics, reference-free evaluation, LLM-as-a-Judge, human evaluation, task-specific metrics, hallucination, faithfulness, groundedness, RAG evaluation, retrieval evaluation, generation evaluation, agent evaluation, tool-calling evaluation, safety evaluation, structured-output evaluation, latency, cost, observability, regression testing, evaluation pipelines, CI/CD quality gates, model comparison, prompt evaluation, fine-tuning evaluation, production monitoring, enterprise AI evaluation architecture, and practical interview questions.
1. Overview¶
LLM Evaluation is the systematic process of measuring whether a Large Language Model or AI system produces outputs that are:
- Correct
- Relevant
- Helpful
- Faithful
- Grounded
- Safe
- Consistent
- Well-formatted
- Cost-efficient
- Fast enough for the application
Traditional Machine Learning evaluation often focuses on metrics such as:
LLM evaluation is more complex because LLM outputs are often:
For example:
There may be many valid ways to answer the same question.
Therefore:
LLM evaluation is not simply about checking whether two strings are identical.
2. Why LLM Evaluation Matters¶
A model can produce fluent and convincing text while still being wrong.
Example:
The answer may sound perfectly reasonable.
But the actual policy may say:
Therefore:
Production AI systems require systematic evaluation.
3. The LLM Evaluation Problem¶
Traditional ML:
LLM:
Evaluation may need to measure:
4. LLM Evaluation Is Multi-Dimensional¶
A useful mental model is:
LLM Quality
│
┌────────────┼────────────┐
↓ ↓ ↓
Correctness Relevance Safety
│ │ │
└────────────┼────────────┘
↓
Production
Quality
A production LLM should not be evaluated using one metric alone.
5. LLM Evaluation Stack¶
A production evaluation stack can be represented as:
flowchart TD
A["Test Dataset"] --> B["Prompt / System"]
B --> C["LLM"]
C --> D["Generated Output"]
D --> E["Automated Metrics"]
D --> F["LLM-as-a-Judge"]
D --> G["Human Evaluation"]
E --> H["Evaluation Report"]
F --> H
G --> H
H --> I{"Quality Gate"}
I -->|Pass| J["Deploy"]
I -->|Fail| K["Improve"]6. What Should Be Evaluated?¶
An enterprise AI system may need to evaluate:
1. Model
2. Prompt
3. Retrieval
4. Generation
5. Tools
6. Agents
7. Safety
8. Structured Output
9. Latency
10. Cost
11. User Experience
This means:
is broader than:
7. Model Evaluation vs System Evaluation¶
Model Evaluation¶
Measures the underlying model.
System Evaluation¶
Measures the complete application.
Production systems should prioritize system-level evaluation.
8. Evaluation Pyramid¶
A practical evaluation hierarchy:
Each layer catches different classes of problems.
9. Evaluation Types¶
Major evaluation categories include:
Offline Evaluation
Online Evaluation
Human Evaluation
Automated Evaluation
Model-Based Evaluation
Component Evaluation
End-to-End Evaluation
Regression Evaluation
Safety Evaluation
Performance Evaluation
10. Offline Evaluation¶
Offline evaluation happens before production deployment.
Advantages:
- Reproducible
- Controlled
- Cost-effective
- Useful for CI/CD
- Safe for experimentation
11. Online Evaluation¶
Online evaluation happens against real or production-like traffic.
Examples:
- User feedback
- Task completion
- Error rate
- Latency
- Abandonment
- Escalation rate
- Human review
12. Offline vs Online Evaluation¶
| Offline | Online |
|---|---|
| Controlled dataset | Real traffic |
| Reproducible | Real-world variability |
| Lower risk | Production risk |
| Faster experimentation | Real user behavior |
| Good for regression | Good for actual performance |
A mature AI platform needs both.
13. Golden Dataset¶
A golden dataset is a curated set of representative evaluation examples.
Example:
{
"question": "What is the refund period?",
"context": "Refunds must be requested within 30 days.",
"expected_answer": "Refunds must be requested within 30 days."
}
A golden dataset should represent:
14. Golden Dataset Structure¶
A practical structure:
{
"id": "refund-policy-001",
"input": "What is the refund period?",
"context": "...",
"expected_output": "30 days",
"category": "policy",
"difficulty": "easy",
"risk": "high"
}
Additional metadata may include:
15. Building an Evaluation Dataset¶
A strong evaluation dataset can come from:
Production Queries
+
Expert-Created Examples
+
Synthetic Examples
+
Historical Failures
+
Edge Cases
+
Adversarial Examples
Architecture:
flowchart LR
A["Production Logs"] --> D["Evaluation Dataset"]
B["Experts"] --> D
C["Synthetic Data"] --> D
E["Failure Cases"] --> D
F["Adversarial Cases"] --> D16. Dataset Splits¶
Similar to ML workflows, maintain:
For LLM applications:
Avoid repeatedly tuning against the same final test set.
17. Evaluation Data Leakage¶
A major risk is accidentally using benchmark or test examples during:
This can make evaluation scores artificially high.
Therefore:
where practical.
18. Evaluation Categories¶
A practical evaluation taxonomy:
Correctness
Relevance
Faithfulness
Groundedness
Completeness
Coherence
Safety
Format
Consistency
Latency
Cost
19. Correctness¶
Correctness measures whether the generated answer is factually or task-wise correct.
Example:
Correctness:
For complex tasks, correctness may require:
20. Relevance¶
Relevance measures whether the answer directly addresses the user's question.
Bad:
Even if factually correct, the answer is irrelevant.
Therefore:
are separate evaluation dimensions.
21. Faithfulness¶
Faithfulness asks:
Does the answer accurately reflect the information provided in the source/context?
Example:
Faithful.
But:
is not faithful.
22. Groundedness¶
Groundedness measures whether generated claims are supported by the provided evidence.
Conceptually:
This is particularly important for:
23. Faithfulness vs Groundedness¶
These terms are sometimes used differently across evaluation frameworks.
A practical distinction:
Groundedness
→ Is the claim supported by available evidence?
Faithfulness
→ Does the answer accurately represent that evidence?
The exact terminology depends on the evaluation framework.
24. Completeness¶
Completeness measures whether the answer contains the important information required by the task.
Example:
If the answer provides only:
it may be correct but incomplete.
25. Coherence¶
Coherence evaluates whether the answer:
- Makes logical sense
- Flows naturally
- Does not contradict itself
- Maintains consistent reasoning
Example:
This is internally inconsistent.
26. Consistency¶
Consistency measures whether similar inputs produce compatible outputs.
Example:
Both should generally produce compatible answers.
27. Hallucination¶
A hallucination occurs when an LLM generates unsupported or incorrect information.
Example:
If Oracle is not supported by the evidence, the additional claim is unsupported.
28. Hallucination Evaluation¶
A useful workflow:
flowchart TD
A["Generated Answer"] --> B["Extract Claims"]
B --> C["Retrieve Evidence"]
C --> D["Claim Verification"]
D --> E{"Supported?"}
E -->|Yes| F["Grounded"]
E -->|No| G["Potential Hallucination"]29. Hallucination Rate¶
A simple conceptual metric:
For example:
This is a simplified metric and should be adapted to the application's evaluation design.
30. Reference-Based Evaluation¶
Reference-based evaluation compares generated output against a known reference.
Examples:
Common metrics include:
- BLEU
- ROUGE
- METEOR
- Exact Match
- Token-level F1
31. Exact Match¶
Exact Match (EM) checks whether prediction and reference are exactly equal.
Example:
But:
This makes exact match unsuitable for many open-ended generation tasks.
32. Token-Level F1¶
For QA-style tasks, token overlap can be measured using precision and recall.
Then:
33. BLEU¶
BLEU measures n-gram overlap between generated text and reference text.
It was originally developed for machine translation evaluation.
Conceptually:
BLEU can be useful for certain translation tasks but is not sufficient as a general-purpose LLM metric.
34. ROUGE¶
ROUGE is commonly used for summarization evaluation.
It measures overlap between:
and:
Common variants include:
35. Limitations of Lexical Metrics¶
Consider:
Generated:
The semantic meaning is similar, but lexical overlap is low.
Therefore:
This is one reason LLM evaluation often requires semantic or model-based evaluation.
36. Semantic Evaluation¶
Semantic evaluation attempts to determine whether two texts express similar meaning.
Possible approaches:
Example:
Lexical overlap may not be perfect, but semantic similarity is high.
37. Embedding Similarity¶
A simple semantic evaluation approach:
Cosine similarity can be used:
Higher similarity generally indicates closer semantic representation.
However:
Semantic similarity does not guarantee factual correctness.
38. LLM-as-a-Judge¶
An LLM can evaluate another LLM's output.
Architecture:
flowchart LR
A["Input"] --> B["Candidate LLM"]
B --> C["Generated Answer"]
C --> D["Judge LLM"]
A --> D
E["Reference / Context"] --> D
D --> F["Evaluation Score"]The judge can evaluate:
39. LLM-as-a-Judge Prompt¶
Example:
You are evaluating an answer.
Question:
{{question}}
Reference:
{{reference}}
Candidate Answer:
{{answer}}
Evaluate the candidate on:
1. Correctness
2. Relevance
3. Completeness
Return:
{
"correctness": 1-5,
"relevance": 1-5,
"completeness": 1-5,
"reason": "..."
}
Structured output makes the evaluation easier to process.
40. Advantages of LLM-as-a-Judge¶
Advantages:
- Handles open-ended answers
- Can evaluate semantic correctness
- Flexible
- Scales better than manual review
- Useful for large evaluation datasets
41. Limitations of LLM-as-a-Judge¶
Potential problems:
- Judge bias
- Position bias
- Verbosity bias
- Model-specific preferences
- Agreement with candidate model
- Difficulty evaluating specialized domains
- Evaluation instability
Therefore:
LLM-as-a-Judge should be validated rather than blindly trusted.
42. Judge Calibration¶
Before using an LLM judge in production evaluation:
If human experts consistently rate outputs differently from the judge, improve:
43. Pairwise Evaluation¶
Instead of assigning an absolute score:
ask the judge:
Example:
Pairwise evaluation is often useful for:
44. Pointwise Evaluation¶
Pointwise evaluation gives each response an independent score.
Example:
This is useful when absolute quality matters.
45. Pairwise vs Pointwise¶
| Pairwise | Pointwise |
|---|---|
| Compare two outputs | Score one output |
| Good for model comparison | Good for absolute quality |
| Relative judgment | Absolute judgment |
| Easier preference decisions | Easier dashboards |
Both can be useful.
46. Human Evaluation¶
Human evaluation remains important for:
Complex Tasks
High-Risk Applications
Subjective Quality
Domain-Specific Evaluation
Judge Calibration
Human reviewers can evaluate:
- Correctness
- Helpfulness
- Safety
- Tone
- Completeness
- Domain accuracy
47. Human Evaluation Rubric¶
Example:
Correctness
5 = Completely correct
4 = Minor issue
3 = Partially correct
2 = Major errors
1 = Incorrect
A good rubric should provide explicit criteria.
48. Human Evaluation Workflow¶
flowchart TD
A["Evaluation Dataset"] --> B["Human Reviewers"]
B --> C["Rubric-Based Scoring"]
C --> D["Aggregate Scores"]
D --> E["Inter-Rater Analysis"]
E --> F["Evaluation Report"]49. Inter-Rater Agreement¶
When multiple reviewers evaluate the same examples, agreement can be measured.
The goal is to determine:
Possible measures include:
- Cohen's Kappa
- Fleiss' Kappa
- Krippendorff's Alpha
The appropriate metric depends on the evaluation design.
50. Evaluation Rubrics¶
A strong rubric should be:
Bad:
Better:
51. RAG Evaluation¶
RAG systems should be evaluated at two levels:
Architecture:
flowchart LR
A["Query"] --> B["Retriever"]
B --> C["Retrieved Context"]
C --> D["LLM"]
D --> E["Answer"]
C --> F["Retrieval Evaluation"]
E --> G["Generation Evaluation"]52. Retrieval Evaluation¶
Important retrieval metrics include:
53. Precision@K¶
Precision@K measures how many of the top K retrieved documents are relevant.
Conceptually:
Example:
54. Recall@K¶
Recall@K measures how many of the relevant documents were successfully retrieved.
Example:
55. Mean Reciprocal Rank¶
MRR focuses on the rank position of the first relevant result.
If the first relevant result appears at:
Higher MRR is better.
56. NDCG¶
Normalized Discounted Cumulative Gain (NDCG) evaluates ranking quality while considering the relevance of results at different positions.
Conceptually:
NDCG is useful when relevance can have multiple grades:
57. Context Precision¶
Context precision evaluates whether the retrieved context contains useful information relative to the question.
Bad retrieval:
Good retrieval:
58. Context Recall¶
Context recall evaluates whether the retrieved context contains the information necessary to answer the question.
A system can retrieve:
but still miss:
Therefore:
should both be considered.
59. RAG Evaluation Matrix¶
| Layer | Metric / Dimension |
|---|---|
| Retrieval | Precision@K |
| Retrieval | Recall@K |
| Retrieval | MRR |
| Retrieval | NDCG |
| Context | Relevance |
| Context | Coverage |
| Generation | Correctness |
| Generation | Faithfulness |
| Generation | Groundedness |
| End-to-End | Answer Quality |
60. RAG Failure Taxonomy¶
RAG failures can be classified as:
This is useful for debugging.
61. Retrieval Failure¶
Example:
User asks:
What is the refund period?
Retriever returns:
Marketing documentation
Product roadmap
Technical architecture
The LLM may hallucinate because the relevant policy was never retrieved.
Therefore:
A generation problem can actually be a retrieval problem.
62. Generation Failure in RAG¶
Retrieved context is correct:
But the LLM generates:
This is a generation/faithfulness problem.
63. RAG End-to-End Evaluation¶
Evaluate:
rather than only:
This provides better root-cause analysis.
64. Prompt Evaluation¶
Prompts should also be evaluated.
Compare:
against:
using the same:
Then compare:
65. Prompt Regression Testing¶
Store important prompts in version control.
Run the same evaluation dataset against each version.
flowchart LR
A["Evaluation Dataset"] --> B["Prompt v1"]
A --> C["Prompt v2"]
A --> D["Prompt v3"]
B --> E["Metrics"]
C --> F["Metrics"]
D --> G["Metrics"]66. Prompt Evaluation Matrix¶
| Prompt | Correctness | Groundedness | Format | Latency |
|---|---|---|---|---|
| v1 | Measure | Measure | Measure | Measure |
| v2 | Measure | Measure | Measure | Measure |
| v3 | Measure | Measure | Measure | Measure |
Never choose a prompt solely because it sounds better.
67. Fine-Tuning Evaluation¶
Fine-tuning should compare:
using the same evaluation suite.
Measure:
68. Fine-Tuning Regression¶
A fine-tuned model may improve:
while degrading:
Therefore:
should both be included.
69. PEFT Evaluation¶
For LoRA / QLoRA:
should be evaluated against:
and potentially:
if available.
70. Quantization Evaluation¶
For quantized models:
Evaluate:
A quantized model should not be considered successful solely because it consumes less memory.
71. Agent Evaluation¶
Agentic systems are more complex because they involve multiple steps.
Evaluate:
72. Agent Evaluation Architecture¶
flowchart TD
A["Task"] --> B["Agent"]
B --> C["Plan"]
C --> D["Tool Call"]
D --> E["Tool Result"]
E --> B
B --> F["Final Answer"]
C --> G["Plan Evaluation"]
D --> H["Tool Evaluation"]
F --> I["Final Answer Evaluation"]73. Agent Success Rate¶
A simple metric:
Example:
But this alone does not explain:
the agent succeeded or failed.
74. Agent Step Efficiency¶
Measure:
An agent that completes a task in:
may be preferable to one requiring:
if quality is equivalent.
75. Tool-Calling Evaluation¶
Evaluate:
Example:
Check:
76. Structured Output Evaluation¶
For JSON or schema-based output:
A response can be semantically correct but technically unusable.
77. Schema Validity Metric¶
A simple metric:
Example:
78. Safety Evaluation¶
Safety evaluation should include:
Production AI systems require continuous safety testing.
79. Safety Evaluation Dataset¶
Build adversarial test cases:
Example:
Evaluate:
80. Prompt Injection Evaluation¶
For RAG systems:
Example:
The evaluation should verify that the model treats retrieved content as data rather than blindly following embedded instructions.
81. PII Evaluation¶
Test whether the system:
Example:
The evaluation should verify appropriate handling.
82. Latency Evaluation¶
LLM evaluation is not only about output quality.
Important latency metrics:
83. Latency Percentiles¶
Average latency can hide slow requests.
Example:
This means the tail experience is significantly worse than the average.
Production systems should monitor:
84. Throughput Evaluation¶
Throughput may be measured as:
or:
depending on the workload.
For LLM serving:
can provide useful performance information.
85. Cost Evaluation¶
Track:
A model that is slightly more accurate but 10× more expensive may not be the right production choice.
86. Quality-Cost Frontier¶
A useful production concept is:
The goal is often to identify the best:
rather than maximize quality without constraints.
87. Evaluation Scorecard¶
A production scorecard might contain:
| Dimension | Score |
|---|---|
| Correctness | Measure |
| Relevance | Measure |
| Groundedness | Measure |
| Faithfulness | Measure |
| Safety | Measure |
| Format Validity | Measure |
| Latency | Measure |
| Throughput | Measure |
| Cost | Measure |
88. Composite Evaluation Score¶
Organizations may create weighted scores.
Example:
Overall Score
=
0.30 × Correctness
+
0.20 × Groundedness
+
0.15 × Relevance
+
0.15 × Safety
+
0.10 × Format
+
0.10 × Efficiency
The weights should reflect business priorities.
Do not blindly use a generic weighting scheme.
89. Hard Quality Gates vs Weighted Scores¶
Some dimensions should be treated as:
rather than tradeable scores.
For example:
Even if:
is high.
90. Evaluation Quality Gates¶
Example:
quality_gates:
correctness:
minimum: 0.90
groundedness:
minimum: 0.95
schema_validity:
minimum: 0.99
safety:
minimum: 0.99
p95_latency_ms:
maximum: 2000
This can be integrated into CI/CD.
91. Evaluation in CI/CD¶
flowchart LR
A["Code / Prompt / Model Change"] --> B["Evaluation Pipeline"]
B --> C["Run Golden Dataset"]
C --> D["Calculate Metrics"]
D --> E{"Quality Gates"}
E -->|Pass| F["Deploy"]
E -->|Fail| G["Block Deployment"]92. Evaluation as a Regression Test¶
Treat important LLM behaviors as regression tests.
Example:
Another:
Test:
PII Protection
Expected:
Do not reveal sensitive information
Current:
Information revealed
FAIL
93. Evaluation Dataset Versioning¶
Evaluation datasets should be versioned.
Each release should document:
94. Evaluation Reproducibility¶
Record:
Model Version
Model Provider
Prompt Version
Generation Config
Retriever Version
Embedding Model
Dataset Version
Evaluation Framework
Judge Model
Timestamp
Hardware
Without this metadata:
95. Evaluation Experiment Tracking¶
A useful experiment record:
experiment:
id: exp-2026-001
model:
name: enterprise-llm
version: "3.2"
prompt:
version: "v7"
generation:
temperature: 0.2
top_p: 0.9
dataset:
version: "eval-v4"
metrics:
correctness: 0.93
groundedness: 0.96
safety: 0.99
96. Evaluation Pipeline Architecture¶
flowchart TD
A["Model Registry"] --> E["Evaluation Runner"]
B["Prompt Registry"] --> E
C["Dataset Registry"] --> E
D["Generation Config"] --> E
E --> F["Model Inference"]
F --> G["Output Collection"]
G --> H["Automated Metrics"]
G --> I["LLM Judge"]
G --> J["Human Review"]
H --> K["Evaluation Store"]
I --> K
J --> K
K --> L["Dashboard"]
K --> M["Quality Gate"]97. Evaluation Store¶
Store:
Input
Context
Output
Expected Output
Model
Prompt
Metrics
Judge Result
Latency
Token Usage
Timestamp
This allows:
98. Evaluation Observability¶
Evaluation systems should expose:
Example dashboard:
Correctness ████████████████ 94%
Groundedness █████████████████ 97%
Safety ██████████████████ 99%
Schema Validity ██████████████████ 99%
99. Evaluation Drift¶
Production traffic can change over time.
This can cause evaluation results to become less representative.
Monitor:
100. Data Drift vs Evaluation Drift¶
Data Drift¶
Production inputs change.
Evaluation Drift¶
The evaluation dataset no longer represents real production behavior.
Both should be monitored.
101. Continuous Evaluation¶
A mature AI system continuously evaluates:
Architecture:
flowchart TD
A["Offline Evaluation"] --> D["Continuous Evaluation"]
B["Production Samples"] --> D
C["Human Feedback"] --> D
E["Failure Cases"] --> D
D --> F["Metrics"]
F --> G["Model / Prompt Improvement"]102. Production Sampling¶
Do not necessarily evaluate every production request with an expensive judge.
Use:
Example:
The sample should be representative.
103. High-Risk Sampling¶
Increase evaluation frequency for:
Evaluation frequency should reflect business risk.
104. Human-in-the-Loop Evaluation¶
For high-risk workflows:
Human feedback can be used to:
105. User Feedback¶
Production applications can collect:
But raw thumbs-up/down is not sufficient.
Where possible, collect:
106. Feedback Taxonomy¶
Example:
Negative Feedback
├── Incorrect
├── Irrelevant
├── Missing Information
├── Hallucination
├── Poor Formatting
├── Too Verbose
├── Too Short
├── Unsafe
└── Tool Failure
This turns user feedback into actionable evaluation data.
107. Evaluation Failure Clustering¶
Group failures by:
Example:
This helps prioritize engineering work.
108. Error Budget for LLM Systems¶
Similar to SRE concepts, define acceptable failure rates.
Example:
If the system exceeds the budget:
may be paused.
109. Evaluation and SLOs¶
Production LLM SLOs may include:
Example:
Quality SLOs are increasingly important for AI systems.
110. Evaluation and A/B Testing¶
A/B testing compares:
under real traffic.
Example:
Compare:
111. A/B Testing Architecture¶
flowchart TD
A["User Traffic"] --> B["Experiment Router"]
B --> C["Variant A"]
B --> D["Variant B"]
C --> E["Metrics"]
D --> F["Metrics"]
E --> G["Experiment Analysis"]
F --> G112. Shadow Evaluation¶
A new model can receive production inputs without serving its output.
This is useful for:
113. Canary Evaluation¶
Deploy a new model to a small traffic percentage.
Monitor:
Increase gradually.
114. Model Comparison¶
When comparing models:
use the same:
Otherwise the comparison may be misleading.
115. Evaluation Confounders¶
Avoid changing multiple variables simultaneously.
Bad experiment:
You cannot determine what caused the difference.
Better:
116. Evaluation of Model Upgrades¶
Before upgrading:
to:
run:
Then compare:
117. Evaluation of Prompt Changes¶
For every prompt change:
run regression evaluation.
Do not rely on:
Few examples can hide regressions.
118. Evaluation of RAG Changes¶
If you change:
re-run end-to-end evaluation.
A retrieval improvement can sometimes cause:
because the context distribution changes.
119. Evaluation of Chunking¶
Compare:
Measure:
This is important in enterprise RAG systems.
120. Evaluation of Reranking¶
If a reranker is introduced:
evaluate:
at both:
121. Evaluation of Embedding Models¶
Embedding model changes can affect retrieval.
Compare:
using:
122. Evaluation of Agents¶
Agent evaluation should measure the complete trajectory:
Useful metrics:
Task Success Rate
Tool Selection Accuracy
Tool Argument Accuracy
Steps per Task
Failure Recovery
Latency
Cost
123. Agent Trajectory Evaluation¶
Store:
{
"task": "...",
"steps": [
{
"action": "search_customer",
"arguments": {
"id": "C123"
}
},
{
"action": "refund_customer",
"arguments": {
"amount": 100
}
}
],
"success": true
}
This allows detailed debugging.
124. Multi-Agent Evaluation¶
For multi-agent systems:
evaluate:
A failure may originate from one agent but appear at the final system level.
125. Evaluation of Memory Systems¶
For conversational AI with memory, evaluate:
Example:
User told system:
Preferred programming language = Java
Later:
System should correctly retrieve the preference
126. Evaluation of Multilingual Systems¶
For multilingual applications evaluate:
Language Detection
Translation Quality
Answer Correctness
Cultural / Locale Accuracy
Language Consistency
Do not assume that strong English performance implies strong multilingual performance.
127. Evaluation of Long-Context Models¶
Test:
Measure:
A model may support a large context window but still perform poorly when critical information is buried deep inside it.
128. Needle-in-a-Haystack Evaluation¶
A common long-context test places a small piece of relevant information inside a large context.
Large Context
│
│
│
├── Irrelevant Information
│
├── Irrelevant Information
│
├── Critical Fact
│
├── Irrelevant Information
│
└── Irrelevant Information
Then test whether the model can retrieve and use the fact correctly.
129. Evaluation of Structured Generation¶
For structured outputs, evaluate:
Example:
This gives a much clearer picture than a single "JSON success" metric.
130. Evaluation of Code Generation¶
Code-generation evaluation can use:
Example:
flowchart TD
A["Generated Code"] --> B["Compile"]
B --> C{"Compiles?"}
C -->|No| D["Fail"]
C -->|Yes| E["Unit Tests"]
E --> F{"Tests Pass?"}
F -->|No| D
F -->|Yes| G["Security / Quality Checks"]
G --> H["Evaluation Pass"]This is generally more reliable than asking another LLM whether code "looks correct."
131. Evaluation of SQL Generation¶
For SQL generation:
Evaluate:
Does SQL parse?
Does it execute?
Does it return the expected result?
Does it avoid unsafe operations?
132. Evaluation of Tool Calls¶
For tool calling:
Metrics:
133. Evaluation of Summarization¶
Measure:
A short summary is not automatically a good summary.
134. Evaluation of Question Answering¶
Evaluate:
For document QA:
should ideally be evaluated together.
135. Evaluation of Classification¶
LLM classification can still use traditional metrics:
Example:
can be evaluated exactly like a conventional classification system.
This is an important principle:
Use traditional ML metrics whenever the LLM task has a well-defined label space.
136. Evaluation of Extraction¶
For information extraction:
Example:
Each field can be evaluated independently.
137. Field-Level Evaluation¶
Example:
| Field | Precision | Recall |
|---|---|---|
| Name | 0.99 | 0.98 |
| Amount | 0.97 | 0.96 |
| Currency | 0.99 | 0.99 |
This is more actionable than a single extraction score.
138. Evaluation of Classification + Generation¶
Many enterprise systems combine:
Example:
Evaluate each component separately and the complete workflow.
139. Component vs End-to-End Evaluation¶
Both are necessary.
140. Evaluation Dependency Graph¶
flowchart TD
A["Embedding Quality"] --> B["Retrieval Quality"]
B --> C["Context Quality"]
C --> D["Generation Quality"]
D --> E["End-to-End Quality"]
F["Prompt Quality"] --> D
G["Model Quality"] --> D
H["Tool Quality"] --> EThis explains why system evaluation requires multiple layers.
141. Evaluation Root-Cause Analysis¶
When quality drops:
End-to-End Quality ↓
↓
Check Generation
↓
Check Context
↓
Check Retrieval
↓
Check Embeddings
↓
Check Prompt
↓
Check Model
Do not immediately replace the model.
142. Evaluation Experiment Registry¶
Store experiments:
This enables:
143. Evaluation Report¶
A useful report should contain:
Executive Summary
Model Information
Dataset Information
Evaluation Method
Metrics
Results
Failure Analysis
Regression Analysis
Cost
Latency
Recommendation
144. Example Evaluation Report¶
Model: Enterprise-LLM v3.2
Dataset:
Enterprise-Eval-v4
Correctness:
93.4%
Groundedness:
96.1%
Schema Validity:
99.2%
Safety:
99.6%
p95 Latency:
1.8 sec
Cost / 1K Requests:
$X
Recommendation:
PASS
145. Evaluation Thresholds¶
Thresholds should be based on:
For example:
146. Risk-Based Evaluation¶
Not every AI task needs the same evaluation rigor.
Low Risk
→ Basic Automated Evaluation
Medium Risk
→ Automated + LLM Judge
High Risk
→ Automated + Expert Human Review
This is more practical than applying maximum evaluation everywhere.
147. Evaluation Cost Optimization¶
LLM evaluation itself can be expensive.
Use a layered strategy:
Only difficult cases should reach the most expensive evaluation layers where possible.
148. Evaluation Cascade¶
flowchart TD
A["All Samples"] --> B["Cheap Automated Checks"]
B --> C["Semantic Metrics"]
C --> D["LLM Judge"]
D --> E["Human Review"]
B -->|Clear Pass| F["Pass"]
B -->|Clear Fail| G["Fail"]
D -->|Uncertain / High Risk| E149. Evaluation Confidence¶
Some evaluations are uncertain.
For example:
Such examples can be routed to:
This creates a more efficient human-in-the-loop evaluation pipeline.
150. LLM Judge Bias¶
Potential judge biases include:
Mitigation:
151. Multiple Judges¶
For important evaluations:
can be aggregated.
Example:
This can reduce dependence on a single evaluator.
152. Judge Model Selection¶
A judge should ideally be:
A small judge may be sufficient for simple checks.
A stronger judge may be required for:
153. Evaluation of Reasoning¶
Reasoning evaluation should focus on:
Do not assume that a long explanation indicates better reasoning.
For many production tasks:
is more important than:
154. Evaluation of Factuality¶
Factuality can be evaluated against:
For high-risk applications, external verification may be required.
155. Evaluation of Citation Quality¶
For RAG systems, evaluate:
Example:
A citation that exists but does not support the claim should be considered a failure.
156. Citation Evaluation¶
flowchart TD
A["Generated Claim"] --> B["Generated Citation"]
B --> C["Source Document"]
C --> D["Evidence Matching"]
D --> E{"Supports Claim?"}
E -->|Yes| F["Valid Citation"]
E -->|No| G["Invalid Citation"]157. Evaluation of Enterprise RAG¶
A strong enterprise RAG evaluation suite should include:
Retrieval Recall
Context Precision
Context Relevance
Answer Correctness
Faithfulness
Groundedness
Citation Accuracy
Citation Completeness
Latency
Cost
158. Evaluation of Long Documents¶
For enterprise documents:
evaluate:
This is particularly important when documents contain tables, headings, footnotes, and structured sections.
159. Multimodal Evaluation¶
For multimodal systems:
evaluation should include:
Example:
Each stage can fail independently.
160. Evaluation of OCR + LLM¶
For document AI:
should be evaluated separately.
Otherwise:
may be incorrectly attributed to:
161. Evaluation of Voice AI¶
For speech systems:
Evaluate:
162. Evaluation of End-to-End AI Systems¶
The most important production question is:
Did the AI system successfully complete the user's task?
This leads to:
which may be more meaningful than individual model metrics.
163. Task Success Rate¶
Conceptually:
Example:
This can become a top-level business metric.
164. Business Metrics¶
Ultimately, evaluate:
Task Completion
Customer Satisfaction
Resolution Rate
Escalation Rate
Cost per Task
Time Saved
Revenue Impact
Technical metrics should connect to business outcomes.
165. Evaluation Hierarchy¶
A useful hierarchy is:
Example:
166. Evaluation and Observability¶
Observability answers:
Evaluation answers:
Production AI needs both.
167. Evaluation and MLOps¶
Evaluation is part of the MLOps / LLMOps lifecycle.
flowchart LR
A["Data"] --> B["Train / Fine-Tune"]
B --> C["Evaluate"]
C --> D["Register"]
D --> E["Deploy"]
E --> F["Monitor"]
F --> G["Collect Failures"]
G --> AThis creates a continuous improvement loop.
168. Evaluation and LLMOps¶
An enterprise LLMOps pipeline can include:
Model Registry
Prompt Registry
Dataset Registry
Evaluation Runner
Judge Models
Experiment Tracking
Deployment Pipeline
Observability
Feedback
169. Production Evaluation Architecture¶
flowchart TD
A["Model Registry"] --> B["Evaluation Platform"]
C["Prompt Registry"] --> B
D["Dataset Registry"] --> B
B --> E["Offline Evaluation"]
E --> F["Quality Gates"]
F -->|Pass| G["Deployment"]
G --> H["Production"]
H --> I["Observability"]
H --> J["User Feedback"]
H --> K["Production Samples"]
I --> L["Continuous Evaluation"]
J --> L
K --> L
L --> M["Improvement Loop"]
M --> B170. Enterprise AI Evaluation Architecture¶
A mature architecture can be:
┌────────────────────┐
│ Evaluation UI │
└─────────┬──────────┘
│
┌─────────▼──────────┐
│ Evaluation Service │
└─────────┬──────────┘
│
┌──────────────────┼──────────────────┐
│ │ │
▼ ▼ ▼
Dataset Store Judge Models Metrics Engine
│ │ │
└──────────────────┼──────────────────┘
▼
Evaluation Results
│
┌────────────┼────────────┐
▼ ▼ ▼
Reports Quality Gate Registry
171. Java / Spring Boot Integration¶
For a Java-first enterprise AI architecture, evaluation can be exposed as a service.
Example:
Example request:
public record EvaluationRequest(
String datasetVersion,
String modelVersion,
String promptVersion
) {}
The implementation can invoke:
172. Evaluation Capability Interfaces¶
A capability-oriented architecture may define:
public interface RetrievalEvaluator {
RetrievalMetrics evaluate(RetrievalEvaluationRequest request);
}
public interface GenerationEvaluator {
GenerationMetrics evaluate(GenerationEvaluationRequest request);
}
This keeps the architecture modular.
173. Evaluation Adapter Pattern¶
flowchart LR
A["Evaluation Service"] --> B["JudgeProvider"]
B --> C["OpenAI Judge"]
B --> D["Local Judge"]
B --> E["Cloud Judge"]
A --> F["RetrievalEvaluator"]
A --> G["GenerationEvaluator"]Provider-specific evaluation logic remains behind adapters.
174. Evaluation API¶
A production evaluation API could expose:
POST /evaluations
GET /evaluations/{id}
GET /evaluations/{id}/metrics
GET /evaluations/{id}/failures
Example request:
175. Evaluation Result Schema¶
{
"evaluationId": "eval-123",
"model": "enterprise-llm-v3",
"dataset": "enterprise-eval-v4",
"metrics": {
"correctness": 0.93,
"groundedness": 0.96,
"safety": 0.99
},
"status": "PASSED"
}
176. Evaluation Storage¶
Store evaluation artifacts in durable storage:
Possible separation:
177. Evaluation Data Governance¶
Evaluation datasets can contain:
Therefore implement:
178. Evaluation Security¶
Evaluation systems themselves can become attack surfaces.
Protect:
Never expose sensitive evaluation artifacts publicly.
179. Evaluation Cost Controls¶
Potential optimizations:
Sample Production Data
Cache Judge Results
Use Cheap Metrics First
Use Strong Judge Only for Hard Cases
Batch Evaluation
Run Full Suites on Releases
Run Small Suites on Every Commit
180. Tiered Evaluation Pipeline¶
Every Commit
↓
Fast Regression Suite
Every PR
↓
Expanded Evaluation
Every Model Release
↓
Full Evaluation
Production
↓
Continuous Sampling
This balances:
181. Evaluation Frequency¶
Suggested pattern:
Code Change
→ Fast Tests
Prompt Change
→ Prompt Regression
Model Change
→ Full Evaluation
Retriever Change
→ RAG Evaluation
Production
→ Continuous Sampling
182. Evaluation Test Pyramid¶
Cheap tests should catch simple failures first.
183. Evaluation Anti-Patterns¶
Avoid:
Avoid:
Avoid:
Avoid:
Avoid:
Avoid:
Avoid:
184. Evaluation Best Practices¶
- Build a representative golden dataset.
- Include difficult and adversarial cases.
- Separate development and final test datasets.
- Evaluate retrieval and generation separately for RAG.
- Use traditional metrics where the task allows them.
- Use semantic evaluation for open-ended tasks.
- Validate LLM judges against human reviewers.
- Use multiple evaluation dimensions.
- Establish quality gates.
- Version models, prompts, datasets, and evaluation configurations.
- Track latency and cost alongside quality.
- Include safety evaluation.
- Evaluate structured output separately.
- Evaluate tools and agents at trajectory level.
- Continuously update evaluation datasets using real failures.
- Monitor production behavior.
- Use risk-based evaluation intensity.
- Keep evaluation reproducible.
- Treat evaluation as part of CI/CD.
185. Common Evaluation Mistakes¶
Mistake 1 — Evaluating Only Fluency¶
A fluent answer can be completely wrong.
Mistake 2 — Using Only ROUGE / BLEU¶
Lexical overlap does not fully capture semantic correctness.
Mistake 3 — Trusting LLM-as-a-Judge Blindly¶
Judges have biases and failure modes.
Mistake 4 — Evaluating Only the Final Answer¶
In RAG and agents, the failure may occur upstream.
Mistake 5 — No Production Evaluation¶
Offline performance does not guarantee production performance.
Mistake 6 — No Versioning¶
Without model, prompt, dataset, and configuration versions, results become difficult to reproduce.
186. Common Failure Modes¶
Evaluation Dataset Too Easy
Evaluation Dataset Too Small
Evaluation Dataset Outdated
Metric Misaligned with Task
Judge Bias
Data Leakage
Prompt Leakage
Production Drift
Missing Edge Cases
No Safety Tests
No Regression Tests
No Cost Measurement
No Latency Measurement
187. Evaluation Debugging Workflow¶
When a model score drops:
flowchart TD
A["Score Regression"] --> B["Check Dataset"]
B --> C["Check Prompt"]
C --> D["Check Model"]
D --> E["Check Retrieval"]
E --> F["Check Generation Config"]
F --> G["Check Judge"]
G --> H["Analyze Failure Examples"]Always inspect actual failures.
A single aggregate score rarely explains the root cause.
188. Failure Example Analysis¶
Suppose:
Do not stop there.
Break failures down:
This makes remediation actionable.
189. Evaluation Dashboard¶
A production dashboard might contain:
Overall Quality
Correctness
Groundedness
Faithfulness
Safety
Schema Validity
Task Success
p50 Latency
p95 Latency
TTFT
TPOT
Token Usage
Cost
Error Rate
Trend these over time.
190. Quality Trend¶
A downward trend may indicate:
191. Evaluation and Model Lifecycle¶
flowchart LR
A["Model Development"] --> B["Evaluation"]
B --> C["Model Registry"]
C --> D["Deployment"]
D --> E["Production"]
E --> F["Monitoring"]
F --> G["Failure Analysis"]
G --> H["New Evaluation Cases"]
H --> BThis creates a continuous quality loop.
192. Evaluation and Continuous Improvement¶
Production failures should become future test cases.
This is one of the most valuable practices in production AI engineering.
193. Evaluation Maturity Model¶
Level 1 — Manual¶
Level 2 — Automated¶
Level 3 — Model-Based¶
Level 4 — Production¶
Level 5 — Continuous AI Quality¶
Evaluation
+
CI/CD
+
Observability
+
Automatic Regression Detection
+
Continuous Dataset Improvement
194. Enterprise AI Evaluation Maturity¶
Manual Testing
↓
Golden Dataset
↓
Automated Evaluation
↓
LLM-as-a-Judge
↓
CI/CD Quality Gates
↓
Production Monitoring
↓
Continuous Evaluation
The goal is not simply:
but:
195. Practical Evaluation Workflow¶
For a new enterprise LLM application:
Step 1:
Define the business task
Step 2:
Define success criteria
Step 3:
Create golden dataset
Step 4:
Create edge and failure cases
Step 5:
Define metrics
Step 6:
Establish baseline
Step 7:
Evaluate model
Step 8:
Evaluate prompt
Step 9:
Evaluate retrieval
Step 10:
Evaluate generation
Step 11:
Evaluate safety
Step 12:
Measure latency and cost
Step 13:
Set quality gates
Step 14:
Deploy to staging
Step 15:
Run shadow / canary
Step 16:
Monitor production
Step 17:
Convert failures into regression tests
196. Production Workflow¶
flowchart TD
A["Define Business Objective"] --> B["Define Evaluation Criteria"]
B --> C["Build Golden Dataset"]
C --> D["Create Baseline"]
D --> E["Evaluate Model"]
E --> F["Evaluate Prompt"]
F --> G["Evaluate Retrieval"]
G --> H["Evaluate Generation"]
H --> I["Evaluate Safety"]
I --> J["Measure Latency & Cost"]
J --> K{"Quality Gates"}
K -->|Pass| L["Staging"]
K -->|Fail| M["Improve"]
M --> D
L --> N["Canary"]
N --> O["Production"]
O --> P["Continuous Evaluation"]
P --> Q["Failure Feedback"]
Q --> C197. Production Evaluation Checklist¶
[ ] Business Objective Defined
[ ] Success Criteria Defined
[ ] Golden Dataset Created
[ ] Edge Cases Included
[ ] Failure Cases Included
[ ] Adversarial Cases Included
[ ] Dataset Versioned
[ ] Prompt Versioned
[ ] Model Versioned
[ ] Generation Config Versioned
[ ] Baseline Established
[ ] Correctness Evaluated
[ ] Relevance Evaluated
[ ] Faithfulness Evaluated
[ ] Groundedness Evaluated
[ ] Safety Evaluated
[ ] Structured Output Evaluated
[ ] Retrieval Evaluated
[ ] Tool Calling Evaluated
[ ] Agent Workflow Evaluated
[ ] Latency Measured
[ ] Throughput Measured
[ ] Cost Measured
[ ] Quality Gates Defined
[ ] CI/CD Integrated
[ ] Shadow Testing Completed
[ ] Canary Testing Completed
[ ] Production Monitoring Enabled
[ ] Failure Feedback Loop Established
198. Interview Questions¶
Beginner¶
- What is LLM evaluation?
- Why is LLM evaluation different from traditional ML evaluation?
- What is a golden dataset?
- What is offline evaluation?
- What is online evaluation?
- What is exact match?
- What is BLEU?
- What is ROUGE?
- What is hallucination?
- What is faithfulness?
- What is groundedness?
- What is relevance?
- What is correctness?
- What is LLM-as-a-Judge?
Intermediate¶
- Why are BLEU and ROUGE insufficient for general LLM evaluation?
- What is semantic similarity?
- How does embedding-based evaluation work?
- What is pairwise evaluation?
- What is pointwise evaluation?
- How would you validate an LLM judge?
- What is retrieval evaluation?
- Precision@K vs Recall@K?
- What is MRR?
- What is NDCG?
- How do you evaluate RAG?
- How do you evaluate structured output?
- How do you evaluate tool calling?
- How do you evaluate an agent?
- How do you evaluate a fine-tuned model?
- How do you evaluate a quantized model?
- How do you evaluate prompt changes?
Advanced¶
- How would you design an enterprise LLM evaluation platform?
- How would you build a golden dataset from production traffic?
- How would you prevent evaluation leakage?
- How would you design LLM-as-a-Judge infrastructure?
- How would you calibrate an LLM judge against human reviewers?
- How would you evaluate a RAG system end-to-end?
- How would you isolate retrieval failures from generation failures?
- How would you evaluate an agent trajectory?
- How would you design LLM evaluation CI/CD?
- How would you implement quality gates?
- How would you handle evaluation drift?
- How would you design continuous production evaluation?
- How would you optimize evaluation cost?
- How would you evaluate high-risk AI workflows?
- How would you compare two LLMs fairly?
- How would you evaluate a model upgrade?
- How would you evaluate quantization without sacrificing quality?
- How would you evaluate long-context performance?
- How would you evaluate tool-calling reliability?
- How would you design an evaluation architecture using Spring Boot and cloud-native services?
199. Scenario-Based Interview Questions¶
Scenario 1 — Model Accuracy Increased but User Satisfaction Decreased¶
Investigate:
A model can score better on a benchmark while performing worse in real workflows.
Scenario 2 — RAG Answer Quality Dropped¶
Break down:
Identify where the regression began.
Scenario 3 — LLM Judge Says Everything Is Excellent¶
Validate:
Check:
Do not assume the judge is correct.
Scenario 4 — New Model Improves General Benchmark but Breaks Enterprise Workflows¶
Run:
The enterprise evaluation suite should be the deployment decision authority.
Scenario 5 — RAG Retrieval Recall Is High but Answers Are Poor¶
The likely problem may be:
High retrieval recall does not guarantee good final answers.
Scenario 6 — Fine-Tuned Model Improves Domain Accuracy but Hallucinates More¶
Compare:
on:
The fine-tuning may have introduced a regression.
Scenario 7 — Production Quality Slowly Degrades¶
Investigate:
Data Drift
Prompt Changes
Model Updates
Retriever Changes
Embedding Changes
User Behavior
Evaluation Dataset Drift
Then add representative failures to the evaluation suite.
Scenario 8 — Cost Increased Without Quality Improvement¶
Analyze:
Input Tokens
Output Tokens
Model Selection
Prompt Length
Retrieval Context
Retries
Agent Steps
Judge Calls
Evaluation should include cost efficiency.
200. 🚀 Quick Revision Sheet¶
Core Evaluation Dimensions¶
Evaluation Types¶
RAG¶
Evaluate:
Production¶
Key Principle¶
And:
The ultimate objective is:
201. Remember¶
LLM evaluation is the discipline of measuring whether an AI system reliably performs its intended task across correctness, relevance, groundedness, safety, reliability, performance, and business outcomes.
Remember:
For RAG:
For Agents:
For Production:
202. Key Takeaways¶
- LLM evaluation is broader than traditional model evaluation.
- A fluent answer is not necessarily a correct answer.
- LLM evaluation should measure multiple dimensions.
- Correctness measures whether the answer is correct.
- Relevance measures whether the answer addresses the task.
- Faithfulness measures whether the answer accurately reflects its evidence or context.
- Groundedness measures whether claims are supported by available evidence.
- Completeness measures whether important information is present.
- Safety measures whether the system behaves appropriately under normal and adversarial inputs.
- A golden dataset is one of the most important assets for LLM evaluation.
- Evaluation datasets should contain common cases, edge cases, failure cases, and adversarial cases.
- Development and final test datasets should be separated where practical.
- Offline evaluation provides controlled and reproducible testing.
- Online evaluation measures real-world system behavior.
- Traditional metrics remain useful for classification and extraction tasks.
- Exact Match is useful for tasks with strict expected outputs but is often unsuitable for open-ended generation.
- BLEU and ROUGE measure lexical overlap and should not be treated as universal LLM quality metrics.
- Embedding similarity provides semantic comparison but does not guarantee factual correctness.
- LLM-as-a-Judge provides scalable semantic evaluation but introduces judge bias and must be calibrated.
- Human evaluation remains important for complex, subjective, and high-risk tasks.
- Pairwise evaluation is useful for comparing models or prompts.
- Pointwise evaluation is useful for measuring absolute quality.
- RAG evaluation must separate retrieval quality from generation quality.
- Precision@K, Recall@K, MRR, and NDCG are important retrieval metrics.
- RAG generation should be evaluated for correctness, faithfulness, groundedness, and citation quality.
- Agent evaluation should measure task success, tool selection, argument correctness, trajectory, efficiency, and final output.
- Structured-output evaluation should include syntax, schema, semantics, completeness, and business-rule validation.
- Tool-calling evaluation should verify tool selection, arguments, schema, execution, and recovery.
- Code generation should be evaluated using compilation, tests, security analysis, and functional correctness where possible.
- SQL generation should be evaluated using syntax, execution, result correctness, and security.
- Long-context models require dedicated context and retrieval evaluation.
- Model upgrades, prompt changes, retrieval changes, embedding changes, and quantization changes should all trigger regression evaluation.
- Evaluation datasets, prompts, models, and generation configurations should be versioned.
- Evaluation results should be reproducible.
- Quality gates should block deployments when critical requirements are not satisfied.
- Not every metric should be treated as a tradeable score; safety and schema correctness may need hard thresholds.
- Evaluation should measure latency, throughput, token usage, and cost alongside quality.
- Production systems should use shadow and canary evaluation for high-risk model changes.
- User feedback and production failures should continuously improve the evaluation dataset.
- Evaluation drift occurs when the evaluation dataset stops representing production behavior.
- Risk-based evaluation is more practical than applying identical evaluation rigor to every AI workflow.
- Evaluation itself should be optimized using cascaded metrics, sampling, caching, and selective human review.
- Observability answers what happened; evaluation answers whether the result was good.
- LLM evaluation belongs inside the broader LLMOps / MLOps lifecycle.
- A mature evaluation platform connects datasets, prompts, models, judges, metrics, deployment, monitoring, and feedback.
- The ultimate objective is not merely a high benchmark score but a reliable AI system that successfully completes real business tasks.
203. Chapter Navigation¶
Previous Chapter¶
Current Chapter¶
16. LLM Evaluation
Next Chapter¶
Related Chapters¶
- 01. Generative AI Fundamentals
- 02. Language Understanding Fundamentals
- 03. Word Embeddings
- 04. Language Modeling
- 05. Attention and Positional Encoding
- 06. GPT and BERT Architecture
- 07. Hugging Face and Transformers
- 08. LLM Data Preparation
- 09. Hugging Face Training Workflow
- 10. Transformer Fine-Tuning Fundamentals
- 11. Supervised Fine-Tuning (SFT)
- 12. Parameter-Efficient Fine-Tuning (PEFT)
- 13. LoRA and QLoRA
- 14. Model Quantization
- 15. LLM Generation Strategies
- 17. Retrieval-Augmented Generation (RAG) Fundamentals
References¶
- IBM AI Engineering Professional Certificate
- Hugging Face Transformers Documentation
- Hugging Face Evaluate Documentation
- Hugging Face Datasets Documentation
- RAGAS Documentation
- LangChain Evaluation Documentation
- LangSmith Documentation
- OpenAI Evaluation / Evals Documentation
- Anthropic Model Evaluation Guidance
- Google Cloud Vertex AI Evaluation Documentation
- NIST AI Risk Management Framework
- Stanford HELM — Holistic Evaluation of Language Models
- Liang et al. — Holistic Evaluation of Language Models
- Papineni et al. — BLEU: a Method for Automatic Evaluation of Machine Translation
- Lin — ROUGE: A Package for Automatic Evaluation of Summaries
- Jurafsky & Martin — Speech and Language Processing
- Retrieval-Augmented Generation research literature
- LLM-as-a-Judge research literature
- RAG evaluation research and benchmarks
Enterprise AI Engineering Handbook
Building Production-Grade Enterprise AI Systems — One Chapter at a Time.