02. Agent Logging¶
Category: Agent Observability Module: AI Agents Prerequisites: Agent Observability Overview Difficulty: Intermediate
Note: Agent Logging is the process of recording important events generated by AI agents during execution. Logs provide a chronological history of what an agent did, why it made certain decisions, which tools it invoked, which prompts were used, and whether execution succeeded or failed. In production AI systems, logs are the primary source for debugging, auditing, compliance, and incident investigation.
Overview¶
Imagine an enterprise AI assistant receives a user request.
Everything appears to work.
However, a customer later reports:
- Wrong answer
- Slow response
- Missing information
- Tool failure
Without logs, engineers have no way to determine what happened.
Instead, every important action should be recorded.
This chronological history is called Agent Logging.
Why Agent Logging Matters¶
Without Logging
Questions remain unanswered.
- Which prompt was sent?
- Which tool failed?
- Which model was used?
- How long did execution take?
- Which agent produced the answer?
With Logging
Benefits
- Easier debugging
- Production monitoring
- Root cause analysis
- Compliance
- Auditability
- Performance optimization
What is Agent Logging?¶
Agent Logging records every significant event that occurs during execution.
User Request
↓
Agent Started
↓
Memory Retrieved
↓
LLM Called
↓
Tool Invoked
↓
Response Generated
↓
Completed
Every step generates log entries.
Unlike traditional applications, AI logs also capture AI-specific information such as prompts, tool usage, token consumption, and reasoning steps.
High-Level Architecture¶
User Request
│
▼
AI Agent
│
┌──────────────────┼──────────────────┐
▼ ▼ ▼
Memory Tool Calls LLM
│ │ │
└──────────────────┼──────────────────┘
▼
Logging Layer
│
┌──────────────────┼──────────────────┐
▼ ▼ ▼
Structured Logs Error Logs Audit Logs
│
▼
Log Aggregation Platform
│
▼
Dashboard
The Logging Layer captures events from every stage of the AI workflow.
Log Lifecycle¶
A production log follows a structured lifecycle.
Logs remain available for debugging, monitoring, and compliance.
Types of Agent Logs¶
Enterprise AI systems generate multiple categories of logs.
1. Execution Logs¶
Record workflow execution.
Typical Uses
- Workflow tracking
- Debugging
- Monitoring
2. Prompt Logs¶
Record prompts sent to the LLM.
Typical Uses
- Prompt debugging
- Prompt optimization
- Evaluation
Security Note: In production systems, sensitive user information and confidential business data should be masked or redacted before prompts are stored.
3. Tool Logs¶
Record external tool execution.
Typical Uses
- Tool debugging
- Performance analysis
- Failure investigation
4. Memory Logs¶
Track memory operations.
Typical Uses
- RAG debugging
- Memory optimization
- Context validation
5. Error Logs¶
Capture failures.
Typical Uses
- Incident response
- Retry analysis
- Root cause investigation
6. Audit Logs¶
Track important business actions.
Typical Uses
- Regulatory compliance
- Security audits
- Governance
Structured Logging¶
Production AI systems should avoid plain text logs.
Poor
Better
{
"timestamp": "2026-08-06T09:30:10Z",
"agent": "RetrieverAgent",
"event": "tool_execution",
"tool": "Vector Search",
"status": "FAILED",
"latency_ms": 820
}
Benefits
- Searchable
- Machine readable
- Dashboard friendly
- Easier analytics
JSON is the preferred format for production AI platforms.
Correlation ID vs Request ID¶
Distributed AI workflows often involve multiple agents.
Each request should include identifiers.
Request ID¶
Identifies one user request.
Example
Correlation ID¶
Tracks the same workflow across multiple services and agents.
Benefits
- End-to-end tracing
- Easier debugging
- Distributed monitoring
- Incident investigation
What Should Be Logged?¶
Enterprise AI platforms typically log the following.
Agent Logs
│
├── Timestamp
├── Request ID
├── Correlation ID
├── Agent Name
├── Workflow ID
├── Prompt Version
├── Model Name
├── Tool Calls
├── Memory Operations
├── Latency
├── Token Usage
├── Status
└── Errors
These fields provide enough information to reconstruct an execution.
Implementation¶
Example 1 – Core Python¶
Structured logging using Python's built-in logging module.
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(message)s"
)
logging.info(
"Planner Agent Started"
)
logging.info(
"Retriever Executed"
)
logging.info(
"LLM Response Generated"
)
Output
2026-08-06 10:00:10 INFO Planner Agent Started
2026-08-06 10:00:12 INFO Retriever Executed
2026-08-06 10:00:15 INFO LLM Response Generated
Example 2 – LangChain Callback Logging¶
LangChain callback handlers allow developers to observe LLM execution.
from langchain_core.callbacks import BaseCallbackHandler
class LoggingCallback(BaseCallbackHandler):
def on_llm_start(self, *args, **kwargs):
print("LLM execution started")
def on_llm_end(self, *args, **kwargs):
print("LLM execution completed")
This callback records the lifecycle of every LLM invocation and can be extended to capture prompts, latency, token usage, and errors.
Example 3 – Production Example (OpenTelemetry + Structured Logs)¶
import logging
from opentelemetry import trace
logger = logging.getLogger("agent")
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("retriever"):
logger.info(
"Retriever started",
extra={
"agent": "RetrieverAgent",
"request_id": "REQ-101",
"correlation_id": "CORR-1001"
}
)
By combining structured logging with OpenTelemetry tracing, engineers can correlate log entries with distributed traces, making it easier to investigate failures across multiple AI agents and services.
Enterprise Use Cases¶
Customer Support AI¶
Customer support agents generate logs throughout the request lifecycle.
Typical logs
- Request received
- Intent identified
- Knowledge retrieved
- Prompt sent
- LLM response generated
- CRM updated
- Response returned
These logs help engineers investigate customer issues and improve response quality.
Enterprise RAG Assistant¶
RAG systems require detailed logging across multiple components.
Typical logs
- Query rewrite
- Retrieved document IDs
- Similarity scores
- Retrieved chunk count
- Reranking score
- Prompt version
- Model used
- Token usage
- Final response
These logs make it easier to identify poor retrieval quality or hallucinations.
Multi-Agent Workflow¶
Enterprise AI platforms coordinate multiple agents.
Logs capture
- Agent started
- Agent completed
- Task assignment
- Task duration
- Retry attempts
- Workflow completion
This provides complete visibility into multi-agent execution.
Financial Services¶
Financial AI systems require audit-quality logging.
Typical audit logs
- Decision timestamp
- Agent version
- Model version
- Rules applied
- Risk score
- Final decision
- User approval
These records support compliance, auditing, and regulatory investigations.
AI Software Engineering Assistant¶
Software engineering agents generate logs throughout development.
Logs include
- Planning completed
- Files generated
- Tests executed
- Deployment initiated
- Deployment completed
These logs simplify debugging and workflow analysis.
Production Insight¶
Logging should capture the complete AI execution lifecycle, not just application errors.
Unlike traditional applications, AI systems should log:
- Prompt versions
- Model versions
- Memory retrieval
- Tool execution
- Agent decisions
- Token usage
- Workflow state
- Execution cost
A production AI platform should allow engineers to reconstruct an entire request from logs.
Centralized Logging Architecture¶
Enterprise AI platforms aggregate logs from every service.
AI Platform
│
┌───────────────┼────────────────┐
▼ ▼ ▼
Agent A Agent B Agent C
│ │ │
└───────────────┼────────────────┘
▼
OpenTelemetry Collector
│
▼
Log Aggregation
│
┌──────────────┼──────────────┐
▼ ▼ ▼
Loki Elasticsearch Splunk
│
▼
Grafana
Centralized logging enables a single place to search, filter, and analyze logs across distributed AI systems.
Log Levels¶
Not every event has the same importance.
| Level | Purpose | Example |
|---|---|---|
| DEBUG | Development diagnostics | Prompt construction |
| INFO | Normal workflow events | Retriever started |
| WARNING | Recoverable issues | Retry initiated |
| ERROR | Failed operations | Tool timeout |
| CRITICAL | System failure | Workflow aborted |
Choosing appropriate log levels reduces noise while preserving useful information.
Architecture Decision¶
| Requirement | Recommended Solution |
|---|---|
| Local Development | Python Logging |
| Structured Logs | JSON Logging |
| Distributed Tracing Integration | OpenTelemetry |
| Centralized Logging | Loki |
| Full-Text Search | Elasticsearch |
| Enterprise Log Analytics | Splunk |
| Cloud Logging | CloudWatch / Azure Monitor / Google Cloud Logging |
| Enterprise AI Platform | OpenTelemetry + Loki + Grafana |
Advantages¶
- Faster debugging
- Complete execution history
- Easier incident investigation
- Better auditability
- Improved compliance
- Supports distributed AI workflows
- Enables performance optimization
- Simplifies production operations
Limitations¶
- Large storage requirements
- Additional infrastructure
- Increased operational costs
- Sensitive data management
- Log retention policies
- Search performance on massive log volumes
Best Practices¶
- Use structured JSON logging.
- Generate Request IDs and Correlation IDs for every request.
- Log significant workflow events rather than every internal function call.
- Include model version, prompt version, and workflow ID.
- Mask sensitive information before storing logs.
- Centralize logs across all AI services.
- Define log retention policies based on business requirements.
- Integrate logging with tracing and metrics for complete observability.
Common Mistakes¶
❌ Logging entire prompts containing sensitive data
❌ Logging every token generated by the LLM
❌ Using plain text instead of structured JSON
❌ Missing Correlation IDs
❌ Logging only errors
❌ Storing logs locally on application servers
❌ No retention or archival policy
❌ Mixing application logs with audit logs
Framework Comparison¶
| Framework | Logging Support |
|---|---|
| Python Logging | Standard application logging |
| Loguru | Simplified structured logging |
| LangChain | Callback Handlers, LangSmith Integration |
| LangGraph | Workflow State & Execution Events |
| OpenTelemetry | Distributed Logging & Correlation |
| Loki | Centralized log aggregation |
| Elasticsearch (ELK) | Search & analytics |
| Splunk | Enterprise log analytics |
| CloudWatch Logs | AWS centralized logging |
| Azure Monitor Logs | Azure logging platform |
Interview Questions¶
What is Agent Logging?¶
Why is logging important in enterprise AI systems?¶
What information should every AI log contain?¶
What is the difference between Request ID and Correlation ID?¶
Why should AI systems use structured logging?¶
Why should sensitive prompt data be masked?¶
How does centralized logging improve debugging?¶
Why should logs be integrated with traces?¶
What is the difference between application logs and audit logs?¶
Why shouldn't AI systems log every LLM token?¶
Quick Revision¶
User Request
│
▼
AI Agent
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
Memory Tool Calls LLM
│ │ │
└─────────────────┼─────────────────┘
▼
Structured Logs
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
Execution Prompt Logs Error Logs
│
▼
OpenTelemetry Collector
│
▼
Loki / ELK
│
▼
Grafana
Key Takeaways¶
- Agent Logging records the complete execution history of AI agents, including prompts, memory operations, tool calls, workflow events, model interactions, and errors.
- Structured JSON logging with Request IDs and Correlation IDs enables efficient searching, debugging, and distributed tracing across multi-agent systems.
- Enterprise AI platforms centralize logs using tools such as OpenTelemetry, Loki, Elasticsearch, Splunk, CloudWatch Logs, or Azure Monitor.
- Logging should balance operational visibility with security by masking sensitive information, separating audit logs from application logs, and enforcing retention policies.
- Effective logging is the foundation for debugging, compliance, monitoring, performance optimization, and production support of enterprise AI applications.
References¶
- Python Logging Documentation
- Loguru Documentation
- OpenTelemetry Documentation
- Loki Documentation
- Elasticsearch Documentation
- Splunk Documentation
- LangGraph Documentation
- LangChain Documentation
- OpenAI Agents SDK Documentation
Next Note¶
03-agent-tracing.md
In the next note, you'll explore Agent Tracing, where you'll learn how to follow an AI request across multiple agents, LLM calls, tools, retrievers, databases, and external APIs using distributed tracing. Topics include spans, traces, context propagation, OpenTelemetry, Jaeger, Tempo, LangSmith traces, LangFuse traces, and production tracing architectures for enterprise AI systems.
Enterprise AI Engineering Handbook
Building Production-Grade Enterprise AI Systems — One Chapter at a Time.