Agent Scaling & Resilience¶
Build AI Agents that can handle increasing workloads, recover from failures, maintain availability, and operate reliably under production conditions.
📖 Overview¶
An AI Agent is not simply an LLM endpoint.
As Agent workloads grow, the system must handle:
More Users
More Sessions
More Tasks
More Model Calls
More Tool Calls
More Data
More Concurrent Executions
At the same time, production systems must tolerate:
Model Failures
Tool Failures
Network Failures
Worker Failures
State Store Failures
Queue Failures
Provider Rate Limits
Traffic Spikes
Resource Exhaustion
This creates two closely related engineering requirements:
A production Agent platform should therefore be designed to:
- Scale horizontally
- Control concurrency
- Apply backpressure
- Isolate workloads
- Recover from transient failures
- Preserve execution state
- Prevent cascading failures
- Handle provider throttling
- Protect against runaway Agents
- Maintain predictable performance
🎯 Learning Objectives¶
After completing this chapter, you will understand:
- Agent scalability fundamentals
- Vertical vs horizontal scaling
- Stateless Agent services
- Agent worker scaling
- Queue-based scaling
- Autoscaling
- Queue-depth-based scaling
- Concurrency management
- Tenant-aware scaling
- Fair scheduling
- Backpressure
- Load shedding
- Admission control
- Resource quotas
- Agent workload isolation
- Model provider scaling
- Tool scaling
- Connection pooling
- Caching
- Retry strategies
- Exponential backoff
- Jitter
- Circuit breakers
- Bulkheads
- Timeouts
- Rate limiting
- Idempotency
- Checkpointing
- Recovery
- Graceful degradation
- Failover
- High availability
- Disaster recovery
- Multi-region deployment
- Failure containment
- Chaos testing
- Resilience testing
- Production scaling architecture
1. Why Agent Scaling Is Different¶
Traditional applications often have a relatively predictable execution path:
An Agent may perform:
One user request can therefore generate many downstream operations.
For example:
At scale:
Therefore Agent scalability must consider execution amplification.
2. Agent Workload Amplification¶
A useful mental model:
Traffic can multiply at every layer.
Therefore:
Scaling the Agent API alone does not guarantee that the entire Agent system can scale.
3. Agent Scaling Dimensions¶
Agent platforms may need to scale across:
Users
Sessions
Tasks
Agent Instances
Workers
Model Requests
Tool Calls
Memory Operations
Database Queries
Queue Depth
Each dimension can become a bottleneck.
4. Scaling Architecture¶
A production architecture can separate request handling from execution:
Clients
│
▼
API Gateway
│
▼
Agent API
│
┌─────┴─────┐
│ │
▼ ▼
Sync Tasks Async Tasks
│
▼
Task Queue
│
┌───────────┼───────────┐
↓ ↓ ↓
Worker A Worker B Worker C
│ │ │
└───────────┼───────────┘
↓
Agent Runtime
│
┌─────────────────┼─────────────────┐
↓ ↓ ↓
Model Memory Tools
This allows different components to scale independently.
5. Vertical Scaling¶
Vertical scaling increases the capacity of a single instance.
Advantages:
- Simple
- Easy to implement
- Useful for memory-intensive workloads
Limitations:
- Hardware limits
- Single-instance failure risk
- Expensive at larger sizes
- Limited elasticity
6. Horizontal Scaling¶
Horizontal scaling adds more instances.
Advantages:
- Higher availability
- Better elasticity
- Independent failure domains
- Suitable for cloud-native deployment
For production Agent APIs and workers, horizontal scaling is often the preferred model.
7. Stateless Agent Services¶
Horizontal scaling is easiest when Agent API instances are stateless.
Load Balancer
│
┌────────────┼────────────┐
↓ ↓ ↓
Pod A Pod B Pod C
│ │ │
└────────────┼────────────┘
↓
External State
External services can store:
This allows any healthy instance to handle a request.
8. Why Statelessness Matters¶
Suppose:
and later:
If state exists only inside Agent A:
With external state:
both can access the required state.
9. Scaling Agent Workers¶
For asynchronous Agents:
Workers can scale based on:
10. Queue-Based Scaling¶
A queue separates workload arrival from execution capacity.
When traffic increases:
When traffic decreases:
This creates elasticity.
11. Queue Depth as a Scaling Signal¶
CPU is not always the best scaling metric for Agent workers.
For example:
The system may still be overloaded from a user perspective.
Queue depth can therefore be a valuable autoscaling signal.
Other useful signals include:
12. Autoscaling¶
A production platform can automatically adjust capacity.
Example:
Autoscaling should include sensible upper and lower bounds.
13. Scaling Limits¶
Unlimited autoscaling is dangerous.
Traffic Spike
↓
More Workers
↓
More Model Calls
↓
Provider Throttling
↓
Retries
↓
More Calls
↓
System Overload
Therefore:
Autoscaling must be combined with concurrency, quota, rate, and budget controls.
14. Concurrency Control¶
The runtime should limit concurrent execution.
Agent Platform
│
├── Maximum Tasks
├── Maximum Model Calls
├── Maximum Tool Calls
└── Maximum Sandbox Jobs
Without limits:
15. Per-Agent Concurrency¶
Different Agents may have different limits.
Customer Agent
→ 100 concurrent tasks
Reporting Agent
→ 20 concurrent tasks
Infrastructure Agent
→ 5 concurrent tasks
The limits should reflect:
16. Per-Tenant Concurrency¶
A large tenant should not consume all platform capacity.
Better:
This provides workload fairness.
17. Fair Scheduling¶
A multi-tenant scheduler can use:
Possible strategies:
The exact strategy depends on business requirements.
18. Backpressure¶
Backpressure prevents the system from accepting more work than it can safely process.
When capacity is exhausted:
instead of allowing unlimited work.
19. Admission Control¶
Before accepting an Agent task:
New Task
↓
Authentication
↓
Authorization
↓
Quota Check
↓
Capacity Check
↓
Budget Check
↓
Risk Check
↓
Accept / Reject
Admission control protects the platform before execution begins.
20. Load Shedding¶
When the platform is overloaded, lower-priority work can be deferred or rejected.
Overload
│
├── Critical Tasks → Continue
│
├── Normal Tasks → Queue
│
└── Low Priority → Defer / Reject
This protects critical workloads.
21. Resource Quotas¶
Quotas can be defined for:
Example:
Quotas help prevent resource abuse.
22. Model Provider Scaling¶
The Agent platform may scale faster than the model provider allows.
Example:
Therefore model provider capacity must be part of the scaling architecture.
23. Model Rate Limiting¶
The runtime can enforce:
before calling the model provider.
24. Model Provider Failover¶
If the primary provider becomes unavailable:
Provider failover must consider:
25. Tool Scaling¶
Tools may become bottlenecks.
The tool layer may therefore require:
26. Tool Gateway Scaling¶
A Tool Gateway can scale independently.
This provides a controlled scaling boundary.
27. Connection Pooling¶
High-concurrency Agent systems can exhaust connections.
Use controlled connection pools.
Important parameters include:
Oversized pools can also overload downstream systems.
28. Caching¶
Caching can reduce repeated operations.
Potential caches:
Example:
Caching must respect:
29. Cache Stampede¶
When many Agents request the same uncached data:
This can overload the backend.
Mitigation strategies include:
30. Agent Runtime Resilience¶
Resilience means the Agent platform can continue operating despite failures.
Not every failure must result in complete task failure.
31. Failure Domains¶
Separate failure domains where practical:
A failure in one component should not automatically bring down the entire Agent platform.
32. Bulkhead Pattern¶
Bulkheads isolate workloads.
Agent Platform
│
┌────────────┼────────────┐
↓ ↓ ↓
Tenant A Tenant B Tenant C
Workers Workers Workers
If Tenant A experiences a workload spike:
other tenants remain protected.
33. Bulkhead by Agent¶
Another option:
A problematic Agent cannot consume all platform capacity.
34. Bulkhead by Risk¶
High-risk Agents can have dedicated infrastructure.
This can improve both security and resilience.
35. Timeout Strategy¶
Every external operation should have a timeout.
Timeouts prevent:
36. Retry Strategy¶
Retries can recover from transient failures.
But retries can also amplify load.
Therefore use:
37. Exponential Backoff¶
Instead of:
use increasing delays:
The actual values should be selected according to the dependency and workload.
38. Jitter¶
If thousands of workers retry simultaneously:
Jitter randomizes retry timing:
This reduces synchronized retry storms.
39. Retry Storm¶
A dangerous pattern:
Model Provider
↓
Failure
↓
10,000 Agents
↓
Retry
↓
10,000 More Requests
↓
Provider Overload
↓
More Failures
Retries can turn a temporary problem into a major outage.
40. Circuit Breaker¶
A circuit breaker stops repeated calls to an unhealthy dependency.
After a recovery period:
41. Circuit Breaker States¶
Closed¶
Open¶
Half-Open¶
This protects both the Agent and the dependency.
42. Graceful Degradation¶
Not every dependency failure needs to produce a complete failure.
Example:
Possible degradation:
Another example:
The Agent may continue without the enrichment.
43. Dependency Criticality¶
Classify dependencies:
Example:
Authentication → Critical
Primary Model → Critical
Customer Database → Critical
Analytics API → Optional
Failure handling should depend on criticality.
44. Dependency Failure Matrix¶
| Dependency | Failure Strategy |
|---|---|
| Primary Model | Fallback / Retry |
| Memory Store | Retry / Degraded Mode |
| Optional Search | Continue Without |
| Payment API | Stop / Escalate |
| Analytics API | Degrade |
| Authorization Service | Fail Closed |
The correct strategy depends on the business context.
45. Fail-Fast vs Fail-Safe¶
Fail-Fast¶
Stop quickly when continuing is unsafe.
Fail-Safe¶
Move to a safe fallback state.
Security-sensitive operations should generally fail closed rather than bypassing controls.
46. Idempotency¶
Distributed systems can execute an operation more than once.
Without idempotency:
With idempotency:
47. Idempotency Keys¶
For side-effecting operations:
can form an idempotency key.
Example:
Repeated attempts use the same key.
48. Checkpointing¶
Long-running Agent execution should persist progress.
If the worker fails:
49. Checkpoint Frequency¶
Too frequent:
Too infrequent:
The checkpoint interval should be selected based on:
50. Recovery¶
Recovery can follow:
For side-effecting operations:
51. Worker Failure¶
Example:
A resilient system:
Without checkpointing:
52. Queue Failure¶
A production queue should have appropriate:
The Agent platform should avoid losing tasks because a worker or process failed.
53. Dead-Letter Queue¶
Repeatedly failing tasks can move to a dead-letter queue.
Operators can inspect and remediate these tasks.
54. Poison Tasks¶
A poison task repeatedly fails because of its content or configuration.
Without limits, it can consume worker capacity indefinitely.
Controls:
55. Agent Loop Resilience¶
An Agent may accidentally loop:
Controls:
56. Runaway Agent Protection¶
A production runtime should enforce:
This protects:
57. Tool Call Amplification¶
An Agent may generate many calls:
This creates downstream load.
Controls:
58. Result Size Limits¶
Tools can return unexpectedly large results.
This can cause:
The runtime should impose:
59. Context Growth¶
Repeated tool calls can grow context:
Eventually:
Controls include:
60. Memory Store Scaling¶
Memory systems can become bottlenecks.
Scale through:
The exact approach depends on the chosen storage technology.
61. Memory Hotspots¶
A popular tenant or session may create disproportionate load.
Controls:
62. Database Scaling¶
Agent workloads may generate:
Potential strategies:
63. Audit Scaling¶
Agent systems can generate large volumes of audit events.
The audit pipeline should therefore be designed to scale independently.
64. Observability Scaling¶
High-volume tracing can become expensive.
Use appropriate strategies:
But retain sufficient information for:
65. Multi-Region Agent Deployment¶
For high availability:
Global Traffic
│
┌─────────┴─────────┐
↓ ↓
Region A Region B
│ │
Agent API Agent API
│ │
Workers Workers
│ │
State / Data State / Data
The architecture must carefully handle:
66. Active-Passive¶
One region handles traffic:
If Region A fails:
Advantages:
Limitations:
67. Active-Active¶
Both regions serve traffic:
Advantages:
Challenges:
68. Regional State¶
Long-running Agent state creates an important question:
Possible approaches:
The right model depends on:
69. Disaster Recovery¶
Agent platforms should define recovery objectives.
RTO¶
RPO¶
For long-running Agents:
directly influences recovery characteristics.
70. Recovery Point Example¶
If checkpoints occur every:
a worker failure could potentially require replaying up to approximately that amount of work, depending on the architecture.
Therefore checkpoint frequency should reflect:
71. High Availability¶
High availability can combine:
No single instance should become a critical single point of failure.
72. Health Checks¶
Agent infrastructure should monitor:
A worker should not receive new tasks if it cannot safely execute them.
73. Graceful Shutdown¶
During deployment or scaling down:
For long-running tasks:
may be preferable to waiting indefinitely.
74. Graceful Degradation Architecture¶
Primary Capability
│
▼
Failure
│
┌─────┴─────┐
↓ ↓
Fallback Optional
Model Capability
↓ ↓
Continue Skip
The fallback path should be explicitly designed rather than improvised during failures.
75. Resilience and Risk¶
High-risk actions require stronger failure handling.
Example:
The runtime should not blindly retry.
Instead:
This is especially important for financial and other irreversible operations.
76. Compensation¶
Some operations cannot simply be retried.
Example:
A compensation action may be required:
The exact compensation strategy belongs to the business workflow.
77. Resilience Patterns Summary¶
Important patterns include:
Timeout
Retry
Backoff
Jitter
Circuit Breaker
Bulkhead
Rate Limit
Backpressure
Load Shedding
Idempotency
Checkpointing
Failover
Graceful Degradation
These patterns should be applied according to failure mode rather than mechanically everywhere.
78. Scaling and Resilience Interaction¶
Scaling without resilience:
Resilience without scaling:
A production platform needs both:
79. Failure Amplification¶
Agent systems can amplify failures.
Example:
This creates cascading load.
The runtime should therefore coordinate:
80. Retry Budget¶
A useful concept is a retry budget.
The platform should avoid allowing retries to consume unlimited capacity.
81. Error Budget¶
Traditional SRE concepts can also apply to Agent platforms.
For example:
An Agent platform can define an error budget around:
Agent-specific SLOs should also consider behavior and quality.
82. Agent SLOs¶
Possible Service Level Objectives:
For example:
The exact target should be defined according to business requirements.
83. Agent SLIs¶
Useful indicators:
Task Success Rate
p50 Task Latency
p95 Task Latency
p99 Task Latency
Queue Age
Tool Error Rate
Model Error Rate
Retry Rate
Cancellation Rate
84. Resilience Testing¶
Test failure scenarios deliberately:
85. Chaos Engineering¶
Chaos testing introduces controlled failures.
Examples:
Worker Termination
Network Delay
Network Failure
Model Timeout
Database Failure
Queue Failure
Credential Expiration
Expected result:
86. Chaos Testing for Agents¶
Agent-specific chaos scenarios include:
Model Returns Invalid Tool Call
Tool Returns Huge Result
Tool Repeatedly Fails
Memory Becomes Unavailable
Agent Enters Loop
Worker Dies Mid-Task
Provider Returns Rate Limit
These tests validate the runtime rather than just infrastructure.
87. Load Testing¶
Agent load testing should model realistic execution.
Not just:
but:
This better reflects actual downstream load.
88. Burst Testing¶
Test sudden workload spikes:
Observe:
89. Soak Testing¶
Run the Agent platform for extended periods.
Look for:
90. Scalability Testing¶
Increase workload gradually:
Measure:
91. Capacity Planning¶
Capacity planning should consider:
Requests/sec
Tasks/sec
Average Steps
Average Model Calls
Average Tool Calls
Average Task Duration
Concurrency
Example:
The actual capacity depends on the runtime and workload characteristics.
92. Bottleneck Identification¶
Typical bottlenecks:
The slowest dependency can limit overall throughput.
93. Little's Law¶
Queueing theory can help reason about workload capacity.
Where:
For Agent systems, this can help reason about:
It is a simplified model and real Agent workloads may have variable execution times and downstream dependencies.
94. Example Capacity Reasoning¶
Suppose:
Then:
This gives a first-order estimate of concurrency requirements.
Actual production capacity should be validated through load testing.
95. Agent Scaling Architecture¶
A mature architecture may look like:
Users
│
▼
Global Gateway
│
┌────────────┴────────────┐
↓ ↓
Region A Region B
│ │
Agent API Agent API
│ │
Task Queue Task Queue
│ │
┌──────┼──────┐ ┌──────┼──────┐
↓ ↓ ↓ ↓ ↓ ↓
W1 W2 W3 W1 W2 W3
│ │ │ │ │ │
└──────┼──────┘ └──────┼──────┘
│ │
└──────────┬──────────────┘
↓
Shared / Replicated
State Services
│
┌────────────────┼────────────────┐
↓ ↓ ↓
Model Memory Tools
Cross-cutting:
96. Production Scaling Strategy¶
A practical scaling strategy:
1. Measure
↓
2. Identify Bottleneck
↓
3. Apply Capacity Control
↓
4. Scale Component
↓
5. Load Test
↓
6. Monitor
↓
7. Reassess
Avoid scaling blindly.
97. Production Resilience Strategy¶
A practical resilience strategy:
Identify Failure
↓
Classify Failure
↓
Detect Quickly
↓
Contain Blast Radius
↓
Retry / Recover / Degrade
↓
Persist State
↓
Escalate if Required
↓
Learn and Improve
98. Production Readiness Checklist¶
Scaling¶
- [ ] Horizontal scaling supported
- [ ] Stateless API design where practical
- [ ] Worker scaling supported
- [ ] Queue-based execution available
- [ ] Autoscaling configured
- [ ] Concurrency limits defined
- [ ] Tenant quotas defined
- [ ] Backpressure implemented
- [ ] Admission control implemented
Resilience¶
- [ ] Timeouts configured
- [ ] Retry policies defined
- [ ] Exponential backoff implemented
- [ ] Jitter implemented where appropriate
- [ ] Circuit breakers configured
- [ ] Bulkheads defined
- [ ] Idempotency implemented
- [ ] Checkpointing implemented where required
- [ ] Dead-letter handling implemented
- [ ] Graceful degradation defined
High Availability¶
- [ ] Multiple Agent instances
- [ ] Multiple workers
- [ ] Durable queues
- [ ] Replicated state where required
- [ ] Health checks
- [ ] Failover strategy
- [ ] Disaster recovery plan
Performance¶
- [ ] Load testing completed
- [ ] Burst testing completed
- [ ] Soak testing completed
- [ ] Capacity limits identified
- [ ] Bottlenecks identified
- [ ] Cost measured
Agent Safety¶
- [ ] Step limits
- [ ] Tool limits
- [ ] Token limits
- [ ] Runtime limits
- [ ] Resource limits
- [ ] Risk-aware execution
- [ ] High-risk actions protected
99. Common Scaling Mistakes¶
Mistake 1 — Scaling Only the Agent API¶
Better¶
Scale:
as a complete system.
Mistake 2 — Unlimited Autoscaling¶
Better¶
Mistake 3 — Aggressive Retries¶
Better¶
Mistake 4 — No Tenant Isolation¶
Better¶
Mistake 5 — Ignoring Downstream Systems¶
Better¶
Scale and protect the complete dependency chain.
Mistake 6 — No Recovery State¶
Better¶
Mistake 7 — Measuring Only CPU¶
does not necessarily mean:
Also measure:
100. Key Engineering Principles¶
1. Scale the Entire Execution Chain¶
2. Prefer Horizontal Scaling¶
Use multiple stateless API instances and scalable workers where practical.
3. Use Queues for Long-Running Work¶
Separate task admission from task execution.
4. Control Concurrency¶
Never assume the platform can safely execute unlimited Agent tasks.
5. Use Backpressure¶
Protect downstream systems from overload.
6. Bound Retries¶
Retries can amplify failures.
7. Use Circuit Breakers¶
Stop repeatedly calling unhealthy dependencies.
8. Use Bulkheads¶
Prevent one workload from consuming all platform capacity.
9. Make Side Effects Idempotent¶
Retries are unavoidable in distributed systems.
10. Persist Execution State¶
Long-running Agents need recoverability.
11. Monitor Agent-Specific Signals¶
Infrastructure health alone is insufficient.
12. Test Failure Deliberately¶
Resilience should be validated, not assumed.
101. Agent Scaling Maturity¶
Level 1 — Prototype¶
Level 2 — Application¶
Level 3 — Production¶
Level 4 — Enterprise¶
Multi-Tenant
Fair Scheduling
Bulkheads
Quotas
Multi-Region
Disaster Recovery
Cost Governance
Chaos Testing
Level 5 — Agent Platform¶
Dynamic Scheduling
Risk-Aware Scaling
Multi-Agent Workloads
Central Policy
Distributed Execution
Advanced Capacity Management
102. Java / Spring Boot Scaling Architecture¶
For a Java-first enterprise Agent platform:
Spring Boot Agent API
│
Load Balancer
│
┌──────────────┼──────────────┐
↓ ↓ ↓
Pod A Pod B Pod C
│ │ │
└──────────────┼──────────────┘
↓
Task Queue
│
┌──────────────┼──────────────┐
↓ ↓ ↓
Worker A Worker B Worker C
│ │ │
└──────────────┼──────────────┘
↓
Agent Runtime
│
┌───────────────┼───────────────┐
↓ ↓ ↓
ModelProvider MemoryProvider ToolProvider
│ │ │
↓ ↓ ↓
Model API State Store Tool Gateway
The application layer should remain independent of infrastructure-specific scaling mechanisms.
103. Capability-Based Interfaces¶
A Java-first architecture can use interfaces such as:
ModelProvider
MemoryProvider
ToolProvider
StateStore
CheckpointStore
PolicyProvider
RateLimitProvider
Infrastructure adapters can implement them.
This allows the Agent runtime to remain cloud-independent.
104. Scaling Control Plane¶
A centralized control plane can manage:
Agent Configuration
Worker Limits
Tenant Quotas
Model Limits
Tool Limits
Risk Policies
Scaling Policies
Execution plane:
Control plane:
105. Scaling vs Agentic AI Boundary¶
This chapter belongs to Part VI — AI Agents because it focuses on making an individual Agent runtime production-ready under increasing workload and failure conditions.
The focus is:
Part VII — Agentic AI & Multi-Agent Systems can extend these principles to:
Advanced topics such as:
- Multi-agent workload orchestration
- Cross-agent scheduling
- Agent supervisor scaling
- Agent-to-agent failure propagation
- Distributed multi-agent recovery
- Swarm scalability
- Hierarchical agent execution
belong primarily in Part VII.
📌 Key Takeaways¶
- Agent scalability must consider the entire execution chain, not just the Agent API.
- A single user request can generate multiple model and tool operations, creating workload amplification.
- Horizontal scaling is generally preferable for cloud-native Agent services.
- Stateless API services make horizontal scaling easier when state is stored externally.
- Long-running Agent tasks benefit from queues and worker-based execution.
- Queue depth, task latency, concurrency, and queue age can be more meaningful scaling signals than CPU alone.
- Autoscaling must be bounded by quotas, concurrency limits, rate limits, and budgets.
- Multi-tenant Agent platforms require fair scheduling and tenant isolation.
- Backpressure and admission control prevent overload from propagating through the system.
- Load shedding can protect critical workloads during severe capacity pressure.
- Model providers and downstream tools can become bottlenecks even when Agent workers scale successfully.
- Retries must be bounded and combined with exponential backoff and jitter.
- Circuit breakers prevent repeated calls to unhealthy dependencies.
- Bulkheads isolate tenants, Agents, or workloads from each other.
- Idempotency is essential when retries can repeat side effects.
- Checkpointing allows long-running Agents to recover after worker failures.
- Dead-letter queues provide a safe destination for repeatedly failing tasks.
- Step, token, tool, runtime, and resource limits protect against runaway Agent behavior.
- Graceful degradation allows non-critical capabilities to fail without taking down the complete task.
- High availability may require multiple workers, durable queues, replicated state, and regional failover.
- Multi-region Agent deployment introduces additional challenges around state, consistency, data residency, and duplicate execution.
- Load, burst, soak, scalability, and chaos testing should be part of production readiness.
- Agent-specific SLOs should include task success, task latency, tool reliability, and queue performance.
- The core principle is:
Scale capacity without scaling failure.
🔗 Related Topics¶
Previous¶
Next¶
04. Production Agent Deployment
Related¶
- 01. Agent Deployment Overview
- 05. Agent Authorization
- 06. Secrets Management
- 07. Data Privacy
- 08. Agent Sandboxing
- 09. Agent Guardrails
- 10. Agent Risk Management
- Planning & Task Decomposition
- Agent Reasoning
- Reflection & Self-Correction
- Agent Evaluation
Enterprise AI Engineering Handbook
Building Production-Grade Enterprise AI Systems — One Chapter at a Time.