Production Agent Deployment¶

Design and deploy AI Agents as secure, scalable, observable, resilient, and production-ready enterprise services using modern cloud-native deployment practices.

📖 Overview¶

Deploying an AI Agent to production is significantly different from deploying a traditional REST API or a simple LLM application.

A production Agent combines:

LLM
+
Reasoning
+
Memory
+
Tools
+
External Systems
+
Runtime State
+
Security
+
Policies

The deployment architecture must therefore support:

Availability
Scalability
Security
Reliability
Observability
Governance
Cost Control

A production Agent can be viewed as:

                         Users / Applications
                                  │
                                  ▼
                         API Gateway / Ingress
                                  │
                                  ▼
                         Authentication
                                  │
                                  ▼
                        Agent Application
                                  │
                         ┌────────┴────────┐
                         ↓                 ↓
                    Sync Tasks        Async Tasks
                                           │
                                           ▼
                                      Task Queue
                                           │
                                ┌──────────┼──────────┐
                                ↓          ↓          ↓
                             Worker A   Worker B   Worker C
                                │          │          │
                                └──────────┼──────────┘
                                           ↓
                                    Agent Runtime
                                           │
                       ┌───────────────────┼───────────────────┐
                       ↓                   ↓                   ↓
                     Model              Memory              Tools
                       │                   │                   │
                       ↓                   ↓                   ↓
                 Model Provider       State Store        Tool Gateway
                                                               │
                                                               ▼
                                                        Enterprise Systems

Cross-cutting capabilities:

Security
Guardrails
Risk Management
Secrets
Observability
Audit
Cost Management

🎯 Learning Objectives¶

After completing this chapter, you will understand:

Production Agent deployment architecture
Deployment environments
Containerized Agent deployment
Kubernetes deployment
Serverless Agent deployment
Synchronous deployment
Asynchronous deployment
Agent worker deployment
API Gateway integration
Load balancing
Health checks
Autoscaling
Configuration management
Secrets management
Model provider deployment
Tool deployment
State and memory deployment
Deployment strategies
Rolling deployments
Blue-green deployments
Canary deployments
Shadow deployments
Agent versioning
CI/CD
Infrastructure as Code
Production networking
Security boundaries
High availability
Disaster recovery
Rollback
Production readiness
Operational governance

1. From Agent Application to Production Service¶

A development Agent may look like:

Application
 ↓
LLM
 ↓
Tool
 ↓
Response

A production Agent requires additional infrastructure:

Client
 ↓
Gateway
 ↓
Identity
 ↓
Agent Service
 ↓
Runtime
 ↓
Model / Memory / Tools
 ↓
Enterprise Systems

And:

Monitoring
Security
Guardrails
Risk
Scaling
Reliability
Audit

The deployment architecture must account for the entire system rather than only the Agent code.

2. Production Deployment Architecture¶

A practical enterprise architecture:

                              Clients
                                 │
                                 ▼
                         ┌───────────────┐
                         │ API Gateway   │
                         │ / Ingress     │
                         └───────┬───────┘
                                 ↓
                         ┌───────────────┐
                         │ AuthN / AuthZ │
                         └───────┬───────┘
                                 ↓
                         ┌───────────────┐
                         │ Agent API     │
                         └───────┬───────┘
                                 │
                     ┌───────────┴───────────┐
                     ↓                       ↓
                Sync Runtime           Async Runtime
                                             │
                                             ▼
                                        Task Queue
                                             │
                                  ┌──────────┼──────────┐
                                  ↓          ↓          ↓
                               Worker A   Worker B   Worker C
                                  │          │          │
                                  └──────────┼──────────┘
                                             ↓
                                      Agent Runtime
                                             │
                          ┌──────────────────┼──────────────────┐
                          ↓                  ↓                  ↓
                        Model              Memory             Tools
                          │                  │                  │
                          ↓                  ↓                  ↓
                    Model Provider      State Store       Tool Gateway
                                                                  │
                                                                  ▼
                                                          Enterprise APIs

Supporting platform:

Secrets
Policies
Guardrails
Risk Controls
Observability
Audit
Cost Management
CI/CD

3. Deployment Environment Strategy¶

A production Agent should normally move through controlled environments:

Development
     ↓
Testing
     ↓
Staging
     ↓
Production

Each environment should have appropriate separation for:

Credentials
Data
Models
Tools
Policies
Configuration
Monitoring

4. Development Environment¶

Development should optimize for:

Fast Feedback
Experimentation
Debugging

Typical components:

Local Agent
Mock Tools
Test Models
Synthetic Data
Local State
Development Sandbox

Production credentials should not be embedded into development environments.

5. Testing Environment¶

The testing environment validates:

Functional Behavior
Tool Integration
Runtime Behavior
Security
Guardrails
Failure Handling

Examples:

Unit Tests
Integration Tests
Agent Evaluation
Security Tests
Load Tests
Failure Tests

6. Staging Environment¶

Staging should resemble production as closely as practical.

Production-like
Infrastructure
        +
Controlled Data
        +
Production-like Configuration

Validate:

Deployment
Scaling
Networking
Model Connectivity
Tool Connectivity
Observability
Rollback

7. Production Environment¶

Production should provide:

High Availability
Scalability
Security
Observability
Reliability
Backup
Recovery
Governance

Production access should be controlled and auditable.

8. Containerized Agent Deployment¶

Containers provide a portable deployment unit.

Agent Application
       ↓
Container Image
       ↓
Container Runtime
       ↓
Cloud / Kubernetes

A container may package:

Agent Runtime
Application Code
Dependencies
Configuration References

Secrets should generally be injected at runtime rather than baked into the image.

9. Container Image Lifecycle¶

A typical lifecycle:

Source Code
 ↓
Build
 ↓
Unit Tests
 ↓
Security Scan
 ↓
Container Image
 ↓
Registry
 ↓
Deployment

The image should be immutable after publication.

10. Container Registry¶

A production platform typically stores images in a controlled registry.

CI/CD
 ↓
Container Registry
 ↓
Deployment Platform
 ↓
Agent Pods / Containers

Image governance should include:

Versioning
Vulnerability Scanning
Access Control
Retention
Signing where required

11. Kubernetes Deployment¶

A Kubernetes architecture can use:

                     Ingress
                        │
                        ▼
                  Agent Service
                        │
              ┌─────────┼─────────┐
              ↓         ↓         ↓
            Pod A     Pod B     Pod C

Worker execution can be separated:

Task Queue
    │
    ▼
Worker Deployment
    │
 ┌──┼──┐
 ↓  ↓  ↓
W1 W2 W3

This allows API and worker capacity to scale independently.

12. Kubernetes Components¶

A production Agent deployment may use:

Ingress
Service
Deployment
Horizontal Pod Autoscaler
ConfigMap
Secret Reference
Service Account
Network Policy
Pod Disruption Budget

The exact Kubernetes resources depend on the deployment requirements.

13. Agent API Deployment¶

The Agent API should generally be stateless where practical.

                 Load Balancer
                      │
          ┌───────────┼───────────┐
          ↓           ↓           ↓
        Pod A       Pod B       Pod C
          │           │           │
          └───────────┼───────────┘
                      ↓
                 State Store

This allows requests to be distributed across instances.

14. Agent Worker Deployment¶

Long-running tasks can execute through dedicated workers.

                  Task Queue
                      │
          ┌───────────┼───────────┐
          ↓           ↓           ↓
       Worker A    Worker B    Worker C
          │           │           │
          └───────────┼───────────┘
                      ↓
                Agent Runtime

Workers can scale independently according to workload.

15. Synchronous Deployment¶

For short tasks:

Client
 ↓
Gateway
 ↓
Agent API
 ↓
Runtime
 ↓
Model / Tool
 ↓
Response

Suitable for:

Short Conversations
Simple Tool Calls
Low-Latency Requests

16. Asynchronous Deployment¶

For longer tasks:

Client
 ↓
Agent API
 ↓
Create Task
 ↓
Task ID
 ↓
Queue
 ↓
Worker
 ↓
Agent Runtime
 ↓
Result Store

The client can retrieve the result later.

This avoids keeping an HTTP connection open for the entire Agent execution.

17. Event-Driven Deployment¶

Agents can also be triggered by events:

Enterprise Event
       ↓
Event Router
       ↓
Agent Task
       ↓
Queue
       ↓
Worker
       ↓
Agent Runtime

Examples:

Order Event
Incident Event
Customer Event
Document Event
Business Process Event

Event-driven Agents require careful authorization and idempotency controls.

18. Serverless Agent Deployment¶

Some Agent workloads can use serverless infrastructure:

API Gateway
 ↓
Serverless Function
 ↓
Agent Runtime
 ↓
Model / Tools

Serverless can be useful for:

Short-Lived Tasks
Event-Driven Agents
Variable Traffic
Low Operational Overhead

It may be less suitable for:

Long-Running Tasks
Large Stateful Runtimes
Heavy Persistent Workloads
Specialized Execution Environments

19. Choosing a Deployment Model¶

Deployment Model	Best Fit
Container	General-purpose Agent services
Kubernetes	Complex enterprise platforms
Serverless	Short event-driven workloads
Worker + Queue	Long-running Agents
Managed Model Endpoint	Model inference
Hybrid	Enterprise Agent platforms

The deployment model should follow workload characteristics rather than technology preference alone.

20. API Gateway¶

The gateway provides the external boundary.

Common responsibilities:

TLS
Authentication
Rate Limiting
Routing
Request Validation
Tenant Identification
Quota Enforcement
Logging

Architecture:

Client
 ↓
API Gateway
 ↓
Agent API

21. Load Balancing¶

Multiple Agent API instances can be placed behind a load balancer.

                   Load Balancer
                        │
          ┌─────────────┼─────────────┐
          ↓             ↓             ↓
       Agent A       Agent B       Agent C

The load balancer should route only to healthy instances.

22. Health Checks¶

Production Agent services should expose health signals.

Liveness¶

Is the process alive?

Readiness¶

Can the service accept traffic?

Dependency Health¶

Are required dependencies available?

Health checks should distinguish between:

Process Health

and:

Dependency Health

23. Graceful Shutdown¶

During deployment:

Shutdown Signal
 ↓
Stop New Requests
 ↓
Finish / Checkpoint Active Tasks
 ↓
Release Resources
 ↓
Shutdown

For asynchronous workers:

Stop Accepting New Tasks
 ↓
Complete Current Task

or:

Checkpoint
 ↓
Requeue
 ↓
Shutdown

depending on the workload.

24. Autoscaling¶

Production deployments should scale according to workload.

Possible signals:

CPU
Memory
Request Rate
Queue Depth
Queue Age
Concurrent Tasks
Task Latency

For Agent workers:

Queue Depth

can be particularly useful.

25. Horizontal Pod Autoscaling¶

A Kubernetes-based deployment can scale:

Low Workload
 ↓
3 Pods

High Workload
 ↓
20 Pods

But autoscaling should have:

Minimum
Maximum
Scaling Thresholds
Cooldown Behavior

26. Scaling the Complete Agent Platform¶

Scaling only Agent API instances is insufficient.

Scale and protect:

Agent API
 ↓
Workers
 ↓
Model Provider
 ↓
Memory
 ↓
Tool Gateway
 ↓
Enterprise APIs

A bottleneck anywhere in this chain can limit the entire platform.

27. Configuration Management¶

Separate configuration from application code.

Examples:

Model Name
Token Limits
Step Limits
Tool Allowlist
Timeouts
Feature Flags
Runtime Settings

Configuration should be:

Versioned
Environment-Specific
Audited

28. Secrets Management¶

Never hard-code:

API Keys
Passwords
Cloud Credentials
Database Passwords
Model Credentials

Use:

Secret Manager
Vault
Cloud Secret Store
Workload Identity

The Agent should receive only the credentials required for its capabilities.

29. Identity and Workload Identity¶

Where supported, prefer workload identity mechanisms over long-lived credentials.

Agent Workload
 ↓
Workload Identity
 ↓
Cloud Resource

This reduces:

Credential Exposure
Credential Rotation Burden
Long-Lived Secrets

30. Network Architecture¶

A production deployment can separate public and private boundaries:

Internet
   │
   ▼
API Gateway
   │
   ▼
Public / Edge Layer
   │
   ▼
Private Agent Network
   │
   ├── Model
   ├── Memory
   ├── Tool Gateway
   └── Enterprise Systems

Sensitive services should not be unnecessarily exposed publicly.

31. Network Segmentation¶

Separate components according to risk.

Public Zone
 ↓
Gateway

Application Zone
 ↓
Agent Runtime

Restricted Zone
 ↓
Databases
Enterprise APIs
Sensitive Services

Network policies can limit communication between workloads.

32. Tool Deployment¶

Tools can be deployed independently from the Agent runtime.

Agent
 ↓
Tool Gateway
 ↓
Tool Service
 ↓
Enterprise System

This allows:

Independent Scaling
Central Authorization
Auditing
Rate Limiting

33. Tool Gateway¶

A Tool Gateway can centralize:

Authentication
Authorization
Validation
Rate Limiting
Guardrails
Audit
Routing

Architecture:

Agent Runtime
      ↓
Tool Gateway
      ↓
Policy
      ↓
Tool Adapter
      ↓
Enterprise API

34. Model Deployment¶

The model can be:

External API
Managed Cloud Endpoint
Self-Hosted Model
Private Model Endpoint

The Agent runtime should access the model through a provider abstraction.

Agent Runtime
      ↓
ModelProvider
      ↓
Model Adapter
      ↓
Model Endpoint

35. Model Provider Failover¶

Production deployments can use controlled fallback:

Agent Runtime
      ↓
Primary Model
      ↓
Failure
      ↓
Fallback Model

Fallback must respect:

Data Residency
Privacy
Model Capability
Cost
Latency
Policy

36. Memory and State Deployment¶

Agent state should generally be externalized.

Agent Workers
      │
      ├── Session Store
      ├── Task Store
      ├── Checkpoint Store
      └── Memory Store

This allows workers to remain disposable.

37. Durable State¶

For long-running Agents:

Task
 ↓
Checkpoint
 ↓
Worker Failure
 ↓
New Worker
 ↓
Resume

State durability should be aligned with:

RPO
Task Cost
Business Criticality

38. Deployment Strategies¶

Common production deployment strategies:

Rolling
Blue-Green
Canary
Shadow
Feature Flag

The correct strategy depends on:

Risk
Traffic
Rollback Requirements
Change Size
Business Criticality

39. Rolling Deployment¶

Instances are replaced gradually.

Version 1
 ↓
Replace Instance
 ↓
Version 2
 ↓
Replace Instance
 ↓
Version 2

Advantages:

Simple
Resource Efficient

Risk:

Multiple Versions Run Together

40. Blue-Green Deployment¶

Two environments:

BLUE
Current Production

GREEN
New Version

After validation:

Traffic
 BLUE
  ↓
 GREEN

Rollback:

GREEN
 ↓
BLUE

41. Canary Deployment¶

A small percentage of traffic goes to the new version.

Version 1 → 95%
Version 2 → 5%

Monitor:

Errors
Latency
Task Success
Tool Failures
Cost
Guardrail Violations

Then gradually increase:

5%
 ↓
25%
 ↓
50%
 ↓
100%

42. Shadow Deployment¶

A new Agent version receives copied traffic without controlling the real user response.

User Request
      │
      ├────────→ Production Agent
      │
      └────────→ Shadow Agent

Compare:

Agent Outputs
Tool Calls
Latency
Cost
Policy Decisions

This can be useful for evaluating major Agent changes before production cutover.

43. Feature Flags¶

Some Agent capabilities can be released behind feature flags.

Agent
 │
 ├── Feature A → Enabled
 │
 └── Feature B → Disabled

Feature flags can control:

New Model
New Tool
New Prompt
New Runtime Behavior
New Guardrail

44. Agent Versioning¶

Version more than application code.

Track:

Agent Version
Runtime Version
Model Version
Prompt Version
Tool Schema Version
Policy Version
Guardrail Version

Example:

Agent: v4
Runtime: v3
Model: model-x
Prompt: v8
Policy: v5

This improves reproducibility.

45. Configuration Versioning¶

Production configuration should also be versioned.

Examples:

max_steps
max_tokens
timeout
tool_allowlist
model
policy

Configuration changes should be auditable.

46. CI/CD Pipeline¶

A production Agent deployment pipeline can be:

Developer
   ↓
Git
   ↓
Build
   ↓
Unit Tests
   ↓
Integration Tests
   ↓
Agent Evaluation
   ↓
Security Scan
   ↓
Container Build
   ↓
Image Scan
   ↓
Registry
   ↓
Deploy Staging
   ↓
Smoke Tests
   ↓
Canary
   ↓
Production

47. Agent-Specific CI/CD Checks¶

Traditional tests are not sufficient.

Include:

Prompt Tests
Tool Tests
Agent Evaluation
Guardrail Tests
Security Tests
Policy Tests
Regression Tests
Cost Tests

The exact evaluation strategy should depend on the Agent's capabilities and risk.

48. Infrastructure as Code¶

Production infrastructure should be reproducible.

Examples of managed infrastructure:

Network
Kubernetes
Queues
Databases
Secrets
IAM
Monitoring
Load Balancers

Infrastructure as Code can provide:

Version Control
Repeatability
Review
Automation
Environment Consistency

49. Deployment Pipeline Separation¶

A mature enterprise platform can separate:

Application Deployment

from:

Infrastructure Deployment

and:

Agent Configuration Deployment

For example:

Infrastructure
      ↓
Agent Runtime
      ↓
Agent Configuration
      ↓
Agent Version

50. Production Smoke Tests¶

After deployment:

Deploy
 ↓
Health Check
 ↓
Authentication Test
 ↓
Simple Agent Task
 ↓
Tool Test
 ↓
Model Test
 ↓
Observability Test

Only then should traffic be increased.

51. Rollback¶

Every production deployment should have a rollback strategy.

New Version
 ↓
Regression
 ↓
Stop Rollout
 ↓
Rollback
 ↓
Previous Version

For Agent systems, rollback may involve:

Application
Model
Prompt
Tool
Policy
Configuration

52. Agent State and Rollback¶

Rolling back code does not necessarily roll back Agent state.

Runtime v4
 ↓
Checkpoint v4
 ↓
Rollback to Runtime v3

Compatibility between:

Checkpoint Schema
Runtime Version
Agent Version

must therefore be considered.

53. Database Migration Strategy¶

Agent deployments may modify:

Session Schema
Task Schema
Memory Schema
Checkpoint Schema
Audit Schema

Use compatible migration strategies.

A common pattern:

Expand
 ↓
Deploy
 ↓
Migrate
 ↓
Contract

Avoid breaking active Agents during deployment.

54. Long-Running Agent Deployment¶

A long-running task may outlive a deployment.

Deployment
 ↓
Worker Restart
 ↓
Task Still Active

The system should support:

Checkpoint
Requeue
Resume

rather than silently losing the task.

55. Deployment Drain¶

Before shutting down workers:

Worker
 ↓
Drain
 ↓
Stop New Tasks
 ↓
Finish Existing Tasks
 ↓
Checkpoint
 ↓
Shutdown

This reduces task interruption during deployments.

56. Agent Deployment Observability¶

Monitor both infrastructure and Agent behavior.

Infrastructure¶

CPU
Memory
Network
Pod Health
Queue

Agent¶

Task Success
Task Latency
Tool Failures
Model Errors
Token Usage
Cost
Guardrail Blocks

57. Deployment Metrics¶

Useful deployment metrics:

Deployment Success Rate
Rollback Rate
Startup Time
Pod Readiness Time
Task Failure Rate
Task Latency
Error Rate
Model Error Rate
Tool Error Rate

58. Canary Metrics¶

During canary deployment compare:

Old Version
vs
New Version

Metrics:

Task Success
Latency
Tool Selection
Tool Failure
Token Usage
Cost
Guardrail Violations
User Feedback

A technically healthy Agent can still be behaviorally worse.

59. Deployment Audit¶

Track:

Who Deployed
What Changed
When
Which Environment
Which Agent Version
Which Model
Which Configuration

This supports:

Incident Investigation
Compliance
Rollback
Governance

60. Production Security¶

Production Agent deployment should include:

Authentication
Authorization
Network Segmentation
Secrets Management
Encryption
Sandboxing
Guardrails
Audit
Vulnerability Management

The Agent should operate with least privilege.

61. Least Privilege¶

The Agent should receive only required capabilities.

Customer Agent
 ↓
Read Customer Profile
 ↓
Read Orders

not:

Customer Agent
 ↓
Full Database Access

Capability boundaries should be explicit.

62. Deployment and Data Privacy¶

Production deployment must consider:

Data Residency
Data Retention
Encryption
Logging
Memory
Model Provider
Backups

Sensitive information should not automatically appear in:

Logs
Traces
Metrics
Prompts
Checkpoints

63. Tenant Isolation¶

Production multi-tenant Agents require:

Tenant Identity
Tenant State
Tenant Memory
Tenant Authorization
Tenant Quotas
Tenant Cost Attribution

Architecture:

Tenant A
 ↓
Agent Runtime
 ↓
Tenant A State

Tenant B
 ↓
Agent Runtime
 ↓
Tenant B State

64. High Availability¶

High availability can combine:

Multiple API Instances
+
Multiple Workers
+
Durable Queue
+
Replicated State
+
Health Checks
+
Failover

Avoid single points of failure in critical paths.

65. Disaster Recovery¶

Define recovery for:

Agent API Failure
Worker Failure
Queue Failure
State Store Failure
Model Provider Failure
Tool Failure
Regional Failure

Recovery strategies can include:

Replication
Backups
Checkpoint Recovery
Provider Failover
Regional Failover

66. RTO and RPO¶

RTO¶

How quickly can service recover?

RPO¶

How much state can be lost?

For long-running Agents:

Checkpoint Frequency

directly influences recoverability.

67. Production Cost Management¶

Track:

Model Tokens
Model Requests
Workers
Compute
Memory
Storage
Tool Calls
Network
Sandbox Runtime

Attribute cost to:

Agent
User
Tenant
Task

68. Cost Controls¶

Use:

Token Budgets
Step Limits
Tool Limits
Concurrency Limits
Model Routing
Caching
Tenant Quotas

Example:

Simple Task
 ↓
Lower-Cost Model

Complex Task
 ↓
Higher-Capability Model

69. Production Failure Handling¶

A production Agent should have explicit responses for:

Model Timeout
Tool Timeout
Authorization Failure
Policy Denial
State Store Failure
Worker Failure
Queue Failure
Rate Limit
Budget Exhaustion

Possible outcomes:

Retry
Fallback
Pause
Resume
Escalate
Terminate

70. Deployment Resilience¶

Deployment itself should not become an outage source.

Use:

Rolling Updates
Blue-Green
Canary
Graceful Shutdown
Health Checks
Rollback
Checkpointing

71. Production Deployment Architecture¶

A mature enterprise architecture:

                               Users
                                 │
                                 ▼
                        Global Load Balancer
                                 │
                     ┌───────────┴───────────┐
                     ↓                       ↓
                  Region A                Region B
                     │                       │
                API Gateway             API Gateway
                     │                       │
                Agent API               Agent API
                     │                       │
                Task Queue              Task Queue
                     │                       │
             ┌───────┼───────┐       ┌───────┼───────┐
             ↓       ↓       ↓       ↓       ↓       ↓
            W1      W2      W3      W1      W2      W3
             │       │       │       │       │       │
             └───────┼───────┘       └───────┼───────┘
                     │                       │
                     └──────────┬────────────┘
                                ↓
                         Agent Runtime
                                │
              ┌─────────────────┼─────────────────┐
              ↓                 ↓                 ↓
            Model             Memory            Tools
              │                 │                 │
              ↓                 ↓                 ↓
        Model Provider      State Store      Tool Gateway
                                                   │
                                                   ▼
                                            Enterprise Systems

Cross-cutting:

Identity
Security
Guardrails
Risk
Secrets
Observability
Audit
Cost

72. Kubernetes Production Architecture¶

A Kubernetes implementation can look like:

                         Ingress
                            │
                            ▼
                       Agent Service
                            │
                 ┌──────────┼──────────┐
                 ↓          ↓          ↓
               Pod A      Pod B      Pod C
                            │
                            ▼
                        Task Queue
                            │
                 ┌──────────┼──────────┐
                 ↓          ↓          ↓
             Worker A   Worker B   Worker C
                 │          │          │
                 └──────────┼──────────┘
                            ↓
                       Agent Runtime
                            │
              ┌─────────────┼─────────────┐
              ↓             ↓             ↓
            Model         Memory          Tool

Platform services:

Secret Manager
Monitoring
Logging
Tracing
Policy
Identity

73. Java / Spring Boot Deployment Architecture¶

For a Java-first enterprise Agent platform:

Spring Boot Agent Service
│
├── API Layer
│
├── Agent Application Layer
│
├── Agent Runtime
│
├── Context Manager
│
├── State Manager
│
├── ModelProvider
│
├── ToolProvider
│
├── MemoryProvider
│
├── PolicyProvider
│
├── GuardrailProvider
│
└── Infrastructure Adapters

Deployment:

Spring Boot
 ↓
Container
 ↓
Container Registry
 ↓
Kubernetes
 ↓
Agent Pods

Capability-based interfaces keep the Agent core independent from specific infrastructure providers.

74. Cloud-Agnostic Deployment¶

The same logical architecture can be deployed across:

AWS
Azure
GCP
Private Cloud
Hybrid Cloud

The application architecture should remain stable while infrastructure adapters change.

Agent Runtime
      │
      ├── ModelProvider
      ├── MemoryProvider
      ├── ToolProvider
      └── StateStore
               │
               ▼
       Cloud-Specific Adapters

75. Deployment Maturity¶

Level 1 — Prototype¶

Local Process
Local State
Manual Deployment

Level 2 — Application¶

Container
API
External Model
Basic Monitoring

Level 3 — Production¶

Kubernetes / Managed Runtime
External State
CI/CD
Scaling
Security
Observability
Rollback

Level 4 — Enterprise¶

Multi-Tenant
High Availability
Risk Governance
Multi-Region
Disaster Recovery
Cost Governance

Level 5 — Enterprise Agent Platform¶

Central Control Plane
Multiple Agent Types
Shared Runtime
Shared Tool Platform
Model Routing
Policy Engine
Evaluation Platform
Governance

76. Production Readiness Checklist¶

Architecture¶

[ ] Agent API boundary defined
[ ] Runtime architecture defined
[ ] Sync vs async execution defined
[ ] Worker architecture defined
[ ] State architecture defined
[ ] Model architecture defined
[ ] Tool architecture defined

Security¶

[ ] Authentication implemented
[ ] Authorization implemented
[ ] Least privilege applied
[ ] Secrets managed securely
[ ] Network boundaries defined
[ ] Encryption configured
[ ] Sandbox implemented where required
[ ] Guardrails configured

Scalability¶

[ ] Horizontal scaling supported
[ ] Autoscaling configured
[ ] Queue-based execution supported
[ ] Concurrency limits configured
[ ] Tenant quotas configured
[ ] Backpressure implemented
[ ] Downstream capacity validated

Reliability¶

[ ] Timeouts configured
[ ] Retry policies configured
[ ] Circuit breakers considered
[ ] Idempotency implemented
[ ] Checkpointing implemented where required
[ ] Dead-letter handling configured
[ ] Disaster recovery defined

Deployment¶

[ ] Development environment
[ ] Testing environment
[ ] Staging environment
[ ] Production environment
[ ] Container image pipeline
[ ] CI/CD pipeline
[ ] Infrastructure as Code
[ ] Deployment strategy
[ ] Rollback strategy

Observability¶

[ ] Logs
[ ] Metrics
[ ] Traces
[ ] Agent execution telemetry
[ ] Cost monitoring
[ ] Alerts
[ ] Deployment audit

Operations¶

[ ] Health checks
[ ] Graceful shutdown
[ ] Capacity planning
[ ] Load testing
[ ] Failure testing
[ ] Chaos testing
[ ] Incident response
[ ] Runbooks

77. Common Production Deployment Mistakes¶

Mistake 1 — Deploying the Agent as a Single Process¶

One Agent
 ↓
One Server

This creates a major availability and scaling limitation.

Better¶

Multiple Instances
+
External State

Mistake 2 — Keeping State Inside Containers¶

Container
 ↓
Session State
 ↓
Container Restart
 ↓
State Lost

Better¶

External State Store

Mistake 3 — No Rollback Strategy¶

Deploy
 ↓
Problem
 ↓
No Rollback

Better¶

Canary / Blue-Green
+
Versioning
+
Rollback

Mistake 4 — Treating Agent Deployment Like Normal Application Deployment¶

Traditional deployment checks:

HTTP
CPU
Memory

Agent deployment should additionally validate:

Task Success
Tool Behavior
Model Behavior
Guardrails
Cost

Mistake 5 — No Long-Running Task Strategy¶

HTTP Request
 ↓
Agent Runs 30 Minutes

Better¶

Create Task
 ↓
Queue
 ↓
Worker
 ↓
Checkpoint
 ↓
Result

Mistake 6 — Mixing Production and Development Credentials¶

Never allow:

Development Agent
 ↓
Production Database

without explicit controlled access.

Mistake 7 — Deploying Without Behavioral Validation¶

A deployment may be:

Infrastructure Healthy

but:

Agent Quality Degraded

Production promotion should therefore include Agent evaluation.

78. Production Deployment Flow¶

A recommended lifecycle:

Developer
   ↓
Git Commit
   ↓
Build
   ↓
Unit Tests
   ↓
Integration Tests
   ↓
Agent Evaluation
   ↓
Security Tests
   ↓
Container Build
   ↓
Image Scan
   ↓
Registry
   ↓
Deploy Staging
   ↓
Smoke Tests
   ↓
Load / Failure Tests
   ↓
Canary
   ↓
Behavioral Monitoring
   ↓
Production
   ↓
Continuous Monitoring

Rollback:

Regression
 ↓
Stop Rollout
 ↓
Rollback
 ↓
Investigate
 ↓
Fix
 ↓
Redeploy

79. Key Engineering Principles¶

1. Treat the Agent as a Production Distributed System¶

It interacts with:

Models
Databases
Tools
Queues
External APIs

and therefore inherits distributed-system failure modes.

2. Keep the Runtime Disposable¶

Externalize:

State
Memory
Checkpoints
Results

3. Separate API and Worker Scaling¶

API Scaling
≠
Execution Scaling

4. Make Deployment Reversible¶

Use:

Versioning
Canary
Blue-Green
Rollback

5. Deploy With Behavioral Validation¶

Infrastructure health is not sufficient.

6. Protect Long-Running Execution¶

Use:

Queues
Checkpoints
Retries
Idempotency

7. Apply Least Privilege¶

The Agent should receive only the capabilities it needs.

8. Treat Configuration as a Versioned Artifact¶

Track:

Model
Prompt
Policy
Tools
Runtime

9. Monitor the Complete Execution Chain¶

Client
 ↓
Agent
 ↓
Model
 ↓
Tool
 ↓
Enterprise System

10. Design for Failure Before Production¶

Assume:

Worker Will Fail
Model Will Timeout
Tool Will Fail
Queue Will Fill
Provider Will Throttle

and design recovery accordingly.

80. Deployment Decision Framework¶

When choosing a deployment architecture, evaluate:

Task Duration
        ↓
Short / Long

Traffic
        ↓
Stable / Bursty

State
        ↓
Stateless / Stateful

Risk
        ↓
Low / High

Concurrency
        ↓
Low / High

Availability
        ↓
Standard / Critical

Data
        ↓
Public / Sensitive

Then select:

Container
Kubernetes
Serverless
Worker + Queue
Multi-Region
Hybrid

based on those requirements.

81. Part VI Production Boundary¶

Production Agent Deployment belongs to Part VI — AI Agents because the focus is the deployment and operation of an individual production-grade Agent.

The progression is:

Agent
 ↓
Runtime
 ↓
Scaling
 ↓
Deployment
 ↓
Production Operation

Part VI establishes the foundation required for reliable individual Agents.

82. Part VI → Part VII Boundary¶

Part VII — Agentic AI & Multi-Agent Systems can extend this deployment foundation into:

Multiple Agents
      ↓
Delegation
      ↓
Agent-to-Agent Communication
      ↓
Supervision
      ↓
Hierarchical Execution
      ↓
Distributed Autonomous Workflows

Advanced deployment concerns such as:

Multi-Agent Scheduling
Agent Supervisor Scaling
Cross-Agent State
Agent-to-Agent Failure Propagation
Distributed Multi-Agent Recovery
Swarm Deployment

belong primarily in Part VII.

📌 Key Takeaways¶

Production Agent deployment requires an architecture that encompasses the Agent runtime, models, memory, tools, state, security, and enterprise systems.
Separate synchronous request handling from asynchronous long-running execution where appropriate.
Containerization provides a portable deployment unit for Agent services.
Kubernetes can provide scalable API and worker infrastructure for enterprise Agent platforms.
Serverless deployment can be useful for short-lived, event-driven Agent workloads.
Long-running Agents should generally use durable task queues, workers, and checkpointing.
Agent API instances should remain stateless where practical.
Session state, execution state, and checkpoints should be externalized when horizontal scaling is required.
Production environments should be separated from development and staging environments.
Secrets should be injected securely rather than embedded in container images.
Workload identity and least privilege reduce credential exposure.
API gateways provide an important external security and traffic-management boundary.
Tools should be accessed through controlled interfaces such as Tool Gateways where appropriate.
Model providers should be treated as external dependencies with explicit timeout, rate, fallback, and data-governance policies.
Production deployment strategies include rolling, blue-green, canary, shadow, and feature-flag-based releases.
Agent versioning should include runtime, model, prompt, tool, policy, and configuration versions where relevant.
CI/CD should include Agent-specific evaluation, security testing, and behavioral regression testing.
Infrastructure as Code improves repeatability and governance.
Production observability must monitor both infrastructure health and Agent behavior.
Rollback must consider not only application code but also Agent configuration, model, prompt, tools, and state compatibility.
Long-running Agents must survive worker restarts and deployments through checkpointing and recovery.
Production systems should define RTO, RPO, high-availability, and disaster-recovery strategies.
Cost controls should operate at task, Agent, user, and tenant levels where appropriate.
The central principle is:

Deploy AI Agents as governed distributed systems, not simply as LLM-powered applications.

Previous¶

03. Agent Scaling & Resilience

Next Section¶

07-agentic-ai

Enterprise AI Engineering Handbook
Building Production-Grade Enterprise AI Systems — One Chapter at a Time.