Building Reliable AI Agents: Retries, State Management, Checkpoints, and Failure Recovery
An AI agent successfully completes a task during testing. It retrieves information, calls the right APIs, processes the results, and produces the expected answer.
Then, in production, the same agent encounters a temporary API outage, loses its connection halfway through execution, or receives an unexpected response from an external service.
What happens next?
Does it start again and repeat every action? Does it remember which operations have already succeeded? Could it accidentally send the same payment request twice? Can it recover from an interrupted workflow without asking the user to repeat everything?
These questions expose one of the most important challenges in agent engineering: reliability.
An AI agent is not production-ready simply because it can complete a task once. It must also handle interruptions, recover from failures, preserve important execution state, and avoid repeating operations that have already produced side effects.
This requires more than better prompts or a more capable model. It requires established software engineering principles such as timeouts, retries, idempotency, persistent state, checkpoints, and fault tolerance.
This article explains how these mechanisms work together and how developers can use them to build AI agents that remain reliable beyond controlled demonstrations.
1. Why AI agents need reliability engineering
Traditional applications usually execute predefined instructions. Developers know which operations should happen and can often predict the execution path.
AI agents introduce a degree of uncertainty.
An agent may dynamically select tools, interpret external responses, decide whether to retry an operation, or delegate work to another agent.
Consider an AI agent responsible for processing customer orders:
- Retrieve the customer's information.
- Verify the order.
- Check inventory.
- Request payment authorization.
- Create a shipping request.
- Notify the customer.
This workflow depends on several independent components:
- The language model.
- The customer database.
- The inventory service.
- The payment provider.
- The shipping API.
- The notification service.
A failure in any one of these components can interrupt the entire task.
Worse, a failure does not always mean that an operation did not happen.
Imagine the agent sends a payment request. The payment provider processes it successfully, but the response never reaches the application because the network connection times out.
From the application's perspective, the result is unknown.
If the agent simply repeats the request, it could charge the customer twice.
This is the fundamental problem that reliability engineering must address: the application needs to distinguish between an operation that failed, an operation that succeeded, and an operation whose outcome is unknown.
Common failure categories
| Failure category | Example | Required response |
|---|---|---|
| Model failure | Invalid or incomplete tool arguments | Validate, correct, or request clarification |
| Tool failure | API returns an error | Classify the error and apply an appropriate policy |
| Network failure | Connection timeout | Check the operation's outcome before retrying |
| Execution failure | Worker crashes | Restore state and resume safely |
| State failure | Corrupted or missing checkpoint | Detect the inconsistency and recover or escalate |
| Logical failure | Agent selects the wrong action | Diagnose and improve the workflow or evaluation |
| Authorization failure | Agent requests a forbidden operation | Reject the action; do not retry without a permission change |
These failures require different solutions. Treating every failure as a reason to restart the agent is a common source of duplicate operations, unnecessary costs, and unpredictable behavior.
2. Understand the difference between conversation state and execution state
Before discussing recovery, it is important to understand what an agent needs to remember.
Developers often use the word memory to describe several different things. However, retaining a conversation is not the same as preserving an unfinished workflow.
Consider a user asking an agent:
"Find the three cheapest suppliers, compare their delivery times, and prepare a purchase recommendation."
The agent may need to preserve:
- The original request.
- The suppliers already discovered.
- The results returned by each API.
- The supplier currently being evaluated.
- The remaining tasks.
- The final recommendation status.
If the application only stores conversation messages, it may know what the user requested but not exactly which operations were completed.
A reliable system distinguishes between several kinds of state.
Conversation state
Contains the interaction history between the user and the agent.
Examples:
- User messages.
- Agent responses.
- Relevant previous exchanges.
- Conversation identifiers.
Workflow state
Represents the current progress of a task.
Examples:
- Current execution step.
- Completed operations.
- Pending operations.
- Intermediate results.
- Error status.
- Retry count.
Application state
Contains information maintained by the surrounding application.
Examples:
- Customer records.
- Orders.
- Account permissions.
- Payment records.
- Business rules.
Long-term memory
Contains information that may be useful across separate tasks or conversations.
Examples:
- User preferences.
- Previously established facts.
- Persistent knowledge.
These categories should not be confused.
An agent can remember an entire conversation and still be unable to resume an interrupted payment workflow safely.
The application needs an explicit representation of execution progress.
3. State management: make execution progress explicit
A reliable agent should not depend entirely on the language model remembering what it has already done.
Instead, represent the workflow as structured application state.
For example, a simplified order-processing workflow might maintain:
{
"task_id": "task_1042",
"status": "in_progress",
"current_step": "inventory_check",
"steps": {
"customer_verification": "completed",
"order_validation": "completed",
"inventory_check": "in_progress",
"payment": "pending",
"shipping": "pending"
},
"retry_count": 0,
"last_error": null
}
This is an illustrative structure. A real implementation should also define schema versions, timestamps, ownership, and validation rules.
The important idea is that execution progress is represented explicitly rather than inferred from a natural-language conversation.
Why structured state matters
Suppose the worker crashes after verifying the customer and validating the order.
When the application restarts, it can inspect the saved workflow state and determine that the first two steps have already completed.
It can then continue from inventory verification instead of repeating the entire workflow.
A good state model should answer:
- What task is being executed?
- What is its current status?
- Which operations have completed?
- Which operations remain?
- What information is required to continue?
- Has an operation produced an external side effect?
- What should happen if execution cannot continue?
State transitions
It is useful to define valid transitions explicitly.
For example:
PENDING
|
v
RUNNING
|
+------> COMPLETED
|
+------> WAITING_FOR_APPROVAL
|
+------> RETRY_SCHEDULED
|
+------> FAILED
The application should control these transitions.
A model may suggest that a task is complete, but the application should verify that required operations actually succeeded before changing the workflow status to COMPLETED.
Engineering principle: The model generates decisions; the application owns authoritative execution state.
4. Retry strategies: recover from temporary failures without creating new problems
Retries are one of the simplest ways to improve application reliability.
However, retrying every failed operation is not a safe strategy.
Some failures are temporary. Others indicate a permanent problem that will not disappear after waiting.
Classify errors before retrying
A useful distinction is between transient and permanent failures.
| Error | Typical treatment |
|---|---|
| Network timeout | Retry cautiously after checking the outcome |
| HTTP 429 rate limit | Respect Retry-After and apply backoff |
| HTTP 500 server error | Often retryable, depending on the operation |
| HTTP 503 unavailable | Usually retryable with limits |
| HTTP 400 invalid input | Correct the input; do not repeat unchanged |
| HTTP 401 unauthorized | Fix authentication or credentials |
| HTTP 403 forbidden | Reject or request an authorized permission change |
| Invalid tool arguments | Validate and correct the arguments |
| Business rule violation | Stop and report the reason |
These are general guidelines, not universal rules. Actual retry behavior depends on the API's contract and error semantics.
OpenAI's Agents API recovery guidance, for example, recommends checking session and turn status, inspecting completed actions, limiting retries, and distinguishing transient failures from invalid input or credential problems.
Source: OpenAI — Errors and Recovery
Exponential backoff
A common retry strategy is exponential backoff.
Instead of retrying immediately, the application increases the waiting time between attempts.
A simplified formula is:
[ d_n = \min(d_{\max}, d_0 \times 2^n) ]
Where:
- (d_n) is the delay before retry (n).
- (d_0) is the initial delay.
- (d_{\max}) is the maximum delay.
For example, using an initial delay of one second:
| Attempt | Approximate delay |
|---|---|
| First retry | 1 second |
| Second retry | 2 seconds |
| Third retry | 4 seconds |
| Fourth retry | 8 seconds |
This reduces the pressure on an overloaded service.
Add jitter
If thousands of agents fail at the same time and all retry after exactly four seconds, they may overwhelm the same service again.
Jitter introduces a random variation into retry delays, spreading requests over time.
A simplified full-jitter strategy is:
[ d_n = U(0,\min(d_{\max},d_0 \times 2^n)) ]
Here, (U(0,x)) means a random delay between zero and (x).
This technique is discussed in Amazon's engineering guidance on timeouts, retries, and backoff.
Source: AWS Builders' Library — Timeouts, Retries, and Backoff with Jitter
Implement retry limits
Retries should always have limits.
Possible limits include:
- Maximum number of attempts.
- Maximum total elapsed time.
- Maximum execution cost.
- Maximum number of tool calls.
- Overall task deadline.
For example, an inventory API might allow three attempts with a total timeout budget of 15 seconds.
If the service remains unavailable, the agent should stop and report that it cannot verify inventory.
It should not continue calling the same endpoint indefinitely.
Example retry logic
The following Python example demonstrates the general pattern:
import random
import time
def call_with_retry(operation, max_attempts=3):
base_delay = 1
max_delay = 8
for attempt in range(max_attempts):
try:
return operation()
except TemporaryServiceError:
if attempt == max_attempts - 1:
raise
delay = min(
max_delay,
base_delay * (2 ** attempt)
)
time.sleep(random.uniform(0, delay))
except PermanentServiceError:
raise
TemporaryServiceError and PermanentServiceError are illustrative application-defined exceptions.
In a production implementation, also consider request deadlines, cancellation, Retry-After, logging, and whether the operation is safe to repeat.
Important: Retry logic should generally live at a clearly defined layer. Multiple independent layers retrying the same request can multiply attempts unexpectedly.
5. Idempotency: prevent duplicate actions
Retries become particularly dangerous when tools modify external systems.
Consider an agent that creates an order through an API.
The request reaches the server and creates the order successfully. However, the response is lost.
The agent cannot determine whether the order was created.
If it sends the same request again, the server might create a second order.
This is where idempotency becomes essential.
What is idempotency?
An operation is idempotent when repeating the same operation produces the same intended effect as executing it once.
For example:
- Reading an order is normally safe to repeat.
- Setting an order status to
confirmedcan be designed to be idempotent. - Creating a new order is not automatically idempotent.
- Charging a payment is not automatically idempotent.
For non-idempotent operations, an application can use an idempotency key.
How idempotency keys work
Before sending a sensitive request, the application generates a unique key.
For example:
idempotency_key = task_1042_payment_01
The payment service stores the result associated with that key.
If the same request arrives again with the same key, the service can return the existing result instead of processing a second payment.
Conceptually:
Agent requests payment
|
v
Payment API receives request
|
v
Check idempotency key
|
+---+---+
| |
Already New
processed request
| |
Return Process
existing payment
result |
| v
| Store result
| |
+---+---+
|
v
Response
Important implementation details
An idempotency system should:
- Generate a stable key for the logical operation.
- Reuse that key when retrying the same operation.
- Associate the key with the intended request parameters.
- Persist the operation's result and status.
- Reject reuse of the same key with conflicting parameters.
- Define how long idempotency records remain valid.
A new logical operation should receive a new key.
For example, retrying a payment request should reuse the original key, while charging a different order should use another key.
Idempotency does not guarantee exactly-once execution
This distinction is important.
In distributed systems, an operation may execute more than once even if the application ultimately exposes only one logical result.
A system can use deduplication and idempotency to prevent duplicate business effects, but it should not assume that network delivery or internal execution happens exactly once.
Temporal's documentation explicitly warns that activities may execute more than once during retries and recommends designing them to be idempotent.
Sources: Temporal — Activity Definition · Temporal — Error Handling and Idempotence
6. Checkpointing: save progress so the agent can resume
Retries handle temporary failures of individual operations.
But what happens when an entire workflow is interrupted?
Imagine an agent conducting a 30-minute research task involving multiple websites, document extraction, data analysis, and report generation.
The application server crashes after 20 minutes.
Without persistent execution state, the application may need to repeat the entire task.
Checkpointing provides a better solution.
What is a checkpoint?
A checkpoint is a saved snapshot of workflow state at a particular point during execution.
It can include:
- Current workflow position.
- Completed task outputs.
- Pending operations.
- Relevant agent context.
- Intermediate results.
- State needed to resume execution.
For example:
Task started
|
v
Search documents
|
v
Extract information
|
v
Analyze results
|
X Worker crashes
|
v
Restore latest checkpoint
|
v
Continue unfinished work
|
v
Generate report
Instead of restarting from the beginning, the system restores the latest valid state.
Checkpoints versus ordinary conversation history
Conversation history stores messages.
A workflow checkpoint stores execution progress.
For example, a conversation might contain:
"Searching the first supplier's catalog now."
But a useful workflow checkpoint should contain structured information such as:
- Supplier A: completed.
- Supplier B: completed.
- Supplier C: pending.
- Extracted prices: saved.
- Next operation: compare delivery times.
The second representation is much more useful for recovery.
When should checkpoints be created?
There is no single checkpoint frequency suitable for every application.
A reasonable strategy is to save state:
- After completing an important workflow stage.
- Before an expensive or irreversible operation.
- After receiving a significant external result.
- Before waiting for human approval.
- At boundaries between long-running tasks.
More frequent checkpoints can reduce lost work but increase storage and processing overhead.
The appropriate balance depends on task duration, recovery requirements, and the cost of repeating completed operations.
Checkpoint storage
Checkpoint data must survive the failure it is intended to recover from.
An in-memory checkpoint disappears when the process or machine is restarted.
For production workloads, developers generally need durable storage such as a database or persistent workflow service.
Examples include:
- PostgreSQL.
- Azure Cosmos DB.
- Other durable database systems.
- Dedicated workflow orchestration platforms.
The storage mechanism should support the required durability, concurrency, and retention characteristics.
Framework implementations
Several agent frameworks provide checkpointing or persistence capabilities.
LangGraph
LangGraph uses checkpointers to save graph state across execution steps. Its persistence system supports conversation continuity, human-in-the-loop workflows, replay, and fault tolerance.
It also distinguishes between:
- Checkpointers for short-term, thread-specific execution state.
- Stores for longer-term application-defined data shared across threads.
The documentation describes recovery from the latest successful checkpoint and preservation of successful task writes when another task fails within the same execution step.
Documentation: LangGraph — Persistence
Microsoft Agent Framework
Microsoft Agent Framework supports workflow checkpointing, allowing developers to persist workflow state and resume execution.
Its documented storage implementations include:
InMemoryCheckpointStoragefor tests and short-lived workflows.FileCheckpointStoragefor local or single-machine use.CosmosCheckpointStoragefor distributed, durable scenarios.
Custom executors must also save and restore their own state appropriately.
Documentation: Microsoft Agent Framework — Checkpoints
Temporal
Temporal approaches the problem through durable workflow execution.
Instead of relying on a model to reconstruct its progress, the workflow engine records execution history and coordinates activities, retries, and recovery.
Documentation: Temporal — Documentation
These solutions address related problems but are not interchangeable. Their persistence models, execution semantics, operational requirements, and integration patterns differ.
7. Durable execution: recover without relying on a running process
Checkpointing saves progress. Durable execution takes the idea further.
A durable execution system is designed to preserve workflow progress even when the application process or worker responsible for execution disappears.
This is particularly useful for:
- Long-running research agents.
- Financial processing workflows.
- Document-processing pipelines.
- Multi-step enterprise automation.
- Workflows requiring human approval.
- Tasks that may remain active for hours or days.
The difference between a simple agent loop and durable execution
A basic agent loop might look like this:
while not completed:
action = agent.decide(state)
result = execute(action)
state = update(state, result)
This is a useful abstraction, but if the process crashes, the in-memory state disappears.
A durable workflow needs additional infrastructure:
Workflow Engine
|
v
Persisted Execution History
|
v
Schedule Work
|
v
Execute Activity
|
v
Persist Result
|
v
Continue Workflow
When a worker becomes unavailable, another worker can continue the workflow using its persisted history and state.
Activities and workflow logic
Durable workflow systems often separate orchestration from external operations.
The workflow defines the sequence of activities.
Activities perform operations such as:
- Calling external APIs.
- Reading databases.
- Sending notifications.
- Processing documents.
This separation helps the system recover from individual failures without repeating the entire workflow.
Temporal is one example of this architecture.
Its documentation emphasizes that workflow executions and activities have different retry behavior. Activities are generally retried according to their policies, while whole workflows are not automatically retried by default.
This is an important design consideration: retry the failed unit of work rather than restarting an entire business process unnecessarily.
Source: Temporal — Retry Policies
8. Failure recovery: design what happens after something goes wrong
A reliable agent needs explicit recovery procedures.
The system should not simply catch every exception and ask the model to try again.
Different failure scenarios require different responses.
Scenario A: A temporary API failure
The agent requests inventory information, but the inventory service returns an HTTP 503 response.
A sensible recovery process:
- Record the failure.
- Determine whether it is temporary.
- Apply a limited retry policy.
- Respect the service's retry instructions.
- Continue if the request succeeds.
- Escalate or return a controlled failure if the limit is reached.
Scenario B: A worker crashes
An agent is processing a document when its execution process stops.
Recovery process:
- Detect the interrupted workflow.
- Retrieve the latest valid checkpoint.
- Restore workflow state.
- Identify unfinished activities.
- Resume from the appropriate execution boundary.
- Verify that previously completed side effects are not repeated.
Scenario C: An operation succeeds but the response is lost
An agent creates an order, but the network connection fails before the application receives confirmation.
Recovery process:
- Mark the operation as having an unknown outcome.
- Query the order service using a stable operation identifier.
- Check whether the order already exists.
- Retrieve the existing result if successful.
- Retry only if the operation is confirmed not to have happened and the API contract permits it.
This is one of the most important cases to design for.
Scenario D: The model makes a logical mistake
An agent selects the wrong tool or generates invalid arguments.
This is not necessarily a transient infrastructure failure.
Repeatedly retrying the same request may simply reproduce the same mistake.
Instead:
- Capture the model decision and tool arguments.
- Validate the arguments against the tool schema.
- Return a structured error if necessary.
- Allow a bounded correction or clarification step.
- Add the failure to the evaluation dataset.
Scenario E: A workflow requires human approval
An agent prepares a purchase order but is not authorized to submit it without approval.
The system should:
- Save the current workflow state.
- Mark the workflow as waiting for approval.
- Notify the authorized person.
- Receive the approval decision.
- Validate that the approval is still applicable.
- Resume execution.
The workflow should not remain dependent on an active model call while waiting for a human.
Recovery decision matrix
| Situation | Recommended strategy |
|---|---|
| Temporary service outage | Bounded retry with backoff |
| Rate limiting | Respect Retry-After |
| Invalid tool input | Correct or reject the input |
| Unknown write outcome | Query operation status or use idempotency |
| Worker crash | Restore checkpoint and resume |
| Repeated permanent failure | Stop and escalate |
| Unauthorized operation | Reject and enforce policy |
| Long-running approval | Persist state and wait |
| Repeated logical error | Diagnose and update evaluations |
9. Advanced reliability patterns
Beyond retries and checkpoints, several established distributed-systems patterns can make agent workflows more robust.
Circuit breakers
A circuit breaker prevents an application from repeatedly calling a service that is consistently failing.
It generally has three states:
Closed: Requests are allowed normally.
Open: Requests are blocked temporarily because failures have exceeded a threshold.
Half-open: A limited number of test requests are allowed to determine whether the service has recovered.
For example, if the shipping API repeatedly returns server errors, the agent application can temporarily stop calling it rather than consuming resources on requests that are unlikely to succeed.
This is especially useful when several agents share the same external dependency.
Deadlines and time budgets
Every operation should have an appropriate timeout.
But individual timeouts are not enough.
A workflow should also have an overall deadline.
For example:
- Order lookup: 3 seconds.
- Inventory check: 5 seconds.
- Payment authorization: 8 seconds.
- Entire workflow: 25 seconds.
These are illustrative budgets.
The application should account for time already spent before starting another operation. Otherwise, a series of individually acceptable retries can make the overall workflow excessively slow.
Bulkheads
Bulkheads isolate failures between different parts of an application.
For example, a large number of document-processing tasks should not consume all available worker capacity and prevent payment-related workflows from executing.
Separate queues, worker pools, or concurrency limits can reduce this risk.
Saga pattern and compensating actions
Some business processes involve several operations that cannot be wrapped in one database transaction.
Imagine:
- Reserve inventory.
- Authorize payment.
- Create a shipping request.
If the shipping request fails after payment authorization, the system may need to release the inventory reservation or cancel the payment authorization.
The Saga pattern handles this by defining compensating actions for previously completed steps.
A simplified example:
| Forward operation | Possible compensation |
|---|---|
| Reserve inventory | Release reservation |
| Authorize payment | Void authorization |
| Create shipment | Cancel shipment, if supported |
| Send notification | Send a correction notification, if appropriate |
Not every operation can be perfectly reversed. Sending an email, for example, cannot be undone in the same way as releasing inventory.
Compensation is therefore a business-level recovery strategy, not a universal rollback mechanism.
Source: Temporal — Error Handling and Saga Pattern
10. Observability: understand why an agent failed
Reliability mechanisms are only useful when developers can understand what happened during execution.
A final error message such as "The task failed" provides very little diagnostic information.
A production system should record the execution path.
What should be traced?
For each workflow, useful information includes:
- Workflow and task identifiers.
- Model and configuration versions.
- Model requests and responses, subject to privacy controls.
- Tool names and execution status.
- Input and output references.
- Retry attempts.
- Checkpoint identifiers.
- Error categories.
- Execution duration.
- Token consumption and cost.
- Human approval decisions.
Trace the complete execution
Consider the following execution trace:
Workflow: order_processing_1042
10:00:00 Customer verification SUCCESS
10:00:02 Order validation SUCCESS
10:00:05 Inventory API TIMEOUT
10:00:06 Retry scheduled SUCCESS
10:00:09 Inventory API SUCCESS
10:00:11 Payment authorization UNKNOWN OUTCOME
10:00:12 Payment status lookup SUCCESS
10:00:13 Existing authorization found SUCCESS
10:00:15 Shipping request SUCCESS
10:00:16 Workflow COMPLETED
This example illustrates why execution traces should capture more than the final answer.
Without the payment-status lookup, developers might not understand how the application avoided a duplicate operation.
OpenAI's tracing documentation describes how traces can record model generations, tool calls, delegated agents, timing, and execution status. Microsoft Agent Framework and LangGraph also provide workflow checkpoint and state-inspection capabilities.
Sources: OpenAI — Tracing · Microsoft Agent Framework — Checkpoints · LangGraph — Persistence
Observability data should be protected. Avoid storing credentials, access tokens, or unnecessary sensitive information in traces and checkpoints.
11. A practical reliability architecture
The following architecture combines the concepts discussed in this article.
User Request
|
v
Agent Runtime
|
v
Workflow Engine
|
+---------+---------+
| |
v v
State Store Policy Engine
| |
v v
Checkpoints Authorization
| |
+---------+---------+
|
v
Tool Gateway
|
+---------+---------+
| | |
v v v
Database External Business
APIs Services
|
v
Result Validation
|
v
State Persistence
|
v
Workflow Progress
|
v
Final Result
Observability monitors all major components.
Each component has a distinct responsibility:
| Component | Responsibility |
|---|---|
| Agent runtime | Interprets tasks and proposes actions |
| Workflow engine | Controls execution and recovery |
| State store | Maintains durable workflow progress |
| Policy engine | Enforces access and business rules |
| Tool gateway | Controls external operations |
| Retry mechanism | Handles eligible temporary failures |
| Idempotency layer | Prevents duplicate business effects |
| Observability | Records execution and failures |
This separation is important because a model should not be responsible for maintaining the application's entire reliability architecture.
12. Choosing the right reliability tools
Different frameworks solve different parts of the problem.
| Technology | Main strength | Suitable use |
|---|---|---|
| LangGraph | Graph state persistence and checkpointing | Stateful agent workflows |
| Microsoft Agent Framework | Workflow orchestration and checkpoints | Applications using Microsoft's agent ecosystem |
| Temporal | Durable execution and activity recovery | Long-running business processes |
| OpenAI Agents SDK | Agent execution and built-in tracing | Agent applications using OpenAI's SDK |
| PostgreSQL | Durable application and workflow state | Custom state-management implementations |
These technologies should not be treated as direct substitutes.
For example, PostgreSQL can store workflow state, but it does not automatically provide a complete durable execution engine. Similarly, an agent framework's checkpointing feature does not automatically make every external API operation safe to repeat.
A developer may combine several technologies depending on the application's requirements.
A practical selection guide
For a simple agent:
- Explicit state management.
- Input validation.
- Tool-level timeouts.
- Bounded retries.
- Structured logging.
For a stateful agent application:
- Persistent state storage.
- Checkpointing.
- Execution tracing.
- Idempotent external operations.
- Human approval support.
For long-running business automation:
- Durable workflow orchestration.
- Persistent execution history.
- Activity-level retries.
- Idempotency keys.
- Recovery and compensation strategies.
- Monitoring and alerting.
The goal is not to introduce the largest possible infrastructure stack. It is to select the minimum architecture that satisfies the application's reliability requirements.
13. A production-readiness checklist
Before deploying an AI agent that performs meaningful operations, engineering teams should verify the following:
State management
- Workflow state is represented explicitly.
- Conversation history and execution state are separated.
- State survives application restarts where required.
- Checkpoints are stored durably.
- State transitions are validated.
- Checkpoint retention and access policies are defined.
Retry and error handling
- Errors are classified before retrying.
- Transient failures use bounded retries.
- Exponential backoff and jitter are considered.
- Service retry instructions are respected.
- Individual operations have timeouts.
- Workflows have overall deadlines.
- Permanent failures are escalated instead of repeated.
External operations
- Write operations have appropriate idempotency controls.
- Unknown operation outcomes can be investigated.
- Duplicate requests are handled safely.
- Destructive operations have explicit authorization.
- Compensation procedures exist where appropriate.
Recovery and observability
- Interrupted workflows can resume.
- Previously completed operations are not blindly repeated.
- Execution traces capture important events.
- Failures can be reproduced.
- Recovery behavior is tested.
- Sensitive information is protected in logs and checkpoints.
Conclusion: reliability is designed, not prompted
The reliability of an AI agent cannot be measured only by how often it produces a correct answer.
It must also be measured by how it behaves when something goes wrong.
A production agent needs to preserve execution state, recover from interruptions, retry temporary failures safely, avoid duplicate side effects, and provide enough observability for engineers to understand its behavior.
These are not uniquely AI problems. They are established distributed-systems challenges that become especially important when combined with dynamic model-driven execution.
The most useful architectural principle is simple:
Never make the language model responsible for remembering whether the application has safely completed an important operation.
Use structured state to track progress, checkpoints to preserve it, idempotency to protect external actions, retries to handle temporary failures, and durable execution when workflows must survive process interruptions.
An agent that succeeds once is a demonstration. An agent that can recover safely and consistently is an engineered system.
Sources and further reading
Reliability and distributed systems
- AWS Builders' Library — Timeouts, Retries, and Backoff with Jitter
- Temporal — Activity Definition and Idempotency
- Temporal — Retry Policies
- Temporal — Error Handling and Saga Pattern
Agent frameworks and persistence
Agent operations and observability
Technical note: Framework APIs and capabilities evolve. The examples in this article illustrate architectural principles; consult the linked official documentation for current implementation details.