Back to articles
Table of Contents Tap to expand

Building Reliable AI Agents: Retries, Checkpoints, and Failure Recovery

D
Dave Dotio Content Editor & AI Advocate

Quick Summary

Extractable

An AI agent is not production-ready simply because it can complete a task once. It must also handle interrupted execution, failed APIs, duplicate requests, invalid tool responses, and unexpected model behavior without losing control of the workflow. This article focuses on the reliability engineering behind AI agents: how to preserve execution state, retry safely, recover from failures, make operations idempotent, and design workflows that can resume instead of starting over.

Category
AI Agents & Automation
Published
Nov 20, 2026
Tags
None
Decision support

Turn this guide into a shortlist decision.

TipJournal articles should lead back into product evaluation. Use the recommended compare pages or jump into a custom comparison from here.

Browse compare hub
Building Reliable AI Agents: Retries, Checkpoints, and Failure Recovery

Building Reliable AI Agents: Retries, State Management, Checkpoints, and Failure Recovery

An AI agent successfully completes a task during testing. It retrieves information, calls the right APIs, processes the results, and produces the expected answer.

Then, in production, the same agent encounters a temporary API outage, loses its connection halfway through execution, or receives an unexpected response from an external service.

What happens next?

Does it start again and repeat every action? Does it remember which operations have already succeeded? Could it accidentally send the same payment request twice? Can it recover from an interrupted workflow without asking the user to repeat everything?

These questions expose one of the most important challenges in agent engineering: reliability.

An AI agent is not production-ready simply because it can complete a task once. It must also handle interruptions, recover from failures, preserve important execution state, and avoid repeating operations that have already produced side effects.

This requires more than better prompts or a more capable model. It requires established software engineering principles such as timeouts, retries, idempotency, persistent state, checkpoints, and fault tolerance.

This article explains how these mechanisms work together and how developers can use them to build AI agents that remain reliable beyond controlled demonstrations.

1. Why AI agents need reliability engineering

Traditional applications usually execute predefined instructions. Developers know which operations should happen and can often predict the execution path.

AI agents introduce a degree of uncertainty.

An agent may dynamically select tools, interpret external responses, decide whether to retry an operation, or delegate work to another agent.

Consider an AI agent responsible for processing customer orders:

  1. Retrieve the customer's information.
  2. Verify the order.
  3. Check inventory.
  4. Request payment authorization.
  5. Create a shipping request.
  6. Notify the customer.

This workflow depends on several independent components:

  • The language model.
  • The customer database.
  • The inventory service.
  • The payment provider.
  • The shipping API.
  • The notification service.

A failure in any one of these components can interrupt the entire task.

Worse, a failure does not always mean that an operation did not happen.

Imagine the agent sends a payment request. The payment provider processes it successfully, but the response never reaches the application because the network connection times out.

From the application's perspective, the result is unknown.

If the agent simply repeats the request, it could charge the customer twice.

This is the fundamental problem that reliability engineering must address: the application needs to distinguish between an operation that failed, an operation that succeeded, and an operation whose outcome is unknown.

Common failure categories

Failure category Example Required response
Model failure Invalid or incomplete tool arguments Validate, correct, or request clarification
Tool failure API returns an error Classify the error and apply an appropriate policy
Network failure Connection timeout Check the operation's outcome before retrying
Execution failure Worker crashes Restore state and resume safely
State failure Corrupted or missing checkpoint Detect the inconsistency and recover or escalate
Logical failure Agent selects the wrong action Diagnose and improve the workflow or evaluation
Authorization failure Agent requests a forbidden operation Reject the action; do not retry without a permission change

These failures require different solutions. Treating every failure as a reason to restart the agent is a common source of duplicate operations, unnecessary costs, and unpredictable behavior.

2. Understand the difference between conversation state and execution state

Before discussing recovery, it is important to understand what an agent needs to remember.

Developers often use the word memory to describe several different things. However, retaining a conversation is not the same as preserving an unfinished workflow.

Consider a user asking an agent:

"Find the three cheapest suppliers, compare their delivery times, and prepare a purchase recommendation."

The agent may need to preserve:

  • The original request.
  • The suppliers already discovered.
  • The results returned by each API.
  • The supplier currently being evaluated.
  • The remaining tasks.
  • The final recommendation status.

If the application only stores conversation messages, it may know what the user requested but not exactly which operations were completed.

A reliable system distinguishes between several kinds of state.

Conversation state

Contains the interaction history between the user and the agent.

Examples:

  • User messages.
  • Agent responses.
  • Relevant previous exchanges.
  • Conversation identifiers.

Workflow state

Represents the current progress of a task.

Examples:

  • Current execution step.
  • Completed operations.
  • Pending operations.
  • Intermediate results.
  • Error status.
  • Retry count.

Application state

Contains information maintained by the surrounding application.

Examples:

  • Customer records.
  • Orders.
  • Account permissions.
  • Payment records.
  • Business rules.

Long-term memory

Contains information that may be useful across separate tasks or conversations.

Examples:

  • User preferences.
  • Previously established facts.
  • Persistent knowledge.

These categories should not be confused.

An agent can remember an entire conversation and still be unable to resume an interrupted payment workflow safely.

The application needs an explicit representation of execution progress.

3. State management: make execution progress explicit

A reliable agent should not depend entirely on the language model remembering what it has already done.

Instead, represent the workflow as structured application state.

For example, a simplified order-processing workflow might maintain:

{
  "task_id": "task_1042",
  "status": "in_progress",
  "current_step": "inventory_check",
  "steps": {
    "customer_verification": "completed",
    "order_validation": "completed",
    "inventory_check": "in_progress",
    "payment": "pending",
    "shipping": "pending"
  },
  "retry_count": 0,
  "last_error": null
}

This is an illustrative structure. A real implementation should also define schema versions, timestamps, ownership, and validation rules.

The important idea is that execution progress is represented explicitly rather than inferred from a natural-language conversation.

Why structured state matters

Suppose the worker crashes after verifying the customer and validating the order.

When the application restarts, it can inspect the saved workflow state and determine that the first two steps have already completed.

It can then continue from inventory verification instead of repeating the entire workflow.

A good state model should answer:

  1. What task is being executed?
  2. What is its current status?
  3. Which operations have completed?
  4. Which operations remain?
  5. What information is required to continue?
  6. Has an operation produced an external side effect?
  7. What should happen if execution cannot continue?

State transitions

It is useful to define valid transitions explicitly.

For example:

PENDING
   |
   v
RUNNING
   |
   +------> COMPLETED
   |
   +------> WAITING_FOR_APPROVAL
   |
   +------> RETRY_SCHEDULED
   |
   +------> FAILED

The application should control these transitions.

A model may suggest that a task is complete, but the application should verify that required operations actually succeeded before changing the workflow status to COMPLETED.

Engineering principle: The model generates decisions; the application owns authoritative execution state.

4. Retry strategies: recover from temporary failures without creating new problems

Retries are one of the simplest ways to improve application reliability.

However, retrying every failed operation is not a safe strategy.

Some failures are temporary. Others indicate a permanent problem that will not disappear after waiting.

Classify errors before retrying

A useful distinction is between transient and permanent failures.

Error Typical treatment
Network timeout Retry cautiously after checking the outcome
HTTP 429 rate limit Respect Retry-After and apply backoff
HTTP 500 server error Often retryable, depending on the operation
HTTP 503 unavailable Usually retryable with limits
HTTP 400 invalid input Correct the input; do not repeat unchanged
HTTP 401 unauthorized Fix authentication or credentials
HTTP 403 forbidden Reject or request an authorized permission change
Invalid tool arguments Validate and correct the arguments
Business rule violation Stop and report the reason

These are general guidelines, not universal rules. Actual retry behavior depends on the API's contract and error semantics.

OpenAI's Agents API recovery guidance, for example, recommends checking session and turn status, inspecting completed actions, limiting retries, and distinguishing transient failures from invalid input or credential problems.

Source: OpenAI — Errors and Recovery

Exponential backoff

A common retry strategy is exponential backoff.

Instead of retrying immediately, the application increases the waiting time between attempts.

A simplified formula is:

[ d_n = \min(d_{\max}, d_0 \times 2^n) ]

Where:

  • (d_n) is the delay before retry (n).
  • (d_0) is the initial delay.
  • (d_{\max}) is the maximum delay.

For example, using an initial delay of one second:

Attempt Approximate delay
First retry 1 second
Second retry 2 seconds
Third retry 4 seconds
Fourth retry 8 seconds

This reduces the pressure on an overloaded service.

Add jitter

If thousands of agents fail at the same time and all retry after exactly four seconds, they may overwhelm the same service again.

Jitter introduces a random variation into retry delays, spreading requests over time.

A simplified full-jitter strategy is:

[ d_n = U(0,\min(d_{\max},d_0 \times 2^n)) ]

Here, (U(0,x)) means a random delay between zero and (x).

This technique is discussed in Amazon's engineering guidance on timeouts, retries, and backoff.

Source: AWS Builders' Library — Timeouts, Retries, and Backoff with Jitter

Implement retry limits

Retries should always have limits.

Possible limits include:

  • Maximum number of attempts.
  • Maximum total elapsed time.
  • Maximum execution cost.
  • Maximum number of tool calls.
  • Overall task deadline.

For example, an inventory API might allow three attempts with a total timeout budget of 15 seconds.

If the service remains unavailable, the agent should stop and report that it cannot verify inventory.

It should not continue calling the same endpoint indefinitely.

Example retry logic

The following Python example demonstrates the general pattern:

import random
import time

def call_with_retry(operation, max_attempts=3):
    base_delay = 1
    max_delay = 8

    for attempt in range(max_attempts):
        try:
            return operation()

        except TemporaryServiceError:
            if attempt == max_attempts - 1:
                raise

            delay = min(
                max_delay,
                base_delay * (2 ** attempt)
            )

            time.sleep(random.uniform(0, delay))

        except PermanentServiceError:
            raise

TemporaryServiceError and PermanentServiceError are illustrative application-defined exceptions.

In a production implementation, also consider request deadlines, cancellation, Retry-After, logging, and whether the operation is safe to repeat.

Important: Retry logic should generally live at a clearly defined layer. Multiple independent layers retrying the same request can multiply attempts unexpectedly.

5. Idempotency: prevent duplicate actions

Retries become particularly dangerous when tools modify external systems.

Consider an agent that creates an order through an API.

The request reaches the server and creates the order successfully. However, the response is lost.

The agent cannot determine whether the order was created.

If it sends the same request again, the server might create a second order.

This is where idempotency becomes essential.

What is idempotency?

An operation is idempotent when repeating the same operation produces the same intended effect as executing it once.

For example:

  • Reading an order is normally safe to repeat.
  • Setting an order status to confirmed can be designed to be idempotent.
  • Creating a new order is not automatically idempotent.
  • Charging a payment is not automatically idempotent.

For non-idempotent operations, an application can use an idempotency key.

How idempotency keys work

Before sending a sensitive request, the application generates a unique key.

For example:

idempotency_key = task_1042_payment_01

The payment service stores the result associated with that key.

If the same request arrives again with the same key, the service can return the existing result instead of processing a second payment.

Conceptually:

Agent requests payment
        |
        v
Payment API receives request
        |
        v
Check idempotency key
        |
    +---+---+
    |       |
 Already   New
 processed  request
    |       |
Return     Process
existing   payment
result      |
    |       v
    |     Store result
    |       |
    +---+---+
        |
        v
     Response

Important implementation details

An idempotency system should:

  1. Generate a stable key for the logical operation.
  2. Reuse that key when retrying the same operation.
  3. Associate the key with the intended request parameters.
  4. Persist the operation's result and status.
  5. Reject reuse of the same key with conflicting parameters.
  6. Define how long idempotency records remain valid.

A new logical operation should receive a new key.

For example, retrying a payment request should reuse the original key, while charging a different order should use another key.

Idempotency does not guarantee exactly-once execution

This distinction is important.

In distributed systems, an operation may execute more than once even if the application ultimately exposes only one logical result.

A system can use deduplication and idempotency to prevent duplicate business effects, but it should not assume that network delivery or internal execution happens exactly once.

Temporal's documentation explicitly warns that activities may execute more than once during retries and recommends designing them to be idempotent.

Sources: Temporal — Activity Definition · Temporal — Error Handling and Idempotence

6. Checkpointing: save progress so the agent can resume

Retries handle temporary failures of individual operations.

But what happens when an entire workflow is interrupted?

Imagine an agent conducting a 30-minute research task involving multiple websites, document extraction, data analysis, and report generation.

The application server crashes after 20 minutes.

Without persistent execution state, the application may need to repeat the entire task.

Checkpointing provides a better solution.

What is a checkpoint?

A checkpoint is a saved snapshot of workflow state at a particular point during execution.

It can include:

  • Current workflow position.
  • Completed task outputs.
  • Pending operations.
  • Relevant agent context.
  • Intermediate results.
  • State needed to resume execution.

For example:

Task started
    |
    v
Search documents
    |
    v
Extract information
    |
    v
Analyze results
    |
    X  Worker crashes
    |
    v
Restore latest checkpoint
    |
    v
Continue unfinished work
    |
    v
Generate report

Instead of restarting from the beginning, the system restores the latest valid state.

Checkpoints versus ordinary conversation history

Conversation history stores messages.

A workflow checkpoint stores execution progress.

For example, a conversation might contain:

"Searching the first supplier's catalog now."

But a useful workflow checkpoint should contain structured information such as:

  • Supplier A: completed.
  • Supplier B: completed.
  • Supplier C: pending.
  • Extracted prices: saved.
  • Next operation: compare delivery times.

The second representation is much more useful for recovery.

When should checkpoints be created?

There is no single checkpoint frequency suitable for every application.

A reasonable strategy is to save state:

  • After completing an important workflow stage.
  • Before an expensive or irreversible operation.
  • After receiving a significant external result.
  • Before waiting for human approval.
  • At boundaries between long-running tasks.

More frequent checkpoints can reduce lost work but increase storage and processing overhead.

The appropriate balance depends on task duration, recovery requirements, and the cost of repeating completed operations.

Checkpoint storage

Checkpoint data must survive the failure it is intended to recover from.

An in-memory checkpoint disappears when the process or machine is restarted.

For production workloads, developers generally need durable storage such as a database or persistent workflow service.

Examples include:

  • PostgreSQL.
  • Azure Cosmos DB.
  • Other durable database systems.
  • Dedicated workflow orchestration platforms.

The storage mechanism should support the required durability, concurrency, and retention characteristics.

Framework implementations

Several agent frameworks provide checkpointing or persistence capabilities.

LangGraph

LangGraph uses checkpointers to save graph state across execution steps. Its persistence system supports conversation continuity, human-in-the-loop workflows, replay, and fault tolerance.

It also distinguishes between:

  • Checkpointers for short-term, thread-specific execution state.
  • Stores for longer-term application-defined data shared across threads.

The documentation describes recovery from the latest successful checkpoint and preservation of successful task writes when another task fails within the same execution step.

Documentation: LangGraph — Persistence

Microsoft Agent Framework

Microsoft Agent Framework supports workflow checkpointing, allowing developers to persist workflow state and resume execution.

Its documented storage implementations include:

  • InMemoryCheckpointStorage for tests and short-lived workflows.
  • FileCheckpointStorage for local or single-machine use.
  • CosmosCheckpointStorage for distributed, durable scenarios.

Custom executors must also save and restore their own state appropriately.

Documentation: Microsoft Agent Framework — Checkpoints

Temporal

Temporal approaches the problem through durable workflow execution.

Instead of relying on a model to reconstruct its progress, the workflow engine records execution history and coordinates activities, retries, and recovery.

Documentation: Temporal — Documentation

These solutions address related problems but are not interchangeable. Their persistence models, execution semantics, operational requirements, and integration patterns differ.

7. Durable execution: recover without relying on a running process

Checkpointing saves progress. Durable execution takes the idea further.

A durable execution system is designed to preserve workflow progress even when the application process or worker responsible for execution disappears.

This is particularly useful for:

  • Long-running research agents.
  • Financial processing workflows.
  • Document-processing pipelines.
  • Multi-step enterprise automation.
  • Workflows requiring human approval.
  • Tasks that may remain active for hours or days.

The difference between a simple agent loop and durable execution

A basic agent loop might look like this:

while not completed:
    action = agent.decide(state)
    result = execute(action)
    state = update(state, result)

This is a useful abstraction, but if the process crashes, the in-memory state disappears.

A durable workflow needs additional infrastructure:

Workflow Engine
      |
      v
Persisted Execution History
      |
      v
Schedule Work
      |
      v
Execute Activity
      |
      v
Persist Result
      |
      v
Continue Workflow

When a worker becomes unavailable, another worker can continue the workflow using its persisted history and state.

Activities and workflow logic

Durable workflow systems often separate orchestration from external operations.

The workflow defines the sequence of activities.

Activities perform operations such as:

  • Calling external APIs.
  • Reading databases.
  • Sending notifications.
  • Processing documents.

This separation helps the system recover from individual failures without repeating the entire workflow.

Temporal is one example of this architecture.

Its documentation emphasizes that workflow executions and activities have different retry behavior. Activities are generally retried according to their policies, while whole workflows are not automatically retried by default.

This is an important design consideration: retry the failed unit of work rather than restarting an entire business process unnecessarily.

Source: Temporal — Retry Policies

8. Failure recovery: design what happens after something goes wrong

A reliable agent needs explicit recovery procedures.

The system should not simply catch every exception and ask the model to try again.

Different failure scenarios require different responses.

Scenario A: A temporary API failure

The agent requests inventory information, but the inventory service returns an HTTP 503 response.

A sensible recovery process:

  1. Record the failure.
  2. Determine whether it is temporary.
  3. Apply a limited retry policy.
  4. Respect the service's retry instructions.
  5. Continue if the request succeeds.
  6. Escalate or return a controlled failure if the limit is reached.

Scenario B: A worker crashes

An agent is processing a document when its execution process stops.

Recovery process:

  1. Detect the interrupted workflow.
  2. Retrieve the latest valid checkpoint.
  3. Restore workflow state.
  4. Identify unfinished activities.
  5. Resume from the appropriate execution boundary.
  6. Verify that previously completed side effects are not repeated.

Scenario C: An operation succeeds but the response is lost

An agent creates an order, but the network connection fails before the application receives confirmation.

Recovery process:

  1. Mark the operation as having an unknown outcome.
  2. Query the order service using a stable operation identifier.
  3. Check whether the order already exists.
  4. Retrieve the existing result if successful.
  5. Retry only if the operation is confirmed not to have happened and the API contract permits it.

This is one of the most important cases to design for.

Scenario D: The model makes a logical mistake

An agent selects the wrong tool or generates invalid arguments.

This is not necessarily a transient infrastructure failure.

Repeatedly retrying the same request may simply reproduce the same mistake.

Instead:

  1. Capture the model decision and tool arguments.
  2. Validate the arguments against the tool schema.
  3. Return a structured error if necessary.
  4. Allow a bounded correction or clarification step.
  5. Add the failure to the evaluation dataset.

Scenario E: A workflow requires human approval

An agent prepares a purchase order but is not authorized to submit it without approval.

The system should:

  1. Save the current workflow state.
  2. Mark the workflow as waiting for approval.
  3. Notify the authorized person.
  4. Receive the approval decision.
  5. Validate that the approval is still applicable.
  6. Resume execution.

The workflow should not remain dependent on an active model call while waiting for a human.

Recovery decision matrix

Situation Recommended strategy
Temporary service outage Bounded retry with backoff
Rate limiting Respect Retry-After
Invalid tool input Correct or reject the input
Unknown write outcome Query operation status or use idempotency
Worker crash Restore checkpoint and resume
Repeated permanent failure Stop and escalate
Unauthorized operation Reject and enforce policy
Long-running approval Persist state and wait
Repeated logical error Diagnose and update evaluations

9. Advanced reliability patterns

Beyond retries and checkpoints, several established distributed-systems patterns can make agent workflows more robust.

Circuit breakers

A circuit breaker prevents an application from repeatedly calling a service that is consistently failing.

It generally has three states:

Closed: Requests are allowed normally.

Open: Requests are blocked temporarily because failures have exceeded a threshold.

Half-open: A limited number of test requests are allowed to determine whether the service has recovered.

For example, if the shipping API repeatedly returns server errors, the agent application can temporarily stop calling it rather than consuming resources on requests that are unlikely to succeed.

This is especially useful when several agents share the same external dependency.

Deadlines and time budgets

Every operation should have an appropriate timeout.

But individual timeouts are not enough.

A workflow should also have an overall deadline.

For example:

  • Order lookup: 3 seconds.
  • Inventory check: 5 seconds.
  • Payment authorization: 8 seconds.
  • Entire workflow: 25 seconds.

These are illustrative budgets.

The application should account for time already spent before starting another operation. Otherwise, a series of individually acceptable retries can make the overall workflow excessively slow.

Bulkheads

Bulkheads isolate failures between different parts of an application.

For example, a large number of document-processing tasks should not consume all available worker capacity and prevent payment-related workflows from executing.

Separate queues, worker pools, or concurrency limits can reduce this risk.

Saga pattern and compensating actions

Some business processes involve several operations that cannot be wrapped in one database transaction.

Imagine:

  1. Reserve inventory.
  2. Authorize payment.
  3. Create a shipping request.

If the shipping request fails after payment authorization, the system may need to release the inventory reservation or cancel the payment authorization.

The Saga pattern handles this by defining compensating actions for previously completed steps.

A simplified example:

Forward operation Possible compensation
Reserve inventory Release reservation
Authorize payment Void authorization
Create shipment Cancel shipment, if supported
Send notification Send a correction notification, if appropriate

Not every operation can be perfectly reversed. Sending an email, for example, cannot be undone in the same way as releasing inventory.

Compensation is therefore a business-level recovery strategy, not a universal rollback mechanism.

Source: Temporal — Error Handling and Saga Pattern

10. Observability: understand why an agent failed

Reliability mechanisms are only useful when developers can understand what happened during execution.

A final error message such as "The task failed" provides very little diagnostic information.

A production system should record the execution path.

What should be traced?

For each workflow, useful information includes:

  • Workflow and task identifiers.
  • Model and configuration versions.
  • Model requests and responses, subject to privacy controls.
  • Tool names and execution status.
  • Input and output references.
  • Retry attempts.
  • Checkpoint identifiers.
  • Error categories.
  • Execution duration.
  • Token consumption and cost.
  • Human approval decisions.

Trace the complete execution

Consider the following execution trace:

Workflow: order_processing_1042

10:00:00  Customer verification       SUCCESS
10:00:02  Order validation             SUCCESS
10:00:05  Inventory API                TIMEOUT
10:00:06  Retry scheduled              SUCCESS
10:00:09  Inventory API                SUCCESS
10:00:11  Payment authorization        UNKNOWN OUTCOME
10:00:12  Payment status lookup        SUCCESS
10:00:13  Existing authorization found SUCCESS
10:00:15  Shipping request             SUCCESS
10:00:16  Workflow                     COMPLETED

This example illustrates why execution traces should capture more than the final answer.

Without the payment-status lookup, developers might not understand how the application avoided a duplicate operation.

OpenAI's tracing documentation describes how traces can record model generations, tool calls, delegated agents, timing, and execution status. Microsoft Agent Framework and LangGraph also provide workflow checkpoint and state-inspection capabilities.

Sources: OpenAI — Tracing · Microsoft Agent Framework — Checkpoints · LangGraph — Persistence

Observability data should be protected. Avoid storing credentials, access tokens, or unnecessary sensitive information in traces and checkpoints.

11. A practical reliability architecture

The following architecture combines the concepts discussed in this article.

                  User Request
                       |
                       v
                Agent Runtime
                       |
                       v
                Workflow Engine
                       |
             +---------+---------+
             |                   |
             v                   v
        State Store         Policy Engine
             |                   |
             v                   v
         Checkpoints       Authorization
             |                   |
             +---------+---------+
                       |
                       v
                  Tool Gateway
                       |
             +---------+---------+
             |         |         |
             v         v         v
          Database   External   Business
                     APIs       Services
                       |
                       v
                Result Validation
                       |
                       v
                State Persistence
                       |
                       v
                Workflow Progress
                       |
                       v
                  Final Result

        Observability monitors all major components.

Each component has a distinct responsibility:

Component Responsibility
Agent runtime Interprets tasks and proposes actions
Workflow engine Controls execution and recovery
State store Maintains durable workflow progress
Policy engine Enforces access and business rules
Tool gateway Controls external operations
Retry mechanism Handles eligible temporary failures
Idempotency layer Prevents duplicate business effects
Observability Records execution and failures

This separation is important because a model should not be responsible for maintaining the application's entire reliability architecture.

12. Choosing the right reliability tools

Different frameworks solve different parts of the problem.

Technology Main strength Suitable use
LangGraph Graph state persistence and checkpointing Stateful agent workflows
Microsoft Agent Framework Workflow orchestration and checkpoints Applications using Microsoft's agent ecosystem
Temporal Durable execution and activity recovery Long-running business processes
OpenAI Agents SDK Agent execution and built-in tracing Agent applications using OpenAI's SDK
PostgreSQL Durable application and workflow state Custom state-management implementations

These technologies should not be treated as direct substitutes.

For example, PostgreSQL can store workflow state, but it does not automatically provide a complete durable execution engine. Similarly, an agent framework's checkpointing feature does not automatically make every external API operation safe to repeat.

A developer may combine several technologies depending on the application's requirements.

A practical selection guide

For a simple agent:

  • Explicit state management.
  • Input validation.
  • Tool-level timeouts.
  • Bounded retries.
  • Structured logging.

For a stateful agent application:

  • Persistent state storage.
  • Checkpointing.
  • Execution tracing.
  • Idempotent external operations.
  • Human approval support.

For long-running business automation:

  • Durable workflow orchestration.
  • Persistent execution history.
  • Activity-level retries.
  • Idempotency keys.
  • Recovery and compensation strategies.
  • Monitoring and alerting.

The goal is not to introduce the largest possible infrastructure stack. It is to select the minimum architecture that satisfies the application's reliability requirements.

13. A production-readiness checklist

Before deploying an AI agent that performs meaningful operations, engineering teams should verify the following:

State management

  • Workflow state is represented explicitly.
  • Conversation history and execution state are separated.
  • State survives application restarts where required.
  • Checkpoints are stored durably.
  • State transitions are validated.
  • Checkpoint retention and access policies are defined.

Retry and error handling

  • Errors are classified before retrying.
  • Transient failures use bounded retries.
  • Exponential backoff and jitter are considered.
  • Service retry instructions are respected.
  • Individual operations have timeouts.
  • Workflows have overall deadlines.
  • Permanent failures are escalated instead of repeated.

External operations

  • Write operations have appropriate idempotency controls.
  • Unknown operation outcomes can be investigated.
  • Duplicate requests are handled safely.
  • Destructive operations have explicit authorization.
  • Compensation procedures exist where appropriate.

Recovery and observability

  • Interrupted workflows can resume.
  • Previously completed operations are not blindly repeated.
  • Execution traces capture important events.
  • Failures can be reproduced.
  • Recovery behavior is tested.
  • Sensitive information is protected in logs and checkpoints.

Conclusion: reliability is designed, not prompted

The reliability of an AI agent cannot be measured only by how often it produces a correct answer.

It must also be measured by how it behaves when something goes wrong.

A production agent needs to preserve execution state, recover from interruptions, retry temporary failures safely, avoid duplicate side effects, and provide enough observability for engineers to understand its behavior.

These are not uniquely AI problems. They are established distributed-systems challenges that become especially important when combined with dynamic model-driven execution.

The most useful architectural principle is simple:

Never make the language model responsible for remembering whether the application has safely completed an important operation.

Use structured state to track progress, checkpoints to preserve it, idempotency to protect external actions, retries to handle temporary failures, and durable execution when workflows must survive process interruptions.

An agent that succeeds once is a demonstration. An agent that can recover safely and consistently is an engineered system.


Sources and further reading

Reliability and distributed systems

  1. AWS Builders' Library — Timeouts, Retries, and Backoff with Jitter
  2. Temporal — Activity Definition and Idempotency
  3. Temporal — Retry Policies
  4. Temporal — Error Handling and Saga Pattern

Agent frameworks and persistence

  1. LangGraph — Persistence
  2. Microsoft Agent Framework — Checkpoints
  3. Temporal — Documentation

Agent operations and observability

  1. OpenAI — Errors and Recovery
  2. OpenAI — Tracing
  3. OpenAI — Observability and Usage

Technical note: Framework APIs and capabilities evolve. The examples in this article illustrate architectural principles; consult the linked official documentation for current implementation details.

Related Reading

More articles with the same topic or audience.

Browse articles
AI Agents & Automation • Nov 22, 2026

AI Agent Security: Permissions, Tool Access, and Autonomous Actions

A practical security guide to controlling what AI agents can access, what actions they can perform, and when they must ask for human approval. The central threat scenario: a browser agent encounters a malicious webpage containing hidden instructions, follows them, and attempts to access private information or perform an unauthorized action. The article will cover least privilege, sandboxed execution, credential management, approval gates, user and task isolation, audit trails, and emergency stops.

AI Agents & Automation • Nov 21, 2026

The Data Layer Behind AI Agents: RAG, Memory, Databases, and Knowledge Graphs

An AI agent's intelligence is constrained by the information it can access, retrieve, understand, and retain. Connecting a model to a vector database is only one part of the problem. This article explains how to design the data architecture behind production agents: where information should live, how RAG retrieves it, when to use relational databases or knowledge graphs, how agent memory differs from a knowledge base, and how to keep retrieved information accurate, secure, and useful.

AI Agents & Automation • Nov 19, 2026

The AI Agent Development Lifecycle: From Prototype to Production

Building an AI agent is not a single coding task. It is an engineering lifecycle involving problem definition, architecture, development, evaluation, deployment, monitoring, and continuous improvement. The article will focus on the practical decisions developers need to make at each stage, including when to use a simple workflow instead of an autonomous agent, how to evaluate behavior before release, and how to manage failures after deployment.

AI Agents & Automation • Nov 18, 2026

AI Agent Identity and Authorization: A Practical Guide to Secure Agent Access

When an AI agent accesses an enterprise API, updates a database, or initiates a payment, the system needs to establish three things: which agent is acting, on whose behalf it is acting, and exactly what it is authorized to do. This is more complicated than giving an agent an API key or reusing a human employee's credentials. An important development for this topic is NIST's October 27, 2026 publication of ten proposed design principles for agent identity management. They cover distinct agent identities, delegation, least privilege, revocable authorization, human approval, and auditable evidence. These are proposed design principles, not a finalized technical standard.

Discussion (0)

Please sign in with Google to join the conversation.

No discussions yet. Be the first to comment!