AI Agent Memory vs Context Windows: How Agents Retain Information Across Long Tasks

An agent can process hundreds of thousands of tokens and still forget what it learned yesterday. Understanding the difference between context and memory is essential to building systems that can work over time.

A language model can analyze a long document, inspect a codebase, and reason across thousands of lines of information. Modern models also support increasingly large context windows.

Yet a simple problem remains: what happens when the conversation ends, the context window fills up, or the agent needs to resume a task tomorrow?

The model does not automatically retain everything it has previously processed.

This is why agent architectures increasingly distinguish between context windows, session history, working memory, and long-term memory.

These concepts are related, but they solve different problems. Understanding their roles is essential when designing agents that must preserve knowledge, resume work, and make consistent decisions across long-running tasks.

1. Context window: what the model can see right now

A context window is the amount of information a language model can process in a single inference request.

It is measured in tokens, which are units of text and other model input representations. A token may correspond to a word, part of a word, punctuation, or another encoded element.

The context window can contain:

• System instructions.
• User messages.
• Conversation history.
• Retrieved documents.
• Tool descriptions.
• Tool outputs.
• Intermediate reasoning-related inputs.
• The current task and relevant state.

All this information competes for the same available context capacity, together with the output budget defined by the model and API.

Consider a coding agent asked to fix a bug.

It might receive:

1. The developer's task description.
2. Instructions describing the repository.
3. Relevant source files.
4. Previous tool results.
5. Test output.
6. A summary of earlier work.

The model reasons using the information available in that request.

The context window is therefore the model's immediate workspace, not a permanent storage system.

A larger context window does not mean unlimited memory

Increasing context capacity allows a model to process more information at once. It does not automatically provide persistence between sessions.

A model with a very large context window can still lose continuity when:

• A conversation is restarted without restoring its history.
• A task is moved to a new session.
• Older information is removed during context compaction.
• A system fails to preserve important intermediate results.
• The model receives a new request without the relevant previous information.

There is another limitation: fitting information into a context window does not guarantee that the model will use every detail equally well.

Large inputs can contain irrelevant information, repeated tool outputs, conflicting instructions, and important details buried among less useful content.

The engineering challenge is not simply maximizing context size. It is deciding what information belongs in each model call.

2. Memory: what the agent can retain and retrieve later

Memory is the mechanism through which an agent preserves information beyond the immediate model call.

Unlike context, memory is not necessarily presented to the model all at once.

It can be stored externally, updated over time, and retrieved when relevant.

For example, a customer-support agent might need to remember:

• The customer's preferred communication language.
• A previously reported issue.
• The resolution attempted last week.
• A company-specific support procedure.
• The current status of an unresolved case.

Instead of including every historical interaction in every request, the system can store relevant information and retrieve it when needed.

This is a key architectural distinction:

Memory is not simply a larger context window. It is a separate information-management system surrounding the model.

3. The three layers: short-term, working, and long-term memory

A useful architecture separates memory into three complementary layers.

3.1. Short-term memory: what happened during the current session

Short-term memory preserves information from an ongoing interaction.

It can include:

• Recent conversation messages.
• Previous tool calls and results.
• Current task status.
• Temporary decisions.
• Intermediate findings.

For example, a research agent may spend 20 minutes comparing five products.

Its short-term memory can preserve which products have already been investigated, which sources have been checked, and which comparisons remain incomplete.

This avoids repeating work during the same session.

Short-term memory is usually session-scoped and constrained by retention policies and the context capacity of the model.

3.2. Working memory: what the model receives for the current step

Working memory is slightly different.

It is the information actually assembled and supplied to the model for one inference call.

It may combine:

• System instructions.
• Relevant parts of the current session.
• Retrieved long-term memories.
• Relevant documents.
• Current task state.
• Tool outputs needed for the next decision.

Working memory is not necessarily a permanent database. It is a temporary composition of information.

A useful analogy:

• Short-term memory is the collection of notes from the current project session.
• Long-term memory is the archive containing useful information from previous sessions.
• Working memory is the small set of documents currently open on the desk.

The model reasons over working memory. The other layers determine what information can be assembled for it.

3.3. Long-term memory: what survives between sessions

Long-term memory preserves useful information beyond the current interaction.

Examples include:

• Stable user preferences.
• Project architecture decisions.
• Previously completed tasks.
• Known constraints.
• Reusable procedures.
• Important events and outcomes.

Imagine a coding agent that works on the same application over several weeks.

It may need to retain:

• The project uses Laravel and Livewire.
• Database migrations follow a particular naming convention.
• The authentication system has custom middleware.
• A previous attempt to change a component caused a regression.
• A specific implementation decision was approved by the engineering team.

Without persistent memory or accessible project documentation, the agent may repeat investigations or propose incompatible changes.

Long-term memory provides continuity, but it must be carefully managed. Old, incorrect, or irrelevant information can be worse than no memory at all.

4. Different kinds of memory: facts, experiences, and procedures

Not all information should be stored in the same way.

A useful design distinguishes between semantic, episodic, and procedural memory.

Semantic memory: what the agent knows

Semantic memory stores facts and relationships.

Examples:

• The application uses PostgreSQL.
• The project has three user roles.
• A customer prefers email communication.
• The company operates in a particular market.

These entries describe information that may remain useful across many tasks.

Episodic memory: what happened

Episodic memory stores events, interactions, and outcomes.

Examples:

• A deployment failed on a particular date.
• A customer reported a billing issue.
• A developer approved a specific architecture proposal.
• An agent attempted a database migration and encountered an error.

The value of episodic memory is that it preserves historical context.

Knowing that a deployment failed is different from knowing the current deployment configuration. One is an event; the other is a fact about the system.

Procedural memory: how to perform a task

Procedural memory stores reusable instructions, workflows, and skills.

Examples:

• How to prepare a release.
• How to investigate a failed test.
• How to format a customer report.
• How to deploy a particular service.
• How to follow a company's code-review process.

This type of memory is especially useful when an agent repeatedly performs similar work.

These categories are useful design patterns rather than a mandatory classification for every agent. A production system may use additional categories for tasks, entities, relationships, or domain-specific knowledge.

5. How an agent memory system works

A memory system is not just a database where conversations are saved.

It needs a process for deciding what to remember, how to store it, and when to retrieve it.

A simplified architecture looks like this:

          User interaction
                 |
                 v
          Agent execution
                 |
                 v
       Identify useful information
                 |
                 v
        Memory extraction
                 |
                 v
       Validate and organize
                 |
                 v
        Persistent storage
                 |
          Later interaction
                 |
                 v
          Memory retrieval
                 |
                 v
       Relevance and permission checks
                 |
                 v
          Working memory
                 |
                 v
             Model call

This creates two important processes: writing memory and reading memory.

5.1. Writing memory: deciding what to preserve

At the end of an interaction, or during a task, the system can identify information worth retaining.

For example, consider this conversation:

We tried deploying the application using the previous configuration, but the deployment failed because the database migration ran before the required environment variable was available.

A memory extraction process might produce:

Event: Deployment failed.

Cause: Required environment variable was unavailable when the migration executed.

Scope: Project deployment process.

Potential reusable lesson: Validate required environment variables before running migrations.

The system should not automatically save every sentence. It needs to identify information that is durable, useful, and supported by the interaction.

5.2. Reading memory: retrieving what matters

When the agent receives a new task, it can search its memory store.

Suppose the next request is:

Prepare another deployment.

The memory system can retrieve the previous deployment failure and include the relevant lesson in the working context.

However, if the user asks for a general explanation of database indexing, that deployment event may be irrelevant.

Memory retrieval should therefore be based on the current task, not simply on whether a record exists.

5.3. Updating and correcting memory

Memory is not static.

Suppose a project changes from PostgreSQL to MySQL.

The previous database fact should no longer be treated as current.

A well-designed memory system needs to support:

• Adding new information.
• Updating existing facts.
• Resolving conflicting information.
• Tracking when information was valid.
• Expiring temporary information.
• Deleting information when required.

Without these mechanisms, long-term memory becomes an accumulation of potentially contradictory records.

6. Where does RAG fit into agent memory?

Retrieval-Augmented Generation (RAG) is often discussed alongside memory, but they are not identical.

RAG retrieves external information and places relevant content into the model's context.

For example, a company agent may need to answer a question using internal documentation.

A RAG pipeline can:

1. Search the company's knowledge base.
2. Retrieve relevant documents or passages.
3. Rank the results.
4. Add the selected information to the model's context.
5. Generate an answer grounded in the retrieved material.

Agent memory can use similar retrieval technologies, but it focuses on retaining information about interactions, preferences, events, decisions, or procedures over time.

The distinction is about purpose, not necessarily infrastructure.

A vector database can support both systems, but using a vector database does not automatically create a complete memory architecture.

The system still needs extraction, metadata, permissions, retrieval policies, updates, and evaluation.

7. The main techniques for managing long-running context

Even with external memory, an agent still needs to manage its active context window.

Several techniques help.

7.1. Trimming: removing low-value history

Trimming removes information that is no longer useful.

For example, an agent may discard repeated tool outputs after extracting the relevant result.

It may preserve:

• The current task.
• Important decisions.
• Relevant code changes.
• Unresolved issues.
• The latest test results.

It can remove:

• Repeated instructions.
• Redundant logs.
• Irrelevant intermediate outputs.
• Information already represented elsewhere.

Trimming is simple, but it must be careful not to remove information needed later.

7.2. Summarization and compaction

When a session becomes too long, the system can summarize earlier interactions into a smaller representation.

For example, a coding session might contain hundreds of messages and tool results.

A compacted summary could preserve:

Project: Customer management application

Task: Implement customer export.

Completed:
- Added export endpoint.
- Implemented CSV generation.
- Added authorization check.

Current state:
- Unit tests pass.
- Integration test fails for empty customer lists.

Important constraint:
- Existing API response format must remain unchanged.

Next step:
- Investigate the empty-list integration test.

The new context contains less information but preserves the details needed to continue.

Compaction is not lossless. A summary can omit an important detail or introduce an inaccurate interpretation. Critical decisions and facts should therefore be stored in structured form where possible, rather than relying exclusively on summaries.

OpenAI's Agents SDK cookbook documents trimming and compression techniques for managing short-term session memory. .

7.3. Retrieval on demand

Instead of including every historical detail in the prompt, the agent can search for information when it needs it.

For example:

• Retrieve the previous deployment incident when investigating a deployment.
• Retrieve the project architecture when modifying a major component.
• Retrieve a customer interaction when answering a support question.

This reduces unnecessary context and can improve cost and latency.

7.4. Hierarchical memory

A hierarchical system organizes information at different levels.

For example:

Organization memory
      |
      +---- Project memory
      |         |
      |         +---- Architecture decisions
      |         +---- Development conventions
      |
      +---- User memory
      |         |
      |         +---- Preferences
      |         +---- Interaction history
      |
      +---- Task memory
                |
                +---- Current progress
                +---- Intermediate results

The agent can retrieve information at the appropriate level instead of searching one large collection of unrelated memories.

This also helps enforce boundaries between different users, projects, and organizations.

8. Long-running agents: the problem of continuing work across sessions

Long-running tasks expose the difference between context and memory particularly clearly.

Imagine an agent assigned to refactor a large software repository.

The work may involve:

• Understanding the architecture.
• Identifying dependencies.
• Modifying several modules.
• Running tests.
• Fixing regressions.
• Updating documentation.
• Preparing a final report.

This work may exceed a single context window or execution session.

When the agent starts a new session, it needs a reliable way to understand what happened previously.

A useful approach is to maintain an explicit task state:

{
  "task": "Refactor authentication module",
  "status": "in_progress",
  "completed": [
    "Mapped authentication dependencies",
    "Updated token validation"
  ],
  "pending": [
    "Update integration tests",
    "Verify backward compatibility"
  ],
  "constraints": [
    "Do not change public API behavior"
  ],
  "last_verified": "Unit tests passed"
}

The next session can load this state, retrieve relevant files and decisions, and continue.

This is more reliable than asking the model to remember an entire conversation.

Anthropic's approach to long-running agents

Anthropic's November 2025 engineering article, Effective harnesses for long-running agents, describes the challenge of maintaining progress across multiple context windows.

It highlights the importance of structured handoffs between sessions, persistent project state, clear task boundaries, and mechanisms that allow new agent sessions to understand previous work.

The central lesson is that long-running agents need more than conversation history. They need a durable representation of project progress.

.

9. Choosing a memory architecture

There is no single storage technology that solves every memory problem.

Different types of information require different storage and retrieval strategies.

A practical system often combines multiple approaches.

For example, a customer-support agent might use:

• A relational database for customer identity and case status.
• A vector index for finding relevant past interactions.
• A document repository for company procedures.
• A structured memory record for customer preferences.
• A session store for current conversation state.

The goal is not to choose the most sophisticated database. It is to match the storage model to the information being managed.

10. The hidden problems: memory can make agents less reliable

Persistent memory sounds like an obvious improvement, but poorly designed memory can introduce new failure modes.

Stale information

An agent may retrieve an old fact that is no longer true.

A project may have changed its database, a customer may have updated their preferences, or a procedure may have been replaced.

Memory needs timestamps, update policies, and source information.

Incorrect memory extraction

A model may incorrectly interpret a conversation and store an inaccurate fact.

For example, a temporary suggestion could be remembered as an approved company policy.

Memory extraction should distinguish between confirmed decisions, tentative ideas, user statements, and inferred conclusions.

Irrelevant retrieval

A vector search can return semantically similar information that is not actually relevant.

Similarity is not the same as correctness.

Retrieval should consider metadata, recency, source reliability, access permissions, and the current task.

Privacy and cross-project leakage

An agent that remembers information from one customer or project must not silently expose it in another context.

Memory needs clear scope boundaries:

• User.
• Organization.
• Project.
• Agent.
• Session.
• Task.

Permissions should be enforced during retrieval, not merely during storage.

Memory poisoning

Untrusted input can introduce false instructions or misleading information into a persistent memory store.

For example, a malicious document could attempt to convince an agent to remember an unauthorized procedure.

Persistent memory therefore needs validation and provenance. The system should not treat every retrieved memory as a trusted instruction.

11. How to evaluate an agent's memory

A memory system should be tested like any other production component.

It is not enough to demonstrate that an agent can recall a preference in a short conversation.

A useful evaluation should examine several dimensions.

Benchmarks such as LongMemEval investigate long-term conversational memory, including the ability to answer questions about information distributed across extended interaction histories.

The important point is that memory quality is multidimensional. A system that remembers more is not necessarily a system that remembers better.

12. A practical architecture for production agents

For many applications, a hybrid architecture is a sensible starting point.

It combines a small amount of always-available context with structured state and selective retrieval.

                 User request
                      |
                      v
                Agent runtime
                      |
          +-----------+-----------+
          |                       |
          v                       v
    Session state           Memory retrieval
    Current task             Relevant facts
    Recent history           Past events
    Tool results             Procedures
          |                       |
          +-----------+-----------+
                      |
                      v
                Working context
                      |
                      v
                  LLM call
                      |
                      v
               Tool execution
                      |
                      v
                Task outcome
                      |
          +-----------+-----------+
          |                       |
          v                       v
     Update session         Extract memory
     and task state         and validate it
                                  |
                                  v
                           Persistent store

A good implementation can follow these principles:

1. Keep the active context focused.
2. Preserve task progress in structured state.
3. Store only information that has future value.
4. Retrieve long-term memories when relevant.
5. Separate facts, events, and reusable procedures.
6. Track sources, timestamps, and confidence.
7. Enforce permissions and memory scope.
8. Evaluate recall, updates, and forgetting.
9. Allow users or administrators to correct and delete stored information.

This architecture does not require a dedicated vector database from day one. A relational database, structured records, and targeted retrieval may be sufficient for an initial application.

Conclusion: the future of agents depends on managing information over time

A context window gives a model the information it can process now. Memory gives an agent a way to preserve useful information and retrieve it later.

The distinction becomes essential when agents move beyond short interactions and begin working on projects, customers, and workflows over extended periods.

Larger context windows help, but they do not eliminate the need for memory architecture. Persistent storage alone is not enough either: agents must decide what to remember, how to organize it, when to retrieve it, and when to stop using outdated information.

The most capable long-running agents will not simply accumulate more history. They will manage information selectively, preserve task state, distinguish facts from experiences, and retrieve the right knowledge at the right time.

The real challenge is not giving an agent more information. It is giving it continuity without overwhelming its ability to reason.

Sources and further reading

1. Microsoft — Multi-Agent Reference Architecture: Memory — A practical distinction between short-term memory, long-term memory, and working memory.
2. Anthropic — Effective harnesses for long-running agents — How agents preserve progress across sessions and context windows.
3. OpenAI Cookbook — Short-Term Memory Management with Sessions — Practical context trimming and compression techniques.
4. ACM — A Survey on the Memory Mechanism of Large Language Model-based Agents — Research covering agent memory mechanisms, design patterns, and evaluation.
5. Google Cloud — Agentic AI Architecture Components — Short-term and long-term memory architecture for agents.
6. Cloudflare — Agent Memory Documentation — A practical example of persistent, scoped memory with extraction and retrieval.
7. Google DeepMind — A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts — Research on compressing long documents into retrievable memory episodes.
8. LongMemEval — Benchmark for Long-Term Memory in LLMs — Evaluation of memory and recall over long conversational histories.