The Hidden Bottleneck in Agentic AI Is Recomputing the Same Context

Subtitle: As AI agents run longer workflows, inference infrastructure must do more than generate tokens quickly. It must recognize what has already been processed, preserve useful state, and reuse computation without compromising correctness or isolation.

An AI agent can spend a surprising amount of computing power processing information it has already seen.

Imagine a coding agent working on a large repository. It reads the project instructions, examines the directory structure, opens several source files, runs tests, and proposes a fix. The tests fail. The agent inspects the error, revises its plan, and tries again.

Some of that work is new. But much of the surrounding context—the system instructions, repository guidelines, relevant files, and earlier conversation—may be unchanged.

If the inference system must process the same long prefix repeatedly, it spends resources reconstructing context instead of advancing the task.

This problem becomes more significant when an agent performs dozens of steps, when multiple agents share the same instructions, or when thousands of requests arrive at once.

The bottleneck is not always the model's ability to generate the next token. Sometimes, it is the cost of preparing the model to generate that token.

1. Why repeated context becomes expensive

When a large language model receives a request, it must process the input before generating its response. This initial stage is commonly called prefill.

During prefill, the model processes the prompt and computes intermediate representations used by its attention mechanism. These are stored in a structure called the key-value cache, or KV cache, so the model can generate subsequent tokens without recomputing the entire preceding sequence at every generation step.

The KV cache is important because a model's attention calculations depend on earlier tokens. As the sequence grows, retaining useful intermediate results avoids repeatedly performing the same work within that sequence.

But there is a distinction between two situations:

• Within one generation: The model reuses its KV cache as it generates additional tokens.
• Across requests or agent steps: The serving infrastructure may need to recognize that a new request shares a prefix with an earlier one and determine whether previously computed KV-cache blocks can be reused.

The second case is particularly relevant to agentic workloads.

Consider an agent that repeatedly submits a prompt containing 20,000 tokens of instructions, repository context, and conversation history, followed by a small new observation. If the prefix remains identical and the serving system supports cache reuse, it may avoid processing much of that shared input again.

Without reuse, the shared context can repeatedly consume prefill computation, increasing latency and reducing the number of requests the infrastructure can handle efficiently.

The exact cost depends on the model, sequence length, hardware, batch composition, and serving implementation. Reusing context is not a universal shortcut for every workload, but it can be valuable when substantial portions of requests remain identical.

2. The KV cache is useful, but it is not free memory

The KV cache stores intermediate model state associated with processed tokens. It can make generation more efficient, but that state occupies memory—often a scarce resource on GPUs and other accelerators.

Longer contexts and more concurrent requests can increase cache demand substantially.

This creates a practical trade-off:

• Keep more cached context available, and future requests may avoid recomputation.
• Evict cached context to free memory, and a later request may need to process it again.
• Move cache data to another memory tier, and the system may save accelerator memory but incur transfer latency and bandwidth costs.

A system that caches everything indefinitely is not practical. A system that discards useful state too aggressively wastes computation.

The infrastructure must decide which state is worth keeping, where it should live, and which request is most likely to benefit from it.

That is a scheduling and resource-management problem, not merely a model problem.

3. Prefix caching: reuse what has already been processed

One established technique is automatic prefix caching.

The idea is straightforward: when a new request begins with the same token prefix as an earlier request, the inference engine can reuse compatible KV-cache blocks instead of recomputing the shared portion.

For example, imagine three requests that all include the same long set of application instructions:

• Request A: shared instructions + question about authentication.
• Request B: shared instructions + question about database access.
• Request C: shared instructions + question about test failures.

The questions differ, but the common prefix may be reusable.

The open-source vLLM documentation on automatic prefix caching explains how cache blocks can be identified and reused when requests share compatible prefixes.

This is particularly useful in workloads involving repeated queries against the same long document or multiple rounds of a conversation.

However, prefix caching has limits.

The shared prefix must actually match. Small changes early in the prompt can prevent reuse of subsequent blocks. Reordering instructions, changing serialized tool descriptions, or inserting new conversation content into the middle of a prefix may reduce cache hits.

Only part of the workload may benefit. Prefix caching primarily reduces the work required to process shared input. It does not automatically accelerate the generation of every new output token.

Cached state must remain valid. Different model configurations, relevant input modalities, or other cache-key components can affect whether stored state is reusable.

The practical lesson is that context reuse depends partly on infrastructure and partly on how applications construct their requests.

4. Agent workflows make cache reuse more complicated

A traditional chatbot often follows a relatively simple pattern: receive a question and generate an answer.

An agent may follow a longer loop:

1. Read instructions and available context.
2. Decide which tool to call.
3. Execute the tool.
4. Inspect the result.
5. Update its plan.
6. Call another tool.
7. Evaluate whether the task is complete.

The loop can continue for many iterations.

Some context remains stable throughout the workflow. Other context changes with every observation, tool result, or planning decision.

A useful serving system must distinguish between these two categories.

Stable instructions, repeated tool definitions, and shared repository context may be candidates for reuse. New tool outputs and changed conversation history require appropriate updates to the model's context.

The goal is not to freeze the agent's context. It is to avoid recomputing portions that have not changed while ensuring the model still receives the correct information for its next decision.

There is also a coordination problem when many agents run concurrently.

Suppose 500 coding agents work on different tasks using the same project instructions. If their requests reach arbitrary model replicas, a request may land on a server that does not have the relevant cached state.

The system can then recompute a prefix that another server has already processed.

This is why cache reuse cannot be considered independently of request routing.

5. Why ordinary load balancing can miss the point

A conventional load balancer often distributes requests according to general routing rules, such as round-robin selection or current load.

That can be adequate for stateless services. But inference servers may hold valuable cached state associated with earlier requests.

If the next request in an agent's workflow is sent to a different replica, the destination may have to rebuild context that was available on the original server—unless the cache has been shared, transferred, or otherwise made accessible.

The least-loaded server is not necessarily the fastest destination for a particular request.

An inference router therefore faces a more complicated decision: balance computational load while accounting for where useful context is already available.

This is the motivation behind cache-aware routing.

On October 20, 2026, Google Cloud announced llm-d 1.0, an open-source distributed inference stack that includes prefix-aware routing, session-aware routing, KV-cache offloading, and traffic-control mechanisms. The project's description explicitly addresses multi-step agents that share long prefixes and generate bursts of requests.

In the architecture described by Google Cloud, the router tracks which replicas hold cached prefixes and considers both cache locality and load when selecting a destination.

The objective is not simply to route a request to the server with the most spare capacity. It is to select a destination that can execute the request efficiently while respecting the system's latency and capacity constraints.

That is a meaningful shift: the router becomes aware of the computational history of a request.

6. Keeping useful state when accelerator memory fills up

Even with good routing, a busy serving cluster can run out of accelerator memory for cached context.

One response is to use a hierarchy of storage tiers.

Frequently used cache data can remain in high-bandwidth accelerator memory. Less frequently accessed state can potentially be moved to host memory or another storage tier, depending on the serving architecture.

When the context is needed again, the system can retrieve it rather than recomputing everything from scratch.

This introduces another trade-off. Moving data consumes bandwidth and adds latency, so offloading is beneficial only when the cost of retaining or retrieving the state compares favorably with recomputation and the pressure on accelerator memory.

Google Cloud's llm-d announcement describes integrations for moving KV-cache data through accelerator memory, host memory, and storage tiers, with the aim of preserving useful context under demanding workloads.

The right cache policy depends on the workload.

An interactive assistant with strict response-time requirements may prioritize keeping its active session's context close to the model. A background agent processing a long report may tolerate more retrieval latency in exchange for lower accelerator-memory pressure.

The important point is that memory management, routing, and scheduling need to work together.

7. The performance gains are real—but workload-dependent

Infrastructure vendors increasingly publish results showing improvements from cache-aware routing and KV-cache reuse.

For example, Google Cloud reports that its Gemini platform doubled its prefix-cache hit rate from 35% to 70%, alongside improvements in time to first token for coding-agent workloads. Its llm-d 1.0 announcement also reports substantial latency and throughput improvements in specific benchmarks using prefix-aware routing and cache offloading.

These figures should be read as results for the reported infrastructure, workload, and benchmark configuration—not as guaranteed gains for every AI application.

Cache reuse helps most when requests contain substantial identical prefixes and the serving system can locate and reuse the corresponding state. It may provide little benefit when requests are mostly unique, prefixes change frequently, or generation rather than prefill dominates the total workload.

For engineering teams, the relevant questions are therefore:

• How much of the incoming context is repeated?
• How often do requests find a usable cached prefix?
• How much time is spent in prefill compared with decoding?
• Does routing preserve useful cache locality?
• How often is cached state evicted before it can be reused?
• Does offloading reduce total cost after accounting for transfer overhead?

Without those measurements, enabling a cache feature is not the same as demonstrating a performance improvement.

8. Context reuse introduces correctness and isolation requirements

Caching intermediate state is an optimization, but it must never become a shortcut around correctness or security.

A cache entry should only be reused when the relevant context and model configuration are compatible. Multimodal inputs may require additional identifiers for image or other input representations. Systems must also account for configuration differences that affect the computed state.

Shared infrastructure introduces a separate concern: one tenant must not gain access to another tenant's private context through careless cache reuse or observable cache behavior.

The vLLM prefix-caching documentation describes cache-key components and optional cache salting to isolate reuse between requests that do not belong to the same trust group.

These mechanisms illustrate a broader principle: cache reuse must preserve the same security boundaries that would apply if the system recomputed the request from scratch.

Application developers also need to consider whether prompts are constructed consistently enough to benefit from caching, while avoiding unsafe sharing of sensitive or user-specific state.

Performance optimization is useful only when the system remains correct and properly isolated.

9. What developers should do now

Teams building agentic applications do not need to deploy a distributed inference platform immediately. The first step is to determine whether repeated context is a significant cost in their own workload.

A practical approach is:

1. Measure prompt repetition. Inspect representative requests and identify stable instructions, repeated documents, tool definitions, and conversation prefixes.

2. Measure prefill and decoding separately. A long prompt and a long generated answer create different performance profiles. Identify which stage is responsible for the latency.

3. Test prefix caching. Use a representative workload with repeated long prefixes and compare latency, throughput, and memory consumption with caching enabled and disabled.

4. Keep stable context stable where appropriate. Avoid unnecessary changes to the beginning of a prompt if those changes prevent otherwise useful prefix reuse. Do not sacrifice clarity, security, or correctness merely to maximize cache hits.

5. Inspect routing behavior. If requests move across multiple replicas, determine whether the serving layer can route based on cached state or otherwise share the necessary context efficiently.

6. Evaluate memory pressure. Track cache occupancy, eviction rates, and the cost of offloading state. A high cache-hit rate is not enough if the system spends too much time moving data.

7. Validate isolation. Confirm that cache reuse respects tenant boundaries, model configurations, and the application's data-handling requirements.

The goal is not to optimize a single metric. It is to reduce the total cost of completing useful work while maintaining predictable latency and correct behavior.

The next inference bottleneck may be what the system already knows

Agentic AI changes the shape of inference traffic. Instead of isolated prompts arriving independently, applications increasingly produce sequences of related requests, repeated instructions, tool outputs, and long-running sessions.

That creates an opportunity to reuse computation—but only if the serving infrastructure can recognize shared context, retain useful state, and route requests intelligently.

Prefix caching addresses repeated input processing. Cache-aware routing helps requests reach useful state. Offloading provides additional options when accelerator memory is constrained. Together, these techniques make context a resource that infrastructure can manage rather than something every request must rebuild from scratch.

The broader lesson is that faster models alone will not solve every scaling problem in agentic AI.

As workflows grow longer and more concurrent, efficiency will depend increasingly on how well the entire system remembers what has already been computed.

The next performance improvement may not come from generating tokens faster.

It may come from not having to compute the same context again.

Further reading

• Announcing llm-d 1.0: Open distributed inference for agents — Google Cloud
• Automatic Prefix Caching — vLLM documentation
• llm-d v1.0: State-of-the-art inference, ready for Day 2 — llm-d
• Automatic Prefix Caching — vLLM feature guide

TipJournal covers the engineering decisions behind AI tools and systems—not just the announcements.