Skip to main content
Back to timeline
OllamaSource publication:

Long-running agents stop rereading the whole transcript: Ollama and Together AI split prefix-cache upkeep across harness and engine

Synopsis

Coding agents resend the entire session on every step, so long inputs and short outputs make prefix caching decide whether long sessions slow down. Ollama and Together AI describe the problem from the request-construction and inference-engine sides: a cache matches only up to the first differing token, so one per-turn field in the system prompt forces the whole session to be reprocessed. In an idealized estimate where each turn adds the same amount, ten turns cost 55 prompt units uncached versus 10 cached; a hit also depends on routing the request back to the node holding that cache.

Source-provided article image: Making open models work for long-horizon agentic workloads
Ollama

Interpretation

The cost of long-horizon agents sits mainly on the input side: each turn carries the system prompt, every tool definition, the user request, files already read and commands already run, while the reply is often a tool call a few dozen tokens long. Without a cache the engine reprocesses the whole history every turn, and the rework grows with session length; in an idealized estimate where each turn adds the same amount, ten turns process 55 prompt units and a hundred turns process 5,050. Under the same idealized setting, prefix caching cuts ten turns to 10 units and a hundred turns to 100, removing the pattern in which repeated work grows with the square of session length.

A prefix cache matches token by token from the start of the prompt and stops at the first difference, so everything after that point is recomputed and keeping the front of the prompt stable is what preserves the cache. Appending tool definitions at the end of the conversation, moving per-turn values such as timestamps and git status into the task message, and fixing reasoning effort at session start all avoid touching the front of the prompt. The GLM-5.3-Flash chat template writes reasoning effort into the very first system block, so raising effort mid-session makes the next request differ almost at the first token and a long run reprocesses hundreds of thousands of tokens.

On the engine side the cache has to still exist and the next turn has to reach the machine that holds it. A session's cache lives on a particular node in a particular data center; a request landing elsewhere reprocesses all earlier turns and leaves two nodes storing the same prefix. Together AI routes requests by prompt_cache_key or by hashing the prefix to the node with the longest match, offloads the cache to CPU memory while a session waits on tool calls, hands out cache memory in small blocks as needed (PagedAttention), and rebuilds the running batch each pass (continuous batching) so new requests take a free slot immediately.

Perspective

The discussion covers agent traffic with long inputs and short outputs, spanning prompt layout on the request side and cache routing, offloading and memory allocation on the engine side; the 55-versus-10 and 5,050-versus-100 figures are idealized numbers for equal per-turn increments, and the authors state that real turns vary in size.

A cache hit is not free, since every new token still has to attend through the cached content; even with a perfectly stable prompt, an overloaded warm node forces a choice between waiting and paying to reprocess the prefix on an idle node, and a cache miss is sometimes the better deal.