Long-running agents stop rereading the whole transcript: Ollama and Together AI split prefix-cache upkeep across harness and engine
Coding agents resend the entire session on every step, so long inputs and short outputs make prefix caching decide whether long sessions slow down. Ollama and Together AI describe the problem from the request-construction and inference-engine sides: a cache matches only up to the first differing token, so one per-turn field in the system prompt forces the whole session to be reprocessed. In an idealized estimate where each turn adds the same amount, ten turns cost 55 prompt units uncached versus 10 cached; a hit also depends on routing the request back to the node holding that cache.