billion-context folds long agent sessions into layered summaries, letting a 100K window carry billions of tokens
Synopsis
billion-context is a compression proxy that sits between an AI agent and its model API, folding consumed conversation into layered summaries so a single session keeps running inside a 100K-scale context window. Rather than hard truncation, the model itself decides when and what to compress; summaries are written in small ranges, can be decompressed on demand, and the prefix cache stays intact. In a 1,500-item cross-benchmark evaluation it reached 0.3097 Utilized F1 on ContextBench at 58.3% lower cost than full context, and survived all of a 1.48M-token MemoryAgentBench TTL run where full context failed 199 of 199 times.
Interpretation
Long coding sessions keep inflating their context, model APIs charge per token, and once the context window is exceeded the session degrades or dies; billion-context compresses consumed conversation into layered summaries so one session can run for days with billions of tokens passing through a single context window. A host's built-in summarizer also compresses, but compression here is incremental, reversible and prefix-cache friendly, which sets it apart from the built-in summarizer. A production-scale longitudinal study covered 4.5 months, three hosts, 174,327 model calls and 18.76B cumulative input tokens, with zero window violations on 204,800-token models and marathon sessions of 8,584 to 12,049 calls.
The model initiates compression itself: the proxy injects four context-management tools into the conversation, the model calls them as context grows, and the proxy executes the folding server-side so folded ranges stay summarized in history until restored. The four tools are compress, which folds a message range into a detailed summary; decompress, which restores a compressed range when exact details are needed; search_context, which keyword-searches compressed summaries and visible messages; and acp_status, which gives a context-usage overview plus which ranges are still compressible. The proxy parses Anthropic- or OpenAI-shaped requests, runs acp-kernel compression, injects the tools and compression philosophy, forwards to the real model API and rewrites the streaming response; a healthy session keeps a 95–97% prefix-cache hit rate, with compression itself costing at most 2%.
Perspective
The demonstrated scope is the coding-agent setting: the proxy sits between the agent and the model API, the evaluation uses DeepSeek V4.1 Flash, Qwen3-Embedding-8B, Qwen3-Reranker-4B and Pi Agent, and the comparators are full context and three RAG arms (bm25, embedding, embedding-rerank).
If the prefix-cache hit rate stays below 95–97%, the listed usual causes, in order, are upstream cache TTL expiry, a model switch, a bili bug and other or unknown causes; host-native plugins and standalone in-process extensions are mutually exclusive, and enabling both causes double compression.
