Skip to main content
Back to timeline
ranxianglei/billion-contextSource publication:

billion-context folds long agent sessions into layered summaries, letting a 100K window carry billions of tokens

Synopsis

billion-context is a compression proxy that sits between an AI agent and its model API, folding consumed conversation into layered summaries so a single session keeps running inside a 100K-scale context window. Rather than hard truncation, the model itself decides when and what to compress; summaries are written in small ranges, can be decompressed on demand, and the prefix cache stays intact. In a 1,500-item cross-benchmark evaluation it reached 0.3097 Utilized F1 on ContextBench at 58.3% lower cost than full context, and survived all of a 1.48M-token MemoryAgentBench TTL run where full context failed 199 of 199 times.

Source-provided article image: ranxianglei / billion-context
GitHub Trending

Interpretation

Long coding sessions keep inflating their context, model APIs charge per token, and once the context window is exceeded the session degrades or dies; billion-context compresses consumed conversation into layered summaries so one session can run for days with billions of tokens passing through a single context window. A host's built-in summarizer also compresses, but compression here is incremental, reversible and prefix-cache friendly, which sets it apart from the built-in summarizer. A production-scale longitudinal study covered 4.5 months, three hosts, 174,327 model calls and 18.76B cumulative input tokens, with zero window violations on 204,800-token models and marathon sessions of 8,584 to 12,049 calls.

The model initiates compression itself: the proxy injects four context-management tools into the conversation, the model calls them as context grows, and the proxy executes the folding server-side so folded ranges stay summarized in history until restored. The four tools are compress, which folds a message range into a detailed summary; decompress, which restores a compressed range when exact details are needed; search_context, which keyword-searches compressed summaries and visible messages; and acp_status, which gives a context-usage overview plus which ranges are still compressible. The proxy parses Anthropic- or OpenAI-shaped requests, runs acp-kernel compression, injects the tools and compression philosophy, forwards to the real model API and rewrites the streaming response; a healthy session keeps a 95–97% prefix-cache hit rate, with compression itself costing at most 2%.

Perspective

The demonstrated scope is the coding-agent setting: the proxy sits between the agent and the model API, the evaluation uses DeepSeek V4.1 Flash, Qwen3-Embedding-8B, Qwen3-Reranker-4B and Pi Agent, and the comparators are full context and three RAG arms (bm25, embedding, embedding-rerank).

If the prefix-cache hit rate stays below 95–97%, the listed usual causes, in order, are upstream cache TTL expiry, a model switch, a bili bug and other or unknown causes; host-native plugins and standalone in-process extensions are mutually exclusive, and enabling both causes double compression.