Baidu caps the decoder's attention window, letting Unlimited OCR transcribe dozens of pages in one forward pass
Synopsis
Baidu released Unlimited OCR, which replaces every attention layer in the DeepSeek OCR decoder with Reference Sliding Window Attention (R-SWA), so the KV cache stops growing with output length during generation. Previously, end-to-end OCR models could not parse ten pages in a single forward pass and instead processed pages in a loop, resetting memory at each step. On OmniDocBench v1.5, Unlimited OCR scores 93.23 overall, 6.22 above the DeepSeek OCR baseline, and transcribes dozens of pages in one pass under a 32K maximum length.
Interpretation
Unlimited OCR lets end-to-end document parsing cover dozens of pages in a single forward pass instead of a page-by-page loop. Earlier end-to-end OCR models could not parse ten pages in one forward pass and instead processed pages in a loop, resetting memory at every step. On OmniDocBench v1.5, Unlimited OCR scores 93.23 overall against 87.01 for DeepSeek OCR, a gap of 6.22; on v1.6 it scores 93.92 overall.
Each newly generated decoder token sees all visual and prompt tokens but only the most recent stretch of generated tokens. Reference Sliding Window Attention (R-SWA) confines attention to a two-segment window made of a prefix window m and a causal sliding window n, with n defaulting to 128. Training continues from the DeepSeek OCR checkpoint for 4,000 steps on about 2M document OCR samples, freezing DeepEncoder and training only LLM parameters at a 32K sequence length.
The generation-stage KV cache is bounded by prefix length plus window width rather than growing linearly with output length. Standard multi-head attention cache size grows linearly with the number of generated tokens T, while the R-SWA decode-side cache is capped at prefix length plus n. With a fixed prefix length and ideal concurrency, DeepSeek OCR trails Unlimited OCR by 35% in throughput at 6,000 output tokens; on OmniDocBench, Unlimited OCR reaches 5580 TPS against 4951 TPS for DeepSeek OCR.
Perspective
The demonstrated scope is document OCR: with DeepSeek OCR as the baseline, it keeps DeepEncoder's 16x token compression and handles single-page and multi-page PDFs under a 32K maximum length; multi-page evaluation uses an in-house set grouped at 2, 5, 10, 15, 20 and 40+ pages, with no fewer than ten books per group. The authors propose training 128K contexts in the short term to prefill more pages, and building a prefill pool in the long term so the model fetches KV chunks itself.
The authors state that truly unlimited parsing is not achievable under a finite context such as 32K, since prefill still grows as pages accumulate. Repeated errors at 40+ pages mostly occur where small PDF text is hard to discern, which the authors attribute to the 1024x1024 Base encoder mode under multi-page conditions rather than to R-SWA losing track during long-horizon parsing.
