Skip to main content
Back to timeline
arXivSource publication:

A small model reads first: SkimPrefill cuts prefill time 3.7 to 11.6 times on long image-text inputs

Synopsis

On long interleaved image-text inputs of about 100K tokens, vision-language models spend most of the wait for the first output token in prefill. SkimPrefill has Qwen3.5-0.8B read the whole input first at half image resolution, stopping at layer 7, then scores every image and paragraph with one attention head of that layer and hands the top whole units to the target as an ordinary shorter prompt. Keeping 10% of the input tokens on MMLongBench at 128K, it speeds up the prefill of Qwen3.5-9B, Gemma-4-12B and Qwen3-VL-32B by 3.7, 11.6 and 8.2 while holding their average accuracy.

Source-provided article image: Let a Small Model Read First: Faster Prefill for Long Interleaved Image-Text Inputs
arXiv

Interpretation

Prefill on long interleaved inputs no longer needs the target model to read every token before deciding what to keep; a 0.8B model reads once and picks the whole images and paragraphs that hold the evidence. Earlier pruning methods scored the input with the target's own early layers, or with a same-family small model that first wrote an answer and was scored on that basis, and that scoring consumed much of the time pruning saved. On MMLongBench at 128K, across 19 datasets of five task types with 10% of the input tokens kept, prefill speeds up by 3.7, 11.6 and 8.2 on Qwen3.5-9B, Gemma-4-12B and Qwen3-VL-32B, with average accuracy no lower than the full input.

The selector runs only layer 7 of the first 8 layers of Qwen3.5-0.8B, six of which are Gated DeltaNet layers whose time grows linearly with input length, so this pass is short. Scoring is the average attention that one head of layer 7 pays to each unit, with images read at half resolution; the chosen whole units are reassembled in their original order at full resolution as an ordinary shorter prompt. On Qwen3-VL-32B, selection takes 0.71 s while the target needs 17.58 s to prefill the full input, and one untrained selector serves targets of three different model families.

Perspective

The demonstrated scope is prefill speedup and average accuracy on MMLongBench at 128K with 10% of the input tokens kept, with all three targets running in SGLang and the selector using one setting for every target.

Summarization needs the whole document, and accuracy there falls by 3.4 to 4.2 points at a 10% budget; on LongDocURL, 18 of 100 questions keep no answer page, which accounts for most of the gap to SGP.