Keeping only 5% of visual tokens still holds 91.6% of performance: Glance-ETD glances at low resolution, then selects tokens by question evidence
Synopsis
In multimodal large models, visual tokens dominate prefill cost for high-resolution images, and earlier pruning at extreme budgets only shortened the language model's input while the vision encoder still attended over every patch. Glance-ETD uses one question-conditioned scorer of about 1.9M parameters to drive two steps: a quarter-resolution glance that re-encodes only the 25% of cells relevant to the question at native resolution, then an evidence-tempered diversity rule that picks the final tokens. At 5% of the visual tokens on Qwen3.5-4B, it retains 91.6% of full-token performance across fourteen benchmarks, 17.3 points above the prior best LT-OPD, and reaches the first token 3.1 times sooner on 4K images.
Interpretation
Visual token compression at extreme budgets leaves two bottlenecks that earlier work did not address together: pruning happens after the vision encoder, which still attends over every patch and dominates cost at high resolution, and the few surviving tokens often miss the small regions that hold the answer. Earlier token pruning acts after the encoder and only shortens the language model's prefill, barely changing time to first token; diversity-based selection covers the scene but often drops small text such as a street sign or phone number, while relevance-only selection loses the context that counting and existence questions need. On HR-Bench 4K the encoder accounts for 2.64 s of a 4.03 s time to first token, and 89% of its FLOPs come from global attention over about 50k patches, so methods that prune only the language model input stay at 0.87 to 1.28 times speed-up.
Glance-ETD also cuts encoder cost: the image is first glanced at a quarter of its patches, the scorer picks the 25% of cells with the highest relevance, only those cells' native patches are encoded a second time, and every other cell keeps its coarse encoding. The low-resolution glance plus selective re-encoding makes the encoder cost about a sixth of the dense FLOPs, and the output stays on the same token grid as the dense encoder, so any post-encoder token selector applies unchanged. On Qwen3.5-4B, adding the glance encoder cuts HR-Bench 4K prefill from 373 to 65 TFLOPs and time to first token from 3355 to 1290 ms, raising retention from 92.5% to 93.3%.
The second step, Evidence-Tempered Diversity (ETD), keeps the greedy diversity loop of DivPrune but weights each candidate token by the scorer's evidence, so a flat evidence map reduces it to DivPrune and a question pointing at a small region tilts selection toward it while still covering the scene. The scorer has about 1.9M parameters, reads the model's own token features and question embeddings, takes about 2 ms per call, and is trained from box annotations without an extra encoder; it is used only at inference, while the model is trained with diversity-chosen cells, so the model does not depend on the scorer. With the MLLM frozen, ETD retains 82.0% at the strict 5% budget, 10.7 points above the strongest training-free baseline DivPrune; on TextVQA, ETD keeps a token inside the answer box for 99.1% of questions against 80.2% for DivPrune.
The full pipeline retains 91.6% of full-token performance at the strict 5% budget, 17.3 points above the prior best LT-OPD and 6.0 points above the same training recipe with DivPrune, and shortens time to first token on 4K images by 3.1 times. The gains concentrate on reading and document questions, where diversity alone fails; the same method also leads on Gemma 4, GLM-4.6V and InternVL3.5. The fourteen-benchmark retention is a mean over three training seeds with a standard deviation of 0.3 points; the full pipeline retains 83.6% on Gemma 4 E4B-it, 88.2% on GLM-4.6V-Flash and 83.8% on InternVL3.5-8B.
Perspective
The demonstrated setting is image question answering at a strict 5% visual token budget: the main experiments use Qwen3.5-4B in non-thinking mode over fourteen benchmarks, with the comparison repeated on Gemma 4 E4B-it, GLM-4.6V-Flash and InternVL3.5-8B. The glance encoder applies only to images with at least 4096 patches; smaller images stay on the dense encoding path.
Text and document benchmarks remain the largest absolute gap, with OCRBench at 590 of 848, InfoVQA at 48.1 of 79.4 and DocVQA at 82.6 of 94.6. On Gemma 4 the glance encoder adds no gain, and the full pipeline retains 83.6%, slightly below 84.2% for ETD alone.
