Varest cuts image tokens to 5.56% and still keeps over 80% of performance by allocating resolution by semantics
Synopsis
Vision-language models pay decoder compute for the hundreds or thousands of image tokens each image produces, and Varest splits compression into two steps: variable-resolution merging guided by object boundaries and text locations, then pruning by learned [REL] registers after the early decoder layers. Across three backbones and 13 benchmarks it beats 15 baselines in mean performance retention at every evaluated budget, keeping 86.10–92.33% of full-token performance with only 5.56% of the original layer-averaged image-token count.
Interpretation
Image-token compression no longer decides merging and removal from feature similarity or attention scores alone; it first judges whether a region can be summarized coarsely, then decides where detail must stay. Earlier merging and pruning relied on feature geometry, attention scores, learned retention utility or object membership, signals that do not map directly onto semantics: merging across two neighbouring objects creates a token that loses both meanings, and removing a token damages semantic coverage. Across three backbones and 13 benchmarks, Varest has the highest mean performance retention at every evaluated budget against 15 baselines; at 5.56% of the original layer-averaged image-token count it retains 86.10–92.33%.
In practice, a lightweight object localizer supplies boundary strength between neighbouring cells and a text localizer supplies a textness probability per cell, and together they set the merging granularity. Merging runs as constrained mean-boundary-linkage agglomeration: starting from individual cells, it repeatedly merges the pair of regions with the weakest shared boundary until each group reaches its allocated region count, and each region is pooled into one image token with a position tag. The localizers are distilled from the cosine-distance form of DINOv3 features and from EasyOCR boxes, then frozen; on LLaVA-1.5-7B, removing the object localizer lowers retention at the smallest budgets by 5.76–11.99 percentage points.
On the decoder side, four learned [REL] registers sit after the user prompt and before the assistant turn, and a bilinear scoring head ranks image tokens so only the top-scoring ones are kept. Pruning happens at the decoder midpoint, so more tokens pass through the early layers while the layer-averaged token budget stays the same; during training the pruning is an attention mask, so gradients flow through. At fixed pruning layers and retained token counts, learned selection beats random selection at every budget on all three backbones; at the smallest budgets, joint training gains 2.41–3.26 retention points over a [REL] variant trained with the tokenizer only.
Perspective
Evaluation covers the LLaVA-1.5-7B, LLaVA-1.5-13B and LLaVA-NeXT-8B backbones and 13 benchmarks: MMBench, AI2D, MMStar, POPE, GQA, ChartQA, DocVQA, TextVQA, the RefCOCO family, CountBench and V*, with retention normalized against the uncompressed backbone over 15 metrics and averaged. The authors list adapting to other vision-language model families such as Qwen-VL and InternVL as future work, and note that turning compute savings into lower latency also requires an efficient implementation.
Adaptive halting is optional and, when triggered, cuts a further 6.9–10.2% of image-token propagation across layers at a mean score loss of 0.11–0.24 points on the benchmarks' native 0–100 scale; on LLaVA-1.5-13B, 20.3% of inputs halt before selection runs. The authors note that latency depends heavily on the code implementation and is therefore not reported, and the aggregate gap for the text localizer masks larger changes on individual benchmarks, such as TextVQA dropping 1.21–2.24 points at the smallest budgets across the three backbones.
