Varest cuts image tokens to 5.56% and still keeps over 80% of performance by allocating resolution by semantics
Vision-language models pay decoder compute for the hundreds or thousands of image tokens each image produces, and Varest splits compression into two steps: variable-resolution merging guided by object boundaries and text locations, then pruning by learned [REL] registers after the early decoder layers. Across three backbones and 13 benchmarks it beats 15 baselines in mean performance retention at every evaluated budget, keeping 86.10–92.33% of full-token performance with only 5.56% of the original layer-averaged image-token count.