Multi-image vision-language models drop attention-based pruning: RI-Prune scores tokens in projector space and keeps 80% of performance with 4.6% of visual tokens
Synopsis
Multi-image vision-language models pack several high-resolution images into one sequence, so the visual token count balloons, and pruning methods built for single images lose their footing. RI-Prune instead scores each visual token at the projector output, before the decoder, using query relevance, cross-image correspondence and intra-image representativeness, then keeps a per-image Top-K share of a budget split evenly across images, with no training. On Qwen2.5-VL-7B retaining only 4.6% of visual tokens, it averages 50.7 across four multi-image benchmarks, 80.4% of the vanilla 63.1, above SCOPE at 48.7 and DivPrune at 48.4.
Interpretation
Visual token pruning for multi-image vision-language models no longer relies on attention saliency; it selects the retained tokens in one shot at the projector output, before the decoder, from three complementary signals. Earlier pruners were designed mainly for single images and estimated token importance from decoder query attention or vision-encoder CLS attention; those methods keep 82.2% accuracy on average on single-image benchmarks but fall to 69.1% on multi-image benchmarks, a 13.1 percentage-point gap. On Qwen2.5-VL-7B retaining only 4.6% of visual tokens, RI-Prune averages 50.7 across Mantis-Eval, MIBench, MIRB and MUIRBench, against 63.1 for the vanilla model, 48.7 for SCOPE and 48.4 for DivPrune.
Each visual token's score is the sum of query relevance, cross-image correspondence and intra-image representativeness, each z-normalized within the sample and fused with equal weights. Query relevance is the maximum cosine similarity to any question token; cross-image correspondence averages each image into 16 contiguous prototypes and takes the maximum cosine similarity to prototypes from other images; intra-image representativeness multiplies a token's alignment with its assigned prototype by the log inverse frequency of that prototype, suppressing repeated background. On Qwen2.5-VL-7B at 4.6% retention, equal weighting gives 50.7 average accuracy, query relevance alone 44.6, query relevance plus cross-image correspondence 48.6, and random selection 40.1.
Perspective
The demonstrated scope is three backbones, Qwen2.5-VL-7B, Qwen3-VL-8B and LLaVA-OneVision-7B, evaluated with official metrics on four multi-image benchmarks, Mantis-Eval, MIBench, MIRB and MUIRBench, plus video-frame results on Video-MME and TempCompass; pruning runs once at the projector output with default 16 prototypes and all three weights set to 1. The authors propose next steps of content-adaptive per-image budget allocation, exploiting decoder layer-wise redundancy, and extending to larger models, long video and generation-heavy tasks.
The authors note that a near-uniform per-image budget may be suboptimal when relevant evidence sits in only a few images; the method acts only at the projector-decoder interface and does not exploit layer-wise redundancy inside the decoder; the cross-image correspondence signal may become costly for long interleaved video or document inputs; and evaluation is limited to 7-8B models and multiple-choice multi-image benchmarks. The prototype sweep shows P=64 reaching 51.0, slightly above 50.7 at P=16, at nearly double the compute time, while P=128 falls back to 48.7.
