Pruned visual tokens keep the answer but shift the odds: DiFiT trains selection on the full model's response distribution
Synopsis
Visual token pruning for multimodal large language models is usually judged by whether answers stay correct, yet pruning can change the probabilities of alternative responses even when greedy answers do not move. DiFiT learns which visual tokens to keep from both the full-token model's response distributions and reference answers, with one ranking serving several retention ratios. On Qwen2.5-VL-7B it cuts sampling-induced negative flips by 30.3% relative to FastV at 10% retention and preserves 98.74% of baseline performance on MME at 30% retention.
Interpretation
Visual token pruning can preserve greedy answers while still changing how the model distributes probability across candidate responses, and that change can be measured on its own. Earlier pruning methods were mainly compared by the task score retained at a given budget, and such scores show whether answers stay correct but reveal little about changes in response probabilities. Under shared reference prefixes, with fixed samples that the full and pruned models answer identically, average predictive drift grows with stronger compression for every evaluated pruning method.
DiFiT aligns the pruned model's vocabulary probabilities at response positions with the full-token model while reference answers constrain task quality. Dual-View Distillation matches probabilities across the whole vocabulary under shared reference prefixes, and answer supervision rewards the correct response tokens, so the two signals complement each other. Evaluation covers Qwen2.5-VL-7B, Qwen2.5-VL-3B and LLaVA-OneVision-1.5-8B at 30%, 20% and 10% retention across eight image-language benchmarks and MVBench, with a separately trained scorer per backbone and one scorer serving all three budgets on that backbone.
Perspective
Evaluation covers the Qwen2.5-VL-7B, Qwen2.5-VL-3B and LLaVA-OneVision-1.5-8B backbones at 30%, 20% and 10% retention across eight image-language benchmarks and the MVBench video benchmark; drift and behavior analyses center on Qwen2.5-VL-7B, with behavior tests spanning four tasks and two budgets.
Hard selection discards entire tokens, so evidence can be lost when retained tokens do not represent the discarded regions, and the frozen visual encoder cannot adapt its features to a smaller token budget. Response-position matching under shared reference prefixes does not capture error accumulation when an early token change shifts the context for later tokens during free generation.
