Skip to main content
Back to timeline
arXivSource publication:

Images Stay in Context but Slip Out of Reach: VPR Shifts Only Cached Visual-Key Rotary Phases, Lifting 97 of 100 Model–Task Scores

Synopsis

Keeping an image inside a multimodal model's context window does not mean its details stay usable: with 32K text tokens inserted between image and question, Qwen2.5-VL-7B's DocVQA score falls from 93.5 to 52.6 ANLS. Visual Position Re-Anchoring (VPR) shifts the rotary phases of cached visual keys toward a short-context reference before question processing, leaving visual values, text cache entries and causal masks untouched and adding no forward pass. Across ten checkpoints and ten benchmarks, static VPR improves 97 of 100 model–task scores; in the TextVQA length study, all six models remain below their short-context references at 32K, so recovery is incomplete.

Source-provided article image: Keep Images in Sight: Visual Position Re-Anchoring for Long-Context MLLMs
arXiv

Interpretation

A long-context multimodal model can keep an image inside its context window and still lose access to the image's details once enough text accumulates between image and question, a failure the authors call visual evidence drift. Earlier routes to stronger visual grounding include attention intervention, decoding-time correction, visual revisitation and positional design, some of which need training adaptation, intermediate-attention processing, auxiliary predictions or repeated visual-token processing. With image and question held fixed and only the intervening text varied, Qwen2.5-VL-7B's DocVQA score drops from 93.5 in the short context to 52.6 ANLS with 32K intervening text tokens, even though the input fits the checkpoint's configured context window.

VPR translates the rotary phases of cached visual keys after context prefill and before question processing, restoring their positional relation to later queries to the short-context reference. The method acts only on the existing KV cache, preserving visual values, text-cache entries and physical cache slots, without extracting attention maps or adding model forward passes. Of two compensation schedules, static and continuous, the static schedule holds the offset fixed by default; across ten checkpoints, ten benchmarks and 500 questions per model–task, static VPR improves 97 of 100 scores, with the decreases on GQA and AI2D.

The phase shift redistributes attention toward image regions that contain answer evidence, rather than only raising overall attention to the image. Normalizing by total image attention separates concentration on answer-bearing text from a general rise in visual attention. Across six checkpoints, static VPR raises the mean share of visual attention assigned to text containing the reference answer by 0.70 to 2.57 percentage points, with the share rising on 88% to 95% of images per checkpoint and every paired 95% bootstrap interval for the mean gain above zero.

Perspective

The demonstrated setting is single-image question answering with up to 32K text tokens inserted between image and question, across ten checkpoints from Qwen2.5-VL, Qwen3-VL, InternVL3 and LLaVA; the authors propose extending re-anchoring to evidence spread across multiple images and documents and revisited throughout generated reasoning as a next step.

In the TextVQA length study, all six models remain below their short-context references at 32K, and recovery is described as substantial but incomplete; position does not explain every loss, since InternVL3-2B stays below its short-context reference even when all text precedes the image; compensation strengths beyond the reference reduce accuracy, and the authors treat those settings as exploratory rather than as isolating one cause of failure.