Skip to main content
Back to timeline
arXivSource publication:

Vision-language models see the match but cannot say it: training 16 to 32 cross-image heads lifts SPair-71k from 0.430 to 0.712

Synopsis

Vision-language models often fail to match a point or object across two images even when the answer is decodable from their hidden states. Across Qwen3-VL, InternVL3.5 and Gemma-3, the failure splits in two: exact copies are compared but not read out at the answer position, while semantic correspondence sits in the visual tokens and is never compared. Training only the query and key projections of 16 to 32 cross-image attention heads raises Qwen3-VL-8B on SPair-71k from 0.430 to 0.712 and transfers to HPatches and BLINK, which were not used for training.

Source-provided article image: Seen but Unsaid: Locating and Repairing Cross-Image Comparison in Vision-Language Models
Figure 1 · arXiv · Page 2

Interpretation

Vision-language models fail when matching a point or object across two images, and the failure sits at different points: in exact-copy tasks the comparison is already computed but the answer position does not read it out, while in semantic correspondence the correspondence is present in the visual tokens but the comparison stage never delivers it to the answer position. An earlier account attributed the gap to entities that cannot be named and often used probe readouts over annotated regions as a reference; on synthetic shapes those readouts are near ceiling because reference and target are pixel-identical, so they measure pixel identity rather than computation inside the language model. On synthetic shapes with Qwen3-VL-8B, behavioural accuracy is 41% while the answer-position probe reaches 64%; editing the answer-position residual stream along the probe's direction changes accuracy by no more than random subspaces, whereas adding a letter's output direction at the same layer switches the answer to that letter.

The comparison between two images is carried by a small set of cross-image attention heads: option tokens of the second image retrieve evidence from the reference region of the first through these heads, with attention first landing on salient candidate locations and then weighted by content similarity. Earlier analyses mostly concerned text-to-image grounding or within-image computation and did not identify the step at which region-to-region comparison between two images is lost, nor a concrete trainable component. Under interventions that keep total attention mass fixed and only redistribute it, Qwen3-VL-8B loses 17.2 points on synthetic shapes and 32.2 points on HPatches, and InternVL3.5-8B loses 20.2 and 16.8 points, all beyond random head sets and attention-mass-matched head sets.

Training only the query and key projections of these heads recovers most of the semantic-correspondence deficit and transfers to benchmarks not used for training; an unconstrained all-layer adapter also routes part of its gain through the same heads. The base models sit near chance on SPair-71k, and knocking out these heads barely affects their semantic answers, so the base models do not yet use these heads for that task. SPair-71k accuracy rises from 0.430 to 0.712 for Qwen3-VL-8B and from 0.354 to 0.605 for InternVL3.5-8B with about 1.68M and 1.50M trainable parameters; transplanting the image-2 attention rows of the all-layer adapter D into the base model recovers 33.9% of its gain in Qwen3-VL-8B and 27.0% in InternVL3.5-8B.

Perspective

Mechanistic analysis and repair cover the two 8B-scale models Qwen3-VL-8B and InternVL3.5-8B, with Gemma-3-12B entering measurement results only; tasks are four-way two-image multiple-choice questions with 1,000 test items each for synthetic shapes, HPatches same-instance point correspondence and SPair-71k semantic correspondence, of which Syn-known has 500, and transfer is evaluated on the BLINK semantic and visual correspondence validation splits.

Knocking out these heads barely affects SPair-71k in the base models, so causal evidence for semantic correspondence comes from the trained models; mass-preserving interventions move some HPatches items outside the natural residual-norm band, with reported out-of-band rates between 5.6% and 20.6% across operations and models; the attention-mass-matched control heads are not layer-matched, and three of four learning-rate sweeps selected the largest rate in the grid; BLINK splits contain fewer than 200 items each, so their intervals are wide.