WhereBits picks which VLM layers localize without box labels: 0.72 COCO pointing against 0.22 for mean attention
Synopsis
Where a vision-language model looks can be read from attention itself, but attention sinks make raw maps unreliable, and contrastive methods localize only in some layers, which prior work chose with box annotations or fixed rules. WhereBits splits the question's effect on attention into whether a head looks at the image and where it looks, turns the second part, measured in bits on unlabeled questions, into a layer prior, and combines it with per-head log-ratio scores. Across eight vision-language models it reaches 0.72 pointing accuracy on COCO against 0.22 for mean attention, and matches layers tuned with box labels on COCO and RefCOCO.
Interpretation
Choosing the layers no longer needs box annotations: which layers of a vision-language model localize the object a question refers to can be read directly from how attention changes on unlabeled questions. Earlier approaches either picked one layer per model with box annotations, read a fixed band of deep layers, or chose a layer per question from the magnitude of the attention change, and none of these rules was derived from how well a given model's layers actually localize. Layer scores track held-out localization accuracy across depth, with positive Spearman correlation over the 12 COCO and RefCOCO model-dataset pairs; using only the whether-to-look bits lowers COCO pointing accuracy by 0.108.
The method separates the attention change a question induces relative to a generic prompt and keeps only the part that describes where the head looks within the image, converting it into a weight per layer. Each head scores a visual token by the log-ratio of its routing probability under the question and under the generic prompt, and an additive offset shared by both prompts cancels when two tokens are compared, so sinks need no detection step. The layer prior is estimated from 385 COCO calibration questions, and its three design constants were fixed once on labeled COCO data and then frozen; a new model or resolution needs only unlabeled questions, and ten questions estimate the prior within 0.011 pointing accuracy of all 385.
Perspective
The results cover causal decoders from six models and four families with the image placed before the text, and the invariance argument for per-head log-ratios relies on that layout; the V* zoom-in gains are measured under forced-choice evaluation, while generated answers often contain no option letter.
On small targets WhereBits points less often than the box-tuned layer, with the largest deficit on the smallest third of V* targets; the choice of reference prompt matters, and a short-caption prompt lowers pointing by 0.018-0.191 in three models; calibrating on V* questions drops COCO pointing for SmolVLM-500M from 0.718 to 0.455.
