WhereBits picks which VLM layers localize without box labels: 0.72 COCO pointing against 0.22 for mean attention
Where a vision-language model looks can be read from attention itself, but attention sinks make raw maps unreliable, and contrastive methods localize only in some layers, which prior work chose with box annotations or fixed rules. WhereBits splits the question's effect on attention into whether a head looks at the image and where it looks, turns the second part, measured in bits on unlabeled questions, into a layer prior, and combines it with per-head log-ratio scores. Across eight vision-language models it reaches 0.72 pointing accuracy on COCO against 0.22 for mean attention, and matches layers tuned with box labels on COCO and RefCOCO.