Skip to main content
Search

# WhereBits

1 related articles

Subscribe to updates
  1. arXiv

    WhereBits picks which VLM layers localize without box labels: 0.72 COCO pointing against 0.22 for mean attention

    Where a vision-language model looks can be read from attention itself, but attention sinks make raw maps unreliable, and contrastive methods localize only in some layers, which prior work chose with box annotations or fixed rules. WhereBits splits the question's effect on attention into whether a head looks at the image and where it looks, turns the second part, measured in bits on unlabeled questions, into a layer prior, and combines it with per-head log-ratio scores. Across eight vision-language models it reaches 0.72 pointing accuracy on COCO against 0.22 for mean attention, and matches layers tuned with box labels on COCO and RefCOCO.

You’re all caught up