A probe for vision-language models' "conscious" access: MC-Lens reads image concepts and rewrites multi-step answers
Vision-language models can answer questions about an image, but the answer does not show where a concept appears inside the model. MC-Lens maps intermediate states from the vision encoder and the language decoder into the decoder's vocabulary space, reads concepts by vocabulary rank, and runs counterfactual replacements along concept directions. On flag-and-capital questions, a greedily selected concept direction set redirected nearly every one-hop answer; the same set also redirected direct answers, but only about 15% of one-hop cases passed both the necessity and sufficiency tests.