Skip to main content
Back to timeline
arXivSource publication:

Vision-language models stop inventing findings for blank images: support completion cuts the invalid-evidence mirage rate from 0.759 to 0.097

Synopsis

Vision-language models often answer questions about images they were never shown, a failure called mirage that can be correct yet unjustified by the input. TIDE shows each training question with a blank image, a same-domain wrong image and an out-of-domain image, with targets that decline, so an image-blind predictor is no longer optimal. On Qwen2.5-VL-7B the three-condition predecessor cut the invalid-evidence mirage rate from 0.759 to 0.097 and the blank-image rate from 0.700 to 0.003, at a false-abstention cost rising from 0.018 to 0.126.

AI-generated editorial illustration: Teaching Vision-Language Models to Decline: Support Completion Against Mirage

Interpretation

A vision-language model will answer about an image it was never given, a behaviour called mirage; such an answer can be correct without being supported by the input the model actually saw. Standard visual instruction tuning places all training mass on images the question is about, so answering a blank or wrong image is never penalised and declining is never rewarded. On Qwen2.5-VL-7B-Instruct under the frozen protocol v4.1 test of 390 groups, the stock model reached a mirage rate of 0.759 over 1,170 invalid-evidence rows and 0.700 on blank-image rows.

Expanding each training question into four image roles, with the three counterfactual images mapped to decline targets, moves where the optimum sits. The objective, TIDE for support completion, adds no auxiliary head, contrastive term, detector or architectural change; the four examples from one question differ only in image and target. The three-condition predecessor, which never trained on out-of-domain images, cut the invalid-evidence rate to 0.097 and the blank-image rate to 0.003 on the 390-group test, with out-of-domain falling from 0.764 to 0.054 and the residual concentrated on in-domain wrong images at 0.233.

Declining partly spills onto valid images, lowering the answered rate and raising false abstention. The objective specifies declining on counterfactual inputs and does not directly constrain behaviour on valid images. On the same test the answered rate fell from 0.982 to 0.874 and false abstention rose from 0.018 to 0.126, while accuracy over all real rows fell from 0.577 to 0.559.

Grounding the vision projector before support completion leaves the result essentially unchanged, while reversing the order re-opens the mirage. Grounding alone raised the invalid-evidence rate to 0.859 and moved blank-image mirage only from 0.700 to 0.646. Grounding then TIDE reached 0.094 against 0.097 for TIDE alone, a paired difference of 0.003 with interval [-0.018, 0.014], whereas TIDE then grounding returned to 0.615 with in-domain wrong images at 0.936.

Perspective

Every measured result is for Qwen2.5-VL-7B-Instruct at seed 0 under protocol v4.1 with the three-condition predecessor, on a 390-group test of 39 groups per domain and 1,560 rows, judged by a response-only judge U and an image-aware rule B. The four-condition TIDE, two further base models, three seeds, component ablations and a human validation of the judges are in progress and report no values here.

Proposition 2 concerns the population optimum and needs supports disjoint in meaning, while the realised training set is disjoint only at the level of image content and question strings. Proposition 3 bounds a weighted sum of string-level answer-or-decline decision errors, not each residual and not a judge-defined mirage rate. Rule B was validated in sample on 38 earlier-era human-labelled rows, and judge U was not validated against humans. The rank-correlation analysis between the in-domain residual and false abstention across the ten domains is a post-hoc analysis of ten points from 39 rows each, not a pre-registered test.