VEGD steers both contrastive-decoding branches to the same image evidence, cutting LLaVA-1.5-7B's CHAIRS from 53.0 to 48.4 without training
Synopsis
Training-free contrastive decoding previously altered only the auxiliary branch's input or computation, leaving uncontrolled which image evidence the main and auxiliary branches used during generation. Visual Evidence-Guided Decoding (VEGD) first locates the query-relevant image region using a few localization attention heads, weakens that region in the auxiliary image, then anchors both branches' attention to the same location in the final decoding layers and adaptively fuses or contrasts the two distributions by their divergence. Across CHAIR, POPE and MME-Hallucination on eight vision-language models, VEGD reports consistent hallucination reductions; at 128-token generation, LLaVA-1.5-7B's CHAIRS falls from 53.0 to 48.4.
Interpretation
Training-free contrastive decoding now treats which image region each branch looks at as something to intervene on, not just as a perturbed auxiliary input to construct. Earlier global-perturbation methods altered the whole image, so query-irrelevant changes entered the output difference; local methods decided only where the auxiliary input was altered, neither directing the main branch toward relevant evidence nor anchoring the auxiliary branch to the corresponding weakened region. On CHAIR, POPE and MME-Hallucination, VEGD reports consistent hallucination reductions across eight vision-language models; at 128-token generation, CHAIRS falls from 53.0 to 48.4 on LLaVA-1.5-7B, from 56.6 to 46.0 on InstructBLIP-7B, and from 29.0 to 24.2 on ShareGPT4V-7B.
The procedure has two steps: locate the query-relevant image evidence, then keep both branches fixed on the same location throughout decoding. Localization does not average all attention heads; it restricts candidates to middle layers, filters heads by image attention mass and spatial fragmentation, and smooths and averages their spatial maps into one continuous evidence map. In an adversarial POPE-COCO diagnostic on LLaVA-1.5-7B, the mean target-mask effect is 3.32 for correct predictions versus 0.57 for errors, a paired gap of 2.75 (95% CI [2.04, 3.48]); the paired gap in evidence enrichment averages 0.79 over layers 10–20 and falls to 0.32 over layers 21–31.
The auxiliary image weakens high-evidence regions at the evidence map's mean threshold while preserving the rest of the scene, so the two branches differ mainly in the evidence under test. In the final seven decoding layers, the same evidence map modulates the visual-carrier attention heads in both branches: the main branch reinforces intact evidence, while the auxiliary branch stays anchored to the same location after its content is weakened. The two branches' predictions are additively fused or contrastively subtracted according to their total variation distance; on MME-Hallucination, VEGD totals 650.00 on LLaVA-1.5-7B, 453.33 on InstructBLIP-7B and 671.67 on ShareGPT4V-7B, gains of 60.00, 43.33 and 100.00 over regular decoding.
Perspective
Evaluation covers three hallucination benchmarks, CHAIR, POPE and MME-Hallucination, and eight vision-language models: LLaVA-1.5-7B, InstructBLIP-7B, ShareGPT4V-7B, DeepSeek-VL2-Small, Janus-Pro-7B, Kimi-VL-A3B-Instruct, InternVL3-9B and Qwen2.5-VL-7B, with all compared methods reimplemented in the same evaluation framework and identical generation settings. The method itself is training-free, requiring one probe forward pass before generation and running on NVIDIA A100 40GB GPUs.
Adding more localization heads leaves POPE nearly unchanged but lowers MME, suggesting extra heads add noisy localization cues; intervention depth is more sensitive for MME than POPE, and the default final seven layers leaves F1 0.03 below its peak. These hyperparameter conclusions come from the reported benchmarks and models; behavior on other tasks is not given in the material.
