VEGD steers both contrastive-decoding branches to the same image evidence, cutting LLaVA-1.5-7B's CHAIRS from 53.0 to 48.4 without training
Training-free contrastive decoding previously altered only the auxiliary branch's input or computation, leaving uncontrolled which image evidence the main and auxiliary branches used during generation. Visual Evidence-Guided Decoding (VEGD) first locates the query-relevant image region using a few localization attention heads, weakens that region in the auxiliary image, then anchors both branches' attention to the same location in the final decoding layers and adaptively fuses or contrasts the two distributions by their divergence. Across CHAIR, POPE and MME-Hallucination on eight vision-language models, VEGD reports consistent hallucination reductions; at 128-token generation, LLaVA-1.5-7B's CHAIRS falls from 53.0 to 48.4.