Skip to main content
Back to timeline
arXivSource publication:

Entropy curves become an alarm for multimodal reasoning: SEED realigns vision mid-decode and lifts six benchmarks by 3.2%–3.7% on average

Synopsis

Multimodal large language models often hallucinate content unfaithful to the image during long-chain reasoning, and SEED treats the entropy of the decoding process as a real-time diagnostic: when a trajectory sits in a relatively low-entropy convergence regime while showing a local entropy spike, it triggers a visual attention realignment. The training-free method improves average accuracy by 3.2%–3.7% across six benchmarks and nine settings on Qwen3-VL-8B, Qwen3-VL-4B and Llama-3.2-11B, adding less than 5% inference latency.

Source-provided article image: Entropogram-guided Visual Realignment Decoding: Capturing the Rhythm of Multimodal Reasoning
arXiv

Interpretation

Hallucination in multimodal large language models is treated not as a single mispredicted token but as an observable dynamical state along the decoding trajectory: the model runs in an overall overconfident low-entropy regime while showing a local entropy spike, marking a hallucination-prone state. Earlier training-free decoding methods mainly identify which tokens are hallucinated and suppress them, treating entropy as a token-level confidence signal rather than examining how multimodal information is integrated over the course of generation. Among 1,228 visual question answering trajectories generated by Qwen3-VL-8B on M3CoT, 763 hallucinated and 465 correct, the mean entropy jump at hallucination onset is 0.245 versus 0.012 for correct trajectories, while differences before onset are substantially smaller.

Once that state is detected, SEED injects an image representation weighted by visual saliency at the first layer of the language model, pulling attention back to visual evidence. The intervention acts at the attention level rather than the final output layer, runs only when triggered, leaves the model architecture unchanged and requires no additional training. Across the Qwen3-VL-8B, Qwen3-VL-4B and Llama-3.2-11B backbones, six benchmarks (VStar, MMVP, VisOnly, M3CoT, POPE and HallusionBench) and nine settings, average gains are 3.2%–3.7%; on POPE and HallusionBench, Qwen3-VL-8B and Qwen3-VL-4B improve by 2.5% and 3.6% on average respectively.

The intervention is cheap: end-to-end latency rises from 3.30 to 3.45 milliseconds per token. By comparison, VCD adds 54% latency, MemVR 157% and OPERA 333%. Measured with a maximum generation length of 512 tokens on four NVIDIA RTX 5090 GPUs, with hyperparameters selected once on a small held-out subset of M3CoT and then fixed across backbones and benchmarks.

Perspective

The demonstrated scope is zero-shot results for the Qwen3-VL-8B, Qwen3-VL-4B and Llama-3.2-11B backbones on six multimodal reasoning and hallucination benchmarks across nine evaluation settings, with intervention hyperparameters selected once on a small held-out subset of M3CoT and then fixed.

The link between entropy dynamics and hallucination is correlational, and SEED uses it only as a detection signal without claiming causality; the injected visual representation is query-independent, needs scale alignment on the cross-attention Llama-3.2-Vision backbone, and shows 0.2–1.8 point drops on MMVP and POPE, where the net POPE decline mainly comes from more outputs failing the yes/no format rather than more object hallucination.