Skip to main content
Search

# Hallucination Mitigation

9 related articles

Subscribe to updates
  1. aclanthology.org

    Vision-language models stop pruning whole layers: head-level attribution cuts CHAIR-i from 18.0 to 8.6 on multi-label COCO

    Vision-language models often describe objects that are not in the image, and earlier fixes intervened at the output level or across whole attention layers, which cannot separate heads that support faithful captions from heads that drive hallucination. HACP scores each attention head at inference time by the spectral change of its perturbed representation, then suppresses hallucination-prone heads and reinforces faithful ones during decoding without training. On a 1000-image multi-label COCO subset it lowers CHAIR-s/i from 66.7/18.0 to 43.4/8.6 and raises F1 from 73.0 to 77.0.

  2. aclanthology.org

    Vision-language hallucinations get a fix at the encoder: three LoRA experts, no retraining per model

    Hallucinations in large vision-language models partly trace to CLIP-style encoders that favor main objects and skip image detail. PLANE attaches three LoRA visual experts to CLIP, each trained on object, attribute or relation captions, and an instruction-aware gate fuses the four image representations at token level. Trained once on LLaVA-1.5 7B, it plugs into other models sharing the same encoder; on LLaVA-1.5 7B the MME hallucination subset total rises from 555.38 to 582.34 and the GIFT hallucination rate falls from 23.39% to 20.30%.

  3. arXiv

    Diffusion language models stop reusing one steering vector: feeding the previous prediction back lifts TruthfulQA multiple-choice average from 30.89 to 45.42

    TraceSteer feeds a diffusion language model's previous denoising prediction back into activation steering, adapting both the correction direction and its strength step by step, and improves factual reliability on TruthfulQA for LLaDA-8B-Instruct and Dream-7B-Instruct.

  4. arXiv

    Visual edits lose their grip mid-decoder: equal-strength interventions fade before answers settle, and calibrated routing turns that into attribute accuracy

    Applying the same visual-residual edit at every decoder layer with matched RMS strength reveals a persistent drop in effect, called the Evidence Anchoring Boundary; routing interventions by question type and calibration improves full-set attribute accuracy across three checkpoints, while universal subtraction is neutral or harmful.

  5. arXiv

    LLMs quietly flip constraints when they write optimization models: post-hoc repair lifts zero-shot accuracy from 44.8% to 64.9% without retraining the generator

    OPT-Kraken treats an LLM's optimization model and solver code as frozen artifacts and repairs them after the fact, lifting zero-shot accuracy from 44.8% to 64.9% across six benchmarks while making only 0.6% of initially correct outputs incorrect.

  6. arXiv

    SAGE suppresses prompt-invariant visual attention sinks to improve grounding and reduce hallucination across VLM families

    Revisiting visual attention sinks in vision-language model decoders, this work finds a layer-dependent structure in which early and final decoder layers repeatedly collapse onto the same few image regions across prompts (Prompt-Invariant Sinks, PIS) while mid layers stay prompt-conditioned and drive vision-language alignment; it then proposes SAGE, a training-free intervention that applies an additive bias to visual-token attention logits using token-aligned ROI masks from standard vision backbones such as CLIP, ViT, and DINOv3, steering attention away from PIS toward query-relevant regions and improving visual grounding, reducing hallucination, and yielding gains on public benchmarks across LLaVA-1.5, Qwen2-VL, InternVL3, and NVILA.

  7. arXiv

    RED, a training-free decoding method, raises accuracy by up to 7% on three audio-visual hallucination benchmarks at about 1.5x standard time to first token

    The work introduces Relevant Evidence Decoding (RED), a training-free method that uses pointwise mutual information to decompose the joint audio-visual prediction into audio, video, and residual interaction components, then runs a question-only inference pass to determine the required evidence type and augments the original prediction with the corresponding contribution; across three audio-visual hallucination benchmarks and three AV-LLMs, RED improves over standard decoding by up to 7% on CMM, 6.3% on AVHBench, and 3.8% on SVHalluc, with an average relative time to first token of 1.5x standard decoding.

  8. arXiv

    EscapeGuard cuts tool-selection hallucination by 9.0 pp across six benchmarks and six models while suppressing cross-configuration hallucination escape

    The work identifies and formally defines hallucination escape, in which existing tool-hallucination mitigations reduce hallucination on the tool configuration they are tuned on but increase it on others; it finds that models hold intrinsic tool-use tendencies whose conflict with the runtime configuration sharply raises hallucination and that existing methods reinforce these tendencies; it then proposes EscapeGuard, a training-free inference-time method combining conflict-aware gating with configuration-derived attention enhancement, which reduces tool-selection hallucination by 9.0 pp, lowers the cross-configuration mean by 23.7 pp, and achieves 89.1% net improvement in paired-query evaluation across six benchmarks and multiple models.

  9. arXiv

    MLLM Hallucinations Arise When Information Distribution Drifts Inside Synergy Heads

    The work proposes HEAL, which applies causal noise intervention to multi-head outputs and counterfactual Difference-in-Differences to categorize attention heads into redundant, visual, language, and synergy types, finds that hallucinations occur when the visual-language information distribution inside synergy heads drifts away from a healthy equilibrium rather than correlating strongly with the quantity or strength of modality-specific heads, and accordingly injects dynamic calibration factors into the value vectors of synergy heads at inference time, reducing hallucinations across multiple MLLMs and benchmarks.

You’re all caught up