Skip to main content
Search

# Multimodal Large Language Models

30 related articles

Subscribe to updates
  1. arXiv

    Images Stay in Context but Slip Out of Reach: VPR Shifts Only Cached Visual-Key Rotary Phases, Lifting 97 of 100 Model–Task Scores

    Keeping an image inside a multimodal model's context window does not mean its details stay usable: with 32K text tokens inserted between image and question, Qwen2.5-VL-7B's DocVQA score falls from 93.5 to 52.6 ANLS. Visual Position Re-Anchoring (VPR) shifts the rotary phases of cached visual keys toward a short-context reference before question processing, leaving visual values, text cache entries and causal masks untouched and adding no forward pass. Across ten checkpoints and ten benchmarks, static VPR improves 97 of 100 model–task scores; in the TextVQA length study, all six models remain below their short-context references at 32K, so recovery is incomplete.

  2. arXiv

    Pruned visual tokens keep the answer but shift the odds: DiFiT trains selection on the full model's response distribution

    Visual token pruning for multimodal large language models is usually judged by whether answers stay correct, yet pruning can change the probabilities of alternative responses even when greedy answers do not move. DiFiT learns which visual tokens to keep from both the full-token model's response distributions and reference answers, with one ranking serving several retention ratios. On Qwen2.5-VL-7B it cuts sampling-induced negative flips by 30.3% relative to FastV at 10% retention and preserves 98.74% of baseline performance on MME at 30% retention.

  3. arXiv

    Composed image retrieval stops squeezing queries into one vector: FactEmbed decomposes the edit first, then reranks candidates

    Composed image retrieval takes a reference image plus a modification text and must return the target image, and the common recipe compresses that multimodal query straight into one retrieval vector, leaving the model no explicit chance to work out what the text changes. FactEmbed inserts a set of learnable factorized latents into Qwen2.5-VL-7B's input and trains them with a target-imagination objective that reconstructs target details, so each latent grounds a semantic region, then carries that decomposition into retrieval through a factor-level contrastive loss and attention distillation. On the refined CIRR and FashionIQ benchmarks it scores above both CSMCIR and ReCALL on Recall@1 and average Recall@10, though the improvement margins stated in the text do not match the table values.

  4. arXiv

    Same object, new camera angle, wrong answer: LieVLM predicts local rotations from geometry tokens and lifts ScanRefer grounding accuracy by 6.9 points

    Multi-view 3D vision-language models suffer viewpoint-induced semantic drift: the same object and query produce weaker, diffuse or shifted target responses when the camera moves. LieVLM has geometry tokens predict local rotation parameters and channel-wise gates, applies length-preserving rotations to projected visual features within learned three-dimensional subspaces, and gates the identity-relative correction back into the original representation. On ScanRefer, LieVLM-8B reaches 64.5 Acc@0.25 and 57.2 Acc@0.5, 6.9 and 6.3 points above the published VG LLM-8B under the same proposal-refinement protocol.

  5. Brain voxels get more than one semantic unit: SAIL merges cross-layer sparse features for lower redundancy and higher alignment

    Similarity between multimodal large language model representations and human visual cortex shows that the two are alike but not which semantics they share. SAIL recasts voxel interpretation as a correspondence between each voxel and multiple sparse semantic units: it trains a unified cross-layer sparse autoencoder on several model layers, matches each voxel to several units, and compares the semantic similarity of their preferred image sets with optimal transport. Across eight subjects in the Natural Scenes Dataset, joint cross-layer training reduced unit response redundancy and raised semantic similarity over layer-wise training, in both whole cortex and selective regions of interest.

  6. arXiv

    A verifiable proving ground for egocentric AI assistants: humans act on instructions, and the best model tops out at 20% on long-horizon guidance

    Testing egocentric AI assistants has usually required real people, and the expensive, scarce interaction data left open whether today's multimodal models can actually help someone in the physical world. InteractiveWorld moves that loop into a verifiable environment: the assistant issues instructions, a human agent observes the scene, understands the instruction and acts independently, and the environment executes the action and updates what both see. InteractiveBench built on it holds 120 multi-turn tasks across six categories, each instruction shaping the outcome and scored by verifiable state-based rewards; across 12 frontier multimodal models, Claude Fable 5.1 reaches at most 20% and 25% on long-horizon Guidance and Planning tasks, respectively.

  7. arXiv

    Visual edits lose their grip mid-decoder: equal-strength interventions fade before answers settle, and calibrated routing turns that into attribute accuracy

    Applying the same visual-residual edit at every decoder layer with matched RMS strength reveals a persistent drop in effect, called the Evidence Anchoring Boundary; routing interventions by question type and calibration improves full-set attribute accuracy across three checkpoints, while universal subtraction is neutral or harmful.

  8. arXiv

    Multimodal models stop guessing at fine-grained recognition: tighter candidate lists plus contrastive hints lift 13 benchmarks by about 15 points

    Multimodal large language models associate visual cues with category concepts too weakly during fine-grained recognition; LIVE tightens the candidate category list and adds category contrast hints at inference time, raising Qwen2.5-VL-7B average accuracy by 14.96 points across 13 benchmarks.

  9. arXiv

    Visual reasoning drops coordinate regression: a probe token samples evidence over visual tokens, hitting 78.0% on 4K with about 1.2% of the tokens

    TiCO replaces coordinate regression in thinking-with-images with a probe token that samples evidence over visual tokens, reaching 78.0% overall accuracy on HR-Bench 4K while generating only 6.2 tokens per query.

  10. arXiv

    Visual token pruning drops single-score ranking: graph propagation plus degree correction keeps 98.7% accuracy at 30% tokens on LLaVA-NeXT

    Visual token selection in multimodal LLMs now propagates scores over a visual graph and corrects for neighborhood degree: across four backbones, ten benchmarks and seven pruning ratios, LLaVA-NeXT retains 98.7% accuracy while keeping 30% of visual tokens, with a net prefill speedup.