Skip to main content
Search

# Visual Question Answering

8 related articles

Subscribe to updates
  1. arXiv

    Alibaba DAMO's HoloCT fuses CT reading and 3D segmentation in one model, surpassing all four radiologists in sensitivity and overall accuracy in a multicentre reader study

    Alibaba DAMO Academy's HoloCT is a 3D CT foundation model that puts scan interpretation and voxel-level segmentation inside one vision–language model, so a textual finding can come with a checkable 3D mask. Previously, segmentation networks delineated without interpreting, and CT vision–language models wrote text without voxel-level localization. On 11,455 clinical CT scans from five hospitals, HoloCT-8B raised mean Dice over VoxTell from 0.107 to 0.488 on laryngeal lesions and from 0.543 to 0.715 on lung cancer, and in retrospective reader studies of thyroid tumours and colon lesions it exceeded all four participating radiologists in sensitivity and overall accuracy.

  2. Multimodal explanations without model internals: MultiSHAP scores each image-patch and text-token pair

    Multimodal models answer questions and retrieve images, yet which image region and which word jointly push a given output score has been hard to quantify. MultiSHAP applies the Shapley Interaction Index from cooperative game theory to estimate, for every image-patch and text-token pair, a second-order synergy or suppression score, using only a query interface that allows separate perturbation of image and text and returns a task-relevant scalar. On 500 VQAv2 samples, deleting patches in the resulting ranking lowered the target-answer probability by 0.1720 on average, versus 0.0032 for the lowest-ranked patches.

  3. arXiv

    Driving VLMs stop distilling every token alike: pick the critical phrases, then trust the teacher only where it agrees

    Knowledge distillation for driving vision-language models previously matched teacher distributions uniformly across answer tokens, diluting planning-relevant semantics and passing on confident teacher errors. RSKD first localizes task-critical phrases in the ground-truth answer, such as traffic participants, states and spatial relations, then uses a frozen sentence encoder to score how semantically consistent the teacher's prediction is with each phrase, so consistent positions take more teacher-distribution supervision and inconsistent ones take more ground-truth supervision. On DrivingVQA, LingoQA and MapLMv2, RSKD reaches 16.60, 0.672 and 81.82, beating the compared SFT, GKD and ToDi student baselines on the two QA benchmarks and nearly matching SFT's 81.85 on MapLMv2.

  4. Fireworks AI

    Llama 3.2 lands on Fireworks: the 11B and 90B vision models cost the same as text models, with images billed as 6,400 text tokens

    Fireworks announced support for Meta's Llama 3.2 family, covering the text-only 1B and 3B models and the multimodal 11B Vision and 90B Vision models, with inference and fine-tuning available on its platform. That generation was not previously offered there; the 11B and 90B multimodal models are now priced the same as text-only models, and images are charged as text tokens, typically counted as 6,400 text tokens for the Llama 3.2 vision models. Fine-tuning is currently open for the 1B and 3B text-only models, while multimodal fine-tuning is not yet available.

  5. arXiv

    ΔRepresentation models phenotypes as representation increments over normal anatomy via counterfactual reasoning, improving lesion grounding and phenotype recognition across six medical VLMs

    The work proposes ΔRepresentation, a visual phenotype representation learning framework for medical VLMs: BaseAnatomy applies geometry supervision to normal anatomical representations using ideal spatial distributions, while Phenotype uses counterfactual reasoning to estimate a normal anatomical reference for each pathological patch and learn phenotype-specific representation increments that cluster by phenotype. On ReXGroundingCT and LIDC-IDRI, it consistently improves lesion grounding and phenotype recognition across six pretrained medical VLMs.

  6. arXiv

    STAVE replaces in-context demos with two task vectors, beating per-layer interventions on six multimodal models at zero-shot inference cost

    The work proposes STAVE, which replaces in-context demonstrations with two task-specific vectors added to the input embeddings of a frozen LLM or LMM, a readout vector updating the answer-producing tokens and a context vector updating the other structural token groups, both trained with answer labels on prompts with and without demos; across six LMMs and five LLMs, STAVE matches or outperforms per-layer interventions with far fewer task parameters, surpasses 15-shot ICL and prior task vectors on 18 text tasks, and runs at zero-shot inference cost.

  7. arXiv

    CLIMB improves multimodal RAG with a confidence-guided complementary evidence pool, consistently beating retrieval-augmented baselines on Encyclopedic-VQA and InfoSeek

    The authors propose CLIMB, a training-free inference-time multimodal RAG framework that first builds a compact complementary evidence pool with an MMR-style objective balancing query relevance and passage-level redundancy, then performs confidence-controlled refinement within this fixed pool: an R/E/C critic scores passages by relevance, evidence specificity, and cross-modal alignment, while an evidence-grounded confidence estimator accepts an updated answer only when estimated confidence increases, yielding a simple stopping criterion; on Encyclopedic-VQA and InfoSeek it consistently improves over retrieval-augmented multimodal baselines, and ablations indicate complementary pooling, critic-based scoring, and iterative confidence-controlled refinement each contribute.

  8. arXiv

    Multimodal Flow unifies language and vision as continuous hyperchunks under one flow-matching backbone, with MF-1 scoring 0.821 on GenEval and 83.44 on DPG-Bench at 150B tokens

    The work introduces Multimodal Flow, which encodes text blocks and images as ordered continuous hyperchunks and models both modalities in their own embedding spaces with a shared chunk-causal Flow Matching backbone, instantiated as MF-1: continued pretraining improves multimodal modeling across 0.6B, 1.2B, and 1.6B scales, and the 1.6B model reaches 0.821 on GenEval and 83.44 on DPG-Bench with only 150B pretraining tokens and an average of 75.3 across VQAv2, MMBench, and POPE, outperforming representative hybrid and discrete models under matched data, optimization, and parameter budgets.

You’re all caught up