Skip to main content
Back to timeline
arXivSource publication:

General vision-language models skip medical retraining: VERA visual harness lifts spleen measurement from 63.3% to 98.3%

Synopsis

VERA is a medical visual harness for general vision-language models that supplies interactive image tools, lossless evidence memory and evidence verification at inference time, so the model inspects images repeatedly and revises its answer. Across 11 tasks and four imaging modalities, the Qwen3.8-27B backbone improves on every task, with spleen measurement rising from 63.3% to 98.3% and pulmonary nodule detection from 0.0% to 28.2%. Stronger API-based models gain more with VERA, and GPT-6.1 Sol raises pulmonary nodule detection from 10.8% to 69.2%.

Source-provided article image: Visual Harness Scales General VLMs to Medical Images
arXiv

Interpretation

General vision-language models perform weakly on medical images, and VERA raises the same model's scores across the board by giving it a workflow for inspecting images and checking evidence at inference time, without changing model weights. Earlier routes ran mainly two ways: continued pretraining or fine-tuning on medical data to build specialized medical vision-language models, or agent frameworks that call tools; both emphasized domain specialization. On the Qwen3.8-27B backbone, VERA beats direct image input on all 11 tasks and reaches best or tied-best performance among agent baselines on 10; pulmonary nodule detection goes from 0.0% with direct input to 28.2% with VERA.

VERA splits a reading into a loop of exploration, answering, verification and re-inspection: the model first inspects images and gives an initial answer, then uses coverage and evidence reliability to decide whether to keep looking. In a conventional tool-calling loop the model decides when to stop, and stopping does not mean the image was adequately inspected or the finding reliably confirmed. The loop runs at most three verification rounds and ends when the estimated number of potentially undetected findings falls within tolerance and no candidate remains undecided; evaluation covers 11 tasks across CT, MRI, ultrasound and whole-slide pathology.

VERA has five components: medical visual tools for navigation, magnification, multiplanar viewing and quantitative measurement; lossless evidence memory that keeps inspected images and measurements verbatim; an evidence map that estimates how many findings each region may still hide; an evidence verifier that maintains a belief for each candidate finding; and an evidence-guided agentic loop that feeds verification back into the next round of inspection. A binary visited/unvisited record is insufficient to describe inspection progress, because the same region yields different detail at different resolutions. Ablation shows spleen measurement falling from 98.3% to 70.0% without the evidence verifier and pulmonary nodule detection from 28.2% to 14.1% without the evidence map, with all 11 tasks dropping when both are removed.

Perspective

Evaluation is limited to public datasets, covering 11 tasks across CT, MRI, ultrasound and whole-slide pathology in four categories: measurement, localization plus measurement, detection and diagnosis; all results are offline benchmark scores and were not validated in real clinical environments.

The four API models used reduced subsets for pulmonary nodule detection, brain tumour MRI and echocardiography because of API costs, limiting the scope of cross-backbone comparison; overlap between public datasets and proprietary model training data cannot be ruled out; VERA's performance still depends on the backbone model's ability to interpret medical images and execute tool calls reliably.