Skip to main content
Back to timeline
arXivSource publication:

MedJev turns medical image Q&A into one scoring pass: 7.93 points more average accuracy and a 6.04x end-to-end speedup over its Qwen3.5-4B backbone

Synopsis

MedJev replaces word-by-word answer generation in medical image understanding with a single forward scoring pass: given images, a question and candidate answers, the model scores every candidate and takes the highest, generating no reasoning or answer text. It keeps all visual tokens, removes only the upper language layers and the vocabulary projection, and scores candidates with pretrained query–key matching plus a zero-initialized residual head. Built on Qwen3.5-4B, MedJev-4B raises average accuracy across four medical benchmarks from 61.13% to 69.06% and cuts end-to-end latency from 685.03 ms to 113.44 ms.

AI-generated editorial illustration: MedJev: A System One Vision-Language Decision Model for Fast and Effective Medical Image Understanding

Interpretation

Medical image question answering no longer needs word-by-word generation: the model assigns each candidate answer a score, so a prediction is always one of the given candidates. Earlier vision-language models autoregressively generated reasoning and answer tokens, so inference cost grew with response length and the answer had to be parsed from free text, with a response naming no candidate counted as an error. On four benchmarks covering dermoscopy, pediatric chest radiographs, multi-image clinical cases and biomedical figures, MedJev-4B gains 7.93 absolute points of average accuracy over Qwen3.5-4B, which shares its vision encoder and connector, and lowers end-to-end latency from 685.03 ms to 113.44 ms.

The speedup comes from shortening the language-side path rather than discarding visual information: the vision encoder, the connector and every visual token are kept, and only upper language computation the decision does not use is removed. Token-pruning methods accelerate by discarding visual tokens, which may remove small lesions and fine structures, while layer-pruned models still generate answers token by token. MedJev-4B keeps the first 23 language layers of Qwen3.5-4B and only the query and key projections of layer 32, reducing parameters from 4.44B to 3.55B; in the ablation, depth reduction costs no accuracy under either scoring head, raising the mean from 68.31% to 68.70% and from 68.93% to 69.06%.

Candidate scores come from pretrained query–key matching, with a lightweight multilayer perceptron adding a residual correction. Pretrained matching alone may not separate heterogeneous medical images and questions, while replacing matching with a per-candidate MLP reaches only a 68.83% mean. The residual head has hidden width 256 and a zero-initialized output layer, adds 1.97M parameters and about 1.4 ms, and lifts the mean to 69.06%; against 20 general and medical vision-language models at the 4B–9B scale, MedJev-4B is both the most accurate and the fastest on average.

Perspective

The demonstrated scope is fixed-candidate selection on four medical image understanding benchmarks with two to seven candidates, covering dermoscopic diagnosis, pediatric chest radiograph binary classification, multi-image clinical cases and biomedical figure questions; latency is measured on a single NVIDIA H20 at batch size one, including image loading and preprocessing. The authors propose extending the work to more general vision-language decision models and complex visual perception tasks.

Among the four benchmarks, the 1.44-point gain on PneumoniaMNIST has a paired bootstrap interval spanning zero, while the other three gains stay clear of zero; permuting candidate order lowers the average accuracy from 69.06% to 68.10%, with larger losses on the free-text-option benchmarks. The authors list extension to more general scenarios as future work.