Skip to main content
Back to timeline
arXivSource publication:

Radiology contrastive pretraining drops paired-identity targets: CorLIP lifts sentence-level AUC from 68.2 to 81.6 and keeps large batches trainable

Synopsis

Radiology image-text contrastive pretraining has inherited CLIP's paired-identity target, treating every other report in a batch as a negative, yet radiology reports share many of the same findings, so that target is systematically biased. CorLIP instead counts a text as positive whenever its clinical assertions agree with the image, changing only the target matrix and leaving architecture and optimizer untouched. On MIMIC-CXR, sentence-level CorLIP raises zero-shot 14-class mean AUC from 68.2 for vanilla CLIP to 81.6 and largely removes the post-peak decline; the same correction transfers to 3D CT.

Source-provided article image: Rethinking Contrastive Targets in Radiology Language-Image Pretraining
arXiv

Interpretation

The definition of a positive pair in radiology image-text contrastive pretraining is rewritten: texts whose clinical assertions agree with the image count as positives, not only the paired report. The prior recipe inherited CLIP's paired-identity assumption, treating every non-paired text in a batch as a negative, even though radiology reports share one anatomy and a small finding vocabulary, so unpaired reports often agree with an image on most assertions. On MIMIC-CXR, sentence-level CorLIP reaches 81.6 zero-shot 14-class mean AUC against 68.2 for vanilla CLIP at sentence level; vanilla loss keeps falling while AUC declines late in training, whereas CorLIP sentence-level drops only 0.2 AUC from peak to final checkpoint.

The correction replaces only the contrastive target matrix; the image encoder, text encoder, optimizer and training schedule stay fixed. The image-side fact state is latent in real data, so an offline parser extracts structured facts from the paired report and compatibility is scored against those assertions. Chest X-ray experiments use a DINOv3-initialized ViT-B image encoder and a BioClinicalBERT text encoder, trained on frontal MIMIC-CXR images and evaluated on MIMIC-CXR, CheXpert, VinDr and MS-CXR.

Correcting the target turns large-batch training from collapse into a usable regime and lets data scale start to pay off. Vanilla medical CLIP at large batch stays near random AUC with loss near the uniform-softmax floor, and common chest X-ray batch sizes are often below 128. Sentence-level CorLIP still trains at batch 2048, with peak AUC rising up to 2048 and then staying stable; doubling data from roughly one scale to two raises corrected sentence-level peak AUC from 81.6 to 86.5, while the best vanilla checkpoint moves only from 68.2 to 70.6.

Perspective

The findings rest on frontal chest X-ray pretraining on MIMIC-CXR plus a transfer experiment on Merlin 3D CT; evaluation covers zero-shot classification and MS-CXR phrase grounding, not report generation, segmentation or supervised fine-tuning. Compatibility is defined by fixed 13-label or 30-label fact schemas, and the schema decides which clinical distinctions are preserved.

The authors state that no multi-seed error bars or formal significance tests are reported, and the conclusions rely on consistent trends across granularities, datasets and batch sizes. A coarse schema may merge left- and right-sided findings and weaken laterality, while a fine schema may make compatible sets sparse; the two schemas differ in both granularity and extractor, so the gap between them cannot be attributed to granularity alone.