Skip to main content
Back to timeline
arXivSource publication:

Frozen features can still segment biomedical images: CAST skips training and beats the strongest baseline by 12.34 Dice points across eight benchmarks

Synopsis

In biomedical images, adjacent structures look alike, so patch-by-patch matching on frozen visual features can rank anatomically wrong regions above correct correspondences. CAST adds two tests to a frozen DINOv3 backbone: a stability test that uses feature variation under geometric transformations for global debiasing and patch-specific calibration, and a coherence test that checks candidate matches against neighborhood feature contrast and layout consistency. Across eight biomedical benchmarks spanning 2D and 3D, cross-subject and cross-modality transfer, CAST reaches 72.39% average Dice, 12.34 percentage points above the strongest baseline SegGPT at 60.05%, and ranks first in all six volumetric settings.

Source-provided article image: CAST: Calibrating Correspondences for Training-Free Biomedical In-Context Segmentation
arXiv

Interpretation

CAST lets biomedical in-context segmentation work without task training: a few annotated exemplars specify a new structure, and segmentation runs on frozen DINOv3 features. Earlier training-free routes matched exemplar and query patches directly by feature similarity, but adjacent anatomical structures in biomedical images share tissue appearance, intensity and texture, with boundaries set by anatomical organization rather than pronounced visual transitions. Across eight biomedical benchmarks, CAST averages 72.39% Dice against 60.05% for the strongest baseline SegGPT, a gap of 12.34 percentage points, and ranks first in all six volumetric settings.

The method replaces single-patch appearance similarity with a joint screen built from two kinds of evidence. The stability test applies seven fixed geometric transformations, including translations, rotations and isotropic scaling, to each source image, aligns transformed features back to the original patch grid, uses the residuals to debias dominant transformation-sensitive directions globally, and builds a patch-specific matching metric for each source patch. The coherence test compares center-relative feature differences around a query patch and a source candidate over eight directions at several radii, and checks neighborhood layout against target and non-target labels from the source mask; retrieved candidates are reranked by both kinds of evidence and then voted into a segmentation.

Each test helps on its own, and together they help more. On LPBA40 and CHAOS, the stability test alone gives 42.54% and 79.44% Dice, and the coherence test alone gives 42.58% and 79.67%. Combining both raises these to 44.88% and 81.51%, gains of 3.79 and 2.41 percentage points over either test alone, while direct nearest-neighbor label transfer gives 37.72% and 77.53% on the same two settings.

Perspective

Evaluation covers eight biomedical benchmarks: 2D chest X-ray, fundus, histology and fluorescence images, plus 3D pelvic CT, brain MRI and abdominal CT/MRI, including cross-subject and cross-modality transfer; volumetric tasks use five annotated slices from one source subject to guide segmentation of another subject's full volume. The atlas label transfer experiment moves Allen Mouse Brain CCFv3 atlas annotations onto DMBM fluorescence images across 11 brain regions under the PSD95 and SAP102 markers.

Cross-modality transfer is especially hard for appearance-based matching, because corresponding anatomical structures can show substantially different contrast and texture. Atlas label transfer is still affected by differences in imaging appearance and by often indistinct boundaries between neighboring brain regions.