Skip to main content
Back to timeline
arXivSource publication:

Frozen DINOv3 heads already separate lesions; SMIA trains just 0.51M parameters to reach 80.99% mean mIoU across 20 medical and industrial tasks

Synopsis

Cross-domain few-shot segmentation transfers natural-image pretrained encoders to medical or industrial images using only a handful of pixel-annotated support masks. SMIA inserts a conditional branch after each attention head's aggregation inside a frozen DINOv3, letting the receiving patch's pre-attention local state decide how that head's message updates, so similar contextual messages can produce different or opposing corrections at different positions. Across 20 tasks covering five medical datasets and all 15 MVTec AD categories, it trains only 0.51M parameters and reaches six-dataset mean mIoUs of 77.40% at 5 shots and 80.99% at 10 shots, 3.61 and 3.93 percentage points above the strongest prior method at each budget.

Source-provided article image: State-Message Interaction Adapter for Cross-Domain Few-Shot Segmentation
arXiv

Interpretation

The adaptation point moves from the encoder's end into each Transformer block: pretrained attention and FFN parameters stay frozen, and a lightweight branch after each head's aggregation lets the same contextual message be rewritten into different residuals depending on the receiving position. Earlier approaches either match frozen features against support exemplars without training or train multi-scale decoders on frozen layer outputs, so support masks affect prediction only after the encoder finishes token processing; parameter-efficient tuning such as LoRA applies token-invariant low-rank increments to selected projections. On BUSI ultrasound, a single frozen DINOv3 Block-9 head already separates the lesion from surrounding stroma, whereas under the same 10-shot support set the frozen probe and LoRA produce diffuse final-layer similarities and miss nearly the entire lesion; on MVTec Capsule the frozen head responds to both the crack and nearby printed digits, the frozen probe over-segments the shell, and LoRA keeps only a small fragment of the crack.

The update of each head's message is written as a linear function of the receiving patch's local state, then confined by a low-rank bilinear map shared across patches and images to low-dimensional state, message and output subspaces. Parameterizing this conditional relation freely in ambient dimensions would require a full weight set per head, while scarce support masks cover only limited combinations of state and message and cannot support that parameter count. On DINOv3 ViT-S+/16, each head's weights comprise state projection, message projection, diagonal pairing and output projection, giving 511,873 trainable parameters together with a 385-parameter linear head; the update sits before each adapted block's frozen FFN, which with deeper blocks refines it for a single linear readout.

Across the 20 tasks, six-dataset mean mIoU is 77.40% at 5 shots and 80.99% at 10 shots, ranking first at both budgets and leading on 16/20 and 18/20 tasks respectively. Under the same fixed-support protocol the strongest prior methods are HERA at 73.79% for 5 shots and DINO-AugSeg at 77.06% for 10 shots, so SMIA is 3.61 and 3.93 percentage points higher while using a ViT-S+/16 backbone smaller than those of INSID3, HERA, RSRM and DINO-AugSeg. On MVTec AD the lead widens to 4.98 and 5.38 percentage points with industrial foreground precision of 82.14% and 83.55% and recall 9.35 and 11.20 points above DINO-AugSeg; prior methods lead on two 10-shot tasks, Kvasir-SEG with HERA and RSRM at 83.45% and 83.00% versus 82.21%, and MVTec Transistor with DINO-AugSeg at 65.33% versus 63.03%.

Perspective

The demonstrated scope is 20 tasks spanning five medical datasets and all 15 MVTec AD categories, with support sets built from five nested random draws, all methods evaluated on identical support sets and the same native-resolution query sets, and SMIA by default adapting all 12 Transformer blocks. The material explicitly proposes combining local state-conditioned updates with non-local relational priors to capture global part geometry as a next step.

On MVTec Transistor, recall stays conservative for missing or displaced leads, where exposed board regions can appear locally intact, despite 78.22% foreground precision; prior methods obtain higher mIoU on two 10-shot tasks, Kvasir-SEG and MVTec Transistor. As subspace rank grows from low to high, three-dataset mean mIoU spans only 79.24% to 79.50%, which the material reads as rapid rank saturation.