Skip to main content
Back to timeline
arXivSource publication:

A frozen vision-language model picks labels but never touches pixels: SEER reaches 58.0 average mIoU across eight segmentation benchmarks

Synopsis

Residual errors in open-vocabulary semantic segmentation are mostly local, and letting a vision-language model edit pixels directly is unreliable. SEER splits the repair in two: a frozen vision-language model only verifies a small set of score-derived candidate labels in closed set, while pixel ownership stays with the segmentation model's own score field. Across eight standard benchmarks it reaches 58.0 average mIoU with ViT-L/14 without updating segmentation weights.

Source-provided article image: SEER: Semantic Evidence for Executable Region Propagation in Open-Vocabulary Semantic Segmentation
arXiv

Interpretation

Repair in open-vocabulary semantic segmentation is recast as an executable action that must supply a visually supported label, keep that label legal under the active vocabulary, and place it on the right pixels. Earlier training-free methods mainly strengthened the initial prediction by modifying CLIP attention, calibrating patch-text similarity or injecting visual priors, so the bottleneck shifted from global recognition to local residues such as boundary spillover, object-part confusion and stuff/object ambiguity. In a diagnostic that fixes the prediction map and score tensor and applies only the evidence-to-execution layer, all eight benchmarks improve, indicating the gain does not come from a change of backbone, resolution or training recipe.

The vision-language model only selects one label from the candidate set or abstains; it neither predicts nor edits a mask. The candidate set is built from the frozen score tensor and by default holds the current base label, up to three non-base alternatives and an uncertain option, so labels outside the vocabulary are not executable. On the four mechanism benchmarks the default top-3 candidates put the target label in the set 99.9% of the time; filtering raises decision precision among executed answers from 57.9% before execution to 82.3%, with 36.2% of audits leaving the prediction unchanged and only 0.2% invalid or out of set.

Once a label is certified, the segmentation score field decides how far it propagates: coherent high-confidence regions may propagate as a whole, while object or boundary-risk regions get only conservative score-supported propagation. Under matched inputs, a score-only rule, an ordinary CLIP crop classifier, a vision-language answer with score support only, and Dense-CRF smoothing average gains of 2.64, 3.69, 1.70 and 0.01 mIoU respectively, all below the full SEER gain of 7.52. The same pipeline applied unchanged to the ProxyCLIP, ResCLIP, Trident and CorrCLIP bases improves every base-benchmark pair, with mean gains between 4.53 and 5.93 mIoU.

Perspective

The demonstrated scope is offline or selective repair on eight natural-image segmentation benchmarks, using a fixed SAM2 proposal bank and a locally deployed Qwen3.5-9B verifier, with coefficients and thresholds selected once on the evaluated domains and then frozen. In the measured configuration the complete system adds 9.3% cached or 19.3% cold latency, and the verifier peaks at 18.7GB of memory.

Residual errors concentrate in mixed proposals, thin structures, candidate misses and fragmented score support; the gate and conservative route reduce harmful writes, but some difficult regions remain unchanged. The verifier's public documentation provides no sample-level pretraining manifest, so zero overlap with segmentation or grounding data cannot be assumed, and the authors instead test replacement by three other vision-language model families and CLIP. The authors also state they do not claim domain-blind development or medical or remote-sensing validation.