Thermal images no longer break CLIP's language alignment: EmergeFuse turns heat cues into unseen-class fixes, lifting unseen mIoU by up to 4.755 points
Synopsis
Visible-infrared fusion can carry thermal information into an image, but CLIP is pretrained mainly on RGB, so adding infrared directly may disrupt vision-language alignment and turn thermal cues into semantic interference. EmergeFuse anchors on the RGB prediction, selects class-relative thermal evidence that can correct RGB errors, and writes those corrections back into the fused image through text contrasts. On FMB, MFNet and SemanticRT it raises unseen-class mIoU by 3.308, 4.755 and 3.046 points over the strongest conventional fusion baselines, with unseen strict complementary conversion up by as much as 15.730 points while fusion quality stays competitive.
Interpretation
Infrared-visible fusion is now used to improve open-vocabulary segmentation of unseen classes, not only image appearance or seen-class tasks. Earlier fusion methods tied fusion to seen-class supervised tasks through task losses or downstream features, and did not separately address whether thermal information helps recognize classes absent from training. On FMB, MFNet and SemanticRT, EmergeFuse reaches unseen mIoU of 33.772%, 37.849% and 30.194%, respectively 3.308, 4.755 and 3.046 points above the strongest conventional fusion baselines.
Infrared evidence enters the fused image only when it can correct the current RGB judgment, rather than being blended in wholesale. Earlier fuse-then-recognize pipelines did not distinguish whether a thermal cue supports or refutes the current RGB-predicted class, so complementary evidence could fail to fix errors while misleading cues could disrupt reliable RGB semantics. On FMB the full model reaches an unseen strict complementary conversion rate of 89.349%, 14.058 points above the strongest competitor; removing the evidence calibrator or the semantic gate lowers it to 83.960% and 82.804%.
Class-relative cue selection takes the highest-scoring RGB class at each pixel as reference, compares RGB and infrared semantic responses with calibrated thermal evidence, and competitively selects a class to correct or a no-rectification option. This step turns thermal evidence into signed class correction requests and keeps a no-rectification option, so weak complementary evidence favors preserving the RGB prediction. On FMB the cue-selection-only variant reaches 30.868% unseen mIoU and 36.579% hIoU, above the 26.384% and 32.294% of the reconstruction-only basic variant.
Text-contrast image rectification translates the selected correction requests into text-contrast directions, updates multi-scale fusion features through an evidence gate, and the same frozen open-vocabulary reader then segments the fused image. The correction acts on fused-image features rather than directly on segmentation predictions, so no retraining is needed for unseen classes. On FMB the full model reaches 33.772% unseen mIoU and 39.379% hIoU, above the 32.118% and 37.839% of the variant with semantic supervision and text-conditioned feature modulation.
Perspective
Experiments cover the FMB, MFNet and SemanticRT benchmarks under a seen-to-unseen protocol, where pixel-level semantic supervision is restricted to seen classes and unseen classes are evaluated without retraining; all fusion methods use the same text queries and the same frozen open-vocabulary reader.
Open-RGBT is excluded from the quantitative comparison because its official implementation is unavailable; cross-dataset zero-shot results are reported only for FMB-trained models transferred to MFNet and SemanticRT; under Hard+8 distractor classes, unseen mIoU drops by 1.355 points while unseen strict complementary conversion stays at 85.906%.
