No target images, still segmenting rainy nights: CROMPT adapts CLIP with cross-modal prompts for adverse weather
Synopsis
Semantic segmentation models lose accuracy in unseen conditions such as night, rain and fog, and zero-shot domain adaptation addresses this using only textual descriptions of the target domain, with no target images during training. CROMPT combines learnable textual prompts with fixed random visual prompts to fine-tune the CLIP vision encoder and adapts in an easy-to-hard two-stage order; with Cityscapes as source and evaluation on ACDC and GTA5, it averages 2.63% and 1.63% mIoU above PØDA and ULDA on ACDC while keeping inference cost equal to the baseline model.
Interpretation
Segmentation accuracy drops when a model meets scenes outside its training distribution, such as night, rain, snow or fog, and CROMPT lets a single model adapt to these unseen scenes from textual descriptions alone, without knowing at inference which domain an image belongs to. PØDA required the target domain identifier in advance and a separate model per target domain guided by CLIP text embeddings, so inference had to identify the domain before selecting a model; ULDA unified the model but still relied on frozen encoders without fine-tuning for the downstream segmentation task. With Cityscapes as the source domain and evaluation on ACDC and GTA5, CROMPT averages 2.63% and 1.63% mIoU above PØDA and ULDA on ACDC and 0.93 and 0.91 mIoU above them on GTA5; in source-to-night and source-to-rain settings, CROMA-VL improves over ULDA by 2.7% and CROMPT gains 2.45% in source-to-rain.
Adaptation runs in two steps: the CLIP vision encoder becomes trainable while fixed-position random visual prompts are added to the input image, and textual prompts from several domains pass jointly through the frozen CLIP text encoder so each iteration sees multiple candidate styles. Random Visual Prompting (RVP) differs from standard visual prompting, which freezes the encoder and optimizes only continuous prompt parameters added to the input; RVP patches are non-learnable and act as noise, while the CLIP vision encoder itself is fine-tuned, so no trainable parameters are added. Textual prompting is hybrid, with learnable soft prompts for some domains and hard-coded descriptions such as "driving at night" for others, and the two domain sets are mutually exclusive; in the visual-prompt ablation, non-learnable random patch prompts reach 39.65 mean mIoU, learnable corner-plus-center prompts give 39.62 and learnable padded prompts give 39.34.
When a large vision-language model is inconvenient to call, CROMPT switches to an easy-to-hard two-stage adaptation: stage one uses learnable prompts for hard domains and hard-coded descriptions for easy domains, and stage two swaps their roles and continues from the intermediate model of stage one. CROMA-VL instead uses Qwen-VL to generate domain-conditioned descriptions, but ablation shows simple hard-coded prompts perform best overall, and extending adaptation to three stages drops performance back to near the ULDA baseline. In the two-stage ablation, the structured progressive adaptation strategy used by CROMPT achieves the best overall performance among the prompt combinations, while extending to three stages consistently lowers performance; CROMA-VL exceeds PØDA and ULDA on ACDC by 2.33% and 1.55%.
Perspective
Evaluation uses Cityscapes as the source domain, the official 406-image ACDC validation split and a 1,000-image GTA5 subset, with all datasets sharing a 19-class label space and target images processed at native resolution at test time; the method builds on DeepLabv3+ with a CLIP-ResNet-50 backbone, the adaptation stage updates the segmentation head and classifier, and CROMPT's first stage also fine-tunes the image encoder.
Ablation shows that adding Qwen-VL does not always help, that simple hard-coded prompts perform best overall, and that extending adaptation to three stages drops performance to near the ULDA baseline; these results come from the two benchmark datasets and a single source-domain setting.
