PhaseCLIP brings the phase spectrum into frequency-domain adaptation of a frozen CLIP: 84.28% mean accuracy across five remote-sensing benchmarks and 4.36 points on the hardest one
Synopsis
PhaseCLIP extends frequency-domain adaptation of a frozen CLIP visual encoder from the amplitude spectrum to the phase spectrum, and lets one set of module weights learn phase first and refine with amplitude, so no parameters are added. Under a matched protocol on EuroSAT, PatternNet, UC Merced, RSSCN7 and LSRS, with 1 to 32 shots and eight baselines, it reaches 84.28% mean accuracy, 1.93 points above the amplitude-only FreqDIMFT and 4.36 points above it on the 113-class LSRS.
Interpretation
Few-shot remote-sensing scene classification gains a further source of information: a frozen CLIP visual encoder can now read the phase spectrum in the frequency domain, not only the amplitude spectrum. Earlier frequency-domain adaptation used the amplitude spectrum or unseparated frequency coefficients, leaving the phase component that carries spatial structure unused on its own. Under a matched protocol on five remote-sensing datasets, 1 to 32 shots and three random seeds, PhaseCLIP averages 84.28% accuracy, against 82.36% for amplitude-only FreqDIMFT and 79.63% for the strongest non-frequency baseline, Tip-Adapter-F, and is best in 23 of 30 dataset-by-shot cells.
The method inserts frequency modules into a frozen CLIP ViT-B/16 visual encoder, and each module can select the amplitude spectrum, the phase spectrum, or both in parallel. The modules take a two-dimensional Fourier transform of image patches and transformer token grids, compress the spectrum by radial average pooling, and train only about 1.30% of the parameters. The amplitude and phase pathways share an identical parameter layout, so one weight set can be trained on phase and transferred to amplitude for a second refinement stage without adding parameters; the parallel two-branch PhaseCLIP-SB instead trains 2.00% of the parameters.
How phase is combined with amplitude matters as much as whether phase is used. The parallel PhaseCLIP-SB gains only 0.53 points over FreqDIMFT on average, while the two sequential transfer variants gain 1.84 and 1.93 points. On LSRS, amplitude-only FreqDIMFT ranks fourth at 59.24%, behind CoOp at 62.52%; adding phase raises it by 4.36 points to a statistical tie with Tip-Adapter-F at 63.62%, the largest phase gain across the five datasets.
Perspective
The conclusions rest on five public remote-sensing scene-classification datasets, a CLIP ViT-B/16 backbone, 1 to 32 shots and a fixed 40/60 train-test split, with one augmentation shared by all methods; the single prompt template is shared only by zero-shot CLIP, the adapter and cache methods and the frequency-domain methods, while prompt-learning methods use their own learned prompts, and optimisation hyperparameters follow each method's authors. The authors propose extending the modules to other sensors, tasks and domains, but this study does not test those settings.
On LSRS, PhaseCLIP's 63.60% mean is a statistical tie with Tip-Adapter-F at 63.62%, and it trails that baseline at 1, 4 and 8 shots, overtaking only from 16 shots; RSSCN7 is near saturation and its 32-shot cell goes narrowly to amplitude-only FreqDIMFT; augmentation gains fall monotonically with shot count and are slightly negative in a few high-shot cells.
