CLIP incremental learning stops relying only on [CLS] and [EOS]: PACE feeds patch and token detail into the decision and leads last accuracy on nine benchmarks
Synopsis
CLIP-based class-incremental learning has mostly matched the global visual [CLS] and textual [EOS] representations across modalities, leaving patch-level visual detail and token-level textual semantics out of the final match. PACE selects the top-k visual patches and textual tokens most related to the global representations from the frozen CLIP encoders, injects that local information into the global representations through a cross-modal fusion module, and constrains updates of the shared fusion weights with subspace projection. Across nine benchmark datasets, PACE reports higher last and average accuracy than the listed comparison methods, for example a last accuracy of 93.94 on Cars B0 Inc10.
Interpretation
CLIP-based class-incremental learning starts bringing patch-level visual features and token-level textual semantics into the classification decision instead of relying only on the two global [CLS] and [EOS] representations. Earlier CLIP-based incremental methods kept the original global-alignment paradigm, optimizing learnable prompts or refining the global [CLS] and [EOS] representations, so the patch and token sequences went unused at the final matching stage. On nine benchmark datasets, PACE reports higher last and average accuracy than the listed comparison methods; under Cars B0 Inc10 its last accuracy is 93.94, above PROOF at 90.74 and HERMAN at 88.30.
The method first scores local features against the global representation, then lets each modality's global representation draw information from the other modality's local sequence. The score is the normalized affinity between each patch or token and the global representation, and the top-k most discriminative local features are selected; the visual side recomputes this every forward pass, while the textual side can precompute and reuse it because the prompt template is fixed. Cross-modal fusion uses single-head attention: the global visual feature queries the local text tokens, the global text feature queries the local visual patches, and the fused global and local features feed separate contrastive losses that are combined with a weight.
The shared fusion weights are easily rewritten by gradients from new tasks, so the method protects the parameter directions critical to old tasks with subspace projection. Each new task adds and trains only its own visual and textual projectors, historical projectors stay frozen, and their outputs are accumulated before entering the fusion module. After each task the fusion weights are decomposed by SVD, an orthogonal projection matrix is built from the minimum rank whose cumulative energy reaches a threshold of 0.9, and at the end of training the new weights are combined with a weight over the old-knowledge subspace and its orthogonal complement; the Cars B0 Inc10 ablation shows that removing subspace projection has a milder early effect but a rapid decline later as interference from new tasks accumulates.
Perspective
Evaluation covers nine benchmarks, CIFAR100, CUB200, ObjectNet, ImageNet-R, FGVCAircraft, StanfordCars, Food101, SUN397 and UCF101, with B-Inc class splits and a ViT-B/16 backbone; the main results use CLIP weights pretrained on LAION-400M, with an additional comparison using OpenAI pretrained weights on UCF B0 Inc10.
The textual branch uses a fixed prompt template, which may limit the model's ability to capture semantic diversity; the authors list richer prompt engineering, such as learnable prompts, as future work. PACE++, which replaces the fixed template with GPT-5.4-generated class-specific visual descriptions, performs comparably overall and improves on several datasets.
