TAP keeps CLIP frozen and still learns new classes, reaching 83.05% final-stage accuracy on ImageNet-R
Synopsis
Class-incremental learning usually keeps changing pre-trained representations through prompt learning or adapters, at the cost of optimization overhead and disturbance to old-class feature structure. TAP freezes both CLIP encoders and accumulates each class's mean and within-class scatter in the frozen embedding space, while learning class-specific text-template weights from current-task images. Across nine datasets under two incremental protocols, TAP attains the highest average incremental and final-stage accuracies among the compared methods, with 83.05% final-stage accuracy on ImageNet-R under B0/Inc20.
Interpretation
TAP removes continual adaptation of the CLIP encoders from class-incremental recognition: both image and text encoders stay frozen, and new-class information accumulates as class statistics plus calibrated text representations. Earlier CLIP-based incremental routes either introduced task information through prompt learning or adapters, which continually update shared parameters, or represented each class only by its mean prototype, which does not explicitly capture within-class variation. Across nine datasets and two incremental protocols, 18 dataset-protocol settings in all, TAP has the highest average incremental and final-stage accuracies among the compared methods; under the B0 protocol the two metrics reach 86.08% and 80.55%, exceeding the strongest baseline in the table by 2.21 and 2.93 percentage points.
On the visual side, each class keeps a mean and a within-class scatter, and the class-specific covariance is shrunk toward a covariance shared by all seen classes before a Gaussian discriminant score accounts for both distance and distribution volume. A class-specific covariance captures the directions of within-class variation, but its estimate is unreliable when training samples are few relative to feature dimension, and shrinking toward the shared covariance suppresses that sampling noise. In a cumulative ablation on ImageNet-R under B0/Inc20, adding the shared covariance raises average incremental accuracy by 7.921 percentage points, the largest gain among the components.
On the semantic side, each class learns a set of text-template weights, with current-task images supervising the weights of new classes, and the visual and semantic posteriors are then combined by a weighted product of experts. CLIP commonly converts class names into text descriptions with predefined templates, yet different templates suit different classes unevenly, and old-class text representations stay unchanged. In the same ablation, visual-semantic posterior fusion adds a further 1.343 percentage points over the shared covariance, and class-adaptive semantic calibration adds another 0.144; at the final ImageNet-R stage fusion turns 3.10% of samples from incorrect visual predictions into correct ones, a net gain of 1.78 percentage points after the 1.32% reverse change.
Perspective
The demonstrated scope is the exemplar-free class-incremental setting on nine image-classification datasets, using a frozen CLIP ViT-B/16 with neither image nor text encoder updated and no old-class images retained after training. The intended readers are researchers and engineers working on continual learning and vision-language model reuse.
The method relies on a class-conditional Gaussian approximation and shrinkage toward a shared covariance, and the material gives no conclusion on how its benefit changes when training samples are even fewer relative to feature dimension. The hyperparameter analysis only shows that neighboring values keep accuracy high on ImageNet-R under B0, which is local robustness within the evaluated neighborhood.
