Skip to main content
Back to timeline
arXivSource publication:

PatchAdaNet does class-incremental learning without storing old samples: frozen patch features plus adapters reach 65.9% on CIFAR-10

Synopsis

PatchAdaNet splits each image into patches, trains an independent patch autoencoder per position on external data and freezes them permanently, then uses learnable adapters to build whole-image representations, enabling unsupervised class-incremental classification without storing any past samples. A fixed-size Global Feature Centroid, a running average of adapter outputs, penalizes representation drift, and EWC constrains the adapters. On the 5-split CIFAR-10 protocol the CNN and ViT variants reach 65.9% and 64.5% respectively, both without a replay buffer.

Source-provided article image: PatchAdaNet: Unsupervised class incremental learning using local patches and global adapters
arXiv

Interpretation

Class-incremental image classification can proceed with no stored past samples by separating shared features from task-specific ones into a frozen layer and a learnable layer. Earlier replay methods need a memory buffer that grows with the number of tasks and retain task-specific noise, while knowledge-distillation methods must keep a copy of the previous task's teacher model. On the 5-split CIFAR-10 protocol the CNN variant reaches 65.9%±0.7 and the ViT variant 64.5%±0.4, both replay-free, while the compared replay methods use a buffer of 100 stored samples.

The construction divides each image into non-overlapping patches and assigns each spatial position its own autoencoder that reconstructs only that patch. These autoencoders are pretrained once on external data, TinyImageNet, disjoint from every evaluation dataset, and after training all encoder weights are frozen and receive no gradient from any task loss. The frozen features are tiled back to full-image scale by patch grid position, followed by learnable adapters and a decoder that reconstructs the whole image, and a linear classifier is trained on the concatenation of the frozen bottleneck features and the Global Feature Centroid.

Drift is held back by two complementary mechanisms: the Global Feature Centroid in feature space and EWC in weight space. The Global Feature Centroid is one running-average tensor per encoding depth, updated only from the last mini-batch of the final training epoch of each task, so its memory footprint is independent of the number of tasks. EWC is applied only to the adapter and reconstruction layers, leaving the frozen patch encoders unconstrained; in the CIFAR-10 ablation the full CNN configuration reaches 65.9% and drops to 51.8% without the Global Feature Centroid, with the ViT counterparts at 64.5% and 61.0%.

Perspective

The demonstrated scope is class-incremental and task-incremental image classification on CIFAR-10, CIFAR-100, miniImageNet and a 5-Datasets sequence of MNIST, SVHN, notMNIST, Fashion-MNIST and CIFAR-10, with patch autoencoders pretrained once on TinyImageNet.

The patch autoencoders and the Global Feature Centroid are evaluated with relatively compact models, three-layer convolutional encoders for CNN and shallow Transformer blocks for ViT, pretrained on a comparatively small external dataset, so whether these low-level features generalize as well under greater visual complexity or domain diversity remains unclear. The Global Feature Centroid is built from only the final mini-batch of each task, which may introduce sample-dependent noise, and the CNN variant shows higher seed and class-order variance than the ViT variant on the longer CIFAR-100 and miniImageNet sequences.