Skip to main content
Back to timeline
arXivSource publication:

Concepts that arrive late can still learn from discarded images: RetroCBM stores only statistics and lifts mean incremental accuracy 18.1% over the strongest exemplar-free baseline

Synopsis

Exemplar-free class-incremental learning requires models to learn new classes in sequence without keeping old training images, and concept bottleneck models often fail to let newly named concepts contribute to old-class predictions once the concept vocabulary expands. RetroCBM stores a small set of statistics linking frozen visual features to vision-language model embeddings before images are discarded, then uses them to recover the same predictor that class-balanced ridge regression on all images seen so far would give, and reprojects those statistics into the expanded concept space to reweight old classes. Across 16 settings on six datasets it beats interpretable methods, with an 18.1% relative gain in mean average incremental accuracy over CI-CBM, the strongest exemplar-free baseline.

Source-provided article image: Retroactive Concept Bottleneck Models for Exemplar-Free Class-Incremental Learning
arXiv

Interpretation

In exemplar-free class-incremental learning, a concept bottleneck model can let late-arriving concepts contribute to old-class scores without retaining any earlier training images. Earlier interpretable incremental methods regularize old concepts, freeze learned concept mappings, or approximate past classes with generated features, which can leave new concepts underused in old-class predictions. Across all 16 class-incremental settings on six datasets, RetroCBM's average incremental accuracy exceeds that of the compared interpretable methods, with an 18.1% relative gain in mean average incremental accuracy over CI-CBM, the strongest exemplar-free baseline.

The method records class-balanced statistics between frozen visual features and vision-language model image embeddings while images are still available, then fits each new concept's predictor from them. This regression rule, retroactive concept regression, yields the same predictor as class-balanced ridge regression on all images seen so far. At the end of the ten-phase experiments on CUB, CIFAR-100 and Aircraft, concept weights and concept scores match an offline reference using per-image features from all training images, and the classifier objective value and subgradient agree up to numerical precision.

Old-class classifiers are refitted in the expanded concept space, so each earlier class learns how much each new concept should contribute to its score. This step, moment reprojection, transforms the stored visual statistics into the current concept space, equivalent to passing earlier images through the updated concept layer to recompute the squared loss. Refitting old-class weights raises mean old-class accuracy by 8.05 to 18.53 percentage points across nine settings on CUB, Aircraft and CIFAR-100; in the ten-phase setting, letting old classes use later concepts adds a further 7.39 points on CUB and 6.60 points on Aircraft.

Perspective

The demonstrated scope is the exemplar-free class-incremental setting on six image-classification datasets, with concepts drawn from a vocabulary shared with CI-CBM and supplemented by GPT-5 candidates filtered with CLIP; on CUB with ten phases the learned parameters used at inference occupy about 2 MiB. The intended readers are researchers and engineers working on continual learning and interpretable models.

The concept regression coefficient affects how well a new concept retrieves earlier-class images the vision-language model ranks highly; on CUB and Aircraft recall peaks near half the baseline value and declines as regularization grows stronger. A weak graph penalty reduces ranking flips between neighboring classes while keeping accuracy close to the no-penalty value, whereas a stronger penalty roughly halves the flip rate on CUB but lowers average incremental accuracy by about 12 percentage points. The case study also shows the same late-arriving concept can support a correct prediction or push up the score of an incorrect class.