Small-data training stops sharpening confidence: model annealing has the network review its past selves, beating cross-entropy in 28 of 33 AdamW configurations
Synopsis
On small labelled datasets a network often memorizes the training set before the budget runs out, and the remaining one-hot fitting mostly sharpens confidence. Model annealing splits training into equal episodes, reheats the learning rate and cools it again, freezes each cooled state into a memory that averages predictions, and has the student learn from labels, from the memory's tempered predictions, and from adversarial inputs sought where student and memory disagree most. Across CIFAR-100, Tiny-ImageNet and Imagewoof, AdamW-trained ConvNeXt, ViT and MLP-Mixer beat cross-entropy in 28 of 33 configurations by 1.3 points on average, matching snapshot distillation in accuracy while improving likelihood, calibration and robustness.
Interpretation
The back half of a small-data training budget is spent on review and consolidation instead of further fitting one-hot labels. Plain supervised training only rereads the same labels; once the network has memorized the training set, the remaining epochs mostly sharpen confidence and accuracy plateaus early. Across CIFAR-100, Tiny-ImageNet and Imagewoof, four architecture families and 5% to 100% of the labels, the AdamW-trained ConvNeXt, ViT and MLP-Mixer reach higher top-1 accuracy than cross-entropy in 28 of 33 configurations, by 1.3 points on average and by up to 4.3 points for ConvNeXt on CIFAR-100 with 10% of the data.
Training is divided into equal episodes, each raising the learning rate to the same peak and cosine-decaying it to nearly zero, and every cooled state is frozen into a memory. Snapshot distillation teaches from the latest snapshot alone, whereas model annealing keeps all cooled states and averages their predictions, not their weights, with geometric decay. The memory enters supervision as tempered predictions, and the student's target is a mixture of the label and the memory prediction; on CIFAR-100, replacing the memory with the latest snapshot moves AdamW top-1 by at most 0.18 points but raises negative log-likelihood.
The student also practices on dreams: inputs searched inside an ℓ∞ ball around a training image where student and memory disagree most, labelled by the memory. Earlier self-distillation matched teacher predictions on training images only, while dreams extend supervision beyond the training set and target current-versus-past disagreement. Removing the dreams lowers accuracy under PGD attack by 2.7 points and under synthetic corruptions by 0.53 points averaged over the four families; for the AdamW families the memory and the dreams each curb cross-episode overconfidence, and together they move validation negative log-likelihood from +0.18 nats under snapshot distillation to −0.04 nats.
Perspective
The findings rest on three image-classification benchmarks, CIFAR-100, Tiny-ImageNet and Imagewoof, with all four architecture families trained from scratch and label amounts of 5%, 10%, 25% and 100%, the full set omitted for Tiny-ImageNet; every data-dependent decision is made on a validation split, and test images are evaluated once after training.
On ResNet-18 accuracy drops by 1.8 points, and the decomposition traces the loss to the reheats rather than to relearning, with training-batch accuracy falling from 98% on average to between 26% and 45% after each reheat; the authors attribute this to peak learning rates that do not match the cooled network and propose matching episode lengths and reheat peaks per network and weighting snapshots by validation likelihood as next steps.
