Small-data training stops sharpening confidence: model annealing has the network review its past selves, beating cross-entropy in 28 of 33 AdamW configurations
On small labelled datasets a network often memorizes the training set before the budget runs out, and the remaining one-hot fitting mostly sharpens confidence. Model annealing splits training into equal episodes, reheats the learning rate and cools it again, freezes each cooled state into a memory that averages predictions, and has the student learn from labels, from the memory's tempered predictions, and from adversarial inputs sought where student and memory disagree most. Across CIFAR-100, Tiny-ImageNet and Imagewoof, AdamW-trained ConvNeXt, ViT and MLP-Mixer beat cross-entropy in 28 of 33 configurations by 1.3 points on average, matching snapshot distillation in accuracy while improving likelihood, calibration and robustness.