Skip to main content
Back to timeline
arXivSource publication:

No need to finish every run: SOTF forecasts the finish from checkpoints, cutting the retained-model gap on heterogeneous zoos from 0.0643 to 0.0017

Synopsis

When training a population of candidate networks, mid-run accuracy rankings need not match final rankings, and learning curves can cross. SOTF training predicts each candidate's final or five-epoch-ahead accuracy from checkpoint weight statistics and SANE latent representations, then stops the lower-ranked runs early. On ResNet-18 and ViT zoos with wide performance spread, pruning by forecast final accuracy lowers the mean best-retained-model gap from 0.0643 to 0.0017 and raises oracle top-5 recall from 0.11 to 0.72; on CNN zoos, pruning by current accuracy already retains a near-optimal model.

Source-provided article image: Nowcasting Future Performance for Population Pruning in Survival-of-the-Fittest Training
arXiv

Interpretation

While a population of candidate models is still training, each candidate can be given a forecast score for future accuracy, and that score decides which runs continue and which stop early. Earlier adaptive resource-allocation methods mainly rank candidates by observed current accuracy or by learning-curve extrapolation, yet learning curves can cross, so the current leader need not finish first. Across nine image-classification model zoos, Kendall's tau between the Tiny ImageNet ResNet-18 zoo and the final ranking falls during training and approaches one only near the end, showing that intermediate rankings disagree with the final ranking.

The forecast score comes from checkpoint weight statistics and SANE latent representations, combined with the observed accuracy and loss history. Weight statistics are summary quantities of the parameter distribution such as mean, variance, percentiles and range; SANE is a transformer autoencoder that encodes a checkpoint as a sequence of latent tokens, and both were previously used mainly to judge current properties of a single checkpoint. For five-epoch nowcasting, the best weight-statistic predictor beats LCNet on all nine zoos; for final-accuracy prediction, the best SANE predictor has lower RMSE than the best weight-statistic predictor on six of nine zoos, and no representation family dominates every zoo.

After pruning by forecast score, the best surviving model in heterogeneous zoos sits much closer to the best model in the whole population. Pruning by current accuracy loses 0.0643 percentage points of final accuracy on average on the ResNet-18 and ViT zoos and retains only 0.11 of the oracle top-5. For final-value selection on the ResNet-18 and ViT zoos, XGBoost on performance features alone lowers the gap to 0.0051, and adding weight statistics lowers it to 0.0017 while raising oracle top-5 recall from 0.11 to 0.72; a paired bootstrap over 44 matched configurations reports a 95% confidence interval for this additional reduction.

Perspective

The evaluation covers nine image-classification model zoos, including CNN, width-varying ResNet-18 and ViT populations on CIFAR-10, STL-10 and Tiny ImageNet, with training horizons of 50 to 300 epochs; pruning experiments run retrospectively on fully recorded trajectories, and predictors are fitted once on the training split and then held fixed for the held-out population.

Savings are proxied by training epochs and exclude the cost of computing weight statistics and of training and applying SANE encoders; SANE backbones are trained separately per zoo, so the experiments show zoo-specific representations rather than a reusable encoder; in zoos varying width or hyperparameters, weight statistics may partly encode the configuration itself; experiments are not repeated across multiple random seeds, so variability from initialization, data ordering or predictor training is not quantified.