Skip to main content
Back to timeline
arXivSource publication:

Cosine schedules get a per-batch nudge: one extra forward pass cuts Cifar10 loss faster and more monotonically

Synopsis

When training Resnets, a cosine learning-rate schedule can be modulated per minibatch instead of followed as a fixed curve. Both proposed methods use one extra forward pass to estimate local quadratic information about the minibatch loss, then slightly raise or lower the current learning rate. On Cifar10 this lowers the loss faster than the perturbed schedule and makes the curve more monotone, with a small test-accuracy gain; on Cifar100 test accuracy stays the same.

Source-provided article image: Stepsize modulation for cross entropy from local quadratic optimization
arXiv

Interpretation

When training convolutional networks, a fixed cosine learning-rate schedule can be replaced by per-batch modulation: each minibatch first yields an estimate of a more suitable step size, which then scales the current scheduled value slightly up or down. Earlier automatic step-size methods mostly rebuilt the learning rate from scratch and were often unstable with momentum; here the existing schedule is kept and only perturbed. Resnet18 and Resnet34 were trained from scratch on Cifar10 and Cifar100 and compared with the unperturbed cosine schedule and its lookahead combination; loss fell faster and more monotonically.

Both estimates cost only one extra forward pass: one minimizes a quadratic Taylor approximation of the minibatch loss under an outer Hessian approximation, the other linearizes the softmax values and minimizes their squared error to the one-hot labels. Second-order step sizes usually need several loss evaluations or gradient-based Hessian approximations; here the needed derivatives are estimated from softmax values at one test update. The two derivations differ, yet on Resnet18 with Cifar10 their average step sizes are close to each other and, after halving the Polyak step size, close during the middle of training.

The overhead can be compressed: estimating the step size from 20 inputs drawn from each 128-example minibatch, with lookahead off and subsampling on in the second experiment series, did not appear to degrade performance. The full estimate needs one extra forward pass, which adds runtime given that a backward pass takes about twice a forward pass. The best Cifar10 results still came from one of the two step sizes, but on Resnet34 the accuracy gains look random.

Perspective

The demonstrated setting is Resnet18 and Resnet34 trained from scratch on Cifar10 and Cifar100, with baselines of standard SGD with cosine decay without warmup and its lookahead combination, and hyperparameters held fixed across experiments.

On Cifar100 test accuracy shows no reliable improvement, and on Resnet34 the accuracy gains look random; improvement slows near the end of training, and the method itself requires extra computation.