Cosine schedules get a per-batch nudge: one extra forward pass cuts Cifar10 loss faster and more monotonically
When training Resnets, a cosine learning-rate schedule can be modulated per minibatch instead of followed as a fixed curve. Both proposed methods use one extra forward pass to estimate local quadratic information about the minibatch loss, then slightly raise or lower the current learning rate. On Cifar10 this lowers the loss faster than the perturbed schedule and makes the curve more monotone, with a small test-accuracy gain; on Cifar100 test accuracy stays the same.