Skip to main content
Search

# Adaptive learning rate stepsize estimation

1 related articles

Subscribe to updates
  1. arXiv

    Cosine schedules get a per-batch nudge: one extra forward pass cuts Cifar10 loss faster and more monotonically

    When training Resnets, a cosine learning-rate schedule can be modulated per minibatch instead of followed as a fixed curve. Both proposed methods use one extra forward pass to estimate local quadratic information about the minibatch loss, then slightly raise or lower the current learning rate. On Cifar10 this lowers the loss faster than the perturbed schedule and makes the curve more monotone, with a small test-accuracy gain; on Cifar100 test accuracy stays the same.

You’re all caught up