Skip to main content
Search

# Softmax Cancellation

1 related articles

Subscribe to updates
  1. arXiv

    Google DeepMind rewrites softmax and cross-entropy: confident logits keep their gradient, and distillation students stop drifting from the teacher

    Google DeepMind presents rewritten implementations of log-softmax, softmax and cross-entropy that remove two floating-point defects: the target logit's gradient rounding to zero when that logit dominates the others, and precision loss when student and teacher logits are close. The rewrite keeps 1 out of the exponential sum and computes the probability difference directly in log space, so gradients stay accurate and sum to zero in bf16, fp32 and fp64, eliminating loss spikes in grokking and student drift in soft-target distillation.

You’re all caught up