Skip to main content
Back to timeline
arXivSource publication:

Google DeepMind rewrites softmax and cross-entropy: confident logits keep their gradient, and distillation students stop drifting from the teacher

Synopsis

Google DeepMind presents rewritten implementations of log-softmax, softmax and cross-entropy that remove two floating-point defects: the target logit's gradient rounding to zero when that logit dominates the others, and precision loss when student and teacher logits are close. The rewrite keeps 1 out of the exponential sum and computes the probability difference directly in log space, so gradients stay accurate and sum to zero in bf16, fp32 and fp64, eliminating loss spikes in grokking and student drift in soft-target distillation.

Interpretation

Standard softmax and cross-entropy compute the wrong gradient at high confidence: once the target logit exceeds the others by enough, the target term's gradient rounds to zero while non-target gradients stay positive, so the gradients no longer sum to zero. Frameworks already subtract the maximum logit to avoid exponential overflow, but the denominator is formed as 1 plus e to the minus margin, and once the margin passes the precision cliff (about 5.5 in bf16, 16.6 in fp32, 36.7 in fp64) that addend is rounded away. On the a+b mod 31 modular-addition task, standard cross-entropy in fp32 repeatedly spikes and reaches 94.3 ± 5.4% test accuracy, while the rewritten cross-entropy crosses the cliff smoothly and reaches 100.0 ± 0.0%.

The rewritten implementation never adds 1 to a tiny exponential term; it fixes the maximum logit position at 0, takes the log-sum-exp over the remaining logits to get the margin, and combines them with softplus. The standard route computes probabilities and subtracts them, whereas the rewrite computes the student-teacher probability difference directly in log space with expm1 and log1p, avoiding subtraction of two nearly equal probabilities. Training a 1-layer, 4-head Transformer on a+b mod 31, the rewrite reaches at least 98.9% test accuracy in bf16, fp32 and fp64 across epsilon from 10^-8 to 10^-16, while the standard version stalls near 93% to 94% in bf16.

In soft-target distillation, the standard implementation still produces a non-zero residual gradient when the student is initialized with the teacher's weights, and Adam amplifies it into parameter drift. The standard implementation uses softmax in the forward pass and log-softmax in the backward pass, and these two paths do not match exactly, so the gradient is non-zero even when student logits equal teacher logits. Distilling on a+b mod 53 at temperature 4, the rewrite holds KL at 0 and 100% accuracy in all three precisions, while the standard version drives the student away from the teacher and lowers test accuracy in bf16 and shows an initial KL spike with a floor near 10^-6 in fp32.

Perspective

The demonstrated scope is small-scale: 1-layer Transformers and 2-layer MLPs on a+b mod 31, a+b mod 113, a2+b2 mod 97 and a3+b mod 97, plus soft-target distillation on a+b mod 53; the authors state that the extreme margins and near-zero student-teacher gaps probed by the experiments are uncommon in standard large-scale training.

The authors note that the extreme margins and near-zero student-teacher gaps probed by the experiments are uncommon in standard large-scale training, so whether the rewrite helps depends on whether a model makes high-confidence predictions or distills from a close teacher; the margin equilibrium at log(1/epsilon) does not hold without weight decay, where margins keep rising.