Skip to main content
Back to timeline
arXivSource publication:

Decoding-aware sequence-level losses are proved unbounded below: greedy and sampling objectives both diverge, so teacher forcing stays the practical choice for supervised autoregressive training

Synopsis

Autoregressive language models are usually trained by teacher forcing, predicting each token from the ground-truth prefix, while at inference they condition on their own generated prefixes, a mismatch called exposure bias. Two sequence-level losses built to remove that mismatch, one based on greedy decoding and one on autoregressive sampling, are both shown to be unbounded below and therefore unusable for learning. An autoregressive Gumbel trick shows the sampling loss is the expected greedy loss under Gumbel perturbations; a stop-gradient moment-matching direction is not the gradient of any loss yet is a descent direction for the autoregressive negative log-likelihood in function space, at the cost of high variance.

Source-provided article image: On the Difficulty of Sequence-Level Supervised Learning for Autoregressive Language Models
arXiv · Page 6

Interpretation

Writing the inference-time decoding procedure directly into the training objective does not replace teacher forcing: both decoding-aware sequence-level losses drive training to diverge. Teacher forcing needs a single forward pass and no sampling, but it conditions on ground-truth prefixes during training and on the model's own prefixes at inference, the exposure-bias mismatch; existing sequence-level methods rely on external reward signals or on maximization and log-partition oracles that are intractable over sequence space. The greedy-decoding loss is proved discontinuous, nonconvex and unbounded below; the autoregressive-sampling loss, though smooth, is proved nonconvex and unbounded below, so both send the loss toward negative infinity.

The two seemingly different decoding-aware losses are the deterministic and stochastic faces of one object. Greedy decoding and ancestral sampling previously corresponded to separate training objectives whose relation was not characterized. The autoregressive Gumbel trick yields that the sampling loss equals the greedy loss in expectation under independent identically distributed Gumbel perturbations, with greedy decoding as its deterministic limit.

Removing the decoded sequence's dependence on the parameters gives a theoretically usable update direction, paid for in variance. The moment-matching direction is not the gradient of any loss function, so it cannot be understood as ordinary loss minimization. In a function-space representation the direction is proved to be a descent direction for the autoregressive negative log-likelihood; its expectation requires Monte Carlo estimation over the full sequence distribution, and sampled prefixes are correlated across time steps, so estimator variance grows with sequence length.

Perspective

The analysis concerns autoregressive models trained by supervised learning from static datasets, and its conclusions cover two intrinsic sequence-level losses, one based on greedy decoding and one on autoregressive sampling, plus the moment-matching direction obtained by applying a stop gradient; the descent property holds in function space, and the authors state explicitly that it does not directly imply descent in parameter space.

Practical use of the moment-matching direction is limited by high variance, because its expectation can only be estimated by Monte Carlo sampling and sampled prefixes are correlated across time steps, so variance grows with sequence length; the function-space descent guarantee does not automatically extend to parameter space, since the parameterization's Gram operator deforms inner products. The authors also note that the stop-gradient reparameterization trick does not transfer to reinforcement-learning post-training, because the reward is usually a separate function and stopping the gradient removes the signal entirely.