Skip to main content
Back to timeline
arXivSource publication:

Latent feedback makes each FLOP go further: up to 48% less compute, or 47% less unique data, than repeating epochs

Synopsis

A latent-feedback Transformer feeds each token's top-layer state into the next token's input; earlier comparisons held training tokens fixed and never repeated data. Across 60M to 778M parameters, 12.5 to 400 unique tokens per non-embedding parameter and one to four epochs, with both models tuned at every size and data amount, latent feedback reaches a Transformer's loss with less compute at fixed unique data and with less unique data at fixed epochs, and both savings grow with unique data per parameter and with repetition. A fitted scaling law predicts held-out 778M losses to within 0.3% and the gap between the two models to within 0.05%.

AI-generated editorial illustration: Latent Feedback Is a Compute and Data Amplifier

Interpretation

At fixed unique data and equal model size, the latent-feedback Transformer reaches a vanilla Transformer's validation loss with less training compute, and the saving grows with unique tokens per parameter and with epochs. Earlier latent-feedback comparisons fixed the training token count and never repeated data; once data runs out the common practice is several epochs, and each repeat is worth less than the last, so the fair baseline is a Transformer spending the same compute on more epochs. On a grid from 60M to 778M parameters, 12.5 to 400 unique tokens per non-embedding parameter and one to four epochs, both models are tuned at every size and data amount; at two epochs latent feedback costs 8% more at 12.5 unique tokens per non-embedding parameter, saves 15% to 18% at 100 and 44% to 48% at 400, and above about 75 it saves compute at every budget trained.

At fixed epochs and equal model size, latent feedback reaches the same validation loss with less unique data, while serving with the same parameters and KV cache. A vanilla Transformer needs more unique data to reach that loss, and latent feedback pays about 1.2 times the training FLOPs per token in exchange for a lower capacity floor. At one epoch and 50 unique tokens per non-embedding parameter, latent feedback saves 16% of the unique data at 271M and 20% at 778M; at three epochs and 100 it saves 41% at 271M; at 271M and two epochs the data multiplier is 1.34, meaning 25% less unique data.

One scaling law fits both models by writing latent feedback as a Transformer with more parameters, and from it the two savings are projected across size, unique data and repetition. The measured multipliers cover only up to 778M, 400 unique tokens per non-embedding parameter and four epochs, so behavior beyond the grid requires extrapolation. Fitted jointly on 128 cells at 60M, 134M and 271M, the law predicts held-out 778M losses with a mean absolute error of 0.13% and at most 0.29%, predicts the gap between the models to within 0.05% in every held-out row, and gives the measured sign in 6 of 7 rows, with the seventh a tie.

Perspective

The result covers a grid from 60M to 778M parameters, 12.5 to 400 unique tokens per non-embedding parameter and one to four epochs, with both models sharing one modern decoder backbone and tuned at every size and data amount; the savings are measured at a fixed model size, latent feedback costs about 1.2 times a Transformer's training FLOPs per token, each generated token costs what a Transformer's does, and prefilling a prompt needs additional fixed-point forward passes.

The authors note the ladder stops at a relatively small scale and uses a single backbone, so larger models, larger data budgets and other architectures may shift the crossover; latent feedback is trained with a single fixed schedule and further optimizations are possible; comparison with recurrent-depth models is left to future work. The size term is not significant within 60M to 271M but is favored by the held-out 778M, whose rows come from three main runs and thus give only three independent signs.