Skip to main content
Search

# Latent Feedback

1 related articles

Subscribe to updates
  1. arXiv

    Latent feedback makes each FLOP go further: up to 48% less compute, or 47% less unique data, than repeating epochs

    A latent-feedback Transformer feeds each token's top-layer state into the next token's input; earlier comparisons held training tokens fixed and never repeated data. Across 60M to 778M parameters, 12.5 to 400 unique tokens per non-embedding parameter and one to four epochs, with both models tuned at every size and data amount, latent feedback reaches a Transformer's loss with less compute at fixed unique data and with less unique data at fixed epochs, and both savings grow with unique data per parameter and with repetition. A fitted scaling law predicts held-out 778M losses to within 0.3% and the gap between the two models to within 0.05%.

You’re all caught up