Skip to main content
Back to timeline
arXivSource publication:

Looped Transformers no longer wait a full loop per token: LoopFill merges suffix retrieval and shallow drafts into one tree, beating LoopAhead on seven benchmarks

Synopsis

Looped Transformers reuse the same layers repeatedly, so decoding each token requires several sequential passes and latency is high; earlier pipelined self-speculative methods drafted only along recurrent depth, one token at a time, and each token had to reach the drafting depth before the next could enter the pipeline. LoopFill combines multi-token continuations retrieved from suffix trees with the model's own shallow-loop predictions in a single draft tree, letting several future tokens enter the pipeline together while full-depth verification preserves the autoregressive output distribution. Across seven benchmarks under greedy decoding, its average speedup over autoregressive decoding exceeds the strongest baseline, LoopAhead, by 40% on Ouro-2.6B and 25% on Huginn-3.5B.

AI-generated editorial illustration: LoopFill: Pipelined Multi-Token Speculative Decoding for Looped Transformers

Interpretation

The decoding pipeline of looped language models can now be filled with several future tokens at once instead of advancing one token per round. Earlier pipelined self-speculative methods drafted only along recurrent depth: each shallow prediction proposed one token, a new token could start at most once every several loop steps, so speedup was bounded by the loop count, and a rejected draft forced a restart. On Ouro-2.6B, pairing the suffix tree with shallow drafts extends the correct draft path from 1.34 to 1.73 tokens, and to 2.28 tokens when the suffix tree is also queried at the loop-1 draft.

The approach attaches both draft sources to one draft tree that shares computation and verification. A suffix tree is a drafter that retrieves continuations from the prompt and previously generated text without model computation, while a shallow prediction is a candidate token the model reads out at a shallower loop depth. Each draft node is scored by the product of shallow-prediction probabilities and suffix frequencies along its path, only the top-scoring nodes are computed each loop step, and when the root completes all loops the matching subtree is kept together with its completed loops.

The speedup holds across several models and benchmarks, with a trade-off between draft budget and concurrency. Drafts decide only which positions start early, never which token is emitted, so full-depth verification preserves the output distribution of autoregressive decoding. On Huginn-3.5B, raising the draft node budget raises decoding speed by 19.5%, while raising it further lowers end-to-end throughput by 43% relative to the lower budget.

Perspective

The demonstrated setting is seven benchmarks on a single NVIDIA L20X GPU in BF16 with the Ouro-1.4B, Ouro-2.6B and Huginn-3.5B models, covering greedy decoding and temperature 1.0 plus five concurrency levels on SpecBench. The intended readers are researchers and inference-engine engineers concerned with looped language model efficiency.

Suffix drafts are less likely to match the model's output when the next token is less predictable, and a larger draft budget can reduce single-request latency while lowering throughput at high concurrency. The source lists adaptive methods that sustain low latency and high throughput as future work.