Skip to main content
Back to timeline
arXivSource publication:

Looped language models no longer wait loop by loop: LoopSkim turns each depth's readout into a verifier, reaching 1.73 times LoopSpec's speedup without training

Synopsis

Looped language models apply one weight-shared block several times per token, so autoregressive decoding runs every loop of every token in sequence. LoopSkim lets draft tokens read out at shallow depths be checked by each later depth's readout, commits only full-depth readouts, and fixes each position's random draws in advance so decoding stays lossless under sampling. Across five Ouro and Raven models, the untrained version averages 1.73 times LoopSpec's speedup over autoregressive decoding, and an optional draft head raises that ratio to 1.99.

Source-provided article image: LoopSkim: Lossless Multi-Depth Speculative Decoding for Looped Language Models
arXiv

Interpretation

A looped language model's decoding pipeline now uses the model's own depth as the verifier of its draft tokens: a shallow readout that contradicts a draft replaces it at that depth, and the tokens behind it stop before full depth. Earlier pipelined self-speculative decoders such as LoopSpec and WaveFront draft one token ahead, so a step commits at most one token; drafting several tokens lifts that cap, but if only full depth finds a wrong token, the pipeline restarts from the correction and discards the loops of every token behind it. Under greedy decoding on Raven-Llama, a replacement after the admission check comes after a mean of 6.7 of its 32 loops, so positions behind a wrong token run only 16% of the loops between admission and full depth.

Draft tokens copied from the context enter the pipeline after the first loops, each check depth compares its readouts with them, and only full-depth readouts commit tokens. The drafts come from an n-gram chain as in prompt lookup decoding, which finds the last earlier occurrence of the longest suffix ending at the newest token and copies the tokens that followed it; the lookup runs on the host and needs no forward pass or training. Evaluation covers Ouro-1.4B, Ouro-2.6B and the Raven-Llama, Raven-OLMo and Raven-TinyLlama checkpoints on GSM8K, MATH500, BBH, HumanEval+ and MBPP+, on one H200 in BF16 with one request at a time.

Losslessness under sampling comes from fixing each position's random draws in advance, so every readout of a position, at any depth and for any draft chain, draws with the same noise. Speculative sampling at each check depth would carry a residual distribution from each replaced token into every later check; with fixed draws, admitting, replacing and redrafting change only what is computed ahead of time, not what is committed. In exact arithmetic, for any number of checks and draft chains, the committed tokens are a prefix of what autoregressive decoding generates with the same noise and end equal; with every operation in FP32, LoopSkim plus the draft head matches autoregressive output token for token on 3,300 prompts across seven settings.

Perspective

Evaluation is limited to looped language models of up to 3.5B parameters on math, code and reasoning tasks, on H200 and RTX 5090 GPUs with up to 32 requests per batch; settings are chosen per batch size, and integrating the method into a production serving engine is future work.

LoopSkim trades compute for latency: per output token it runs 4.74 to 7.74 times as many loops as autoregressive decoding over all token positions, which may raise energy per token and leaves less compute for other requests; its margin over LoopSpec narrows at long contexts, the n-gram chain gains most where an output repeats text from the context or from itself, and the optional draft head is slightly slower than LoopSkim without it in every BBH cell, whose questions its training data exclude.