Skip to main content
Back to timeline
arXivSource publication:

Diffusion language models stop waiting for a whole block to finish: B3 overlaps two blocks in one forward pass for 11.5–14.5% more throughput

Synopsis

Block diffusion language models generate one block at a time, and a block cannot hand off until its slowest token settles, so compute idles at block tails. B3 initializes the next block once the current one is settled enough, denoises both in a shared forward pass, and calibrates the spawn point online from the model's own acceptance behavior, with no training or hyperparameter tuning. On DiffusionGemma 26B, LLaDA 2.1 mini 16B and flash 100B across 10 datasets, B3 raises tokens per second by 11.5–14.5% at on-par generation quality.

AI-generated editorial illustration: Blurring the Block Boundaries: Overlapping Block Denoising for Efficient Diffusion Language Models

Interpretation

Block diffusion language models generate block by block, and each block advances only after its slowest token settles, so block boundaries act as synchronization barriers and easy tokens in the next block wait untouched. Earlier block diffusion inference was strictly sequential: the next block was not initialized until the current block had converged, and one forward pass served only one block. For DiffusionGemma on HumanEval+, the last 20% of a block takes 22% of the steps but yields only 10% of the tokens; the ratio of token share to step share in that tail is 0.35–0.71 for DiffusionGemma and 0.05–0.62 for LLaDA 2.1 mini, where 1 means parity.

B3 pulls the next block into the same forward pass before the current block is fully finalized, denoising both in parallel and overlapping the low-acceptance tail of one block with the start of the next. The spawn time is decided online from the model's own acceptance behavior: a new block opens only when every token in the current block has entropy at most a threshold estimated from earlier frontier-advancing steps, so no training and no per-model or per-task hyperparameter tuning is needed. On DiffusionGemma 26B, LLaDA 2.1 flash 100B and LLaDA 2.1 mini 16B across 10 benchmarks, B3 improves both tokens per second and tokens per forward on all 30 model–benchmark pairs, with average accuracy changing by –0.40 to +0.76 percentage points.

Attention stays block-causal, so the current block's predictions are unchanged by the early spawn; the next block is drafted on a prefix that has not fully converged, and self-editing later corrects those early guesses. At LLaDA 2.1 mini's default block size of 32, B3 is the fastest of the compared methods and is more accurate than FlowBlock and MBD-LMs; at block size 64 it again leads on both speed and accuracy. Across all 164 HumanEval+ problems, tokens drafted in the future block are later edited at rates of 1.98% for DiffusionGemma and 4.29% for LLaDA 2.1 mini, against 0.27% and 2.80% for vanilla decoding, so 98.0% and 95.7% of drafted tokens are kept unchanged.

Perspective

The demonstrated scope is block diffusion language models with self-editing: DiffusionGemma 26B, LLaDA 2.1 mini 16B and LLaDA 2.1 flash 100B, evaluated on 10 English text-only benchmarks, with wall-clock speedups measured on one inference stack per model.

B3 does not guarantee outputs identical to vanilla decoding, and average accuracy changes by –0.40 to +0.76 percentage points; the speedup is bounded by each block's low-acceptance tail, and the authors describe the 11.5–14.5% throughput gain as moderate; wall-clock speedups were measured on one inference stack per model and may differ in other serving regimes such as large-batch inference; ablations and comparisons with concurrent methods used only LLaDA 2.1 mini and a subset of benchmarks because of compute constraints.