Transformers can branch sideways: Multi-Branch Transformer splits a block stack into parallel branches, lifting long-context decode throughput by up to 51.4%
Synopsis
Language models are usually scaled by widening blocks or stacking them deeper; the Multi-Branch Transformer (MBT) takes a third path, composing full Transformer blocks into parallel branches. Across four MoE scales from 0.8B to 2.1B total parameters, a 2-branch Wide-Shallow MBT consistently beats a sequential Transformer at matched parameters and training compute, and the final Balanced MBT with Joint-KV cuts full-model KV-cache storage by up to 34% while raising decoding throughput by up to 51.4% on a single H200 with 64K-token inputs and 8 concurrent requests.
Interpretation
MBT replaces the serial middle stack of a standard Transformer with several independently parameterized branches that run in parallel: the input passes through shared initial layers, the branch outputs are fused, and the result goes through shared final layers, so capacity can be organized along a branch axis rather than only by width or depth. Prior scaling choices concentrated on making each block wider or stacking more blocks in sequence, and the parallel composition of full Transformer blocks was rarely used as a scaling lever for autoregressive language models. Across four MoE scales from 0.8B to 2.1B total parameters, the 2-branch Wide-Shallow MBT beats the sequential Transformer at matched parameters and training compute at all four scales, lowering validation loss by 0.025 to 0.031.
The branch structure also yields an inference-side benefit: branches at the same depth can share one key-value cache instead of each keeping its own. The default Independent-KV gives every block in every branch its own cache, so more branches mean more cache and more bytes read during long-context decoding. Joint-KV builds a shared KV cache from branch representations at each depth while queries stay local to each branch; 2-, 3-, and 4-branch Joint-KV MBTs reduce full-model KV-cache storage by 19% to 34% relative to the standard Transformer, and all 12 Joint-KV models reach lower validation loss than the same-scale standard model.
A companion inference engine batches blocks at the same depth across branches into one execution stage and folds the branch dimension into the query-head dimension, so a single attention call reuses the shared cache. Running branch blocks one at a time adds serial steps and makes each branch read the same shared KV data, which is slower than the standard Transformer. On a single H200, the 4-branch Joint-KV MBT decodes faster than the standard Transformer in every setting across four scales, 4K to 128K inputs, and 1 to 64 concurrent requests, reaching up to 51.4% higher throughput with 64K-token inputs and 8 concurrent requests.
Perspective
The results rest on pretraining on FineWeb-Edu at roughly 200 tokens per parameter with a 4,096-token context, at four MoE scales from 0.8B to 2.1B total parameters; the inference benchmark runs on a single H200 with 4K to 128K-token inputs and 1 to 64 concurrent requests.
The branch benefit varies with scale and configuration: two branches are slower than the standard Transformer in seven settings, six of them at W1 and one at W3 with 4K inputs, and going from 2 to 3 or 4 branches changes validation loss by a mean absolute value below 0.006, which preserves rather than extends the gain. The fitted 1.26x training-compute figure and the 34% cache saving are fitted or theoretical upper values, and the authors list joint scaling laws, combination with cross-layer cache sharing, and multimodal branches as future directions.
