Skip to main content
Search

# Joint-KV

1 related articles

Subscribe to updates
  1. arXiv

    Transformers can branch sideways: Multi-Branch Transformer splits a block stack into parallel branches, lifting long-context decode throughput by up to 51.4%

    Language models are usually scaled by widening blocks or stacking them deeper; the Multi-Branch Transformer (MBT) takes a third path, composing full Transformer blocks into parallel branches. Across four MoE scales from 0.8B to 2.1B total parameters, a 2-branch Wide-Shallow MBT consistently beats a sequential Transformer at matched parameters and training compute, and the final Balanced MBT with Joint-KV cuts full-model KV-cache storage by up to 34% while raising decoding throughput by up to 51.4% on a single H200 with 64K-token inputs and 8 concurrent requests.

You’re all caught up