Skip to main content
Search

Kristian Schwethelm

1 related articles

Subscribe to updates
  1. arXiv

    CDB lets looped language models infer at token-adaptive depth, realizing about 99% of the early-exit speedup available on Ouro 1.4B

    The work introduces and first implements end-to-end continuous depth batching (CDB) for looped language models: it splits inference into prefill, prelude, recurrent-core, and coda stage queues, reforms batches between loop steps, manages a depth-indexed KV cache, and uses a lookahead gate to predict exits one step ahead so tokens with different loop depths can share a forward pass; on Ouro 1.4B and Huginn 3.5B, CDB realizes about 99% and 83%–96% of the estimated maximum speedup, showing fully looped architectures are best suited to depth-adaptive inference.

You’re all caught up