Skip to main content
Search

# Depth Scaling

2 related articles

Subscribe to updates
  1. arXiv

    ANCRe makes residual connections learnable, accelerating convergence in LLM pretraining, diffusion models, and deep ResNets at under 1% overhead

    Revisiting residual connections from an optimization perspective, this work proves that the layout of residual connections can fundamentally shape convergence behavior and even induce an exponential gap in convergence rates, and introduces ANCRe, a principled and lightweight framework that parameterizes and learns residual connectivities from data; ANCRe adaptively reassigns residual connections with negligible computational and memory overhead (<1%) to use network depth more effectively, and extensive numerical tests across large language model pretraining, diffusion models, and deep ResNets show consistently accelerated convergence, boosted performance, and enhanced depth efficiency over conventional residual connections.

  2. arXiv

    DepthBench sweeps width–depth ratio across 10 architectures at fixed parameter budget: HC and Full AttnRes keep lowering validation loss at extreme deep-narrow shapes while Pre-LN degrades

    The work introduces DepthBench, a controlled benchmark that varies the width–depth aspect ratio (d_model/n_layer) from shallow-wide to deep-narrow under a fixed parameter budget and pre-training recipe, comparing 10 representative residual and normalization architectures, and finds that the benefit of depth is strongly architecture-dependent: Pre-LN and most of its norm- and scaling-based variants worsen as models get deeper and narrower (Pre-LN rises from 2.759 to 2.782), whereas HC and Full AttnRes improve consistently (HC from 2.729 to 2.699, Full AttnRes from 2.751 to 2.718), accompanied by stronger cross-layer causal dependence and order sensitivity, though deeper models also incur compute and memory overhead.

You’re all caught up