Skip to main content
Search

Yilang Zhang

1 related articles

Subscribe to updates
  1. arXiv

    ANCRe makes residual connections learnable, accelerating convergence in LLM pretraining, diffusion models, and deep ResNets at under 1% overhead

    Revisiting residual connections from an optimization perspective, this work proves that the layout of residual connections can fundamentally shape convergence behavior and even induce an exponential gap in convergence rates, and introduces ANCRe, a principled and lightweight framework that parameterizes and learns residual connectivities from data; ANCRe adaptively reassigns residual connections with negligible computational and memory overhead (<1%) to use network depth more effectively, and extensive numerical tests across large language model pretraining, diffusion models, and deep ResNets show consistently accelerated convergence, boosted performance, and enhanced depth efficiency over conventional residual connections.

You’re all caught up