Skip to main content
Search

# ProxyProp

1 related articles

Subscribe to updates
  1. arXiv

    ProxyProp trains transformers without storing every layer's activations: local proxies estimate global gradients and cut VRAM by 62%

    Standard backpropagation must keep every layer's activations in memory until the loss gradient has traveled back, so memory grows with depth and becomes a major cost of training large models. ProxyProp attaches to each transformer block a small proxy network distilled from the full network's output, and backpropagates gradients through that proxy, so only one block's activations need to be held at a time. On a 12-layer vision transformer trained on CIFAR-100, ProxyProp comes close to backpropagation accuracy while using 62% less VRAM; when fine-tuning OPT-13B and LLaMA-3.2-3B, it outperforms the zeroth-order optimizer MeZO at comparable memory.

You’re all caught up