ProxyProp trains transformers without storing every layer's activations: local proxies estimate global gradients and cut VRAM by 62%
Synopsis
Standard backpropagation must keep every layer's activations in memory until the loss gradient has traveled back, so memory grows with depth and becomes a major cost of training large models. ProxyProp attaches to each transformer block a small proxy network distilled from the full network's output, and backpropagates gradients through that proxy, so only one block's activations need to be held at a time. On a 12-layer vision transformer trained on CIFAR-100, ProxyProp comes close to backpropagation accuracy while using 62% less VRAM; when fine-tuning OPT-13B and LLaMA-3.2-3B, it outperforms the zeroth-order optimizer MeZO at comparable memory.
Interpretation
Training a transformer no longer requires memory that grows linearly with depth: each block can update its parameters and release its activations once its own forward pass is done, with the gradient estimate supplied by that block's local proxy network. End-to-end backpropagation must retain activations from all layers until the loss gradient has been propagated back, so memory rises with depth and limits the network depth and sequence length that fit in a given budget. On a 12-layer vision transformer trained on CIFAR-100, ProxyProp comes close to backpropagation accuracy while reducing VRAM use by 62%.
Each block carries a small proxy network distilled from the full network's output, and the block's parameters are updated by backpropagation through that proxy rather than by waiting for the downstream blocks. Most earlier local learning algorithms were not designed to approximate the backpropagated gradient and struggle in deep architectures, while DNI-style auxiliary networks that explicitly predict gradients are hard to train because their errors compound. On the vision transformer, replacing the distillation loss with a DNI-style gradient bootstrapping loss brought no significant improvement in classification, despite requiring second-order derivatives and more memory.
A residual parameterization of the proxy makes the local gradient exact at initialization and gives a bounded gradient estimation error when the residual branches are small relative to the skip connections. A linear proxy cannot fully align sample-level gradients in a nonlinear network, and token-wise linear projections cannot track the computation that attention layers perform over a sequence. On a 6-layer MLP with skip connections, gradients align perfectly at all layers under zeroed initialization; away from initialization, the residual proxy's gradient alignment stays above that of the linear proxy and approaches the best achievable alignment.
The same local gradient estimate can fine-tune large language models, converging better and faster than zeroth-order optimization at comparable memory. Zeroth-order methods such as MeZO store almost no activations, but their noisy gradient estimates converge slowly and require repeated parameter perturbations. Fine-tuning OPT-13B and LLaMA-3.2-3B on SST2, WIC and SQuAD, ProxyProp approaches backpropagation performance and exceeds the results reported for MeZO; MeZO did not catch up even with 8 times the wall-clock time.
Perspective
The demonstrated scope is 12-layer vision transformer classification on CIFAR-100 and fine-tuning of OPT-13B and LLaMA-3.2-3B on 1000 examples each from SST2, WIC and SQuAD; the memory comparison is measured at a fixed, relatively large batch size where activation storage dominates. The authors propose extending the method to deeper, looped or diffusion-style architectures and to fully decoupled parallel training across devices, but these are outlooks, not implemented or evaluated results.
The theoretical exactness holds only for deep linear networks; in nonlinear residual networks the error bound depends on skip connections dominating, meaning the residual branch's Jacobian norm is bounded. The authors also note that the algorithm retains a weak coupling from one pre-forward pass, and that the fully unlocked variant has not been studied in a distributed implementation.
