L2 regularization in continual learning gets a sign-weighted twist: sign-weighted L1 error falls linearly with sparsity where plain L1 only crawls
Synopsis
In continual classification with two-layer diagonal linear networks, replacing weight-space L2 regularization with a sign-weighted form induces a sign-weighted L1 penalty in predictor space, so excess risk falls linearly in the sparsity factor for a sparse teacher, while plain L1 decays only near-linearly in sparsity and poly-logarithmically in the task horizon. The result comes from a high-dimensional asymptotic analysis plus synthetic Gaussian, linear MNIST and two-layer ReLU MNIST experiments, and the advantage shrinks as the teacher becomes less sparse.
Interpretation
Continual classification with linear predictors now has a high-dimensional asymptotic characterization: with Gaussian features and labels from a logistic teacher with flip noise, each coordinate of the learned weight behaves like a scalar proximal estimate applied to an effective signal corrupted by Gaussian noise. Earlier theoretical work on continual learning focused largely on continual linear regression, leaving the classification case without a precise characterization. The characterization holds in the proportional high-dimensional limit where dimension and sample count grow together, under the assumption that the state-evolution system has a unique solution.
In the single-layer case, the optimal isotropic L2 strength decays nearly linearly with the number of tasks, and excess risk decreases nearly linearly in the task horizon, mirroring known results for linear regression. This scaling law had previously been derived only for continual linear regression. Synthetic experiments show that L2 excess risk stays roughly fixed as teacher sparsity varies, indicating it does not exploit sparse structure.
Sign-weighted L1 places weights that preserve the coordinate-wise signs of the previous solution into the regularizer, giving excess risk that falls linearly in the sparsity factor for a sparse teacher, whereas plain L1 only achieves near-linear decay. Plain L1 can exploit sparsity but decays only poly-logarithmically in the task horizon, so it converges slowly over long task sequences. The bound is tight within the family of positively homogeneous regularizers: no such regularizer can make excess risk vanish with the task horizon.
Across synthetic Gaussian data, linear MNIST classification and two-layer ReLU MNIST, sign-weighted L1 gains more over L2 and L1 as the teacher becomes sparser. At lower sparsity, L1 plateaus early while L2 keeps improving and eventually reaches lower loss. Results report held-out test loss after a sequence of tasks, with curves averaged over seeds and shaded one-standard-deviation bands; the main text says 5 seeds while the appendix summary table lists 3, and the two disagree.
Perspective
The conclusions target high-dimensional continual classification with Gaussian features and labels from a logistic teacher with flip noise, using a single-layer linear predictor or a two-layer diagonal linear network; experiments cover synthetic Gaussian data, linear MNIST classification and two-layer ReLU MNIST, all with sparse teachers.
The theory assumes the state-evolution system has a unique solution at every task; the sign-weighted L1 advantage is clear only when the teacher is sufficiently sparse, L2 eventually reaches lower loss as sparsity decreases, and positively homogeneous regularizers cannot make excess risk vanish with the task horizon. The reported seed count differs between the main text and the appendix summary table, so the exact figure should be taken from the original.
