Skip to main content
Search

# Sign Weighted L1 Regularization

1 related articles

Subscribe to updates
  1. arXiv

    L2 regularization in continual learning gets a sign-weighted twist: sign-weighted L1 error falls linearly with sparsity where plain L1 only crawls

    In continual classification with two-layer diagonal linear networks, replacing weight-space L2 regularization with a sign-weighted form induces a sign-weighted L1 penalty in predictor space, so excess risk falls linearly in the sparsity factor for a sparse teacher, while plain L1 decays only near-linearly in sparsity and poly-logarithmically in the task horizon. The result comes from a high-dimensional asymptotic analysis plus synthetic Gaussian, linear MNIST and two-layer ReLU MNIST experiments, and the advantage shrinks as the teacher becomes less sparse.

You’re all caught up