A preconditioner pulled out of a transformer's inner computation: RAPID lowers the condition number at every tested step on all 31 SuiteSparse matrices
A softmax-attention transformer trained only to predict in-context linear systems turns out, when its layers are traced, to apply a solution-preserving congruence update that improves the conditioning of the Gram matrix. The researchers distilled this into RAPID, a standalone algorithm with no learned components: it repeatedly applies random orthogonal mixing, then lets each row pick its smallest off-diagonal entry for a sparse correction. On four synthetic spectral families and 31 SuiteSparse symmetric positive definite matrices, RAPID reduced the condition number at every tested step count and cut the conjugate gradient iterations needed to reach a target accuracy.