Masked linear self-attention lets a transformer output the least-squares solution directly, and training does use the layer-norm division
On in-context learning of a simple linear regression task, a transformer with masked linear self-attention is constructed to output the closed-form least-squares estimate directly rather than approach it by gradient descent. Earlier constructions assumed a sufficiently long prompt; the mask separates the prediction point from the in-context samples and removes that assumption. Trained with the least-squares estimate as target and ℓ1 regularization, two of ten independent runs showed a correlation above 0.9 between the reciprocal layer-norm standard deviation and the reciprocal sample variance, and one run reached an average squared test error of 0.000058.