Skip to main content
Back to timeline
arXivSource publication:

Masked linear self-attention lets a transformer output the least-squares solution directly, and training does use the layer-norm division

Synopsis

On in-context learning of a simple linear regression task, a transformer with masked linear self-attention is constructed to output the closed-form least-squares estimate directly rather than approach it by gradient descent. Earlier constructions assumed a sufficiently long prompt; the mask separates the prediction point from the in-context samples and removes that assumption. Trained with the least-squares estimate as target and ℓ1 regularization, two of ten independent runs showed a correlation above 0.9 between the reciprocal layer-norm standard deviation and the reciprocal sample variance, and one run reached an average squared test error of 0.000058.

Source-provided article image: In-context learning of solution to simple linear regression task using transformer with masked linear self-attention
arXiv

Interpretation

In-context learning of a simple linear regression can deliver the least-squares estimate analytically inside the prompt instead of approximating it by iterated gradient descent. Most earlier work on transformers implementing regression solvers found the implemented algorithm to be essentially gradient descent; the least-squares solution needs a matrix inverse, that is, a division, which gradient descent only performs implicitly through repeated multiplication and addition. The constructed TF(4, 2, 2) was trained with input sequence length 51, 50 in-context samples and noise variance 0.04, and its output matches the least-squares estimate at the prediction point.

The analytic solution is realized by handing the division to layer normalization and using a mask to separate the prediction point from the in-context samples. Linear self-attention computes inner products of the sequence, so sequence-position information disappears, and the prediction point would contaminate the computation of the in-context input mean. The mask zeroes the last row of the sequence so the prediction point does not enter the mean and variance statistics, while the standard deviation computed along the embedding dimension supplies the division by the sample variance.

The trained transformer does use this construction, even though its weight representation is not unique. Linear self-attention representations are non-unique, so the same output can arise from different weight combinations, and whether training adopts the construction needs a separate check. In ten independent runs, runs 1 and 5 gave correlations of 0.966911 and 0.988581 between the reciprocal layer-norm standard deviation and 1/V_m; run 1 had an average squared test error of 0.000058, and a linear probe of its output reached R² of 0.999974 with coefficients (α3, α4, α5) near (1, 1, −1).

Perspective

The demonstrated setting is TF(4, 2, 2) on a simple linear regression task with input sequence length 51, 50 in-context samples and noise variance 0.04, trained toward the least-squares estimate at the prediction point with ℓ1 regularization on the output weights. The intended reader is a researcher interested in in-context learning mechanisms; the source proposes next steps including training without ℓ1 regularization, examining how model size affects the implementation, other prompt formats, and using the output sample at the prediction point rather than the least-squares estimate as the training target.

The construction relies on a sufficiently large constant R, which forces weight magnitudes to be either near zero or extremely large; ℓ1 regularization is used to suppress irrelevant weights, which the source considers unnatural and a possible source of prediction-error bias. Only two of ten runs exceeded a correlation of 0.9, and the source notes that comparability with a gradient-descent implementation in TF(4, 2, 2) is limited by the number of layers.