Backpack language models can now veto a word sense: equal-parameter perplexity gap to a Transformer falls from 3.51 to 2.43
Synopsis
Backpack language models predict from a linear combination of non-contextual sense vectors, but the combination weights come from a softmax, so context can select or ignore a sense but cannot cancel a wrong one. A differential Backpack computes those weights as the difference of two softmax maps with a learned per-sense gain, letting context assign a sense a negative weight. At equal parameters on WikiText-103, validation perplexity falls from 24.60 to 23.52, cutting the gap to a plain Transformer from 3.51 to 2.43; the advantage persists on OpenWebText up to 350M parameters, though its relative size shrinks with scale.
Interpretation
The sense-combination weights of a Backpack language model can now take negative values, so context can actively inhibit a sense rather than merely decline to select it. The original Backpack's weights are softmax outputs and therefore non-negative, so when bank appears in a financial sentence its river sense is not simply unused; it leaks into the prediction, and a non-negative combination has no mechanism to cancel that wrong-sense contribution. On WikiText-103 at about 70M parameters and equal training budget, exact validation perplexity falls from 24.60 for the softmax Backpack to 23.52 for the differential Backpack, against 21.09 for a plain Transformer, cutting the gap from 3.51 to 2.43; retraining both headline arms with three seeds gives 23.52 and 24.60, a gap of several standard errors.
The method computes sense weights as the difference of two causal softmax maps, the first exciting senses and the second inhibiting them, so each weight lies between positive and negative values while each row's mass stays at one. The per-sense gain is reparameterized following differential attention, the sense vectors remain frozen and non-contextual, and the readout stays a strictly linear combination of them, so each sense's contribution to each prediction remains an exact, inspectable quantity. Ablations that freeze the gain at zero, add the two maps, or scale a single softmax each remove one ingredient and recover at most about 28% of the gain, all with zero negative weight mass; the alternative signed rule Cog Attention lands at 24.72, at softmax level, indicating that the difference-of-softmax structure rather than the sign alone carries the gain.
Perspective
Experiments span roughly 70M to 350M parameters, with the main configuration trained on WikiText-103 and larger models on OpenWebText, all on English corpora; sense-level interpretations and the bank example are qualitative demonstrations, and removing inhibition at inference raises perplexity sharply, showing that the trained model depends on this pathway.
The advantage shrinks with scale, from the relative size at 70M to a smaller one at 350M, and whether it survives at billion-parameter scale is open; the 350M runs, Transformer baselines, sense-count sweep and control experiment are single-seed; a zero-shot WiC probe shows no measurable advantage for signed sense weights, though at this scale even the contextual hidden-state baseline barely exceeds the majority class, so the experiment does not decide the question; the debiasing experiment is single-seed at each scale, and the reversal of the frontier ordering at 350M is unexplained.
