Skip to main content
Search

# differential attention

1 related articles

Subscribe to updates
  1. arXiv

    Backpack language models can now veto a word sense: equal-parameter perplexity gap to a Transformer falls from 3.51 to 2.43

    Backpack language models predict from a linear combination of non-contextual sense vectors, but the combination weights come from a softmax, so context can select or ignore a sense but cannot cancel a wrong one. A differential Backpack computes those weights as the difference of two softmax maps with a learned per-sense gain, letting context assign a sense a negative weight. At equal parameters on WikiText-103, validation perplexity falls from 24.60 to 23.52, cutting the gap to a plain Transformer from 3.51 to 2.43; the advantage persists on OpenWebText up to 350M parameters, though its relative size shrinks with scale.

You’re all caught up