Skip to main content
Search

# ConvPE

1 related articles

Subscribe to updates
  1. arXiv

    Position encoding gains a convolutional route: ConvPE adds a windowed bias to attention scores and beats RoPE-8K perplexity at 8K context

    ConvPE injects position information into attention scores: hidden states are projected to a low dimension, a causal convolution turns them into a content-dependent windowed bias, and the bias is exactly zero outside the window. Pretrained from scratch on the first 3B tokens of FineWeb-Edu-10BT, it reaches lower training loss than the RoPE baseline and needs fewer tokens to match that baseline's final loss; after extension to 8K context, ConvPE-8K reaches lower perplexity than RoPE-8K and YaRN.

You’re all caught up