Position encoding gains a convolutional route: ConvPE adds a windowed bias to attention scores and beats RoPE-8K perplexity at 8K context
ConvPE injects position information into attention scores: hidden states are projected to a low dimension, a causal convolution turns them into a content-dependent windowed bias, and the bias is exactly zero outside the window. Pretrained from scratch on the first 3B tokens of FineWeb-Edu-10BT, it reaches lower training loss than the RoPE baseline and needs fewer tokens to match that baseline's final loss; after extension to 8K context, ConvPE-8K reaches lower perplexity than RoPE-8K and YaRN.