Skip to main content
Back to timeline
arXivSource publication:

Position encoding gains a convolutional route: ConvPE adds a windowed bias to attention scores and beats RoPE-8K perplexity at 8K context

Synopsis

ConvPE injects position information into attention scores: hidden states are projected to a low dimension, a causal convolution turns them into a content-dependent windowed bias, and the bias is exactly zero outside the window. Pretrained from scratch on the first 3B tokens of FineWeb-Edu-10BT, it reaches lower training loss than the RoPE baseline and needs fewer tokens to match that baseline's final loss; after extension to 8K context, ConvPE-8K reaches lower perplexity than RoPE-8K and YaRN.

Source-provided article image: ConvPE: Convolutional Position Embedding Improves Transformers
arXiv

Interpretation

Transformers gain another place to inject position: instead of only altering token representations or rotating queries and keys, the model adds a local, content-dependent bias directly to the attention score matrix. The established routes were absolute position vectors or RoPE rotations, with long-context extension requiring interpolation or frequency rescaling; among score-bias methods, ALiBi uses a fixed linear decay and DAPE convolves the full score map. Pretrained from scratch on the first 3B tokens of FineWeb-Edu-10BT, ConvPE converges faster and reaches a lower final training loss than the RoPE baseline, needing fewer tokens to match that baseline's final loss.

The bias is produced from hidden states by a low-dimensional projection followed by a causal convolution, each bias head learns a different relative-position pattern, and keys outside the window receive no positional bias. Window size, projection dimension and bias-head count are the three hyperparameters, and the default uses window 128, projection dimension 16 and a single bias head because that setting performed best in the base-model sweep and is the cheapest bias-head option. In hyperparameter sweeps on tiny and base backbones, every ConvPE variant beat the RoPE baseline; the grouped-convolution variants had the worst loss.

As a plug-in, ConvPE coexists with RoPE, attention masks and KV concatenation, and high-dimensional convolutions extend it to image-patch grids. The 2D version, ConvPE2D, replaces the 1D causal convolution with a 2D convolution over the patch grid, so each patch receives a bias over its neighborhood. On CIFAR-100, ConvPE2D scores higher than the eight 2D position-mechanism baselines in the same reproduction, but remains below the convolutional reference network.

Perspective

The demonstrated scope is language-model pretraining, zero-shot downstream evaluation, context extension up to 8K, and CIFAR-10 and CIFAR-100 image classification at ViT-Tiny scale.

The main experiments use a single seed and models of limited size, so how large the gains are at larger scale is unknown; the longest evaluated context is 8K; and the influence of the 2D hyperparameters needs further study on larger models and datasets.