Adding a paragraph coordinate to attention makes real paragraphs compress deeper than random labels, but depth varies by corpus and no corpus-only statistic fully reproduces it
Using a hierarchical rotary positional encoding (hRoPE) that splits paragraph, sentence, and token indices into separate channels, the authors intervene on the paragraph coordinate while holding the token sequence fixed and measure cross-paragraph attention with a token-distance-exact estimator, finding that attention is compressed in all three corpora but that a density-matched random-label channel is compressed too; what distinguishes real structure is the depth of compression, which is greater and corpus-dependent while the control's is not, and across eight corpus-only quantities spanning lexical persistence, paragraph length, and embedding-based coherence, none fully reproduces the cross-corpus ordering of depth, though embedding-based coherence comes closest.