Skip to main content
Back to timeline
arXivSource publication:

Resample the corpus, how much do predictions move? Smoothing cuts disagreement only 5–40%, far short of the 75% scaling implies

Synopsis

A first-order formula predicts how much a masked-text reconstructor's outputs would change if a second corpus were collected from the same population, using only an independent reference corpus and treating each document as one sampling unit. On TinyStories, WikiText-103 and CNN/DailyMail, growing the corpus from 512 to 2,048 chunks lowers measured disagreement by only 5–40%, far short of the 75% implied by inverse-size scaling, and the forecast tracks that slow decline. Small references systematically underpredict disagreement, and a source-split jackknife that uses only reference data reduces this calibration error by 57–59%.

Source-provided article image: Forecasting Corpus-Sampling Disagreement in Masked Text Reconstruction
Figure 1 · arXiv

Interpretation

How much a masked-text reconstructor's outputs would shift under a second corpus draw can now be forecast without training new models or using the target corpora's outcomes. Previously, dependence on the chosen corpus could only be judged from average test error, yet similarly accurate predictors can disagree on the same input and comparable validation performance can conceal differences in deployment behavior. On TinyStories, WikiText-103 and CNN/DailyMail, seven independent corpus pairs per corpus show that growing from 512 to 2,048 chunks lowers measured disagreement by only 5–40%, against 75% under inverse-size scaling, and the forecast tracks this slow decline in every corpus.

The forecast is computed from count statistics on an independent reference corpus rather than from the target corpora. The formula treats each document as one sampling unit and retains document length, random row totals and covariance across token offsets, which explains why smoothing makes disagreement fall more slowly than inverse-size scaling. Reference size governs calibration: when the reference is smaller than the target, both analytic laws underpredict, and a source-split jackknife cuts the source law's error from 0.469 to 0.203 and the multinomial law's from 0.205 to 0.085, reductions of 56.7% and 58.5%.

Comparison with small masked diffusion language models shows the count-based forecasts agree only in part. The forecasts partly identify which inputs are most sensitive to the corpus, but they miss the neural models' much faster decline in disagreement as the corpus grows. Across three corpora, 512 to 2,048 chunks and 12,000 updates, MDLM disagreement falls by 72.9%, 81.5% and 79.9%, while the reconstructor falls by 38.9%, 4.9% and 11.4%; the Spearman correlation between the source-law forecast and MDLM disagreement averages 0.48 on TinyStories, 0.38 on WikiText and 0.18 on news.

Perspective

The forecast applies to masked-text reconstructors built from nearby token-pair counts with smoothing, evaluated over the full 50,257-token GPT-2 vocabulary, 64-token chunks, three corpora and corpus sizes from 512 to 2,048 chunks, with reference and target corpora kept independent. The authors propose extending the approach to n-gram models and normalized co-occurrence or pointwise mutual information estimates, and using the forecasts to evaluate candidate corpus sizes and identify the smallest corpus expected to meet a chosen stability target, but these remain stated goals rather than validated results.

Larger references need not improve calibration, because reference error and linearization error can offset each other; the first-order formula assumes a fixed vocabulary and positive context probabilities, and the jackknife requires a regular bias expansion in the number of reference sources, so neither gives uniform guarantees for sparse rows or growing vocabularies. The correction also leaves rows unseen in the reference unresolved. Intervals capture variation across reference banks and corpus pairs, conditional on the auxiliary quantities and 128 evaluation chunks; six banks limit precision, WikiText paragraph sampling leaves dependence within articles unmodeled, and the fresh-source check uses only six corpus pairs and three banks per corpus. The MDLM comparison covers small Transformers and corpora with three corpus pairs and fixed update budgets, and larger corpora receive fewer passes, so training itself affects the measured size response.