Skip to main content
Search

# Next-Token Prediction

7 related articles

Subscribe to updates
  1. arXiv

    Chain-of-thought solves unseen multi-step tasks by reusing one step predictor, with error growing sublinearly in depth

    Chain-of-thought out-of-distribution generalization is framed as compositional generalization: training supplies only single-step atomic transitions, while the target task requires composing many steps. Theory bounds target risk by the sum of step risks times a depth-independent coefficient, and shows that sharing one predictor makes the estimation term shrink as the inverse square root of depth rather than grow linearly. Atom-Task and GSM8K experiments support short-to-long and unseen-combination transfer, and show explicit state tracking is essential.

  2. aclanthology.org

    Autoregressive training drops strict left-to-right targets: Max(MI(S;t)) reorders by mutual information and lifts arithmetic accuracy to 94.96%

    Autoregressive language models normally learn next-token prediction in a fixed left-to-right order; Max(MI(S;t)) instead ranks target tokens by their mutual information with the source and already-selected targets before training, exposing informative targets earlier. On GPT-style models it reaches 94.96% average accuracy across arithmetic tasks, against 74.84% for next-token prediction and 86.29% for reverse order, close to the 97.33% upper bound from exhaustively testing 24 orders; on WikiText-2 it lowers average perplexity from 13.58 to 10.77. Only the training-target organization changes; architecture and autoregressive decoding stay the same.

  3. arXiv

    Alignment without scale: good views let a 109M text JEPA beat 72B language models on the Platonic representation benchmark

    The Platonic Representation Hypothesis, which holds that separately trained models converge on a shared picture of the world, becomes a theorem for JEPAs (joint-embedding predictive architectures): if each modality's positive pairs link the same world states, the encoders agree up to one rotation. The resulting good-view versus bad-view distinction predicts alignment, and a 109M-parameter text JEPA reaches .184 on the PRH benchmark, above every language model evaluated, up to Qwen2.5-72B at .154.

  4. arXiv

    More Communication Can Widen the Generalization Gap in Streaming Federated Learning, Bounds Grow With √R

    Streaming federated learning trains a shared model from continuously arriving client data, whose streams are often temporally dependent. For clients whose data follow Markov processes, an analysis uses information-theoretic and rate-distortion tools to bound the expected generalization gap of the aggregated model, specializing to linear and self-attention next-token prediction. The bounds show that more communication rounds loosen the guarantee at roughly √(R/N), while temporal dependence enters through the square root of the mixing time; GPT-style pretraining on TinyStories and ETTh2 forecasting both show train and test losses falling with rounds while the gap widens.

  5. A generative ICU twin trained without task labels forecasts 13 endpoints at AUROC 0.83–0.98 and ranks first among 20 predictors

    A 123.5-million-parameter generative model, Didymos, was trained once without task labels on 409,297 intensive care stays from seven databases in five countries and answers prediction questions by sampling a patient's futures. On held-out stays it read 13 primary endpoints from those futures at AUROC 0.83–0.98 and ranked first on average among 20 predictors, including label-trained LightGBM and TabPFN-3.5. Its simulated treatment responses agreed with eight of nine observational contrasts, yet at two hospitals in Russia and China that contributed no training data it needed minutes of continued training on unlabelled local stays to match locally supervised models.

  6. arXiv

    Mid-training a language model on raw video lifts video benchmarks by 2.9 points and image benchmarks by 5.1 points while preserving text ability

    This work asks whether raw web video, with no captions and no text loss, can serve as mid-training data for a pretrained language model: frames are encoded into continuous visual tokens and the model learns to predict the next visual token, Qwen3-1.7B is mid-trained on raw clips from YT-Temporal-1B and then given the same image-text instruction tuning as a model without mid-training, yielding 2.9 points higher on video benchmarks and 5.1 points higher on image benchmarks, with text average rising from 48.0 to 48.9, and caption prediction offering no advantage over visual-token prediction.

  7. PhysBrain 1.5: A Physical Foundation Model Unifying Language, Action, and Future World States in One Autoregressive Vocabulary

    PhysBrain 1.5 builds on a pretrained Qwen3-VL backbone extended with dedicated action and visual-state tokens, formulating language responses, structured spatial outputs, end-effector trajectories, and future world states as discrete tokens jointly learned under a single next-token prediction objective, with embodied pretraining supervision drawn entirely from human interaction videos (egocentric, synchronized ego-exocentric, and panoramic recordings structured into task-centered episodes) and supervised fine-tuning combining human demonstrations, real-robot trajectories, and simulated experience; across 28 embodied spatial-intelligence and planning benchmarks the 8B version reaches an overall average of 72.

You’re all caught up