Skip to main content
Back to timeline
arXivSource publication:

A causal model learns to edit its own draft: insert, replace, delete without recomputing, 59.2 on HumanEval

Synopsis

Autoregressive decoding commits to its output in generation order and cannot revise it, while diffusion models can correct tokens but stay bound to fixed length and absolute positions. AutoIndexer decouples generation order from display order, using a marker head to decide when to edit and a pointer head to localize the edit span, with perturbation training that teaches the model to undo insertions, substitutions, and deletions. On a Qwen3 8B backbone it preserves standard benchmark performance and scores 59.2 on HumanEval without instructions, above the matched LoRA baseline's 51.2 and the diffusion baselines.

Source-provided article image: AutoIndexer: Flexible-Order Decoding by Training Causal Models on Chains of Edits
Figure 1 · arXiv

Interpretation

A generative model can now revisit its own output after writing it, inserting, replacing, or deleting any number of tokens without a length constraint. Autoregressive decoding could not revise once generated, and diffusion models could correct tokens but produced fixed-length output with fixed token positions, preventing arbitrary insertion or deletion. On a Qwen3 8B backbone, AutoIndexer essentially ties the matched Qwen3 LoRA baseline on MMLU, ARC-Challenge, GSM8K, and MATH500 while outperforming the LLaDA 8B and Dream 7B diffusion baselines; its HumanEval score without instructions is 59.2 against the matched LoRA baseline's 51.2.

The model works through a text-editor-like mechanism: it decides when to start an edit, points at the span to change, writes replacement content, and returns. Unlike Fill-in-the-Middle, the edit span is not supplied externally; the model predicts its own start and end cursors. Training perturbs clean sequences with random inverse edit operations and teaches the model to undo them; in a free-generation diagnostic, edit spans for deletion and substitution are localized correctly in nearly seventy percent of cases, with 53 of 100 exact repairs for deletion and 18 of 100 for substitution.

Perspective

The mechanism is validated on a Qwen3 8B backbone, covering general language, math, and code benchmarks plus insertion, deletion, substitution, and no-op edit diagnostics. It targets settings that require repeated revision of output, such as code correction, long-form writing, and structured documents. Code is released in a public repository.

Insertion edits are the hardest to trigger, with fewer than half of corrupted drafts firing an edit; even when they fire, replacement content often falls short of the required length. The free-generation diagnostic uses 100 windows per operation with an average edit size of 5, and results come from a single checkpoint.