Asked to hand-write weights, frontier LLMs skip the Transformer: a sparse fact table beats the same model's written Transformer on ARC with about 1,800 fewer parameters
Synopsis
Given no gradient training, ten labeled examples and a sandbox, three frontier LLMs asked to design their own target model all abandoned the Transformer and instead wrote plain-text facts into a sparse table read by a fixed procedure. Claude Opus 5.5's table model scored higher ARC multiple-choice accuracy than the most accurate Transformer the same LLM wrote, using about 1,800 fewer parameters; the authors then propose an associative-memory model that reads a fixed number of rows and runs up to 17 times faster on a GPU than a same-sized Transformer.
Interpretation
When an LLM is free to choose the target structure, it writes a sparse fact table plus a fixed lookup procedure rather than a Transformer, and all three frontier LLMs converge on this kind of design. Earlier weight-space programming filled LLM-written weights into released Transformers, state-space models or RNNs whose structure was fixed in advance; here the structure is also left to the writing LLM. On ARC four-option science questions, Claude Opus 5.5's designed model beats the most accurate released Transformer the same LLM wrote while using about 1,800 fewer parameters; the same LLM's written two-layer Transformer reaches 36.1% against 49.6% to 54.4% for the written table models.
Without gradients, a written model must trace each prediction to specific written rows and let an edit to one row change only the few answers that depend on it, and a sparse table satisfies both. The same facts compiled into a Transformer score below plain BM25 keyword search over the raw sentences, by 8.0 points on average across Claude Opus 5.5's six written two-layer Transformers, consistent with interference between dense keys. Keeping only the single best-matching row per option in Claude Opus 5.5's model drops accuracy from 50.8% to 49.1%, and deleting one row changes 0.5 to 1.8 answers on average, against 4.8 for the first written Transformer.
To remove the cost of scoring every stored fact for every question, the authors fix the forward pass and leave only the weight writing to the LLM: each question reads 120 pair-table rows and computes one product with the concept table. LLM-designed table models score every stored fact per question, and kernels launched per question grow from 13 in the first generation to 118 in the fifth; only one of 46 variants beats a same-sized Transformer at its fastest input, with a median 9.4 times slower. Claude Opus 5.5's ten associative-memory models reach 49.6% to 52.8% ARC accuracy, within about three points of the designed models, and answer the 3,530 test questions in 18 ms on an RTX 3090, where a same-sized Transformer is 7 times slower at the first-generation size and 17 times slower at the fifth.
Perspective
The claim covers current LLMs writing weights from random initialization without running on data or gradient training, evaluated mainly on ARC four-option science questions, with extensions to SciQ, OpenBookQA, MMLU, SQuAD, SciQ generation, AG News and CIFAR-10. The speed advantage is measured on a single RTX 3090 and holds up to the concept counts tested, between about 15,000 and 32,000 with a dense concept table and about 100,000 with the row-list variant.
Fine-tuned pretrained models remain far more accurate, with Qwen3.5-0.8B at 82.1% on ARC, and the authors limit the claim to which structure to build when writing weights. The LLMs may pick a table because retrieval is a familiar way to answer without training, and the data do not separate a structure suited to writing from one that is merely known. The written models only recall facts and do not chain them across steps, with ARC-Challenge accuracy staying near 38%.
