Skip to main content
Back to timeline
arXivSource publication:

GPT-5.6-Sol ranks solid-state recipes: hiding all chemistry leaves overall accuracy at 0.72, hiding temperature drops it to 0.56

Synopsis

A team built a benchmark from 240 measured solid-state syntheses across eight oxide targets, each run through the same 30 firing conditions, then had GPT-5.6-Sol compare two recipes for the same target and pick the one expected to yield higher target-phase purity. Its pairwise accuracy was 0.72; hiding all chemical identity moved overall accuracy from 0.74 to 0.72, while on the six thermally driven targets accuracy was 0.68 without chemistry and 0.56 without temperature. The authors therefore describe the model as a thermally informed prior rather than a ranker that understands target chemistry.

Source-provided article image: What Do LLMs Use to Judge Synthesis Recipes? An Experimental Analysis on 240 Solid-State Syntheses
arXiv

Interpretation

Large language models are now used to propose solid-state synthesis conditions, but there was no benchmark grounded in measured outcomes that could separate reasoning from recall. Published recipes rarely record failures, seldom report purity, and offer no clear basis for saying one recipe beats another, so existing evaluations largely measure rediscovery of known routes. The benchmark contains 240 synthesis attempts, 233 of which returned a purity measured by X-ray diffraction; 7 samples fused to crucibles, evaporated or were lost. Eight oxide targets each ran the same 30 conditions, and three targets had no prior experimental report of any kind.

GPT-5.6-Sol is only asked which of two recipes for the same target will reach higher target-phase purity; it never writes a route and never returns a number. Every pair is judged in both presentation orders to cancel position preference, and the preferences are aggregated with a regularized Bradley–Terry model into a ranking of that target's recipe bank. Across 2,795 scorable recipe pairs, pairwise accuracy was 0.72; by target it ranged from 0.86 on M6 to 0.45 on M2, the latter below chance.

Ranking is driven predominantly by thermal features, with target chemistry acting as a modulator. Withholding all chemical identity moved overall accuracy from 0.74 to 0.72, whereas on the six thermally driven targets accuracy was 0.68 without chemistry and 0.56 without temperature. Per-target ranking accuracy tracked the purity–peak-temperature correlation, with a Spearman correlation of +0.91 (n = 8, two-sided p = 0.002).

Perspective

The benchmark covers eight oxide targets with 30 firing conditions each, with purity from X-ray diffraction and Rietveld refinement; the authors note the targets are largely thermally driven, and whether the findings carry to systems governed by precursor reactivity must be established case by case.

Ranking performance was higher on known than on unknown targets, but the authors state that eight targets cannot establish that group difference with confidence; they also note the benchmark remains small and the known–novel comparison is bounded the same way.