Tied on benchmarks, 17 points apart on real work: Epoch tests Fable 5.2 and GPT-6.2 Astra on its own tasks
Synopsis
Epoch turned its own real work into evaluation tasks and hand-graded six models against internal standards, with 100% meaning the level expected of Epoch employees. Claude Fable 5.2 and GPT-6.2 Astra roughly tie for a new record on the Epoch Capabilities Index, yet no model met the employee standard on these tasks, and GPT-6.2 Astra averaged 65%, 17 points below Fable 5.2. On the task of adding a data center to Epoch's AI Data Centers database, Fable 5.2 scored 97.5% and Opus 5.5 scored 87.5%, reaching contractor level, while GPT-6.2 Astra scored 65%.
Interpretation
On tasks derived directly from Epoch's day-to-day work, none of the six tested models met the standard Epoch expects of its employees, even though two of them are roughly tied on general benchmarks. The two models had roughly tied and set a new record on the Epoch Capabilities Index, and strong performance on typical benchmarks does not always carry over to real workplace tasks. Hand-graded against Epoch's internal standards, with 100% representing the employee standard, GPT-6.2 Astra averaged 65%, 17 points below Fable 5.2 despite a roughly tied ECI score.
Data-center analysis is the closest of these tasks to automation, with two Claude models judged comparable to what Epoch receives from contractors. That is a clear improvement over Fable 5.1 and GPT-6 Astra, and contractor work also typically needs some revision before publication. On that task Fable 5.2 scored 97.5% and Opus 5.5 scored 87.5%, where a perfect score means Epoch could publish the output without changes, compared with 65% for GPT-6.2 Astra.
Models generally reuse existing work when designing experiments, which limits how well they answer the research question. Good experiment design requires building conditions suited to the question rather than reusing what already exists. In the chess-variants task, every tested model used pre-existing variants instead of creating its own, and those variants are likely in the evaluated models' training data, so they cannot test out-of-distribution generalization.
Models still struggle to judge which findings matter and to present them, and they prefer their own outputs. Research quality depends not only on producing results but on identifying meaningful findings and presenting them clearly. Where a human output met the standard, all models preferred their own output in every case except one, where Gemini 4 Argon preferred the human output on the hidden-messages task, and their justifications often rewarded superficial rigor such as the number of experiments run.
Perspective
The evaluation's tasks come directly from Epoch's own work and are hand-graded by its staff against internal standards to assess open-ended capabilities and progress toward work automation, and the authors say they will keep evaluating new models and updating the task suite.
The authors note a limited task suite and a single run per model per task, so scores are noisy and should not be read as a general measure of what AI can or cannot do across all work tasks; grading involves subjective judgment, so specific numbers could change with different rubric definitions or graders; and because elicitation and setup changed since the previous report, scores between the two reports are not directly comparable.
