Skip to main content
Back to timeline
arXivSource publication:

MedPhysBench puts medical physics hand calculations and error checks on the exam table: four models near perfect on calculation, yet one accepts a correction it always applies itself

Synopsis

MedPhysBench turns medical physics hand calculations and the checking of a colleague's work into an automatically graded benchmark: ten seeded generators cover radiotherapy, diagnostic imaging, nuclear medicine and radiation protection, reference answers come from code implementing the governing protocols, and grading only converts units and applies a tolerance band. On 1,200 held-out test items, 12 models scored 33.3% to 100.0% on calculation, with four at 99.3% or above; five models detected at least 96.7% of planted errors with at most 4% false alarms in both checking formats. The same model can compute correctly and check incorrectly, so the two abilities need separate measurement.

Source-provided article image: MedPhysBench: A Benchmark for Evaluating Language Models on Medical Physics Calculations and Error Detection
arXiv

Interpretation

Medical physics calculations now have a benchmark graded by protocol code, where a model must return a number with its unit rather than pick a letter. Earlier evaluations mostly used examination questions or had physicists rate open-ended answers; a multiple-choice item grades a selected letter, while a physicist's product is a number with a unit or a judgment that a colleague's number is right. On 1,200 held-out test items, the 12 models scored 33.3% to 100.0% on calculation, with GPT-6 Astra, GPT-6.1 Sol, Gemini 4 Argon and Claude Opus 5.5 at 99.3% or above; exact McNemar tests with Holm's adjustment did not separate any two of these four.

The benchmark also turns one set of inputs into two checking tasks: review the colleague's full worked steps, or see only the colleague's final result. The two formats correspond to two ways a clinical check is organized, step-by-step review or recomputation and comparison; no earlier study measured both formats for the same models. Five models detected at least 96.7% of planted errors with at most 4% false alarms in both formats; for four of the other seven, showing only the result raised false alarms significantly, and gpt-oss-20b went from 5.3% to 38.7% on correct results.

Calculation and checking come apart: a model that computes correctly can accept a colleague's worksheet that omits the same correction. The test split planted the omitted isocentric inverse-square correction in a single item, and four of the 10 models that saw it as a result check missed it: Claude Opus 5.5, Claude Sonnet 5.5 and gpt-oss-120b accepted the wrong result, and Mistral Large 3 gave no verdict. In a targeted test designed after the main results, 20 public parameter sets were each presented five ways; the three models at the ceiling named the missing correction in all 20 planted worksheets, while GPT-6 Luna applied the correction in 17 of its own 20 calculations yet accepted 16 worksheets that left it out.

Perspective

The items are text only and supply every input, and the data tables are generated rather than taken from real devices; the protocols are North American (TG-51 rather than the IAEA code of practice, NCRP shielding reports) and the formalisms are the hand-calculation forms in Table 1. Checking covers five of the ten generators, with at most one planted error per item and no errors in the inputs themselves; the targeted test covers one error in one calculation.

After the test runs the author read NCRP Report 151 and found two points on which the level-3 shielding items depart from the report or leave a factor open, so four answers graded incorrect actually followed the report; version 1.1 will correct these items. Each model answered each item once, and when five cheaper models repeated the test split a day later their calculation accuracy fell by 1.0 to 3.5 points, so a single-run score may shift between runs. The author also notes that the generators, the second implementation and the analysis code were written with one AI coding assistant from one specification, so their agreement shows the code computes what the specification says, not that the specification follows the governing documents.