A correct discharge diagnosis is not a safe route: ClinTraj's 300-case audit found 55.2% of hospital trajectories with an acceptable final diagnosis passed through a potentially harmful decision
Synopsis
Clinical AI evaluation has mostly scored final diagnoses, and even stepwise evaluations that release evidence over time ask only whether a system reaches the right answer. ClinTraj rates every test, treatment and disposition for potential harm and clinical validity using only the evidence available at its checkpoint, calibrated against a physician reference standard. Across 300 cases from a Chinese cardiology department, MIMIC-IV and ASCENT, 55.2% of hospital trajectories with an acceptable final diagnosis passed through a decision with moderate or severe potential harm; structured corrections resolved 73.6% of first errors, yet 30.2% of corrected continuations introduced new potentially unsafe decisions.
Interpretation
The object of evaluation moves from the final diagnosis to every decision a patient passes through: among trajectories whose final diagnosis physicians judged acceptable, a substantial share still contained intermediate decisions that could have harmed the patient. Earlier stepwise evaluations counted only whether a system reached the right diagnosis, asked the right questions or kept testing cheap, while safety evaluations rated single responses; neither measured the potential harm of a decision sequence. On 96 cardiology admissions from a Chinese teaching hospital, 55.2% [42.9, 66.9]% of trajectories with an acceptable final diagnosis passed through at least one decision with moderate or severe potential harm and 24.8% [16.3, 36.5]% through at least one clinically unacceptable decision; the shares were 76.2% and 51.0% on 96 MIMIC-IV admissions and 39.6% and 24.3% on 108 ASCENT cases.
Whether corrective feedback repairs an error depends on whether it carries clinical content, and repairing an error is not the same as restoring a safe trajectory. A bare reject-and-reconsider resolved 24.8% of first errors at the next checkpoint, whereas a structured correction stating the problem, the corrected action and its clinical rationale resolved 73.6%. On 150 physician-rated first-error events, defensible recovery rose from 15.0% after rejection alone to 56.0% after a structured correction, but a new potentially unsafe problem followed 30.2% of structured corrections within two checkpoints, against 19.7% of bare rejections.
Potential harm is the dimension on which an automated judge and clinicians diverge most, so headline safety rates need a physician reference standard to set their scale. Judge and reference standard agreed with Cohen's κ of 0.54 on final-diagnosis acceptability, 0.28 on unacceptable decisions and only 0.21 on potentially unsafe decisions. Weighted to all judged decisions, the judge rated 50.0% potentially unsafe and the reference standard 23.5%; even among the judge's severe ratings, the reference standard placed 71.0% at a lower level.
Perspective
The demonstrated scope is 96 inpatient admissions from the cardiology department of a tertiary teaching hospital in China, 96 cardiology emergency admissions from one US centre in MIMIC-IV, and 108 multi-specialty teaching vignettes from ASCENT; each record is replayed checkpoint by checkpoint with only the evidence released by then, and decisions are judged against what was knowable at that time rather than against counterfactual courses of care.
Potential harm is itself a contested clinical judgment: in a pilot study separate from the formal annotation, all 10 annotators with clinical backgrounds rated the same 50 first errors, and their agreement on whether a decision carried moderate or severe potential harm was Fleiss' κ of 0.16; the judge's harm calls also run well above the reference standard. The authors name mapping where model judges and clinicians, and clinicians among themselves, diverge on potential harm as a next question.
