Skip to main content
Back to timeline
JMIR Medical InformaticsSource publication:

SOFA's 1996 expert thresholds give way to outcome-fitted cutpoints, lifting ICU mortality AUROC from 0.751 to 0.811

Synopsis

The SOFA score used in intensive care has kept the thresholds and 0-to-4 point bands set by expert consensus in 1996, never re-estimated from outcome data. In 14,848 adult first ICU admissions from the multicenter eICU database, a study re-learned each organ component's cutpoints and weights while keeping an additive, integer, bedside-computable form, using supervised discretization plus weight optimization under monotonicity and integrality constraints. In 10-fold nested cross-validation grouped by hospital, the recalibrated score reached an ICU mortality AUROC of 0.811 versus 0.751 for a prespecified SOFA-derived reference score, a paired difference of 0.059.

Source-provided article image: Interpretable Recalibration of a Sequential Organ Failure Assessment-Derived Score Using Machine Learning-Guided Optimization: Multicenter Cohort Study.
PubMed

Interpretation

A multicenter retrospective study handed the cutpoints and integer weights of a SOFA-derived score back to outcome data, while keeping the item-by-item additive form that can be tallied at the bedside. SOFA's thresholds and 0-to-4 bands came from 1996 expert consensus and were never re-estimated from outcome data; machine learning models that discriminate better usually abandon this additive, auditable structure. In 10-fold nested cross-validation grouped by hospital, the recalibrated score reached an ICU mortality AUROC of 0.811 (95% CI 0.800-0.819) versus 0.751 (95% CI 0.740-0.761) for the prespecified SOFA-derived reference score, a paired difference of 0.059 (95% CI 0.053-0.067); AUPRC was 0.495 versus 0.393, a paired difference of 0.102.

The procedure runs in two stages: each continuous variable is cut into data-determined intervals, then each interval receives a constrained integer point value, and a logistic layer maps the total to a mortality probability. Supervised discretization builds candidate cutpoints with univariate decision trees, with the interval count chosen by a penalized validation criterion; integer weights are optimized under monotonicity, integrality and scale-anchoring constraints, which prevent weight collapse and keep directions clinically consistent. The analysis included 14,848 adult first ICU admissions in the eICU database with complete day 1 SOFA-related measurements, drawn from 153 hospitals, with an ICU mortality of 17.03% (2529/14,848).

The recalibrated score approaches unconstrained XGBoost in discrimination and exceeds penalized logistic regression fitted on the same folds. On the same candidate predictors and identical hospital-grouped outer folds, XGBoost reached an AUROC of 0.818 and penalized logistic regression 0.775, against 0.811 for the recalibrated score. Across 1000 patient-level resampling replications, the median paired AUROC improvement was 0.060 and was positive in all 1000; in a prespecified sensitivity analysis expanding the cohort to 61,469 encounters, the median paired AUROC difference was 0.037.

Perspective

Results come from 14,848 adult first ICU admissions with complete day 1 SOFA-related measurements across 153 of 208 US hospitals in the eICU Collaborative Research Database for 2014-2015; all performance estimates come from out-of-fold predictions of hospital-grouped cross-validation, and the final score was refitted on the full cohort without contributing performance numbers. The authors state the score is a mortality risk model on its own 0-71 scale, not a replacement for canonical SOFA's description of organ dysfunction, and provide an open-source web calculator; external and prospective validation are named as prerequisites before clinical use.

Complete-case selection left the primary cohort sicker and more intensively monitored, and the authors state the primary performance estimates should not be extrapolated directly to unselected ICU admissions. On calibration, the recalibrated score had a lower Brier score (0.112 vs 0.123) but higher integrated calibration index, E50, E90 and Emax than the reference, which the authors read as no consistent calibration advantage. Vasoactive dose units could not be harmonized across hospitals and urine output was largely unavailable, so the comparator is a SOFA-derived reference score rather than the original SOFA; some cutpoints varied substantially across the 1000 replications.