OmitEHR stops asking an LLM to score every sentence: clinical-summary omission screening runs 1.6 to 1.7 times faster and 5.6 to 6.7 times cheaper
Synopsis
Health systems use large language models to draft patient-record summaries, and evaluations now rank omissions, clinically important record content missing from the summary, as the most common failure mode. OmitEHR splits omission detection in two: it learns from historical clinician-written summaries which source sentences get included, then has an LLM check coverage only for the top-ranked candidates. On 25 blinded admissions, a mean of 5.64 omissions per summary were confirmed by at least one clinician, versus 3.32 for a full LLM pipeline, at 17.7 to 19.5 seconds and $0.03 to $0.13 per summary.
Interpretation
Omission detection no longer needs an LLM to judge sentence by sentence at inference which record content matters; clinical salience becomes a quantity learned offline. Earlier reference-free omission detectors either did not model clinical importance or used an LLM to score every source sentence at inference, so cost grew with record length. On 750 held-out admissions and more than 446,000 sentences, with AUROC and AUPRC computed within each admission and then averaged, the learned contextual salience model beat structural heuristics, lexical models and a prompted LLM judge at predicting which source sentences belong in the brief hospital course.
The pipeline first predicts which source sentences a clinician would write into the summary, then checks only those high-scoring sentences for coverage by the generated summary and surfaces uncovered ones as candidate omissions. The salience model takes BioClinRoBERTa sentence embeddings plus structural features such as position in the note, admission length and repetition count, and passes them through a two-layer Transformer encoder so each sentence's score depends on the whole admission. Training used 5,000 internal medicine admissions and 2.9 million source sentences from a large academic medical center, labeled by GPT-5-mini on a three-level scale of not related, indirectly related and directly included, with the latter two merged as salient.
On 25 brief hospital courses generated by Gemini-2.5-pro, two board-certified clinicians blindly reviewed the ten highest-ranked candidate omissions from each pipeline. Of 250 OmitEHR candidates, 141 (56.4%) were judged clinically meaningful omissions by at least one clinician, versus 83 (33.2%) for the full LLM pipeline; both clinicians confirmed 71 (28.4%) versus 17 (6.8%). Per summary, a mean of 5.64 omissions were confirmed by at least one clinician and 2.84 by both, versus 3.32 and 0.68 for the full LLM pipeline; 24 of 25 admissions had at least one omission confirmed by both clinicians, versus 12 of 25.
Perspective
The demonstrated setting is brief hospital course generation for 5,000 internal medicine admissions at one large academic medical center, with source records built from all hospitalization notes except the discharge summary, and omission validity was reviewed by two clinicians on 25 summaries generated by Gemini-2.5-pro.
The salience labels agree only moderately with clinician annotations, so ranking results measure agreement with an LLM-derived target; the clinician evaluation measures precision among surfaced candidates rather than recall over all clinically important omissions, and 25 admissions, two reviewers unfamiliar with the cases, and accompanying LLM rationales may all have influenced judgments.
