A study conducted across seven hospitals within the Yale New Haven (Conn.) Health System found that Epic's Deterioration Index underperformed compared to other early warning scores in detecting clinical deterioration.
The study, published Oct. 15 in JAMA Open Network, analyzed 362,926 patient encounters between March 9, 2019, and Nov. 9, 2023, and evaluated six early warning scores, including AI-driven and traditional models. These scores are integrated into EHR systems to help clinicians identify and respond to patient deterioration early.
Among the six models assessed, Epic's Deterioration Index received one of the lowest AUROC scores at 0.808. The AUROC, or Area Under the Receiver Operating Characteristic curve, is a critical performance metric for classification models, especially in binary tasks such as predicting whether a patient will deteriorate. This score reflects the model's ability to distinguish between "positive" cases of deterioration and "negative" cases, where no deterioration occurs.
In contrast, the machine learning-based eCARTv5 model performed the best, achieving an AUROC of 0.895 and provided better lead times for predicting adverse events such as ICU transfers or death within 24 hours. The National Early Warning Score followed in the rankings with an AUROC of 0.831.
Given the differences in accuracy and lead time across the tools, greater transparency and oversight of early warning systems are necessary to ensure the best outcomes for patients, the authors wrote.
Becker's reached out to Epic for comment and will update this story if more information becomes available.