A study conducted across seven hospitals within the Yale New Haven (Conn.) Health System found that Epic's Deterioration Index underperformed compared to other early warning scores in detecting clinical deterioration.
The study, published Oct. 15 in JAMA Open Network, analyzed 362,926 patient encounters between March 9, 2019, and Nov. 9, 2023, and evaluated six early warning scores, including AI-driven and traditional models. These scores are integrated into EHR systems to help clinicians identify and respond to patient deterioration early.
Among the six models assessed, Epic's Deterioration Index received one of the lowest AUROC scores at 0.808. The AUROC, or Area Under the Receiver Operating Characteristic curve, is a critical performance metric for classification models, especially in binary tasks such as predicting whether a patient will deteriorate. This score reflects the model's ability to distinguish between "positive" cases of deterioration and "negative" cases, where no deterioration occurs.
In contrast, the machine learning-based eCARTv5 model performed the best, achieving an AUROC of 0.895 and provided better lead times for predicting adverse events such as ICU transfers or death within 24 hours. The National Early Warning Score followed in the rankings with an AUROC of 0.831.
Given the differences in accuracy and lead time across the tools, greater transparency and oversight of early warning systems are necessary to ensure the best outcomes for patients, the authors wrote.
"Epic's Deterioration Index has performed well in other academic studies and in clinical settings. A Vanderbilt University Medical Center study, for example, found that the model performed better than both MEWS and NEWS at predicting deterioration three and six hours early, while a Stanford Health Care study showed the model contributed to a 10% reduction in care escalations for hospitalized patients. At RWJBarnabas Health, the model helped save an estimated 100 lives and reduced mortality rates for patients transferred to intensive care by nearly 30% within six months of implementation," a spokesperson for Epic told Becker's in an emailed statement. "Epic provides an open system so that healthcare organizations, like Yale New Haven Health System, can use whatever model they find performs best with their patient population."