A recent study conducted by the University of Michigan has revealed shortcomings in the effectiveness of the proprietary artificial intelligence software known as the Epic Sepsis Model.
Developed to serve as an early warning system for sepsis, the AI tool, integrated into Epic's EHR software, struggles to differentiate between high- and low-risk patients before they receive treatments, according to the research published in the New England Journal of Medicine AI.
The Epic Sepsis Model automatically generates sepsis risk estimates in the medical records of hospitalized patients every 20 minutes. The intention is to provide clinicians with timely information to identify patients at risk of sepsis before the symptoms become severe.
However, the study pointed out key challenges in the AI's performance.
One hurdle the researchers identified is the timing misalignment between when the AI processes information and when it becomes relevant to clinicians. The study analyzed data from 77,000 adults hospitalized at the Ann Arbor-based University of Michigan Health, revealing that the AI's accuracy dropped significantly when using patient data recorded before sepsis criteria were met.
The researchers found that the Epic model correctly identified high-risk patients 87% of the time when considering predictions made throughout the entire hospital stay. However, this accuracy dropped to 62% when utilizing patient data recorded before the patient met sepsis criteria and further decreased to 53% when predictions were limited to before a blood culture had been ordered.
"We suspect that some of the health data that the Epic Sepsis Model relies on encodes, perhaps unintentionally, clinician suspicion that the patient has sepsis," Jenna Wiens, PhD, corresponding author of the study and associate professor of computer science and engineering at the University of Michigan, said in a Feb. 15 news release.
A spokesperson for Epic told Becker's that multiple organizations have studied the same sepsis model as was included in the NEJM AI paper, but in clinical settings using different workflows.
"They have published on the positive impact that Epic's sepsis model has had on patient outcomes, such as reducing the odds of sepsis mortality by 44% and improving the timeliness of antibiotic administration by about 40 minutes without increasing antibiotic usage," the spokesperson said.