Artificial intelligence drafted patient portal messages of similar quality to those written by human clinicians, according to a new study from New York City-based NYU Langone Health.
Since 2023, in an effort to reduce provider burnout, the health system has used a private version of OpenAI's GPT-4 generative AI technology to draft responses to Epic EHR In Basket inquiries. In the July 16 JAMA Network Open study, 16 primary care physicians were asked to assess 344 pairs of patient portal responses, not knowing which were written by AI or humans. The scores for accuracy, completeness and tone did not differ statistically.
"This work demonstrates that the AI tool can build high-quality draft responses to patient requests," said corresponding author Devin Mann, MD, strategic director of digital health innovation for NYU Langone Health, in a July 16 statement. "With this physician approval in place, GenAI message quality will be equal in the near future in quality, communication style, and usability, to responses generated by humans."
In the study, generative AI outperformed human providers in understandability and tone by 9.5%, and was more than twice as likely to be considered empathetic and 62% more likely to use language that conveyed positivity and affiliation ("we are in this together"). However, AI responses were 38% longer and 31% more likely to use complex language, writing at an eighth-grade level versus a sixth-grade level for the human providers.
The authors said future research is needed to determine whether the use of private patient data improves the AI's performance.