Some refer to generative AI's tendency to spew wrong information as "hallucinating" or "making stuff up." These researchers call it "botsh-t."
Regardless of what it's dubbed, healthcare providers must be on guard against the "epistemic risks" of chatbots dispensing inaccurate or untruthful facts, the researchers wrote July 17 in Harvard Business Review.
They pointed to digital health company Babylon Health's GP at Hand app that used a chatbot for patient triage. Several physicians found the app was giving incorrect information, including wrongly suggesting that two conditions didn't require emergency treatment — they could have indicated a heart attack. The company has since gone out of business.
The authors also cited a 2023 presentation by researchers to the American Society for Health-System Pharmacists that three-fourths of ChatGPT's responses to drug-related questions were wrong or incomplete and a March systematic review that found ChatGPT often delivers untrue healthcare information in a persuasive manner.
Thus, healthcare is a sector that would benefit from "practice-specific" rather than general-purpose chatbots such as ChatGPT, the authors said.
"Promising directions for practice-specific chatbots include ones that employ emerging LLM [large language model] technologies such as retrieval augmented generation as well as users developing a set of prompt engineering practices," the researchers wrote. "To ensure reliability and avoid botsh-t risks, the scope of chatbot use should be more confined and the guardrails (i.e., rules, guidelines, or limitations for chatbot use) more restrictive."
The article was written by Ian McCarthy, PhD, the W.J. VanDusen Professor of Innovation and Operations Management at Burnaby, British Columbia-based Simon Fraser University and a professor at the Center in Leadership, Innovation and Organisation at Rome-based Luiss; Timothy Hannigan, PhD, associate professor of organization theory and entrepreneurship at the (Edmonton) Alberta School of Business, University of Alberta; and André Spicer, PhD, dean and professor of organizational behavior at Bayes Business School at City, University of London.