Physicians and advocates are raising concerns about the use of artificial intelligence in medicine due to the lack of regulation and oversight.
No single entity governs the AI landscape, but the FDA, who approves programs to be used in medicine, is facing criticism for its lackluster process. One research team found evidence of real-world use of AI programs for critically ill patients was "completely absent" or based on computer models. The Philadelphia-based University of Pennsylvania and the Los Angeles-based University of Southern California found some programs are approved based on their similarities to existing devices. Another study found of 118 AI tools cleared by the FDA in 2021, only one described the geographic and racial breakdown of patients the program was trained on.
While drugs and some devices are tested on patients before approval, software programs are not. Jeffrey Shuren, MD, the chief of the FDA's medical device division, acknowledged to The New York Times in an Oct. 30 report that there needs to be continuing efforts to ensure programs live up to their promises after they are cleared.
Patient safety advocates warned that without better oversight, providers could be using AI programs that misdiagnose disease, rely on racially biased data or violate patients' privacy, Politico reported.
The FDA has limited reach in which products are approved for sale and has no authority over programs that health systems build and use internally. Many large systems build their own tools that affect care and coverage with little to no oversight, the report said.
Physicians are raising more questions about AI programs, such as how they are built, how many people they were tested on, and how likely it is to identify something a physician missed, but current laws protect that information from public view due to intellectual property rights.
An October 2022 bill called the Blueprint for an AI Bill of Rights was issued by the Biden administration but has not been picked up by lawmakers. Meanwhile, healthcare systems are racing ahead with AI adoption and some fear physicians will trust the programs' judgements too readily.
The FDA is planning a novel system that will include ongoing audits and certification of AI products, Politico said. Some industry leaders also said physicians are a form of regulator, since they make the final decision with AI co-pilots.
On Oct. 30, the White House released new guidance on safe, secure and trustworthy AI programs.