Although Amazon's artificial intelligence-powered voice assistant Alexa has thousands of health and wellness skills — not to mention experts have touted its potential in the industry — a Quartz investigation revealed many of those skills give poor health advice.
Customers can either use Alexa's voice command search function to ask about their health issue, which Alexa then processes to query answers from an Amazon "verified trusted source" such as Wikipedia or WebMD, or they can download one of Alexa's skills — which function similar to a smartphone app.
While anyone can develop their own Alexa skill and submit it to Amazon, company policy requires health-related skills meet certain criteria, including:
- Skills cannot collect customers' personal health information
- Skills cannot claim to be life-saving in their names or descriptions
- Skills must include a disclaimer stating they aren't "medical advice" and note customers should seek additional information from their healthcare providers
Failure to abide by these policies — namely provide a disclaimer — means Amazon can remove the skill from its library.
However, Quartz reporters Katherine Foley and Youyou Zhou audited 915 health and wellness skills, and identified 65 medical skills without a disclaimer.
When Quartz notified Amazon of its findings July 2, company spokespeople said it "routinely audit[s] Alexa skills, and if they are not compliant, we work quickly to communicate with the developer and take action on behalf of customers," adding that it conducts the audits on an ongoing basis. Those 65 skills were still available as of July 10.
To test how well the skills perform, the reporters then subdivided the 915 health and fitness skills into four categories based on the above Amazon policy:
- Healthcare: skills that offer basic information on health or healthcare
- Trackers: skills that help monitor health
- Guided activities: skills that serve as basic fitness coaches
- Diagnostics and treatment: skills that claim to diagnose diseases or provide treatment options
Skills in Quartz's diagnostics and treatment category — which Quartz claimed were clearly offering health advice despite the disclaimers — frequently picked up on the wrong keywords from user questions, and subsequently offered inaccurate advice. For example, when asked about flu symptoms, one skill diverted to a definition of the bird flu. Other times, skills "rambled on for five minutes, listing out every possible cause," the report states.
Physicians that spoke with Quartz and reviewed the reporters' analysis agreed Alexa skills provide mediocre health advice in the best-case scenarios.
"These answers all sound like they just extract information from Wikipedia (which contains a lot of incorrect information) using very simple 'yes' or 'no' algorithms," Tian Wang, MD, a neurologist at Washington, D.C.-based Georgetown University, told the news outlet via email. "Based on my judgment, these are all bad responses. If this is the best that Silicon Valley can offer, then it will still take a long way for the robotic machine to take over our jobs."
Rumors have been swirling that Amazon is building a health team around Alexa, which is not yet HIPAA-compliant. But since Alexa can't yet parse out actual conversations, it appears she shouldn't be stepping in for human physicians.
To access the full Quartz report, click here.