This appeared last week:
Online symptom checkers: still a long way to go
Authored by Nicole MacKee
ONLINE and mobile application symptom checkers get the diagnosis right first time in just a third of instances, but the only Australian-based checker included in a recent study,, Healthdirect, did slightly better than the rest.
In pre-coronavirus disease 2019 (COVID-19) times (November 2018 – January 2019), Australian researchers put 36 freely available online and mobile application symptom checkers under the spotlight to determine their effectiveness in providing diagnostic and triage advice.
The researchers evaluated the symptom checkers using 48 medical condition vignettes (1170 diagnostic vignettes and 688 triage vignettes). They concluded that symptom checkers may provide unsuitable or incomplete diagnostic or triage advice for users in Australia, resulting in inappropriate care advice.
Their findings, published in the MJA, showed that the correct diagnosis was listed first in only 36% of tests, and the first 10 results in 58% of tests. For triage services, the correct advice was provided in 49% of cases, including 60% of emergency and urgent cases, but only 30–40% of less serious case vignettes.
Healthdirect, which was launched in 2015 and is funded by the federal government, was the only Australian-based symptom checker included in the study. The service, which provides only triage advice, did slightly better than other symptom checkers in the MJA study, providing correct triage advice in 28 of 46 vignettes (61%).
Associate Professor Adam Dunn, Head of the Discipline of Biomedical Informatics and Digital Health at the University of Sydney, said it was important to consider the purpose of symptom checkers in determining their utility.
“Is a symptom checker just something you want to Google online to reassure yourself, or is it something you want to actually action and you want to deploy as a health service to try to get people the right care at the right time?” he asked.
He said it was likely that most symptom checkers included in the study were not designed to replace a full triage service.
“The most important thing is that we really shouldn’t be judging all the symptom checkers together. We don’t need 50 symptom checkers; we only need one and we need one good one. And the best performing [triage tool in the study] … happened to be the Healthdirect one, which is likely to be the most appropriate one for the Australian context.”
Associate Professor Dunn said finding the right risk balance was crucial to the effectiveness of online symptom checkers.
“There needs to be balance between sending too many people to health services that they don’t need, and not directing people to health services who need them,” he said. “It’s a real sensitivity and specificity problem. And that’s why it’s such a critical thing right now [during a pandemic], because we need to play with this balance between being risk averse and making sure people get care when they need it versus not overwhelming the health system.”
He said the UK’s artificial intelligence triage service, Babylon Health, provided a recent example of the potential dangers of not getting this balance right.
When a UK doctor tested the service by inputting symptoms related to a heart attack, the symptom checker advised him to stay home.
Dr Marie-Louise Stokes, Chief Medical Officer of Healthdirect Australia, said the Healthdirect symptom checker aimed to provide information and advice to support someone who was unsure about what to do about their symptoms.
More here:
Here is the Abstract and some additional commentary.
The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia
Michella G Hill, Moira Sim and Brennen Mills
Med J Aust || doi: 10.5694/mja2.50600
Published online: 18 May 2020
Published online: 18 May 2020
Abstract
Objectives: To investigate the quality of diagnostic and triage advice provided by free website and mobile application symptom checkers (SCs) accessible in Australia.
Design: 36 SCs providing medical diagnosis or triage advice were tested with 48 medical condition vignettes (1170 diagnosis vignette tests, 688 triage vignette tests).
Main outcome measures: Correct diagnosis advice (provided in first, the top three or top ten diagnosis results); correct triage advice (appropriate triage category recommended).
Results: The 27 diagnostic SCs listed the correct diagnosis first in 421 of 1170 SC vignette tests (36%; 95% CI, 31–42%), among the top three results in 606 tests (52%; 95% CI, 47–59%), and among the top ten results in 681 tests (58%; 95% CI, 53–65%). SCs using artificial intelligence algorithms listed the correct diagnosis first in 46% of tests (95% CI, 40–57%), compared with 32% (95% CI, 26–38%) for other SCs. The mean rate of first correct results for individual SCs ranged between 12% and 61%. The 19 triage SCs provided correct advice for 338 of 688 vignette tests (49%; 95% CI, 44–54%). Appropriate triage advice was more frequent for emergency care (63%; 95% CI, 52–71%) and urgent care vignette tests (56%; 95% CI, 52–75%) than for non‐urgent care (30%; 95% CI, 11–39%) and self‐care tests (40%; 95% CI, 26–49%).
Conclusion: The quality of diagnostic advice varied between SCs, and triage advice was generally risk‐averse, often recommending more urgent care than appropriate.
The known: Australians are searching the internet for health care advice with increasing frequency. Symptom checkers are common online tools for obtaining diagnostic and triage advice.
The new: Thirty‐six symptom checkers were evaluated with a range of clinical vignettes. The correct diagnosis was listed first in 36% of tests, and included among the first ten results in 58%; triage advice was appropriate in 49% of cases, including about 60% of emergency and urgent cases but only 30–40% of less serious case vignettes.
The implications: Symptom checkers may provide unsuitable or incomplete diagnostic or triage advice for users in Australia, resulting in inappropriate care advice.
Here is the link to the original article.
Clearly these systems don’t really look ready for prime time, or ready to replace your friendly GP in most circumstances!
This does rather make one wonder just how powerful AI can be, and what can be done to make it work better?
David.
No comments:
Post a Comment