This appeared last week.
‘Botched’ aged care AI camera trial generates 12,000 false alerts
Justin Hendry
Editor
The sheer number of alerts created alert fatigue that “overwhelmed” already overworked staff, and in at least one instance, the persistent false alerts meant a staff member did not respond to a true resident fall event.
The CCTV pilot began at Mt Pleasant Aged Care and Hortgate House in March 2021, with cameras and microphones installed in common areas and resident bedrooms. Consent for the recording devices to be turned on in bedrooms was given by 41 of the 57 residents or their guardians.
The project aimed at “exploring the acceptability and viability of using surveillance and monitoring within residential care settings” against the backdrop of improving the quality of support and safety in aged care.
Confronting reports about the sector were heard during the three-year Royal Commission into Aged Care Quality and Safety, which made dozens of recommendations for improvement in its final report in March 2021.
The system was programmed to “detect specific movements or sounds including falls, assist, calls for help, or screams”, with a text message sent to a monitoring centre when an event occurs. The message is then passed onto staff at the aged card facility.
The AI used was also “designed to learn over time and improve its ability to recognise the actions or audio cues specific to the sites and residents”.
But an independent evaluation by PwC found the “AI technology had a high rate of alerts”, meaning the pilot was “not yet sufficiently accurate at detecting incidents in a residential aged care setting”.
“Despite improvements made over time, during the 12 months of the pilot, the system generated over 12,000 arts across the two sites that were not verified as ‘true events’,” the report said, adding that a high percentage of these alerts were for staff crouching – a programmed movement.
“In these cases, while the system was detecting movement or sounds that it was programmed to detect, it was unable to reliably distinguish between the programmed events and similar movements or sounds that are reasonable to expect in residential care.”
The report said that while some ‘false alerts’ during the AI learning period was expected, the high volume was “unexpected”, leading to the introduction of a ‘sit fall’ alert that recognised crouching into the pilot in October 2021.
However, as the motion of dropping to one knee resembles a “common care position used by nurses and care attendants – the ‘knights position’”, the system also detected this movement as a fall, resulting in a spike of false alerts.
Further maturing of the system within the final months of the pilot meant that it was able to “detect some true quality and safety events, including falls by residents”, with 22 per cent of actual events detected, compared with just two per cent at the start of the trial.
But alert fatigue continued, and in the “final months of the pilot, staff were no longer able to respond to every alert”, leading to “at least one instance where staff did not respond to an alert that turned out to be a ‘true’ resident fall”.
“The evaluation found that the number number of ‘false alerts’ experienced at the sites by the CCTV system was unexpected and unacceptable to staff,” the report said.
“The number of false alerts in the first few months of the pilot means that the staff at both sites were overwhelmed by the workload associated with responding to alerts.”
The report found that at the conclusion of the pilot there was no evidence that the “AI-based surveillance used in the pilot had influence either positive or negatively, the quality and safety of the care provided at the sites”.
More here:
https://www.innovationaus.com/botched-aged-care-ai-camera-trial-generates-12000-false-alerts/
There is also coverage here:
South Australia's aged care AI trial produced 12,000 false alarms
By Ry Crozier on Sep 1, 2022 8:35AM
In the space of a year.
A trial of CCTV and AI technology to detect accidents or abuse in two aged care facilities in South Australia produced 12,000 false alarms in a year, a review has revealed.
The Australian-first project was intended to pilot the use of cameras and AI to aid monitoring of residents under care, with a view to making the lives of staff easier.
However, a review of the pilot by PwC [pdf] showed the technology produced false positives at such a rate that alert fatigue among staff set in, and at least one actual incident - a resident falling over - went unresponded to.
The technology was programmed to detect four key incident types, defined as “falls, assist, call for help and/or screams”.
However, PwC found there were concerns from the outset “that the way in which these events had been programmed were not well aligned to the common movement patterns of residents at the sites.”
In addition, the system was tuned to be overly sensitive to noise levels in facilities, and was unable to distinguish between inanimate objects and people until it was patched.
The end result was a flood of “false alerts” that overwhelmed onsite staff and facility managers.
PwC said that “a threshold of 10 false alerts per day were anticipated by SA Health and the pilot sites”.
On average, the number of false alerts a day was triple that amount, and exceeded 12,000 across two sites over the year-long trial.
“A high percentage of these alerts were sit-fall events which involved staff performing a bend to knee (crouching) motion” to mobilise a resident, the review found.
Across the trial, the AI algorithm flagged “movements or sounds that are reasonably expected in residential care” as problematic and repeatedly raised alerts.
While the algorithm did become better at detecting actually problematic events, “it still initiated a high number of false alerts per month across the two sites” even as the 12-month pilot wound down, PwC found.
“In these final months of the pilot, staff were no longer able to respond to every alert,” it said.
More here:
This is a pretty sad outcome I have to say and really was a mess but a few questions go unanswered as far as I can tell.
The first and most obvious is why the trial was not paused or aborted until the AI could be really improved?
Second, and related, is just why there was not real testing period prior to the trial formally started where it could be discovered just what the error rate was and corrective action taken before the trial proper to sort the algorithm out?
You can really understand the frustration of the care home staff with all these ‘false alarms’ disrupting the working day!
I also wonder why, with such a personally invasive study for the patients that the Ethics Committee did not insist on at least monthly reporting on false alarms etc. I guess it is easy to be wise after the event!
Clearly complex and expensive trials – as this on must have been – need better supervision on the way through!
David.
"Second, and related, is just why there was not real testing period prior to the trial formally started where it could be discovered just what the error rate was and corrective action taken before the trial proper to sort the algorithm out?"
ReplyDeleteBecause once the technology had been developed and implemented, the problem had obviously been solved - as far as the project was concerned.
Health and aged care are about much more than just technology. It's about time the bureaucrats and managers got that past their blinkered view of the world and into their heads.
Given the amount of hubris demonstrated at the federal level over the past 10-15 years, that's a rather forlorn hope.
I don't think anyone teaches the need for testing in IT courses these days. Just like the need for parallel running has been discarded. The only thing that hasn't been discarded is that 80% complete right is good enough to implement and deploy, and fix the other 20% and any problems later!
ReplyDelete