This appeared last week:
Don’t judge a mental health app by RCT alone
Authored by Samineh Sanatkar Samuel Harvey
THE randomised controlled trial (RCT) model is a vital tool in clinical practice and helps establish, with reasonable precision, the usefulness of novel drugs and therapeutic techniques. As is implied in the name, RCT models are characterised by employing randomisation techniques and the presence of a control group. Successful randomisation to experimental and control groups ensures that all participant characteristics are equally distributed across the experimental conditions. With that, any confounding factors, both known and unknown, should not unduly influence the outcomes in one group more than in the other. The experimenter can be reasonably certain that any changes in the outcome variable can be linked back to the variable of interest (ie, the treatment method or drug administered). The control group is equally important – only with a control is it possible to establish how big a treatment effect is compared with other options such as treatment as usual or some other treatment option.
While RCTs need to remain the gold standard when deciding if a new treatment or intervention works, a positive RCT result alone does not guarantee that a new program will have a meaningful impact. The limitations of traditional RCTs are particularly stark when evaluating new technologies.
Over the past decade, there has been an exponential increase in the number of digital mental health programs (ie, mental health websites and mobile applications) that have been developed and tested. Many of these new online or app-based interventions have been shown in RCTs to be effective (here, here, and here), leading to the suggestion that digital interventions may be an important part of the solution for persistently high rates of mental health symptoms in most countries. However, very little of this evidence has flowed through into the applications and programs being used in the real world. A recent investigation published in Nature estimated that only 2% of the popular depression smartphone apps had a reasonable evidence base.
Given these concerns, a key question that needs to be addressed is what additional insights are needed, beyond the traditional RCT analysis, in understanding the potential role for evidence-based digital mental health solutions. In answering this, a first step is to examine more closely what occurs within RCTs.
Engagement with a digital mental health tool is frequently defined as adherence to a predefined protocol. Thus, likely real-world uptake and usage often remains unknown and untested before rolling out the digital mental health tool as a public health resource. Differences between these contexts may arise because, for example, individuals who sign up to a digital mental health research project are more motivated to engage with such a program and often receive regular prompts from the research team to take up and continue program usage. These factors in turn can lead to an overestimation of program use – and thus effectiveness – in naturalistic settings.
Lots more discussion here:
This is interesting and useful in deepening our understanding on just where we are with our deployment and use of mental health apps.
It seems clear well designed apps have a place – especially as the rising incidence of all sorts of mental illness and distress is rising remorselessly and the availability of clinicians to deliver care is pretty constrained leading to the need for ‘self help’ (via usage of an app) in many cases.
The traditional RCT compares a treatment – the usage of an app in this case – against either no treatment or the best non app intervention that is available for matched patients. Clearly, as the authors point out, this sort of trial is a good first step, but for overall confidence that success is achieved post trial follow up is clearly needed as is collection of all the relevant aspects of the care delivered in terms of app usage, frequency and so on.
As time goes on we would also hope to see comparisons of different apps to identify what features are best within an app and so on!
For sure, this is the start of a journey which will lead to better outcomes over time! It should be an interesting and important ride.
David.
No comments:
Post a Comment