How to Quickly Assess the Scientific Rigour of a Clinical Trial
To appreciate what makes the first successful placebo-controlled study of a meditation app so special, we must first understand the general sentiment towards mobile healthcare (“mHealth”) in 2015, when Mobio Interactive was founded. Bluntly put, mHealth was not accepted as “real” medicine for mental health (it largely still isn’t). This view was justified in 2015, as at that time, no strong evidence existed to suggest that mHealth for mental illness was doing anything beyond evoking the placebo effect. That’s where Katie’s study made such a difference.
Broadly speaking, there are three levels of rigour within clinical study designs. From least to most rigorous, they are:
- Pre/post studies
- Waitlist-controlled studies
- Placebo-controlled studies
These clinical trial designs are named for the type of comparison group used to explore if a given treatment is effective. As you’ll soon see, the differences are far from subtle and they dramatically impact what can be confidently concluded from a study’s results.
Pre/post studies compare the condition of participants after treatment, to the condition of the same participants before treatment. A good thing about this type of design is that its “within-participant” analysis ensures that the “pre” and “post” groups are inherently balanced for things like gender, age, and initial illness severity, or anything else that might impact treatment efficacy. This consistency in the comparison groups reduces variability, which in turn allows for a higher sensitivity to detect pre/post differences that may result from the treatment.
Sounds good right? Well, kinda. Higher sensitivity may not actually be a good thing because statistically significant effects can be observed even if they are not medically significant. Remember: the use of the word “significant” in statistics only describes the probability that an observed difference is real. It says nothing about why the difference occurred, or even if that difference would be meaningful to a patient.
There are two reasons why the pre/post study is the least rigorous design. First, study participants and their environments routinely change over time, but not always in predictable ways. To illustrate, imagine a study examining how a treatment effects people’s happiness. In this imaginary study, most participants begin the treatment during the winter and finish in the late spring. If we then find that they are happier after the treatment, how do we know if the reason they are happier is the treatment or the warmer weather? Or what if there is a dramatic down turn in the economy and many participants lose their jobs? If we find they are statistically less happy on average, does that mean the treatment failed?
Second, the pre/post design fails to control for the placebo effect. Often described as the “most robust effect in medicine,” the placebo effect describes the very real observation that people tend to feel better when they believe there is a reason for them to feel better. For most study participants, just being part of a clinical trial produces a very real expectation that they will be better off. In a pre/post design, where all the participants know they are receiving treatment, the placebo effect can be extremely large. In the context of studies that measure mental wellbeing, this is a very real concern.
Waitlist-controlled clinical trial designs solve the first problem with pre/post studies by randomising participants into two groups. One group receives the treatment while the other group receives nothing (or nothing different from their standard care). Both groups are then examined before and after the experimental treatment. If a difference is found within the treatment group and not within the waitlist control group, then we know that environmental factors, such as a change in seasons or people losing their jobs, are not the reason for the difference between the treatment and waitlist groups.
Like pre/post designs, however, waitlist designs do not control for the placebo effect.
Placebo-controlled clinical trials, as the name suggests, contain a group of participants that receive a placebo treatment. In the simplest placebo-controlled design, one group of participants will receive the treatment, while the other will receive a fake (placebo) treatment. When the placebo controlled design is done well, neither group knows which group they are in. The participants are “blind” to the “study arm” into which they have been randomly placed. Not only does this type of design control for the placebo effect, it has the potential to eliminate the placebo effect altogether because all participants are aware they may be in the fake (placebo) treatment group. Thus their overall expectation of treatment effect may be reduced. Placebo-controlled studies, often also called “actively controlled” randomised controlled trials (RCTs) are the gold standard in medical research.
Now you are an expert in clinical trial design. At least, you can now roughly judge the rigour of a study by knowing which design was used.
It should be mentioned that sometimes a placebo-controlled study is hard to justify ethically, if it means preventing patients from receiving a treatment that might save their life or otherwise majorly benefit their health. For this reason, placebo-controlled studies often evoke a “cross-over” design in which the participants that are in the placebo group eventually cross-over into the treatment group after the post-placebo measurements have been obtained.
There is a lot more complexity we could get into, but generally speaking, this additional complexity is only nuance. The basic fact is that, all else remaining constant, placebo-controlled studies are a rigorous and fairly simple-to-understand way to research novel medical treatments.
Katie’s study examining the effect of our digital therapeutic on the mental health of young adults was a placebo-controlled study. That’s a major reason why I feel it will be remembered as one of the most important early mHealth studies completed in mental healthcare.
To dive into what Katie discovered, follow me and you’ll be notified soon when the description is published here on Medium.
This article is part of a series:
- Humble Beginnings for a New Era in Medicine
- How to Quickly Assess the Scientific Rigour of a Clinical Trial
- What the First Successful Placebo-Controlled Trial with a Mindfulness App Actually Taught Us
- The More Subtle Message from the First Successful Placebo-Controlled Trial with a Meditation App (Coming soon)
Credits: Ms Annabel Chang, interning at Mobio Interactive, contributed to this article. Copy and line editing completed by Prof. Judith Scholes of Cascadia Editors Collective.
