What the First Successful Placebo-Controlled Trial with a Mindfulness App Actually Taught Us

“Katie’s study was the first successful human trial of its kind. Never before had a meditation app been shown to outperform a placebo control.”

It’s one thing to hear in the news or read a headline somewhere that a “new study” has shown for the “first time” that this or that is possible. It’s another thing to source the original paper and digest the scientific jargon to decipher what that new study actually teaches us. But who got time for that?

My goal here is to save you time. I summarise the findings of the first-ever scientific study that proved mindfulness training from an app can lead to real world benefits, even when compared against placebo.

The placebo-controlled trial completed by Prof. Norm Farb, myself and Norm’s MSc student Katie Walsh was pivotal because it showed that young adults learning mindfulness skills with the mobile mental health platform AmDTx experienced both mental health and cognitive benefits — even when controlling for the all-powerful and ever-present placebo-induced expectation bias. But what actually were the specific benefits observed? And are we so confident in these results?

First, let’s learn a couple key psychological terms:

• Trait Effects: Persisting or long-term impact. More broadly “traits” are elements of your character that are relatively consistent over time, for example a positive disposition or persistent tendency to stay up late.
• State Effects: A term used to describe immediate or short-term impact. More broadly, “state” is how you feel in the moment. Your state can change in an instant, for example when you hear good news, or (if you’re like me) hear that lovely, gentle sound of fresh coffee beans rotating in a hand grinder.

Katie’s study examined both the “trait effects” (long-term impact) of AmDTx on mental wellbeing and cognitive performance, as well as the “state effects” (immediate impact) of AmDTx on stress, mood and heart rate — all relative to a placebo control.

AmDTx is a clinically validated resilience mobile platform, freely available on app stores (iOS, Android) and it has a wide variety of meditation training and music therapy in six languages.

We’ll go through the data from Katie’s study bit-by-bit, starting with the trait effects since these are probably the most meaningful.

Trait Effects: Mental health was measured via subjective psychological questionnaires (also called “surveys” or “scales”), as is still the standard practice in medicine and health research. In Katie’s paper, mental health is usually described as “overall mental wellness”.

Three separate wellbeing traits were examined:

1. Acceptance — not avoiding distressing events or thoughts
2. Awareness — being aware of thoughts, physical sensations, and the outside world
3. Openness — welcoming change, personal growth, and different emotions

These traits were identified via a mathematical process called “principal components analysis” using the results from a battery of psychological questionnaires. Briefly, the PCA approach uses statistics to identify clusters of questions that have similar answers, whereby each cluster is a “principal component”. Clustering is performed without expectation bias because the participants and researchers cannot know how the study participants answer the questions until after the study is completed.

In Katie’s study, the traits of acceptance and awareness were significantly influenced by AmDTx.

Evidence AmDTx improves mental health. “Acceptance” and “Awareness” are important elements of a person’s overall mental wellbeing, specifically referring to being accepting of distressing events or thoughts and being aware of thoughts, internal sensations and the outside world. The “z-score” is a statistical measure indicating how similar each group is at each point in time is to the population average. It shows that training with AmDTx, but not placebo, increases a person’s acceptance and awareness.

Trait Effects: Cognitive performance was examined through a computerised task developed by the Centre for Research on Safe Driving (Lakehead University, Canada). The “Attention Network Test” measures focus (or “attentional control” as used in Katie’s paper) according to three specific abilities:

1. Alerting — achieving and maintaining attention to incoming stimuli
2. Conflict monitoring — resolving conflict amongst responses
3. Orienting — directing attention to sensory input

We use these abilities everyday. For example, imagine you have to complete some task at work. Alerting is the state of concentrating your cognitive resources on that task. Suddenly, your phone buzzes. You must now decide whether or not to ignore the buzzing phone. Is the buzzing relevant for the task you are performing? Conflict monitoring is the decision-making process that governs the decision to ignore or answer, to stay focused or shift attention. Finally, orienting is the specific action that results from the conflict monitoring decision. If the buzzing is part of the work you are performing, then orienting to answer the phone is the appropriate response (from a work task efficiency point of view). Better attentional control across these three specific abilities leads to better overall cognitive performance and productivity.

In the version of the Attention Network Test used in Katie’s study, participants are instructed to focus on a cross in the centre of a computer screen. Then, a row of five vehicles randomly appear either above or below the cross. Participants are asked to decide as fast and accurately as possible whether the vehicle in the middle is facing left or right. Flanking vehicles could be facing the same or different directions. Sometimes, an asterisks appears a half second before the vehicles, serving as a signal for where the vehicles will appear on the screen. Both reaction time and accuracy are recorded.

Alerting is quantified by the change in reaction time that results from priming vehicle location with an asterisks.

Conflict monitoring is quantified by the change in accuracy that results from flipping the direction of the flanking vehicles relative to the direction of the vehicle in the centre.

Orienting is quantified by the total reaction time in trials where an asterisks reveals the location.

Katie’s analysis showed that AmDTx improves conflict monitoring.

Evidence AmDTx improves cognitive performance. “Conflict Effect” is an important measure of a person’s ability to monitor and respond appropriately in the face of conflicting information, and it can be measured computerised tasks like the “Attention Network Test”. This test uses a person’s reaction time to identify the orientation of an object under various scenarios. The reaction time differences between various type of tasks help tease out various aspects of a person’s ability to focus. The “z-score” is a statistical measure indicating how similar each group is at each point in time is to the population average. It shows that training with AmDTx, but not placebo, increases a person’s ability to monitor conflict.

State Effects: The impact of AmDTx on stress, mood and heart rate was examined using highly scalable tools directly embedded within AmDTx and the placebo control. Participants were asked to make use of these embedded tools both before and after their meditation or cognitive training sessions.

Some of the psycho-biometric tools embedded within AmDTx (and the placebo control) used in Katie’s study to quantify mental state and vitals.

State Effects: Stress was quantified subjectively through self-reports on a scale from “none” to “max” through an easy-to-use slider. The technique was successful in revealing an initial difference in stress levels between the participants who were randomised into the AmDTx or placebo arms of the study. Specifically, the participants in the AmDTx group were found to have less stress on average. Ideally, both groups would start with the same level of stress, and the difference between the groups could have made it more difficult to deliver reductions in stress in the AmDTx group compared to the placebo group, since the AmDTx group had comparatively lower stress to begin with. Regardless, the results show that meditation training with AmDTx results in a drop in stress that directly comes from using AmDTx. No effect on stress was observed in the placebo group.

State Effects: Mood was quantified implicitly through the self-selection of words that corresponded to how a participant was feeling in the moment. In the scientific sense, mood is a part of the umbrella term “affect”. Affect describes “states of feeling” comprised of two components: “valence” and “arousal”. Valence represents how pleasant or unpleasant a feeling is. Think “is what I’m feeling a good or bad feeling?”. Meanwhile, arousal is the intensity of the feeling. Think “am I ecstatic or just pleased?”. Mood was measured along these axes with a “circumplex”, and all the participants needed to do was tap on the words that represent how they feel. Each word has its own numerical score associated, but the score is hidden from the participants in order to minimise bias and obtain a more implicit measurement.

The technique was successful in finding that AmDTx results in an improvement in mood that directly comes from using AmDTx. No effect on mood was observed in the placebo group.

Evidence AmDTx improves stress and mood. Stress and mood — both key determinants of overall wellbeing — were measured using daily check-ins (or “Snapshots” as they are called within AmDTx) captured before and after meditation and cognitive training for the AmDTx and placebo groups, respectively. The “z-score” is a statistical measure indicating how similar each group is at each point in time is to the population average. It shows that a 10–15 min meditation with AmDTx, but not cognitive brain training with the placebo, decreases stress while improving mood.

State Effects: Heart Rate was quantified objectively through computer vision with the mobile device camera. For the technical engineers interested in how this was accomplished, the mathematics and initial verification results are published here. To be clear, an effect of meditation on heart rate was not expected. Rather, the measure was included as part of a continuing effort at Mobio Interactive to develop and validate methods that can objectively quantify mental wellbeing at scale. We have recently made a fundamental step forward in this regard by training a deep neural network to use heart rate and associated digital biomarkers to accurately predict stress from a 30 second selfie video, as described in another peer-reviewed paper here.

The computer vision technique was successful in finding that AmDTx has no effect on heart rate. In addition, a trend emerged where heart rate increased in the placebo group after engaging in the cognitive training exercises. This finding may suggest some form of anticipatory arousal in the participants using the placebo control.

Evidence cognitive training increases heart rate. Measurement of human biomarkers from smartphone video is a major recent advancement in health technologies. Here, an increase in heart rate while playing a cognitive training exercise was picked up by algorithms owned by Mobio Interactive. The “z-score” is a statistical measure indicating how similar each group is at each point in time is to the population average. It shows the placebo group, but not the AmDTx group, had an elevated heart rate after cognitive training. Since the publication of Katie’s paper, Mobio Interactive has used the heart rate and stress slider data to build an AI tool that can accurately quantify stress (read more here).

It is just one study, and just one study in isolation is rarely that significant. However, Katie’s study was the first successful human trial of its kind. Never before had a meditation app been shown to outperform a placebo control. Moreover, the results are statistically significant and mutually supportive. Finally, additional trials, including those with patient populations using AmDTx, corroborate Katie’s pivotal findings (e.g., here and here and many more yet unpublished).

The humble conclusion: Effective and accessible healthcare can be delivered at scale.

If meaningful healthcare can be delivered on a smartphone, then it can be accessed by virtually everyone. Moreover, if reliable measurements of wellbeing can be simultaneously delivered at scale, then it may also be possible for healthcare to become personalised at scale. That is the world Mobio Interactive is helping us achieve — a world in which precision psychiatry at scale is not just possible, but commonplace.

With the accelerating speed of technological progress and adoption, it will not take long for these technologies to transform healthcare as we know it.

This article is part of an ongoing series:

1. Humble Beginnings for a New Era in Medicine
2. How to Quickly Assess the Scientific Rigour of a Clinical Trial
3. What the First Successful Placebo-Controlled Trial with a Mindfulness App Actually Taught Us
4. The More Subtle Message from the First Successful Placebo-Controlled Trial with a Meditation App (Coming soon)

Credits: Ms Annabel Chang, interning at Mobio Interactive, contributed to this article. Copy and line editing completed by Prof. Judith Scholes of Cascadia Editors Collective.