Mindfulness study retracted: Problems with MBSR/MBCT paper

Earlier this year Plos One took the step of retracting a well known and widely cited mindfulness related study.

It's getting harder to make sense of mindfulness research
Is it getting harder to make sense of mindfulness research?

This particular study is a first for the Science of Meditation blog. Whilst we have featured a number of papers that have highlighted methodological problems in meditation and mindfulness research, this is the first time that we have drawn attention to a retraction of a peer-reviewed study.

The basis for the retraction is outlined in detail on the Plos One website, but we have paraphrased the three main points.

  • The handling Academic Editor shared an affiliation with three of the authors, although this didn’t emerge until post-publication.
  • Two of the authors hold or had held positions at an institute offering mindfulness related products and services in clinical contexts.
  • The paper has a number of errors including pooling of results which led to double counting and incorrect effect estimates in figures contained in the study.

There’s not a lot more we need to add to the identified issues, they speak for themselves. However, when considered as part of the ongoing crisis in mindfulness research they make troubling reading.

A general defence used in cognitive psychology when the findings of mindfulness studies are criticised is, the peer review system is self-regulating. That when studies are found to be below the expected standard, they are usually rejected during review. Or at the very least other experts working in the field have the opportunity to raise concerns in print. This retraction challenges this basic notion. Significant issues with both the methodology and the editorial process can endure, thus, have the ability to influence the scientific and popular understanding of mindfulness. According to Google Scholar, this Gotink et al. study has been cited over 400 times, the citing publications, in turn, used by thousands more papers. The details provided on the Plos One website indicate the study has received 50,000 views.

Rather than simply criticize this study or the journal, I would like to ask what this retraction show us about the way that meditation technologies are being treated by clinical and scientific institutions?

Authors: Gotink, R. A., Chu, P., Busschbach, J. J., Benson, H., Fricchione, G. L., & Hunink, M. M.

Year: 2015

Title: Standardised mindfulness-based interventions in healthcare: an overview of systematic reviews and meta-analyses of RCTs

Summary: This is a research paper that (at the time of writing) had been retracted by PLOS ONE.

“In light of the methodological issue and concerns about the validity of the study’s results, the PLOS ONE Editors retract this article. We regret that these issues were not fully addressed prior to the article’s publication.”

An extensive explanation of the reasons behind the retraction are published on the Plos One website which can be reached by following the link below.

Link: https://journals.plos.org

How far can we trust meditation research?

Strategic reviews are challenging the popular perception of the beneficial effects of mindfulness

How far can we trust meditation research?
Looking for answers from meditation?

How far can we trust meditation research?

No matter how I tried to write the headline it came out as provocative. My intention wasn’t to be controversial, rather I wanted to articulate concerns that have been rumbling around the science of meditation and mindfulness for decades. At the heart of this story are two important yet unresolved issues. Firstly how does psychology and neuroscience understand meditation and what do the results of meditation research really mean?

The limited prosocial effects of meditation is a recent systematic review of research undertaken by Ute Kreplin, Miguel Farias and Inti Brazil. The study has been discussed in the meditation community at some length so I’m not going to review it here. But to summarize, the positive effects of meditation on prosocial behaviours (compassion, empathy, aggression, connectedness and prejudice) in healthy adults were only observed in compassion and empathy scores. However increases to compassion were just seen when the meditation teacher was one of the co-authors of the research paper or when the study used a ‘passive’ control group (this means the control group were on a waiting list). These findings are suggestive of flaws and possibly ‘bias’ in some of those studies  that demonstrated significant results. In an interview with Ute Kreplin published in the international Buddhist journal Tricycle, a number of broader issues have been highlighted, it’s those that I’d like to push around a little now. Leaving to one side the methodological flaws which are the main focus of the Tricycle interview, let me draw attention to the potential causes of the ongoing limitations in our attempts to evidence the effects of meditation.

woman meditating

It should be stressed that the Kreplin, Farias and Brazil paper is one of a number of reviews that came to similar conclusions, that many (possibly most) of the published studies reporting significant effects in non clinical populations had methodological and/or theoretical flaws. And as Kreplin hinted, published research tends only to be the tip of the iceberg, studies that fail to show measurable changes in meditators rarely see the light of day. So the examples analyzed in strategic reviews are not the full picture of meditation research, they offer a very selective (positive) account of the scientific landscape. And yet the common perception grows that meditation is a panacea able to deliver a range of desirable outcomes to almost anyone willing to practice a method.

“At this moment in time the science generally isn’t helping us to understand the benefits of meditation…”

Stephen Gene Morris

By way of transparency I should make it clear that I’m an experienced meditator and confident of the great benefits of the practice. My interest in contemplative science comes from the perspectives of both a trained cognitive psychologist and a practicing Buddhist. From my experience of teaching traditional meditation systems, it is unrealistic to claim that a few weeks of meditation practice automatically leads to ‘significant’ change. Some practitioners do progress rapidly, embracing the transformative potential of meditation, but others fall away after only a few weeks, sometimes disillusioned and unfulfilled. This is a difficult subject to address coming from a traditional meditation perspective, because judging or criticizing the progress of another practitioner is something of a taboo. But to enhance the wider understanding of meditation this point needs to be stressed. There is no reason to assume that the meditation method alone leads to change, the method is an integral part of a firmly established theoretical framework. The effects of meditation tend to be meditated by several factors such as, individual capacity, participant motivation and qualitative differences between the teacher or teaching systems.

The contemporary scientific investigation of meditation typically takes the reductionist approach, stripping out components that might confound the results of an experiment, such as variability in the method or differences in the environment. But isolating the cause (meditation method) and the effect (empirical change in the participant) is difficult, and in complex aspects of human behaviour such as empathy or compassion it may be beyond the scope of many experiments. Consider that large numbers of the participants in meditation studies are likely to be undergraduates ‘pressed’ into research projects, obliged to participate in return for course credits. If meditation doesn’t always work for the people who choose to attend classes in the wider community why should things be any different in an experimental setting?

ancient architecture art asia

The ‘expectation’ that a meditation method in itself leads to change is not supported by the human history. This idea may eventually be confirmed by science but the data gathered so far is inconclusive. We know that a number of meditation scientists are committed practitioners, so perhaps they have first hand experience of the benefits of meditation or mindfulness. Is this as Kreplin suggests, part of the problem? Could the experiential knowledge of the results of meditating introduce subconscious bias into research methodology? I’m a meditator I know about the benefits of regular practice but I can see dangers to the credibility of meditation systems if claims based on poor science are over-hyped. The lack of long term studies for secular forms of meditation should also be a serious concern.

The failure to establish robust findings in meditation research begs a further question, without reliable replicated science how does the delivery of meditation technologies continue to grow in society? If scientists are raising questions about the claims made in individual studies why isn’t this filtering down more into health care, public policy and the media? If meditation and mindfulness interventions cannot be shown to work, or deliver predictable results, confidence in meditation generally may decline. It might also lead to an erosion in the status of experimental psychology as a provider of independent and reliable data.

These few paragraphs are simply an introduction to the subject, the start of a very long road. It can be argued that the contemporary western scientific investigation of meditation began in the 1970s, since when perhaps as many as 10,000 studies have been published. But based on the findings from recent strategic reviews our scientific understanding of meditation is at a surprisingly preliminary stage.

Notes

The Kreplin, Farias and Brazil study can be found here.