Category: critical mindfulness

What does the replication crisis mean for the science of meditation and mindfulness?

The scientific study of meditation has been limited by a replication crisis and a mindfulness crisis. What does this mean and what is the way forward for contemplative science?

Replication, an important element of the scientific method

For at least the last 20 years psychological science has been facing a replication crisis.1 For those who don’t know, the replication crisis reflects a deep-seated problem in how psychology carries out scientific investigations. In essence, it means that many psychological studies from the past may not be as reliable as we thought they were. This uncertainty has implications for the way psychology is conducted, and it may accelerate the declining public confidence in science more generally.

The replication crisis is visible in social sciences and medicine, but not all disciplines have been affected to the same extent. Although social psychology is regarded as having the most significant replication problem, the phenomenon is present in other areas such as the science of meditation. For an experimental study to be scientifically reliable, it generally has to be repeated, repeated by other scientists in alternative locations. If the results are the same, or at least very similar on each of these occasions, the scientific findings are much more likely to be reliable. However, if scientific claims cannot be replicated, it raises questions about how they were initially established, and the extent to which they can be generalised across populations. So if one scientific study found that regular meditation reduced the effects of hay fever, we’d expect to see the same results in other studies carried out in the same way. If not it could mean that there was an unusual characteristic in the first study or some problem in the method. It is for these reasons meditation scientists, teachers and practitioners are reevaluating what they know about the health benefits of meditation.

A failure to replicate doesn’t necessarily prove that scientific findings in the original study were not reliable, but it raises questions over the extent to which the claims are robust. So any isolated evidence for the health and wellbeing benefits of meditation has to be seen as a pilot study, preliminary in nature. In most cases, without replication, we cannot assume that findings from any individual study could apply to the general population.

For those of us working with meditation, the replication crisis is compounded because we are also facing a ‘mindfulness crisis’. The mindfulness crisis describes systemic problems in meditation research that go back 50 years. At least half a dozen studies published since 2015 have identified and described the meditation and mindfulness research crisis. Its main characteristics are conflicting theoretical understandings of meditation and methodological limitations which include low levels of replication. Although many, perhaps most scientific studies of meditation have been impacted by problems linked to the replication and mindfulness crises. The scientific enthusiasm for meditation technologies since the 1970s has been so great that one-off unreplicated claims for the benefits of meditation have not always been critically evaluated by the scientific community. As Van Dam and colleagues have demonstrated, this has led to the ‘hyping’ of preliminary evidence as robust scientific findings.2

Measures are being taken to address the replication crisis within psychology more generally. These initiatives have had a limited effect so far, and their impact will have to be evaluated over the longer term. To overcome the problems being experienced in Contemplative Science, there are three issues that need to be considered by the scientific and practice communities. Firstly the development of a system where unreplicated, preliminary findings are not treated in the same way as robust, replicated work. Secondly, address the pressing need to understand and resolve the known theoretical and methodological limitations. And finally, to review the procession of the scientific understanding of meditation since the 1930s to make sense of the current crisis and diagnose its underlying causes.

Notes

1 Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?. American Psychologist, 70(6), 487.

2 Van Dam, N. T., van Vugt, M. K., Vago, D. R., Schmalzl, L., Saron, C. D., Olendzki, A., … & Meyer, D. E. (2018). Mind the hype: A critical evaluation and prescriptive agenda for research on mindfulness and meditation. Perspectives on psychological science, 13(1), 36-61.

Advertisement

Some critical thoughts on mindfulness?

The scientific study of meditation has produced 7,000 peer reviewed studies, but our understanding is still described as preliminary. Has the time come for a more critical approach to mindfulness?

Reliable measurement is a central tenet of experimental psychology, but deciding what to measure is a much more complex question.
Accurate measurement is crucial to science, but deciding what to measure is a challenging question in mindfulness research

So broad has the field of mindfulness become that we find conflicting, coexisting and complementary perspectives among practices. Traditional mindfulness methods include explicitly and implicitly nondual understandings that appear abstract (incommensurable) to positivist scientific enquiry. However, the exponential growth in the modern forms of mindfulness sits primarily within the positivist ontology of experimental psychology. Positivism creates understandings linked to established tenets. In particular, it (a) assumes psychological phenomena follow deterministic (causal) patterns, (b) that explanations for behaviour can be generalised beyond narrow experimental settings and (c) elaborate explanations are rejected in favour of parsimonious accounts. Also, (d) that reductive investigations can offer understandings of complex human behaviours and cognitive states and (e) experiments produce events which can be reliably measured. In many respects, the positivist approach has successfully contributed a great deal to our understandings of human behaviour. However, its ability to explain and evaluate traditional and medicalised mindfulness and meditation is facing challenges from within the scientific community1.

This image has an empty alt attribute; its file name is Wilhelm_Wundt.jpg
Wilhelm Maximilian Wundt – Founder of experimental psychology

Wilhelm Maximilian Wundt was the architect of experimental psychology. Towards the end of the 19th century, Wundt became one of the first researchers to conceptualise and investigate psychology as a field of science rather than philosophy2. But Wundt was also very clear about the limitations of the experimental approach, that nuanced human behaviours were not accessible to methodologies rooted in positivism3. After all, we humans are replete with agency, we can take for or against an idea with little rational justification. Given the spectrum of human experience, Wundt’s position seems to hold some merit. How can experiments be created that fully explain and generalise highly individualised behaviour? Behaviour created and maintained within abstract inner worlds, which is supported by unique environmental conditions? Consider that meditation is the mediation of consciousness, of which psychology only has a rudimentary understanding. One of the issues that Wundt’s concerns highlight is ‘fitness for purpose’. That experimental psychology requires (among other things) the reliable measurement of at least two fixed points to meet the requirements of empiricism. Criticisms of the science of mindfulness include the contention that establishing ‘fixed points’ when dealing with universal human consciousness is problematic. That is not to say that individual studies cannot identify their own fixed points to generate data. But the extent to which different studies use the same, constructs, scales and understandings is highly variable.1

Reviews of the scientific literature have indicated that there are multiple understandings of the mental states and traits described as mindfulness. That congruence between contemporary and traditional forms of mindfulness have not been established at operational or theoretical levels. There are widespread methodological problems in how mindfulness is observed by and integrates with the scientific method. But the uncertainty surrounding mindfulness is not a new issue. The term mindfulness in the context of contemplative science was first translated into English in 1881. Since which time understandings have been continually proposed, developed, corrected and reconsidered4. Today, contemplative science appears no closer to a clear definition of precisely what mindfulness might be or how interventions can meditate it. The pressing questions asked by critical mindfulness are, how can positivism alone make sense of behaviour that defies an authoritative description at the theoretical and operational level? And how can psychology develop a more rigorous approach to testing the findings and claims produced by 7,000 published meditation studies over the last eight decades?

The traditional Buddhist account of mindfulness plays on aspects of remembering, recalling, reminding and presence of mind that can seem underplayed or even lost in the context of MBSR and MBCT.

Rupert Gethin 4

References

1 Van Dam, Nicholas T., Marieke K. van Vugt, David R. Vago, Laura Schmalzl, Clifford D. Saron, Andrew Olendzki, Ted Meissner et al. “Mind the hype: A critical evaluation and prescriptive agenda for research on mindfulness and meditation.” Perspectives on Psychological Science 13, no. 1 (2018): 36-61.

2 Danziger, Kurt. “The positivist repudiation of Wundt.” Journal of the History of the Behavioral Sciences 15, no. 3 (1979): 205-230.

3 Wundt, Wilhelm. “Über Ausfrageexperimente und über die Methoden zur Psychologie des Denkens.” Psychologische Studien 3 (1907): 301-360.

4 Gethin, Rupert. “On some definitions of mindfulness.” Contemporary Buddhism 12, no. 01 (2011): 263-279.

Notes

Image of Wundt – Weltrundschau zu Reclams Universum 1902 [Public domain], via Wikimedia Commons

Meditation, mindfulness, quantum physics and duality

Mindfulness meditation finds itself under sustained scientific criticism, could quantum physics explain why?

Meditation and quantum physics
How far can a reductive mechanistic scientific ontology measure organic processes?

Experimental psychology has been the main object of The Science of Meditation project. The scientific papers featured in this blog are recent, generally published after 2010. But yesterday evening I came across a study from 1975 that addresses many of the issues which are currently limiting the research and practice of contemporary meditation. Fritjof Capra’s paper, Modern Physics and Eastern Mysticism describes the world views of Buddhism, Hinduism and Taoism, comparing them with Western equivalents.1 In doing so, Capra highlights more than a dozen problems manifest in the contemporary (positivist) scientific understanding of Buddhist meditation. One of which I’m going to discuss here briefly; world views as either organic or mechanistic.

person walks towards temple

Having experienced the benefits of meditation first hand, I find the failure of psychology to demonstrate the potential of meditation both wasteful and confusing. As many as ten thousand meditation and mindfulness experiments have been conducted over the last forty years. Yet cognitive psychology describes research in this area as preliminary! Over time two questions have shaped my academic and scientific work; i) how does a spiritual practice become a secular (scientific) practice and ii) what is lost and gained in this transition? Put concisely, how well has the West understood traditional meditation systems?

Strategic reviews of research published since 2016 generally identify two limitations in the science of meditation. An absence of theoretical frameworks and widespread methodological flaws. The lack of a cohesive ontology (framework) is the greater of the two problems. Without a guiding rationale, the scientific method can become directionless, entangling the means with the ends. Capra’s paper sets out his interpretation of the characteristics of ‘Eastern’ spiritual understandings, thus offering signposts to how the West could shape meditation research.2 

high angle photo of robot

So what are these organic and mechanistic world views that could alter the trajectory of research? Capra’s paper is 45 years old; much has changed in physics, psychology and our understandings of meditation.3 But as a theoretical study, Modern Physics and Eastern Mysticism deals with overarching concerns that are timeless. Capra argues that the view of ‘reality’ developed in the West rests on certain principles, such as those set out by the anatomist Democritus. It was the progression of this view that led to the creation of classical physics and established dualism as the ‘Western’ way of understanding almost everything.

Conversely Eastern understandings see nature as much more interconnected, that the categories and laws of nature are constructs, built by mental processes rather than absolute ‘truths’. Capra offers a deal of evidence from quantum physics to demonstrate how this proposition might work with the inanimate. But for the psychological sciences, the value of this insight is self-evident, humans rarely respond to complex phenomena in a universally predictable manner. And where experiments reveal ‘universality’ in complex human behaviours, there are generally several factors influencing the data, including society and the experimental method.

So what does this piece of ‘dated’ quantum physics mean for our contemporary understanding of meditation? The essence of this work highlights fundamental differences between ontologies (theories of being) between the societies where meditation was created and is now investigated. That the West follows a culturally relevant mechanistic presumption of causality, even when considering human nature.4  Not to suggest that Newtonian physics doesn’t ‘work’, rather that it is part of a much more sophisticated understanding of nature. Psychology’s failure to recognise that different ontologies exist in different cultures, even when appropriating their technologies has implications to the study of meditation and mindfulness.5

 

Notes:

1 Capra, F. (1976). Modern physics and eastern mysticism. Journal of Transpersonal Psychology, 8(1).

2 Capra also discusses Hinduism and Taoism in this paper. Although grouping ideas from different Buddhist schools or diverse religio-philosophical systems can lead to over-generalisations, each of the points made needs to considerer on its individual merit.

3 I’m unfamiliar with Capra’s later studies; his views may have changed radically since this paper was published. I’d be delighted to hear from you if you are familiar with his recent work, feel free to email me or post comments in the text box below.

4 Capra’s thinking embraces physics generally, the emphasis on human behaviour here is my focus rather than a reflection of the paper under discussion.

5 While the existing positivist ontologies present in cognitive psychology offer investigatory potential, there are two problems if traditional meditation is based on a Western world view.  Firstly without cognisance of the spiritual frameworks, the contemporary interpretation of the original practices may lack elements foundational to its understanding. Secondly, while positivist approaches will produce data, what is being measured, and how it is understood may be unrelated to the spiritual meditation.

Mindfulness study retracted: Problems with MBSR/MBCT paper

Earlier this year Plos One took the step of retracting a well known and widely cited mindfulness related study.

It's getting harder to make sense of mindfulness research
Is it getting harder to make sense of mindfulness research?

This particular study is a first for the Science of Meditation blog. Whilst we have featured a number of papers that have highlighted methodological problems in meditation and mindfulness research, this is the first time that we have drawn attention to a retraction of a peer-reviewed study.

The basis for the retraction is outlined in detail on the Plos One website, but we have paraphrased the three main points.

  • The handling Academic Editor shared an affiliation with three of the authors, although this didn’t emerge until post-publication.
  • Two of the authors hold or had held positions at an institute offering mindfulness related products and services in clinical contexts.
  • The paper has a number of errors including pooling of results which led to double counting and incorrect effect estimates in figures contained in the study.

There’s not a lot more we need to add to the identified issues, they speak for themselves. However, when considered as part of the ongoing crisis in mindfulness research they make troubling reading.

A general defence used in cognitive psychology when the findings of mindfulness studies are criticised is, the peer review system is self-regulating. That when studies are found to be below the expected standard, they are usually rejected during review. Or at the very least other experts working in the field have the opportunity to raise concerns in print. This retraction challenges this basic notion. Significant issues with both the methodology and the editorial process can endure, thus, have the ability to influence the scientific and popular understanding of mindfulness. According to Google Scholar, this Gotink et al. study has been cited over 400 times, the citing publications, in turn, used by thousands more papers. The details provided on the Plos One website indicate the study has received 50,000 views.

Rather than simply criticize this study or the journal, I would like to ask what this retraction show us about the way that meditation technologies are being treated by clinical and scientific institutions?

Authors: Gotink, R. A., Chu, P., Busschbach, J. J., Benson, H., Fricchione, G. L., & Hunink, M. M.

Year: 2015

Title: Standardised mindfulness-based interventions in healthcare: an overview of systematic reviews and meta-analyses of RCTs

Summary: This is a research paper that (at the time of writing) had been retracted by PLOS ONE.

“In light of the methodological issue and concerns about the validity of the study’s results, the PLOS ONE Editors retract this article. We regret that these issues were not fully addressed prior to the article’s publication.”

An extensive explanation of the reasons behind the retraction are published on the Plos One website which can be reached by following the link below.

Link: https://journals.plos.org

%d bloggers like this: