What does the replication crisis mean for the science of meditation and mindfulness?

The scientific study of meditation has been limited by a replication crisis and a mindfulness crisis. What does this mean and what is the way forward for contemplative science?

Replication, an important element of the scientific method

For at least the last 20 years psychological science has been facing a replication crisis.1 For those who don’t know, the replication crisis reflects a deep-seated problem in how psychology carries out scientific investigations. In essence, it means that many psychological studies from the past may not be as reliable as we thought they were. This uncertainty has implications for the way psychology is conducted, and it may accelerate the declining public confidence in science more generally.

The replication crisis is visible in social sciences and medicine, but not all disciplines have been affected to the same extent. Although social psychology is regarded as having the most significant replication problem, the phenomenon is present in other areas such as the science of meditation. For an experimental study to be scientifically reliable, it generally has to be repeated, repeated by other scientists in alternative locations. If the results are the same, or at least very similar on each of these occasions, the scientific findings are much more likely to be reliable. However, if scientific claims cannot be replicated, it raises questions about how they were initially established, and the extent to which they can be generalised across populations. So if one scientific study found that regular meditation reduced the effects of hay fever, we’d expect to see the same results in other studies carried out in the same way. If not it could mean that there was an unusual characteristic in the first study or some problem in the method. It is for these reasons meditation scientists, teachers and practitioners are reevaluating what they know about the health benefits of meditation.

A failure to replicate doesn’t necessarily prove that scientific findings in the original study were not reliable, but it raises questions over the extent to which the claims are robust. So any isolated evidence for the health and wellbeing benefits of meditation has to be seen as a pilot study, preliminary in nature. In most cases, without replication, we cannot assume that findings from any individual study could apply to the general population.

For those of us working with meditation, the replication crisis is compounded because we are also facing a ‘mindfulness crisis’. The mindfulness crisis describes systemic problems in meditation research that go back 50 years. At least half a dozen studies published since 2015 have identified and described the meditation and mindfulness research crisis. Its main characteristics are conflicting theoretical understandings of meditation and methodological limitations which include low levels of replication. Although many, perhaps most scientific studies of meditation have been impacted by problems linked to the replication and mindfulness crises. The scientific enthusiasm for meditation technologies since the 1970s has been so great that one-off unreplicated claims for the benefits of meditation have not always been critically evaluated by the scientific community. As Van Dam and colleagues have demonstrated, this has led to the ‘hyping’ of preliminary evidence as robust scientific findings.2

Measures are being taken to address the replication crisis within psychology more generally. These initiatives have had a limited effect so far, and their impact will have to be evaluated over the longer term. To overcome the problems being experienced in Contemplative Science, there are three issues that need to be considered by the scientific and practice communities. Firstly the development of a system where unreplicated, preliminary findings are not treated in the same way as robust, replicated work. Secondly, address the pressing need to understand and resolve the known theoretical and methodological limitations. And finally, to review the procession of the scientific understanding of meditation since the 1930s to make sense of the current crisis and diagnose its underlying causes.

Notes

1 Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?. American Psychologist, 70(6), 487.

2 Van Dam, N. T., van Vugt, M. K., Vago, D. R., Schmalzl, L., Saron, C. D., Olendzki, A., … & Meyer, D. E. (2018). Mind the hype: A critical evaluation and prescriptive agenda for research on mindfulness and meditation. Perspectives on psychological science, 13(1), 36-61.

Some critical thoughts on mindfulness?

The scientific study of meditation has produced 7,000 peer reviewed studies, but our understanding is still described as preliminary. Has the time come for a more critical approach to mindfulness?

Reliable measurement is a central tenet of experimental psychology, but deciding what to measure is a much more complex question.
Accurate measurement is crucial to science, but deciding what to measure is a challenging question in mindfulness research

So broad has the field of mindfulness become that we find conflicting, coexisting and complementary perspectives among practices. Traditional mindfulness methods include explicitly and implicitly nondual understandings that appear abstract (incommensurable) to positivist scientific enquiry. However, the exponential growth in the modern forms of mindfulness sits primarily within the positivist ontology of experimental psychology. Positivism creates understandings linked to established tenets. In particular, it (a) assumes psychological phenomena follow deterministic (causal) patterns, (b) that explanations for behaviour can be generalised beyond narrow experimental settings and (c) elaborate explanations are rejected in favour of parsimonious accounts. Also, (d) that reductive investigations can offer understandings of complex human behaviours and cognitive states and (e) experiments produce events which can be reliably measured. In many respects, the positivist approach has successfully contributed a great deal to our understandings of human behaviour. However, its ability to explain and evaluate traditional and medicalised mindfulness and meditation is facing challenges from within the scientific community1.

This image has an empty alt attribute; its file name is Wilhelm_Wundt.jpg
Wilhelm Maximilian Wundt – Founder of experimental psychology

Wilhelm Maximilian Wundt was the architect of experimental psychology. Towards the end of the 19th century, Wundt became one of the first researchers to conceptualise and investigate psychology as a field of science rather than philosophy2. But Wundt was also very clear about the limitations of the experimental approach, that nuanced human behaviours were not accessible to methodologies rooted in positivism3. After all, we humans are replete with agency, we can take for or against an idea with little rational justification. Given the spectrum of human experience, Wundt’s position seems to hold some merit. How can experiments be created that fully explain and generalise highly individualised behaviour? Behaviour created and maintained within abstract inner worlds, which is supported by unique environmental conditions? Consider that meditation is the mediation of consciousness, of which psychology only has a rudimentary understanding. One of the issues that Wundt’s concerns highlight is ‘fitness for purpose’. That experimental psychology requires (among other things) the reliable measurement of at least two fixed points to meet the requirements of empiricism. Criticisms of the science of mindfulness include the contention that establishing ‘fixed points’ when dealing with universal human consciousness is problematic. That is not to say that individual studies cannot identify their own fixed points to generate data. But the extent to which different studies use the same, constructs, scales and understandings is highly variable.1

Reviews of the scientific literature have indicated that there are multiple understandings of the mental states and traits described as mindfulness. That congruence between contemporary and traditional forms of mindfulness have not been established at operational or theoretical levels. There are widespread methodological problems in how mindfulness is observed by and integrates with the scientific method. But the uncertainty surrounding mindfulness is not a new issue. The term mindfulness in the context of contemplative science was first translated into English in 1881. Since which time understandings have been continually proposed, developed, corrected and reconsidered4. Today, contemplative science appears no closer to a clear definition of precisely what mindfulness might be or how interventions can meditate it. The pressing questions asked by critical mindfulness are, how can positivism alone make sense of behaviour that defies an authoritative description at the theoretical and operational level? And how can psychology develop a more rigorous approach to testing the findings and claims produced by 7,000 published meditation studies over the last eight decades?

The traditional Buddhist account of mindfulness plays on aspects of remembering, recalling, reminding and presence of mind that can seem underplayed or even lost in the context of MBSR and MBCT.

Rupert Gethin 4

References

1 Van Dam, Nicholas T., Marieke K. van Vugt, David R. Vago, Laura Schmalzl, Clifford D. Saron, Andrew Olendzki, Ted Meissner et al. “Mind the hype: A critical evaluation and prescriptive agenda for research on mindfulness and meditation.” Perspectives on Psychological Science 13, no. 1 (2018): 36-61.

2 Danziger, Kurt. “The positivist repudiation of Wundt.” Journal of the History of the Behavioral Sciences 15, no. 3 (1979): 205-230.

3 Wundt, Wilhelm. “Über Ausfrageexperimente und über die Methoden zur Psychologie des Denkens.” Psychologische Studien 3 (1907): 301-360.

4 Gethin, Rupert. “On some definitions of mindfulness.” Contemporary Buddhism 12, no. 01 (2011): 263-279.

Notes

Image of Wundt – Weltrundschau zu Reclams Universum 1902 [Public domain], via Wikimedia Commons

A history of meditation; from the temple to the laboratory

The scientific history of meditation reveals that we may be overlooking many important findings from the past.

The history of meditation and mindfulness
The history of meditation and mindfulness

A history of meditation and mindfulness

In the last century, we have seen four or five (it depends on how you categorise the research) waves of engagement between science and spiritual/meditation technologies. Careful attention to the successes and failures of each of these waves gives us important insights into the current crisis in meditation research. Understanding how the curative potential of meditation has been altered by its relocation to science should accelerate the development of more effective interventions. However, there are few reliable accounts of the scientific history of meditation from which to evaluate our progress. This short introduction highlights some of the important and often disregarded progress from past decades.

Buddhist meditation had been migrating for two and a half thousand years. From its conceptual birthplace in Northern India, Buddhism developed many schools and approaches, all loosely tied to foundational theoretical frameworks.[1] Buddhist meditation methods became transnational spiritual practices, frequently adapted to local conditions whenever they established a foothold.[2]

D. T Suzuki (image from Wikipedia)

The relocation of Buddhism from Asia to the West gained momentum in the first half of the twentieth century, bringing Buddhist knowledge and meditation to many Westerners for the first time. But as well as introducing Zen Buddhism to many in the West, academic and Zen teacher Daisetsu Teitaro Suzuki was also redrawing the boundaries between psychology and religion.[3] Suzuki’s ideas were shaped by the time he spent living and working with Paul Carus at the start of the twentieth century. Carus was an advocate of monism, a concept expressed through the Religion of Science (RoS). [4]  The RoS held that a positivist symmetry existed between elements of science and religion. By the middle decades of the twentieth century, Suzuki had become increasingly influential with Western academics and opinion formers such as Alan Wallace and Erich Fromm.

Fromm drew on Suzuki’s reformed, psychologised version of Zen to develop Zen psychotherapy in the late 1950s.[5] Fromm’s interest in Zen was part of a broader movement, and during the 1960s Eastern spiritual traditions became increasingly important in America and Britain. Promoted by poets, philosophers and global celebrities such as The Beatles, meditation in general and Transcendental Meditation, in particular, expressed the aspirations of a generation.[6] The growing Counter Culture became increasingly suspicious of mechanistic approaches to health and wellbeing.[7] Meditation and spiritual world views were seen by many as an antidote to the restlessness caused by an overbearing ‘technocracy’ and processes of dehumanisation.[8]

History of meditation
EEG test (image from Wikipedia)

Alongside the Carus-Zuzuki-Fromm confluence of psychology and traditional spiritual thought, there was a second, more scientific engagement with Eastern meditation traditions. Early accounts of the effects of meditation on the brain can be found in the electroencephalographic (EEG) literature from the 1930s.[9] The following decades were punctuated with several important peer reviewed papers from philosophical, psychoanalytical, physiological and psychological perspectives. A key breakthrough in the scientific understanding of the effects of meditation came with NN Das’s and Henri Gastaut’s research in 1955.[10] By studying brain wave activity alongside other physiological indicators, Das and Gastaut helped to establish the methodology that dominated meditation research for the following two decades.

EEG studies became much more commonplace during the 1960s. Researchers from several countries (France, Germany, India and Japan in particular) confirmed a correlation between the frequency and strength of alpha brain waves and meditative states. But in 1970, a study of Transcendental Meditation (TM) published in Science took the scientific and public interest in meditation to new levels.[11] Keith Wallace’s claims that alongside a range of potential health benefits, TM practitioners could access a novel state of consciousness, inspired meditation researchers for a decade. The 1970s saw a rapid growth in TM studies in experimental and applied settings. In the early 1970s, Wallace joined forces with the cardiologist and health researcher Herbert Benson. Using novel methodological approaches, Benson and Wallace provided further evidence that the practice of TM could lead to improved health and wellbeing.[12]

By 1974 Benson’s research had changed direction, he went on to describe the relaxation response (RR), a ‘grand theory’ that made sweeping claims for a relaxed cognitive state.[13] Benson aggregated operational elements from many different spiritual practices into four essential components that could deliver the relaxation response. Despite a successful book about the RR (written by Benson and Miriam Klipper), the science underpinning the hypothesis never received universal scientific acceptance.[14] By 1980 the scientific community had begun to evaluate‘progress’ made in the study of meditation. Deep-seated methodological and theoretical issues with TM research led to critical reviews from scientists such as Michael West.[15]

However, setbacks to the reputation of meditation research did not thoroughly dampen enthusiasm for the technology. Several new approaches were delivering positive findings, such as Vikram Patel’s combination of biofeedback and meditation to reduce stress.[16] At the start of 1980, there was a growing acceptance of the curative potential of Eastern non-positivist interventions, such as meditation (as well as yoga and acupuncture).[17] But a critical view of meditation research had raised questions about how psychology might best harness the curative potential of meditation.

When viewed from a history of science perspective, many of the strengths and weaknesses in contemporary meditation research are visible in previous ‘waves’. I acknowledge the Western-centric nature of this account. Some potentially important research originating outside of America and Britain, remains to be fully reviewed. But a clear conclusion from this brief summary is that if we don’t pay attention to reliable science from the past, we risk repeating the same mistakes and also missing important findings from earlier work.

(Based on a history of meditation paper prepared for the ERC BodyCapital conference October 2020).

Biblography


[1] Ronkin, Noa, Early Buddhist Metaphysics: The Making of a Philosophical Tradition (London: Routledge, 2005). https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.695.8390&rep=rep1&type=pdf

[2] Stephen C. Berkwitz, Buddhism in World Cultures (ABC-CLIO, 2006).

[3] Robert Sharf, ‘The Zen of Japanese Nationalism’, History of Religions, 33.1 (1993), 1–43. https://www.journals.uchicago.edu/doi/abs/10.1086/463354

[4] Carl T. Jackson, ‘The Meeting of East and West: The Case of Paul Carus’, Journal of the History of Ideas, 29.1 (1968), 73. https://www.jstor.org/stable/2708466?seq=1

[5] Erich Fromm, Psychoanalysis and Zen Buddhism (New York: Open Road Media, 2013). https://philpapers.org/rec/FROPAZ

[6] Anne Harrington and John D. Dunne, ‘When Mindfulness Is Therapy’, American Psychologist, 70.7 (2015), 621–31. https://dash.harvard.edu/bitstream/handle/1/25757884/97605608.pdf?sequence=1

[7] Saks, Mike, ‘Medicine and the Counter Culture’, in Companion to Medicine in the Twentieth Century, ed. by Roger Cooter and John Pickstone (Abingdon: Routledge, 2003), pp. 113–24

[8] Theodore Roszak, The Making of a Counter Culture: Reflections on the Technocratic Society and Its Youthful Opposition. (Oakland: University of California Press, 1995).

[9] Walter, W. G., ‘Critical Review: The Technique and Application of Electro-Encephalaography’, Journal of Neurology, Neurosurgery & Psychiatry, 1.4 (1938), 359–85. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1088109/pdf/jnpsychiatry00020-0059.pdf

[10] N Das and H Gastaut, ‘Variations in the Electrical Activity of the Brain, Heart, and Skeletal Muscles during Yogic Meditation and Trance’, Electroencephalography and Clinical Neurophysiology, 6 (1955), 211-219.

[11] Robert Keith Wallace, ‘Physiological Effects of Transcendental Meditation’, Science, 167.3926 (1970), 1751–54. https://science.sciencemag.org/content/167/3926/1751.abstract

[12] Robert Keith Wallace, Herbert Benson, and Archie Wilson, ‘A Wakeful Hypometabolic Physiologic State.’, The American Journal of Physiology, 221.3 (1971), 795–99. https://journals.physiology.org/doi/abs/10.1152/ajplegacy.1971.221.3.795?journalCode=ajplegacy

[13] Herbert Benson, John F. Beary, and Mark P. Carol, ‘The Relaxation Response’, Psychiatry, 37.1 (1974), 37–46. https://search.proquest.com/docview/1301437094

[14] Herbert Benson and Miriam Klipper, The Relaxation Response (New York: Collins, 1976).

[15] Michael West, ‘Meditation.’, The British Journal of Psychiatry : The Journal of Mental Science (Cambridge University Press, 1979), 457–67. https://www.cambridge.org/core/journals/the-british-journal-of-psychiatry/article/abs/meditation/BCF552D109C0184ADCF979EBAE736915

[16] C. H. Patel, ‘Yoga and Bio-Feedback in the Management of Hypertension’, The Lancet, 302.7837 (1973), 1053–55. https://www.sciencedirect.com/science/article/abs/pii/S0140673673926603

Meditation, mindfulness, quantum physics and duality

Mindfulness meditation finds itself under sustained scientific criticism, could quantum physics explain why?

Meditation and quantum physics
How far can a reductive mechanistic scientific ontology measure organic processes?

Experimental psychology has been the main object of The Science of Meditation project. The scientific papers featured in this blog are recent, generally published after 2010. But yesterday evening I came across a study from 1975 that addresses many of the issues which are currently limiting the research and practice of contemporary meditation. Fritjof Capra’s paper, Modern Physics and Eastern Mysticism describes the world views of Buddhism, Hinduism and Taoism, comparing them with Western equivalents.1 In doing so, Capra highlights more than a dozen problems manifest in the contemporary (positivist) scientific understanding of Buddhist meditation. One of which I’m going to discuss here briefly; world views as either organic or mechanistic.

person walks towards temple

Having experienced the benefits of meditation first hand, I find the failure of psychology to demonstrate the potential of meditation both wasteful and confusing. As many as ten thousand meditation and mindfulness experiments have been conducted over the last forty years. Yet cognitive psychology describes research in this area as preliminary! Over time two questions have shaped my academic and scientific work; i) how does a spiritual practice become a secular (scientific) practice and ii) what is lost and gained in this transition? Put concisely, how well has the West understood traditional meditation systems?

Strategic reviews of research published since 2016 generally identify two limitations in the science of meditation. An absence of theoretical frameworks and widespread methodological flaws. The lack of a cohesive ontology (framework) is the greater of the two problems. Without a guiding rationale, the scientific method can become directionless, entangling the means with the ends. Capra’s paper sets out his interpretation of the characteristics of ‘Eastern’ spiritual understandings, thus offering signposts to how the West could shape meditation research.

high angle photo of robot

So what are these organic and mechanistic world views that could alter the trajectory of research? Capra’s paper is 45 years old; much has changed in physics, psychology and our understandings of meditation.3 But as a theoretical study, Modern Physics and Eastern Mysticism deals with overarching concerns that are timeless. Capra argues that the view of ‘reality’ developed in the West rests on certain principles, such as those set out by the anatomist Democritus. It was the progression of this view that led to the creation of classical physics and established dualism as the ‘Western’ way of understanding almost everything.

Conversely Eastern understandings see nature as much more interconnected, that the categories and laws of nature are constructs, built by mental processes rather than absolute ‘truths’. Capra offers a deal of evidence from quantum physics to demonstrate how this proposition might work with the inanimate. But for the psychological sciences, the value of this insight is self-evident, humans rarely respond to complex phenomena in a universally predictable manner. And where experiments reveal ‘universality’ in complex human behaviours, there are generally several factors influencing the data, including society and the experimental method.

So what does this piece of ‘dated’ quantum physics mean for our contemporary understanding of meditation? The essence of this work highlights fundamental differences between ontologies (theories of being) between the societies where meditation was created and is now investigated. That the West follows a culturally relevant mechanistic presumption of causality, even when considering human nature.4  Not to suggest that Newtonian physics doesn’t ‘work’, rather that it is part of a much more sophisticated understanding of nature. Psychology’s failure to recognise that different ontologies exist in different cultures, even when appropriating their technologies has implications to the study of meditation and mindfulness.5

 

Notes:

1 Capra, F. (1976). Modern physics and eastern mysticism. Journal of Transpersonal Psychology, 8(1).

2 Capra also discusses Hinduism and Taoism in this paper. Although grouping ideas from different Buddhist schools or diverse religio-philosophical systems can lead to over-generalisations, each of the points made needs to considerer on its individual merit.

3 I’m unfamiliar with Capra’s later studies; his views may have changed radically since this paper was published. I’d be delighted to hear from you if you are familiar with his recent work, feel free to email me or post comments in the text box below.

4 Capra’s thinking embraces physics generally, the emphasis on human behaviour here is my focus rather than a reflection of the paper under discussion.

5 While the existing positivist ontologies present in cognitive psychology offer investigatory potential, there are two problems if traditional meditation is based on a Western world view.  Firstly without cognisance of the spiritual frameworks, the contemporary interpretation of the original practices may lack elements foundational to its understanding. Secondly, while positivist approaches will produce data, what is being measured, and how it is understood may be unrelated to the spiritual meditation.

Mindfulness study retracted: Problems with MBSR/MBCT paper

Earlier this year Plos One took the step of retracting a well known and widely cited mindfulness related study.

It's getting harder to make sense of mindfulness research
Is it getting harder to make sense of mindfulness research?

This particular study is a first for the Science of Meditation blog. Whilst we have featured a number of papers that have highlighted methodological problems in meditation and mindfulness research, this is the first time that we have drawn attention to a retraction of a peer-reviewed study.

The basis for the retraction is outlined in detail on the Plos One website, but we have paraphrased the three main points.

  • The handling Academic Editor shared an affiliation with three of the authors, although this didn’t emerge until post-publication.
  • Two of the authors hold or had held positions at an institute offering mindfulness related products and services in clinical contexts.
  • The paper has a number of errors including pooling of results which led to double counting and incorrect effect estimates in figures contained in the study.

There’s not a lot more we need to add to the identified issues, they speak for themselves. However, when considered as part of the ongoing crisis in mindfulness research they make troubling reading.

A general defence used in cognitive psychology when the findings of mindfulness studies are criticised is, the peer review system is self-regulating. That when studies are found to be below the expected standard, they are usually rejected during review. Or at the very least other experts working in the field have the opportunity to raise concerns in print. This retraction challenges this basic notion. Significant issues with both the methodology and the editorial process can endure, thus, have the ability to influence the scientific and popular understanding of mindfulness. According to Google Scholar, this Gotink et al. study has been cited over 400 times, the citing publications, in turn, used by thousands more papers. The details provided on the Plos One website indicate the study has received 50,000 views.

Rather than simply criticize this study or the journal, I would like to ask what this retraction show us about the way that meditation technologies are being treated by clinical and scientific institutions?

Authors: Gotink, R. A., Chu, P., Busschbach, J. J., Benson, H., Fricchione, G. L., & Hunink, M. M.

Year: 2015

Title: Standardised mindfulness-based interventions in healthcare: an overview of systematic reviews and meta-analyses of RCTs

Summary: This is a research paper that (at the time of writing) had been retracted by PLOS ONE.

“In light of the methodological issue and concerns about the validity of the study’s results, the PLOS ONE Editors retract this article. We regret that these issues were not fully addressed prior to the article’s publication.”

An extensive explanation of the reasons behind the retraction are published on the Plos One website which can be reached by following the link below.

Link: https://journals.plos.org