The safety and efficacy of acceptance and commitment therapy against psychotic symptomatology: a systematic review and meta-analysis

Abstract

Objective:

Acceptance and commitment therapy (ACT) is a third-wave psychological intervention that has attracted considerable clinical and research attention. A previous meta-analysis of ACT trials in psychosis reported a large effect size of ACT against overall psychotic symptomatology. However, there were critical methodological issues in the review that justify replication.

Methods:

Systematic review and meta-analysis of randomized controlled trials (RCTs) testing ACT vs. any comparator condition in a sample of adults with psychosis. The outcome of interest was overall psychotic symptomatology.

Results:

The search identified seven published and eight unpublished trials (of which we were able to obtain data from one). Data on symptomatology were extracted from six trials that involved 274 participants. The summary effect size (Hedge’s G) for overall symptomatology was small and not significant (-0.21, 95%CI -0.60-0.18). Trials were generally rated as having a high risk of bias. Safety reporting was inadequate across included trials.

Conclusions:

Our observed effect size contrasted with that reported in a previous meta-analysis; differences were likely explained by errors in data extraction. The findings of this review suggest that there is currently inadequate evidence to conclude that ACT is a safe and effective treatment against psychotic symptomatology.

Systematic review registration:

CRD42018097200

acceptance and commitment therapy; ACT; systematic review; meta-analysis; randomized controlled trial; RCT

Introduction

There has been considerable interest in acceptance and commitment therapy (ACT) in the treatment of various psychiatric disorders, including psychosis.¹ Belonging to the third wave of cognitive behavioral therapy (CBT),² ACT was first defined and described by Hayes et al.,² who set out six basic principles – cognitive diffusion, expansion and acceptance, contact and connection with the present moment, the observing self, values clarification, and committed action – that work in conjunction with one another towards the overarching goal of effectively handling painful thoughts and experiences and creating a vibrant, vital life.³ ACT has been manualized as a specific treatment for psychosis.⁴ However, whether it is safe and effective for this purpose remains unclear. To date, there have been two systematic reviews⁵,⁶ and two systematic reviews and meta-analysis⁷,⁸ of ACT for psychosis.

Ost⁵ reported a review and meta-analysis of 60 ACT trials, involving 4,234 participants across a range of mental and physical disorders (as well as stress at work). Of note, four trials testing ACT against psychotic symptoms were included.⁹-¹² A small effect size (0.42) across all conditions was reported. However, Ost⁵ did not report a meta-analysis of ACT against psychotic symptoms, and notes that only one trial⁹ reported a significant difference in favor of ACT. Methodological issues with the included trials were identified, leading the authors to conclude that ACT was possibly effective for psychotic symptoms. This review has been heavily criticized¹³ for potential bias and lack of methodological rigor; for example, screening, data extraction, and risk of bias analysis (using a measure developed by the review author) was undertaken by only one researcher.⁵

Wakefield et al.⁶ reported a systematic review – but not a meta-analysis – of ACT for psychosis. The authors included 13 controlled and uncontrolled trials and concluded that ACT was beneficial for patients with first-episode or chronic psychosis. However, the authors included papers that reported reanalyses of combined datasets. For example, Bach et al.¹⁴ examined the mediating role of symptom believability on hospitalization outcomes by combining datasets from two previous trials.⁹,¹⁰ As a consequence, Wakefield et al.⁶ conflated the number of ACT studies because the same datasets were included more than once.

Two meta-analyses of the safety and efficacy of acceptance and mindfulness-based interventions for psychosis have been reported.⁷,¹⁵ Neither review was pre-registered. Authors of both meta-analyses included trials that reported against several different outcomes, including overall symptomatology, positive/negative symptoms, hospitalization/days in hospital, depressive symptoms, social functioning, mindfulness skills, and acceptance. As a consequence, there is a risk of selective outcome reporting bias, with preference given to outcomes with significant associations. Four ACT trials⁹-¹² were included in the original review⁷ and two¹⁰,¹¹ in their meta-analysis. A small and non-significant effect size (-0.16) was reported against psychotic symptomatology. Low risk of bias was reported for the included studies. Jansen et al.¹⁵ identified two additional trials (for a total of six) that evaluated ACT in people with psychosis. Their meta-analysis of ACT against overall symptomatology reported a large effect size (-1.07).¹⁵ The effect size for one of the included trials in particular¹⁶ is considerable (-4.45) and may be incorrect (we wrote to the editor of Schizophrenia Research on March 9, 2020, informing them of this potential error). Seemingly, the authors extracted the standard error rather than the standard deviation (SD), and used this in their meta-analysis. The authors conclude that ACT is a safe intervention and state that no study reported any serious adverse events that were directly related to the intervention. Safety data were only reported in two¹⁰,¹² of seven included ACT trials. Using the Cochrane risk of bias tool,¹⁷ the authors again concluded that the studies had a low risk of bias.¹⁵

To date, there have been four previous systematic reviews of ACT against psychotic symptomatology that generally affirm the use of ACT in this population. As we have discussed, there are potential errors in two⁶,¹⁵ and a lack of methodological rigor in one.⁵ There is a pressing need for a rigorous systematic review and meta-analysis testing the safety and efficacy of ACT against psychosis symptoms.

Methods

Review design

This was a systematic review and meta-analysis of randomized controlled trials (RCT) of ACT in the treatment of patients diagnosed with psychosis. The main outcome was psychotic symptomatology at the end of treatment; we note that while this outcome may be dissonant with the intended aims of ACT, it is the primary measure used in previous reviews.⁵ We focused on symptoms at the end of treatment, as this was more likely to have been reported consistently in included studies. It was anticipated that there might be considerable variation in timings of follow-up measures, making these difficult to include in a meta-analysis. We have published a protocol¹⁸ for this review, which includes consideration of the issues related to outcome selection and a detailed description of our methodology. The review was registered on PROSPERO on May 21, 2018 (registration number: CRD42018097200; web address:

https://www.crd.york.ac.uk/prospero/display_record.php? RecordID=97200). This report complies with Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.¹⁹

Inclusion and exclusion criteria

Trials were included that:

• - Compared full-length, therapist-delivered ACT, regardless of medium (e.g., individual, group, telephone, online), with any comparator condition, using a randomized controlled design;

• - Included adult participants (age 18 years or older) with a diagnosis of schizophrenia or other psychotic disorder determined using any standard diagnostic criteria;

• - Reported psychotic symptomatology as an outcome;

• - Were written in English and published after 1999 (the year ACT was first described).

• - Trials were excluded if:

• - Participants had developmental impairment, intellectual disability, affective psychosis, organic psychosis, primary diagnosis of major depression with psychotic symptoms, primary drug or alcohol addiction, or any clinically significant medical diseases;

• - Authors reported a follow-up or extension of a previously published trial (to avoid the risk of counting the same participants twice).

Data sources and search strategy

The following databases were searched for eligible studies: Cochrane Central Register of Controlled Trials (CENTRAL), Education Resources Information Center (ERIC), Excerpta Medica Database (Embase), MEDLINE via Ovid, and PsycINFO via Ovid. The following search strategy was employed (keywords are italicized; subject headings are emboldened; Boolean operators are capitalized): ‘acceptance commitment therapy’ OR act OR ‘clinic* behavi?r analy*', psychos?s OR ‘psychotic disorder*' OR schizophrenia, exp Cognitive Therapy/, exp Psychotic Disorders/ OR exp Schizophrenia/, 1 OR 3, 2 OR 4, 5 AND 6, and limit 7 to RCT.

Schizophrenia Research on subject headings in a highly sensitive syntax. The search strategy was initially executed on May 20, 2018 and updated on February 23, 2020.

As possible publication bias has been noted in some reviews of ACT research,⁵ we sought to identify unpublished trials. This may provide critical insight into the current state of the evidence, as publication bias typically exaggerates the magnitude of the treatment effect.²⁰ On May 21, 2018, we searched the following trial registries with psychosis and schizophrenia as condition and acceptance and commitment therapy and ACT as intervention: ClinicalTrials.gov, the Australian New Zealand Clinical Trials Registry (ANZCTR), and the International Standard Randomized Controlled Trial Number (ISRCTN) Registry.

Review management software

We used the online Covidence software package to manage title and abstract and full-text screening.

Screening

Titles and abstracts of studies retrieved using the search strategy were screened independently by two researchers to identify trials that met the inclusion/exclusion criteria. A third researcher resolved any disagreements. Full texts of these potentially eligible studies were retrieved and independently assessed for inclusion by two researchers, with any further disagreements resolved by the third researcher.

Data extraction

We extracted the following data from included studies: study citation, country, setting, dose of intervention, delivery medium of intervention, primary endpoint, number of participants allocated to intervention and control groups, number of analyzed participants allocated to intervention and control groups, calculation of sample size, pre- and post-intervention/control summary statistics, harms and registration status (an International Committee of Medical Journal Editors [ICMJE] requirement for trials that started recruiting after July 2005²¹), and country of origin. Extracted data were entered directly into a spreadsheet. We calculated effect sizes using any reported, post-treatment standard measure of psychotic pathology that included, but was not limited to, the Positive and Negative Syndrome Scale (PANSS)²² and the Brief Psychiatric Rating Scale (BPRS),²³ even where this was not the stated primary outcome. Data on hospitalization as a surrogate marker of psychotic symptoms were also extracted.

Risk of bias

Study quality was determined using version 2 of the Cochrane risk of bias tool (RoB 2).²⁴ The updated tool was produced because of developments in the understanding of bias in RCTs and feedback from researchers that used version 1 of the measure. Bias was rated high, low, or some concerns, in five domains: 1) Risk of bias arising from the randomization process; 2) Risk of bias due to deviations from the intended interventions (effect of assignment to intervention and effect of adhering to intervention); 3) Missing outcome data; 4) Risk of bias in the measurement of the outcome; 5) Risk of bias in the selection of the reported result. The overall default rating of risk of bias is high risk if one of the domains is rated high. Ratings were completed by two researchers, with discrepancies resolved by a third.

Data synthesis

We used Review Manager version 5 (http://community.cochrane.org/help/tools-and-software/revman-5/revman-5-download) to compute the meta-analyses. The most common summary statistic across studies was the standardized mean difference. These were calculated to determine the effect of ACT against comparator treatments. Because four of the included studies had small group sizes (< 20), Hedge’s G was considered a more appropriate metric of effect size.²⁵ Where applicable, an intent-to-treat analysis was used.

Heterogeneity across the studies was assessed using the Cochrane chi-square (χ²) and quantified with theI ²statistic.²⁶ I ²is the proportion of total variation provided by between-study variation, andI ²values of 25, 50 and 75% were considered to represent low, moderate, and high heterogeneity, respectively.²⁷ We checked for potential publication bias by visual inspection of a funnel plot.

Results

Search results

Figure 1 shows the flow of papers through the review. The initial search identified 1,305 articles; this was reduced to 1,152 following removal of duplicates. Title and abstract and full-text screening resulted in seven trials that met our inclusion criteria.

Figure 1
Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram.

Unpublished trials

The search of trial registries identified a further eight studies, involving at least 126 participants, which met the inclusion criteria. Table 1 summarizes the results of this search. Principal investigators were contacted on May 22, 2018, with a request to release data for our meta-analysis. Two authors representing three trials responded. The corresponding author for NCT02336581 stated that analysis was incomplete, but that the aim of the trial was primarily feasibility and acceptability. The corresponding author for ISRCTN 68540929 and ISRCTN 73327972 – trials that ended in December 2013 and August 2018 respectively – stated that the authors were not in a position to share data. As of February 25, 2020, data from these trials had not yet been published.

Table 1
Summary of unpublished trials

Principal investigator(s)	Study title	Registry	Registration status	Registration number	Start date	End date	Recruitment status	Study population	Number enrolled	Arms	Primary outcome	Availability of data
Gaudiano	Researching the effectiveness of acceptance-based coping during hospitalization (REACH)	ClinicalTrials.gov	Prospective	NCT02336581	March 2015	September 2017	Completed	Psychosis	62	ACT/enhanced treatment as usual	Symptoms determined using the BPRS	May 22, 2018: response received, data not available. Rechecked Feb 26, 2020: data not published.
Lundgren	Acceptance and commitment therapy for psychotic inpatients	ClinicalTrials.gov	Retrospective	NCT02458105	April 2015	December 2017	Withdrawn	Psychosis	Not reported	ACT/attentional control	Rehospitalization	May 22, 2018: no response received (as of Feb 26, 2020). Rechecked Feb 26, 2020: data not published.
Fatouros-Bergman	Cognitive behavior therapy for patients with psychoses	ClinicalTrials.gov	Retrospective	NCT03501888	October 2015	August 2016	Completed	Psychosis	16	ACT/cognitive behavior therapy	Symptoms determined using the BPRS	May 22, 2018: no response received (as of Feb 26, 2020). Rechecked Feb 26, 2020: data not published.
Landa	Acceptance and commitment therapy for delusions	ClinicalTrials.gov	Prospective	NCT00657631	April 2008	June 2009	Completed	Schizophrenia/ delusional disorder	4	Unable to determine from the information provided in the registry	Distress (measure not stated)	May 22, 2018: no response received (as of Feb 26, 2020). Rechecked Feb 26, 2020: data not published.
Ftanou	Treating acute psychotic symptoms in an inpatient service using acceptance and commitment therapy, measuring acceptance-based attitudes and actions, health outcomes and level of disability	Australian and New Zealand Clinical Trials Registry	Retrospective	ACTRN12609000603213	July 2009	Withdrawn	Withdrawn	Acute psychotic symptoms	0	ACT/treatment as usual	Voices actions and acceptance scale	May 22, 2018: no response received (as of Feb 26, 2020). Rechecked Feb 26, 2020: data not published.
Johns & Jolley	Acceptance and commitment therapy for recovery	ISRCTN	Retrospective	ISRCTN68540929	February 2013	December 2013	Completed	Any service user or caregiver within the Promoting Recovery service	Not reported	ACT/waitlist control	WEMWBS, CORE-10 (psychological distress)	May 22, 2018: Response received. Data not available ahead of publication. Results not published as of Feb 26, 2020.
O’Donoghue, & Jolley	Balancing ACT: evaluating the effectiveness of psychoeducation and acceptance and commitment therapy groups for people with bipolar disorder	ISRCTN	Prospective	ISRCTN73327972	May 2017	August 2018	Completed	Psychosis with bipolar disorder/symptoms	36	ACT/waitlist control	Quality of life determined using the Brief Quality of Life in Bipolar Disorder scale	May 22, 2018: Response received. Data not available ahead of publication. Results not published as of Feb 26, 2020.
Boden	Acceptance and commitment therapy for the inpatient treatment of psychosis	ClinicalTrials.gov	Prospective	NCT01981356	September 2014	April 2015	Completed	Psychotic disorders	18	ACT/treatment as usual	Symptoms determined using the BPRS	Data were reported in registry entry. Results extracted and included in the review.

• ACT = acceptance and commitment therapy; BPRS = Brief Psychiatric Rating Scale; ISRCTN = International Standard Randomized Controlled Trial Number; WEMWBS = Warwick-Edinburgh Mental Well-being Scale.

Usable data were available from one unpublished trial.²⁸ The authors published a summary of the feasibility of doing an ACT trial based on their experiences from the study, but did report results in the manuscript.²⁹

The final number of trials included in this review and meta-analysis was therefore eight.

Characteristics on included trials

Table 2 shows the characteristics of the eight included trials. Trials were published between 2002 and 2018. Three were conducted in the United States, two in Australia, and one each in Sweden, Canada, and the United Kingdom.

Table 2
Characteristics of trials included in the meta-analyses

Citation	Country	Registration status	Population (mean age [years] male/ schizophrenia [%])	Setting	ACT	Control condition	Number of participants allocated (ACT, control)	Number of participants included in analysis (ACT, control)	Sample size calculation	Symptom measure	Baseline symptoms measure mean (SD) ACT, control	End point symptoms measures mean (SD) ACT, control	Reported harms ACT, control
Bach & Hayes9	United States	Not registered	39, 63/54	Psychiatric inpatients	Individual, four sessions, lasting 45-50 min	Treatment as usual	40, 40	35, 35	No	Hospitalization	Not applicable	4 months following discharge from inpatient care, 7, 14 participants rehospitalized	Psychiatric hospitalization reported as an outcome
Gaudiano & Herbert10	United States	Not registered	Not possible to extract data from the manuscript. Data not reported.	Psychiatric inpatients	Individual, one to five sessions (average of three), lasting 60 min	Enhanced treatment as usual	19, 21	19, 21	No	BPRS: total scoreHospitalization	58.4 (8.6),59.0 (8.0)	Prior to discharge from hospital41.6 (7.8), 45.6 (8.5) Nine control and five ACT patients hospitalized	Psychiatric hospitalization reported as an outcome. Authors report “no adverse events were observed from participation in the study.”
White12	United Kingdom	The trial was retrospectively registered (NCT01003132), but the registration number was not reported in the manuscript.	34, 78/48	Community-dwelling	Individual, 10 sessions	Waitlist	14, 13	14, 10	No	Positive symptoms determined using the positive symptoms subscale of PANSS	11.36 (2.62), 12.75 (4.16)	3 months post baseline9.75 (3.60), 11.70 (4.72)	Participant in the trial did have contact with crisis contacts that would constitute an adverse event, but numbers not reported. Authors stated that “There were no suspected unexpected serious adverse reactions over the course of the trial.”
Shawyer11	Australia	Retrospectively registered (after the first participant had been enrolled). Reason for retrospective registration not given.	40, 56/74	Community-dwelling	Individual, 12 sessions, weekly (plus three assessment and engagement sessions)	Befriending (n=17 participants were initially randomized to a waiting list and then to ACT or befriending)	21, 22	The authors reported that between 18-20 participants were followed up in each group, depending on the measure	Yes	PANSS: total score	67.9 (13.6), 58.1 (8.4)	Completion of therapy60.1 (15.9), 57.5 (10.4)	Harms not reported
Boden29*	United States	Prospectively registered	Not possible to extract data from the manuscript. Data not reported.	Inpatients	Individual, four sessions	The control condition is treatment as usual in registry. In the manuscript, the authors describe attentional control (not described in detail).	12, 6	5, 4	No	Symptoms determined using the BPRS	Data are not reported in the registry	Discharge from hospital-8.0 (13.8),-11.6 (5.8)	No harms reported
Shawyer30	Australia	Prospectively registered	34, 61/75	Community-dwelling	Individual, eight sessions, 50 min, weekly to fortnightly, over 3 months, delivered by a clinical psychologist	8, 50 min sessions of befriending delivered by a clinical psychologist	49, 47	49, 47	Yes	PANSS: total score	79.2 (14.1),78.6 (14.9)	Post-therapy74.6 (16.6), 72.6 (14.7)	The authors state similar rates of hospital admissions in both groups (serious adverse event), but do not report numbers. No other harms reported.
Spidel16	Canada	Not registered	Not possible to extract data from the manuscript. Data not reported. Authors state that participants had psychosis and childhood trauma.	Not specified	Group, eight sessions (up to eight participants), lasting 90 min	Waitlist (treatment as usual)	30, 20	30, 15	No	BPRS (expanded) total score	50.2 (1.6),51.0 (1.9)†	End of treatment 43.1 (1.6), 51.1 (1.9)	Not reported
Tyrberg31	Sweden	Not registered	41, 62/29	Inpatients	Individual, one to four sessions (average of two), lasting 45 min	Treatment as usual	12, 10	11, 10	No	Rehospitalization	Not applicable	4-month follow-up from trial entry 1, 4	Not reported

• ACT = ACT = acceptance and commitment therapy; BPRS = Brief Psychiatric Rating Scale; PANSS = Positive and Negative Syndrome Scale; SD = standard deviation.
• ^*
  Data were published on ClinicalTrials.gov. The registry entry was last updated on April 14, 2016 (checked on July 30, 2019). A short feasibility report²⁹ summarizes the feasibility of ACT in an inpatient setting, but does not report raw data.
• ^†
  Standard error (SE) reported in the manuscript. SD was calculated using the formula SD = SE.√n.

Disorders of interest and measurement

As per our inclusion criteria, trials focused on patients experiencing psychosis. Of these, Spidel et al.¹⁶ also required participants to have experienced childhood trauma. The demographics and clinical profile of participants were not reported across all trials; notably, we were not able to extract necessary information from three studies.¹⁰,¹⁶,²⁹ Where demographic and clinical data were reported, participants were typically in their 30s, there was a slight preponderance of males. Psychotic symptoms were reported as an outcome in six published trials: three used the PANSS and three the BPRS. The authors of three trials reported hospitalization as an outcome. Primary outcomes were specified in two of seven published trials¹¹,³⁰; the remaining studies merely listed outcomes used in the trial.

Trial arms and participants

Four of the trials were conducted in inpatient settings and three in the community. Overall, there were 197 and 179 participants in the ACT and comparator groups, respectively. The smallest trial involved 18 participants,²⁹ and the largest, 96.³⁰

In the Gaudiano & Herbert¹⁰ trial, we identified an error in the results table (which was confirmed by the corresponding author): the number of participants in the experimental and control groups is reversed. The authors published a corrigendum confirming this.³² We have reported the correct numbers in our data extraction (i.e., control = 21, experimental = 19), as shown in Table 1.

ACT

Seven trials⁹-¹²,²⁹-³¹ tested individual ACT, and one tested group ACT.¹⁶ Individual treatment comprised one³¹ to 15¹¹ sessions, typically lasting between 45 minutes and 1 hour. Group ACT¹⁶ comprised eight sessions lasting 90 minutes, with eight participants in each session. The ACT approach used varied between studies. Four authors based their intervention on the original² Hayes manual,⁹,¹⁰,²⁹,³¹ while in the other four studies,¹¹,¹²,¹⁶,³⁰ ACT manuals were developed or adapted especially for the trial. The Spidel¹⁶ trial was the only study that explicitly stated metaphors were not used because of the cognitive problems associated with psychosis. In other studies (e.g., Gaudiano & Herbert¹⁰ und Tyrberg et al.³¹), metaphors were applied as part of the ACT approach. In three studies,¹⁰,³⁰,³¹ participants were given homework tasks to complete between sessions. The psychology background of clinicians providing ACT was stated in two studies.¹¹,³⁰ That ACT was provided by a psychologist could be worked out or inferred (e.g., the affiliation of the author that provided the ACT) in other studies, but was not explicitly stated. In all but two trials,¹²,³¹ the clinician providing treatment received additional training and supervision to deliver ACT.

Comparator interventions

The comparator intervention was treatment as usual or waiting list control in four⁹,¹⁰,¹²,¹⁶ of the trials. Befriending was used as a comparator intervention in two trials.¹¹,³⁰ Boden states in their registry entry that there was an attentional control component in their trial, whereby ACT facilitators met with participants for 15 minutes every other day to provide additional support and answer questions, ensuring not to discuss ACT techniques; however, this is not reported in the published feasibility report.²⁹

Sample size calculations

Two trials reported sample size calculations.¹¹,³⁰ In both cases, researchers failed to recruit the required number of participants.

Numbers completing treatment and included in the analysis

A CONSORT diagram³³ of participant flow through the trial was provided in four studies.¹⁰-¹²,³⁰ Of 328 participants invited to take part, 215 (66%) consented, suggesting that around two patients needed to be asked to get one to consent to participate in the trial. In the ACT and comparison groups, respectively, 183 (93%) and 162 (91%) participants were included in the analysis. Data on the feasibility of delivering ACT to patients with psychosis (e.g., number of sessions completed, duration of sessions) was inconsistently reported. Shawyer et al.³⁰ reported that participants completed seven of eight sessions in both the ACT and befriending groups.

Harms reporting

Adverse and serious adverse events that occurred during the trial were not reported in adequate detail in any of the included trials. Shawyer et al.³⁰ reported admission to hospital as a serious adverse event. Gaudiano & Herbert¹⁰ stated that no harms occurred. White et al.¹² stated that there were no serious adverse reactions over the course of the trial.

Meta-analyses

Figure 2 is a forest plot showing the effect sizes for each study individually and pooled as an aggregate for psychotic symptoms at the end of ACT treatment. An effect size below 0 (left of the line of no effect) favors ACT; one above 0 (right of the line of no effect) favors the comparator interventions(s). The pooled effect size for the six included trials was -0.21 (95% confidence interval [95%CI] -0.60 to 0.18), suggesting a small effect size in favor of ACT. Moderate statistical heterogeneity (I ² > 50%) was observed.

Figure 2
Meta-analysis of studies reporting psychotic symptoms as outcome. 95%CI = 95% confidence interval; df = degrees of freedom; IV = inverse variance; SD = standard deviation.

Three trials⁹,¹⁰,³¹ reported rehospitalization as a surrogate measure of psychotic relapse. A relative risk (RR) less than 1 favors ACT; above 1, the comparator condition. The RR was 0.37 (95%CI 0.17 to 0.80), suggesting that ACT is associated with a significant reduction in the risk of admission (Figure 3).

Figure 3
Meta-analysis of studies reporting hospitalization. 95%CI = 95% confidence interval; df = degrees of freedom; M-H = Mantel-Haenszel.

Subgroup analyses

Post-hoc, we conducted subgroup analyses to examine differences in post-treatment symptom outcomes between individually delivered and group-delivered ACT. Five trials testing individual ACT and one¹⁶ testing group ACT reported psychotic symptoms at the end of treatment as an outcome. In a separate meta-analysis (Figure 4), a medium to large effect size for group ACT was observed (standard mean difference [SMD] = -0.91, 95%CI -1.50 to -0.31). Individual ACT was seemingly not effective (SMD = -0.04, 95%CI -0.33 to 0.26).

Figure 4
Subgroup analysis (group vs. individual ACT). 95%CI = 95% confidence interval; ACT = acceptance and commitment therapy; df = degrees of freedom; IV = inverse variance; SD = standard deviation.

Sensitivity analyses

Of the six trials included in primary meta-analysis, three¹¹,²⁹,³⁰ had a comparator intervention (e.g., befriending) that was intended to control for the nonspecific effects of treatment, while three¹⁰,¹²,¹⁶ used treatment as usual. In post-hoc sensitivity analyses (Figure 5), there was a medium effect size (SMD= -0.64, 95%CI -1.02 to -0.27) where treatment-as-usual was the comparator. In trials with active control, no effect of ACT on post-treatment psychosis symptoms was observed; conversely, there was a nonsignificant trend toward the control (SMD = 0.16, 95%CI -0.15 to 0.48).

Figure 5
Sensitivity analysis (comparator intervention vs. treatment as usual). 95%CI = 95% confidence interval; df = degrees of freedom; IV = inverse variance; SD = standard deviation.

Prospective trial registration

Of the eight trials included in this review, two were registered prospectively and one retrospectively. Four were not registered. White et al.¹² registered their trial, but did not report the registration number in the manuscript.

Risk of bias

Table 3 shows the risk of bias for the eight included trials. One trial was considered to have a low risk of bias, and one as some concerns. Six trials were rated as having a high risk of bias. This mostly related to domain 5 (bias in selection of the reported result), i.e., results were selected from multiple outcomes. For example, Spidel et al.¹⁶ listed six outcomes – emotional regulation-acceptance, psychiatric symptoms, trauma symptoms, anxiety symptoms, service engagement, and feedback interview – and no prespecified plan for analysis was available. Tyrberg³¹ was rated as having a high risk of bias arising from the randomization process. The authors state that they used pairwise randomization to ensure an even flow of participants; consequently, it seems reasonable to suspect that the enrolling researcher would therefore have knowledge of the upcoming group allocation. In the Cochrane risk of bias algorithm³⁴ (page 6), this requires a high risk of bias rating. Three studies⁹,¹⁶,²⁹ were rated some concerns against domain 1, because essentially no information was given about the randomization process. For example, Bach & Hayes⁹ state that “those who agreed to participate were randomly assigned” (page 1130); no other information is provided. We rated four studies some concern against domain 4 (risk of bias in measurement of the outcome), due to potential for outcome assessment to have been influenced by knowledge of the assigned intervention.

Table 3
Risk of bias determined using the revised Cochrane risk of bias tool for randomized trials (RoB 2)

Study	Domain 1: Risk of bias arising from the randomization process	Domain 2: Risk of bias due to deviations from the intended interventions (effect of assignment to intervention)	Domain 2: Risk of bias due to deviations from the intended interventions (effect of adhering to intervention)	Domain 3: Missing outcome data	Domaine 4: Risk of bias in measurement of the outcome	Domain 5: Risk of bias in selection of the reported result	Overall risk of bias*
Bach & Hayes9	Some concerns	Low	Low	Low	Low	High	High
Gaudiano & Herbert10	Low	Low	Low	Low	Some concerns	High	High
White12	Low	Low	Low	Low	Low	High	High
Shawyer11	Low	Low	Low	Low	Low	High	High
Boden unpublished	Some concerns	Low	Low	Low	Some concerns	Some concerns	Some concerns
Shawyer30	Low	Low	Low	Low	Low	Low	Low
Spidel16	Some concerns	Low	Low	Low	Some concerns	High	High
Tyrberg31	High*	Low	Low	Low	Some concerns	High	High

• ^*
  The default overall rating of risk of bias is high risk if even one of the domains is judged as such.²⁴

Publication bias

Figure 6 shows a funnel plot for the six studies included in the meta-analysis of ACT against psychotic symptoms. The modest number of trials means that it is challenging to determine asymmetry. Visual interpretation of the plot is, however, suggestive of possible publication bias. This publication bias is supported by our search of trial registries, with seven trials remaining unpublished, despite considerable time elapsing since they were registered.

Figure 6
Funnel plot of studies reporting the effect of acceptance and commitment therapy (ACT) on psychotic symptoms. SE = standard error; SMD = standard mean difference.

Discussion

This systematic review aimed to establish the safety and efficacy of ACT in the treatment of schizophrenia-spectrum and other psychotic disorders. Our meta-analysis suggests ACT has a small, nonsignificant effect size vs. comparator interventions against psychotic symptoms. We observed a significant reduction in the risk of relapse, but only three trials were included in the meta-analysis. Sub-group analysis provided limited evidence that group ACT may be more effective than individual ACT. Sensitivity analysis suggested that, in more methodologically robust studies (with an attentional control), ACT was not an effective treatment against psychosis symptoms. The safety of ACT was not adequately reported in the included trials, and this is of notable concern. We identified seven completed trials where researchers would not release data to include in our meta-analysis. We therefore advise caution in applying ACT in clinical practice.

Our cautious conclusion about the safety and efficacy of ACT in psychosis is consistent with the review by Ost,⁵ but discrepant with the three other reviews of ACT in psychosis.⁶,⁷,¹⁵ As we have already noted, Wakefield et al.⁶ included some data sets twice in their qualitative synthesis, which may have influenced their generally positive conclusion. Jansen et al.¹⁵ made critical errors in data extraction; consequently, we believe the effect size they report against overall symptomatology in their paper is an overestimate. We note that the authors may have also made the same error when extracting data on acceptance and anxiety.

Risk of bias

Six trials were rated as having a high risk of bias. Our rating is notably discordant from that of the recent Jansen et al.¹⁵ review, which stated that ACT trials had a low risk of bias. In part, the discrepancy between the two reviews may be related to the version of the Cochrane risk of bias measure that was used; Jansen et al.¹⁵ used the original version,¹⁷ while we used the revised (RoB 2) tool.²⁴ Specifically, Jansen et al.¹⁵ reported that included studies had a low risk of selective reporting bias – an item that has been revised in version 2. A judgment of low risk of bias in selection of the reported result now requires that the trial was analyzed following a prespecified plan that was finalized before unblinded outcome data were available for analysis. Only one trial met these criteria.³⁰

Trial registration

Only two trials included in this review were prospectively registered. Pre-registration is intended to minimize the risk of selective reporting bias.³⁵ Previous systematic reviews of ACT have not noted trial registration as an issue.

Harms reporting

Authors have previously noted the need for better harms reporting in clinical trials.³⁶ We found that there was virtually no discussion of potential harms associated with ACT in the included studies. Authors that did mention adverse events simply stated that no harm occurred during the conduct of the trial. Jansen et al.¹⁵ concluded their systematic review by stating that ACT appears to be a safe intervention. We disagree with this statement. There was little evidence that researchers in included trials actively monitored participants for adverse events, including serious adverse events. According to Good Clinical Practice (GCP) guidance, an adverse event is “any untoward medical occurrence in a participant, which does not necessarily have a causal relationship with the trial intervention.”³⁷ It seems extremely unlikely that no events that met this definition occurred during the conduct of the included trials. In our view, there is a need for investigators to be more proactive in monitoring harms in future ACT trials.

Risks with underpowered trials

The sample size of the included trials was modest, and included trials were potentially underpowered. The sample size calculation in the Shawyer et al.¹¹ trial, for example, required 60 participants, yet only 44 were recruited; a limitation the authors acknowledge. Intuitively, researchers are most concerned with an underpowered trial resulting in a type II error (where the hypothesis is true, but not rejected). However, should underpowered trials deliver a positive result, there may be an increased risk of type I error (where the hypothesis is false but accepted³⁸). Small sample sizes can undermine the reliability of trials; this should be borne in mind when interpreting results.

Limitations of this review

Seven completed yet unpublished trials were identified in our search, and our funnel plot suggested the possibility of important publication bias. The identification of unpublished work is a strength of this review. This finding also raises concerns about the validity of our – indeed anyone’s – observations. It is possible that further unregistered and unpublished ACT trials have been completed, but not reported. Authors of unpublished trials either did not respond to our e-mails or were unwilling to share their data. It is understandable that research groups typically first publish their work in a peer-reviewed journal before making data publicly available; that said, one of the unpublished trials (ISRCTN68540929) was – according to the registry entry – completed in 2013, and another in 2009 (NCT00657631), yet their data are still not publicly available.

Several further limitations of this review warrant mention. Firstly, we used an inclusive definition of ACT, to an extent this was unavoidable given the small number of trials currently published. Consequently, there was considerable variation in the duration and mode of delivery; ideally, we would have preferred trials that tested a prescribed ACT treatment program (e.g., 12 weekly sessions of individual therapy delivered by a psychologist following a standard treatment manual).

A further limitation pertains to heterogeneity. For example, Shawyer et al.¹¹ only included participants experiencing problematic command hallucinations, while Spidel et al.¹⁶ focused on individuals with psychosis who had experienced childhood trauma. Similarly, comparators varied across studies as well (e.g., treatment as usual, befriending). We only included studies published in English. There may be relevant studies that were published in other languages. Finally, we did not extract follow-up data and can consequently not comment on the long-term benefits of ACT.

Conclusion

Our systematic review and meta-analysis provide limited evidence that ACT is an effective treatment for people experiencing psychosis. Included trials were modest in scale and had a high risk of bias. In our view, there is a need for methodologically rigorous, appropriately powered, multicenter trials, with an appropriate comparator. Additionally, we encourage researchers to ensure that their trials are registered, and that data are made available promptly. Our advice to clinicians is that caution be exercised when applying a treatment whose efficacy has not yet been established, and where there is a paucity of data concerning the potential for harm.

References

• ¹
  Webster M. Introduction to acceptance and commitment therapy. Adv Psychiatr Treat. 2011;17:309-16.
• ²
  Hayes SCSC, Strosahl KD, Wilson KG. Acceptance and commitment therapy: an experiental approach to behaviour change. New York: Guildford; 1999.
• ³
  Hayes SC, Luoma JB, Bond FW, Masuda A, Lillis J. Acceptance and commitment therapy: model, processes and outcomes. Behav Res Ther. 2006;44:1-25.
• ⁴
  O’Donoghue EK, Morris EMJ, Oliver JE, Johns LC. ACT for psychosis recovery: A practical manual for group-based interventions using acceptance and commitment therapy. Oakland: Context Press/New Harbinger Publications; 2018.
• ⁵
  Ost LG. The efficacy of acceptance and commitment therapy: an updated systematic review and meta-analysis. Behav Res Ther. 2014;61:105-21.
• ⁶
  Wakefield S, Roebuck S, Boyden P. The evidence base of acceptance and commitment therapy (ACT) in psychosis: a systematic review. J Contextual Behav Sci. 2018;10:1-13.
• ⁷
  Cramer H, Lauche R, Haller H, Langhorst J, Dobos G. Mindfulness- and acceptance-based interventions for psychosis: a systematic review and meta-analysis. Glob Adv Health Med. 2016;5:30-43.
• ⁸
  Jansson MM, Syrjala HP, Ala-Kokko TI. Association of nurse staffing and nursing workload with ventilator-associated pneumonia and mortality: a prospective, single-center cohort study. J Hosp Infect. 2019;101:257-63.
• ⁹
  Bach P, Hayes SC. The use of acceptance and commitment therapy to prevent the rehospitalization of psychotic patients: a randomized controlled trial. J Consult Clin Psychol. 2002;70:1129-39.
• ¹⁰
  Gaudiano BA, Herbert JD. Acute treatment of inpatients with psychotic symptoms using acceptance and commitment therapy: pilot results. Behav Res Ther. 2006;44:415-37.
• ¹¹
  Shawyer F, Farhall J, Mackinnon A, Trauer T, Sims E, Ratcliff K, et al. A randomised controlled trial of acceptance-based cognitive behavioural therapy for command hallucinations in psychotic disorders. Behav Res Ther. 2012;50:110-21.
• ¹²
  White R, Gumley A, McTaggart J, Rattrie L, McConville D, Cleare S, et al. A feasibility study of acceptance and commitment therapy for emotional dysfunction following psychosis. Behav Res Ther. 2011;49:901-7.
• ¹³
  Atkins PW, Ciarrochi J, Gaudiano BA, Bricker JB, Donald J, Rovner G, et al. Departing from the essential features of a high quality systematic review of psychotherapy: a response to Öst (2014) and recommendations for improvement. Behav Res Ther. 2017;97:259-72.
• ¹⁴
  Bach P, Gaudiano BA, Hayes SC, Herbert JD. Acceptance and commitment therapy for psychosis: intent to treat, hospitalization outcome and mediation by believability. Psychosis. 2013;5:166-74.
• ¹⁵
  Jansen JE, Gleeson J, Bendall S, Rice S, Alvarez-Jimenez M. Acceptance- and mindfulness-based interventions for persons with psychosis: a systematic review and meta-analysis. Schizophr Res. 2020;215:25-37.
• ¹⁶
  Spidel A, Lecomte T, Kealy D, Daigneault I. Acceptance and commitment therapy for psychosis and trauma: improvement in psychiatric symptoms, emotion regulation, and treatment compliance following a brief group intervention. Psychol Psychother. 2018;91:248-61.
• ¹⁷
  Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.
• ¹⁸
  Gray R, Hillel SA, Brown E, Al Ghareeb A. Safety and efficacy of acceptance and commitment therapy (ACT) in schizophrenia spectrum and other psychotic disorders: protocol for a systematic review and meta-analysis. Methods Protoc. 2018;1:38.
• ¹⁹
  Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6:e1000097.
• ²⁰
  Franco A, Malhotra N, Simonovits G. Social science. Publication bias in the social sciences: unlocking the file drawer. Science. 2014;345:1502-5.
• ²¹
  De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. N Engl J Med. 2004;351:1250-1.
• ²²
  Kay SR, Fiszbein A, Opler LA. The positive and negative syndrome scale (PANSS) for schizophrenia. Schizophr Bull. 1987;13:261-76.
• ²³
  Overall JE, Gorham DR. The brief psychiatric rating scale. Psychol Rep. 1962;10:799-812.
• ²⁴
  Sterne JA, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:4898.
• ²⁵
  Faraone SV. Interpreting estimates of treatment effects: implications for managed care. P T. 2008;33:700-11.
• ²⁶
  Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21:1539-58.
• ²⁷
  Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327:557-60.
• ²⁸
  Acceptance and Commitment Therapy for the Inpatient Treatment of Psychosis (ACT-IP) [Internet]. 2016 Apr [cited 2020 Feb 26]. clinicaltrials.gov/ct2/show/NCT01981356
  » clinicaltrials.gov/ct2/show/NCT01981356
• ²⁹
  Boden MT, Gaudiano BA, Walser RD, Timko C, Faustman W, Yasmin S, et al. Feasibility and challenges of inpatient psychotherapy for psychosis: lessons learned from a veterans health administration pilot randomized controlled trial. BMC Res Notes. 2016;9:376.
• ³⁰
  Shawyer F, Farhall J, Thomas N, Hayes SC, Gallop R, Copolov D, et al. Acceptance and commitment therapy for psychosis: randomised controlled trial. Br J Psychiatry. 2017;210:140-8.
• ³¹
  Tyrberg MJ, Carlbring P, Lundgren T. Brief acceptance and commitment therapy for psychotic inpatients: a randomized controlled feasibility trial in Sweden. Nord Psychol. 2017;69:110-25.
• ³²
  Gaudiano BA, Herbert JD. Corrigendum to ‘Acute treatment of inpatients with psychotic symptoms using Acceptance and Commitment Therapy: Pilot result’ [Behaviour Research and Therapy 44 (2006) 415-437]. Behav Res Ther. 2020;124:103534. Published Erratum.
• ³³
  Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c869.
• ³⁴
  Higgins JPT, Savović J, Page MJ, Sterne JAC; RoB 2 Development Group. Revised Cochrane risk-of-bias tool for randomized trials (RoB 2). Short Version (CRIBSHEET) [Internet]. 2019 Aug 22. [cited 2020 Apr 19]. drive.google.com/file/d/1Q4Fk3HCuBRwIDWTGZa5oH11OdR4Gbhdo/view?usp=drive_open&usp=embed_facebook
  » drive.google.com/file/d/1Q4Fk3HCuBRwIDWTGZa5oH11OdR4Gbhdo/view?usp=drive_open&usp=embed_facebook
• ³⁵
  Gray R, Gray G, Brown E. A review of prospective registration of trials published in nursing science journals in 2017. J Adv Nurs. 2019;75:3263-71.
• ³⁶
  Ioannidis JP, Evans SJ, Gøtzsche PC, O’Neill RT, Altman DG, Schulz K, et al. Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med. 2004;141:781-8.
• ³⁷
  Barton A. Handbook for good clinical research practice (GCP): guidance for implementation. J Epidemiol Community Health. 2007;61:559.
• ³⁸
  Button KS, Ioannidis JP, Mokrysz C, Nosek BA, Flint J, Robinson ES, et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 2013;14:365-76.

Publication Dates

• Publication in this collection
  03 Aug 2020
• Date of issue
  May-Jun 2021

History

• Received
  8 Mar 2020
• Accepted
  22 Apr 2020