Replicability in Psychotherapy Research: Uncontrolled Risk Factors and Proposed Measures
Authors: Falk Leichsenring, Christiane Steinert
Journal: Psychotherapy and Psychosomatics (2016)

Showing that results can be replicated by using either the same or different methods is an essential characteristic of research [1,2]. However, problems in replicability are widespread in both basic and clinical research [1,2,3]. Low replicability of clinical research is particularly alarming because of the implied consequences for clinical practice (e.g. questionable treatment recommendations). Several shortcomings that are likely to reduce replicability have previously been identified (e.g. small samples, publication bias, or selective reporting) [4]. Based on recent empirical findings, we identified additional risk factors for non-replicability in psychotherapy research not yet systematically controlled for. Figure 1 highlights these risk factors and relates them to those already discussed.Clinicians vary in their efficacy, not only in psychotherapy, but also when delivering pharmacotherapy [[5], p. 170]. As a consequence, observations are not independent (e.g. the outcomes of patients X and Y treated by the same therapist Z) [5]. For this reason, therapists need to be statistically taken into account as a nested random factor [5]. Failure to do so may result in increased type I errors and overestimating treatment effects [[5], p. 164]. Thus, ignoring therapist effects may lead to false conclusions about treatment efficacy and results that are not replicable (e.g. 'treatment A is superior to B'). As recently shown by Monte Carlo studies, this effect heavily depends on the size of therapist effects [5], i.e. the more variance in outcome is due to the therapist, the higher the rate of type I error and the larger the estimated treatment effect [5]. This effect increases with fewer therapists and more patients treated per therapist [[5], p. 168]. Many trials are underpowered to detect therapist effects. Thus, even though such effects are not statistically significant, the pernicious effects on error rates, effect sizes, and replicability may be present. When comparing treatments, it is essential that for each treatment a bona fide version is implemented (treatment integrity). However, there are several examples for using 'strawman' or 'sham treatments' as comparators [ [5,6], p. 119], giving the favoured treatment a higher chance to be successful. Treatment integrity may not only be impaired by deviations from the treatment protocol, but also, for example, by conducting fewer sessions in the comparison condition, using less trained or competent therapists, or excluding essential treatment components [[5], p. 120]. Impaired treatment integrity of a comparator may result in overestimating the effects of the favoured treatment and underestimating the effects of the comparison treatment [5]. Results may not be replicated when using valid comparators. In psychotherapy research, small differences between bona fide treatments are rather the rule than the exception [5]. A recent meta-analysis addressing psychotherapy of anxiety disorders may serve as an example [7]. This meta-analysis reported a between-group effect size of 0.14 (p = 0.21) for anxiety-specific measures [7]. Thus, obviously differences were very small (< 0.20) and non-significant [7]. From this effect size, the author concluded to have found a 'signal' that one treatment is superior to others [[7], p. 315]. Firstly, it is questionable whether small effect sizes such as 0.14 are replicable [4] - they may be sheer randomness or noise. Secondly, it is questionable whether such differences are clinically meaningful [5]. Thirdly, if researchers do not a priori define the difference they are planning to regard as clinically meaningful (e.g. ≥0.25), the post hoc interpretation of a difference leaves room for arbitrary decisions (e.g. 'treatment X is superior to others'), constituting another factor that may reduce replicability.Researcher allegiances have been found to heavily influence the results of comparative psychotherapy studies [5,8,9], explaining up to 69% of variance in outcome [5,8]. Conflicts of interest often related to researcher allegiance are common in biomedical research but only sparsely reported [4], let alone controlled for. Researcher allegiance may be involved in all risk factors discussed (e.g. selective reporting or impaired treatment integrity) by introducing advantages or disadvantages to a favoured or an unfavoured treatment. Thus, on all levels of research allegiance may distort results and reduce replicability (fig. 1).Reviewers are usually researchers themselves and may be biased since they may have a stake in the outcome. A 'reviewer allegiance' may lead to biased recommendations for both publication and funding. As a consequence, the published evidence may be unbalanced (e.g. in favour, or not in favour, of a specific treatment: 'A superior to B') and not be replicable. As with researcher allegiance [1], reviewer allegiance may operate on different levels of consciousness. Reviewer allegiance has hardly been studied; however, research on investigator allegiance [5,8] suggests similar biases since the same researchers may be involved.Several measures to improve replicability have been proposed, for example, carrying out higher powered studies, taking all evidence into account, or preregistration of studies [4]. New methods for data analysis ('triple-blind', 'crowdsourcing') and peer review to deflate bias are discussed, for example, reviewing a study design prior to knowing the results, thus controlling for reviewer allegiance [1]. Guidelines to improve the reporting of interventions and ultimately the replicability of research have been developed [10]. Applying these measures to psychotherapy research can be expected to enhance replicability. However, the results reported here suggest that specific risk factors are not yet systematically controlled for. Researcher allegiance in particular, a powerful risk factor [8,9], has not yet been explicitly addressed in any of the available standards. It may affect, for example, treatment integrity in a way that is not identified by recent guidelines [10]: a comparison treatment may be set up, for example, as a version of psychodynamic therapy or cognitive-behavioural therapy which is neutered by excluding essential treatment elements [[5], p. 120]. However, at the same time it could still be properly described according to recent guidelines. In studies comparing rival approaches, allegiance can best be controlled for by including researchers of the rival approaches on an equal basis (adversarial collaboration) [1]. Thus, additional steps seem to be required. We propose including the following measures in CONSORT and journal guidelines:• addressing researcher and reviewer allegiance (e.g. by adversarial collaboration or reviewing designs prior to knowing results); • assuring a valid implementation of each treatment, including all relevant treatment elements (e.g. by including specific items in guidelines and/or adversarial collaboration); • systematically taking therapist effects into account as a nested random factor; • a priori defining clinically meaningful differences (e.g. >0.25 or >0.30) to prevent overestimating small effect sizes and arbitrary conclusions.The authors have no conflicts of interest to declare.