The Utility of Systematic Reviews: The Case of Psychological Debriefing after Trauma
Authors: Tom Sensky
Journal: Psychotherapy and Psychosomatics (2003)

The burgeoning of the research literature has made it increasingly difficult for clinicians to keep up to date with recent advances. It is well recognised in general medicine that knowledge from traditional sources, such as expert clinicians and textbooks, may be out of date, unreliable, or even frankly erroneous [1]. There is no reason to assume that the situation would be different in psychiatry, psychology or psychosomatic medicine. Given the difficulties in keeping up with all the individual papers published, one solution is to rely on secondary sources, notably overviews and meta-analyses. However, even these knowledge sources have grown exponentially (fig. 1).The paper by Rose et al. [2]in this issue is the second recent review, following that of van Emmerik et al. [3], to report that routine debriefing after trauma has no demonstrable efficacy and may indeed lead to adverse outcomes. Another review published in 2000 recommended the routine use of debriefing following trauma as an effective method of reducing subsequent psychological distress [4]. It is telling that this last review, unlike the two already cited, was not systematic. A systematic review is one that 'has been prepared using some kind of systematic approach to minimising biases and random errors, and ... the components of the approach will be documented in a methods and materials section' [5].It is not often the case that two systematic reviews with such similar aims are published within a few months. Further consideration of these reviews not only yields valuable information about psychological treatment following trauma, but also highlights the strengths of systematic reviews in developing evidenced-based practice.From the papers identified in their review, Rose et al. [2]concluded that routine debriefing following trauma was at best ineffective, and at worst potentially harmful. In the two studies with the longest follow-up [6, 7], the group given debriefing fared worse than did patients receiving standard clinical care.van Emmerik et al. [3]carried out a meta-analysis of the results of the studies they identified for their review, using pooled effect size as the outcome measure. For each study, effect size was assessed as the difference between pre- and post-intervention values of the outcome measure, divided by the study standard deviation. Results from individual studies were weighted according to the sample size before being combined into a mean pooled effect size. Using post-traumatic stress disorder (PTSD) ratings as the outcome measure, the mean effect size for debriefing was only 0.13. More significantly, the 95% confidence interval (CI) of this estimate (–0.29 to 0.55) crosses zero, indicating that the true effect could be absent or even negative. This outcome was worse than for non-debriefing interventions (mean effect size 0.65, 95% CI 0.14–1.16). Even control conditions yielded better outcomes than debriefing, giving a pooled effect size of 0.47 (95% CI 0.28–0.66). However, the authors were appropriately cautious in interpreting the differences between the three treatment groups, noting that the CIs overlapped.Despite the 2 systematic reviews yielding the same results, on first reading, the differences between them appear somewhat disconcerting. Thus, while Rose et al. [2]found 11 studies that met their inclusion criteria, van Emmerik et al. [3]only included 7, and only 4 studies were included in both overviews.The reason for these differences lies in the differing inclusion and exclusion criteria used in the 2 reviews. For Rose et al. [2], the key criterion for studies to be included was that they should be randomised controlled trials. van Emmerik et al. [3]included studies which did not involve randomisation of patients to debriefing or a control intervention, but insisted that for inclusion in their review, studies had to have made assessments of the outcome measures both before and after the intervention, since the change between these two assessments was the basis of their meta-analysis.This highlights that there is not necessarily a single set of plausible inclusion and exclusion criteria for studies in a systematic review. On the one hand, it could be argued that randomised controlled trials should be selected because they are widely regarded as the most rigorous source of research evidence for an intervention, and there is evidence that where randomisation is absent or can be subverted, studies more frequently yield results favouring the experimental outcome [8, 9, 10]. On the other hand, a small minority of psychotherapists have argued that randomised controlled trials should not be regarded as the gold standard for intervention studies involving psychotherapy [11, 12]. More importantly, most published psychological interventions involve small patient numbers. Under such circumstances, randomisation may fail to achieve its main purpose, namely to make the treatment groups the same at baseline for the important variables, i.e. confounders as well as those chosen to measure outcome. It might be suggested that both sets of criteria should be applied, i.e. studies should be randomised and have pre- and post-intervention assessments. However, this not only limits the number of studies included to 4, but might exclude some results which are arguably valid. One solution to this problem, particularly where data from included studies have been combined in a meta-analysis, is to do a sensitivity analysis, i.e. to repeat the meta-analysis with, then without, studies whose validity is open to question, and determine whether the overall results remain unchanged. For example, van Emmerik et al. [3]queried whether or not to include a study involving women who had had an early miscarriage, but found that excluding this study made no difference to their overall results.The reviews of both Rose et al. [2]and van Emmerik et al. [3]were well conducted. A key consideration is, based on the details given in the published papers, would an independent reviewer find and include the same studies for review, and reach the results and conclusions? This and the comprehensive search for studies to include in the review distinguish systematic reviews from others. However, the differences in the methods outlined above indicate the need for careful and systematic appraisal of all reviews, taking advantage of appraisal checklists and other available resources [13, 14, 15, 16]. The need to provide better and more consistent information in publications of therapeutic interventions led to the CONSORT statement that specified information required in publications, and there is evidence that adoption of the CONSORT principles has improved the quality of relevant publications [17]. More recently, the QUORUM guidelines have been issued, which are equivalent to the CONSORT guidelines but for overviews [18].Well-presented systematic reviews provide full details not only of their methods, but also the data contributing to their results. While some might argue that this level of detail is excessive, it not only makes the results more transparent but also offers an opportunity to try to answer questions not included in the reviewers' aims, provided that the data collected are appropriate. Thus, for example Rose et al. [2]applied two measures of methodological quality – one generic, the other specific for PTSD – to the studies included in their review. They commented that there was little agreement between the two measures. In fact, a Spearman rank correlation between the estimates calculated from their data indicates that the correlation might actually be zero (rho = 0.56, but 95% CI –0.06 to 0.87). However, Rose et al. [2]drew attention to 2 studies in obstetric populations which are rather different from others in the nature of the trauma investigated. Removing these 2 studies, rho becomes 0.85, and the 95% CI is 0.44–0.97, i.e. a much more reassuring result!There is a particular risk of being 'seduced' by the methodology of critical reviews and focussing on plots of the results, while failing to exercise appropriate critical judgement of the remaining details. Some commentators have drawn attention to problems that have arisen in particular meta-analyses [19]. Unfortunately, there are numerous examples of flawed systematic reviews. For example, an early meta-analysis comparing cognitive therapy with other interventions for depression [20]included only studies which used the Beck Depression Inventory as the outcome measure. There is no a priori reason to exclude studies that used other depression measures, and because Beck is so closely associated with cognitive therapy, it is possible that selecting only those studies relying on the Beck Depression Inventory led to a biased sample. Another example appeared in the Cochrane Library but has since been withdrawn, comparing brief hospitalisation with standard in-patient care for serious mental disorders. It will be obvious to most clinicians that early discharge from in-patient care will depend crucially on what other treatment resources are available, such as day hospital, or assertive outreach. However, the reviewers acknowledged that their review contained none of this information, which effectively rendered any conclusions invalid. Similarly, clinicians will know that in considering the evidence for the efficacy of antidepressant treatment for depression, it may not be valid to extrapolate from studies in primary care to those in psychiatric out-patient clinics. However, a recent review not only failed to test this, but also gave no information about which studies were conducted in primary care settings [21]. The same review, of the efficacy and tolerability of venlafaxine compared with other antidepressants, grouped together data from studies where venlafaxine was compared with trazodone or mirtazepine. The only rationale for pooling these data appears to be that these two drugs are classified neither as tricyclic antidepressants nor as serotonin-selective reuptake inhibitors. There appears to be no valid pharmacological or clinical reason for considering these two drugs together. In fact, data presented in the paper suggest that although outcomes with trazodone appear less favourable than with venlafaxine, the opposite appears to be the case with mirtazepine, but any advantage for mirtazepine disappears when the data on the two drugs are combined.A charitable explanation for the drafting of such flawed systematic reviews is that to date they have often involved the collaboration between specialists who have knowledge of the subject of the review, and others who have special expertise in the methodology of meta-analyses. One might expect occasional communication problems between these two groups of experts. However, this does not explain how such reviews have come to be published.It is unusual for the studies that contribute to a meta-analysis to have used identical methods with patient samples having identical profiles. Accordingly, combining data from different studies inevitably involves making assumptions. A well-conducted review will list and justify these assumptions, and they will, where appropriate, be tested using sensitivity analyses (see above). Even with all this information, critical interpretation is still required, particularly of reviews of psychotherapy or other complex interventions. Thus, a meta-analysis of cognitive therapy for depression concluded that cognitive therapy was superior to a miscellaneous group of other interventions [22]. This group of 'other' therapies included some for which there was a therapeutic rationale, and others that were used as control interventions without any therapeutic rationale. The original review noted that there was heterogeneity among the studies in this 'other therapies' group. A re-analysis of the original data, but separating studies involving interventions with a therapeutic rationale from those without such a rationale, showed that these two types of intervention yielded different results [23].Debriefing has been widely practised because it appeared to well-meaning clinicians and therapists to be 'a good thing'. The current review, and that of van Emmerik et al. [2]clearly indicates the risks of following intuition rather than evidence. Interestingly, Rose et al. [2]reported that where patient satisfaction was rated in the studies they reviewed, the responses to debriefing tended to be positive. Despite the lack of any benefits for PTSD symptoms, it remains possible that debriefing has some other benefit for some patients. This appears reminiscent of the continuing controversy about outcomes of counseling in primary care [24], which remains generally well regarded by patients despite difficulties in consistently demonstrating its efficacy.Both Rose et al. [2]and van Emmerik et al. [3]pointed out that while their results indicated that single-session debriefing should not be offered routinely in the aftermath of trauma, it is possible that a subgroup of those who have experienced trauma might benefit from debriefing. If such a subgroup exists, its characteristics have yet to be identified. However, it is also important to remember that debriefing may have functions that vary according to the nature of the trauma and the person experiencing it. For the innocent victim of a rail accident or an earthquake, debriefing is likely to function as outlined by Rose et al. [2], with the risk that talking about the trauma could cause retraumatisation. However, for a psychiatric nurse who has been assaulted on duty by an in-patient, debriefing offers an opportunity to try to learn how the assault occurred despite nursing procedures in order to minimise the risks. Making sense of what had happened may for that nurse mean the difference between an early return to work and a prolonged sickness absence. Neither of the two reviews discussed provide data to distinguish between these two scenarios. It would be premature to conclude that debriefing should be abandoned entirely, and the challenge remains to determine whether there are some circumstances when debriefing should be considered or even offered. However, given the current evidence that debriefing may be detrimental to perhaps a majority of people who have suffered trauma, researchers will need to be circumspect in devising further intervention studies.