Methodological Issues in a Meta-Analysis
Authors: Kristian Thorlund, Tahany Awad
Journal: Hepatology (2010)

We thank Dr. Crémieux for bringing this issue to our attention, and we agree that sensitivity analysis is an important and necessary component of all systematic reviews. We are, however, concerned about the validity of the method proposed by Tian et al.1 that Crémieux has chosen for his re–meta-analysis of sustained virological response. Currently, the method by Tian et al. has only been sparsely validated for meta-analysis scenarios (>30 trials with event rates < 10%) that are far from representative of the sustained virological response meta-analysis (8 trials with event rates ranging from 20% to 80%).2 Although the method by Tian et al. may in time be proven statistically superior, at the current stage, preferring this method over the conventional meta-analysis methods is analogous to preferring a phase 1 drug over a Food and Drug Administration–approved drug in clinical practice. Crémieux points out that the lack of evidence for heterogeneity (I2 = 0%) makes the fixed-effects model appropriate. In other words, Crémieux suggests that the choice of model (fixed versus random) may be determined by the estimate of heterogeneity (I2). This is highly inappropriate meta-analytic conduct that the Cochrane Collaboration abandoned some years ago.3 Findings from empirical studies and simulation studies also testify to the inappropriateness of this conduct.4-6 However, since we do not have evidence of heterogeneity, we cannot preclude the possibility that a fixed-effects model is appropriate. Using the conventional Mantel-Haenszel fixed-effects model, we obtained a pooled relative risk of 1.10 (95% confidence interval = 1.03-1.18). Excluding low-dose peginterferon alfa-2b from the Individualized Dosing Efficacy Versus Flat Dosing to Assess Optimal Pegylated Interferon Therapy (IDEAL) trial, we obtained a pooled relative risk of 1.10 (95% confidence interval = 1.03-1.19). In his commentary, Crémieux interprets his findings as discrepant with the original findings of our systematic review. Although we do agree that some caution should be exercised, the word discrepant seems too strong to describe the observed statistical difference. In fact, this claim of discrepancy is based on an unfair comparison. In our original analyses, we took precautions to interpret the statistical inference according to the strength of the evidence. In this vein, we constructed adjusted thresholds for statistical significance by using an approach (trial sequential monitoring boundaries) analogous to approaches used for interim analysis in clinical trials.7-10 Roughly speaking, we can state that this analysis translates into an adjusted P value less than 5% but not less than 1% (the unadjusted meta-analyzed P value was 0.8%). In comparison, the confidence intervals proposed by Tian et al.1 would translate into a P value of roughly 6%. Several authors have previously warned against relying on the conventional criterion for statistical significance.7, 8, 11-14 In this vein, a P value of 1% to 5% and a P value of 6% should be interpreted similarly. We thank Crémieux for bringing up issues that may have been hidden to the statistically inexperienced reader. However, even if one were to take a leap of faith and believe that Crémieux's analyses were based on a valid method, the conclusions of our analysis and his analysis still do not differ: there is statistical evidence, albeit moderate, that peginterferon alfa-2a is superior to peginterferon alfa-2b in achieving sustained virological response. Kristian Thorlund M.Sc.*, Tahany Awad M.Sc.*, * Cochrane Hepatobiliary Group, Copenhagen Trial Unit, Center for Clinical Intervention Research, Rigshospitalet, Copenhagen University Hospital, Copenhagen, Denmark.