Use and Abuse of Statistics in Evidence-Based Medicine
Authors: John M. Kirkwood, Joseph Ibrahim, Vernon K. Sondak, Marc S. Ernstoff, Lawrence Flaherty, Frank J. Haluska, M.B. Lens, M. Dawes
Journal: Journal of Clinical Oncology (2002)

Article Tools SPECIAL DEPARTMENTS Article Tools OPTIONS & TOOLS Export Citation Track Citation Add To Favorites Rights & Permissions COMPANION ARTICLES No companion articles ARTICLE CITATION DOI: 10.1200/JCO.2002.99.132 Journal of Clinical Oncology - published online before print September 21, 2016 PMID: 12351612 Use and Abuse of Statistics in Evidence-Based Medicine John M. KirkwoodxJohn M. KirkwoodSearch for articles by this author , Joseph IbrahimxJoseph IbrahimSearch for articles by this author , Vernon K. SondakxVernon K. SondakSearch for articles by this author , Marc S. ErnstoffxMarc S. ErnstoffSearch for articles by this author , Lawrence FlahertyxLawrence FlahertySearch for articles by this author , Frank J. HaluskaxFrank J. HaluskaSearch for articles by this author M.B. LensxM.B. LensSearch for articles by this author , M. DawesxM. DawesSearch for articles by this author Show More for the Eastern Cooperative Oncology Group, Southwestern Oncology Group, Cancer and Leukemia Group B, and the US Melanoma IntergroupUniversity of Oxford, John Radcliffe Hospital, Oxford, United Kingdom https://doi.org/10.1200/JCO.2002.99.132 First Page Full Text PDF Figures and Tables © 2002 by American Society of Clinical OncologyjcoJ Clin OncolJournal of Clinical OncologyJCO0732-183X1527-7755American Society of Clinical OncologyResponse01102002In Reply:Kirkwood et al have criticized a meta-analysis in a systematic review.1 For reasons made clear in the article, we have not undertaken meta-analysis but confined ourselves to a systematic review. Meta-analysis is a statistical method for combining the results from two or more independent studies (quantitative synthesis), whereas a systematic review describes any review of a body of data that uses clearly defined methods and criteria.2,3 Furthermore, we were explicit with the statement in the Discussion of our article that the data from included individual studies were not pooled.The fact that our systematic review omitted the results from the United States (US) Intergroup trial E1694,4 published within the window of our review, should not be considered as an error. The US Intergroup trial E1694 simply did not fulfill our inclusion criteria. It is clearly stated in the Methods section of our article that the studies were excluded if they used combination therapy or compared interferon-alfa treatment with some other form of systemic therapy. The trial E1694 compared interferon-alfa therapy with the GM2-KLH/QS vaccine. Although Kirkwood et al accentuated the importance of the positive results from this trial, these results should be considered in context with other studies that compare interferon with other systemic therapies.We agree with Kirkwood et al that use of the raw data in the analysis is very important. However, although we had requested that authors from included trials provide us with raw data, the trialists decided not to make their data available. Therefore, we were limited to including only published data and adhering to one of the principal recommendations of the Quality of Reporting of Meta-Analyses (QUOROM) statement—to present simple summary results from each treatment group in each trial, for each primary outcome.5 Some authors advocated the importance of publishing the raw data used in medical research articles.6,7We recognize that choosing an adequate statistical method when analyzing the data is important. The log-rank test may be preferable, but in the absence of the raw data, we were not able to use it. However, we must be precise that we did not run Fisher’s exact test on the published data. We calculated odds ratios with 95% confidence intervals, as clearly described in the Methods section of the article. Although both methods can be used with 2 × 2 tables when the question is about an association, Fisher’s exact test is a test statistic which two-tailed P value rejects or does not reject the null hypothesis. On the contrary, the odds ratio is an effect estimate reported with the 95% confidence interval (there is no null hypothesis for confidence intervals). Kirkwood et al gave an example using data from the E1684 trial advocating the importance of obtained evidence based on an arbitrary division of results as “significant” or “nonsignificant” according to the commonly used threshold of P value (P = .05). Kirkwood et al used one-sided P values, which are rarely appropriate. Only a small number of one-sided t tests are found in the published articles. Kirkwood et al stated only in their letter that the one-sided tests were preplanned, but from the original publication of the trial,8 the reader cannot determine whether this decision was made before the data were analyzed and whether it is not dependent on what the results were. Furthermore, the authors did not justify why they have considered that a real difference can occur in only one direction. We would like to note that if we use as an example the end point of overall survival, the corresponding two-sided P value for the log-rank test would be .047.The validity of P values is important, but P values smaller than .05 cannot be considered as proof of conclusive, or even strong, evidence against the null hypothesis. Reporting of medical research should continue to move from the idea that results are significant or nonsignificant to the interpretation of findings in the context of the type of study and other available evidence.9The validity of the statistical analysis used in the randomized trial and the clinical relevance of the results are important parameters when assessing the quality of the trials, but the design, conduct, and quality of reporting are as, if not more, important.10 It is essential that randomized trials are reported adequately. Proper methodology should be used and reported explicitly so that readers should not have to infer what methodology was probably used; they should see that it have been used.11 There is no described analysis that can compensate for biases introduced through poor trial design. The achievement of allocation concealment is crucial when conducting and reporting randomized trials.12 Inadequate or unclear concealment of treatment allocation may overestimate the treatment effect by as much as 30%.13 Although the Eastern Cooperative Oncology Group and Intergroup trials were planned by US cooperative groups and the National Cancer Institute, published reports of the trials included in our review (E1684 and E1690) did not have sufficient description of the study design to suggest that adequate concealment of allocation had taken place.It is better to address weaknesses in research design and to consider their possible effects on the results and their interpretation than to ignore them in the hope that they will not be noticed.14 In order to avoid biased findings of systematic reviews, there is a need to improve the quality of the included trials and on a routine basis conduct sensitivity analyses to assess the methodologic quality of controlled trials.13 Highlighting the importance that trial design and analysis can have on results is one of the major roles of evidence-based medicine.1. Lens MB, Dawes M: Interferon alfa therapy for malignant melanoma: A systematic review of randomized controlled trials. J Clin Oncol 20:: 1818,2002-1825, Link, Google Scholar2. Egger M, Smith GD: Meta-analysis: Potentials and promise. BMJ 315:: 1371,1997-1374, Crossref, Medline, Google Scholar3. Chalmers I, Altman DG: Foreword, in Chalmers I, Altman DG (eds): Systematic Reviews. London United Kingdom, BMJ Publishing, 1995 Google Scholar4. Kirkwood JM, Ibrahim J, Sosman JA, et al: High-dose interferon alfa-2b significantly prolongs relapse-free and overall survival compared with the GM2-KLH/QS-21 vaccine in patients with resected stage IIb-III melanoma: Results of intergroup trial E1694/S9512C509801. J Clin Oncol 19:: 2370,2001-2380, Link, Google Scholar5. Moher D, Cock DJ, Eastwood S, et al: Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement—Quality of Reporting of Meta-Analyses. Lancet 354:: 1896,1999-1900, Crossref, Medline, Google Scholar6. Hutchon DJR: Publishing raw data and real time statistical analysis on e-journals. BMJ 322:: 530,,2001 Crossref, Medline, Google Scholar7. Eysenbach G, Sa ER: Code of conduct is needed for publishing raw data. BMJ 323:: 166,,2001 (letter) Google Scholar8. Kirkwood JM, Strawderman MH, Ernstoff MC, et al: Interferon alfa-2b adjuvant therapy of high-risk resected cutaneous melanoma: The Eastern Cooperative Oncology Group trial EST 1684. J Clin Oncol 14:: 7,1996-17, Link, Google Scholar9. Sterne JA, Smith GD, Cox DR: Sifting the evidence: What’s wrong with significance tests? BMJ 322:: 226,2001-231, Crossref, Medline, Google Scholar10. Juni P, Altman DG, Egger M: Assessing the quality of controlled clinical trials, in Egger M, Smith GD, Altman DG (eds): Systematic Reviews in Health Care: Meta-Analysis in Context ( ed 2 ). London United Kingdom, BMJ Books, 2001 Google Scholar11. Altman DG: Better reporting of randomised controlled trials: The CONSORT statement. BMJ 313:: 570,1996-571, Crossref, Medline, Google Scholar12. Moher D, Schulz KF, Altman DG: The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 357:: 1191,2001-1194, Crossref, Medline, Google Scholar13. Juni P, Altman DG, Egger M: Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ 323:: 42,2001-46, Crossref, Medline, Google Scholar14. Altman DG, Gore SM, Gardner MJ, et al: Statistical guidelines for contributors to medical journals, in Altman DG, Machin D, Bryant TN, et al (eds): Statistics With Confidence ( ed 2 ). London United Kingdom, BMJ Books, 2000 Google Scholar