Closing in on the Truth About Recombinant Human Bone Morphogenetic Protein-2: Evidence Synthesis, Data Sharing, Peer Review, and Reproducible Research
Authors: Christine Laine, Eliseo Guallar, Cynthia Mulrow, Darren B. Taichman, John E. Cornell, Deborah Cotton, Michael E. Griswold, A. Russell Localio, Anne R. Meibohm, Catharine B. Stack, Sankey V. Williams, Steven N. Goodman
Journal: Annals of Internal Medicine (2013)

Editorials18 June 2013Closing in on the Truth About Recombinant Human Bone Morphogenetic Protein-2: Evidence Synthesis, Data Sharing, Peer Review, and Reproducible ResearchFREEChristine Laine, MD, MPH, Editor in Chief, Eliseo Guallar, MD, PhD, Associate Editor, Statistics, Cynthia Mulrow, MD, MSc, Senior Deputy Editor, Darren B. Taichman, MD, PhD, Executive Deputy Editor, John E. Cornell, PhD, Associate Editor, Statistics, Deborah Cotton, MD, MPH, Deputy Editor, Michael E. Griswold, PhD, Associate Editor, Statistics, A. Russell Localio, JD, MPH, MS, Associate Editor, Statistics, Anne R. Meibohm, PhD, Associate Editor, Statistics, Catharine B. Stack, PhD, MS, Deputy Editor, Statistics, Sankey V. Williams, MD, Deputy Editor, and Steven N. Goodman, MD, PhD, MHS, Associate Editor, StatisticsChristine Laine, MD, MPH, Editor in ChiefSearch for more papers by this author, Eliseo Guallar, MD, PhD, Associate Editor, StatisticsSearch for more papers by this author, Cynthia Mulrow, MD, MSc, Senior Deputy EditorSearch for more papers by this author, Darren B. Taichman, MD, PhD, Executive Deputy EditorSearch for more papers by this author, John E. Cornell, PhD, Associate Editor, StatisticsSearch for more papers by this author, Deborah Cotton, MD, MPH, Deputy EditorSearch for more papers by this author, Michael E. Griswold, PhD, Associate Editor, StatisticsSearch for more papers by this author, A. Russell Localio, JD, MPH, MS, Associate Editor, StatisticsSearch for more papers by this author, Anne R. Meibohm, PhD, Associate Editor, StatisticsSearch for more papers by this author, Catharine B. Stack, PhD, MS, Deputy Editor, StatisticsSearch for more papers by this author, Sankey V. Williams, MD, Deputy EditorSearch for more papers by this author, and Steven N. Goodman, MD, PhD, MHS, Associate Editor, StatisticsSearch for more papers by this authorAuthor, Article, and Disclosure Informationhttps://doi.org/10.7326/0003-4819-158-12-201306180-00012 SectionsSupplemental MaterialAboutVisual AbstractPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinkedInRedditEmail This article has been corrected. The original version (PDF) is appended to this article as a supplement.Readers may think they are seeing double. They are not. This issue includes 2 systematic reviews (1, 2) that use the same data to address the same question: Compared with iliac crest bone grafting, does recombinant human bone morphogenetic protein-2 (rhBMP-2) safely improve outcomes of spinal fusion surgery? The Yale University Open Data Access Project commissioned the reviews, believing that confidence in the findings would be greatest if both reached the same conclusions. Both include patient-level meta-analyses of data from randomized, controlled trials (RCTs) obtained from Medtronic (Minneapolis, Minnesota), the manufacturer of rhBMP-2 (3–5). Annals’ review involved 2 internal teams of physician and statistical editors and distinct sets of external reviewers. Each team handled 1 manuscript, and neither had access to the other manuscript or associated reviews until both were accepted for publication. The clinical question is of great interest to orthopedic surgeons and relevant to internists who encounter patients who have had or are considering having spinal fusion. Yet, beyond the clinical issues, these dual reviews spotlight the power of evidence synthesis, data sharing, peer review, and reproducible research.The Sum Really Is More Than Its PartsAlthough the RCT may be the “king” of study designs, 1 RCT is rarely definitive. The first published rhBMP-2 trial (n = 14) reported that disability scores improved sooner with rhBMP-2 than with bone graft, quality of life improved, and no adverse events occurred (6). Although few would consider the results of this small, initial trial as a solid basis for treatment decisions, some may rely on the largest, later trial. That trial (n = 577) reported that rhBMP-2 improved outcomes, including function, pain, and return to work (7). Yet, after systematic evaluation and synthesis of all available evidence, both systematic reviews published here independently conclude that rhBMP-2, compared with iliac crest bone grafting, does not improve pain or function and increases adverse events, possibly including cancer (1, 2). For the outcome of fusion, the focus of initial enthusiasm for rhBMP-2, 1 review concluded that it improved by a marginally statistically significant amount with rhBMP-2 (overall fusion, 1.14 [95% CI, 1.03 to 1.25]), and the other concluded that it did not (anterior fusion, 1.05 [CI, 0.88 to 1.24]; posterior fusion, 1.16 [CI, 0.96 to 1.41]). Of importance, the CIs for both sets of estimates substantially overlap, and neither review found differences in pain or function, the outcomes that reflect patients’ well-being.This shows us yet again that conclusions not based on a careful synthesis of all available evidence can be misleading. For that reason, Annals considers systematic reviews to be the best evidence for clinical practice.Share and Share Alike (or a Problem Shared Is a Problem Halved)The reviews in this issue are not the first to synthesize evidence about rhBMP-2 in spine surgery. A 2011 review by Carragee and colleagues (8) suggested that adverse events associated with rhBMP-2 use ranged from 10% to 50% depending on the surgical approach. Because these authors did not have access to all patient-level RCT data, they had limited ability to precisely determine the benefits and harms of rhBMP-2.Although access to individual-patient data cannot overcome inherent biases in individual studies derived from design or implementation flaws, our experience illustrates that routine sharing of individual-patient RCT data, perhaps planned for prospectively with other researchers as exemplified by the Early Breast Cancer Trialists’ Collaborative Group (9) or efforts to promote sharing of National Institutes of Health–funded research (http://grants.nih.gov/grants/policy/data_sharing/data_sharing_guidance.htm), could lead to more rapid, precise, generalizable, and trustworthy knowledge about medical interventions than the “closed data” world that currently defines medical research. To learn how to access the data used in the current reviews, visit http://medicine.yale.edu/core/projects/yodap/index.aspx.What You See Is Not What We GotThese reviews provide an opportunity to reflect on peer review. As new methods of rapidly disseminating research proliferate, some question the benefits of traditional peer review and editing (10). The published reviews are accompanied by supplemental material that includes the initial submitted manuscript and correspondence between editors and authors. The changes from submission to acceptance included greater focus on individual-patient data, more sophisticated handling of missing and sparse data, more detailed methodological appendices (flow diagram, search strategy, details on analyses, and SAS codes), and more thorough assessment of study quality.The perfect peer-review process does not exist, but this peek into the black box of our procedures should show readers that what they see in the journal can be very different (and, we contend, better) than what the authors initially submit. We see helping authors improve their papers to make their methods more transparent and better align the evidence and the conclusions as one of our core missions. Improving reports of medical science facilitates effective postpublication review by the broader medical community, arguably the most important part of the review process, which ultimately improves the quality of the information available for patient care. The substantive changes we noted between submission and publication of these 2 reviews are typical for most articles we and other major journals publish; consistent with the positive changes seen when we formally evaluated the Annals peer review and editing process nearly 2 decades ago (11); and relevant to concerns about alternatives currently being proposed to prepublication, journal-based peer review, and editing. Although peer review continues after publication, the publication of cursorily vetted research can materially impede the attainment of truth.The Value of Reproducible ResearchWas there added value in soliciting 2 separate systematic reviews on this topic? As advocates of “reproducible research” (12), we think so. Despite the frequent use of rhBMP-2, there remained substantial controversy about its benefits and harms. The fact that 2 independent groups armed with the same question and the same patient-level data, along with the guidance of 2 independent sets of reviewers and editors, arrived at essentially the same conclusions should greatly temper enthusiasm for the intervention. Beyond replicating results, this exercise demonstrates additional value in having different scientists tackle the same data. The Table summarizes key characteristics of each review. There were modest differences in approach, and in fact, they came to somewhat different conclusions about the effect of the 2 interventions on bone fusion. This occurred because of differences in evidence selection and different decisions about analyzing posterior and anterior lumbar interbody fusion separately or together. There was another difference worth noting. Fu and colleagues (2) included a second aim of assessing bias in the original trial reports, and the editors handling their manuscript did not discourage this effort. Simmonds and colleagues (1) focused their efforts on analysis of the data rather than on identifying bias in the initial reports, a focus supported by the editorial team handling their manuscript. Different scientists (and different editors) can approach the same question and even the same data in different ways. This variety of perspectives can only enhance our understanding of the evidence and clarify evidence-based disagreement.Table. Key Features of Systematic Reviews and Meta-analyses of rhBMP-2In summary, this Yale University Open Data Access–orchestrated project is a novel exercise that illustrates the value of evidence synthesis, data sharing, peer review, editing, and reproducible research in helping us get closer to the truth. Although it is infeasible to apply this elaborate process to every clinical question, we owe it to our patients to reduce the obstacles that prevent similar efforts from being more common.Christine Laine, MD, MPHEditor in ChiefEliseo Guallar, MD, PhDAssociate Editor, StatisticsCynthia Mulrow, MD, MScSenior Deputy EditorDarren B. Taichman, MD, PhDExecutive Deputy EditorJohn E. Cornell, PhDAssociate Editor, StatisticsDeborah Cotton, MD, MPHDeputy EditorMichael E. Griswold, PhDAssociate Editor, StatisticsA. Russell Localio, JD, MPH, MSAssociate Editor, StatisticsAnne R. Meibohm, PhDAssociate Editor, StatisticsCatharine B. Stack, PhD, MSDeputy Editor, StatisticsSankey V. Williams, MDDeputy EditorSteven N. Goodman, MD, PhD, MHSAssociate Editor, StatisticsReferences1. Simmonds MC, Brown JV, Heirs MK, Higgins JP, Mannion RJ, Rodgers MA, et al. Safety and effectiveness of recombinant human bone morphogenetic protein-2 for spinal fusion. A meta-analysis of individual-participant data. Ann Intern Med. 2013;158:877-89. LinkGoogle Scholar2. Fu R, Selph S, McDonagh M, Peterson K, Tiwari A, Chou R, et al. Effectiveness and harms of recombinant human bone morphogenetic protein-2 in spine fusion. A systematic review and meta-analysis. Ann Intern Med. 2013;158:890-902. LinkGoogle Scholar3. Krumholz HM, Ross JS, Gross CP, Emanuel EJ, Hodshon B, Ritchie JD, et al. A historic moment for open science: the Yale University Open Data Access Project and Medtronic [Editorial]. Ann Intern Med. 2013;158:910-1. LinkGoogle Scholar4. Resnick D, Bozic KJ. Meta-analysis of trials of recombinant human bone morphogenetic protein-2: what should spine surgeons and their patients do with this information? [Editorial]. Ann Intern Med. 2013;158:912-3. LinkGoogle Scholar5. Kuntz RE. The changing structure of industry-sponsored clinical research: pioneering data sharing and transparency [Editorial]. Ann Intern Med. 2013;158:914-5. LinkGoogle Scholar6. Boden SD, Zdeblick TA, Sandhu HS, Heim SE. The use of rhBMP-2 in interbody fusion cages. Definitive evidence of osteoinduction in humans: a preliminary report. Spine (Phila Pa 1976). 2000;25:376-81. [PMID: 10703113] CrossrefMedlineGoogle Scholar7. Gornet MF, Burkus JK, Dryer RF, Peloza JH. Lumbar disc arthroplasty with Maverick disc versus stand-alone interbody fusion: a prospective, randomized, controlled, multicenter investigational device exemption trial. Spine (Phila Pa 1976). 2011;36:1600-11. [PMID: 21415812] CrossrefMedlineGoogle Scholar8. Carragee EJ, Hurwitz EL, Weiner BK. A critical review of recombinant human bone morphogenetic protein-2 trials in spinal surgery: emerging safety concerns and lessons learned. Spine J. 2011;11:471-91. [PMID: 21729796] CrossrefMedlineGoogle Scholar9. Early Breast Cancer Trialists' Collaborative Group. Treatment of Early Breast Cancer Volume 1: Worldwide Evidence 1985–1990. Oxford, UK: Oxford University Pr; 1990. Google Scholar10. Cohen P. A digital helping hand for scholarly work. The New York Times. 25 June 2011. Accessed at http://query.nytimes.com/gst/fullpage.html?res=9A0CE1D61F3EF936A15755C0A9679D8B63&ref=patriciacohen on 13 May 2013. Google Scholar11. Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann Intern Med. 1994;121:11-21. [PMID: 8198342] LinkGoogle Scholar12. Laine C, Goodman SN, Griswold ME, Sox HC. Reproducible research: moving toward research the public can really trust. Ann Intern Med. 2007;146:450-3. [PMID: 17339612] LinkGoogle Scholar Comments0 CommentsSign In to Submit A Comment Author, Article, and Disclosure InformationAffiliations: Disclosures: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M13-1163.Corresponding Author: Annals of Internal Medicine, American College of Physicians, 190 N. Independence Mall West, Philadelphia, PA 19106. PreviousarticleNextarticle Advertisement FiguresReferencesRelatedDetailsSee AlsoSafety and Effectiveness of Recombinant Human Bone Morphogenetic Protein-2 for Spinal Fusion Mark C. Simmonds , Jennifer V.E. Brown , Morag K. Heirs , Julian P.T. Higgins , Richard J. Mannion , Mark A. Rodgers , and Lesley A. Stewart Effectiveness and Harms of Recombinant Human Bone Morphogenetic Protein-2 in Spine Fusion Rongwei Fu , Shelley Selph , Marian McDonagh , Kimberly Peterson , Arpita Tiwari , Roger Chou , and Mark Helfand A Historic Moment for Open Science: The Yale University Open Data Access Project and Medtronic Harlan M. Krumholz , Joseph S. Ross , Cary P. Gross , Ezekiel J. Emanuel , Beth Hodshon , Jessica D. Ritchie , Jeffrey B. Low , and Richard Lehman Meta-analysis of Trials of Recombinant Human Bone Morphogenetic Protein-2: What Should Spine Surgeons and Their Patients Do With This Information? Daniel Resnick and Kevin J. Bozic The Changing Structure of Industry-Sponsored Clinical Research: Pioneering Data Sharing and Transparency Richard E. Kuntz Metrics Cited ByOverview and experience of the YODA Project with clinical trial data sharing after 5 yearsWNT16 induces proliferation and osteogenic differentiation of human perivascular stem cellsComparison of two independent systematic reviews of trials of recombinant human bone morphogenetic protein-2 (rhBMP-2): the Yale Open Data Access Medtronic ProjectUse of bone morphogenetic protein among patients undergoing fusion for degenerative diagnoses in the United States, 2002 to 2012Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and ElaborationKarel G.M. Moons, PhD, Douglas G. Altman, DSc, Johannes B. Reitsma, MD, PhD, John P.A. Ioannidis, MD, DSc, Petra Macaskill, PhD, Ewout W. Steyerberg, PhD, Andrew J. Vickers, PhD, David F. Ransohoff, MD, and Gary S. Collins, PhDAlveolar Ridge and Maxillary Sinus Augmentation Using rhBMP-2: A Systematic ReviewA review of the current published spinal literature regarding bone morphogenetic protein-2: an insight into potential biasAcceso público a los resultados de los ensayos clínicos con medicamentos: ¿es esto suficiente para clínicos y pacientes?Time to ensure that clinical trial appropriate results are actually publishedLettersWhen money talks 18 June 2013Volume 158, Issue 12Page: 916-918KeywordsAdverse eventsBoneOsteocalcinPeer reviewRecombinant proteinsScientistsSpinal fusionSurgerySystematic reviews ePublished: 18 June 2013 Issue Published: 18 June 2013 CopyrightCopyright © 2013 by American College of Physicians. All Rights Reserved.PDF DownloadLoading ...