Comparative Effectiveness Research: A Report From the Institute of Medicine
Authors: Harold C. Sox
Journal: Annals of Internal Medicine (2009)

Medicine and Public Issues4 August 2009Comparative Effectiveness Research: A Report From the Institute of MedicineFREEHarold C. Sox, MD, Editor and Sheldon Greenfield, MDHarold C. Sox, MD, EditorFrom the American College of Physicians, Philadelphia, Pennsylvania, and University of California, Irvine, Irvine, California. and Sheldon Greenfield, MDFrom the American College of Physicians, Philadelphia, Pennsylvania, and University of California, Irvine, Irvine, California.Author, Article, and Disclosure Informationhttps://doi.org/10.7326/0003-4819-151-3-200908040-00125 SectionsAboutVisual AbstractPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinkedInRedditEmail Join the dialogue on health care reform. Comment on the perspectives published in Annals and offer ideas of your own. All thoughtful voices should be heard.On 30 June 2009, another milestone was achieved in progress toward a national system to promote medical research that focuses on decision making by physicians and patients. The first steps occurred in late 2007 and early 2008 with a seminal article (1) and an Institute of Medicine (IOM) report that called for a national initiative of research that would support better decision making about interventions in health care (2). One of us was a coeditor of the IOM report. A third milestone was reached when both presidential candidates endorsed this concept. A fourth milestone came when the president signed into law the American Recovery and Reinvestment Act of 2009 (ARRA); this act allotted $1.1 billion to support this form of research, which had become known as comparative effectiveness research (CER). The legislation created a federal council on CER and asked the IOM to elicit input from a broad array of stakeholders on which research topics should have the highest-priority for funding through the ARRA and to then develop a list of the highest-priority topics for the Secretary of Health and Human Services to consider. By law, the Federal Coordinating Council and the IOM committee to set priorities for comparative effectiveness research reported to the Secretary on 30 June 2009. This issue contains both this article, which is a commentary on the IOM committee report (3) by the committee's cochairs, and a perspective on better research methods for CER (4).Definition of CERThe IOM committee quickly settled on a working definition of CER, which consisted of the elements of earlier definitions reduced to 2 sentences:CER is the generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat and monitor a clinical condition, or to improve the delivery of care. The purpose of CER is to assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve health care at both the individual and population levels.Two key elements that are embedded in this definition are the direct comparison of effective interventions and their study in patients who are typical of day-to-day clinical care. These features would ensure that the research would provide information that decision makers need to know, as would a third feature, research designed to identify the clinical characteristics that predict which intervention would be most successful in an individual patient. The same research design would also help policymakers by identifying subpopulations of patients that are more likely to benefit from one intervention than the other.High-Priority Research TopicsThe IOM committee sought advice from a broad range of stakeholders and received it in 3 forms: approximately 90 letters; 54 oral presentations at a day-long hearing in Washington, DC; and a Web-based nomination form. Overall, the committee received more than 2606 nominations from 1758 individual responders within 3 weeks. In a 3-step voting process, the committee identified 100 high-priority topics, which it ranked in quartiles. At each stage, the committee sought to have a balanced portfolio of topics. Table 5-1 in the report (available at www.iom.edu/cerpriorities) lists the 100 highest-ranked topics.Establishing a Sustainable National Program of CERThe U.S. Congress is currently debating legislation that would guarantee health insurance to more Americans while reforming the U.S. health system. A key aim of the legislation is to moderate the rate at which the cost of health care increases while improving the quality of care. Many leaders believe that reaching this goal will require a sustained effort to produce better evidence to inform decision making. Government officials have characterized the $1.1 billion allocated to CER as a down payment on a national program of CER. As of this writing, the House of Representatives bill proposes a national program of CER, and the Senate Finance Committee white paper does likewise. Accordingly, the IOM committee made several recommendations aimed at a sustainable, trustworthy national CER initiative. The Appendix provides the full text of these recommendations.GovernanceThe 2008 IOM committee recommended establishing a national program "with authority, overarching responsibility, sustained resources, and adequate capacity to ensure production of credible, unbiased information about what is known and what is not known about clinical effectiveness" (2). The current committee expanded this recommendation by calling upon the secretary of the Department of Health and Human Services "to establish a mechanism—such as a coordinating advisory body—with the authority to strategize, organize, monitor, evaluate, and report on the implementation and impact of the CER Program." The committee recognized that many agencies of the federal government are engaged in research that fits some or all of the key elements of CER. To make the best use of the funds allocated for CER, these agencies should be jointly accountable for spending CER funds on research that reflects the same conceptualization of CER. Therefore, these disparate efforts should use a single definition of CER and the same definitions of outcomes and measures of function and illness. This form of cross-agency coordination is compatible with each agency focusing on the form of research with which it has the greatest experience.Active Involvement of Consumers in CERThe 2009 ARRA created a means for coordinating federal CER efforts: the Federal Coordinating Council. Seeking to complement the functions of the Council, the IOM committee focused on the potential contribution of consumers of health care to CER and recommended involving them at all levels of the national CER initiative. The recommendation is as follows: "The CER Program should fully involve consumers, patients and their caregivers in key aspects of CER, including strategic planning, priority setting, research proposal development, peer review, and dissemination."The concept that animates this recommendation is a radical but logical departure from past practice: When a principal aim of research is to inform decision makers, listen to the decision makers. Meaningful participation will require consumers to learn more about research, just as they will teach researchers about how to frame their investigations to satisfy the needs of consumers who are also decision makers. This approach has precedent. For example, the National Institute for Health and Clinical Effectiveness in the United Kingdom teaches decision making to consumers who participate in the institute's program.Methodology of CERRemembering that the purpose of CER is to inform decision making, consider the principal forms of clinical research. Systematic reviews of the literature summarize a body of evidence. Over the past 2 decades, they have become indispensable to expert panels that formulate practice guidelines or policymakers who make insurance coverage decisions. These "macro" decision makers are starting to use decision models for the same purposes (5). Two broad categories of research are the starting point for systematic reviews: observational research and randomized trials. Large established databases provide an opportunity to link current health care practices to the outcome of care. These databases are representative of critical groups often omitted from randomized trials, but they contain limited data; of note, they seldom specify the rationale for medical decisions. These observational research methods have many advantages—speed, real-world decisions, large numbers of decisions and outcomes, and low cost—but they cannot escape a key limitation: Characteristics of the patient that drive real-life clinical decisions may also influence clinical outcomes, leading to uncertainty about whether they or the intervention itself causes the outcomes. Overcoming the limitations of observational research is the most important frontier of research on study methods.To overcome these limitations, researchers often randomly assign patients to different interventions. This simple action eliminates much of the uncertainty that plagues the interpretation of observational research. However, we have too often used the conceptual elegance of randomization to answer the wrong questions. We ask, "Does this work?" when our readers want to know, "Is this better than that?" Thus, too often, randomized trials are so-called efficacy studies: They are designed to create near-ideal circumstances to see whether the intervention can possibly work. To this end, researchers exclude many types of patients from efficacy trials and often compare the intervention with a placebo. In these respects, efficacy trials are unlike the conditions that physicians and patients face in daily practice. Another article in this issue (4) describes several exciting new methods for making randomized trials more suitable for CER.The examples of observational research and new randomized trial methods are the basis for the IOM committee's strong recommendation that a national CER program should support research in the methods of clinical research.The Infrastructure for Observational ResearchThe committee believes that very large collections of the electronic records of patients can be a valuable resource for CER. Using these data sets, it is possible to compare the outcomes of using several effective interventions in a population that is representative of daily care, 2 features that are aligned with the goals of CER. The great numbers of patients in these data sets also makes it possible to study subgroups with precision and perhaps identify key predictors of response to an intervention, both of which would facilitate decision making at the individual and population level. However, using these data sets will require researchers to surmount several technical barriers, to say nothing of the analytic challenges of making crisp inferences from observational data.The committee's recommendation is worded as follows: "The CER Program should help to develop large-scale, clinical and administrative data networks to facilitate better use of data and more efficient ways to collect new data to yield CER findings."The committee recognized the need to protect patients' privacy; it therefore called for development of methods for linking patient-level data from multiple sources. It also recognized that participating in a data network requires institutions to invest in the technical requirements and make a commitment to provide high-quality data. Accordingly, the national CER program should create incentives to participate and rewards for good performance.The CER WorkforceDepending on the scale of the public's investment in CER, the workforce for CER may prove to be inadequate. The committee recommended that a national CER program invest in developing a research workforce with the requisite skills for CER. The national CER program should develop a strategic plan for meeting the CER workforce needs and invest in career development support at every level from graduate school to mid-career.Translating Research Findings into Patient CareThe main reason for public interest in CER is the belief that better information will result in better health outcomes and more effective use of resources. Comparative effectiveness research is an important cog in a larger mechanism for promoting better health care practices. The committee believes that a national CER program will be perceived as successful if it funds research that improves the uptake of new knowledge and its translation into better decision making. Indeed, many of the high-ranked research topics addressed questions about the best methods to translate research results into action. In the end, whether CER will realize its potential will depend on other features of health care reform legislation, the profession's ability to mobilize the best in its members, and the ability of individual patients to engage their physicians in a dialogue about their own care.Personal ReflectionsResearch that informs clinical decisions is everywhere, yet a national research initiative to improve decision making by patients and their physicians is a novel concept. We should all hope that the program goes well beyond the goal of providing better information for decision making. That mission is important. Indeed, it is the principal motivation for a national CER program. However, we believe that it is not enough. Somehow, and in concert with national professional organizations and local opinion leaders, a national program should promote the professional ethos that places the interests of patients and the larger community above all other considerations (6). Together, a national initiative that engages the medical profession and patients can justify a large public investment in CER.References1. Wilensky GR. Developing a center for comparative effectiveness information. Health Aff (Millwood). 2006;25:w572-85. [PMID: 17090555] CrossrefMedlineGoogle Scholar2. Eden J, Wheatley B, McNeil B, Sox H, eds. Institute of Medicine. Knowing What Works in Health Care: A Roadmap for the Nation. Washington, DC: National Academies Pr; 2008. Google Scholar3. Ratner R, Eden J, Wolman D, Greenfield S, Sox H, eds. Institute of Medicine. Initial national priorities for comparative effectiveness research. Washington, DC: National Academies Pr; 2009. Google Scholar4. Luce BR, Kramer JM, Goodman SN, Connor JT, Tunis S, Whicher D, et al. Rethinking randomized clinical trials for comparative effectiveness research: the need for transformational change. Ann Intern Med. 2009;151:206-9. LinkGoogle Scholar5. Zauber AG, Lansdorp-Vogelaar I, Knudsen AB, Wilschut J, van Ballegooijen M, Kuntz KM. Evaluating test strategies for colorectal cancer screening: a decision analysis for the U.S. Preventive Services Task Force. Ann Intern Med. 2008;149:659-69. [PMID: 18838717] LinkGoogle Scholar6. Gawande A. The cost conundrum. The New Yorker. 2009;1 June. Accessed at www.newyorker.com/reporting/2009/06/01/090601fa_fact_gawande on 23 June 2009. Google ScholarAppendix: Recommendations to Establish the Infrastructure for a Successful CER InitiativeRecommendation 5: The HHS Secretary should establish a mechanism—such as a coordinating advisory body—with the authority to strategize, organize, monitor, evaluate, and report on the implementation and impact of the CER Program.Recommendation 6: The CER Program should fully involve consumers, patients and their caregivers in key aspects of CER, including strategic planning, priority setting, research proposal development, peer review, and dissemination.The CER Program should develop strategies to reach out to, engage, support, educate, and, as necessary, prepare consumers, patients and their caregivers for leadership roles in these activities.The CER Program should also encourage broad participation in CER research in order to create a representative evidence base that could help identify health disparities and inform decisions by patients in special population groups.Recommendation 7: The CER Program should devote sufficient resources to research and innovation in the methods of CER, including the development of methodological guidance for CER study design such as the appropriate use of observational data and more informative, practical, and efficient clinical trials.Recommendation 8: The CER Program should help to develop large-scale, clinical and administrative data networks to facilitate better use of data and more efficient ways to collect new data to yield CER findings.The CER Program should ensure that CER researchers and institutions implement privacy and security protections in all their activities.The CER Program should support the development of methodologies for linking patient-level data from multiple sources, and for analyzing the resulting data for CER.The CER Program should promote widespread participation and provide incentives to data holders to participate in CER.Recommendation 9: The CER Program should develop and support the workforce for CER to ensure the nation's capacity to carry out the CER research mission. Important next steps include:Development of a strategic plan for research workforce development.Long-term, sufficient funding for career development including expanding grants for graduate and postgraduate training opportunities in comparative effectiveness methods as well as career development grants and mid-career merit awards.Recommendation 10: The CER Program should promote rapid adoption of CER findings and conduct research to identify the most effective strategies for disseminating new and existing CER findings to health care professionals, consumers, patients, and their caregivers and for helping them to implement these results in daily clinical practice. Comments0 CommentsSign In to Submit A Comment Author, Article, and Disclosure InformationAuthors: Harold C. Sox, MD, Editor; Sheldon Greenfield, MDAffiliations: From the American College of Physicians, Philadelphia, Pennsylvania, and University of California, Irvine, Irvine, California.Disclosures: None disclosed.Corresponding Author: Harold C. Sox, MD, American College of Physicians, 190 N. Independence Mall West, Philadelphia, PA 19106.Current Author Addresses: Dr. Sox: American College of Physicians, 190 N. Independence Mall West, Philadelphia, PA 19106.Dr. Greenfield: Center for Health Policy Research, University of California, Irvine, 111 Academy, Suite 220, Irvine, CA 92697. PreviousarticleNextarticle Advertisement FiguresReferencesRelatedDetailsSee AlsoRethinking Randomized Clinical Trials for Comparative Effectiveness Research: The Need for Transformational Change Bryan R. Luce , Judith M. Kramer , Steven N. Goodman , Jason T. Connor , Sean Tunis , Danielle Whicher , and J. Sanford Schwartz Metrics Cited byTranslational ScienceSequential and Comparative Evaluation of Pain Treatment Effectiveness Response (SCEPTER), a pragmatic trial for conservative chronic low back pain treatmentIntegrated rapid-cycle comparative effectiveness trials using flexible point of care randomisation in electronic health record systemsComputer clinical decision support that automates personalized clinical care: a challenging but needed healthcare delivery strategySlow recruitment in the HIMALAIA study: lessons for future clinical trials in patients with delayed cerebral ischemia after aneurysmal subarachnoid hemorrhage based on feasibility dataAn evaluation of stakeholder engagement in comparative effectiveness research: lessons learned from SWOG S1415CDImpact of Missing Data on Identifying Risk Factors for Postoperative Complications in Hand SurgeryImpact of energy saving on the financial performance of industrial enterprises in China: An empirical analysis based on propensity score matchingThe Pain Outcomes Comparing Yoga vs. Structured Exercise (POYSE) Trial in Veterans With Fibromyalgia: Study Design and MethodsComparative efficacy trials in inflammatory bowel disease: current and future implications for practiceDevelopment of an Efficient ETL Technique for Data WarehousesEthical care requires pragmatic care research to guide medical practice under uncertaintyRisk factors for progression or death after first-line platinum-based chemotherapy in real-world patients in the USA with ovarian cancer from 2011 to 2018Optimizing the Start Time of Biologics in Polyarticular Juvenile Idiopathic Arthritis: A Comparative Effectiveness Study of Childhood Arthritis and Rheumatology Research Alliance Consensus Treatment PlansPragmatic Considerations in Incorporating Stakeholder Engagement Into a Palliative Care Transitions StudyInforming Healthcare Decisions with Observational Research Assessing Causal Effect: An American Thoracic Society Statement Not Ready for ImplementationDesign and methods of the Care Management for the Effective Use of Opioids (CAMEO) trialPostextubation Respiratory Support: Is High-Flow Oxygen Therapy the Answer?*A comparison of parametric propensity score‐based methods for causal inference with multiple treatments and a binary outcomeFactors associated with participation over time in the Avon Longitudinal Study of Parents and Children: a study using linked education and primary care dataEvidence-Based Global HealthEvidence-Based Global HealthPlayers in the Development of Personalized MedicineMatching comparisons of therapeutic efficacy suggest better clinical outcomes for patients treated with peginterferon beta-1a than with glatiramer acetateOnline tools to synthesize real-world evidence of comparative effectiveness research to enhance formulary decision makingTreatment Preferences for Chronic Low Back Pain: Views of Veterans and Their ProvidersAutomated Identification of Common Disease-Specific Outcomes for Comparative Effectiveness Research Using ClinicalTrials.gov: Algorithm Development and Validation StudyAssessing the practice of data quality evaluation in a national clinical data research network through a systematic scoping review in the era of real-world dataA typology of useful evidence: approaches to increase the practical value of intervention researchReal-world experiences with generating real-world evidence: Case Studies from PCORI's pragmatic clinical Studies programAdoption of Innovation in Herpes Simplex Virus KeratitisStrategies for Evaluation Research: InterventionsCystic fibrosis drug trial design in the era of CFTR modulators associated with substantial clinical benefit: stakeholders' consensus viewComparing the effects of manual acupuncture, electroacupuncture, and transcutaneous electrical nerve stimulation on chronic tinnitus: a randomized controlled trialMissing Data in the National Surgical Quality Improvement Program Database: How Does It Affect the Identification of Risk Factors for Shoulder Surgery Complications?Time to treatment failure following initiation of fingolimod versus teriflunomide for multiple sclerosis: a retrospective US claims studyComparative Effectiveness ResearchOral health–related quality of life (OHRQoL) before and after endodontic treatment: a systematic reviewEffectiveness of the automated drug dispensing system: systematic review and meta-analysisTreatment destinations and visit frequencies for patients seeking medical treatment overseas from the United Arab Emirates: results from Dubai Health Authority reporting during 2009–2016Understanding Payer Perspectives on Value in the Use of Pharmaceuticals in the United StatesDoes evidence matter? Comparative effectiveness research and prescribing of Type 2 diabetes mellitus drugsBaseline Characteristics of Randomized Participants in the Glycemia Reduction Approaches in Diabetes: A Comparative Effectiveness Study (GRADE)Comparative effectiveness of a human fibroblast-derived dermal substitute and a viable cryopreserved placental membrane for the treatment of diabetic foot ulcersA Culture-Centered Community-Grounded Approach to Disseminating Health Information among African AmericansGenerating Real-World Tumor Burden Endpoints from Electronic Health Record Data: Comparison of RECIST, Radiology-Anchored, and Clinician-Anchored Approaches for Abstracting Real-World Progression in Non-Small Cell Lung CancerMeasuring the value of MRI: Comparative effectiveness & outcomes researchUnique aspects of clinical trials of invasive therapies for chronic painEvidence-based diabetes care for older people with Type 2 diabetes: a critical reviewA Comparative Effectiveness Trial to Reduce Burnout and Improve Quality of CareIntervention Comparative Effectiveness for Adult Cognitive Training (ICE-ACT) Trial: Rationale, design, and baseline characteristicsA transformative translational change programme to introduce genomics into healthcare: a complexity and implementation science study protocolAdvancing Clinical Research Through Natural Language Processing on Electronic Health Records: Traditional Machine Learning Meets Deep LearningClinic-Based Delivery of the Young Men's Health Project (YMHP) Targeting HIV Risk Reduction and Substance Use Among Young Men Who Have Sex with Men: Protocol for a Type 2, Hybrid Implementation-Effectiveness TrialUsing a society database to evaluate a patient safety collaborative: the Cardiovascular Surgical Translational StudyPatient centered research to improve community involvement (PaRTICIpate) in diabetes self-management: a conference series for developing collaborations between researchers, stakeholders, and patientsPatient Governance in a Patient-Powered Research Network for Adult Rheumatologic ConditionsPatient-reported quality of life progression in men with prostate cancer following primary cryotherapy, cyberknife, or active holistic surveillanceN-of-1 Randomized Intervention Trials in Health Psychology: A Systematic Review and Methodology CritiqueEthics of research methodology requires a methodology of research ethicsOptimal choice of prevention and cure under uncertainty on disease effect and cure effectivenessBayesian comparative effectiveness study of four consensus treatment plans for initial management of systemic juvenile idiopathic arthritis: FiRst-Line Options for Systemic juvenile idiopathic arthritis Treatment (FROST)Intravitreal aflibercept versus intravitreal ranibizumab in patients with age-related macular degeneration: a comparative effectiveness studyIntroduction to the Cancer Intervention and Surveillance Modeling Network (CISNET) Breast Cancer ModelsChoosing important health outcomes for comparative effectiveness research: An updated systematic review and involvement of low and middle income countriesComparative Effectiveness Research in Pediatric Respiratory Disease: Promise and PitfallsComparative effectiveness clinical trials to advance treatment of myasthenia gravisImproving cardiometabolic and mental health in women with gestational diabetes mellitus and their offspring: study protocol for MySweetHeart Trial , a randomised controlled trialRevising our approach to conducting therapeutic clinical research in COPDIncreasing uptake of comparative effectiveness and patient-centered outcomes research among stakeholders: insights from conference discussionDesign characteristics of the Corrona Japan rheumatoid arthritis registryOverview of Health Complexity and the Interaction Between Physical and Behavioral Health ConditionsAutomated Contradiction Detection in Biomedical LiteratureBiomedical Research in the 21st Century: Multiple Challenges in Resolving Reproducibility IssuesDesign of Clinical Trials and StudiesHow do geometry-related parameters influence the clinical performance of orthodontic mini-implants? A systematic review and meta-analysisReply to: comparative effectiveness medicines research cannot assess efficacyDynamic-ETL: a hybrid approach for health data extraction, transformation and loadingPilot study comparing the Childhood Arthritis & Rheumatology Research Alliance (CARRA) systemic Juvenile Idiopathic Arthritis Consensus Treatment PlansComparative Effectiveness Research in Integrative OncologyComparative Toxicity and Effectiveness of Trastuzumab-Based Chemotherapy Regimens in Older Women With Early-Stage Breast CancerTraining in patient-centered outcomes research for specific researcher communitiesComparative effectiveness of antihypertensive drugs in nondiabetic patients with hypertension: A population-based studyMicrobial Modulation of Stem Cells and Future Directions in Regenerative EndodonticsPractical implications of using real-world evidence (RWE) in comparative effectiveness research: learnings from IMI-GetRealMinimally invasive surgery and its impact on 30-day postoperative complications, unplanned readmissions and mortalityCoupling Timing of Interventions With Dose to Optimize Plasticity and Participation in Pediatric Neurologic PopulationsCost-effectiveness Analysis Appraisal and Application: An Emergency Medicine PerspectiveUsing Procedure Codes to Define Radiation Toxicity in Administrative DataPotential Pitfalls of Reporting and Bias in Observational Studies With Propensity Score Analysis Assessing a Surgical ProcedureIranian clinical trials: An analysis of registered trials in International Clinical Trial Registry Platform (ICTRP)Patient, caregiver and clinician use of comparative effectiveness research findings in care decisions: results from a national studyPatient, caregiver and clinician views on engagement in comparative effectiveness researchHealth researcher views on comparative effectiveness research and research engagementQuerying stakeholders to inform comparative effectiveness researchCross-design synthesis for extending the applicability of trial evidence when treatment effect is heterogenous: Part II. Application and external validationPublication of comparative effectiveness research has not increased in high-imp