Systematic Reviews: Time to Address Clinical and Policy Relevance As Well As Methodological Rigor
Authors: Andreas Laupacis, Sharon Straus
Journal: Annals of Internal Medicine (2007)

Editorials21 August 2007Systematic Reviews: Time to Address Clinical and Policy Relevance As Well As Methodological RigorFREEAndreas Laupacis, MD and Sharon Straus, MDAndreas Laupacis, MDFrom Keenan Research Centre, Li Ka Shing Knowledge Institute of St. Michael's Hospital, Toronto, Ontario M5B 1W8, Canada, and University of Calgary, Calgary, Alberta T2N 2T9, Canada. and Sharon Straus, MDFrom Keenan Research Centre, Li Ka Shing Knowledge Institute of St. Michael's Hospital, Toronto, Ontario M5B 1W8, Canada, and University of Calgary, Calgary, Alberta T2N 2T9, Canada.Author, Article, and Disclosure Informationhttps://doi.org/10.7326/0003-4819-147-4-200708210-00180 SectionsAboutVisual AbstractPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinkedInRedditEmail Compared with other study designs, well-done randomized trials provide the most valid estimate of the benefits of health interventions because they minimize bias. Systematic reviews of randomized trials identify all studies that have addressed a particular question, and meta-analyses combine the results by using methods that minimize bias (1). Many consider systematic reviews to be the best source of information for making clinical and policy decisions.Such groups as the Cochrane Collaboration have established standards for the conduct and reporting of systematic reviews (2). In this issue, Shojania and colleagues (3) extend this tradition of subjecting research methods to close scrutiny. They describe how quickly the conclusions of 100 systematic reviews published in ACP Journal Club became out of date because of newly completed randomized trials that changed either the statistical significance or the magnitude of the summary treatment effect when they were added to the original review. Fifty percent of reviews were out of date within 5.5 years after publication and 23% were out of date within 2 years. Not surprisingly, the results were most likely to change for cardiovascular interventions (presumably because of the large number of trials in cardiovascular medicine) or if the trials in the original review were heterogeneous.Unfortunately, Shojania and colleagues' findings do not allow authors to accurately identify the time at which their specific systematic review will be out of date. Because outdated reviews can be misleading, we believe that researchers should update a literature search annually to identify studies that might change the results of the original review, unless a search of clinical trial registries assures them that no randomized trials addressing the topic of the review are ongoing. Because the updating search should use the same search strategy as the original review, updating should take relatively little time.In the past 2 decades, studies like that of Shojania and colleagues have improved the methodological quality of systematic reviews. We suggest that the time has come to devote similar energy to increasing the likelihood that the content and format of reviews are useful to a variety of decision makers. Despite advances in the conduct and reporting of systematic reviews, current evidence suggests that they are used less frequently by clinicians and policymakers than one might think. Patients use them even less frequently. A systematic review of the information-seeking behavior of physicians found that textbooks (many of which do not rely on evidence from systematic reviews) are still the most frequently consulted source of information, followed by advice from colleagues (4). Similarly, nurses and other health professionals seem to refer to evidence from systematic reviews infrequently in decision making (5, 6). Analyses of selected policymaking processes in Canada and at the World Health Organization found that evidence from systematic reviews was used infrequently (7, 8).Given that systematic reviews of randomized trials are less susceptible to bias than the opinions of experts and observational data, why do people fail to use them when making policy, clinical, or personal decisions? In trying to answer these questions, we consider the clinical relevance of the questions addressed by the systematic reviews, the format of systematic reviews, and the failure of authors to place their findings in a clinical context.Most systematic reviews address highly specific questions that interest the author. Authors seldom consult with policymakers, clinicians, patients, or health care managers about what's important to them. Clinicians often struggle with broad questions, such as, "What's the most effective treatment for insomnia?", rather than the narrow questions that researchers prefer, such as, "What's the evidence that benzodiazepines are effective for the treatment of insomnia?" Also, many important clinical questions are not addressed by systematic reviews. For example, a review of the Cochrane Collaboration databases for "dizziness" and "syncope" yielded no completed reviews of the management or investigation of these common symptoms.Systematic reviews are entirely dependent on the relevance of the randomized trials that have been conducted. Except for relatively rare pragmatic trials, the trials that most systematic reviews comprise involve highly selected patients who receive care from highly selected physicians, which in some circumstances raises important questions about their generalizability. (Actually, we feel that people sometimes worry too much about generalizablity. Clinical trials will never be done in every conceivable group of patients; therefore, clinicians must learn the art of applicability—that is, asking, "Is our patient really so different from patients included in the trials that we can't apply the results?"). Randomized trials generally underreport adverse events (9) and often do not study patients for a sufficient period to detect important side effects. These shortcomings can make them, and therefore systematic reviews, insufficient to drive clinical practice on their own.Systematic reviews are often off-putting to clinical readers. To meet the criteria established for the reporting of systematic reviews, most reviews are lengthy (Cochrane reviews are often longer than 30 pages), appear complicated to those not trained in systematic reviews, and take a long time to read and appraise. Thus, it is not surprising that clinicians use them infrequently and prefer more user-friendly formats for accessing evidence.Authors of systematic reviews tend to focus on documenting methodological rigor and fail to describe the policy or clinical context. Therefore, systematic reviews often do not provide crucial information for policymakers (10). Glasziou and Shepperd (11) found that fewer than 15% of systematic reviews published in Evidence-Based Medicine provide sufficient information about the intervention to allow clinicians or policymakers to implement it. Policymakers are interested in the governance, financial, and health care delivery strategies that are most likely to achieve their particular goals. Systematic reviews of studies that address these issues exist (see the Cochrane Effective Practice and Organization of Care group at www.epoc.uottawa.ca/), but the body of evidence is still quite small. In addition, systematic reviews generally do not consider cost-effectiveness or budget impact, which are crucial for policymakers.While recognizing these shortcomings of existing systematic reviews, how can we encourage clinicians and policymakers to use them? First, we could provide more exposure to systematic reviews (which, despite their length and apparent complexity, are relatively easy to interpret) during clinical training. Second, we could encourage more interaction among researchers, policymakers, and frontline clinicians, so that trialists might design pragmatic studies aimed at answering questions of interest to policymakers and clinicians. Third, policymakers must be more willing to commission randomized trials of policy-relevant questions, which can then be incorporated into systematic reviews. Fourth, the authors of systematic reviews should provide more contextual information or refer readers to easy-to-find articles that contain this information. Finally, authors must develop user-friendly formats so that busy clinicians and health care managers can use systematic reviews in real-time decision making. We suggest making systematic reviews available in 3 formats: 20-second, 2-minute, and 2-hour versions. The 20-second version is a 1-paragraph to 1-page bottom-line summary. The 2-minute version is a 2- to 3-page summary that focuses on details of validity and applicability (with sufficient information to facilitate implementation of the evidence). The 2-hour version contains all of the relevant methodological and contextual details.Because systematic reviews are comprehensive, they are undoubtedly a far more reliable basis for decision making than the traditional review article, with its selective citation of articles and the nonsystematic opinion of 1 or 2 experts. However, the traditional review has its virtues. The expert reviewers often go beyond the evidence from randomized trials to discuss issues that are of crucial importance to clinicians and policymakers. Moreover, the experts are usually willing to make recommendations in circumstances in which the evidence base is poor; in contrast, systematic reviews tend to call for more studies, and more than half of completed Cochrane reviews state that the evidence in support of the intervention is limited or poor (Glasziou P. Personal communication). We urgently need a new type of review. It would combine the scientific rigor of systematic reviews with the clinically nuanced contextualization and opinion of traditional review articles while clearly distinguishing between evidence and opinion (12).Andreas Laupacis, MDKeenan Research Centre, Li Ka Shing Knowledge Institute of St. Michael's HospitalToronto, Ontario M5B 1W8, CanadaSharon Straus, MDUniversity of CalgaryCalgary, Alberta T2N 2T9, CanadaReferences1. Oxman AD, Cook DJ, Guyatt GH. Users' guides to the medical literature. VI. How to use an overview. Evidence-Based Medicine Working Group. JAMA. 1994;272:1367-71. [PMID: 7933399] CrossrefMedlineGoogle Scholar2. The Cochrane Collaboration. The reliable source of evidence in health care. Accessed at www.cochrane.org/index.htm on 28 June 2007. Google Scholar3. Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med. 2007;147:224-33. LinkGoogle Scholar4. Dawes M, Sampson U. Knowledge management in clinical practice: a systematic review of information seeking behavior in physicians. Int J Med Inform. 2003;71:9-15. [PMID: 12909153] CrossrefMedlineGoogle Scholar5. Olade RA. Evidence-based practice and research utilization activities among rural nurses. J Nurs Scholarsh. 2004;36:220-5. [PMID: 15495490] CrossrefMedlineGoogle Scholar6. Kajermo KN, Nordström G, Krusebrant A, Lützén K. Nurses' experiences of research utilization within the framework of an educational programme. J Clin Nurs. 2001;10:671-81. [PMID: 11822518] CrossrefMedlineGoogle Scholar7. Lavis JN, Ross SE, Hurley JE, Hohenadel JM, Stoddart GL, Woodward CA, et al. Examining the role of health services research in public policymaking. Milbank Q. 2002;80:125-54. [PMID: 11933791] CrossrefMedlineGoogle Scholar8. Oxman AD, Lavis JN, Fretheim A. Use of evidence in WHO recommendations. Lancet. 2007;369:1883-9. [PMID: 17493676] CrossrefMedlineGoogle Scholar9. Ioannidis JP, Lau J. Completeness of safety reporting in randomized trials: an evaluation of 7 medical areas. JAMA. 2001;285:437-43. [PMID: 11242428] CrossrefMedlineGoogle Scholar10. Lavis JN, Posada FB, Haines A, Osei E. Use of research to inform public policymaking. Lancet. 2004;364:1615-21. [PMID: 15519634] CrossrefMedlineGoogle Scholar11. Glasziou P, Shepperd S. Inadequate descriptions of treatments in published reports: a common but correctable barrier to research uptake [Abstract]. Presented at the 36th Annual Scientific Meeting of the Society for Academic Primary Care, London, United Kingdom, 5 July 2007. Google Scholar12. Reviews: making sense of an often tangled skein of evidence [Editorial]. Ann Intern Med. 2005;142:1019-20. [PMID: 15968018] LinkGoogle Scholar Comments0 CommentsSign In to Submit A Comment Author, Article, and Disclosure InformationAuthors: Andreas Laupacis, MD; Sharon Straus, MDAffiliations: From Keenan Research Centre, Li Ka Shing Knowledge Institute of St. Michael's Hospital, Toronto, Ontario M5B 1W8, Canada, and University of Calgary, Calgary, Alberta T2N 2T9, Canada.Disclosures: None disclosed.Corresponding Author: Andreas Laupacis, MD, Li Ka Shing Knowledge Institute of St. Michael's Hospital, 80 Bond Street, Toronto, Ontario M5B 1W8, Canada.Current Author Addresses: Dr. Laupacis: Li Ka Shing Knowledge Institute of St. Michael's Hospital, 80 Bond Street, Toronto, Ontario M5B 1W8, Canada.Dr. Straus: Foothills Medical Centre, 1403 29th Street NW, Calgary, Alberta T2N 2T9, Canada. PreviousarticleNextarticle Advertisement FiguresReferencesRelatedDetailsSee AlsoHow Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis Kaveh G. Shojania , Margaret Sampson , Mohammed T. Ansari , Jun Ji , Steve Doucette , and David Moher Metrics Cited byA Human Machine Hybrid Approach for Systematic Reviews and Maps in International Development and Social Impact SectorsSystematic reviews should be at the heart of continuing medical educationAssessing the format and content of journal published and non-journal published rapid review reports: A comparative studyThe Decline of Pluralism in Medicine: Dissent Is WelcomeWhat should governments be doing to prevent diabetes throughout the life course?Engaging knowledge users in a systematic review on the comparative effectiveness of geriatrician-led models of care is possible: A cross-sectional survey using the Patient Engagement Evaluation ToolA comprehensive review of the clinical utility of and a combined analysis of the clozapine/norclozapine ratio in therapeutic drug monitoring for adult patientsThe Journal's MissionHarnessing Energies, Resolving Tensions: Acknowledging a Dual Heritage for Qualitative Evidence SynthesisA systematic review and combined analysis of therapeutic drug monitoring studies for oral paliperidoneAn evaluation of the comparative effectiveness of geriatrician-led comprehensive geriatric assessment for improving patient and healthcare system outcomes for older adults: a protocol for a systematic review and network meta-analysisApplicable or non-applicable: investigations of clinical heterogeneity in systematic reviewsDissemination of Evidence From Systematic Reviews Through Academic CME Providers: A Feasibility StudyUsing a systematic review in clinical decision making: a pilot parallel, randomized controlled trialFactors influencing citations to systematic reviews in skin diseases: a cross-sectional study through Web of Sciences and ScopusA usability study of two formats of a shortened systematic review for cliniciansConflicts of interest and critiques of the use of systematic reviews in policymaking: an analysis of opinion articlesAn iterative evaluation of two shortened systematic review formats for clinicians: a focus group studyConsensus-based recommendations for investigating clinical heterogeneity in systematic reviewsDevelopment of two shortened systematic review formats for cliniciansBibliometrics of systematic reviews: analysis of citation rates and journal impact factorsA systematic review of the non-operative treatment of nightstick fractures of the ulnaAssessing new strategies for TB diagnosis in low- and middle-income countriesClinical judgment in psychiatry. Requiem or reveille?Investigating clinical heterogeneity in systematic reviews: a methodologic review of guidance in the literatureInsertion torque and success of orthodontic mini-implants: A systematic reviewEvidence-Based Practice for Rehabilitation Professionals: Concepts and ControversiesBisphosphonates for periprosthetic bone loss after joint arthroplasty: a meta-analysis of 14 randomized controlled trialsAvaliação da utilidade clínica de novos testes diagnósticos em tuberculose: o papel dos ensaios clínicos pragmáticosCaries-preventive effect of glass ionomer and resin-based fissure sealants on permanent teeth: An update of systematic review evidenceAbsence of carious lesions at margins of glass-ionomer cement and amalgam restorations: An update of systematic review evidenceHow well are gic product labels related to current systematic review evidence?Pragmatic vs. explanatory: An adaptation of the PRECIS tool helps to judge the applicability of systematic reviews for daily practicePublicações na área de tisiologia no Jornal Brasileiro de Pneumologia entre 2004 e 2011: tipos de artigos, modelos de estudo, grau de evidência científica e impacto socialInterventions Encouraging the Use of Systematic Reviews in Clinical Decision-Making: A Systematic ReviewProfiling Dizziness in Older Primary Care Patients: An Empirical StudyTranslating Evidence from Systematic Reviews for Policy MakersAHRQ Series Paper 3: Identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the Effective Health-Care programWhy are Cochrane hepato-biliary reviews undervalued by physicians as an aid for clinical decision-making?Facilitating Evidence-Based Practice: Process, Strategies, and ResourcesNo short-cut in assessing trial quality: a case studyThe Policy Debate over Public Investment in Comparative Effectiveness ResearchSystematic review: impaired drug absorption related to the co-administration of antisecretory therapyThe Value of "Traditional" Reviews in the Era of Systematic ReviewingPoor reporting of search strategy and conflict of interest in over 250 narrative and systematic reviews of two biologic agents in arthritis: A systematic reviewMeta-analyse met gegevens van individuele patiënten: een utopisch toekomstscenario?Why would physicians undervalue reviews by the Cochrane Collaboration?What Have We Learned about Optimal Induction Therapy for Lupus Nephritis (III through V) from Randomized, Controlled Trials?: Table 1.Sistema de clasificación de los estudios en función de la evidencia científica. Dietética y Nutrición Aplicada Basadas en la Evidencia (DNABE): una herramienta para el dietista-nutricionista del futuroMotivating authors to update systematic reviews: practical strategies from a behavioural science perspectiveMeasuring the Care of the Patient in Context 21 August 2007Volume 147, Issue 4Page: 273-274KeywordsClinical trialsDecision makingGlobal healthHealth careInformation technologyInsomniaPatientsRandomized trialsSyncopeSystematic reviews ePublished: 21 August 2007 Issue Published: 21 August 2007 Copyright & PermissionsCopyright © 2007 by American College of Physicians. All Rights Reserved.PDF downloadLoading ...