Reporting of systematic reviews and meta-analysis of observational studies
Authors: Mical Paul, Mariska M. Leeflang
Journal: Clinical Microbiology and Infection (2020)

Although randomized controlled trials (RCTs) are generally seen as providing more certainty about the evidence of an effect, many interventions in infectious diseases are examined in observational studies. These include rare cases of treatment that cannot be examined in a RCT (e.g. appropriate vs. inappropriate empirical antibiotic treatment) [[1]Neemann K. Olateju E.K. Izevbigie N. Akaba G. Olanipekun G.M. Richard J.C. et al.Neonatal outcomes associated with maternal recto-vaginal colonization with extended-spectrum β-lactamase producing Enterobacteriaceae in Nigeria: a prospective, cross-sectional study.Clin Microbiol Infect. 2020; 26: 463-469Abstract Full Text Full Text PDF PubMed Scopus (2) Google Scholar,[2]Giannella M. Bartoletti M. Campoli C. Rinaldi M. Coladonato S. Pascale R. et al.The impact of carbapenemase-producing Enterobacteriaceae colonization on infection risk after liver transplantation: a prospective observational cohort study.Clin Microbiol Infect. 2019; 25: 1525-1531Abstract Full Text Full Text PDF PubMed Scopus (16) Google Scholar], complex interventions that are difficult to examine in RCTs (e.g. infection control or antibiotic stewardship) [[3]Onorato L. Macera M. Calò F. Monari C. Russo F. Iovene M.R. et al.The effect of an antimicrobial stewardship programme in two intensive care units of a teaching hospital: an interrupted time series analysis.Clin Microbiol Infect. 2020; 26 (e1–6): 782Abstract Full Text Full Text PDF PubMed Scopus (8) Google Scholar,[4]Biehl L.M. Higgins P. Wille T. Peter K. Hamprecht A. Peter S. et al.Impact of single-room contact precautions on hospital-acquisition and transmission of multidrug-resistant Escherichia coli: a prospective multicentre cohort study in haematological and oncological wards.Clin Microbiol Infect. 2019; 25: 1013-1020Abstract Full Text Full Text PDF PubMed Scopus (9) Google Scholar], comparative effective research and quality improvement programmes (comparisons between commonly used, approved, antibiotics, monotherapy vs. combinations or treatment durations) for which resources for RCTs are lacking [5Wiggers J.B. Sehgal P. Pinto R. MacFadden D. Daneman N. The association of adequate empirical treatment and time to recovery from bacteraemic urinary tract infections: a retrospective cohort study.Clin Microbiol Infect. 2019; 25: 1253-1258Abstract Full Text Full Text PDF PubMed Scopus (10) Google Scholar, 6Tauzin M. Ouldali N. Lévy C. Béchet S. Cohen R. Caeymaex L. Combination therapy with ciprofloxacin and third-generation cephalosporin versus third-generation cephalosporin monotherapy in Escherichia coli meningitis in infants: a multicentre propensity score-matched observational study.Clin Microbiol Infect. 2019; 25: 1006-1012Abstract Full Text Full Text PDF PubMed Scopus (6) Google Scholar, 7Ko J.H. Lim J.U. Choi J.Y. Oh H.S. Yoo H. Jhun B.W. et al.Early cidofovir administration might be associated with a lower probability of respiratory failure in treating human adenovirus pneumonia: a retrospective cohort study.Clin Microbiol Infect. 2020; 26 (e9–14): 646Abstract Full Text Full Text PDF PubMed Scopus (7) Google Scholar, 8Gentry C.A. Nguyen P.K. Thind S. Kurdgelashvili G. Skrepnek G.H. Williams 2nd, R.J. Fidaxomicin versus oral vancomycin for severe Clostridium difficile infection: a retrospective cohort study.Clin Microbiol Infect. 2019; 25: 987-993Abstract Full Text Full Text PDF PubMed Scopus (10) Google Scholar, 9Crespillo-Andújar C. López-Vélez R. Trigo E. Norman F. Díaz-Menéndez M. Monge-Maillo B. et al.Comparison of the toxicity of two treatment schemes with benznidazole for chronic Chagas disease: a prospective cohort study in two Spanish referral centres.Clin Microbiol Infect. 2020; 26 (e1–4): 384Abstract Full Text Full Text PDF PubMed Scopus (5) Google Scholar, 10Abbas M. Rossel A. de Kraker M.E.A. von Dach E. Marti C. Emonet S. et al.Association between treatment duration and mortality or relapse in adult patients with Staphylococcus aureus bacteraemia: a retrospective cohort study.Clin Microbiol Infect. 2020; 26: 626-631Abstract Full Text Full Text PDF PubMed Scopus (6) Google Scholar], or studies rapidly launched during outbreaks, such as the coronavirus disease 2019 (COVID-19) pandemic [11Lian N. Xie H. Lin S. Huang J. Zhao J. Lin Q. Umifenovir treatment is not associated with improved outcomes in patients with coronavirus disease 2019: a retrospective study.Clin Microbiol Infect. 2020; 26: 917-921Abstract Full Text Full Text PDF PubMed Scopus (102) Google Scholar, 12Rinott E. Kozer E. Shapira Y. Bar-Haim A. Youngster I. Ibuprofen use and clinical outcomes in COVID-19 patients.Clin Microbiol Infect. 2020; 26 (e5–7): 1259Abstract Full Text Full Text PDF Scopus (42) Google Scholar, 13Fiolet T. Guihur A. Rebeaud M. Mulot M. Peiffer-Smadja N. Mahamat-Saleh Y. Effect of hydroxychloroquine with or without azithromycin on the mortality of COVID-19 patients: a systematic review and meta-analysis.Clin Microbiol Infect. 2021; 27: 19-27Abstract Full Text Full Text PDF PubMed Scopus (91) Google Scholar]. Summarizing these in systematic reviews is appealing; however, systematic reviews of observational studies suffer from the same shortcomings as primary observational studies when causal inferences need to be made. Consistent errors in observational studies may even be magnified by narrowing the confidence intervals through meta-analysis. In this editorial note, we provide our recommendations and expectations for systematic reviews and meta-analyses of observational studies examining an exposure or intervention. Guidance on performance and reporting of such studies has been published, and we recommend following these [[14]Dekkers O.M. Vandenbroucke J.P. Cevallos M. Renehan A.G. Altman D.G. Egger M. COSMOS-E: guidance on conducting systematic reviews and meta-analyses of observational studies of etiology.PLoS Med. 2019; 16e1002742Crossref PubMed Scopus (90) Google Scholar,[15]Liberati A. Altman D.G. Tetzlaff J. Mulrow C. Gøtzsche P.C. Ioannidis J.P.A. et al.The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration.Ann Intern Med. 2009; 151: W65-W94Crossref PubMed Scopus (3516) Google Scholar]. We explain the specific pitfalls of such studies and what we require from authors of systematic reviews of observational studies. As for any research, systematic reviews of observational studies should start with a protocol. We expect to see the protocol for the systematic review, preferably published in advance in PROSPERO (International Prospective Register of Systematic Reviews) or other database. This protocol should start with the review question, containing Patients, Intervention/Exposure, Comparator, Outcomes, Study design (PICOS or PECOS). The PECOS format has been proposed to distinguish between the active intervention in a RCT and the observation of an 'exposure' in observational studies [[14]Dekkers O.M. Vandenbroucke J.P. Cevallos M. Renehan A.G. Altman D.G. Egger M. COSMOS-E: guidance on conducting systematic reviews and meta-analyses of observational studies of etiology.PLoS Med. 2019; 16e1002742Crossref PubMed Scopus (90) Google Scholar]. In interventional trials, the dose and duration of the treatments are fixed by protocol, and restrictions on patient inclusion on the basis of time from disease onset are defined. Similarly, in observational studies, there should be criteria to define exposure [[16]Paul M. Leibovici L. Observational studies examining patient management in infectious diseases.Clin Microbiol Infect. 2017; 23: 127-128Abstract Full Text Full Text PDF PubMed Scopus (12) Google Scholar], and systematic reviews can and probably should restrict inclusion criteria to observational studies examining the exposure of interest. For example, in a systematic review of hydroxychloroquine therapy for COVID-19, rather than defining for inclusion all studies examining hydroxychloroquine among patients with COVID-19, an informative review can restrict inclusion to studies where hydroxychloroquine therapy was initiated early after diagnosis, examining an appropriate dose, for a minimal duration of treatment. Too tight a definition (e.g. one specific hydroxychloroquine dose) might result in zero studies included, but too broad a definition will result in less meaningful estimates and heterogeneity. Alternatively, this can be investigated in sensitivity or subgroup analyses, and would address the time hydroxychloroquine was started relative to the diagnosis of COVID-19, acceptable dosing and the minimal duration that would be considered as treatment. The study designs included should be addressed. Inclusion of RCTs should always be considered. The types of observational studies considered and restrictions on observational study design for inclusion in the review should be specified. Possible exclusion criteria based on observational study design may include small sample sizes, before-and-after studies or lack of adjusted analysis. An important feature of the protocol for a systematic review of observational studies is the need to tailor the risk of bias assessment tools to the review. We recommend the Cochrane tool for risk of bias of nonrandomized studies of interventions, ROBINS-I [[17]Sterne J.A.C. Hernán M.A. Reeves B.C. Savović J. Berkman N.D. Viswanathan M. et al.ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.BMJ. 2016; 355: i4919Crossref PubMed Scopus (3180) Google Scholar]. ROBINS-I is a domain-based assessment tool with signaling questions for each domain. It is comparable to the Cochrane ROB-2 tool for RCTs, which may be useful when including both RCTs and non-RCTs in the systematic review. The Newcastle-Ottawa scale is an alternative tool [[18]Wells G.A. Shea B. O'Connell D. Peterson J. Welch V. Losos M. et al.The Newcastle-Ottawa scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses.2013http://www.ohri.ca/programs/clinical_epidemiology/oxford.aspGoogle Scholar]. Both require advance planning and adaptation to the review question to enable assessment of the 'confounding' and 'deviations from intended interventions' domains (https://www.riskofbias.info/welcome/home/current-version-of-robins-i/robins-i-tool-2016). The important risk factors for the outcomes and variables presumed to affect both the presence of the exposure and the outcome need to be defined. Review authors should assess whether these predefined variables were examined when analysing the association between exposure and outcome. For example, a systematic review assessing appropriate vs. inappropriate empirical antibiotic therapy for sepsis requires the assessment of age, any sepsis severity score and the place of acquisition as potential confounders in primary studies [[19]Paul M. Shani V. Muchtar E. Kariv G. Robenshtok E. Leibovici L. Systematic review and meta-analysis of the efficacy of appropriate empiric antibiotic therapy for sepsis.Antimicrob Agents Chemother. 2010; 54: 4851-4863Crossref PubMed Scopus (405) Google Scholar]. For a study of hydroxychloroquine and COVID-19, confounding domains should probably include age, comorbidities and disease stage/severity. In addition, a list of potential cointerventions that could be different between intervention groups and that could affect outcomes needs to be specified in advance in order to compare their use between the treatment groups and to score studies on bias related to the interventions. For example, corticosteroids and availability of intensive support would be important cointerventions in the observational assessment of antiviral therapy for COVID-19, especially when comparing data among different centres. Well-conducted risk-of-bias assessment of observational studies within a systematic review is highly informative and could actually be the focus of the systematic review, as was the case in a systematic review of studies comparing monotherapy to combination therapy for carbapenem-resistant Gram-negative infections, where the main objective was to highlight the bias in existing studies to enable an evidence-based statement that these studies should not direct practice [[20]Zusman O. Altunin S. Koppel F. Dishon Benattar Y. Gedik H. Paul M. Polymyxin monotherapy or in combination against carbapenem-resistant bacteria: systematic review and meta-analysis.J Antimicrob Chemother. 2017; 72: 29-39Crossref PubMed Scopus (84) Google Scholar]. The main challenge in meta-analyses of observational studies is pooling the adjusted results. Pooling the crude unadjusted numbers is simple, but it is not more informative than the univariate analysis in the original observational studies. Primary observational studies typically use different methods for adjustment of the analysis, such as multivariable regression analyses, propensity scores that can be used in several ways or instrumental variable analyses. Results from these analyses should be used in systematic reviews of observational studies. However, a commonly encountered problem is that depending on inclusion criteria of the review, only some of the included studies may report the adjusted association for the exposure variable of interest. Included studies might not include an adjusted analysis at all; the authors might have performed an adjusted analysis without analysing the exposure of interest, or they might not include quantitative results for factors that were not retained or that were not statistically significant in the final regression model. We encountered all these possibilities in a systematic review and meta-analysis on appropriate vs. inappropriate empirical antibiotic treatment and mortality in sepsis. Because the studies that did not find an association between inappropriate empirical antibiotic treatment and mortality in the univariate analysis did not include it in the adjusted risk model or did not quantitatively report the results of these adjusted analyses, these studies were highly skewed towards a lack of association [[19]Paul M. Shani V. Muchtar E. Kariv G. Robenshtok E. Leibovici L. Systematic review and meta-analysis of the efficacy of appropriate empiric antibiotic therapy for sepsis.Antimicrob Agents Chemother. 2010; 54: 4851-4863Crossref PubMed Scopus (405) Google Scholar]. In this case, restricting the analysis to studies reporting the adjusted odds ratio for inappropriate empirical antibiotic treatment and mortality would have strongly biased this analysis in favour of appropriate empirical antibiotic treatment. A standard solution to all these obstacles is not available. (A simple solution would be not to perform a systematic review of observational studies.) Several options and the limitations of each are presented in Table 1. Furthermore, even for a focused question, the studies will use different variables and variable definitions for adjustment. To address this source of variability in the adjusted estimates of association, review authors may limit the meta-analysis to studies adjusting for confounders that were defined as being important in the tailored ROBINS-I. Additionally, sensitivity or subgroup analyses by optimal, nonoptimal or no adjustment can be performed [[14]Dekkers O.M. Vandenbroucke J.P. Cevallos M. Renehan A.G. Altman D.G. Egger M. COSMOS-E: guidance on conducting systematic reviews and meta-analyses of observational studies of etiology.PLoS Med. 2019; 16e1002742Crossref PubMed Scopus (90) Google Scholar]. Finally, adjusted results of the original studies might be reported as odds ratios, hazard ratios or risk ratios with different dispersion measures (e.g. 95% confidence intervals, standard errors). A meta-analysis should pool the same type of measure.Table 1Meta-analysis of adjusted associations from observational studiesAnalysis methodLimitationsInclude only studies that reported adjusted associationEmpty review; selected set of studies most likely showing significance for exposureUse adjusted OR when available and crude, unadjusted result otherwiseThis analysis may suffer from confoundingUse uniform null effect estimate (OR = 1), with standard error depending on study size/event rate for studies not reporting adjusted estimateProbable bias towards null associationUse other uniform effect estimate (e.g. pooled univariate OR), with standard error depending on study size/event rate for studies not reporting adjusted estimateDepending on study question, this analysis may suffer from bias due to confoundingRequest raw data from study authors or ask authors to conduct adjusted analysis and provide adjusted associationOptimal solution that will allow possible homogeneity of variables used for adjustment in different studiesOR, odds ratio. Open table in a new tab OR, odds ratio. The optimal strategy will probably be to conduct sensitivity analyses using different assumptions. None of these sensitivity analyses may be without bias, but if the bias works in different directions, this may be informative. Of six systematic review including observational studies published recently in CMI [[13]Fiolet T. Guihur A. Rebeaud M. Mulot M. Peiffer-Smadja N. Mahamat-Saleh Y. Effect of hydroxychloroquine with or without azithromycin on the mortality of COVID-19 patients: a systematic review and meta-analysis.Clin Microbiol Infect. 2021; 27: 19-27Abstract Full Text Full Text PDF PubMed Scopus (91) Google Scholar,21Bellos I. Karageorgiou V. Pergialiotis V. Perrea D.N. Acute kidney injury following the concurrent administration of antipseudomonal β-lactams and vancomycin: a network meta-analysis.Clin Microbiol Infect. 2020; 26: 696-705Abstract Full Text Full Text PDF PubMed Scopus (9) Google Scholar, 22Buckley B.S. Henschke N. Bergman H. Skidmore B. Klemm E.J. Villanueva G. et al.Impact of vaccination on antibiotic usage: a systematic review and meta-analysis.Clin Microbiol Infect. 2019; 25: 1213-1225Abstract Full Text Full Text PDF PubMed Scopus (18) Google Scholar, 23Ko J.H. Kang C.I. Cornejo-Juárez P. Yeh K.M. Wang C.H. Cho S.Y. et al.Fluoroquinolones versus trimethoprim–sulfamethoxazole for the treatment of Stenotrophomonas maltophilia infections: a systematic review and meta-analysis.Clin Microbiol Infect. 2019; 25: 546-554Abstract Full Text Full Text PDF PubMed Scopus (24) Google Scholar, 24Rodrigo C. Weeratunga P. Fernando S.D. Rajapakse S. Amphotericin B for treatment of visceral leishmaniasis: systematic review and meta-analysis of prospective comparative clinical studies including dose-ranging studies.Clin Microbiol Infect. 2018; 24: 591-598Abstract Full Text Full Text PDF PubMed Scopus (11) Google Scholar, 25Saha S. Tariq R. Tosh P.K. Pardi D.S. Khanna S. Faecal microbiota transplantation for eradicating carriage of multidrug-resistant organisms: a systematic review.Clin Microbiol Infect. 2019; 25: 958-963Abstract Full Text Full Text PDF PubMed Scopus (26) Google Scholar], only two addressed the adjusted results of primary observational studies (presenting these without a meta-analysis in a review associating vaccines with antibiotic use [[22]Buckley B.S. Henschke N. Bergman H. Skidmore B. Klemm E.J. Villanueva G. et al.Impact of vaccination on antibiotic usage: a systematic review and meta-analysis.Clin Microbiol Infect. 2019; 25: 1213-1225Abstract Full Text Full Text PDF PubMed Scopus (18) Google Scholar] and using multivariate odds ratios when available and univariate results otherwise in a meta-analysis of hydroxychloroquine used to treat COVID-19 [[13]Fiolet T. Guihur A. Rebeaud M. Mulot M. Peiffer-Smadja N. Mahamat-Saleh Y. Effect of hydroxychloroquine with or without azithromycin on the mortality of COVID-19 patients: a systematic review and meta-analysis.Clin Microbiol Infect. 2021; 27: 19-27Abstract Full Text Full Text PDF PubMed Scopus (91) Google Scholar]). A review of fluoroquinolones vs. trimethoprim/sulfamethoxazole used to treat Stenotrophomonas maltophilia infections reported a lack of adjusted results and pooled univariate results [[23]Ko J.H. Kang C.I. Cornejo-Juárez P. Yeh K.M. Wang C.H. Cho S.Y. et al.Fluoroquinolones versus trimethoprim–sulfamethoxazole for the treatment of Stenotrophomonas maltophilia infections: a systematic review and meta-analysis.Clin Microbiol Infect. 2019; 25: 546-554Abstract Full Text Full Text PDF PubMed Scopus (24) Google Scholar]. We would like to see both the crude results and the adjusted effect estimate presented. A comparison between the nonadjusted and adjusted associations can attest to the degree of confounding. Similarly, in systematic reviews including both RCTs and observational studies, their meta-analysis should be explicitly separated. If you believe that a meta-analysis of crude, unadjusted results is valid, then you must convince readers that confounding is negligible in this particular review. The complicated nature of an optimal meta-analysis of adjusted associations from observational studies limits the scope of a systematic review of observational studies typically to a single outcome of interest. Certainty of the overall evidence presented in the systematic review can be determined using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework, as for RCTs [[26]Schünemann H.J. Cuello C. Akl E.A. Mustafa R.A. Meerpohl J.J. Thayer K. et al.GRADE guidelines, 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence.J Clin Epidemiol. 2019; 111: 105-114Abstract Full Text Full Text PDF PubMed Scopus (137) Google Scholar]. Because risk of bias will rarely, if ever, be low with observational studies, the certainty of the evidence will often be automatically downgraded to moderate, low or very low certainty of the evidence. GRADE allows upgrading the certainty of the evidence of observational studies when all confounding factors (including unobserved, residual confounding) work in the opposite direction of the observed association. This requires knowledge of the residual confounding and was impossible in the case of the systematic review on inappropriate empirical antibiotic treatment. Authors of systematic reviews of observational studies are advised to follow the GRADE-PRO handbook when performing the grading [[27]Schünemann H. Brożek J. Guyatt G. Oxman A. Handbook for grading the quality of evidence and the strength of recommendations using the GRADE approach.2013https://gdt.gradepro.org/app/handbook/handbook.htmlGoogle Scholar]. As for the Discussion and Conclusion sections, we will be strict on overoptimistic interpretation of the results of systematic reviews relying on observational studies. If the aim of a systematic review is to provide evidence about causal effects of interventions and exposures, then this should be made explicit. But at the same time, the conclusions will usually need to be cautious and include a warning about the observational nature of the presented evidence. In summary, performing a systematic review of observational studies requires knowledge of the review field, advanced understanding of observational study design and analysis, good planning and an understanding that the evidence summarized will most probably not reach high quality. We present in Table 2 subheadings for the Methods section that we expect to see in systematic reviews of observational studies in CMI; we expect judicious analysis of data from observational studies.Table 2Items to be reported in Methods section of systematic reviews of observational studiesItemExplanationDesignPresent as systematic review, with or without meta-analysis and general type of studies considered (observational, with or without RCTs).PECOS: PatientsDescribe targeted disease and patient populationPECOS: Exposure/interventionMinimal requirements for exposure/intervention definition should be defined but not too restrictive, considering that many observational studies do not define exposure well, and considering heterogeneity between studies' definitions.PECOS: ComparisonDefine requirements for nonexposed cohort. When including case–control studies, define whether drawing exposed and nonexposed subjects from same population is a requirement.PECOS: OutcomeDefine primary outcome of review. This will be the outcome summarized appropriately if possible through adjusted analysis. Other endpoints can be listed as secondary, addressing adverse events also.PECOS: Eligibility criteriaStudy eligibility criteria should be detailed. Restrictions on inclusion of studies by design features, without adjusted analysis, exclusion of certain types of adjustment or by sample size should be defined.Search strategyDatabases searched and search string adapted for each database should be presented (possibly as supplementary material). Study flowchart (in Results) should start transparently from results of described search strategy. Restrictions on study years, publication status or language should be avoided or justified.Risk of bias assessmentTool used for risk of bias assessment should be defined and its adaptation to review topic should be presented in supplementary material, including definition of relevant confounding domains and cointerventions.DataTypes of data to be extracted should be defined, including nonadjusted and adjusted outcomes. Data on specific confounders extracted and adjustment methods used in study are special to systematic reviews of observational studies.Data extractionMethods of data extraction and risk of bias assessment, including whether duplicate independent extraction was performed. Data extraction and risk of bias assessment of observational studies are more complicated than in RCTs; a duplicate process with a consensus strategy is highly recommended.AnalysisIf performing a meta-analysis, univariate and adjusted analyses must be addressed, including methods of handing different adjustment methods and studies that did not report an adjusted analysis. Describe approach of pooling studies (fixed/random effects meta-analysis), heterogeneity assessment and planned subgroup and sensitivity analyses.GradingPreferably, quality of evidence summarized in review should be graded formally.RCT, randomized controlled trial. Open table in a new tab RCT, randomized controlled trial. Neither author reports a conflict of interest relevant to this editorial. The following is the Supplementary data to this article: Download .docx (.07 MB) Help with docx files Multimedia component 1