Apples and oranges: international comparisons of COVID-19 observational studies in ICUs
Authors: Jonathan E Millar, Reinhard Busse, John F Fraser, Christian Karagiannidis, Daniel F McAuley
Journal: The Lancet Respiratory Medicine (2020)

Multiple observational cohorts describing the outcome of patients with COVID-19 from across the world have been published.1Ferrando C Suarez-Sipmann F Mellado-Artigas R et al.Clinical features, ventilatory management, and outcome of ARDS caused by COVID-19 are similar to other causes of ARDS.Intensive Care Med. 2020; (published online July 29.)https://doi.org/10.1007/s00134-020-06192-2Crossref PubMed Scopus (275) Google Scholar, 2Karagiannidis C Mostert C Hentschker C et al.Case characteristics, resource use, and outcomes of 10 021 patients with COVID-19 admitted to 920 German hospitals: an observational study.Lancet Respir Med. 2020; (published online July 28.)https://doi.org/10.1016/S2213-2600(20)30316-7Summary Full Text Full Text PDF PubMed Scopus (562) Google Scholar, 3Armstrong RA Kane AD Cook TM Outcomes from intensive care in patients with COVID-19: a systematic review and meta-analysis of observational studies.Anaesthesia. 2020; (published online June 30.)https://doi.org/10.1111/anae.15201Crossref Scopus (284) Google Scholar Typically, these studies have reported regional or national cohorts and no two countries have had the same experience. The reasons for these differences are complex and difficult to quantify. Nonetheless, to be able to draw meaningful inferences from these data we must tackle the issues associated with international comparison. Initial reports of outcomes in COVID-19, which emerged from China early in the pandemic, reported a range of mortality rates from intensive care units (ICUs) (0–78%).3Armstrong RA Kane AD Cook TM Outcomes from intensive care in patients with COVID-19: a systematic review and meta-analysis of observational studies.Anaesthesia. 2020; (published online June 30.)https://doi.org/10.1111/anae.15201Crossref Scopus (284) Google Scholar Case series from North America and Europe have been equally variable (with ICU mortality ranging 0–85%).3Armstrong RA Kane AD Cook TM Outcomes from intensive care in patients with COVID-19: a systematic review and meta-analysis of observational studies.Anaesthesia. 2020; (published online June 30.)https://doi.org/10.1111/anae.15201Crossref Scopus (284) Google Scholar A major issue has been the large number of patients in these series who had incomplete outcomes at the time of reporting, a factor that has commonly resulted in mortality being overestimated or underestimated. For example, in UK Intensive Care National Audit and Research Centre (ICNARC) data, early reports from March, 2020, estimated ICU mortality for COVID-19 to be 79 (48%) of 165 patients admitted, when 610 (79%) of 775 patients had an incomplete outcome (ie, were still in the ICU). In the latest report, from July 31, 2020, ICU mortality had decreased to 40% in 10 341 patients with complete outcomes.4Intensive Care National Audit and Research CentreICNARC report on COVID-19 in critical care 31 July 2020. Intensive Care National Audit and Research Centre, LondonJuly 31, 2020Google Scholar In the appendix (pp 1–2), we have summarised European data on COVID-19 mortality, as of Aug 8, 2020, highlighting the range of outcome measures reported. Another key difference is the status of the health systems in which these patients have been managed, in particular the degree of so-called stress that those systems were under.5Ma X Vervoort D Critical care capacity during the COVID-19 pandemic: global availability of intensive care beds.J Crit Care. 2020; 58: 96-97Crossref PubMed Scopus (99) Google Scholar This factor is more difficult to adjust for. Variations in clinical decision making between health-care systems, reflected in the characteristics of patients admitted to ICUs and in the methods of ventilation used, also confound direct comparison. This confounding is potentially evident when comparing ICU admissions between the UK and Germany, where the median age of patients receiving invasive mechanical ventilation was 72 years in a large German series2Karagiannidis C Mostert C Hentschker C et al.Case characteristics, resource use, and outcomes of 10 021 patients with COVID-19 admitted to 920 German hospitals: an observational study.Lancet Respir Med. 2020; (published online July 28.)https://doi.org/10.1016/S2213-2600(20)30316-7Summary Full Text Full Text PDF PubMed Scopus (562) Google Scholar versus 60 years in the latest ICNARC report.4Intensive Care National Audit and Research CentreICNARC report on COVID-19 in critical care 31 July 2020. Intensive Care National Audit and Research Centre, LondonJuly 31, 2020Google Scholar However, ICU mortality was similar, emphasising the role of admission criteria. Regardless, the wide variation observed suggests the possibility that some factors are modifiable. Therefore, making comparisons between countries and systems is important. Beyond careful epidemiological analysis, we could improve comparisons in several ways. The most obvious way to improve comparisons is via a multinational collaboration. Indeed, it is difficult to see how we can mount an effective response to a global pandemic without such collaboration. The fight against COVID-19 has already produced some commendable examples, including the work of the Coalition for Epidemic Preparedness Innovations, the Global Alliance for Vaccines and Immunizations, and the International Severe Acute Respiratory and Emerging Infection Consortium. However, global comparative data on the outcomes from COVID-19 are lacking because a single observational study of global data, with consistent outcomes and definitions used in all sites, has not yet been reported. An issue with the current reports is the inconsistency of the hospital, ICU, and individual patient-level data definitions. Similarly, fundamental differences are often present in the design of these studies. Together, these limitations make linkage or comparison difficult, which restricts our ability to generate sufficiently robust data to form conclusions. Although a single global study might be logistically and politically improbable, simpler measures could be taken that might ease the task of generating international data with fewer limitations. First, the development of harmonised case report forms and data dictionaries would permit straightforward comparison of studies. This approach would also allow investigators to create tiers of data collection, with increasing levels of complexity. This method of data collection is of real importance in resource-limited settings, in which the advanced monitoring, diagnostics, and research infrastructure required to perform extensive study protocols are not available. The use of core outcomes in ICU observational studies should also be addressed—eg, the WHO Working Group on the Clinical Characterisation and Management of COVID-19 Infection outcome set.6Marshall JC Murthy S Diaz J et al.A minimal common outcome measure set for COVID-19 clinical research.Lancet Infect Dis. 2020; 20: e192-e197Summary Full Text Full Text PDF PubMed Scopus (1064) Google Scholar Second, the inclusion of measures of system stress and resource availability have been largely missing from published studies to date. These measures are essential to understand the observational data collected in the middle of a pandemic. For example, evidence of national variation in ICU outcomes has been described between Feb 8 and May 22, 2020, during the COVID-19 pandemic in England by Qian and colleagues.7Qian Z Alaa AM van der Schaar M Ercole A Between-centre differences for COVID-19 ICU mortality from early data in England.Intensive Care Med. 2020; (published online June 22.)https://doi.org/10.1007/s00134-020-06150-yCrossref PubMed Scopus (39) Google Scholar In this study, the authors hypothesised that national variation might have occurred as a result of local strain or resource constraints in the face of a surge in admissions to hospital. Likewise, a recent study from Brazil has highlighted disparities in the outcomes of patients admitted to hospital between the south of the country and the economically poorer north.8Baqui P Bica I Marra V Ercole A van der Schaar M Ethnic and regional variations in hospital mortality from COVID-19 in Brazil: a cross-sectional observational study.Lancet Glob Health. 2020; 8: e1018-e1026Summary Full Text Full Text PDF PubMed Scopus (361) Google Scholar In practice, this disparity might mean collecting hospital-level and unit-level information on resource availability and staffing over time, in addition to patient-level data. Third, integrating ICU observational studies with those that capture patient-level data before and after admission to critical care, and linkage to clinical trial data, would be advantageous. Such integration would better characterise patients and reduce the replication of data collection. Another benefit would be the ability to assess the influence of variations in clinical decision making on ICU outcomes, which hinder the comparison of existing studies. Furthermore, the effective linkage of large observational studies and clinical trials would allow investigators to better explore subgroup effects and identify heterogeneity of treatment effect, such as was identified in the RECOVERY dexamethasone study.9Horby P Lim WS Emberson JR et al.Dexamethasone in hospitalized patients with COVID-19 - preliminary report.N Engl J Med. 2020; (published online July 17.)https://doi.org/10.1056/NEJMoa2021436Google Scholar Current reports of ICU outcomes for patients with COVID-19 vary between countries. The reasons for these variations are unknown but are unlikely to be attributable to differences in the virus or host response alone. Understanding the reasons that are modifiable would be a major step forward in improving care. JFF is the Chief Investigator of the COVID-19 Critical Care Consortium observational study, an ongoing international study of patients admitted to Intensive Care Units with COVID-19. DFM reports personal fees from consultancy for GlaxoSmithKline, Boehringer Ingelheim, and Bayer; his institution has received funds from grants from the UK National Institute for Health Research (NIHR), Wellcome Trust, Innovate UK, HSC Research and Development Division Northern Ireland, Northern Ireland Chest Heart & Stroke, and UK Medical Research Council; he is a named inventor on a patent (US8962032)covering the use of sialic acid–bearing nanoparticles as anti-inflammatory agents issued to his institution; and he is a Director of Research for the Intensive Care Society and NIHR Efficacy and Mechanism Evaluation Programme Director. All other authors declare no competing interests. Download .pdf (.17 MB) Help with pdf files Supplementary appendix