Strengthening evidence-based medicine with real-world evidence
Authors: Nancy A Dreyer
Journal: The Lancet Healthy Longevity (2022)

People conducting clinical trials often express concerns that real-world evidence is not reliable enough to form a basis for conclusions about causal relationships, because treatments are neither randomised nor blinded, and real-world data are not subject to the rigorous quality control practices used in randomised controlled trials (RCTs).1Eichler H-G Pignatti F Schwarzer-Daum B et al.Randomized controlled trials versus real world evidence: neither myth nor magic.Clin Pharmacol Ther. 2021; 109: 1212-1218Crossref PubMed Scopus (69) Google Scholar Yet real-world data often are the only source for information about treatment outcomes for patients with complications or other vulnerabilities, such as a high-risk family history, lifestyle factors, limited access to health care, and economic hardship. In this issue of The Lancet Healthy Longevity, Yen Yi Tan and colleagues2Tan YY Papex V Chang WH Mueller SH Denaxas S Lai A Comparing clinical trial population representativeness to real-world populations: an external validity analysis encompassing 43 895 trials and 5 685 738 individuals across 989 unique drugs and 286 conditions in England.Lancet Healthy Longev. 2022; (published online Sept 20.)https://doi.org/10.1016/S2666-7568(22)00186-6Summary Full Text Full Text PDF PubMed Scopus (13) Google Scholar report on the external validity of populations studied in RCTs. The authors used data from electronic health records (EHRs) of prescriptions and medical care visits to quantify the proportion of people with a condition under study, but who are excluded from RCTs due to their vulnerabilities. The exclusion criteria identified by review of 43 895 trials were applied to national EHRs. A 15% sample of UK EHRs was used to estimate the prevalence of people who would have been excluded from RCT due to three possible vulnerabilities: multimorbidity, polypharmacy, and age (either adolescents or older individuals). The authors reported population exclusion rates exceeding 50% for adolescents who use various medications, and similar figures for individuals older than 80 years. Multimorbidity had a median exclusion proportion of 91·1% (IQR 88·9–91·8) and for concomitant medications the proportion was 52·5% (IQR 50·0–53·7). Tan and colleagues noted large evidence gaps for cardiovascular disease and psychiatric conditions and they noted that these evidence gaps have important implications for the generalisability of RCTs and highlight the value of real-world evidence. Their analysis demonstrates the sizeable proportions of patients who are excluded from RCTs, and along with it, the use of data collected during routine care. The void of information in RCTs on the most vulnerable patients is a concern. Although RCTs have the central role in determining the extent to which a treatment works, there is little evidence about vulnerable subgroups. Clinicians, patients, and payers typically use the assumption that treatments will be as safe and effective in vulnerable groups treated in real-world settings as they are in patients without complications treated in optimal practices and settings. But we know that this is not always true. For example, aging is associated with a reduction in first-pass metabolism. Dosing adjustments are often needed because the concentration of an oral drug might be reduced before reaching systemic circulation. Real-world evidence has a growing track record among regulators for providing reliable information about comparative risks and benefits in diverse patients.3Dreyer NA Advancing a framework for regulatory use of real-world evidence: when real is reliable.Ther Innov Regul Sci. 2018; 52: 362-368Crossref PubMed Scopus (51) Google Scholar, 4Slattery J Kurz X Assessing strength of evidence for regulatory decision making in licensing: what proof do we need for observational studies of effectiveness?.Pharmacoepidemiol Drug Saf. 2020; 29: 1336-1340Crossref PubMed Scopus (7) Google Scholar, 5Nishioka K Makimura T Ishiguro A Nonaka T Yamaguchi M Uyama Y Evolving acceptance and use of RWE of regulatory decision making on the benefit/risk assessment of a drug in Japan.Clin Pharmacol Ther. 2022; 111: 35-43Crossref PubMed Scopus (19) Google Scholar Advances in the methodology for non-interventional studies of comparative effectiveness afford indispensable tools to inform treatment risks and benefits, including considerations for patients with more complications.6Simon GE Bindman AB Dreyer NA et al.When can we trust real-world data to evaluate new medical treatments?.Clin Pharmacol Ther. 2022; 111: 24-29Crossref PubMed Scopus (7) Google Scholar, 7Velentgas P Dreyer NA Nourjah P Smith SR Torchia MM Developing a protocol for observational comparative effectiveness research: a user's guide. Agency for Healthcare Research and Quality, Rockville, MD2013Google Scholar Regulators also look to real-world evidence to characterise the natural history of disease, providing benchmarks, if not formal external comparators, for single arm trials.8Baumfeld Andre E Reynolds R Caubel P Azoulay L Dreyer NA Trial designs using real-world data: the changing landscape of the regulatory approval process.Pharmacoepidemiol Drug Saf. 2020; 29: 1201-1212Crossref PubMed Scopus (103) Google Scholar External comparators are being used with increasing frequency as we move toward more personalised medicine and target populations become smaller. For example, 27% of all marketing authorisation applications to the European Medicines Agency (EMA) in 2021 were for orphan medical products.9Jonker CJ Bakker E Kurz X Plueschke K Contribution of patient registries to regulatory decision making on rare diseases medicinal products in Europe.Front Pharmacol. 2022; 13924648Crossref PubMed Scopus (7) Google Scholar This need for real-world evidence on treatment effectiveness and safety for diverse and rare populations calls attention to the value of large networks, especially those with established data governance, vetted quality, and demonstrated experience, such as EMA's Data Analytics and Real-World Interrogation Network EU.10European Medicines AgencyData Analysis and Real World Interrogation Network (DARWIN EU).https://www.ema.europa.eu/en/about-us/how-we-work/big-data/data-analysis-real-world-interrogation-network-darwin-euDate accessed: August 10, 2022Google Scholar Payers' interests are driven by the populations for whom they provide health-care coverage, and might include niche coverage, such as psychiatric care. Both payers and biopharmaceutical companies use systematic collection of real-world data to monitor the effectiveness of newly marketed products, some of which are covered by risk sharing models. Real-world evidence is also used to bolster formulary positioning, especially for second-line and third-line treatments, by documenting long-term pragmatic benefits that are unlikely to be studied in RCTs. We have long known that many populations are underrepresented in RCTs and that it is unrealistic to expect head-to-head trials for every intervention and combination, in every patient subgroup of interest, and in situations that closely mimic routine care. Calling this situation a gap sounds more like a crack in the sidewalk than the chasm described here.2Tan YY Papex V Chang WH Mueller SH Denaxas S Lai A Comparing clinical trial population representativeness to real-world populations: an external validity analysis encompassing 43 895 trials and 5 685 738 individuals across 989 unique drugs and 286 conditions in England.Lancet Healthy Longev. 2022; (published online Sept 20.)https://doi.org/10.1016/S2666-7568(22)00186-6Summary Full Text Full Text PDF PubMed Scopus (13) Google Scholar Why not guide decision making with both RCTs and real-world evidence? The guidance might be imperfect, but nonetheless preferable to assuming that all patients will benefit equally, regardless of vulnerability. If we treat RCTs and real-world evidence as complementary, it will strengthen evidence-based medicine for all patients. I report membership Of the Duke Margolis Real-World Evidence Collaborative Advisory Group since 2017, and the International Society of Pharmacoeconomics and Outcomes Research Real-World Data Steering Committee since 2021. Comparing clinical trial population representativeness to real-world populations: an external validity analysis encompassing 43 895 trials and 5 685 738 individuals across 989 unique drugs and 286 conditions in EnglandOverly stringent RCT exclusion criteria do not appropriately account for the heterogeneity of vulnerable characteristics observed in real-world populations. Treatment decision information is scarce for such individuals, which might affect health outcomes. We discuss the challenges facing the inclusivity of such individuals and highlight the strength of real-world evidence as an integrative solution in complementing RCTs and increasing the completeness of evidence-based medicine assessments in evaluating the effectiveness of treatment decisions. Full-Text PDF Open Access