COVID-19 vaccine efficacy and effectiveness—the elephant (not) in the room
Authors: Piero Olliaro, Els Torreele, Michel Vaillant
Journal: The Lancet Microbe (2021)

Approximately 96 COVID-19 vaccines are at various stages of clinical development.1Zimmer C Corum J Wee S-L Covid-19 Vaccine Tracker.https://www.nytimes.com/interactive/2020/science/coronavirus-vaccine-tracker.htmlDate accessed: March 10, 2021Google Scholar At present, we have the interim results of four studies published in scientific journals (on the Pfizer–BioNTech BNT162b2 mRNA vaccine,2Polack FP Thomas SJ Kitchin N et al.Safety and efficacy of the BNT162b2 mRNA COVID-19 Vaccine.N Engl J Med. 2020; 383: 2603-2615Crossref PubMed Scopus (9742) Google Scholar the Moderna–US National Institutes of Health [NIH] mRNA-1273 vaccine,3Baden LR El Sahly HM Essink B et al.Efficacy and safety of the mRNA-1273 SARS-CoV-2 Vaccine.N Engl J Med. 2021; 384: 403-416Crossref PubMed Scopus (6728) Google Scholar the AstraZeneca–Oxford ChAdOx1 nCov-19 vaccine,4Voysey M Clemens SAC Madhi SA et al.Safety and efficacy of the ChAdOx1 nCoV-19 vaccine (AZD1222) against SARS-CoV-2: an interim analysis of four randomised controlled trials in Brazil, South Africa, and the UK.Lancet. 2021; 397: 99-111Summary Full Text Full Text PDF PubMed Scopus (3331) Google Scholar and the Gamaleya GamCovidVac [Sputnik V] vaccine)5Logunov DY Dolzhikova IV Shcheblyakov DV et al.Safety and efficacy of an rAd26 and rAd5 vector-based heterologous prime-boost COVID-19 vaccine: an interim analysis of a randomised controlled phase 3 trial in Russia.Lancet. 2021; 397: 671-681Summary Full Text Full Text PDF PubMed Scopus (1226) Google Scholar and three studies through the US Food and Drug Administration (FDA) briefing documents (on the Pfizer–BioNTech,6US Food and Drug AdministrationVaccines and Related Biological Products Advisory Committee meeting: FDA briefing document.https://www.fda.gov/advisory-committees/advisory-committee-calendar/vaccines-and-related-biological-products-advisory-committee-december-10-2020-meeting-announcementDate: Dec 10, 2020Date accessed: March 10, 2021Google Scholar Moderna–NIH,7US Food and Drug AdministrationVaccines and Related Biological Products Advisory Committee meeting: FDA briefing document.https://www.fda.gov/advisory-committees/advisory-committee-calendar/vaccines-and-related-biological-products-advisory-committee-december-17-2020-meeting-announcementDate: Dec 17, 2020Date accessed: March 10, 2021Google Scholar and Johnson & Johnson [J&J] Ad26.COV2.S vaccines).8US Food and Drug AdministrationVaccines and Related Biological Products Advisory Committee meeting: FDA briefing document.https://www.fda.gov/advisory-committees/advisory-committee-calendar/vaccines-and-related-biological-products-advisory-committee-february-26-2021-meeting-announcementDate: Feb 26, 2021Date accessed: March 10, 2021Google Scholar Furthermore, excerpts of these results have been widely communicated and debated through press releases and media, sometimes in misleading ways.9Olliaro P What does 95% COVID-19 vaccine efficacy really mean?.Lancet Infect Dis. 2021; (published online Feb 17.)https://doi.org/10.1016/S1473-3099(21)00075-XSummary Full Text Full Text PDF Scopus (33) Google Scholar Although attention has focused on vaccine efficacy and comparing the reduction of the number of symptomatic cases, fully understanding the efficacy and effectiveness of vaccines is less straightforward than it might seem. Depending on how the effect size is expressed, a quite different picture might emerge (figure; appendix). Vaccine efficacy is generally reported as a relative risk reduction (RRR). It uses the relative risk (RR)—ie, the ratio of attack rates with and without a vaccine—which is expressed as 1–RR. Ranking by reported efficacy gives relative risk reductions of 95% for the Pfizer–BioNTech, 94% for the Moderna–NIH, 91% for the Gamaleya, 67% for the J&J, and 67% for the AstraZeneca–Oxford vaccines. However, RRR should be seen against the background risk of being infected and becoming ill with COVID-19, which varies between populations and over time. Although the RRR considers only participants who could benefit from the vaccine, the absolute risk reduction (ARR), which is the difference between attack rates with and without a vaccine, considers the whole population. ARRs tend to be ignored because they give a much less impressive effect size than RRRs: 1·3% for the AstraZeneca–Oxford, 1·2% for the Moderna–NIH, 1·2% for the J&J, 0·93% for the Gamaleya, and 0·84% for the Pfizer–BioNTech vaccines. ARR is also used to derive an estimate of vaccine effectiveness, which is the number needed to vaccinate (NNV) to prevent one more case of COVID-19 as 1/ARR. NNVs bring a different perspective: 81 for the Moderna–NIH, 78 for the AstraZeneca–Oxford, 108 for the Gamaleya, 84 for the J&J, and 119 for the Pfizer–BioNTech vaccines. The explanation lies in the combination of vaccine efficacy and different background risks of COVID-19 across studies: 0·9% for the Pfizer–BioNTech, 1% for the Gamaleya, 1·4% for the Moderna–NIH, 1·8% for the J&J, and 1·9% for the AstraZeneca–Oxford vaccines. ARR (and NNV) are sensitive to background risk—the higher the risk, the higher the effectiveness—as exemplified by the analyses of the J&J's vaccine on centrally confirmed cases compared with all cases:8US Food and Drug AdministrationVaccines and Related Biological Products Advisory Committee meeting: FDA briefing document.https://www.fda.gov/advisory-committees/advisory-committee-calendar/vaccines-and-related-biological-products-advisory-committee-february-26-2021-meeting-announcementDate: Feb 26, 2021Date accessed: March 10, 2021Google Scholar both the numerator and denominator change, RRR does not change (66–67%), but the one-third increase in attack rates in the unvaccinated group (from 1·8% to 2·4%) translates in a one-fourth decrease in NNV (from 84 to 64). There are many lessons to learn from the way studies are conducted and results are presented. With the use of only RRRs, and omitting ARRs, reporting bias is introduced, which affects the interpretation of vaccine efficacy.10Brown RB Outcome reporting bias in COVID-19 mRNA vaccine clinical trials.Medicina (Kaunas). 2021; 57: 199Crossref PubMed Scopus (20) Google Scholar When communicating about vaccine efficacy, especially for public health decisions such as choosing the type of vaccines to purchase and deploy, having a full picture of what the data actually show is important, and ensuring comparisons are based on the combined evidence that puts vaccine trial results in context and not just looking at one summary measure, is also important. Such decisions should be properly informed by detailed understanding of study results, requiring access to full datasets and independent scrutiny and analyses. Unfortunately, comparing vaccines on the basis of currently available trial (interim) data is made even more difficult by disparate study protocols, including primary endpoints (such as what is considered a COVID-19 case, and when is this assessed), types of placebo, study populations, background risks of COVID-19 during the study, duration of exposure, and different definitions of populations for analyses both within and between studies, as well as definitions of endpoints and statistical methods for efficacy. Importantly, we are left with the unanswered question as to whether a vaccine with a given efficacy in the study population will have the same efficacy in another population with different levels of background risk of COVID-19. This is not a trivial question because transmission intensity varies between countries, affected by factors such as public health interventions and virus variants. The only reported indication of vaccine effectiveness is the Israeli mass vaccination campaign using the Pfizer–BioNTech product. Although the design and methodology are radically different from the randomised trial,2Polack FP Thomas SJ Kitchin N et al.Safety and efficacy of the BNT162b2 mRNA COVID-19 Vaccine.N Engl J Med. 2020; 383: 2603-2615Crossref PubMed Scopus (9742) Google Scholar Dagan and colleagues11Dagan N Barda N Kepten E et al.BNT162b2 mRNA COVID-19 vaccine in a nationwide mass vaccination setting.N Engl J Med. 2021; (published online Feb 24.)https://doi.org/10.1056/NEJMoa2101765Crossref PubMed Scopus (1762) Google Scholar report an RRR of 94%, which is essentially the same as the RRR of the phase 3 trial (95%) but with an ARR of 0·46%, which translates into an NNV of 217 (when the ARR was 0·84% and the NNV was 119 in the phase 3 trial). This means in a real-life setting, 1·8 times more subjects might need to be vaccinated to prevent one more case of COVID-19 than predicted in the corresponding clinical trial. Uncoordinated phase 3 trials do not satisfy public health requirements; platform trials designed to address public health relevant questions with a common protocol will allow decisions to be made, informed by common criteria and uniform assessment. These considerations on efficacy and effectiveness are based on studies measuring prevention of mild to moderate COVID-19 infection; they were not designed to conclude on prevention of hospitalisation, severe disease, or death, or on prevention of infection and transmission potential. Assessing the suitability of vaccines must consider all indicators, and involve safety, deployability, availability, and costs. This online publication has been corrected. The corrected version first appeared at thelancet.com/microbe on June 11, 2021 This online publication has been corrected. The corrected version first appeared at thelancet.com/microbe on June 11, 2021 We declare no competing interests. Download .pdf (.22 MB) Help with pdf files Supplementary appendix Correction to Lancet Microbe 2021; published online April 20. https://doi.org/10.1016/S2666-5247(21)00069-0Olliaro P, Torreele E, Vaillant M. COVID-19 vaccine efficacy and effectiveness—the elephant (not) in the room. Lancet Microbe 2021; published online April 20. https://doi.org/10.1016/S2666-5247(21)00069-0—In this Comment, the relative risk reduction for the Gamaleya vaccine should have been 91%, and the numbers needed to vaccinate should have been 81 for the Moderna–NIH, 108 for the Gamaleya, and 119 for the Pfizer–BioNTech vaccines. These corrections have been made to as of June 11, 2021. Full-Text PDF Open AccessCOVID-19 vaccines: effectiveness and number needed to treatIn a Lancet Microbe Comment, Piero Olliaro and colleagues1 suggest that reporting relative risk reduction (RRR) for vaccination does not reflect entirely its therapeutic performance and consider the solw use of RRR a reporting bias. In addition, they propose that absolute risk reduction (ARR) should be reported as a measure of the vaccine's effectiveness. The authors end up comparing the numbers needed to vaccinate to prevent one case of COVID-19 among the vaccines, which derives from the absolute reductions. Full-Text PDF Open AccessCOVID-19 vaccines: effectiveness and number needed to treat – Authors' replyWe thank Luis C L Correia and Denise Matias1 for the opportunity to clarify why both relative and absolute vaccine effects should be reported when projecting individual and population-wide benefits from clinical trials2 and why this reporting is necessary for well informed public health decision making. Beyond statistical disputes, it is about how health research data are generated, presented, understood, and used for policies. Full-Text PDF Open Access