Use of natural experiments to evaluate public health policy
Authors: Lizbeth Burgos Ochoa, Jasper V Been
Journal: The Lancet Public Health (2023)

A key task of public health research is investigation of the causal effect of policies aimed at improving health outcomes. However, such a task is often challenging because a study in which a population is randomised into exposed and unexposed groups is generally not feasible owing to practical and ethical constraints. In certain settings, a so-called natural experiment resulting from a policy implementation can be used to estimate the causal effect of non-experimental interventions.1de Vocht F Katikireddi SV McQuire C Tilling K Hickman M Craig P Conceptualising natural and quasi experiments in public health.BMC Med Res Methodol. 2021; 21: 32Crossref PubMed Scopus (37) Google Scholar, 2Craig P Campbell M Bauman A et al.Making better use of natural experimental evaluation in population health.BMJ. 2022; 379e070872Google Scholar In The Lancet Public Health, Adina Epure and colleagues3Epure AM Courtin E Wanner P Chiolero A Cullati S Carmeli C Effect of covering perinatal health-care costs on neonatal outcomes in Switzerland: a quasi-experimental population-based study.Lancet Public Health. 2023; 8: e194-e202Summary Full Text Full Text PDF Scopus (2) Google Scholar used the natural experiment resulting from implementation of an expansion of health insurance in Switzerland to examine the effect of full coverage of illness-related costs during pregnancy on birth outcomes. To estimate the effect of this policy, the investigators used a variation of the difference-in-discontinuity design formalised by Grembi and colleagues.4Grembi V Nannicini T Troiano U Do fiscal rules matter?.Am Econ J Appl Econ. 2016; 8: 1-30Crossref Scopus (194) Google Scholar This approach can be used when allocation of the intervention is based on a cutoff value for a running variable that cannot be precisely manipulated by the individuals.4Grembi V Nannicini T Troiano U Do fiscal rules matter?.Am Econ J Appl Econ. 2016; 8: 1-30Crossref Scopus (194) Google Scholar Epure and colleagues used date of childbirth as the running variable and the cutoff was the date of the policy implementation (March 1, 2014). The intervention assignment for individuals close to the cutoff value can be assumed to be as good as random, and a causal effect was estimated by comparing outcomes for groups of individuals just before and after the cutoff.4Grembi V Nannicini T Troiano U Do fiscal rules matter?.Am Econ J Appl Econ. 2016; 8: 1-30Crossref Scopus (194) Google Scholar However, in this setting, a change in outcomes around the cutoff might also occur owing to other factors, such as seasonal patterns. To address this issue, the difference-in-discontinuity design uses the information of births that occurred during the same months around the cutoff date in control years with no policy change. 61 910 children were born 9 months before March 1, 2014 and 63 991 were born 9 months after June 1, 2014 (a 3-month censoring was used from March 1, 2014). 382 861 children were born in the same time period around the three control dates.3Epure AM Courtin E Wanner P Chiolero A Cullati S Carmeli C Effect of covering perinatal health-care costs on neonatal outcomes in Switzerland: a quasi-experimental population-based study.Lancet Public Health. 2023; 8: e194-e202Summary Full Text Full Text PDF Scopus (2) Google Scholar Epure and colleagues3Epure AM Courtin E Wanner P Chiolero A Cullati S Carmeli C Effect of covering perinatal health-care costs on neonatal outcomes in Switzerland: a quasi-experimental population-based study.Lancet Public Health. 2023; 8: e194-e202Summary Full Text Full Text PDF Scopus (2) Google Scholar found that implementation of the policy increased mean birthweight by 23 g (95% CI 5–40) and decreased the predicted proportion of low birthweight births by 0·81% (95% CI 0·14–1·48) and of very low birthweight births by 0·41% (0·17–0·65). No statistically significant effects were observed overall for preterm birth and neonatal death. The observed effect sizes were modest, which could partly be explained by the exposed group also including births after pregnancies only partially covered by the policy, diluting the effect. Another contributing factor was that the estimation was done at a population level, whereas many pregnant women are not affected by the policy as they never need illness-related medical care. Unfortunately, the effect of the policy on health-care use, the presumed mediating variable, could not be estimated. Assessment of underlying mechanisms is still necessary to understand the pathways through which the policy might be acting. Although the policy provided additional health care coverage, it did not reduce socioeconomic health inequalities, and in fact even widened them, as babies with parents not at risk of poverty benefited more from the policy. That is, the policy lowered the predicted proportion of extremely preterm births (–0·19%, 95% CI –0·36 to –0·02) and neonatal deaths (–0·13%, –0·26 to 0·01) in those not at risk of poverty, but not among those at poverty risk. Although overall the intervention had a positive effect, these undesirable consequences are commonly observed when policies aimed at improving health outcomes are implemented at a population level. According to the latest European Perinatal Health Report, Switzerland achieved substantial reductions in adverse birth outcomes in the last few years.5Euro-PeristatEuropean Perinatal Health Report: Core indicators of the health and care of pregnant women and babies in Europe from 2015 to 2019.https://www.europeristat.com/images/Euro-Peristat_Fact_sheets_2022_for_upload.pdfDate accessed: February 2, 2023Google Scholar Although there have been improvements at a population level, previous work has pointed out the presence of health inequalities,6Wanner P Adverse perinatal outcomes among children in Switzerland: the impact of national origin and socio-economic group.Int J Public Health. 2020; 65: 1613-1621Crossref PubMed Scopus (5) Google Scholar and policies specifically targeting health inequalities at birth are needed. To be able to make any causal claims, designs based on natural experiments require a detailed assessment of the mechanism allocating the intervention.1de Vocht F Katikireddi SV McQuire C Tilling K Hickman M Craig P Conceptualising natural and quasi experiments in public health.BMC Med Res Methodol. 2021; 21: 32Crossref PubMed Scopus (37) Google Scholar Issues like manipulation of the running variable could affect the validity of the results.7Craig P Cooper C Gunnell D et al.Using natural experiments to evaluate population health interventions: new Medical Research Council guidance.J Epidemiol Community Health. 2012; 66: 1182-1186Crossref PubMed Scopus (577) Google Scholar Such manipulation would be present if there would be a change in behaviour in anticipation of the policy intervention.8Lee DS Lemieux T Regression discontinuity designs in economics.J Econ Lit. 2010; 48: 281-355Crossref Scopus (2097) Google Scholar Bias would arise, for example, if particular groups decided to actively postpone their pregnancy to benefit from the expansion. Although the investigators cannot rule out such bias, the expansion might have been seen by the target population as a minor change that would not have led to a change in pregnancy planning. In conclusion, the nationwide study by Epure and colleagues3Epure AM Courtin E Wanner P Chiolero A Cullati S Carmeli C Effect of covering perinatal health-care costs on neonatal outcomes in Switzerland: a quasi-experimental population-based study.Lancet Public Health. 2023; 8: e194-e202Summary Full Text Full Text PDF Scopus (2) Google Scholar found evidence of modest reductions in key birth outcomes after the implementation of a policy fully covering illness-related costs during pregnancy. The assessment of public health policies provides information on plausible strategies to improve perinatal health and how these could be improved. The natural experiments arising from implementation of policies can be used to estimate causal effects of public health interventions in real-world settings. JVB is principal investigator of a project exploring the effect of the COVID-19 lockdown on perinatal outcomes that is funded by ZonMw (governmental funding) and Chiesi Pharmaceuticals. LBO declares no competing interests. Effect of covering perinatal health-care costs on neonatal outcomes in Switzerland: a quasi-experimental population-based studyFree access to prenatal care in Switzerland reduced the risk of some adverse health outcomes in newborn babies. Expanding health-care coverage is a relevant health system intervention to reduce the risk of adverse health outcomes in the newborn baby and, potentially, across the life course. Full-Text PDF Open Access