Using administrative data to assess early-life policies
Authors: Katie Harron, Jenny Woodman
Journal: The Lancet Public Health (2023)

In The Lancet Public Health, Ronan McCabe and colleagues1McCabe R Katikireddi SV Dundas R Craig P The health impact of Scotland's Baby Box Scheme: a natural experiment evaluation using national linked health data.Lancet Public Health. 2023; 8: e504-e510Summary Full Text Full Text PDF Google Scholar show how observational data that are routinely collected when individuals interact with services can be used to build evidence when a randomised control trial is not feasible, efficient, or acceptable. The authors report that the introduction of the universal Baby Box Scheme in Scotland, UK—a national programme offering a box of essential items that doubles as a baby bed free of charge to all pregnant women—was associated with a small reduction in infant and primary carer tobacco smoke exposure in the early postnatal period, with some evidence of an increase in breastfeeding for mothers younger than 25 years, but no evidence of an effect on sleeping position or hospital admissions of the child or mother. In interpreting these findings of small and no effect, it is important to consider the challenges of using a non-randomised design and relying on administrative data for outcome measures. The first challenge is to create a suitable comparison group. McCabe and colleagues used a quasi-experimental design to compare all babies born after the introduction of the Baby Box Scheme (intervention group) with babies born before the policy (control group). This intention-to-treat interrupted time series analysis makes sense given the high uptake of the scheme by parents (>85%) and absence of data on which families did not receive the box. The authors exploited longitudinal data to account for background trends in outcomes and tested for the effect of other external events using temporal falsification and negative controls, increasing the confidence that any changes in outcomes are indeed the result of the intervention. This approach also overcomes the limitation that exposed and unexposed populations can differ in characteristics related to the outcome; crude comparisons between intervention and control groups can result in underestimation of intervention effect or even spurious associations because families might be targeted for the intervention because they are at systematically higher risk of worse outcomes (indication bias). But even when intervention and control groups are matched on some characteristics, there might be residual confounding because the data available for matching do not capture all relevant differences.2Cavallaro FL Cannings-John R Lugg-Widger F et al.Lessons learned from using linked administrative data to evaluate the Family Nurse Partnership in England and Scotland.Int J Popul Data Sci. 2023; 8: 10Google Scholar A further challenge with administrative data is the availability and interpretation of outcome measures. All-cause hospital admission is a readily available and easy-to-use proxy for maternal and child health and wellbeing, but might not be affected by a low-dose universal intervention. Additionally, differences in hospital admissions might reflect differences in either underlying health needs or health-care-seeking behaviours, making interpretation difficult. Disaggregating hospital admissions into planned and unplanned, assessing reasons for admission, and investigating emergency department attendances could provide further insight into the effect of interventions on health-care use. Despite advances made in linking health data with education and social care data across the UK,3Mc Grath-Lone L Libuy N Harron K et al.Data resource profile: the Education and Child Health Insights from Linked Data (ECHILD) database.Int J Epidemiol. 2022; 51 (17f): 17Crossref PubMed Scopus (7) Google Scholar, 4Fleming M Salim EE Mackay DF et al.Neurodevelopmental multimorbidity and educational outcomes of Scottish schoolchildren: a population-based record linkage cohort study.PLoS Med. 2020; 17e1003290Crossref PubMed Scopus (15) Google Scholar benefits of early interventions that accrue across the wider system and social determinants of health remain hard to detect—eg, those effecting housing services, access to childcare, and drug and alcohol support services. If effects are detected, they are difficult to attribute to the policy or intervention. Administrative data do not routinely measure more proximal and perhaps more appropriate outcomes that are expected to be amenable to intervention and which effect long-term outcomes, such as child–parent interaction, secure attachment, regular bedtimes, or confident parenting. Supplemental data sources are needed but, as McCabe and colleagues allude to, additional outcome data can only be collected if the evaluation is built into the roll-out of the policy. Finally, expecting to detect effects of an early-life intervention on inequalities might be unrealistic in the context of the challenges that socially disadvantaged families currently face in the UK. Increasing child poverty and adversity, and retraction of preventive services across the board, might have a much larger impact on outcomes and inequalities than can be addressed by any single early-years intervention. Cattan and colleagues suggest that even three decades of early-years reform and policy might have only been enough to stabilise inequalities in child development, given the context of growing adversity.5Cattan S Fitzsimons E Goodman A Phimister A Ploubidis GB Wertz J Early childhood inequalities. IFS Deaton Review of Inequalities.https://ifs.org.uk/inequality/wp-content/uploads/2022/06/Early-childhood-inequalities-IFS-Deaton-Review-2.pdfDate: 2018Date accessed: June 13, 2023Google Scholar Strategies to address the root causes of social disadvantage are required if inequalities are to be reduced. But beyond that, policies and interventions such as Scotland's Baby Box Scheme can play a role in signalling values such as a commitment to equality and fairness and should perhaps be judged on these grounds, too.6Scottish GovernmentScotland's Baby Box: evaluability assessment.https://www.gov.scot/publications/evaluability-assessment-scotlands-baby-box-report-scottish-government/Date: 2018Date accessed: June 13, 2023Google Scholar McCabe and colleagues’ study is an important example of the power of administrative data, combined with robust analytical approaches, for adding to the evidence base. The challenges of retrospectively evaluating early-years policy mean that even the small effects in only some outcomes should be viewed as a promising result for Scotland's Baby Box Scheme. We declare no competing interests. The health impact of Scotland's Baby Box Scheme: a natural experiment evaluation using national linked health dataSBBS reduced infant and primary carer tobacco smoke exposure, and increased breastfeeding among young mothers in Scotland. However, absolute effects were small. Full-Text PDF Open Access