metadata: authors: 0: author: Croke,Kevin
1: author: Hicks,Joan Hamory
2: author: Hsu,Eric
3: author: Kremer,Michael Robert
4: author: Miguel,Edward A.
count: World
docna: 0: docna: Does mass deworming affect child nutrition ?
            meta-analysis, cost-effectiveness, and statistical power
docty: Policy Research Working Paper
subtopic: Pharmaceuticals Industry,Health Care Services Industry,Disease Control & Prevention,Public Health Promotion,Reproductive Health,Early Child and Children's Health,Pharmaceuticals & Pharmacoeconomics,Nutrition,School Health,Educational Populations,Education for Development (superceded),Education For All
teratopic: Education
abstracts: cdata!: The WHO has recently debated whether to
            reaffirm its long-standing recommendation of mass drug
            administration (MDA) in areas with more than 20 percent
            prevalence of soil-transmitted helminths (hookworm,
            whipworm, and roundworm). There is consensus that the
            relevant deworming drugs are safe and effective, so the key
            question facing policymakers is whether the expected
            benefits of MDA exceed the roughly $0.30 per treatment cost.
            The literature on long run educational and economic impacts
            of deworming suggests that this is the case. However, a
            recent meta-analysis by Taylor-Robinson et al. (2015),
            (hereafter TMSDG), disputes these findings. The authors
            conclude that while treatment of children known to be
            infected increases weight by 0.75 kg (95 percent CI: 0.24,
            1.26; p=0.0038), there is substantial evidence that MDA has
            no impact on weight or other child outcomes. This paper
            updates the TMSDG analysis by including studies omitted from
            that analysis and extracting additional data from included
            studies, and finds that the TMSDG analysis is underpowered:
            Power is inadequate to rule out weight gain effects that
            would make MDA cost effective relative to comparable
            interventions in similar populations, and underpowered to
            reject the hypothesis that the effect of MDA is different
            from the effect that might expected, given deworming's
            effects on those known to be infected. The hypothesis of a
            common zero effect of multiple-dose MDA deworming on child
            weight at longest follow-up is rejected at the 10 percent
            level using the TMSDG dataset, and with a p value < 0.001
            using the updated sample. In the full sample, including
            studies in settings where prevalence is low enough that the
            WHO does not recommend deworming, the average effect on
            child weight is 0.134 kg (95 percent CI: 0.031, 0.236,
            random effects). In environments with greater than 20
            percent prevalence, where the WHO recommends mass treatment,
            the average effect on child weight is 0.148 kg (95 percent
            CI: 0.039, 0.258). The implied average effect of MDA on
            infected children in the full sample is 0.301 kg. At 0.22 kg
            per U.S. dollar, the estimated average weight gain per
            dollar is more than 35 times that from school feeding
            programs as estimated in RCTs. Under-powered meta-analyses
            are common in health research, and this methodological issue
            will be increasingly important as growing numbers of
            economists and other social scientists conduct meta-analysis.
repnme: repnme: Does mass deworming affect child nutrition
            ? meta-analysis, cost-effectiveness, and statistical power
display_title: Does mass deworming affect child
            nutrition meta-analysis, cost-effectiveness, and statistical power
listing_relative_url: /research/2016/12/27035515/mass-deworming-affect-child-nutrition-meta-analysis-cost-effectiveness-statistical-power
topicv3: Deworming m1326778 1541,Nutrition m1326779 644,Pharmaceuticals m1327740 1350,Vitamin A m1504373 3634,Health m1504661 3480
publishtoextweb_dt: 2016-12-19T00:00:00Z
historic_topic: Education
no_of_pages: 61
author_keys: 0: author_key: 1161784
1: author_key: 95012
2: author_key: 606929
model_extractions: WPS7921 Policy Research Working Paper 7921 Does Mass Deworming Affect Child Nutrition? Meta-Analysis, Cost-Effectiveness, and Statistical Power _Kevin Croke Joan Hamory Hicks Eric Hsu Michael Kremer Edward Miguel_ Development Research Group Impact Evaluation Team December 2016
Policy Research Working Paper 7921

# **Abstract**

The WHO has recently debated whether to reaffirm its long-standing recommendation of mass drug administration (MDA) in areas with more than 20 percent prevalence of soil-transmitted helminths (hookworm, whipworm, and roundworm). There is consensus that the relevant deworming drugs are safe and effective, so the key question facing policymakers is whether the expected benefits of MDA exceed the roughly $0.30 per treatment cost. The literature on long run educational and economic impacts of deworming suggests that this is the case. However, a recent meta-analysis by Taylor-Robinson et al. (2015), (hereafter TMSDG), disputes these findings. The authors conclude that while treatment of children known to be infected increases weight by 0.75 kg (95 percent CI: 0.24, 1.26; p=0.0038), there is substantial evidence that MDA has no impact on weight or other child outcomes. This paper updates the TMSDG analysis by including studies omitted from that analysis and extracting additional data from included studies, and finds that the TMSDG analysis is underpowered: Power is inadequate to rule out weight gain effects that would make MDA cost effective relative to comparable interventions in similar populations, and underpowered to reject the hypothesis that the effect of MDA is different from the effect that might expected, given deworming’s effects on those known to be infected. The hypothesis of a common zero effect of multiple-dose MDA deworming on child weight at longest follow-up is rejected at the 10 percent level using the TMSDG dataset, and with a p value < 0.001 using the updated sample. In the full sample, including studies in settings where prevalence is low enough that the WHO does not recommend deworming, the average effect on child weight is 0.134 kg (95 percent CI: 0.031, 0.236, random effects). In environments with greater than 20 percent prevalence, where the WHO recommends mass treatment, the average effect on child weight is 0.148 kg (95 percent CI: 0.039, 0.258). The implied average effect of MDA on infected children in the full sample is 0.301 kg. At 0.22 kg per U.S. dollar, the estimated average weight gain per dollar is more than 35 times that from school feeding programs as estimated in RCTs. Under-powered meta-analyses are common in health research, and this methodological issue will be increasingly important as growing numbers of economists and other social scientists conduct meta-analysis.

This paper is a product of the Impact Evaluation Team, Development Research Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at kcroke@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
Does Mass Deworming Affect Child Nutrition?

# Meta-Analysis, Cost-Effectiveness, and Statistical Power

Kevin Croke<sup>1</sup> , Joan Hamory Hicks<sup>2</sup> , Eric Hsu<sup>2</sup> , Michael Kremer<sup>3</sup> , Edward Miguel<sup>2</sup> 1World Bank and Harvard T.H. Chan School of Public Health 2University of California, Berkeley 3Harvard University

## December 16, 2016

**Abstract** : The WHO has recently debated whether to reaffirm its long-standing recommendation of mass drug administration (MDA) in areas with more than 20% prevalence of soil-transmitted helminths (hookworm, whipworm, and roundworm). There is consensus that the relevant deworming drugs are safe and effective, so the key question facing policymakers is whether the expected benefits of MDA exceed the roughly $0.30 per treatment cost. The literature on long run educational and economic impacts of deworming suggests that this is the case. However, a recent meta-analysis by Taylor-Robinson et al. (2015), (hereafter TMSDG), disputes these findings. The authors conclude that while treatment of children known to be infected increases weight by 0.75 kg (95% CI: 0.24, 1.26; p=0.0038), there is substantial evidence that MDA has no impact on weight or other child outcomes. This paper updates the TMSDG analysis by including studies omitted from that analysis and extracting additional data from included studies, and finds that the TMSDG analysis is underpowered: Power is inadequate to rule out weight gain effects that would make MDA cost effective relative to comparable interventions in similar populations, and underpowered to reject the hypothesis that the effect of MDA is different from the effect that might expected, given deworming’s effects on those known to be infected. The hypothesis of a common zero effect of multiple-dose MDA deworming on child weight at longest follow-up is rejected at the 10% level using the TMSDG dataset, and with a p value < 0.001 using the updated sample. In the full sample, including studies in settings where prevalence is low enough that the WHO does not recommend deworming, the average effect on child weight is 0.134 kg (95% CI: 0.031, 0.236, random effects). In environments with greater than 20% prevalence, where the WHO recommends mass treatment, the average effect on child weight is 0.148 kg (95% CI: 0.039, 0.258). The implied average effect of MDA on infected children in the full sample is 0.301 kg. At 0.22 kg per U.S. dollar, the estimated average weight gain per dollar is more than 35 times that from school feeding programs as estimated in RCTs. Under-powered meta-analyses are common in health research, and this methodological issue will be increasingly important as growing numbers of economists and other social scientists conduct meta-analysis.

### JEL classifications: I10, I15, I38, H43.

Keywords: deworming, nutrition, meta-analysis, cost-effectiveness

**Acknowledgments** : We are grateful to the Campbell Collaboration deworming team, especially Vivian Welch, for generously sharing data and information on their work. We thank Harold Alderman, Shally Awasthi, Don Bundy, Serene Joseph, Chengfang Liu, Scott Rozelle, and Walter Willett for help interpreting their studies, searching for original data, or supplying us with additional results. We thank Egor Abramov, Kamran Jamil, Spencer Ma, Cole Scanlon, Wen Wang, and Kevin Xie for research assistance. We thank Amrita Ahuja, Harold Alderman, Sarah Baird, Matthew Basilico, Peter Hotez, Rachael Meager, Antonio Montresor, and Dina Pomeranz for helpful comments.
Croke et al. (2016)

# **1 Introduction**

Soil transmitted helminths (STH; including hookworm, whipworm, and roundworm) are estimated to affect 1 in 4 people in endemic countries (Pullan et al., 2014) and at least 1.3 billion people worldwide (Global Atlas of Helminth Infections, 2016). The intensity of infection is correlated with prevalence, and highly skewed, so while many infected individuals have light infections, only a minority have the moderate to severe intensity infections believed to account for the bulk of morbidity (Anderson, Truscott and Hollingsworth, 2014).

Because of the damage worms can cause, there is consensus that those known to be infected should be treated. There is also consensus that deworming drugs are safe and effective; indeed, they are the standard of care for those known to be infected. Because individual collection and testing of stool samples prior to treatment is prohibitively expensive and logistically impractical in many low-income contexts, the World Health Organization (WHO) has long recommended mass drug administration (MDA) in endemic areas. In particular, the WHO recommends annual MDA in areas with more than 20% prevalence of soil-transmitted helminths, and multiple annual MDA where prevalence is above 50% (World Health Organization, 2012).

Studies of the long-term impact of deworming in multiple environments suggest that even under conservative assumptions and allowing for uncertainty, the expected benefits of MDA exceed the $0.30 per treatment costs (Ahuja et al., 2015; Baird et al., 2
Croke et al. (2016) 2016; Bleakley, 2007; Croke, 2014; Ozier, 2015).<sup>1</sup> Many organizations that have examined the evidence have judged deworming to be highly cost effective, including the Copenhagen Consensus (Hall and Horton, 2008), the Disease Control Priorities Project (Hotez et al., 2006)<sup>2</sup> , Givewell (Givewell, 2013; Givewell, 2014), the Abdul Latif Jameel Poverty Action Lab (J-PAL Policy Bulletin, 2012) and the World Bank (World Bank, 1993).

However, a recent Cochrane Review (Taylor-Robinson et al., 2015; hereafter “TMSDG”) argues against this long-standing consensus. TMSDG conduct a meta-analysis of randomized studies focusing on the short-term impacts of deworming. The authors express support for treating those known to be infected (p. 30), and estimate that single dose treatment for those known to be infected (and to have moderate intensity infections on average) increases weight by 0.75 kg (95% CI: 0.24,1.26; p=0.0038). However, they take a strong stand on mass drug administration in areas where worms are endemic, arguing that there is “substantial evidence” that mass treatment has no impact on child weight or other outcomes (p. 2). This has led a subset of the TMSDG authors to conclude in other work that the belief in long-term educational and economic impacts discussed above are “delusional'' (Garner, Taylor-Robinson and Sachdev, 2015).

Of course, failure to reject the null hypothesis of no effect does not constitute evidence of no effect. Particularly when treatment is inexpensive and side-effects are mild, studies and meta-analyses must have adequate power in order to rule out possibly modest effect sizes that would still render treatment cost-effective. Unfortunately, underpowered

> 1 Givewell calculates the cost of deworming for soil-transmitted helminthes in India at $0.30 per child per treatment, which includes both drug and delivery costs, including the value of staff time (Givewell (2016)). 2 We note that while analysts at Givewell identified errors in the calculations of deworming's cost effectiveness presented in the Disease Control Priorities Project, their revised calculations also find it to be a cost effective intervention at $82.51 -- $138.28 per DALY (Givewell, 2011).

3
Croke et al. (2016) meta-analyses are extremely common in many areas of health research (Turner, Bird and Higgins, 2013), and it is important to examine when this is the case. In the case of deworming, the cost of MDA for STH is estimated at just $0.30 per treatment (Givewell, 2016). Since the majority of those treated in MDA programs will either be uninfected or have only light intensity infections rather than the moderate to severe infections thought to account for the bulk of STH morbidity (De Silva et al. 2015, Montresor et al. 2015), statistical power to pick up population-wide effects is typically limited (Bundy, Walson and Watkins, 2013). TMSDG include studies in environments with prevalence below the 20% threshold at which the WHO recommends deworming, further weakening statistical power by including estimates from settings where there are few worm infections to begin with.

In this paper, we first assess statistical power in TMSDG, and conclude that power is inadequate to rule out weight gain effects that would make MDA cost effective relative to school feeding programs aimed at similar populations. It is also underpowered to reject the hypothesis that the effect of MDA is different from the effect that might expected, given deworming’s effects on those known to be infected, and given STH prevalence in the populations where the MDA trials took place. We then update the work of TMSDG to create a more comprehensive, and thus better-powered, meta-analysis. The update includes 22 estimates from 20 studies examining the impact of multiple-dose MDA on child weight at longest follow-up, twice as many as TMSDG. It includes four studies not identified in TMSDG and additional data from six studies discussed by TMSDG but not included in their meta-analysis for the child weight outcome, either using procedures in the _Cochrane Handbook for Systematic Reviews of Interventions_ (Higgins and Green, 2011) to extract 4
Croke et al. (2016) additional data or contacting the original authors to obtain additional information.<sup>3</sup> Additionally, in three cases, the updated analysis includes improved estimates, for example, by obtaining information on intra-cluster correlation directly from the original study authors rather than by imputing data from other studies.

With the full data set, the hypothesis of a common zero effect of multiple-dose deworming on weight is rejected with p<0.001. With the TMSDG sample, this hypothesis is rejected at the 10% level, but applying either a study classification approach used in the previous 2012 Cochrane Review or a study classification approach used in earlier Cochrane Reviews leads to rejection with p<0.01.<sup>4</sup> Any one of five other individual updates to the TMSDG data leads to rejection of the null hypothesis of a common zero effect at the 5% level.

Using our updated sample, and following TMSDG in including studies from lowprevalence environments where the WHO does not recommend MDA, mass deworming is estimated to increase child weight by 0.134 kg (95% CI: 0.031,0.236; p=0.01; random effects estimation). This effect remains robust (at p<0.05) when any individual trial estimate is dropped from the meta-analysis. The result is also nearly unchanged even when simultaneously dropping any two of the 22 estimates: among the 231 possible combinations of two studies that could be dropped simultaneously, in 96% of cases the estimated effect remains statistically significant at p<0.05, and the largest p-value is just 0.067. In areas with prevalence above 20%, where the WHO recommends MDA, the average estimated impact on child weight is 0.148 kg (CI: 0.039,0.258; p=0.008; random

> 3 In several cases we received data from the Campbell Collaboration deworming team, who had themselves directly contacted the original authors. See Appendix A for details on the source of all studies included in our sample.

> 4 See section 5.1 for more detailed discussion of these study classification issues.

5
Croke et al. (2016) effects). Where the WHO recommends multiple annual MDA (areas with prevalence above 50%), the average estimated weight gain is 0.182 kg (CI: 0.070,0.293; p=0.001; random effects).

The implied average effect of MDA deworming on infected children in the full sample (calculated by dividing estimated impact by worm prevalence for each study and applying a random effects model) is 0.301 kg.<sup>5</sup> This average effect likely conceals substantial heterogeneity. Light infections are often asymptomatic, and only between 2 and 16 percent of the population experience moderate to severe intensity infections in the studies in our sample that report this information,<sup>6</sup> so implied effects in the subpopulation of those with moderate to severe intensity infections are likely much larger. For more general context on the implied average effect size of 0.301 kg, the difference in weight gain for boys at the 25th versus at the 50th percentile of the weight-for-age distribution between ages 2 and 3 is 0.2 kg (World Health Organization Multicentre Growth Reference Study Group, 2006).<sup>7</sup> Moreover, this gain comes at modest cost compared to some other common interventions. The implied weight gain per U.S. dollar of expenditure is 0.22 kg, assuming

> 5 The implied average effect on infected children in the subsamples of studies conducted in areas of 20% and 50% prevalence are similar, at 0.249 and 0.276 respectively. The confidence intervals overlap with that of the full sample.

> 6 In particular, Joseph et al. (2015) reports 15% prevalence of any helminthic infection among preschool aged children at baseline in Peru, with only 1.8% of children exhibiting moderate or heavy infections, while Miguel and Kremer (2004) report hookworm, whipworm, and roundworm prevalences of 77%, 42%, and 55%, with 15%, 16%, and 10% prevalence of moderate to severe infection respectively. Moreover, Miguel and Kremer (2004) use thresholds that are lower than the WHO thresholds used by Joseph et al. (2015) for moderate to severe intensity infections, suggesting that the comparable prevalence of moderate-severe infections in the Miguel and Kremer (2004) sample is lower than the figures presented here.

> 7 According to WHO growth charts, a boy at the 25th percentile of the weight for age distribution grows 1.9 kg between age 2 and age 3, while a boy at the 50th percentile grows 2.1 kg over that year. Note that the difference in weight gain for boys at the 25th versus at the 50th percentile of the weight-for-age distribution between ages 3 and 4 is also 0.2 kg (World Health Organization Multicentre Growth Reference Study Group, 2006). In our full sample, the median duration of follow up at which weight gain is measured is 1 year.

6
Croke et al. (2016) two MDA treatments per year. For comparison, the weight gain per dollar of expenditure estimated in RCTs by Galloway et al. (2009) for school feeding programs is less than 0.01 kg, suggesting that relative to school feeding, deworming is highly cost effective in increasing weight in school-age children in low-income countries. This echoes a recent epidemiological study that similarly finds deworming to be highly cost-effective (Lo et al., 2016).

This paper is organized as follows. Section 2 provides background information on soil-transmitted helminths, mass drug administration, and earlier literature, including TMSDG, and assesses whether TMSDG is adequately powered to rule out costeffectiveness. Section 3 discusses the sample, including criteria for study inclusion, the procedure used to identify studies, and the general principles guiding data extraction and determination of which estimates to use in the meta-analysis. Appendix A details the application of these principles to individual studies. Section 4 describes our hypothesis testing and estimation strategy. Section 5 replicates the TMSDG analysis, tests the hypothesis of a common zero effect of multiple dose MDA, and estimates the impact of multiple dose MDA on child weight, both in environments where the WHO recommends mass deworming and more broadly. Section 6 concludes with a discussion of implications, methodology, and directions for future research.

# **2 Background**

STH are spread via eggs deposited in the local environment when individuals defecate in their surroundings or do not practice proper hygiene after defecating. Due to the transmission mechanism, school-aged children are especially vulnerable to these worm 7
Croke et al. (2016) infections, and also play an important role in spreading them in the local community (Hotez et al., 2006). The potential health consequences of worm infections are generally agreed to depend on the number of worms in the body (i.e., infection intensity), rather than a simple binary indicator of infection status. Infection intensity is highly skewed and is strongly correlated with disease prevalence: in low prevalence populations relatively few people have severe infections, while many more do in high prevalence populations (Anderson, Truscott and Hollingsworth, 2014).

Because of these health impacts, there is widespread acceptance that those who are known to be infected with intestinal helminths should be treated. Indeed, this is the standard of medical care (Horton, 2000; Keiser and Utzinger, 2008; Perez del Villar et al., 2012), and some consider not treating individuals known to be infected as unethical. New trials of this type are therefore typically not conducted, but TMSDG identify five trials of singledose treatment on what the authors term “infected children,”<sup>8</sup> and one trial of multiple dose treatment. The TMSDG meta-analysis found a near-term child weight gain effect of 0.75 kg across the single dose trials (95% CI: 0.24,1.26; p=0.0038).<sup>9</sup> These trials were largely conducted in settings with considerable infection intensity: for the 3 (out of 5) single dose studies that report infection intensities as measured by eggs per gram (epg), all three report mean epg values which are equivalent to a moderate intensity infection for at least one type of worm (Stephenson et al. 1989; Stephenson et al. 1993; Yap and Steinmann 2014).<sup>10</sup>

> 8 As described in more detail below, this terminology is slightly different than “screened for infection,” which was the classification used in the previous Cochrane Review (Taylor-Robinson et al., 2012).

> 9 The degree of confidence that TMSDG have in these findings is unclear. At one point in the text they note that the case for treating infected children is “obvious” (p. 30). However, they elsewhere describe the metaanalysis results as “low quality evidence” (Taylor-Robinson et al. 2015, p. 2).

> 10 These are our calculations, using the infection intensity thresholds from World Health Organization (2002).

8
Croke et al. (2016) Because deworming treatment is inexpensive and safe but diagnosis is comparatively expensive (necessitating lab analysis of a stool sample) and logistically difficult in many contexts, the WHO recommends annual mass treatment in areas where worm infections are above 20% and multiple treatments annually where prevalence is greater than 50%. Screening for worm infections requires testing stool samples, which in turn necessitates skilled staff, laboratory facilities, and re-contacting infected individuals for treatment, which can be challenging in many contexts where worm infections are endemic. Furthermore, the Kato-Katz test - the most commonly used method for testing for worms in these regions - has an estimated specificity between 52% and 91% (Barda et al., 2013; Assefa et al., 2014), suggesting that many infections would go undetected, and thus presumably untreated, even with proper screening. TMSDG note that screening for worm infections is not recommended by the WHO because the cost of screening is 4 to 10 times that of the treatment itself (Taylor-Robinson et al., 2015, p. 7). Taken together, this suggests that a policy of screened treatment for worm infections would be costly, logistically complicated, and imprecise.

Subsequent to the WHO recommendation for MDA, a social science literature emerged measuring the longer-term educational and economic impact of mass deworming. Four studies in three moderate to high prevalence settings -- in Kenya, Uganda, and the historical southern United States -- all find substantial long-run impacts of deworming on educational outcomes (Croke, 2014; Ozier, 2015; Bleakley, 2007; Baird et al., 2016). Two of these studies also report economic outcomes and both find positive effects.

9
Croke et al. (2016)

- Croke (2014) finds that Ugandan children exposed to a deworming program originally studied in Alderman et al. (2006) have higher math test scores nearly a decade later, with effect sizes of over 0.2 standard deviation units.

- Ozier (2015) finds that infant children who lived in Kenyan communities where older school-age children were dewormed show large cognitive test score improvements ten years later, presumably due to reduced infection through beneficial spillover effects. The magnitude of the effect is 0.2 to 0.3 standard deviation units, which is equivalent to between 0.5 to 0.8 years of schooling.

- Using a difference-in-difference estimation methodology rather than a randomized design to study a deworming campaign in the U.S. South in the early 1900s, Bleakley (2007) finds that deworming led to increased school enrollment and attendance for children, and improved literacy and boosted income by 17% for adults who were treated as children.

- Finally, Baird et al. (2016) estimate that a decade after treatment, males who participated in mass deworming in Kenya worked 17% more hours per week and had higher living standards, missing approximately one fewer meal per week. Females were approximately one-quarter more likely to have passed the primaryschool leaving exam and attended secondary school. The estimated value of benefits, in terms of the net present value of future earnings net of increased schooling costs, exceeds the cost by more than one hundred fold.

These results suggest that the expected benefits of deworming would greatly exceed its costs even if one took a conservative approach, assuming a very low probability of effects 10
Croke et al. (2016) of the measured magnitude or assuming that true effects are considerably less than the measured effects (Ahuja et al. 2015).

While TMSDG estimate positive impacts of deworming treatment on weight of children known to be infected, they argue in contrast that there is “substantial evidence” that mass deworming does not improve weight or other child outcomes. Note that this is a considerably different -- and far more demanding -- statement than a claim that the null hypothesis of no effect cannot be rejected (for instance, due to a lack of statistical power), or that MDA has a positive impact in some settings but not others.

TMSDG has been controversial. Campbell et al. (2016) note substantial changes over time in the way the meta-analysis in the Cochrane Review on deworming is presented (see the evolution across Dickson et al. (2000), Dickson et al. (2007), Taylor-Robinson et al. (2012), and Taylor-Robinson et al. (2015)). Campbell et al. (2016) also note that although the text refers to a protocol (Taylor-Robinson et al., 2015, p.29), they were unable to find a publicly available pre-specified protocol for the updated Review, leading to confusion over the precise hypothesis being tested, uneasiness over how studies are grouped for analysis, and concern about which studies are included versus excluded. A number of authors (Campbell et al., 2016; De Silva et al., 2015; Montresor et al., 2015; Michael et al., 2000) have expressed concern over lack of consideration given to the effects of different STH species, treatments, and drug distribution strategies.

# **2.1 Is TMSDG adequately powered?**

A simple calculation suggests that the TMSDG analysis is underpowered to rule out the possibility of effects that would make mass drug administration cost effective relative to 11
Croke et al. (2016) school feeding. As noted, the estimated weight gain in kilograms per dollar spent from school feeding is less than 0.01 (from RCT studies; Galloway et al. (2009)). Given the $0.60 per year treatment cost of deworming in environments where two annual doses are required, an effect of just 0.006 kg would make deworming cost effective relative to school feeding (setting aside, for the time being, the issue of other outcomes).

Examining the random effects meta-analysis estimator that is the focus in TMSDG and taking as given the variance of the effect across studies that TMSDG estimate, and the standard errors TMSDG report for the underlying studies, the implied Minimum Detectable Effect (MDE) size at 95% confidence and 80% power is 0.28 kg.<sup>11</sup> As illustrated in Figure 3, a wide range of possible effect sizes lies above the magnitude at which deworming would be cost effective relative to other common interventions for similar populations (~0.006 kg), yet below the magnitude which TMSDG would be adequately powered to detect (~0.28 kg). For example, this range includes TMSDG's random effects estimate of 0.08 kg, for which there is only 13% statistical power to detect a significant effect at 95% confidence.

TMSDG also appears to be underpowered to detect effects that might be reasonable to expect from MDA given the estimated effects TMSDG report in studies of those known to be infected. This is notable because the suggestion that there are positive effects from treating individuals known to be infected, yet no impact from treating populations

> 11 These estimates were obtained using the method of Hedges and Pigott (2001). In particular, for a given E <u>E</u> E E E E <u>E</u> E E E effect size, we estimate power as: P P Φ �1.96 − E S S S S �+ Φ(−1.96 − E S S S S ~~)~~ , where Φ is the cumulative distribution function for a standard normal random variable, and S S S P P is the standard error for the average effect size under the random effects model. Reported PP= 1 − EES EES power for a given effect size is the probability that the null hypothesis that the average effect size is zero is rejected at the 0.05 level of significance. The reported MDE is an estimate of the effect size that would PPS deliver a test with 80% power.

12
Croke et al. (2016) containing infected individuals, creates an apparent paradox, since if there are effects of deworming among those known to be infected then one would expect a smaller but still positive average effect of MDA in endemic populations. A simple exercise suggests that this hypothesis cannot be rejected using TMSDG's data. Under the assumption that MDA does not affect weight in uninfected children,<sup>12</sup> the expected average effect of MDA on weight is given by multiplying prevalence by the expected effect on infected children. Thus we calculate each study’s expected MDA treatment effect by multiplying the studyspecific STH prevalence by TMSDG’s random effects estimate for the impact of deworming on infected children, which is 0.75 kg.<sup>13</sup> We generate a standard error for this expected MDA treatment effect by multiplying the study-specific prevalence by TMSDG’s standard error of 0.26 kg. These hypothetical data represent MDA effects that would be reasonable to expect in each setting, with standard errors that reflect our uncertainty about the expected treatment effects.<sup>14</sup> We apply a random effects model to these expected MDA effects, which yields a point estimate of 0.20 kg and a confidence interval of (0.13 kg to 0.28 kg). We cannot reject that this expected average effect of MDA is different from TMSDG’s estimate of 0.08 kg (with p=0.23). Thus, TMSDG’s MDA results are consistent with their estimates of the effect of deworming on infected

> 12 This assumption may not hold if deworming creates epidemiological externalities and also affects weight, but it will hold under the null hypothesis that worms have no impact on weight.

> 13 For the expected effect on infected children, we use TMSDG’s random effects estimate of 0.75 kg (95% CI: 0.24, 1.26). Note that these calculations use estimated prevalence for two studies in TMSDG's sample that do not report prevalence within their study – Alderman et al. (2006) and Awasthi et al. (2008). See Section 4 for more details. 14 In particular, suppose that our beliefs about the average treatment effect on infected children are informed by TMSDG’s results and are characterized by ). Under the assumptions of this exercise, in an MDA setting with prevalence , our beliefs about the MDA treatment effect would be characterized by ). The hypothetical data we consider, then, are NNP NNSSNN(0.75, 0.26<sup>2</sup> consistent with our beliefs about MDA treatment effects. Results we obtain using these hypothetical pp data are results that we consider reasonable to expect given TMSDG’s results on infected children.NNP NNSSNN(0.75pp, 0.26<sup>2</sup> pp<sup>2</sup>

13
Croke et al. (2016) children. Moreover, the TMSDG analysis is underpowered to detect these expected MDA effects. Nearly all expected MDA effects that lie in the 95% confidence interval are smaller than the 80% MDE of 0.28 kg.

Furthermore, it is commonly believed that the health impact of worms is much greater for those with moderate to severe infections than with light infections. Since TMSDG’s estimate for effects on infected children is based on studies with moderate intensity infections on average, while the majority of those infected in most MDA studies are likely to have light intensity infections (see footnote 6 above), we may actually expect the average MDA effect to be even smaller than the 0.20 kg estimated above. This underscores the points that (1) TMSDG’s MDA and screen-and-treat results are consistent with one another, and (2) TMSDG is underpowered to detect effect sizes that we would reasonably expect given TMSDG’s estimated effects on infected children.

Finally, power is also an issue in many of the individual studies which make up the meta-analysis. In low worm infection prevalence populations, very large samples are needed to have sufficient statistical power to pick up effects of mass drug administration. Since low-intensity infections are expected to have little impact, and since intensity rises non-linearly with prevalence (Anderson and May, 1985), only a small fraction of the population in low prevalence environments will have moderate to heavy infections and the necessary increase in sample size may be much larger if weight gain effects are concentrated among those with moderate-severe intensity infections.<sup>15</sup>

> 15 Anderson and May (1985) explain that the relationship between prevalence and intensity is well described by: _Mean Intensity = k*(1-Prevalence)_<sup>_-1/k_</sup> _-k_ , where _k_ is a parameter representing the degree of aggregation or dispersion of the parasite in the population. Estimates of _k_ lie in the range 0.11 to 0.81 (Anderson, Truscott and Hollingsworth, 2014). Taking the derivative with respect to prevalence implies that intensity is expected to rise more than linearly in prevalence.

14
Croke et al. (2016) Together, these factors imply that it will be important to use the most efficient estimators possible, for example, by including baseline values of weight when possible rather than just endline values, in order to improve statistical power. It will also require efforts to employ all the available studies and data.

# **3 Sample and data extraction procedures**

This section describes trial inclusion criteria, the search procedure for identifying studies, and the procedures for extracting data from included trials.

# **3.1 Trial inclusion criteria**

Our analysis includes randomized controlled trials of deworming MDA with multiple doses that include child body weight as an outcome. Following what TMSDG term their “main comparison” (Taylor-Robinson et al. 2015, p. 4), we consider only trials in which multiple doses of deworming treatment were administered, and include treatment effect estimates from the longest follow-up reported.<sup>16</sup> We focus on child weight gain because it is an important nutritional outcome which could potentially be improved over relatively short time horizons.<sup>17</sup> Moreover, there is substantial evidence for this outcome available in existing studies, which we expect to lead to relatively more adequate statistical power.<sup>18</sup> Weight is also highlighted as one of the three primary outcomes examined in

> 16 The median length of study in both our sample and the TMSDG sample is 12 months.

> 17 The studies we consider generally take place in low-income settings where child obesity is not considered a widespread issue.

> 18 For many of the other outcomes they examine, TMSDG include relatively few studies in their metaanalysis. Only weight (10 studies), height (7), and hemoglobin (7) have more than three studies that are aggregated in formal meta-analysis. As height deficiencies and stunting are generally conceived in the nutrition literature as the result of cumulative undernutrition over extended periods, we considered that height was unlikely to respond to deworming over the course of relatively short run trials (the median length is 12 months). Hemoglobin is an important outcome but it is most closely associated with

15
Croke et al. (2016) TMSDG (p.11).<sup>19</sup> Only trials for which a proper intention-to-treat estimate can be obtained are included. Therefore, we require that the study (or trial authors, through personal communication) report outcomes for the population assigned to treatment and comparison groups, independent of whether they received treatment or not.

When estimating the mean effect of MDA on weight, we report results both in the set of trials that take place in settings where the WHO recommends deworming (i.e., those where prevalence of either hookworm, whipworm, or roundworm is over 20%, which is the threshold for annual MDA, or 50%, which is the threshold for multiple annual MDA), and in the full sample (for completeness and comparability to TMSDG).

# **3.2 Search procedure**

We start with the sample of studies included in TMSDG, as a well-known and oft-cited systematic review, for their analysis of the impact of multiple-dose deworming treatment of “all children living in an endemic area” (i.e. mass drug administration, or MDA) at longest follow-up on child weight. We supplement this sample with additional studies we could identify that meet the trial inclusion criteria above. The Campbell Collaboration deworming team generously shared information on additional trials which they identified for use in their own forthcoming systematic review on the impacts of mass deworming. All studies we include were identified by the Campbell Collaboration and any study that they

> hookworm; hookworm infection is the third leading contributor to the global burden of anemia, whereas neither of the other two STH species rank among the top causes of anemia (Kassebaum et al., 2013; Smith and Brooker, 2010). Of the 7 trials in the TMSDG meta-analysis of hemoglobin (which produce 9 treatment estimates, since two are factorial), only one appears to have any significant hookworm prevalence (Dossa and Ategbo, 2001); the other six either do not report hookworm prevalence or report very low values, between 1% — 11%.

> 19 The other two are hemoglobin and cognition.

16
Croke et al. (2016) identified for their analysis of the impact of multiple-dose MDA on weight was considered for inclusion in our sample.<sup>20</sup>

# **3.3 Data extraction and choice of estimator**

Given the importance of statistical power, we sought to use the most precise unbiased estimator available. We also followed guidelines in the _Cochrane Handbook for Systematic Reviews of Interventions_ (Higgins and Green, 2011). We use the following principles for extraction of data and selection of estimates from included trials in our analysis below:

- i. If treatment effects are presented without standard errors, standard errors are calculated using other presented data (e.g., t-statistics, p-values, or 95% confidence intervals), where possible following the formulas provided in the _Cochrane Handbook_ (Higgins and Green 2011, section 7.7.3.3).

- ii. If results are reported in figures rather than in the text or in a table, _Web Plot Digitizer_ software (Rohatgi, 2015) is used to extract numerical estimates from the figures.

- iii. If key information on treatment impacts is missing from a paper (and cannot be derived from what is presented), original microdata (where available) is used to obtain relevant estimates. We also obtain information from trial authors in several cases, through either direct communication or thanks to the generosity of the Campbell Collaboration research team.

- iv. Where studies report multiple treatment impact estimates, we follow the standard in TMSDG and the medical literature of favoring unadjusted estimates. If studies

> 20 The Campbell Collaboration report just became publicly available.

17
Croke et al. (2016) do not report unadjusted estimates and we are unable to obtain them directly, but the studies do report treatment effect estimates adjusted with standard covariates or baseline values (such as child age and sex), these estimates are included in the analysis. (Note that since expected weight gain varies with age, including age as a covariate should generally improve precision of the estimates, and that including age or other pre-determined variables as a covariate should not induce bias).

v. When there was a choice between treatment effect estimates based on a comparison of endline differences and treatment effect estimates based on a comparison of changes from baseline to endline, the “changes” estimate is used, since it typically is more precise. Using the single difference treatment effect typically leads to a substantial loss of statistical precision: when outcomes are highly autocorrelated over time (as is the case for body weight), failure to use baseline values in measurements of treatment effects results in far less precise estimates (Gerber and Green, 2012; McKenzie, 2012). Estimates that take into account baseline information remain unbiased, while typically improving precision, and thus are preferable under standard statistical criteria, such as under the goal of minimizing mean squared error. Following the _Cochrane Handbook_ , when baseline and endline means and measures of variance were present but variance of the changes are missing in the original text, the standard error for changes is calculated using a correlation coefficient for the value between baseline and endline imputed from other studies (Higgins and Green 2011, section 16.1.3.2).

18
Croke et al. (2016)

- vi. In the event of apparent textual contradictions about key parameter values in a trial (for example, language in the study text which reports significant effects versus reported standard deviation values which imply non-significant results), we first try to obtain the original microdata to perform the estimation ourselves. Where this is not possible, we assess which statistics were the primary focus of reporting in the text and contact the original authors for clarification.

- vii. When possible, treatment effect estimates are extracted based on an Analysis of Covariance (ANCOVA) model, rather than estimates based on difference-indifference estimators. The _Cochrane Handbook_ states that since ANCOVA estimates “give the most precise and least biased estimates of treatment effects they should be included in the analysis when they are available” (Higgins and Green (2011), section 9.4.5.2). Properly including baseline weight measures in the analysis is also critical in contexts with baseline imbalance (Kerwin, 2015).

The full sample includes 22 estimates from 20 studies, twice as many as TMSDG. In particular, the full sample includes four studies not identified in TMSDG or unpublished when their review was compiled (Gateff, Lemarinier and Labusquiere 1972; Ostwald et al. 1984; Joseph et al. 2015; Liu et al. 2016); data extracted from six studies discussed by TMSDG but not included in their meta-analysis for the MDA child weight outcome (Willett, Kilama and Kihamia 1979; Miguel and Kremer 2004; Ndibazza et al. 2012; Wiria et al. 2013; Stephenson et al. 1993; Gupta and Urrutia 1982); and improved estimates from three studies that are included in the TMSDG meta-analysis (Sur et al. 2005; Hall et al. 2006; Awasthi and Pande 2001). Note that we classify Stephenson et al. (1993) as an MDA trial; the Cochrane authors classify it in this way in their 2012 Review but change the 19
Croke et al. (2016) classification system in the 2015 update; we retain the 2012 classification in this analysis.<sup>21</sup> See Appendix A for detailed information on each individual study in the full sample.

# **4 Hypothesis tests and estimation strategy**

In light of TMSDG's conclusion that there is “substantial evidence” of no impact of deworming MDA, we first report a test of the hypothesis that the true impact of multipledose deworming on weight is zero in all settings. This involves testing the null hypothesis that β=0 in a standard fixed effects meta-analysis estimate:<sup>�</sup> ıı EE where is the estimated effect in study i; YY = ββ+ σσ is the true deworming treatment effect, and is a random variable, representing measurement error, assumed to be distributed YY<sup>�</sup> ββ EE σσnormally, with mean zero and a standard deviation equal to the standard error of the estimated treatment effect in each study. Rejection of this null hypothesis implies that deworming affects child weight in at least some circumstances.

We then report the estimated average impact using a random effects model:

> 21 TMSDG state that “We changed the classification of Stephenson et al. (1989) and Stephenson et al. (1993). Previously these trials were in the ‘all children in an endemic area’ category, whereas now they are classified in the ‘children with infection.’ This decision was based on reviewing the trials with parasitologists and examining the prevalence and intensity of the infection where clearly the whole community was heavily infected” (Taylor-Robinson et al., 2015, p. 154). It is worth noting that although TMSDG exclude Stephenson et al. (1993), they include Watkins, Cruz and Pollitt (1996); the highest recorded worm baseline prevalence in Watkins, Cruz and Pollitt (1996) by STH species is 92% (for ascaris); the highest prevalence in Stephenson et al. (1993) is also 92% (for whipworm). Thus this reclassification does not appear to have been done systematically by worm prevalence. In our view, assessing the merits of the WHO policy by including studies in environments with prevalence below WHO thresholds while excluding MDA studies in areas with high prevalence may lead to risk of bias. Since our goal is to examine the effect of MDA, we retain the 2012 classification. Stephenson et al. (1989) is a single dose deworming trial so does not enter into the present meta-analysis but Stephenson et al. (1993) has one multi-dose treatment arm, which we incorporate. See Appendix A.3 for more detail.

20
Croke et al. (2016) where is the underlying true average effect, is a normally distributed random variable EE denoting the difference between the average effect and the effect in the particular context, ββ μμ and represents measurement error due to sampling variation, which is assumed to be EE captured in the study-specific standard errors. σσ Finally, under the assumption that deworming has no effect on uninfected children, one can calculate for each study an implied estimated effect on weight for infected children as the estimated intention-to-treat effect divided by prevalence. If one takes the estimated prevalence to be accurate and not subject to measurement error, then standard errors for these estimated effects are straightforward to compute. One can then apply a random effects model to estimate the average treatment effect on infected children. In a few cases, a study does not report an exact value for prevalence, but we are able to identify whether the study has below or above 50% prevalence. If prevalence is above 50%, we then compute the average prevalence among all studies in the sample reporting greater than 50% prevalence and assign that value to the study. We proceed similarly for studies with below 50% prevalence that do not report an exact value for prevalence.<sup>22</sup>

# **5 Results**

Subsection 5.1 first replicates the results of the TMSDG subsample. Subsection 5.2 tests the hypothesis of a common zero effect. Finally, subsection 5.3 reports estimated effects of MDA on weight.

# **5.1 Verifying replication of results in the TMSDG sample**

> 22 See Appendix A.7 for more detail on how prevalence categories were assigned when STH prevalence was not reported in the study text.

21
Croke et al. (2016) We call the sample of 11 treatment effect estimates from 10 studies used in the TMSDG meta-analysis for the impact of MDA on child weight gain the “TMSDG sample.” See Table 1 for a list of these studies. Figure 1 verifies that our estimation procedure yields results similar to TMSDG when using this sample. In particular, this figure presents a forest plot with TMSDG estimates; inserting the effect sizes, standard errors, and sample sizes reported for each of these studies in TMSDG's text and figures into the relevant formulas provided in Borenstein, Hedges and Rothstein (2007) using the R statistical software package replicates the TMSDG results.

Using the data presented in TMSDG, the hypothesis of a common zero effect of deworming on child weight gain is rejected at the 10% level (p=0.089). Figure 1 shows the fixed effect estimate used to test this hypothesis.

The null hypothesis of a common zero effect is more strongly rejected when applying either of two study classification approaches used in previous Cochrane Reviews (prior to TMSDG).

1. The first versions of the Cochrane Review (Dickson et al. 2000; Dickson et al. 2007) did not create separate categories for test-and-treat (i.e. treatment only of children who have been diagnosed with STH) and MDA studies (i.e. treatment of the whole population), but rather considered all the data together. At a minimum, reporting results with the full sample before turning to the subgroup analysis seems reasonable, since if deworming has a positive effect on infected individuals, and if there is no effect on uninfected individuals, then deworming must have a positive effect on weight in a population that includes infected individuals. In the case of the TMSDG dataset, using all of the multiple dose at longest follow-up studies

22
Croke et al. (2016) would only involve the addition of the one multiple dose study the authors identify but classify as separate from the MDA studies – Stephenson et al. (1993).<sup>23</sup> 2. The 2012 Cochrane Review changed this approach, however, introducing a distinction that effectively distinguished between test-and-treat and MDA studies (Taylor-Robinson et al. 2012). Applying this classification used in the 2012 Cochrane Review to the studies in TMSDG also leads to the inclusion of Stephenson et al. (1993), which was classified as an MDA study by TaylorRobinson et al. 2012 (or, as a “target population treated” study, using the language in that review) rather than a test-and-treat study.

Thus, applying either of these procedures from Cochrane Reviews prior to TMSDG results in the addition of Stephenson et al. (1993) to the TMSDG sample. With Stephenson et al. (1993) classified in this way but otherwise using the data set in the TMSDG study, the null hypothesis of a common zero effect is strongly rejected with a p-value of 0.009 (see Table 2, row 2).

# **5.2 Testing the hypothesis of a common zero effect**

We now turn to the full sample of 22 treatment effect estimates. Table 3 shows that the null hypothesis of a common zero effect is rejected at p<0.001 in the full sample, in areas with prevalence above the thresholds at which WHO recommends MDA (20% prevalence), and in areas where it recommends multiple annual dose MDA (50% prevalence). This result is not reliant on the addition of any one study to the TMSDG sample: any one of six updates leads to rejection of the hypothesis of a common zero effect with p<0.05 (see Table 2).

> 23 Note that we believe that Stephenson et al. (1993) should be included in the set of MDA studies, since individuals were treated without first being screened, as discussed in Appendix A.3.

23
Croke et al. (2016) Figure 2 shows point estimates and confidence intervals for the effect of mass deworming drug administration on weight from each of the studies included in the full sample.

The rejection of a common fixed effect of zero implies that MDA deworming affects child weight in at least some circumstances. If effects are positive in some circumstances, then unless they are negative in other circumstances, average effects must be positive. Yet, there is no scientific reason to believe that deworming has negative side effects on weight. With only one negative estimate significant at the 5% level out of 22 estimates in the full sample (Figure 2), the patterns in the data seem consistent with the hypothesis that the true effect of MDA on weight is never negative. In future work, we hope to more formally examine if this hypothesis can be rejected. However, for the sake of comparability, we follow TMSDG in the next section by imposing that the distribution of true effects is normal around some unknown mean.

# **5.3 Estimating the impact of deworming**

Table 3 reports random effects estimates. In the full sample, the estimated weight gain effect is 0.134 kg [CI: 0.031,0.236; p=0.01]. Of course, the full sample includes trials conducted in low infection prevalence areas where the WHO does not currently recommend mass deworming.<sup>24</sup> In areas with greater than 20% prevalence, where the

> 24 TMSDG examine outcomes by subgrouping based on infection prevalence, but they split the data into three subgroups, one containing only two studies, thus limiting statistical power for each test. In the weight gain multiple dose comparison, TMSDG analyze 5 studies from low prevalence settings (defined as less than 50% infection), 4 from medium prevalence settings (50% to 70% prevalence), and 2 from high prevalence settings (over 70% prevalence). They take this tripartite division from an earlier WHO framework (World Health Organization, 2002). Creating groupings with only a few studies makes the resulting estimates far less precise. The _Cochrane Handbook_ notes that “when there are few studies or if the studies are small, a randomeffects analysis will provide poor estimates of the width of the distribution of intervention effects” (Higgins and Green, 2011, section 9.5.4). Note that this tripartite division was not pre-specified in the original

24
Croke et al. (2016) WHO recommends MDA deworming, the estimated treatment effect is 0.148 kg [95% CI: 0.039,0.258; p=0.008]. In areas with more than 50% prevalence, where the WHO recommends multiple doses annually, the estimated effect is 0.182 kg [CI: 0.070,0.293; p=0.001].

Our full sample estimate has more statistical power than TMSDG. Using the same approach as above, we find an MDE of 0.15 kg in the full sample, roughly half the MDE of 0.28 kg using the TMSDG sample. Using the random effects estimate of 0.13 kg, we find that the full sample analysis has 72% statistical power to detect a significant effect at 95% confidence. This compares to 18% statistical power to detect an effect of 0.08 kg, or 31% statistical power to detect an effect of 0.13 kg using the TMSDG sample (see Figure 3).<sup>25</sup> This full sample effect remains robust (at p<0.05) when any individual trial estimate is dropped from the meta-analysis, as shown in Table 4. The level of significance with which the null hypothesis that the mean effect is zero can be rejected remains high even when simultaneously dropping any two of the 22 estimates: among the 231 possible combinations of two studies that could be dropped simultaneously, in 96% of cases the estimated effect remains statistically significant at p<0.05, and the largest p value is just 0.067 (not shown). The results are similarly robust to dropping any one or two studies in

> Cochrane pre-analysis plan (Dickson, Awasthi, and Demmellweek, 1997). Examining all studies conducted in environments in which the WHO recommends MDA while excluding studies where the WHO does not recommend MDA would enhance policy relevance.

> 25 Using an effect of 0.13 kg and the value of estimated in the full sample (0.038 in the full sample as opposed to 0.074 in the TMSDG sample), we find that the TMSDG analysis has 44% statistical power to detect a significant effect at 95% confidence. ττ̂<sup>2</sup>

25
Croke et al. (2016) the subsample of studies with prevalence greater than 20% and those with prevalence greater than 50%.<sup>26</sup> The full sample estimate of 0.134 kg comes from studies with average worm prevalence of 51%. Assuming treatment only affects weight in the worm-infected population, this implies an average effect of roughly 0.301 kg among those with worms (calculated by dividing estimated impact by worm prevalence for each study and applying a random effects model).<sup>27</sup> This in turn likely represents the average of a considerably larger effect among the small proportion of those infected who have moderate or severe intensity infections and a considerably smaller effect among the majority of those infected, who have light-intensity infections.

While the average weight gain is fairly modest, it is far from negligible relative to the very low cost of deworming. Given the estimated effect of 0.134 kg and the estimated cost of $0.60 per person treated for two treatments per year (in the multiple dose context we focus on in this meta-analysis), the estimated cost per kg gained is ($0.60)/(0.134kg) = $4.48 per kg. For reference, it is worth comparing this to nutritional programs aimed at similar populations, in particular, school feeding programs. Kristjansson et al. (2007) conduct a Cochrane Review of the impact of school feeding, and Galloway et al. (2009) combine these results with data on costs to estimate the cost effectiveness of school feeding. They estimate that over a one year period, “the cost of an extra kilogram of weight ranged from $112 to $252 in the RCTs and $38 to $86 in the [controlled before-and-after

> 26 Results are also robust to maintaining the treatment effects and standard errors reported in TMSDG (rather than using updating treatment effects or standard errors for Awasthi 2001, Hall 2006, and Sur 2005) and only altering the TMSDG sample by incorporating previously-excluded studies.

> 27 Note that this calculation includes estimated prevalence for three studies in the full sample which do not report prevalence within their study – Gateff, Lemarinier and Labusquiere (1972), Alderman et al. (2006), Awasthi et al. (2008). See Table 1 and Section 4 for more details.

26
Croke et al. (2016) studies]” (p.177-8). This suggests that just focusing on the weight outcome, deworming is highly cost effective relative to another widely implemented intervention. As noted by Galloway et al. (2009), school feeding is implemented in over 72 countries by the World Food Programme alone. To the extent that school feeding programs aim to produce child weight gain, deworming is likely to be a highly cost effective option for policymakers who already support school feeding.<sup>28</sup> Of course, a complete cost-effectiveness analysis of mass deworming treatment would also need to consider effects on later educational and labor market outcomes, in addition to child weight gains, and as noted above these are often substantial (Ahuja et al., 2015; Baird et al., 2016; Bleakley, 2007; Croke, 2014; Ozier, 2015).

# **6 Conclusion**

We began with the question of whether the expected benefits of mass drug administration according to WHO guidelines for deworming exceed the cost.

To summarize, the null hypothesis of a common effect of zero weight gain from multiple-dose mass drug administration is rejected at the 10% level using the TMSDG data. Employing either of two classification systems used in previous Cochrane Reviews would lead to rejection at p<0.01. Any one of five other updates to the data leads to rejection of the null hypothesis at the 5% level. Combining all updates leads to rejection with p<0.001.

> 28 Note that this comparison is imperfect for a number of reasons, including that the cost figures for deworming are for India whereas those for school feeding are based in three African countries. Moreover, both school feeding and deworming may affect other outcomes. However, it is unlikely that appropriate adjustments for these factors would overturn the conclusion that deworming is highly cost effective in increasing weight relative to school feeding programs, given that the cost per kg of weight gain is an order of magnitude higher for school feeding. Moreover, while school feeding may also promote school participation, deworming, too, has been found to be highly cost effective in increasing school participation (Miguel and Kremer, 2004).

27
Croke et al. (2016) Reasonable people may disagree about statistical methods for analyzing data. However, at a minimum, it seems clear that implementing MDA generates child weight gains in some circumstances. Since the null hypothesis of a common zero effect is rejected, MDA must have positive impacts in at least some environments. This implies that if one accepts the standard view that anthelmintic drugs have no substantial negative side effects, the expected effect of deworming on child weight is positive.

Applying standard approaches from the _Cochrane Handbook_ to a larger set of studies yields an estimated average effect of deworming on weight of 0.134 kg [95% CI: 0.031,0.236], corresponding to an estimated average effect of 0.301 kg [CI: 0.071, 0.530] among those with worm infections. While this effect is modest, it is substantial relative to the cost of deworming, and suggests that deworming is many times as cost effective as widely implemented school-feeding programs at enabling weight gain among school age children in low-income populations. Moreover, the findings that deworming improves nutrition in at least some environments implies that the literature on the long-run educational and economic impacts of deworming cannot be dismissed on _a priori_ grounds, and this literature suggests that the expected benefits of MDA greatly exceed the cost.

Our results also suggest that the data from studies of mass deworming are consistent with the data from studies of deworming of those known to be infected, given the much lower prevalence and intensity of infection in MDA studies than in test-and-treat studies, and the substantial confidence intervals around both the MDA and test-and-treat estimates.

A key lesson of this paper is that large samples are needed to have adequate statistical power to pick up a minimum detectable effect (MDE) that corresponds to what would be needed to form a policy judgment that the expected benefit of deworming is less 28
Croke et al. (2016) than the cost.<sup>29</sup> This implies that analyses which divide up the set of available studies will likely be underpowered, limiting the scope for further subsample tests. While the results here suggest that overall, multiple-dose MDA increases child weight, they also suggest that if MDA had similar impacts on weight across drug type or worm species, a meta-analysis focusing on any one species of worm or drug may well be underpowered. We therefore think it would be appropriate for any future studies designed to explore heterogeneity, across worm species or drug type, for example, to report a test of the hypothesis that the average effect of MDA for each worm species or each drug is the same, rather than to simply test the hypothesis that the effect of any one individual drug or MDA against any one species has zero impact on weight. Beyond its relevance for health research, greater awareness of the limitations of under-powered meta-analyses will become increasingly important as more social scientists conduct meta-analyses (Vivalt, 2015).

A further methodological lesson is that it is appropriate to explicitly test the null hypothesis of a common zero effect. Finally, in evaluating policy, it is appropriate to focus on all mass-drug administration studies conducted in environments where mass deworming is recommended under WHO guidelines rather than to mix in studies from environments in which worm prevalence is sufficiently low that the WHO does not recommend mass treatment, or to selectively exclude studies of mass drug administration conducted in highprevalence environments.

> 29 The estimated effect in the full sample is about 50% greater than the estimate in the TMSDG sample, though well within the confidence interval of (-0.11 kg, 0.27 kg) found by TMSDG in their meta-analysis of the impacts of multiple-dose MDA on weight. However, the confidence interval shrinks by 47% to (0.03 kg, 0.24 kg) with the full data set. Thus, incorporation of additional information into the analysis helps address the problem of insufficient power in TMSDG.

29
Croke et al. (2016) While we have argued that deworming MDA is cost effective based on its impact on child weight alone, there is evidence that deworming also leads to gains in other outcomes (Ahuja et al. 2015). TMSDG aggregate data from a more limited number of studies for outcomes other than weight, and some of these are from low-prevalence environments where the WHO does not recommend deworming, further reducing statistical power. This means that for each outcome assessed individually, it is typically impossible to reject either the hypothesis of no average effect, or the hypothesis of effects large enough to make deworming cost effective. In such a setting, it may be appropriate to consider the joint hypothesis that there is no impact on any of the child outcomes considered. We hope to do this in future work.

We have begun to explore heterogeneity of impact with covariates suggested by the scientific literature, including prevalence and intensity, age, and whether the study design captures epidemiological spillovers. One could also examine heterogeneity by helminth type, drugs used, and co-morbidities. However, given the limited number of studies and hence limited degrees of freedom, scope for examining heterogeneity while maintaining statistical power is limited. In future work, we hope to systematically examine heterogeneity, but here we simply capture heterogeneity using random effects estimation.

We follow TMSDG in assuming a normal distribution of the effect of MDA across study environments, but future work could relax the assumptions of symmetry and normality on the distribution of effects across studies and estimate these from the data. There is no reason, _a priori_ , to expect that the distribution of deworming effects follows such a distribution. In fact, the underlying science naturally suggests a non-symmetric 30
Croke et al. (2016) distribution, with positive effects in some cases and negligible effects in cases with low prevalence and intensity of infections.

We hope to examine a Bayesian, rather than frequentist, approach to meta-analysis for policy analysis in future work. The implicit loss function implied by requiring 95% confidence to undertake MDA without regard to the statistical power of the test is one in which there is a high cost of a false positive and a low cost of a false negative. That might be appropriate if, for example, the Food and Drug Administration were considering a drug that might have side effects. However, in the deworming context, the drugs have already been through regulatory approval, the monetary cost of deworming is low, and there is no evidence of serious side effects, while there is at least some potential that deworming has large long-run benefits (Ahuja et al. 2015). Thus, the cost of a false positive is low while the cost of a false negative is potentially quite substantial.

A Bayesian approach to estimating whether the expected benefit of MDA according to WHO guidelines exceeds the cost would start with a prior on the effect of deworming. Studies of the impact of treatment on those known to be infected provide a natural prior. It would factor in the range of benefits that have been estimated from deworming, each with an attached probability weight in order to assess whether summing across the range of potential benefits times their likelihood yields a benefit greater than the estimated cost. Since the net present value of the long-run educational and economic benefits has been estimated as more than one-hundredfold that of the cost (Baird et al. in press), assessing even the subjective probability of these benefits would likely lead to the conclusion that the expected benefits of MDA exceed their cost.

31
Croke et al. (2016) 32