# 7. Implementing Successful Small Interventions at a Large Scale Is Hard

_<mark>Robert Cull and David McKenzie</mark>_ The last decade has seen a large increase in the number of policy evaluations, with many pilot programs tested rigorously for their impacts. This is a boon to governments looking to practice evidence‐based policy making. However, even when pilots and local development interventions have proven very successful, they have often been difficult to scale up in a cost‐effective way to achieve development impact on a large scale. Conversely, several development interventions that have managed to achieve impressive scales at relatively low costs are increasingly under scrutiny for their lack of transformative impacts on the lives of the poor. This raises key questions about why promising programs are so hard to scale, and what can be done to enhance the impacts of large‐scale programs.

## **Taking successful pilots to scale is challenging**

A recent meta‐analysis of more than 600 research papers covering 20 different types of development interventions found that the larger the study, the smaller the size of the effect, and that programs implemented by governments tend to have smaller effect sizes than academic or NGO‐implemented programs.<sup>75</sup> Moreover, several recent experiences provide stark examples of successful pilot programs not having the intended impacts when taken to scale:

- A first example comes from efforts to reform education in Kenyan primary schools by changing the way teachers are hired.<sup>76</sup> New teachers offered a fixed‐term contract by an international NGO significantly raised test scores. However, when the Kenyan government offered identical contracts, this produced zero impact.

- A second example is seen in efforts to help low‐income agricultural workers in rural Bangladesh to be less vulnerable to seasonal food insecurity during the agricultural lean season. Researchers tested the impact of offering small grants and interest‐free loans to enable these workers to temporarily migrate to cities in search of work.<sup>77</sup> This was incredibly successful in increasing internal migration flows, delivering higher income and consumption to program participants. The evidence was so striking that Givewell, a charity evaluator, recommended this program as one of the best values for money approaches of helping the poor, and an NGO scaled up the program dramatically to reach over 150,000 households in 2017. However, evaluations of this scaled‐up program found it was unsuccessful in increasing migration rates, and the program was shut down.

> 75 <mark>“ ” Vivalt, Eva (forthcoming).</mark> <u><mark>How Much Can We Generalize from Impact Evaluations? ,</mark></u> _<mark>Journal of the European Economics Association</mark>_ <mark>.</mark>

> 76 <mark>Bold, Tessa, with Mwangi Kimenyi, Germano Mwabu, Alice Ng'ang'a and Justin Sandefur. “</mark> Experimental Evidence on Scaling Up Education Reforms in Kenya” <mark>,</mark> _<mark>Journal of  Public Economics</mark>_ <mark>, December 2018.</mark>

> 77 G. <mark>Bryan, S. Chowdhury and A. M. Mobarak, “Under‐investment in a Profitable Technology: The Case of Seasonal Migration in Bangladesh,</mark> ” _<mark>Econometrica ̧</mark>_ 82(5): 1671‐1748. September 2014.

24