# **3. Challenges in Measuring Impact**

Impact evaluations which measure the effects of business training programs on business performance rely primarily on survey data to measure outcomes. In order to obtain credible and useful estimates, studies must have sufficient power, measure impacts over an appropriate time horizon, deal with survey attrition and selective survival and start-up of firms, and address the possibility that training changes how firms report business outcomes even if it doesn’t change those outcomes. We discuss each of these challenges and assess how well existing studies have met them in order to provide context for the results discussed in the next section.

# _Power_

The power of a statistical test is the probability that it will reject a null hypothesis given that the null hypothesis is false. A starting point for most business training evaluations is to test the null hypothesis that the intervention had no effect, so the power of the experiment is a measure of the ability to detect an effect of training if such an effect does exist. The key determinants of the power of a study are the size of the sample, how much heterogeneity there is in the sample (the more diverse the set of firms the harder it is to measure a change for them), whether the intervention takes place at an individual or group level (power is lower for a given sample size when treatments are allocated at the group level), and the size of the treatment effect. Low take-up rates dilute the treatment effect, reducing power.

Table 5 compares different studies in terms of these components of power. Most studies conduct individual level randomization, with the exception of several studies working with microfinance groups, where different repayment groups are allocated to treatment or control, and Calderon et al. (2012) who first randomize at the village level, and then at the individual level within the treatment villages. A typical study involves approximately 200 to 400 individuals or groups in each of the treatment and control, although sample sizes have been smaller for the studies based on specific industrial clusters (Mano et al, 2012; Sonobe et al, 2011). A useful summary statistic of the cross-sectional heterogeneity in baseline firms is the coefficient of variation (C.V.) of profits or revenues, which is the ratio of the standard deviation and the mean. The two studies with lowest C.V.s are both studies which restrict the heterogeneity in firms eligible for the study – de Mel et al. (2009) required firms to have baseline profits below 5000 Rs per month (US$43), while Berge et al. (2011) restricted training to firms with loan sizes in a reasonably narrowly defined range. In contrast, most studies of microfinance clients or firms in an industrial cluster contain a much wider mix of firms, resulting in coefficients of variation which can exceed 2 or more. The more different the firms are, the harder it is to detect changes in their average outcomes arising from treatment.

To give a sense of the power of these different studies, we calculate the power to detect a 25% and a 50% increase in profits and revenues, given the C.V., sample size, and training take-up rate of each study.<sup>4</sup> We assume analysis is by an ANCOVA specification (outcome regressed on treatment and the

> 4 Power calculations were done using the sampsi command in Stata, and a test size of 0.05. See McKenzie (2012) for more discussion of the validity of the underlying assumptions needed for these calculations in firm experiments.

8