metadata: authors: 0: author: McKenzie, David
count: World
docna: 0: docna: Beyond baseline and follow-up : the case for
            more t in experiments
docty: Policy Research Working Paper
subtopic: Private Sector Economics,Pharmaceuticals Industry,Health Care Services Industry,Rural Microfinance and SMEs,Microfinance,Pharmaceuticals & Pharmacoeconomics,Educational Sciences,Private Sector Development Law,Marketing
teratopic: Macroeconomics and Economic Growth,Health, Nutrition and Population,Science and Technology Development,Education
abstracts: cdata!: The vast majority of randomized
            experiments in economics rely on a single baseline and
            single follow-up survey. If multiple follow-ups are
            conducted, the reason is typically to examine the trajectory
            of impact effects, so that in effect only one follow-up
            round is being used to estimate each treatment effect of
            interest. While such a design is suitable for study of
            highly autocorrelated and relatively precisely measured
            outcomes in the health and education domains, this paper
            makes the case that it is unlikely to be optimal for
            measuring noisy and relatively less autocorrelated outcomes
            such as business profits, household incomes and
            expenditures, and episodic health outcomes. Taking multiple
            measurements of such outcomes at relatively short intervals
            allows the researcher to average out noise, increasing
            power. When the outcomes have low autocorrelation, it can
            make sense to do no baseline at all. Moreover, the author
            shows how for such outcomes, more power can be achieved with
            multiple follow-ups than allocating the same total sample
            size over a single follow-up and baseline. The analysis
            highlights the large gains in power from ANCOVA rather than
            difference-in-differences when autocorrelations are low and
            a baseline is taken. The paper discusses the issues involved
            in multiple measurements, and makes recommendations for the
            design of experiments and related non-experimental impact evaluations.
repnme: repnme: Beyond baseline and follow-up : the case
            for more t in experiments
display_title: Beyond baseline and follow-up : the
            case for more t in experiments
listing_relative_url: /research/2011/04/14103583/beyond-baseline-follow-up-case-more-t-experiments
topicv3: Private Sector Development m1326060 1921,Education m1327048 470,Microfinance m1327227 367,Pharmaceuticals m1327740 1350,Health m1504661 3480
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Macroeconomics and Economic Growth,Health, Nutrition and Population,Science and Technology Development,Education
no_of_pages: 36
model_extractions: WPS5639 Policy Research Working Paper 5639 Beyond Baseline and Follow-up The Case for More T in Experiments _David McKenzie_ The World Bank Development Research Group Finance and Private Sector Development Team April 2011
Policy Research Working Paper 0

# **Abstract**

The vast majority of randomized experiments in economics rely on a single baseline and single follow-up survey. If multiple follow-ups are conducted, the reason is typically to examine the trajectory of impact effects, so that in effect only one follow-up round is being used to estimate each treatment effect of interest. While such a design is suitable for study of highly autocorrelated and relatively precisely measured outcomes in the health and education domains, this paper makes the case that it is unlikely to be optimal for measuring noisy and relatively less autocorrelated outcomes such as business profits, household incomes and expenditures, and episodic health outcomes. Taking multiple measurements of such outcomes at relatively short intervals allows the researcher to average out noise, increasing power. When the outcomes have low autocorrelation, it can make sense to do no baseline at all. Moreover, the author shows how for such outcomes, more power can be achieved with multiple follow-ups than allocating the same total sample size over a single follow-up and baseline. The analysis highlights the large gains in power from ANCOVA rather than difference-in-differences when autocorrelations are low and a baseline is taken. The paper discusses the issues involved in multiple measurements, and makes recommendations for the design of experiments and related non-experimental impact evaluations.

This paper is a product of the Finance and Private Sector Development Team, Development Research Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at dmckenzie@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Beyond Baseline and Follow-up:**

**The Case for More T in Experiments**<sup>*****</sup>

David McKenzie, _World Bank_ _Keywords:_ Randomized Experiments; Multiple Measurements; Program Evaluation.

_JEL codes:_ O12, C93.

> * I thank Chris Woodruff and participants at the Warwick University summer workshop on Firms in Development for conversations and questions which lead to this article; Chris Blattman, Miriam Bruhn, Jishnu Das,  John Gibson and Berk Özler for helpful comments; Matthew Groh for research assistance; and Miriam Bruhn, Jishnu Das, and Markus Goldstein for sharing their data.
# **1. Introduction**

The number of randomized experiments being conducted in economics has exploded over the past decade, especially in development economics. The vast majority of these studies use only a single baseline and single follow-up survey, or a single follow-up with no baseline. This has been the case in the early randomized experiments looking at education (e.g. Glewwe et al, 2004) and health outcomes (e.g. Miguel and Kremer, 2004), and has remained true as experiments have expanded to consider other interventions and outcomes, such as recent high-profile experiments in microfinance (Karlan and Zinman, 2010; Banerjee et al, 2010). In the rare cases where multiple post-treatment survey waves have been conducted, they have been typically taken relatively far apart in time, with the goal of examining whether the treatment effect differs in the short-term and medium-term. For example, Banerjee et al. (2007) examine impacts of educational interventions at one year and two year horizons. Indeed, so much is the paradigm of baseline plus follow-up accepted that the excellent toolkit for randomization of Duflo et al. (2008) does not discuss at all the possibility of doing more than one pre-treatment or posttreatment round of surveying, let alone the choice of how many such rounds.

In contrast, the clinical trials literature has noted the potential advantages of taking repeated measures of outcomes of interest not just to study the time course of treatment effects, but to obtain more precise estimates of effects around particular endpoints (Frison and Pocock, 1992). Vickers (2003) argues that the number of repeat measures should be a key design choice in conducting experiments, and concludes that the benefit of such additional measures is of greatest value when the autocorrelation of measures is low, such as with episodic conditions like headaches. Such a description would certainly seem to fit key economic outcomes like business profits, and incomes and consumption of the poor. As the recent work by Collins et al. (2009) makes abundantly clear, one of the key difficulties of living on $2 a day is that people don’t receive $2 every day, but rather a highly irregular stream of income. Measuring microenterprise income, in particular, can be difficult, with the resulting data typically having large heterogeneity in reported profits among reasonably similarly sized firms (de Mel et al, 2009). While some of this likely reflects measurement error, Fafchamps et al. (2010a) report that Ghanaian microenterprise owners confirm 85 percent or more of changes in profits above 150 percent or 2
below -60 percent as genuine, reflecting seasonality and the high degree of idiosyncratic variability facing microenterprise owners.

As a result of the high variability and low autocorrelation in economic outcomes like firm profits, income, and expenditure among poor households, there is much to be potentially gained by taking multiple measures of these outcomes at relatively short intervals and averaging over them when estimating treatment effects. This is the approach used in de Mel et al. (2008), who use a baseline and 8 quarterly follow-up waves of business profits and in Fafchamps et al. (2010) which uses two survey rounds before randomization and a further 4 quarterly follow-up surveys.<sup>1</sup> However, this approach seems to be the exception rather than the rule. The premise of this paper is that many field experiments are making a suboptimal choice of how many rounds of surveys to collect, and that the default choice of a single baseline and single follow-up study is unlikely to be optimal in many cases. This paper aims to combine insights from the medical literature on repeated measurement, new analysis of the formulae underlying power calculations, and experience from these studies using multiple measures in economics to provide a practical guide for researchers designing experiments. While our discussion will be in terms of experiments, many of the same issues and therefore lessons also apply for design of non-experimental impact evaluation designs, such as matched difference-in-difference estimation.<sup>2</sup> An additional contribution of the paper is to draw to the attention of empirical researchers in economics the large improvement in power that can arise when estimating treatment effects via an Analysis of Covariance (ANCOVA) estimation compared to using the more common difference-in-difference specification. The improvement in power is greatest when the autocorrelation is low – intuitively when the baseline data have little predictive power for future

> 1 Bloom et al. (2010) provides an example of very large T, with 114 weekly observations on firm output, quality, and inventory levels.

> 2 Gibson and McKenzie (2010) provide an example where multiple rounds of follow-up data are averaged to get more precise measurements of consumption and income in a matched difference-in-differences impact evaluation of a seasonal migration program. One key difference with non-experimental evaluations is that the presence of a baseline becomes more valuable to allow one to control for baseline differences across individuals; whereas randomization ensures these baseline differences are balanced on average. In addition, multiple rounds of pretreatment data are also of additional use in non-experimental studies to show parallel trends in difference-indifferences analysis, or to rule out or control for the presence of pre-program effects like an Ashenfelter dip. Multiple pre- and post-treatment data can also be combined to conduct overidentification tests to distinguish between different data generating processes (Ashenfelter and Card, 1985).

3
outcomes, it is inefficient to fully correct for baseline imbalances between treatment and control groups.

Section 2 begins with a theoretical discussion of what power calculations tell us about the gain to be had from using more rounds of data, and provides examples for common types of economic data. Section 3 then derives implications for the choice of how many pre-treatment and post-treatment waves to use. The bottom line is that collecting multiple measurements posttreatment will make most sense when the data have low autocorrelation; whereas baseline data is of most use when the autocorrelation is high (although there are of course other important uses of baseline data apart from improving the power to estimate treatment effects, such as for examining treatment heterogeneity, and so one may still wish to do a baseline for these other reasons – something I discuss at the end of the paper). It would appear that for many economic outcomes, experiments with a fixed budget to conduct a total of K surveys would have greater power when dividing that K over multiple post-treatment rounds than having a baseline and single follow-up. Section 4 then discusses additional practical issues which are likely to arise when considering multiple measurements in economic experiments, many of which are not common concerns in clinical trials. Section 5 concludes.

# **2. What is the gain from more rounds?**

Let _Yi,t_ be an outcome of interest for household or firm _i_ in survey round _t_ . Consider an intervention which assigns units to receive a binary treatment (such as getting business training or not, or getting a conditional cash transfer or not). Suppose there are _m_ pretreatment survey rounds (labeled –( _m_ -1) through 0) and _r_ post-treatment survey rounds (labeled 1 through _r_ ). A common method in economics for estimating the treatment effect<sup>3</sup> of the intervention on the outcome of interest is via the following difference-in-differences specification: where _EVERTREATi_ is a dummy variable which takes value one if unit _i_ is assigned to the treatment group and zero if it is assigned to the control group, the are time dummies, and

> 3 I focus on estimation of intention-to-treat effects here, but the same points made here also apply for measurement of the treatment effect on the treated and other treatment effects of interest.

4
_TREATi,t_ takes value one if unit _i_ has been assigned to receive treatment by time _t_ (that is for t=1,2,...r), and zero otherwise. The treatment effect of interest is then given by γ.

Note that γ does not contain a _t_ subscript: either the treatment is assumed to have a constant level effect, or the treatment effect of interest is taken to be the average treatment effect over the _r_ post-treatment rounds. Clearly a further, very important, use of multiple postintervention surveys is to collect information on the trajectory of treatment impacts (e.g. Woolcock, 2009), since the short and long-run impacts of some policies may differ dramatically. However, unless one imposes some structure on the form these impact trajectories can take, collecting more rounds of data merely increases the number of points in time at which these impacts can be measured, but does not improve the power of an experiment for measuring these impacts. Even in this case where impacts at different points in time are of interest, one can consider multiple measurements located around each time horizon of interest, and thereby improve power. For example, if a study is interested in knowing whether the impact of microfinance differs at a one year horizon versus a three year horizon, taking measures of microenterprise profits at 11, 12 and 13 months after treatment, and at 35, 36, and 37 months after treatment can be used to improve the accuracy of impacts at around these horizon points, by assuming the treatment effect is constant or considering an average treatment effect in the neighborhood of the horizon of interest. The analytics and ideas of the remainder of the paper can then be applied to estimating the impact at a particular horizon by taking more measurements in neighborhoods around these horizons.

# **2.1.The case of a single baseline and single follow-up**

Consider first the case of a single baseline and single follow-up. Then we see that the least squares estimator of γ is simply the difference in means between the treatment and control groups at follow-up less the difference in means between the treatment and control groups at baseline: Then sample size to detect a given treatment effect at a specified power level and significance level is a linear function of the variance of this estimator. Assume the are independent across individual units with cross-sectional variance σ<sup>2</sup> , but may be autocorrelated over time for the 5
same unit, with autocorrelation ρ. The variance of the difference-in-differences estimator is then easily seen to be 4σ<sup>2</sup> (1- ρ)/ _n_ where _n_ is the size of the treatment and control groups. Compare this to the simple “post” estimator, which uses only the difference in means for the follow-up survey, ignoring the baseline: which has variance 2σ<sup>2</sup> / _n_ . From this we observe the following:

**_Implication 1_** _:_ With a single baseline and follow-up, difference-in-differences only gives more power than the post estimator which ignores the baseline data when the autocorrelation is greater than 0.5.

A more efficient method of estimation than difference-in-differences is to control for the baseline levels of the outcome of interest and estimate the following<sup>4</sup> : Let be the least squares estimator of _θ_ , then the estimator of the treatment effect coming from equation (4) is called the ANCOVA estimator Frison and Pocock (1992) show that ANCOVA is more efficient than either difference-indifferences or analysis using only the post data, with a variance approximately proportional to 2σ<sup>2</sup> (1- ρ<sup>2</sup> )/ _n_ .<sup>5</sup> We therefore deduce:

**_Implication 2_** _:_ When the data have zero autocorrelation, there is no gain from using baseline data. Otherwise the ANCOVA estimation in equation (4) offers more power than either the commonly used difference-in-differences estimation or estimation based on follow-up.

Intuitively, when the autocorrelation is low, the baseline data are not very informative for what future values of the outcome of interest will be. Controlling fully for observed baseline

> 4 Such a specification is estimated in Banerjee et al. (2007) among others.

> 5 This is an approximation as the need to estimate θ causes a loss of one degree of freedom and also adds a term to the variance to reflect the sampling error. Both of these terms are negligible in any reasonably-sized randomized experiment, and hence are ignored here.

6
differences in the mean outcome via difference-in-differences is therefore over-correcting for differences which don’t have much predictive power, whereas the ANCOVA estimation adjusts the degree of correction for baseline differences in means according to the degree of correlation between past and future outcomes actually observed in the data.

The ratio of the difference-in-differences variance to the ANCOVA variance is 2/(1+ ρ). So when ρ=0, with a single baseline and follow-up, one would need twice the sample size when using difference-in-differences to get the same power as obtained with ANCOVA. When ρ=0.25, which we will see to be a reasonable estimate for several economic outcomes, the sample size needed is still 60% higher with difference-in-differences than with ANCOVA to get the same power. There are therefore important gains in power to be had from not using difference-indifferences to estimate treatment effects with standard economic variables.

This comparison of the ANCOVA and difference-in-difference estimators in terms of efficiency is appropriate for experimental estimation, where both estimate the same average treatment effect. However, in non-experimental estimation, the estimators also require different assumptions for efficiency. Imbens and Wooldridge (2009, p. 70) discuss this issue.

# **2.2. Multiple rounds pre- or post-treatment**

Now consider the general case of _m_ pre-treatment survey rounds and _r_ post-treatment survey rounds. The difference-in-differences estimator is obtained by estimation of equation (1). This can be written as: Where for is the mean of the outcome for treatment or control group _g_ in the post-treatment period, and _n_ is the number of cross-sectional observations assigned to treatment and to control. The pretreatment means are defined analogously.

7
Frison and Pocock (1992) give a general formula for the variance of , which depends on the full pattern of autocorrelations across the different time periods. For power calculations, one needs to make simplifying assumptions as to how these autocorrelations behave. Frison and Pocock assume equal variances for all time points, equal correlations between all pairs of time points, and that autocorrelations are equal for the treatment and control groups. They show for physical health measures like cholesterol, blood pressure, and CD4 cell count that this assumption of constant autocorrelation is reasonable. In the next subsection I show such an assumption may also hold approximately for key economic outcomes. Then assuming the treatment and control groups are each of cross-sectional size _n_ , the variance of the difference-indifferences estimator is shown by Frison and Pocock to be: Under these same assumptions, the variance without any baseline surveys of the difference in post-treatment means is then: Equations (7) and (8) are respectively the variances used in power calculations under the “POST” and “CHANGE” methods reported in STATA’s _sampsi_ command. We therefore see that:

**_Implication 3_** _:_ With _m_ baselines and _r_ follow-ups, difference-in-differences only gives more power than the post estimator which ignores the baseline data when the autocorrelation is greater than _1/(m+1)._

Not surprisingly, fact 3 shows that the more baseline rounds of data there are, the more likely it is that it is costly to ignore them. However, as with the single baseline case, if the data are only weakly autocorrelated, it can be costly in terms of power to fully control for baseline differences in means via difference-in-differences.

8
As with the single baseline and follow-up case, ANCOVA estimation will be more efficient than both difference-in-differences and estimation using only the follow-up data. The ANCOVA estimator with multiple pre- and post-treatment rounds is: This can be estimated via the following least squares regression for t=1,2,.., _r_ for individual _i_ : where It is very rare for economists to collect more than one round of data pre-treatment, so such an equation has not been typically estimated in practice for more than one pre-treatment round.

Under the same assumptions as used to derive (7) and (8), the variance of the ANCOVA estimation is then approximately: This is the formula used in power calculations under the ANCOVA option using STATA’s sampsi command.

**_Implication 4_** _:_ If the autocorrelation is zero, then only the number of post-treatment survey waves affects the power of the ANCOVA estimator, there is no gain from baseline surveys, and it reduces to the POST estimator. Intuitively, an autocorrelation of zero means that each round the outcome is a mean plus noise. The treatment changes the mean, and more post-treatment rounds enables one to better average out this noise.

# **2.3. Autocorrelations in practice for economic experiments**

As noted above, the formulae given for the variances of the different estimators assume that the autocorrelations are equal between all points in time, and that they are equal for the treatment and control groups. For power calculations to be useful, we need not have these assumptions 9
hold exactly, but we need them to be a reasonable approximation. The next sub-section then discusses what to do in cases when this assumption doesn’t hold.

The assumption that the autocorrelations are equal between all points in time is one that many economists may have second thoughts about for both theoretical and statistical reasons. On the theoretical side, if we think that a large source of the period-to-period variation in economic outcomes like profits, income or consumption comes from temporary shocks, we should expect the impact of these shocks to die out over time, so that observations closer apart in time will be more highly correlated than those further apart in time. Statistically, constant (non-zero) autocorrelations imply that although the data generating process is covariance-stationary, it is non-ergodic for the mean and thus a law of large numbers will not apply to the mean of as the time dimension approaches infinity.

However, while such an assumption may therefore not be appropriate globally over all horizons, it may be a reasonable assumption within the likely time horizons of many applications. That is, while we might not believe the correlation between business profits this month and next month is the same as that between this month and 10 years from now, the correlation between this month and next month may be close to that between this month and two months from now. For example, in a local neighborhood of a particular time horizon, a reasonable data generating process for might be , where is i.i.d. noise. Thus, for example, within a few months around the one year after an intervention horizon we might expect profits for a particular firm to be higher or lower depending on some longer-term shock , and then in addition from month to month the firm gets idiosyncratic demand shocks which cause its profits to move around further.

Ultimately it is therefore an empirical question as to whether this assumption seems reasonable over the periods of typical economic experiments, and especially over short-term neighborhoods of a few months around a particular horizon of interest. Figures 1A, 1B, and 1C investigate this assumption for microenterprise profits and for household expenditure. Figure 1A shows autocorrelations for monthly profits from the Sri Lankan microenterprises in de Mel et al. (2008) which were measured at three month intervals, and uses data from similar microenterprises measured monthly in de Mel et al. (2009) to show the autocorrelation in monthly profits at the one month and two month horizons. A horizontal line is drawn at 0.38, which is the average 10
autocorrelation over the different interval measures. Pointwise confidence intervals for these autocorrelations are obtained via 1000 bootstrap replications. While Figure 1A does suggest some tendency for the autocorrelation to fall as the time horizon increases, the data do not deviate much from the 0.38 level over all time horizons between 1 month and 18 months, and so the assumption of constant autocorrelation seems a reasonable approximation here.

Figure 1B shows the autocorrelations in monthly microenterprise profits in urban Ghana, which were measured at three month intervals in the experiment of Fafchamps et al. (2010b), and Figure 1C shows the autocorrelation in three monthly household expenditures from the same experiment, also measured at quarterly intervals. Horizontal lines are drawn at 0.32, the mean autocorrelation over these horizons for profits, and 0.20, the mean autocorrelation over these horizons for household expenditures. Again both figures suggest the assumption of constant autocorrelation holds approximately over these horizons.<sup>6</sup> Gibson et al. (2003) also provide evidence from Zambia and urban China that the autocorrelation of expenditure does not change very much within a year as one increases the number of months between surveys.

Table 1 then considers the evidence for the second part of the assumption, which is that the autocorrelations are the same for treatment and control, using the microenterprise profits data from Sri Lanka and Ghana. The point estimates in most periods are reasonably similar between treatment and control, and the bootstrapped confidence intervals for the difference in autocorrelation between the two always contain zero. Evidence from another setting comes from using the test score data of Banerjee et al. (2007). The verbal test score autocorrelation is 0.58 for the treatment group and 0.61 for the control group; and the math test score autocorrelation is 0.47 for the treatment group and 0.55 for the control group. Therefore in these samples this assumption seems reasonable.

Table 2 provides autocorrelations from other data sets for a range of different economic outcomes. The purpose of this is twofold. First, it demonstrates that for many economic outcomes, the autocorrelations are typically lower than 0.5, with many around 0.3. An exception is test scores, which have higher autocorrelations. Second, it provides some parameters that researchers can use when conducting their own power calculations, since it is rare to have

> 6 The Sri Lankan microenterprise surveys only measured household expenditure annually, which is why household expenditure is not shown for this sample.

11
autocorrelation data available for the study populations being considered. We see from Table 2 that autocorrelations are often in the 0.2-0.3 range for household income and consumption. In rural Ghana, the autocorrelation in Boozer et al (2010)’s data is 0.32 when wives report on both their own and their husband’s consumption, but increase to 0.58 to 0.66 when husbands and wives report separately on consumption and the results are combined. The autocorrelations are lower for the poor – in both Mexico and Argentina the autocorrelation of labor income is around 0.2-0.3 for individuals with first period labor income below the median, compared to autocorrelations of 0.5-0.8 for those with this income above the median. This is consistent with the evidence in Collins et al. (2009) that incomes for the poor are very volatile.

The autocorrelation will typically be lower in the more homogeneous samples typical of many randomized experiments than it is in general samples of the population. It will also be lower if one stratifies the randomization or uses fixed effects and thereby controls for time-invariant reasons why one household or firm may have higher income, expenditure or profit levels than another. To see this in a simple example, write the error term , where _X_ is a characteristic that is used to stratify the randomization on, or to restrict an experiment only to individuals with certain values of _X._ Then: And Controlling for _X_ via stratified randomization, or restricting the sample only to those with certain values of _X_ will therefore reduce the variance of , which increases the power of an experiment. However, it also reduces the autocorrelation (by removing a factor which would otherwise cause the error term to be correlated from one period to the next), thereby increasing the value of more measurements.

# **2.4. What should one do if the autocorrelations are not constant?**

The empirical evidence therefore suggests that the assumption that the autocorrelation is constant over different time horizons and is the same for the treatment and control is likely to be a 12
reasonable approximation in a number of economic experiments. How do violations from this assumption affect power calculations?

For post and difference-in-difference analysis, one can simply use the mean autocorrelation in the power calculations. Since the variances of these estimators are linear in , power calculations using the mean will still give the correct power for these estimators even when the autocorrelation differs between treatment and control groups or over different time periods. In contrast, since the variance of the ANCOVA estimate is decreasing in , using the average will understate the sample size one would actually need to achieve a given power. In such cases using the minimum expected among follow-up rounds would be conservative.

# **3.  Implications for choosing the number of pre-treatment and post-treatment rounds**

Equations (7), (8) and (11) can be used to help researchers decide how many rounds of data they should collect before and after the treatment. We use these formulae to examine implications for choice of the number of pre-treatment and post-treatment rounds.

# **3.1. Given a fixed T of 3 or more rounds, how should they be split between pre-treatment and post-treatment?**

The first question we ask is how a researcher who has decided on fielding multiple survey rounds should split these rounds before and after treatment. To do this, we solve for the choice of _r_ and _m_ which minimizes the variance given _r+m=T_ .

**_Implication 5:_** The optimal choice of _r = T/2_ = _m_ for difference-in-differences estimation, and this choice does _not_ depend on the autocorrelation in the data. When is odd, the power is the same when choosing _m-r=1 as r-m=1_ , that is, when allocating the extra odd wave to pretreatment or to post-treatment.

_Proof_ : Setting _r+m=T_ in equation (7) and solving for the optimal choice of _r_ to minimize the variance gives the first part of this result. Re-writing the variance in (7) as 13
Shows that _r_ and _m_ contribute symmetrically to the variance of the difference-in-differences estimator, and so that choosing _r = (T+1)/2_ and _m=(T-1)/2_ will yield the same results as choosing _r=(T-1)/2_ and _m=(T+1)/2_ Intuitively, in a difference-in-differences design, differences in pre-treatment means have exactly as much weight in the estimator as differences in post-treatment means, and so the optimal strategy is to have the same number of time periods to estimate the difference pre-treatment as post-treatment. Note from (12) that while the autocorrelation does not affect the choice of how to allocate the survey rounds between pre-treatment and post-treatment in a difference-indifferences design, it does affect the cross-sectional sample size needed – a higher autocorrelation reduces the variance, thereby requiring a smaller _n_ .

**_Implication 6:_** For ANCOVA estimation, the optimal choice of _r_ given fixed _T_ is given by

So when ρ=0.5, the optimal choice is _r = (T+1)/2_ , whereas when ρ=0.25, the optimal choice of _r = (T+3)/2_ .

_Proof:_ Expanding out the variance in (11), one sees that _r_ and _m_ are symmetric in the numerator. Minimizing the variance therefore amounts to maximizing the denominator, _r[1+(m-1)_ ρ] _,_ which can be solved to give the expression in (13).

Thus with ANCOVA estimation, one chooses fewer pre-treatment survey rounds the lower is the autocorrelation. Intuitively, we see in equation (9) that the pre-treatment difference in means contributes less to the estimator than the post-treatment difference in means, and so it is less important to accurately measure the pre-treatment difference than to accurately measure the posttreatment difference. When the autocorrelation is low, this can result in choosing to have no baseline. For example, when T=3 and ρ=0.25, the optimal choice is _r=3_ , so one is better to have 3 follow-up waves and no baseline than a baseline and two follow-up waves.

14
# **3.2. Given a fixed total sample size nT, what is the trade-off between a larger cross-section and more survey rounds?**

In many cases a researcher will face a fixed total budget, which can fund _K=nT_ total surveys. They must then decide between carrying out a larger cross-sectional sample, which gives information on more units, and carrying out more survey rounds, which gives more information on each unit. Taking _K_ as fixed, we derive the optimal division into more rounds versus more individuals per round.

**_Implication 7:_** For POST estimation, the optimal choice is _T=1_ and _n=K_ when ρ>0 _,_ and all values of _T_ and _n_ such that _nT=K_ yield the same power when ρ=0. When ρ<0 _,_ the power is higher using _T=2_ and _n=K/2_ than it is using _T=1_ and _n=K_ , while the assumption of constant autocorrelation across different points in time doesn’t make sense for negative autocorrelation and _T>2_ .

_Proof:_ From equation (8) we can see that when _rn=K_ , the variance depends on _(r-1)_ ρ _,_ so the optimal choice of _r_ to minimize this will depend on ρ in the way stated.

This fact is analogous to the decision in a clustered randomized trial of whether to collect more observations per cluster, or more clusters. With clustered randomization, since randomization is at the level of the cluster, if there is positive intra-cluster correlation, there is more power to be had in randomizing over more clusters than in having fewer clusters and a larger sample per cluster. Likewise here the randomization occurs in the cross-section, and so with positive autocorrelation, there is more gain to having more cross-sectional units than more observations in the time dimension on each unit.

**_Implication 8:_** For difference-in-differences estimation, the choice of how to allocate a fixed sample _K_ between _n_ and _T_ does not affect the variance provided that one then chooses to allocate _T_ equally between pre-treatment and post-treatment rounds (as in Fact 5), and that there is at least one baseline and one follow-up to enable difference-in-differences estimation.

_Proof:_ Setting _r=m=T/2_ and _nT=K_ in (12) yields a variance of which does not depend on either _T_ or _n._ 15
Thus the power of difference-in-differences estimation is the same with a single baseline and follow-up and _n=100_ , as it is with two pre-treatment rounds and two post-treatment rounds and _n=50_ . Intuitively, with difference-in-differences, every observation counts equally in calculating the pre-treatment and post-treatment means which form the basis of estimation.

This result of course depends on the assumption of equal correlation being a reasonable one. So this implication is likely to make sense when choosing between one or two follow-up surveys, or even between one and five follow-up surveys, but should not be taken to the extreme of saying we should expect the same power from _T=100_ and _n=2_ as we would from _T=2_ and _n=100_ .

**_Implication 9:_** With ANCOVA estimation, when the autocorrelation is high it is better to do a larger cross-section and fewer survey rounds, whereas when the autocorrelation is low, it is better to do relatively more survey rounds and a smaller cross-section. Holding nT fixed, with a single baseline: (i) the optimal value of _r_ holding _nT_ fixed is ; (ii) two follow-up surveys will offer more power than a single follow-up survey for ρ<0.5; (iii) three follow-up surveys will offer more power than one or two follow-up surveys for ρ<1/6.

_Proof:_ Set _m=1_ and _nT=K_ in (11), and then take the derivative with respect to _r_ to show (i). Compare the variances with r=1, 2, and 3, to show (ii) and (iii).

Therefore when the total sample size is fixed at _K_ , power will typically be greater using a baseline and 2 post-treatment surveys with a cross-sectional sample of K/3 than a single baseline and follow-up for many types of economic data. But it will be rare in practice for more than two post-treatment surveys to be optimal with a fixed total sample size. However, the standard practice of a single baseline and single follow-up used in so many economics experiments is unlikely to be the optimal choice for all.

In practice the cost of adding another cross-sectional unit versus surveying the same unit a second time can differ. In many experiments there can be large fixed costs of adding additional units to the experiment. These fixed costs can include the costs of screening and enrolling more subjects in the study, the cost of the intervention itself (many times this is by far the largest cost), and potentially the cost of more handheld units for surveying them. Denote the fixed costs of another cross-sectional unit by _a_ . There can also be fixed costs of conducting another survey round, such as the need to pay field managers for more months of work. Denote the fixed cost of 16
another time round by _b_ . Finally denote the marginal cost of another survey, whether crosssectional or temporal, by _c_ . Then, assuming only one baseline, with a fixed budget _D,_ one can solve for the optimal number of follow-up rounds and cross-sectional units by minimizing the ANCOVA variance in (11) subject to _an+br+cn(r+1) = D_ .<sup>7</sup> This does not have a closed form solution, but can easily be solved numerically for specific cases. For example, take ρ=0.30, b = $100 and c = $50, and D = $500,000. Then with a relatively cheap treatment of a = $100, the optimal is to take _n=1200_ and _r=2_ follow-ups, whereas with an expensive treatment of a = $10,000, the optimal is to take _n=27_ and _r=21_ follow-ups. Thus studies with more expensive treatments should optimally collect more data on each unit treated.

# **3.3. What is the marginal gain to another post-treatment round?**

Finally, in many experiments researchers have a fixed cross-sectional sample, often dictated by the pool of eligible applicants for some pilot program they are investigating. For example, an NGO has money to offer a treatment to 1000 farmers, or a government wishes to pilot a program in 50 villages. The question then facing the researcher is how many survey waves to collect. Consider the difference-in-differences estimator, and the gain in power from collecting _r+1_ rounds compared to _r_ rounds of post-treatment data. This gain in power is proportional to the difference in the variances, and so using (12) we have: Note that the gain in power from adding more post-treatment rounds to difference-in-differences is the same as the gain to adding more post-treatment rounds to estimation by the post method, and thus the gain in (14) is the same as that calculated by Vickers (2003) for the case of estimation by the post method. Using (11), we see that this is also the gain in power from adding more post-treatment rounds to ANCOVA estimation.

> 7 This assumes that the treatment and control groups are the same size. When the cost of adding another treatment greatly exceeds that of adding another control, optimal power with a given budget can be achieved by allocating more units to the control group than to the treatment group (See Duflo et al, 2008). Conditional on doing this, it may still be optimal to allocate fewer cross-sectional units to more survey rounds.

17
**_Implication 10:_** There are diminishing marginal returns to adding more post-treatment rounds in terms of the gain in power they give. The greatest gain is from moving from one to two posttreatment rounds, and the gains are smaller the higher is the autocorrelation.

_Proof:_ One can easily determine that the derivatives of (14) with respect to both _r_ and _ρ_ are both negative.

Given the symmetry of the role of pre-treatment and post-treatment surveys in difference-indifference estimation, we see the same holds for adding more pre-treatment rounds. For ANCOVA estimation, the gain from adding pre-treatment rounds is different from that from adding post-treatment rounds in general, and the gain from moving from one to two pretreatment rounds is less than that from moving from one to two post-treatment rounds when

# **3.4. Numerical illustrations**

To illustrate how much the required cross-sectional sample size can be reduced when more survey rounds are undertaken, Table 3 presents the minimum treatment group size _n_ obtained under power calculations with different values of _r_ , _m_ , and ρ _._ I consider a hypothetical experiment intended to detect a 10 percent increase in business profits from a baseline value of 100. Microenterprise profits are noisy, so it is common for the standard deviation to be of the same order of magnitude as the mean. I therefore take the standard deviation as 100, and fix the power at 80 percent and size at 5%. Panel A then shows the treatment group sample sizes required for post estimation, Panel B for difference-in-differences estimation, and Panel C for ANCOVA estimation, for ρ ε{0, 0.25, 0.5, 0.7, 0.95}. It is assumed that the control group is the same size as the treatment group in these calculations.

Table 3 illustrates numerically Implications 1-6, and Implication 10. Comparing the sample sizes in Panel A and B, we see Implication 1 and 3. When there is only one baseline, difference-indifferences requires larger sample sizes than post analysis when ρ<0.5, the same sample sizes for ρ=0.5, and lower sample sizes for ρ>0.5. The difference in sample size can be large. For example, with a single baseline and follow-up, when ρ=0.25, one would need a treatment group size of 2355 with difference-in-differences, compared to 1570 with post analysis. Adding a second pre-treatment survey reduces the sample size for difference-in-differences to 1766, 18
consistent with Implication 3 (with 2 baselines, difference-in-differences only has more power than post for ρ>1/3). Comparing Panels A and B to Panel C we see consistent with Implications 2 and 4 that ANCOVA requires the same sample size as post when ρ=0, and otherwise requires lower sample sizes than either difference-in-differences or post. Thus in the case of a single baseline and follow-up and ρ=0.25, ANCOVA requires a treatment group size of 1472, compared to the 1570 with POST and 2355 with difference-in-differences. We see in particular that ANCOVA does substantially better than difference-in-differences when ρ is low, and substantially better than POST when ρ is high.

Panel B also illustrates Implication 5. For a fixed number of waves T, the required sample size is smallest when the numbers of pre-treatment and post-treatment waves are equal. Thus with T=4, the treatment sample size required with ρ=0.5 is 785 with 2 pre-treatment waves and 2 followups, compared to 982 with one baseline and three follow-ups. We also see the sample sizes required when _m=1_ and _r=2_ are the same as those with _m=2_ and _r=1_ . In contrast, as per Implication 6, we see ANCOVA favors more post-treatment waves than pre-treatment waves when ρ is low. Thus the treatment sample size required with ρ=0.5 is 785 with one baseline and 2 follow-ups, compared to 1047 with two baselines and 1 follow-up.

Finally in Table 3 one can also observe Implication 10, the diminishing gain from adding more survey rounds, with the amount of the gain lower the higher is ρ. Thus for difference-indifferences with one baseline, when ρ=0 the gain from going from one to two post-treatment waves is a reduction in the treatment group size of 785, whereas the gain in going from two to three post-treatment waves is only 261. When ρ=0.95, the gains are only a reduction in treatment size of 39 when going from one to two post-treatment waves, and 13 in going from two to three post-treatment waves.

Table 4 illustrates the differences in power obtained in changing the allocation of a fixed total sample size between _n_ and _T._ We fix _nT_ = 1000, and consider different combinations of crosssectional and time series samples. Again we assume a mean and standard deviation of 100, and a treatment effect of 10 percent. Panel A illustrates Implication 7, showing that T=1 is always best when ρ>0 for POST analysis. Panel B illustrates Implication 8, demonstrating that the power is the same regardless of how _n_ and _T_ are split, so long as _r=m=T/2_ . Finally, Panel C illustrates Implication 9, that for a given total sample, sometimes power can be greater when doing a 19
baseline and multiple follow-ups, than by doing a single baseline and single follow-up. For example, when ρ=0.25, power is greater doing one baseline and two follow-ups with _n=333_ , than doing a single baseline and single follow-up with _n=500_ , but also greater than doing one baseline and three follow-ups with _n=250._ As a final illustration, consider the power to detect an impact of microfinance on business profits. Banerjee et al. (2010) report a mean of 550 and standard deviation of 46604 for business profits. With such noisy data, the treatment and control group sample sizes required to detect a 10% increase in profits with 90 percent power using a single baseline and follow-up are over 15 million! Suppose then that they were able to consider a more homogeneous set of firms and measure profits more accurately<sup>8</sup> , reducing the standard deviation to 550. Then assuming an autocorrelation of profits of 0.25, with their sample of approximately 1150 treatment and 1150 control, the power for detecting a 10% increase in profits would be 0.669 with a single crosssection post-treatment, 0.697 with a baseline and single follow-up, 0.892 for a baseline and two follow-ups, and 0.952 with a baseline and three follow-ups.<sup>9</sup> Likewise, taking the example of Karlan and Zinman (2010), who have a control group mean of 17,000, if the standard deviation were the same as the mean, the power to detect a 10% increase in profits would rise from 0.68 with a single post-treatment survey as they have to 0.902 for a baseline and two follow-ups and 0.958 with a baseline and three follow-ups. In both cases then, assuming the standard deviation of profits can be kept or reduced to a reasonable level, the choice of a second or third posttreatment measure would allow them to detect a treatment effect of interest with considerably more power than possible with only a single survey round.

# **4. Recommendations for practice and other practical issues**

The above analysis has shown that there are gains to be had from moving beyond the paradigm of single baseline and follow-up. In this section I discuss first some basic guidelines for using these results in practice, and then discuss other practical issues that might affect the choice of how many survey waves to carry out.

> 8 They construct profits as revenue less expenses. De Mel et al. (2009) show that this leads to considerably noisier profit measures than directly asking for a single profits number in their experiments.

> 9 These power calculations ignore intra-cluster correlation for simplicity, and because it is typically low for an outcome like profits.

20
# **4.1. Guidelines for practice**

1. _Highly autocorrelated outcomes:_ For outcome measures like anthropometric measures or test scores, for which the autocorrelation is high (e.g. ρ=0.6 to 0.8), always include at least one baseline. Difference-in-differences and ANCOVA have much greater power than POST in these cases. Moreover, Bruhn and McKenzie (2009) also show that the power improvements from stratified or matched randomization are highest when ρ is high. However, if the total sample size is fixed, a single baseline and follow-up will offer more power than multiple pre-treatment or post-treatment rounds when ANCOVA is used. If the treatment and control sizes are fixed, the gains from going from one to two post-treatment surveys can still be non-trivial in these cases, but there is little gain from more than say 3 post-treatment surveys.

_2. Outcomes with low autocorrelation:_ For outcome measures like business profits, incomes, or expenditure, for which the autocorrelation is typically low (e.g. ρ=0.20 to 0.40) , it can be optimal to have no baseline at all, and just do a single follow-up survey if the total sample size is limited. If the treatment group size is fixed, researchers can dramatically increase power by doing multiple post-treatment surveys. Even if a baseline is taken, researchers should not use difference-in-differences in such cases, since doing so has much lower power than either POST or ANCOVA analysis.

# **4.2. Other practical issues**

In practice there are several other factors that should guide researchers in choosing how many survey rounds to conduct. I note some of the key factors here, and their implications for choice of the number of pre-treatment and post-treatment waves:

1. _Over what horizon is a pooled treatment effect relevant and of interest?_ The analysis in this paper has assumed a common treatment effect γ. Of course if the treatment effects vary over individuals, researchers are usually content to estimate an average treatment effect, and the analysis here will still hold. However, in some experiments the effect of the treatment may also vary over time. In this case estimation of γ by pooling multiple post-treatment survey waves will yield an average treatment, where the average is over multiple waves. If the survey waves are relatively close together in time (e.g. monthly or quarterly), then combining several post-treatment waves and getting the average

21
treatment effect over a period 9-15 months post-treatment is likely to be reasonable.<sup>10</sup> In contrast, if there is reason to believe the effects differ dramatically with time since treatment, one will not want to average over long periods. High seasonality may provide another reason to conduct multiple measures – it may be of more interest to get the average impact over high and low demand periods than to get the impact only at one particular point in time.

2. _Survey compliance and attrition._ Conducting multiple survey waves increases the burden on respondents, which might increase attrition. Using multiple pre-treatment waves can be beneficial in this respect, by eliminating the individuals most likely to attrit before the randomization is done, one can improve internal validity by randomizing only over individuals that are likely to be found in a follow-up survey. In contrast, if multiple rounds of post-treatment surveys cause significant attrition, this will limit the extent to which researchers will wish to conduct many rounds of the survey. If the survey is just collecting measurement of a few key outcomes, these additional rounds of post-treatment surveys can be short and less burdensome for respondents, which can minimize this dropout. Secondly, by conducting multiple post-treatment rounds, researchers stand a better chance of capturing at least once post-treatment individuals who are harder to track down. For example, a business owner who travels temporarily to other towns might get missed in a one-off survey, but may get found when going back at monthly or quarterly intervals. Finally, there is also the possibility that waiting for a long time to re-contact people may make it harder to find them, and make respondents think you have lost interest in them, whereas regular follow-ups may instead these risks.

3. _Will multiple measures change people’s reporting or people’s behaviors?_ Experience with measuring microenterprise profits suggests that respondents may report more accurately after one or two rounds of surveys, perhaps as they better understand the concepts being asked and can recall them better when asked for a second or third time (Samphantharak and Townsend, 2009; Fafchamps et al, 2010a). This suggests a benefit to researchers of conducting more than one pre-treatment survey. However, in some circumstances there will be a concern that asking the same question multiple times will

> 10 Researchers interested in time since treatment effects can then combine multiple measures around 1 year, around 2 years, etc. See De Mel et al. (2008) for an example, in which quarterly waves are combined to look at how treatment effects vary over the first year since treatment, second year since treatment, etc.

22
either change the way people respond, or their behavior. For example, Zwane et al. (2010) summarize the results of five experiments in which the frequency of surveying was varied. They find being surveyed more frequently lowers reported child diarrhea rates and leads to more use water treatment products and take-up of medical insurance. However, they find no effect of being surveyed on borrowing behavior. The risk is therefore that in some settings and for some outcomes, repeated surveying may serve as reminders to increase the salience of neglected actions. Finally, repeatedly asking individuals the same questions may change reporting due to fatigue. For example, individuals who are repeatedly asked about expenditure may quickly learn that they can reduce the length of the questionnaire by saying “no” to questions which when answered positively lead to lengthy detailed questions about what was spent.

**4.** _Cost considerations._ Surveying a smaller cross-section over multiple survey waves can result in a different cost than doing a larger cross-section single baseline and follow-up. Whether the cost is higher or lower will depend on country context. On one hand, a smaller cross-sectional sample with more survey waves allows a smaller survey team to be used for longer periods of time, potentially resulting in higher average quality enumerators. In addition, it can lower the costs of listing since a smaller cross-sectional sample is used. Of greatest cost savings in many experiments is that it also avoids the costs of paying for the intervention for as many units. On the other hand, the team is in the field for a longer period of time, which can raise costs. As a general rule, the higher the per-unit treatment cost, the more likely it is that doing multiple measures per unit treated is optimal.

**5.** _Learning by doing._ Even in cases where researchers have some power to detect a treatment effect with one post-treatment survey round, it can be useful to plan multiple survey rounds relatively close together in time. This allows researchers to quickly analyze the early results from the first follow-up survey, and based on these, ask followup questions in the next post-treatment survey to explore new hypotheses generated by the results.

**6.** _How do we know what ρ is?_ Power calculations already require substantial insights from researchers in terms of what the likely mean, standard deviation, and treatment effect size of interest are. Pilot surveys or existing data are often used to give some sense of likely

23
parameters. In most cases researchers will not have data on the autocorrelation of the outcome of interest. It is hoped that the data provided in this paper for different outcomes from different surveys will therefore provide a starting point for researchers, and that more researchers will report these autocorrelations in future experiments.

**7.** _Can we just obtain multiple measures more cheaply by recall?_ A cheaper alternative to conducting multiple post-treatment or pre-treatment rounds is to ask multiple measurements in a single survey, with recall. While intuitively appealing, the value of this approach is likely to be limited in practice, since it is precisely the types of variables that are noisiest and hardest to measure in practice for which the data are typically less autocorrelated and for which multiple measures are of most value. For small informal firm owners who do not keep books, asking profits with recall over multiple months is difficult, while asking expenditure month by month or quarter by quarter with recall is likely to be highly inaccurate. Gibson and Kim (2010) show even with wage workers in the United States that retrospective recall is problematic, with workers underreporting transitory variation in earnings, creating non-classical measurement errors. However, such a strategy could be used in cases where good records are available, such as in experiments with larger firms which have accurate books or records.<sup>11</sup>

**8.** _To baseline or not?_ The power calculations above suggest that, in a number of cases, it can be optimal in terms of power to conduct a single post-treatment survey of size _n=K_ than to do a baseline and follow-up with _n=K/2_ . However, there are several other factors to consider in choosing whether or not to use a baseline. A baseline can be used to stratify the randomization on key variables, improving power and providing a basis for examination of treatment effect heterogeneity. It is often used to examine what determines take-up of some intervention. It can be useful for verification of randomization in cases where there is a risk of the randomization not being implemented perfectly, and can also be used to test whether attrition is non-random in terms of baseline characteristics. Finally a baseline, and especially multiple pre-treatment surveys, offer the possibility of matched difference-in-differences as a back-up evaluation strategy in cases

> 11 See e.g. Bloom et al. (2010) who collect daily and weekly production data from plants with over 100 workers each once a month.

24
where there is a risk that the randomization may not get implemented in practice or a risk that take-up of a program may be lower than anticipated.

However, when the outcome has low autocorrelation, the baseline does not reveal very much about likely future outcomes, and thus can be an expensive undertaking. Researchers attempting to implement large-scale evaluations with governments or NGOs, where there is reasonable uncertainty as to whether randomization will actually be carried out and whether they will have sufficient take-up. For example, consider a micro-savings intervention, where a government or NGO agree to randomly choose villages in which to introduce a new savings product. There is a risk that after doing baseline surveys in the treatment and control villages that the government changes its mind, and decides to introduce the product in all villages (or decides not to introduce the product at all). There is also a risk that no one uses the new product, in which case there is no possibility of measuring the impact of savings accounts on household outcomes. In such a case, practical considerations may suggest it is optimal to not carry out a baseline, and then wait and only carry out post-treatment surveys if the product is indeed randomly introduced and take-up is high.

# **5. Conclusions**

This paper has shown that when the autocorrelation in outcome data is low, as is common with outcomes of interest like business profits, expenditure, and income, the standard paradigm of single baseline and follow-up, followed by difference-in-differences analysis, is unlikely to be optimal. Large improvements in power can be obtained from multiple post-treatment measures in experiments with fixed treatment and control group sizes, and from using ANCOVA instead of difference-in-differences. Researchers choosing how to allocate a fixed budget over multiple surveys may find they can obtain more power by not conducting a baseline at all, and if they use a baseline, will often get more power doing two follow-up waves with a smaller cross-sectional sample size than a single follow-up with a larger cross-sectional sample.

These findings are particularly likely to be of interest and use to researchers conducting experiments with interventions to help the poor, since the profits, incomes, and expenditures of 25
the poor are typically more volatile and less autocorrelated than those of stable wage earners for example. In many cases the size of the treatment group is determined by the number of units eligible for some pilot initiative, and so hopefully researchers can extract much more out of these samples by considering multiple measurements.

26