metadata: authors: 0: author: Bedoya Arguelles,Guadalupe
1: author: Bittarello,Luca
2: author: Davis,Jonathan Martin Villars
3: author: Mittag,Nikolas Karl
count: World
docna: 0: docna: Distributional impact analysis: toolkit and
            illustrations of impacts beyond the average treatment effect
docty: Policy Research Working Paper
subtopic: Health Care Services Industry,Inequality,Gender and Development,Social Impacts and Poverty Mitigation,Poverty and Social Impact Analysis,Social Analysis,Quality of Life & Leisure
teratopic: Industry,Poverty Reduction,Gender,Social Development
abstracts: cdata!: Program evaluations often focus on
            average treatment effects. However, average treatment
            effects miss important aspects of policy evaluation, such as
            the impact on inequality and whether treatment harms some
            individuals. A growing literature develops methods to
            evaluate such issues by examining the distributional impacts
            of programs and policies. This toolkit reviews methods to do
            so, focusing on their application to randomized control
            trials. The paper emphasizes two strands of the literature:
            estimation of impacts on outcome distributions and
            estimation of the distribution of treatment impacts. The
            article then discusses extensions to conditional treatment
            effect heterogeneity, that is, to analyses of how treatment
            impacts vary with observed characteristics. The paper offers
            advice on inference, testing, and power calculations, which
            are important when implementing distributional analyses in
            practice. Finally, the paper illustrates select methods
            using data from two randomized evaluations.
repnme: repnme: Distributional impact analysis: toolkit and
            illustrations of impacts beyond the average treatment effect
display_title: Distributional impact analysis:
            toolkit and illustrations of impacts beyond the average
            treatment effect
listing_relative_url: /research/2017/07/27708913/distributional-impact-analysis-toolkit-illustrations-impacts-beyond-average-treatment-effect
topicv3: Poverty m1326687 695,Poverty and Social Impact Analysis
            m1327421 709,Gender m1327621 615,Health m1504661 3480
publishtoextweb_dt: 2017-07-06T00:00:00Z
historic_topic: Industry,Poverty Reduction,Gender,Social Development
no_of_pages: 70
author_keys: 0: author_key: 1242417
1: author_key: 1375441
2: author_key: 1357060
3: author_key: 1377979
model_extractions: WPS8139 Policy Research Working Paper 8139 Distributional Impact Analysis Toolkit and Illustrations of Impacts Beyond the Average Treatment Effect _Guadalupe Bedoya Luca Bittarello Jonathan Davis Nikolas Mittag_ Development Research Group Impact Evaluation Team July 2017
Policy Research Working Paper 8139

# **Abstract**

Program evaluations often focus on average treatment effects. However, average treatment effects miss important aspects of policy evaluation, such as the impact on inequality and whether treatment harms some individuals. A growing literature develops methods to evaluate such issues by examining the distributional impacts of programs and policies. This toolkit reviews methods to do so, focus-ing on their application to randomized control trials. The paper emphasizes two strands of the literature: estimation of impacts on outcome distributions and estimation of the distribution of treatment impacts. The article then discusses extensions to conditional treatment effect heterogeneity, that is, to analyses of how treatment impacts vary with observed characteristics. The paper offers advice on inference, testing, and power calculations, which are important when implementing distributional analyses in practice. Finally, the paper illustrates select methods using data from two randomized evaluations.

This paper is a product of the Impact Evaluation Team, Development Research Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at gbedoya@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
Distributional Impact Analysis: Toolkit and Illustrations : Policy Evaluation, Distributional Impact Analysis, Heterogeneous Treatment Effects, Impacts on Outcome Distributions, Distribution of Treatment Effects, Random Control Trials. David Evans, and Caio Piza for making this toolkit possible. We are grateful to Moussa P. Blimpo and participants at the DIME seminar and the Course on Distributional Impact Analysis held at the World Bank. We also benefited from the work and data from “The Impact of High School Financial Education: Experimental Evidence from Brazil” and “Parental Human Capital and Effective School Management: Evidence from the Gambia” and from conversations with members of both teams. We thank Kristoffer Bjarkefur for assistance with programpact Evaluation to Development Impact (i2i) Trust Funds.
<!-- Start of picture text -->
1. Definitions and Notation ...........eccccccesssccecesssceececseaececssseeeceseeeeceessaaeeeessaeeecesstesecesssseeeee4<br>2. Two Unconditional Approaches ............cccccccccccececessessecececeesseessseaeeeceeseessessaseeeeesessessseeeD<br>3. From Unconditional to Conditional AnallySis...........c.ccccccccccssssssssceceeeessssssssseseesessssssssseeesO<br>1. INCFOCUCTION......... ce ceectececessseceeeseneeececeenaececseaueeecessueeececseaaececessaeeecsseeesecessesueeeessseeeessseaeee@<br>2. Randomized Control Trials Without Endogenous Selection .............ceceesesseeeeeeeeeeeseeeeLO<br>3. Applications with Selection on Observables: Inverse Probability Weighting................11<br>4. Applications with Selection on Unobservables: Instrumental Variables .....................12<br>5. Interpreting Quantile Effects ....... ce ceccccesssscececeeecesssssseeeceesesssessssseeeesessssssstssseeseeseeesLG<br>1. INCFOCUCTION........ ee eeeccececeesseceeseseeececeeeeececseaaececseueeecesseneeeeceesaeeececaeeecesseeeessstieessssseresLO<br>2. Bounding the Distribution of Treatment Effects 2.0.00... cecccesssscceceeeessssssssseeeeeesesssssseeeLZ<br>3. Point Identification of Features of the Distribution of Treatment Effects...................4.20<br>A. Estimation Methods ..........cccescscccssssececsessececesseecececseaececsesaeeecssseeeecesseseeessesaeeecssseeeeess24<br>1. INtFOCUCTION......... ce eeettececessseceeeseeeeeeeceeaaececscaueeececsueeececseaseeecessueeeceseesaeceseeteesessseeeessseeesOL<br>3. Conditional Average Treatment Effects ...........ccccescsscececeeesssssstsseeeesesesssssstsseeessessessss<br>1. INCFOCUCTION.........ceeeeececeeeeseceeseseeececeeeeececseaaececseaueeeceseaeececeeaueeecseaueeeceseaeeesseesaeeseesseeesOD<br>2. Statistical INFEreNCe «0... eeccscecssssececesssececeeseeeececseauececseueeecesseeeceeseaseeesseaeeeessseeeesssseesDO<br>A. Power Calculations .0........cccccccesssececssssecececseeaececssuececssseeeecesssaecesseaeeecssseeeecesssseessesseeessl<br>1. Financial Education RCT in BraZil.........e ce eeeeecceceeesstececsessececessseeecesssssecesssseeecsssteeeesseeees44<br><!-- End of picture text -->
<!-- Start of picture text --> Introduction<br>Traditional methods to evaluate the impacts of social programs and the vast majority of ap-<br>plied econometric policy evaluations focus on the analysis of means (Carneiro, Hansen and<br>pact. While less frequently applied, these methods can provide information that is valuable<br>The purpose of this toolkit is to provide an overview of the questions such methods can ad-<br>sions of the assumptions they require, and practical issues in their implementation.<br>Mean impacts are a natural first summary statistic to describe the effect of a policy. The<br>mean impact of a policy or intervention tells us by how much the outcome would increase<br>or decrease on average when every member of a particular population is exposed to the pol-<br>icy or intervention. Thereby, they provide the central piece in any cost-benefit analysis.<br>However, a decision-maker usually requires information on the effects of a policy beyond its<br>or policy, but do not allow us to say anything about the distribution of the gain or how the<br>outcome distribution is affected by the program beyond changes in its mean. A positive av-<br>on distributional concerns, such as whether inequality is affected by the program, whether<br>some people are harmed by the policy or whether a particular demographic group benefits.<br>Even a purely welfare-maximizing social planner with no normative concerns for particu-<br>lar demographic groups, inequality or not harming anyone will often need information on<br>assumes that the welfare consequences of the distributional aspects of programs are either<br>assumption is strong. Many outcomes, such as educational attainment and health status,<br>rived from such outcomes, one needs to know the relation between the outcomes and indi-<br>vidual utility, which can usually at best be approximated. In practice, transfers may be costly<br>and implementing the optimal transfer scheme requires some knowledge of the distribution<br>of gains and losses, i.e., an evaluation that goes beyond the mean impact. Finally, some in-<br>terventions may work well for particular subgroups of the target population, such as those<br>living in urban areas, while there may be better options for rural populations. Knowing which<br>groups benefit more or less can help improve the targeting of policies and programs and<br>thereby help allocate limited resources more effectively.<br>lection of distributional parameters rather than a single scalar parameter such as the mean.<br>Hence, we refer to these types of analyses as “distributional impact analysis” (DIA). DIA con-<br>cerns the study of the distributional consequences of interventions due to participants’ het-<br>erogeneous responses or participation decisions. In particular, DIA investigates features be-<br>any — were produced, and who wins or loses as a result of program participation. The goal of<br>this toolkit is to provide guidance for researchers who want to use RCTs to answer questions<br><!-- End of picture text -->
simplify the exposition of the methods through the use of randomization as a statistical solution to the selection problems that are central to impact evaluation. Subject to addressing these selection problems, the methods we discuss are applicable to non-experimental analIn this toolkit, we study two related approaches to distributional impact analysis. The ences between (statistics of) the distribution of outcomes with the policy and the distribution of outcomes without it, such as the impact of a policy on the variance or a specific quantile of the outcome distribution. The second approach is to examine the . This approach answers questions such as what fraction of the population is harmed by the policy or what the bottom quartile of the impact of a policy is. It requires (statistics of) the distribution of the policy gains or losses, i.e., the distribution of differences between the outcomes of a given individual with the policy and without it. What distinguishes these two approaches is that the first focuses on how the intervention affects the distribution of the outcome in the population (e.g., how would the bottom quintile of finantire population), but disregards how any particular individual is affected by the program. The goal of the second approach is to analyze the distribution of these individual treatment effects (e.g., how individual gains from financial literacy training vary in the population). Due to this additional ambition, the second approach requires stronger assumptions for identification and estimation. To simplify the exposition, we focus on questions concerning unconditional distributional impact parameters first. That is, we start by discussing questions regarding treatment effect heterogeneity in the entire population and only introduce additional covariates to make the underlying assumptions hold. In practice, one may also be interested in how treatment impacts differ between observable sub-populations such as males and females, or how they vary with continuous covariates such as age or income. Therefore, we then review ways to subpopulations or along continuous covariates. This allows us to answer questions such as pacts are increasing in the baseline outcome. Such conditional analyses usually require additional assumptions and (often substantially) larger samples. If this makes them infeasible, conditional means are usually simple to estimate and can still be informative about heteroThe benefits of DIA outlined above beg the question why we do not see more of these methods applied in practice. Recent studies that estimate distributional impacts have examand education (e.g., Carneiro, Hansen and Heckman, 2003; Cunha, Heckman and Schennach, 2010). However, the literature is still nascent compared to the importance of the topic and the tools available. There are often good reasons to focus primarily on mean impact and the motives to not look beyond it depend on the application. For example, DIA usually requires
larger sample sizes, and many of the methods below are justified asymptotically and are not necessarily unbiased in finite samples, which can be problematic in applications with small samples such as most RCTs. Some methods rely on additional assumptions that are reasonable in some applications, but not in others. However, part of the hesitation to apply these methods also seems to stem from inertia and the fact that they are relatively new or have only recently become computationally feasible. Inertia may be an obstacle if researchers or their audiences are more used to and hence comfortable interpreting mean impacts and the conditions under which they are valid. To mitigate these issues, we review the practical considerations of statistical inference and power calculations for DIA. Finally, we illustrate select We focus on the types of questions introduced in the previous paragraphs, but there are many other parameters of interest, and questions that go beyond mean impacts. The methods and questions we discuss are not meant to be exhaustive, but are a selection to illustrate and how to detect them, and to shed light on practical issues such as required sample sizes and estimation of standard errors in (potentially dependent) samples. Programs to impleThis paper is organized as follows. Part 2 introduces notation and distinguishes the types of DIA questions we examine: We first introduce unconditional analyses of impacts on outcome distributions and the distribution of treatment effects. We then point out how these two approaches extend to questions of conditional heterogeneity. Next, we discuss key methods for each approach. Part 3 considers questions on the impact on outcome distributions. Part 4 considers questions on the distribution of treatment effects. Part 5 considers conditional analyses to answer questions of how treatment impacts differ with observables. Part 6 considers practical considerations for RCTs related to statistical inference and power calculations. Part 7 presents our applications.

# Questions of Interest and Definitions

In this part, we introduce notation and discuss the similarities and differences between the We then discuss questions that require each approach to be extended to conditional analyses. Unless stated otherwise, we consider an RCT with full compliance with treatment assignment to illustrate each method without concerns about selection issues.

Definitions and Notation observations, indexed by if observation
tion, although ost methods in this toolkit extend o more complex int rventions. The indienotes treatment participation: if observation ev inth reatment group participates and no one nthe contro group participates, ssume that outcom tre t, with cumulative s are contin ous. fr tenti | comes without istribution nctio thq tile . Potential outle Ea h__ these potential , th rib tion of ou om

F llowing the practice o e the population f interest th pop ation of individuals w under evaluation. For this reason, estimates from RCTs are often considered to be impacts “on the treated” and we follow this practice. This terminology only depends on what population one considers the available sample to represent, and not on which individuals from this sample participate or the methods used. In practice, there is usually a second selection step, which is compliance with We consider methods to adjust for this second selection step in Sections 3.3 and 3.4. These methods can be extended to the methods in Sections 4 and 5 or to correct for selective application to participate in the RCT. For mean impacts, researchers often settle for intent-to-treat parameters instead, to avoid further assumptions. Methodoin treatment effects. It will usually be more informative to analyze these two sources of hetTwo Unconditional Approaches , consider questions such as: nswer this question, we would estimate the aver t m ct s hospital regulation raise minimum levels of patie f While the e e ie nuo te or binary outcomes, they add little in the binary case s fe) tso and is completely characterized by its mean. treatment c g sa ula n terms of Note that this is take-up given participation in the RCT. For prediction about policy impacts in the population, we need the unconditional take-up model unless the RCT sample i representative of the entire population.
S question, we would estimate the treatme , or on a quantile in the left tail, such as e reform decreased persion student’s e question, we could stimate etreatme t on the minimum, a measure of inThe answer to the first question is based nthem_ n,a particular feature of th outco e istribution that is th focus fmostco v ti c stoevau e r, ndomization identi es the ull distrib ti ntreat t onsequently, under ull co pliance, w n stributi_ n oO S, , from the treatme t grou andthe s b e ed outc S ons to answer the remaining questions. We can measure the impact of ap licy on the median, minimum or low quantiles of the distribution of realized outcomes by taking the differcontrol group. By the same token, we can measure the impact of an intervention on a particular inequality measure, such as the standard deviation or the Gini coefficient, as in the third question.

informative about how a program’s impact varies across individuals, because they ignore who within the population belongs in different segments of the outcome distribution. We may want to study distribution of treatment effects, discussed in Part 4, can answer questions like: What proportion of students benefitf a ducational reform? For this question, we would compute Are the improvements in average pat comes from health facility inspections e que ti e ompute n ofthe m t ti, ethe minimum or maximum program impact? For this question, we would compute the relevant quantile of treatment effects.

Unlike evaluating policy impacts on outcome distributions, studying the distribution of impacts requires additional, sometimes strong, assumptions about how individuals would fare in a counterfactual treatment state. Even a perfect RCT cannot yield this information since the same person cannot be in both the treatment and control group at the same time. fictional example.
difference in means between the two potential outcomes. In our example, it is 1.6. Now tributions of potential outcomes. In our example, it is 2. Why is it larger than the ATE? Beindividual mobility across the distribution. For example, student E would have earned the ttle formalism is useful for clar fying this po pose hat th potential outcomes of observation correspond to quantiles , respectively. Then This individual im n e e a c n nideal RCT will o ly provide an estimate of either quantile of outcomes d e to the movement of individuals to different quantiles within the same distribution.

This equation clarifies why quan le effe ts are only equal to individual treatme e show the potential outcomes for a p rticula person in the treated (Treatment) andun _ t- population benefits, because her relative rank in the distribution is much lower in the treatment group than in the control group. While RCTs identify the quantile treatment effect, untional assumptions. This is again because we can never observe the same person in both the treatment and control group at the same time. Even if we observe the same person in the <!-- Start of picture text --> potential outcomes<br><!-- End of picture text -->
<!-- Start of picture text --> ae se<br>j | Group<br>= | Control<br>2 { Treatment<br><!-- End of picture text -->

treatment and in the control group over time, we must still make an assumption about how untreated outcomes vary over time and how they depend on prior treatment status. The key difference between studying impact distributions in Part 4 and impacts distributions in Part 3 is modeling how an individual’s relative performance changes when hey are in the decomposition. Once we have a credible model of the relationship between the right data, we can identify the entire distribution of impacts! From Unconditional to Conditional Analysis The discussions so far concern treatment effect heterogeneity in the entire population. We are often also interested in how treatment effects vary with observable characteristics, such as gender, income or baseline outcomes. As a few concrete examples, consider questions Part 5 provides tools to analyze such questions by discussing methods to study heterogeneity across and, to a lesser extent, within subpopulations defined by observable characteristics. It is important to note that these questions are about how treatment effects correlate with observable characteristics, not necessarily the causal impact of these observable characteristics on the treatment effects. To be sure, the average causal impact of treatment is still identified among subgroups, so long as treatment is randomly assigned within the subgroup. However, the subgroup’s characteristics, such as being poor or female, are not correlation.

Nonetheless, answering such questions can provide useful information to policymakers. For example, understanding how treatment effects vary with observable characteristics sugutes to the understanding of how particular subgroups of interest, like women or the poor-
Pitt, Rosenzweig and Hassan, 2012) and better predict a program’s effects in a population with different characteristics from the original experimental population (e.g., O’ MuirchearImpact on Outcome Distributions Introduction Policymakers often worry about aggregate treatment effects. For example, they may wish to level and the shape of the marginal outcome distribution matter more than individual responses to the intervention. This part provides tools to quantify these effects. There are different approaches to this task. We may analyze simple statistics, such as the and familiarity. For a more detailed picture, we may consider shifts in the CDF or its quantiles. We discuss quantiles for the sake of concreteness, but the methods in this part extend to other quantities under minimal conditions and with minimal adjustments. Quantile treatment effects are the difference between the quantiles of potential outtions (Firpo, 2007). We for ally q t e cts on the treated (QTT) as: th en u S cover t rfactua qu ie from untreated observations, which in turn will allow u timate We focus on uncond on-= re ffects. These effects are particularly relevant for policy evaluation, becau e t capture changes in the dispersion of outcomes both between and within subgroups of the population (Firpo, Fortin and Lemieux, 2009). Since the pioneering work of Koenker and Bassett (1978), a related literature has explored conditional quantile regression (CQR). CQR estimates quantile effects within subgroups under restrictive assumptions. Part 5 discusses conditional effects. Note that unlike average effects, unconditional quantile effects are not weighted averages of subgroup effects. This part continues as follows. Section 3.2 considers RCTs without endogenous selection. This perfectly randomized setup ensures that the distributions of potential outcomes do not depend on treatment status, so that randomization identifies QTTs without further assumptions. However, many applications deviate from this benchmark. For example, participants account for sample selection, as it does for average treatment effects. Section 3.3 surveys inverse probability weighting, which exploits the assumption of selection on observables

| nalogy to average treatment effects, we can also define the quantile treatment effect (QTE): . See Firpo and Pinto (2016) for additional discussion. ractice, it is unfeasible to analyze all infinite points of a continuous distribution. Therefore, we | m n vor to particular quantiles of interest. If outcomes are discrete, in principle we could measure treatment
Section 3.4 presents an instrumental-variable approach. Section 3.5 concludes with remarks on the interpretation of quantile effects.

<!-- Start of picture text --> Randomized Control Trials Without Endogenous Selection<br><!-- End of picture text --> This section considers ran ed trials without selection problems, i.e., we ass me: Potential outcomes, This is a strong assumptio ch is only likely to hold under idealized conditio . The probability of assignment to ent must be equal for all participants. bility of treatment take ust be constant. This condition requires full compliance with for all participants with ame probability. Should there be nonparticipation or nonrewhich satisfy assumption (i) as RCTs with perfect compliance or RCTs without endoge ous selection. Assumption (i) e s hat the counterfactual distribution of potential ou o Therefore, a consist n ator of the QTT is simply the difference in quantie een (t) =q )-4q qd is any consi tent estimat he -th e of outcomes such as mpirical quantile. This estimator is s raightforward to extend to the statistics for treated and untreated b . Note that we may also estimate from the control group under as u It is also possible o recover th a quantile regres on, in the same way that To do so, r na quantile regression of whereas the intercep gives the quantile q Assumption (i) is trong and fails in m plications. For ample, the probability of assignment to treatment may depend on ipants’ attributes, such as region of residence. The equivalence between the distributions of potential outcomes then breaks down, leading to composition bias. For the same reason, endogenous selection into treatment is a concern when participants do not comply with treatment assignment. Similar to average effects, we may either settle for intent-to-treat effects or account for selection and estimate effects on the treated under alternative frameworks, as the next sections show.

If the probability of assignment to treatment differs across design, reweighting the sample yield mators. See Section 3.3.

in the treatment group to estimate these di
Applications with Selection on Observables: Inverse Probability Weighting In the ideal RCT of Section 3.2, treated and untreated observations have the same distribution of potential outcomes. Many applications deviate from this benchmark, though. This section considers the weaker assumption of selection on observables. Selection on observables is an assumption of conditional independence between potential outcomes and which relaxes our previous assumption of unconditional independence. ables. Therefore, treated and untreated participants differ only in observed characteristics (other than treatment status). Correcting imbalances in covariates should restore the equivalence between outcome distributions and allow us to estimate treatment effects. This intuition motivates two classes of estimators: inverse probability weighting (IPW) and matching. IPW consists of reweighting the sample to balance covariates across treated and untreated observations, which should then have the same distribution of potential outwhich Firpo (2007) extends to quantile effects. Donald and Hsu (2014) consider CDFs, observations according to covariates and differencing outcomes to estimate treatment Matching does not readily extend from average to distributional effects, because it relies on the law of iterated expectations (Frdlich, 2007), which fails for quantiles and other nonlinear Statistics. The consistency of IPW estimation of distributional effects relies on the same assumptions and formulas as average effects. It requires: Selection on obs | outcome Assumption (i) is the key umption. Assu_ ptio (ii) is crucial, albeit standard. We implement the IPW estimato n two ste rst, we he weight function: is the unconditio bili m ic stimate with t of the treated in the sample. The conditional probability prope e, a building block of many estimators of average effects. S ix 2 for a discussion of estimation. We abstract henceforth from nonparticipation and nonresponse. This assumption is also known as unconfoundedness, ignorability and conditional independence. DiNardo, Fortin and Lemieux (1996) use reweighting to decompose changes in the density of wages in the U.S. in an early and influential paper. However, they do not investigate the properties of their estimator. Cattaneo (2010) considers multivalued treatments. Additional regularity conditions are required. See Firpo (2007, p. 263).
To bu din n for , remember t h to estima e antile treatment eff to treat d. n still estimat function untreate ho , their distribution of potenti smustb c le to that of the treat .H we reweight this subsample. sumption e on observables, we only need to balance the distribution of in the propensity score: we give more weight to treated observati_ s resemble the treated and less weight to ations with cha cteristics that are nc n among the After constructing @ e proceed ove. The QTT estimator is the difference between the quantile qd we compute separately for nd untreat servations. extends to other statistics. It ccommoda ternative treatment effects, such as effects on the entire population: it suffices to adjust the weight function (Firpo and PinIt is also possible to estimate quantile effects with quantile regressio e should runa @ included as control variables, we would estimate a conditional quantile e see Part 5). The assumption of selection on observables is strong and has receiv ensive discussion in the program evaluation literature. The covariate set must be rich enough that the unexplained component of treatment take-up is independent of potential outcomes. This approach is popular in observational studies, and may also be useful in many RCTs, since designers often collect extensive data on participants’ backgrounds. Imbens (2015) suggests placebo tests to assess plausibility. The assumption becomes testable if a valid instrument is Applications with Selection on Unobservables: Instrumental Variables The previous section leveraged the assumption of selection on observables to address selection bias. This section explores an alternative strategy: instrumental variables (IV). IV estimation of average effects has a long tradition: see Heckman and Vytlacil (2007) extend it to effects on distributions. Important early contributions for quantile effects are Melly (2013a,b) propose estimators of unconditional distributional effects. As Angrist and Imbens (1994) argue, the IV framework only identifies treatment effects on compliers — the subpopulation whose treatment status depends on the value of the instrument. Further assumptions are necessary for identification of effects on the treated. Froparticipants never take the intervention. For intuition, consider a clinical trial of a vaccine. Our candidate instrument is randomization. Participants in the treatment group might refuse the vaccination. On the other hand, the control group has no access to it. Noncompliance is If the data include sampling weights, multiply @ importance of verifying the assumption of one-sided ompliance in practice.
treated are compliers, which allows us to identify effects on the treated from effects on This section presents the ramework o ch and M 013a). We assume the existtion.

d tion: If the assumption of on ompliance fail this proce c nsistent estiC ndition (i) ensures that the instrument is valid. We need full independence, which is stron er than the assumption of uncorrelatedness of the linear IV model. In an RCT, it means does not depend on treatment assignment, which is reasonable. Condition (iii) is analo ous to the assumption of common support in the previous subsection. We implement the | estimator in two steps. First we compute ei ix es th on of the conditional build f , suppose that the instrument is randomization, i.e., . The nof o utcomes is initially balanced across the treatment and | groups, due to ran o ment. Noncompliance distorts the distribution for the

compliers, but we o ow who they are. Note however that the distribution of noncompliers is th s n the treatment and control groups, because of randomiz tion. Therefore, giving negati e weights to the outcomes of noncompliers from the trea ment with the distribution of Accounting for cova iates accommodates applications in which the instrument is only conditionally exogenou such as RCTs with stratified randomization and observational data. We might also want to purge indirect effects of an intervention to focus on particular mechcompliers, the identific ti Frdlich and (2013a) holds nonetheless. After constructing we procee ove. The QTT estimator is the difference between the qua til qd The estimator extends to multivalue ndcontinuous!IV _ ee Section 3.1 in Frélich and Melly (2013a). Additional regularity conditions are required. See and Melly (2013a, p. 391). In particular, we assume that there is no endogenous attrition. The authors pr adjustments for attrition in their Section 4. If the data include sampling weights, multiply @
we compute separately for treated and untreated observations. other statistics, such as the variance.

Interpreting Quantile Effects Quantile effects are often misinterpreted, which can result in unwarranted conclusions for policy. This section discusses and illustrates two common pitfalls: implicit assumptions of rank invariance and extrapolating treatment effects to different populations. It is easy to conflate quantile effects (i.e., changes in the distribution of outcomes) and individual effects (i.e., the distribution of changes in outcomes). One might naively reason: ment.” Recall that the quantile effect is the difference in quantiles between treated and untreated participants. Thus, we implicitly assumed that John’s outcome would be equal to the (cf. Part 2). Rank invariance is a strong assumption, which is implausible when treatment effects differ across observably identical participants. Assumptions of rank invariance are often subtle. For example, researchers might argue that all participants benefitted from the intervention if all quantiles effects are strict y positive. They would have invoked rank invariance across treatment status. One mig t also equate quantile effects and changes with respect to baseline outcomes instead of interpretation requires rank invariance both across treat states and over time. For illustration, consider Figure 2. It shows estimate eatment effects, based n data from Simulation 1 (see Appendix 1 for details). We set the average treatment effect at each quantile of basel tcomes is close to the overall average effect, as can be seen from the line labeled “ATE at Quantile of Baseline Y”. Y_ , quantile effects are increasing. The QTT (dashed line) clearly differs from both avera e effects at baseline values (thin line) and quantiles of the distribution of individual effects (thick line). This discrepancy underlines the notion that implicitly assuming rank invarian e and misinterpreting quantile effect may lead to incorrect conclusions. Note, moreover, th t all quantile effects are positive, even though some participants are worse off after treatA different concern is the comparison of treatment effects across (sub-)populations. This part has surveyed the estimation of changes in the distribution of potential outcomes. As Section 4.2.2 argued, RCTs ide these marginal distributions, which allows us to compute treatment effects under mini a ssumptions. However, these effects depend on the unidentified joint distributio of pare results between diff ren ( b-)populations, unle s we — n account for differences in both the distributions of a e relation between these discrepancies arise rom differences in respons s tot intervention between men they can stem fro gender differences in the unknown distribution of treatment effects, in the distribution of | or both. We can estimate the marginal distribution of outcomes, which As before, it is possible to estimate the QTT from a weighted quantile regression. The slope gives the QTT.
<!-- Start of picture text --> wv<br>o<br>3<br>i= nN - _~ _-——"<br>E eo<br>s |~—<br>e<br>Oo<br>0 20 40 60 80 100<br>Quantile<br>——-— QTT —— Quantiles of F,<br>—— ATE at Quantile of Baseline Y<br>Note. Based on 10,000 observations drawn from Simulation 1. See appendix 1 for details,<br><!-- End of picture text -->

lows us to compare treatment effects on quantiles that correspond to the same value of nd strategies to compare quantile effects across groups.

Distribution of Treatment Impacts Introduction Part 3 discusses methods to estim te the impact of a policy or program on (functions of) the marginal distribution of outcome __n this part, we are interested in the distribution of indiThe distribution of treatment effects is required to answer questions such as: What proportion of the population was hurt by the program? As described in Part 2, these questions cannot be answered by the methods in Part 3, because, rather than effects on distributions, they concern effe ts on individuals. The central difficulty of tudying the distribution of individual treatment effects i that they require a . In order to estimate the impact of treatment o person p antly, the ind vidual estimates are only of interest n thedistributio of treatment effects in or er to answer qu_ tions such as those ve. The estimated
, is of less interest both because it is noisily estimated and because this individual has are dy been treated, so this treatment effect provides little information about future impleme ations of the policy. In Part 5, we discuss methods to analyze heterogeneity, which is informative about the types of people who benefit from a program. In contrast to studying average treatment effects or di rence in marginal distributions as in Part 3, individual counterfactuals cannot be identifi d by r domization alone. Randomly selecting treatment and control groups identifies ginal distributions and quantile treatment effects from Pa t 3 inf rm us of changes in the frequency of outcomes and inequality, but they only provide limited information about idiosyncratic responses to treatment (Bitler, Gelbach and Hoynes, 2014). The distribution of impacts only equals the difference in marginal potential outcome distributions (or quantile outcome distributions. This condition implies that observations with the same rank in the treatment and control outcome distributions are appropriate counterfactuals for each other. When the rank invariance condition is satisfied, estimating the distribution of treatment effects only requires the methods discussed in Part 3. Rank invariance is a strong assumption. If it does not hold, the parameters from the previous part are still identified, but their interpretation can be difficult. The interpretation of the distribution of treatment effects is always clear, but the distribution is no longer identified by randomization if rank invariance fails. As a result, we must either make additional, sometimes strong, assumptions that imply individual counterfactuals to point identify the distribution of treatment effects, or we must settle for partial identification, where only a range of parameters is identified. The empirical strategy depends on the validity of assumptions that we need to assess for the case at hand, so we first provide some background and then discuss more general estimation principles before we discuss a specific model that applies to many evaluations. In particular, we first illustrate the identification problem using the variance of treatment effects as an example. In Section 4.2, we discuss partial identification and how additional assumptions can narrow the bounds from this approach. Section 4.3 discusses point identification. We first illustrate the required assumptions and provide an overview of common approaches to justify them. We then introduce methods to estimate features (moments) of the distribution or, under more stringent assumptions, the entire distribution. In practice, we need to adapt the methods to justify the assumptions and the estimation method to our application and data availability, so we close by giving an overview of a specific panel data model. The model is general enough to cover many common settings and can serve as a blueprint which is adaptable to other situations.

majority of impact evaluations focus on average treatment effects: . This is the first moment of the treatment effect distribution, which pro of location of the distribution, i.e., how much individuals benefit on ave As Bitler, Gelbach and Hoynes (2014) point out, quantile effects contain additional information about the distribution of individual effects. If one or more quantile effects are positive, at least one participant benefited from the intervention. The converse is also true. Note that Makarov bounds allow us to quantify the shares of winners and losers under minimal assumptions: see Section 4.3.
distribution. For example, the variance of treatment effects is the second (centered) moment of this distribution and provides a measure of the dispersion of the treatment i.e., how much they vary across individuals. In the fictional RCT in Part 2, the varife) b ons and can be estimated treat ent andco tates simultaneously. As t he data do not identify , Without additional assumptions. Consider the ata in Table 1, but more realistically, suppose we do not k h —~(Y (Y (Y Thus, the va ce of treatment effec fi face an analogo s iden fication problem when estimating the entire distribution of individual treatme effecs r other features of this distribution: while features of the marginal distributions an be calculated from the data, parameters that describe the relationship are necessary for point identification of the distribution of treatment point esti ates a discussed in Section 4.2. Bounds are often simple to obtain, but inference can be difficult. They also tend to be too wide to be informative and narrowing them requires additional assumptions. Second, we can estimate the (parameters of the) relationship between counterfactual outcomes, as we discuss in Section 4.3. This requires further assumptions and modeling, such as amending the RCT with a model of participation choice.

Bounding the Distribution of Treatment Effects While the data cannot identify higher order moments of the treatment effects distribution without additional assumptions, they can identify a range of values that must include the tion. This is distinct from the variance of the estimate of the average treatment effect due to sampling that is To keep the example simple, we abstract from sampling variation throughout.
Ss ontin o t evarianc and then extend this idea to edist bu on is the correlati n between he standar d is the sta dard deviation of .Wecane timat from the tr a n group. T e only remaining nknown is he cor on , which m tlie tween -—1 and 1 by definitio Therefore, withou addiOnal assum tions, Substituting these bounds for

## <u>—_—</u>

(v

# ~~—_—_~~

(Y only 1.3 standard deviations from zero, so the data do not seem to rule out negative treatpotheses. For example, the classic approach to impact evaluation assumes that In Simulation 1, where tr ects are normally distributed with unit mean and variance, the bounds on the standard deviation of treatment effects tell us that it must include estimates over three times higher than the true standard deviation. Th bounds can be tightened using additional assumptions. The results in Heckman, Smith nd Clements (1997) suggest assuming that potential outcomes are positively correSubstituting these bounds into the ( (Y return to this problem at the end of this section.
How does the example exten to learning about the distribution of treatment effects without making assumptions beyond what was required in Part 3? Just as in the case of the variance above, one can stillu eth margi al ou come distributions to calculate bounds on the entire distribution of trea en effects A estimates of the marginal pontial outcome distributions en the distribution of treat ent ffects, Bounds may also be informative about specific questions of interest. Fo example, marginal potential outcome distributions are sometimes sufficient to find wheth anyone was hurt by the program or the share of treatment effects that are negative, i.e., bounds indicate the range of values for the joint distribution that are co — stent with the observed marginal distributions. We break down the pieces of the lower — und as an illustration. The maximum function in the lower bound just imposes the logical restriction that a CDF cannot be negative. If we ignore the a now, the bound simplifies to: which is just the largest vertical differe ce b treatment and control group CDFs. As with the lower bound, the minimum in the upper bound imposes the restriction that The first panel of Figure 3 illustrates Makarov Bounds for the two data generating processes described in simulation 2. In both cases, the average treatment effect is 1, but the standard deviation of treatment effects is 0 in the first case and 5 in the second case. Despite The second panel of Figure 3 plots the densities of outcomes generated from the second data generating process. The treated outcome density has a slightly higher mean than the ed density’s mass falls below the minimum value of the untreated potential outcome densihurt by the program in this hypothetical example. We cannot say who was hurt gram, but we know that the proportion of people hurt . In this simulation, this bound indicates that at least conditional distributions. Note that we have simplified the notation by assuming continuous CDFs.
<!-- Start of picture text --> &<br>2<br>2oed,<br>e ee ——<br>2 ee<br>$ c<br>EOo<br>i<br>0 200 400 600 800 1000<br>Quantiles<br>——_ QTT, o=0 QTT, o=5<br>Quantiles of F,, o=0 Quantiles of F,, 0=5<br>——— Makarov, o=0 Makarov, o=5<br><!-- End of picture text --> <!-- Start of picture text --> +<br>I\<br>/\<br>| |<br>| |<br>2 | |<br>ea { |<br>fa | \<br>{| \<br>| |<br>“ l 4<br>° eT/ \<br>-20 -10 it] 10 20<br>Outcomes<br><!-- End of picture text -->

In these examples, we have ignored sampling variation, i ., that th marginal distributions are estimates rather than the true population distributions. Sampling variation exacerbates the problem that bounds are often wide (Heckman, Smith and Clements 1997), since estimation error makes the possible range of parameters even wider than the point estioften difficult (see Subsection 6.2.1). Nonetheless, bounds are usually easy to calculate and clearly demonstrate what the data imply about the distribution. Thereby, they can provide a useful informal assessment of what can, and (more often) what cannot, be learned from the data alone. For example, if the point estimates of the variance bounds include zero, the data are not informative about heterogeneity of treatment effects without further assumptions.

Point Identification of Features of the Distribution of Treatment Effects When we are willing to make additional assumptions, we can make more progress in point identifying the distribution of treatment effects. We first continue the example of the variance of treatment effects and illustrate what kind of assumptions are required for point identification and what previous studies have done to justify them. The conditions under which these assumptions are plausible crucially depend on the substantive problem and the
<!-- Start of picture text --> available data. We use cross-sectional not tion to keep the exposition simple. In practice,<br>methods to their cases, we discuss two ge eral estimation frameworks in the next subsec-<br>tion. We conclude with an example of asp _ific panel data model that can easily be adapted<br>y common empirical settings.<br>tinuing the example of the variance of treatment effects, we can point identify<br>we are willing to assume a particular value of the lation between the treated<br>untreated outcome. For example, if potentia tcomes are uncorrelated<br>~(y<br>Vv seen above, the true covariance in our example is itive, so assuming that it is<br>s to an overestimate of the variance of treatment e .<br>More generally, we need to be able to identify the dependence between the treated and<br>untreated outcomes to identify the distribution of treatment effects. So far, we have primari-<br>plicitly chooses values for the parameters that determine the dependence of potential out-<br>comes. While additional assumptions are always required, a much better case for them can<br>be made in practice for two reasons. Fi e esearcher may have a model that includes<br>the relevant dependence parameters a e able to estimate them from this model<br>under weaker conditions. For example,<br>vidual choice such as the (generalized) R |, which can be estimated under conditions<br>tion, we may be able to estimate the required parameters under plausible as umptions by<br>amending the RCT with a model of individual choice. Second, the assumptio s are usually<br>only required to hold conditional on observables, i.e., after controlling for<br>tion 4.3.2 describes.<br>Identification of the distribution of treatment effects comes from restrictions of the depend-<br>the size of the treatment effect. In this subsection, we discuss why these assumpti ns ar<br>necessary, how they solve the identification problem and why cova iates are importa t eve<br>interchangeably. We define parameters and estimators in terms of here, as the dist butio<br>of individual “intent-to-treat” parameters is unlikely to be interes ng and is not cle rly d<br>fined. If perfect compliance fails, one needs to adapt the re-weig ting or IV methods from<br>part 3. For simplicity, we assume that the effect of covariates is linear and additively separa-<br>one equation as:<br><!-- End of picture text -->
This model is simple to extend: e.g., This model is simple to extend: e.g., rfunctionso is raightforward. It is also m re general than the resembla eto astan ardr gressio qu tion suggests. In impact e aluation, we are interested in as the linear projection coeffic nt, so that tion (but ot necessarily independent). If , th contro grou still identifies so that t e treatment group identifies th (not neces lyc_ sal) r lation he treatme t effect to observables. Thus, as long as r domization orks, wec purge eobs vabe part of the model by partial regression, i.e., by regressing In practice, including in the model and estimating it in one step e more c nvenient han partial regressio . Howev al regression provides a use ought device, as . This is a benefit of rando that helps to iden y the distribution eatm cts. To see that it is not su n , con ider the anal gy to identifying m ffects, one would compute the mean of difference. xtend ng thi istri utions, the treatment group identifi s the distr ution of and ontro group identifies the distribution of not back o ‘the distribution of | from these two distributions: c rary to means, the difference b twee two distributio sis not the distributi n of the diffe nces. To make progress, additional strictions on the de endence of

quired. We saw above that, if we are willing to assume hat individua s’ potential outcomes point identified. Technically, we have achieved identification by imposing a moment restriction. All three terms in the formula for the variance of treatment effects above are secdistribution. Restricting it to zero le ves only terms that we can estimate, since the other , only depend on the marginal distributions. Similarly, onal case, we can i ntify the variance of individual treatment effects if We can esti the control determined by the marginal distributions are zero, for example, the third moment of the distribution of treatment effects is identified. We provide more detail on moment estimation in Subsection 4.4.1. The limiting case of this idea is to assume that tribution only depend on the moments of the marginal distribution. This implies that treatNote that if randomization has been compromised, e.g., by noncompliance, the same has to be done for
<!-- Start of picture text --> tional on. Then the entire distribution of treatment effects can often be estimated using<br>the meth | of deconvolution, as discussed in Subsection 4.4.2.<br>The discussion above shows that identification of (features of) the distribution of treatment<br>effects requires independence assumptions. How can we justify such assumptions? They are<br>satisfie if treatment effect heterogeneity is unrelated to any unobservable aspects of indi-<br>domization, conditioning on cov riates is typically required for identification here. The co-<br>variate control for variation in that is potentially correlated with impact heterogeneity.<br>Consequently, the required assu_ ptions to estimate (moments of) the distribution of treat-<br>ment effects may be pla sible w en the data include a rich set of individual characteristics.<br>This is because the condi onal independence assumption allows<br>characteristics of person but not on any unobservable characteristics or variables that have<br>excluded from t emo el. Such a restriction seems more plausible the better the mod-<br>are u correlated amounts to the (usually unrealistic) assump-<br>hat levels ( ) are not related. However, RCTs in economics often col-<br>etailed info m nincuding aseline values. Including baseline outcomes changes this<br>restriction to as that gains f om the program are unrelated to deviations of the out-<br>come from its expected path. While the independence assumption usually seems unrealistic<br>in cross-sectional applications, it often seems more plausible in panel data settings.<br>Consequently, a key component of making the conditional independence assumption<br>and Pischke, 2009), this suggests controlling for many characteristics in a flexible way and<br>assessing the robustness of results to changes in the conditioning variables. However, just as<br>tion can be justified or not depends nthe application at hand.<br>If we find the assumption that is conditionally independent of<br>Vytlacil (2005) show that this assum _ tion can be weakened by taking a andom effects factor<br>isa vector fall obser e potentially co lated with<br>, the distribution of<br>using d convolution Aakv , Heckman and Vytlac ( 005) assume tha<br>specific random effe with particular distribution an estimate the mo — using maximum<br>using a less restri tive fixed effects<br>approach, as in the example in Subsection 4.4 4 below.<br>If we have access to multiple related m asure o important omitted confounders (or<br>proxies), we can estimate the distribution of<br><!-- End of picture text -->
| ine and identically dis buted across inditericall i can be recovered using deconv_ ution. The supplemental appe x ac 2011) provides more detailed in ructions on how to use Kotlarski’s Theorem to identi

Estimation Methods This section presents two approaches to identifying and estimating features of the distribution of treatment effects. In Subsection 4.4.1, we extend our discussion about estimating the variance of treatment effects to higher order moments of the distribution. This first approach relaxes some of the assumptions required to estimate the full distribution of treatment effects. It is therefore preferable for questions that can be answered by moments of the distribution of eatment effects. For example, what is the variance of treatment effects? In Subsection 4.4.2 we show how to use deconvolution to estimate _e enti e distribution of , when our model sufficiently captures the vation n the untreated ent. All of the poli y questions raised at the beginning of this secti n can e answered by computing features of thi distribution. Moreover, it is simple to calc late th variance of the estimated distribution to sess the variability of treatment effects or the fraction of individ. We present simulation results to illustrate these two methods in Subsection 4.4.3. We continue to use o r basic model from the previous section to keep the discussion simple. However, caution is warranted when applying the methods in practice, as the required conditional independence assumption is strong. This assumption likely requires a richer underlying model and may thereby limit how useful these methods are in practice. Subsection 4.4.4 presents additional details of these methods in a specific panel data context where the conditional independence assumption may be plausible.

As the example of the variance shows, moments of the distribution of treatment effects are identified under weaker conditions and often provide sufficient information to answer policy questions. The first two moments of a distribution imply its variance and the first three moments imply its skewness. Skewness is informative about how lopsided the distribution is. For example, a high and negative skewness indicates that there is an important number of with treatment effects far below the mean impact. With many vaccines and medications,

-th moment of the distribution of treatment effects is defined as E(A )
one may be worried about very lopsided distributions where most people benefit modestly, tive provides evidence on this issue. While moments a eus lly easy to estimate by their sample analog, sO we Cannot estima its ome direc y. However, we can calculate e residuals rom partial regressio , the residual from a partial regression of for eatm nt gro p observations and tions.

treatment effect Ww ca S t artial residuals: rst two ter c e d t rtia residuals and is ju age treatment effect. Thus, each of these expectati ns ca mated par Is. This is possible because we have assumed that This is just a conditional version of the assumption hat that we u tify the variance of treatment effects in our exampl abo T s of the distributio y: Once we know the ance of treatment effects, ly unknown term in this equation is . To esti it from rtial regression residuals, we need to extend our assump that terms h do no p dsolely on the marginal distributions of potential outcom re zero to . This is a restriction on how the varition, the third moment h fe) p iven by: Thus, we can estimate S e f ent effects from moments of our partial residuals. Appendix 3 shows how to use the bino mula to solve for In theory, we can use this method to estimate moments that are not determined by the marginal distributions to . Of course, es mating infinitely many moments is impractical. Thus, one may pref timate the first moments and select one of the (usually many) distributions they are consistent with. For e ample, in Section 7.2, we estimate the entire distribution of treatment effects by assumin they are normally distributed and by plugging in mean and variance estimates. Wu and Perloff (2006) estimate the first four moments and recommend selecting the distribution according to the Principle of Maximum Entropy. This distribution is unique and is “maximally noncommittal with regard to missing information” (Jaynes, 1957; Wu and Perloff, 2006). See
Deconvolution methods estimate the distribution of treatment effects by removing the variation due to the error term from treatment group observations so the only remaining variation is due to the treatment effect. Disentangling the variation of the treat ent effect from f or term requires a strong conditional independence assum tion. More prec nal independence requires that the random component of conditional on st n of the sum of two random variables is called the convoluti n of these ra a Deconvolution m thods_ ndo this convolution by removing the distributio of one of the random variables fr mthe m.In our simple model, deconvolution requires to be statistically independent of A testable implic fe) an be tested th of the simulations in Subsection 4.4.3 below, even thou derlying assu ns fu. tions (e.g., BonDeconvolution is often implemented via estimated characteristic fu. Arellano and Bonhomme (2012) find works well in practice. It estimates times. To build intuition for this algorithm, consider again the partial esi uals from above. ment gr up, th partial residual is the sum of the treatment effe t and the errort m, are independent by assumption, we can create pseudo-draws of ximate s dist bution using: m, is a control-group partial resi is a treatment-group partial residual. Mallow ’ Algorithm “shrinks” estimates of tial resi uals under the assumption that lar e va f treatment group partial residuals are associated with larger than usual error ter and small treatment group partial esiduals are sample of treatment residuals by randomly matching its shrinkage estimate of uals and estimating a new shrinkage distribution. Mallows’ Algorithm is described i e conventional linear m del below: The model also requires additive separability and restrictions on the dependence of cussed above. However, these assumptions are specific to the simple model that we use__rclar y he e. They can potentially be relaxed, e.g., when many periods of pre- and post-treatment observatio sare vailable, one
To be er di tinguish treatmen r al re s, we __ Il fo no and Bonh mme’ (2012) notation: are just vectors o si stimat hereas eddr wsfro the(un nown d fe) ment S. To prepa ethe da for allows’ Algorithm, use estimates from he co al linear tively.

rmutation of =sor A- C). Repeat step 2 through 4m times.

. Th empirical distribution of the full set of sis herefore an estimate of the distrib on of Table 2 shows the results of Simulation 3. We present results using 6 different combinations of parameters which determine the correlation between treatment effects and omitted variables and the independence of treatment effects from potential outcomes. The moment and deconvolution estimation methods always yield the same average effect, but the differences are particularly pronounced in the respective estimates of the standard deviation of treatment effects. As we would expect, both the deconvolution and moment methods perform relatively better when their assumptions are satisfied, which is the case for deconvolution in column one only and for the moment-based approach in column two only. In our simulation setup, the moment approach yields standard deviation estimates, which are closer to the truth when an omitted variable is negatively correlated with treatment effects, regardless of whether the covariance satisfies the moment or deconvolution assumptions. In contrast, the deconvolution standard deviation estimates are a better approximation of the truth when the omitted variable is positively correlated with treatment effects.

vect rc mplete repli atio softhev ctor houldbe sedi theal_ rit m since it is mean zero by constru tion replication whe lication or with Arellano and homme (2012) ea th pro edur 2000 times and discar the first 500 iterations. In practice, the lar t number of iteratio s that omp ting and time constraints allow is desirable.
orrelation with O_ itted Variable Negative Positive Standard deviation Standard deviation Standard deviation stimates based on Simulation 3, with treatm nt effects are independent of poten al outcomes, s_ the de onvolution assumptions are sati fied. is 0, the moments assumptions are atisfied. When or—1, here is an omitted variable which is positively or neg tively correlated with both treatment outc mes and tre tment effects, respectively.

In practice, we likely need to extend the simple model from above and adapt the methods to a specific setting. To provide some guidance for this, we next discuss the panel data setting from Arellano and Bonhomme (2012), which can easily be adapted if the assumptions above do not seem plausible or the available data set has a different structure. The conditional independence assumption required for deconvolution is more likely to be satisfied when the researcher can control for a rich set of individual characteristics. Panel data are a special As discussed previously, the central difficulty in studying the distribution of treatment panel, the same individual can often be observed in the treated and control states at different times. This allows one to model individual heterogeneity quite flexibly, but instead requires restrictions on how the treatment effect and the residual are allowed to vary over time, as we show below. Arellano and Bonhomme (2012) show how to estimate higher order moments of the treatment effect distribution and apply deconvolution methods to estimate the full distribution of treatment effects when panel data are available. Consider the panel er o e Oo wewor ghout this section: Outcomes are sh_ ed by the treatment effect in all p riods after an individual is treated o this model imp es that the treatment effect consta t over the time horizon of
This may be a strong assumption, since treatment effects may not begin immediately and/or may fade over time. For example, when analyzing the impact of a school management intervention on students’ test scores, this amounts to assuming the intervention had constant impacts over the entire period of data collection (i.e., three years in the application in Section 7.2). With very long panels, this can be relaxed by modeling decay early over time a cording to the (non-stochastic) line equation re mme (2012) show that the variance and distribution of ) are ti the followin two assumptions: Conditional independence: _ is statistically independent o The mode equires restriction of the d dence of errors over time, so that we can estima e the time series process of In their application, they sume that the errors follow an autoregressive or moving-average process with in ependent and identically distributed innovations in each period. Note that the conditional independence sumption is now conditional on an individual fixed effect and the time varying controls, |. The only remaining source of potentially problematic residual variation is time- arying nobservables. While this is still a strong independence assumption, it is likely m e pla sible than in the cross-sectional setting. The second assumption requires that the error terms. Importantly, the relati nship between is left unrestricted. With panel data, the identification of the distri tion individual treatment effects reWhile this allows us to estimate particular tribution of treatment effects, as these individual e mates will be quite noisy in most applications and most policy relevant questions relate t future realizations of treatment effects. Therefore, estimating the distribution of treatment effects using these models requires the data to have several features. Here, we will focus on model (1) under the assumptions of Arellano and Bonhomme (2012). In particular, we assume someone is “treated” if they have ever received the treatment. In order to id ntify this model, individuals must be observed at least three times, including at least one p riod before and another after the individual reis observed three times and receives the treat-

Alternatively, we could define In non-experimental settings, t e relationship between Control group variation aids estimation of common para e rstha_ ary over time.
Consider evari o inth t to tify fthe a ove parameters. First, ’s observations ar_ informative a tth values. If not for could be estimated epara ly for each individu |. For this reason, only identifi fort subpopulation of dividu Is who are observed in both the treat nt and control ates. co trast, untreated observations are informative about the com on

<!-- Start of picture text --> In principle, the distributi nof = s if ind lis only treate i one period<br>or, even in a cross-section, if th errors are independ nt and denticallyd t uted. This<br>assumption implies that the res ualfro anyindividua at any me period<br>t time<br>sectio this reduces to the assumptio that residuals r m ano her individu | t the same<br>time, . ploiti g el data t use only within individual<br>variati s we do here, elaxes this su _ ptiont uire only residuals from the same<br>’s observations _ esti<br>below which is generated f om Simulation 1 with left panel) and<br>In particular, notice that the distribution e estimates of the en cts is<br>quite different from the true distribution e nvolution estima n<br>estimates are much more similar when In general, estimates of<br>i<br>3 3 5 3 i $ 's 3 5 5 7 3<br>Treatment Effect Treatment Effect<br>—_——_ True Distribution Regression Estimate ——— True Distribution Regression Estimate<br>—-—-— Deconvolution Estimate —-—-— Deconvolution Estimate<br><!-- End of picture text -->

<!-- Start of picture text --> i<br>3 3 5 3 i $<br>Treatment Effect<br>—_——_ True Distribution Regression Estimate<br>—-—-— Deconvolution Estimate<br><!-- End of picture text --> <!-- Start of picture text --> 's 3 5 5 7 3<br>Treatment Effect<br>——— True Distribution Regression Estimate<br>—-—-— Deconvolution Estimate<br><!-- End of picture text --> when individuals are only treated in one, or even a few, peri Moreov mates of The required number of time periods depends on the SEs of er the variation due to the estimation error is minimal can easil be examined in any given application.
treatment impacts on a particular individual are not of particular policy interest, since the individual has already received the treatment in question. The distribution of treatment effects is potential more informative about how individuals are likely to be affected by future applications o a program. Estimates of individual treatment effects may be of interest varies by individual characteristics, which we discuss in more detail in Part 5. For this pur ose, the individual estimates can be regressed on covariates, but standtimated.

A less ambitious goal thane mating r each treated ndividual is to estimate the distribution of treat ent effects, the distribution o stimated in vidual t amentim acts, 4 approx mation of Insuch ases,onem yonlyn edto_ Icu teor plot the distribution of he estimated ea_ ent effects. Unfortunately, is ty icall relatively small in RCTs. is estimated using few obse vations,so . Therefore, o needs to correct for this es mation er r using deconvolution as When the data requirements and assumptions discussed above are satisfied, this model can be estimated using the following procedure: Set up the data so that there is one observation for each unique individualtime period combination.

Estimate regression: Estimate feature of interest Arellano and Bonhomme ( eq. (47). If interested in the full distribution of treatment effects, estimate B-@ Applications of these methods are o — using data from an actual RCT conducted in The Gambia in Section 7.2 and using simulated data in Section 7.3. Example code appears in the Distributional Impacts and Conditional Analyses Introduction Thus far, we have considered unconditional heterogeneity, i.e., how treatment impacts vary in the population overall. We may also be interested in questions such as whether and how creasing or decreasing with baseline income. These questions concern (features of) the joint distribution of treatment impacts and other variables. Thus, as discussed in Part 2, they are discuss how to study conditional treatment effect heterogeneity.

populations, such as by gender. Conditional versions of both questions concerning impacts on outcome distributions, such as whether the effect on inequality varies by gender, and
<!-- Start of picture text --> questions on the distribution of outcomes, such as whether the fraction of individuals who<br>benefit from the program varies by gender, may be of interest. Section 5.2 shows that for<br>discrete covariates, such as gender, the problem is tractable, but often requires larger sam-<br>ple sizes than those typically available. For continuous variables, such extensions would re-<br>quire estimating the joint distribution of two continuous variables, such as treatment effects<br>and baseline income — which requires even stronger assumptions than the one discussed<br>above and exceptionally large data. However, estimating features of this joint distribution,<br>such as conditional means, may still answer important questions. For example, how would<br>the average treatment effect change if the 1985 age distribution were replaced with the cur-<br>rent distribution, holding all else constant (Rothe, 2012)? Therefore, researchers often esti-<br>mate conditional average treatment effects. We discuss their estimation and interpretation<br>in Section 5.3.<br>two or more groups of interest. While straightforward in theory, this approach requires<br>common support and large sample sizes. In particular, we can implement exactly the same<br>ditional approach can be applied to a wide range of methods, including average treatment<br>effects, quantile treatment effects and distributions of treatment effects. For example, the<br>between the treatment and control marginal distributi ns at p_ ticular quantile, the sub-<br>group QTT for, say, males, is the difference in the marginal distributions of outcome<br>at a particular quantile. These subgroup distributional impacts are of interest be-<br>cause, as in the full sample, means may mask substantial within-group treatment heteroge-<br>neity. Note that if the distribution of the outcome variable is very different across groups,<br>quantile treatment effects for each g 0 ~=—may look different even if the effects are identical<br>at the same values of the outcome _r_ie. This is due to the fact that the quantiles of the<br>distribution of untreated outcomes,<br>Section 3.5 and around Figure 6 of Bi e oynes and Domina (2014).<br>In principle, the above approach f timating parameters separately by subgroup works<br>whenever the subgroup has both treatment and control observations, i.e.,<br>Note that subgroup QTTs differ from the estimates of c nditional quantile regression (CQR). CQR models<br>each quantile as a parametric function of co iates. The C R coefficient on the interaction of the subpopula-<br>tion indicator and T would indicate how the quantile of differs between the treated in the subpopulation<br>population (rather than the treated sub opulation). It assumes that treatment effects are<br>constant across subgroups. Note that condi onal quantile ffects do not average to the unconditional effect,<br>so that CQR provides little information about treatment effects for the population of participants.<br><!-- End of picture text -->
analyses are being conducted across each subgroup of interest. Thus, it may often be infeasible, particularly when subgroups are defined by several characteristics or continuous ment and one control family. Even at income levels where there are multiple observations, there generally will not be enough observations to yield a precise enough estimate of the mean, not to mention the entire conditional distribution. Adjusting for multiple hypotheses partially overcome this issue by collapsing continuous variables into a small number of catethe cost of potentially missing some heterogeneity, for example, within observations in the same income decile. These assumptions may allow learning about heterogeneity, but they can also mask it if they are not chosen well, as we discuss further in Section 5.3.

Conditional Average Treatment Effects In this section, we focus on conditional average treatment effects. In practice, most rebecause learning about how the full distribution changes with covariates is not of interest, but because the subgroup sample sizes are too small to estimate separate distributions within each subgroup. However, we may still be able to estimate the average effect of interest vary across men and women, we can simply estimate Conditional Average Treatment Effects observations in both treatment and control groups. If perfect compliance fails, dogenous and bias CATEs, just as it biases ATEs. Consequently, reweighting o IV methods are required to recover effects on the treated, as in Part 3. One can also still us =randomization instead of treatment status to obtain intent-to-treat parameters. However, for their interpretation it is important to take into account that both take-up and treatment effects may vary between groups. Subpopulation means share most of the advantages and shortcomings means across subpopulations can shed some light on heterogeneity. Djebbari and Smith (2008) call heterogeneity explained by differences in subgroup means “systematic heterogeneity” and heterogeneity remaining after controlling for these differences “idiosyncratic heterogeneity”. Subgroup means are therefore informative about systematic heterogeneity, but not idiosyncratic heterogeneity. In some cases, this can lead to incorrect conclusions about Selecting the relevant subgroups is a critical step in conditional analysis. It is recommended to define the groups of interest at the design stage and include them as part of the data collection. See Section 6.4. Despite extensive planning, however, researchers may not anticipate the most relevant subgroups.
<!-- Start of picture text --> Hoynes (2014) re-analyze a welfare experiment to investigate the extent to which allowing<br>for heterogeneity in CATEs can explain the heterogeneity in quantile treatment effects in Bit-<br>ler, Gelbach and Hoynes (2006). Using a simulation exercise, they conclude heterogeneity<br>As pointed out above, estimating how average treatment effects vary with covariates<br>non-parametrically is often infeasible when there are many subgroups or when treatment<br>effects vary with a continuous variable. These problems can be mitigated by making para-<br>metric assumptions on the treatment effect heterogeneity. We often have prior beliefs that<br>suggest a certain functional form for treatment effect heterogeneity. For example, we may<br>be willing to assume treatment effects vary linearly or quadratically with income. At the risk<br>of misspecification, these assumptions increase power and allow for identification of effects<br>by interacting covariates with trea<br>is unlikely to be exog pliance fails, so re-weighting or IV<br>methods a e necessary to obtain consistent estimates. The methods from sec ns 3.3 and<br>3.4 are di ctly a plicable, since conditional means are features of the margi al outcome<br>distributions. One may be tempted to continue to use randomization status<br>in order to estimate an intent-to-treat parameter. Howeve as pointed<br>estimate ta -up given p rticipation in the RCT and treatment effects for the treated sepa-<br>inter ctions,<br>s several parametric assumptions. Most importantly, t come<br>. When the true data generating process is nonlinear, th canb i -<br>a linea approximation of the conditional expectation f n provi ea<br>mpressi of the existence and direction of heterogeneity. Also, covariates in-<br>teraction terms, they may still struggle to detect complex relationships. A common exercise<br>we can add additional covariates to<br>Conditional average treatment effects provide a simple measure of how treatment<br>effects vary with observable characteristics. As with the average treatment effect, however,<br><!-- End of picture text -->
<!-- Start of picture text --> there is a distribution around the conditional mean. For example, even if all conditional<br>means are positive, some individuals may still be hurt by the program. This idiosyncratic het-<br>group effects to inform future targeting of a program may backfire if the impact on the mar-<br>ginal participant is very different from the average treatment effect among the current set of<br>participants, even within each subgroup. Smith (2015) provides the following example. Sup-<br>the cost of participating in the program is 5, only the “big responders” will choose to partici-<br>pate and the program will appear more effective for women than men. However, the impact<br>on the marginal male participant (4) is larger than the impact on the marginal female partic-<br>We can attempt to overcome the above issue by estimating a model of participation<br>of this participation model. The methods in this section can<br>age treatment effect at a specific value of the unobservable component of the participation<br>equation:<br>This is a conditional a erage rea e se described below, but 0 e<br>conditioning variable, , is not observed. Thus, estimation is more complex. Under the a -<br>sumptions of Heckman and Vytlacil, it can be estimated as the derivative of the outcome,<br>with respect to the ex genous variation in the propensity to participate. Brinch, Mogst<br>discuss the relation to LATE.<br>Statistical Inference and Power Calculations<br>Introduction<br>Parts 3 and 4 discuss identification and estimation of treatment effects. However, meaning-<br>ful policy evaluation must also account for the precision of point estimates. This part surveys<br>some tools for formal hypothesis tests and power calculations in DIA: Section 6.2 discusses<br>statistical inference for DIA. It focuses on repeated testing and functional hypotheses. Sec-<br>tion 6.3 surveys tests of a particular distributional hypothesis: whether individual responses<br>to treatment were heterogeneous. Section 6.4 considers the choice of sample size for RCTs.<br>It aims to help researchers design RCTs to address distributional hypotheses.<br>These three sections draw on approximations to the finite-sample behavior of estimators<br>of treatment effects. Inference and power calculations would ideally use the exact distribu-<br>tion of the estimator in question for each sample size. However, finite-sample results are on-<br><!-- End of picture text -->
Hence, we must rely on asymptotic approximations, for which there are two main strategies. Traditional asymptotic theory exploits the limiting distribution of estimators as the sample size goes to infinity, which is normal in most cases. This approach has two drawbacks. First, the resulting approximation might be poor. Second, the asymptotic distribution might be impractical. For example, the asymptotic variance of the IPW estimator of QTTs (Section 3.3) depends on a conditional expectation (Firpo, 2007), which is difficult to estimate due to the curse of dimensionality. Therefore, we recommend simulation-based inference instead. The procedure is straightforward: estimate the effect of interest on many different samples; pool the estimates across replications to construct a sample of estimators; then use the distribution of the estimator from this sample for inference. Statistical Inference When researchers conduct DIA, they often wish to address multiple hypotheses or hypotheses about multiple parameters. However, repeated testing increases the probability of false positives, distorting significance and power levels. Therefore, critical values need careful adjustment according to the question at hand. This section surveys three categories of hypotheses, which often occur in DIA: inference, although critical values need adjustment to correct significance levels. The next subsections discuss each category in turn, including examples of relevant hypotheThis subsection focuses on pointwise inference: testing a single hypothesis about a finite number of points. For instance, consider the following questions: Is the distribution of treatment effects symmetric? Is the change in the first quartile of outcomes larger than the third quartile? The first hypothesis concerns a single statistic of the distribution of treatment effects. The second also concerns a single statistic of this distribution if we reformulate it in terms of tion. Thus, they all require pointwise inference.

For instance, finite-sample theory offers exact formulas for the distribution of quantile estimators at any sample size if the data are independent and identically distributed. See Koenker (2005) and Chernozhukov, The design of the RCT should also account for multiple testing. For example, optimal sample sizes are larger. See Section 6.4. Note that no skewness is a necessary condition for symmetry, but it is not sufficient.
<!-- Start of picture text --> Following the discussion in the introduction ofthis part, we recommend basing inference<br><!-- End of picture text --> structing confidence intervals. We accep the null hypothesis if it falls entirely within the observations. To construct a confidence interval Horowitz (2001) recom ends the percentile method: observations at random with replacement from the data. You now have a Estimate hetreatme effect of interest on the bootstrap sample and sto _ it. times. You now have a sample of estimates of size mp the quanti s 0.025 and 0.975 of the sample of estimators, . ou now h e bootstrap critical values. Your confidence _ terval is: For diff ent nificance levels, adjust step 4. Alternative resampling procedures in step 1 are possible. For example, we can sample clusters instead of individual observations to acstatistics. For example, compute the standard deviation of the sample of estimators in step 4 to estimate standard rrors. How many boots ap repetitions are necessary? Computation is often trivial, so that a conservatively large Cameron and Trivedi 2005) suggest at least 348 repetitions for confidence intervals at level thousand repetitions and often as many as ten thousand.

tion between different sources of estimation noise. Consider, for example, the IPW estimator of treatment effects (see Section 3.3). We first compute weights for each observation, before taking the difference between weighted quantiles (or averages, variances, etc.). One should not overlook estimation error from the first step, which depends on the weight estimator. In the case of parametric models, it is sufficient to re-estimate the weights at each iteration in step 2 to adjust percentile intervals. Nonparametric or semiparametric estimators are the kernel bandwidth). Most data-dependent procedures pick optimal tuning parameters, in the sense that they minimize mean squared error. However, optimal convergence rates often generate asymptotic bias, which invalidates inference (whether it draws on asymptotic theorameter. Moreover, these procedures themselves introduce additional estimation noise. It is unclear, however, whether one should correct for it. In a study of kernel density estimators, for example, Hall and Kang (2001) argue against re-estimating the bandwidth at each bootThe usual caveats about cluster-robust estimation apply. We refer the reader to the excellent survey of Cameron and Miller (2015) for details, in particular Subsection VI.C.3 (“Bootstrap with caution”). For most estimators, analytical corrections for asymptotic confidence intervals are possible. See Newey and McFadden (2001) for a general discussion of inference for two-step estimators.
Note also that the bootstrap requires continuity conditions.

The estimators of changes in the outcome distribution in Part 3 generally satisfy these assumptions. The exception are extremal effects: very low and very high quantiles, for which the scarcity of data affects convergence rates. Zhang (2016) considers inference for such extremal quantile effects. Moment estimators for the distribution of treatment effects (Subsection 4.4.1) also satisfy the bootstrap regularity conditions, as long as this distribution is not degenerate. On the other hand, the validity of resampling for the deconvolution estimator of Subsections 4.4.2 and 4.4.4 is an open question, because the asymptotic properties of Mallows’ Algorithm are unknown. As the literature stands, the bootstrap provides some insight into the precision of this estiThe theory of pointwise inference focuses on point estimates. However, it is also possible to test hypotheses about partially identified parameters. For example, Section 4.2 discussed the estimation of bounds on features of the distribution of treatment effects. As Imbens and Manski (2004) and Stoye (2009) note, inference raises additional questions in this case. They the asymptotic distribution of the bound estimators. This task is often difficult because many Section 4.2. These discontinuities violate the smoothness assumptions of standard methods Makarov bounds, which is valid under weak conditions.

Policy evaluation often entails multiple hypothesis tests. For example, consider the questions: Which quantile treatment effects are positive?

Repeated testing distorts significance levels, necessitating adjustments to critical values for ty across ethnicities. They estimate average treatment effects for five ethnic groups and test each estimate against the zero null hypothesis at the 5-percent level. Thus, the probability of a false positive is five percent for each test. Across all tests, however, it is larger. If the tests Adjusting critical values requires us to extend the concept of test size (i.e., probability of making a Type | error) to multiple hypotheses. Several extensions exist and researchers should base their choice of rejection rule on substantive considerations.

It is often possible to adjust the simulation algorithm to obtain valid inference under weaker assumptions. For instance, Otsu and Rai (forthcoming) develop a valid weighted bootstrap approach for matching estimators. Different resampling methods, such as subsampling, are often valid under weaker conditions than the bootstrap, although their theoretical properties are not as advantageous. See Horowitz (2001). The nonparametric bootstrap is valid for the kernel deconvolution estimator of the distribution of treatment
considers the Familywise Error Rate (FWER). The FWER is the probability of falsely rejecting false rejections. Also popular is the False Discovery Rate (FDR): the expected proportion of false positives across all hypothesis tests. As Romano and Wolf (2010) point resampling and an iterative step-down algorithm to preserve power.

The resulting intervals We assume that the are estimators a nondegenerate di ribution, where is anonneg vese_ enc be the average treatment effect oraq__ntile effect, in which case in a series of single-step tests. At eac iteration, we remove rejec d sideration. Thes_ gle-step al orithm is: ~~7~~ ple, theses f om conEstimate bservatio at random with replacement from the data. You now have a Estimate the bootstrap sample. Use the same estimator of step 1. 6 6| tie |6 —6 | acrossb tsra_ eplic tions. For each a dboo trap c ( |~ —~ |) For each bo t trap re lication t 6 -6 ) -th qu _ntile of th quantile f |@ -6 | strap replicate . This uantile th value for the

parameters, ignore previously rejected hypotheses. Step 7 will yield a different percentile for use as a Critical value in step 8. Continue in this fashion until you stop rejecting.

This subsection discusses uniform inference: testing a single hypothesis about a continuous object, such as a function. Uniform inference allows us to address such questions as: Are all quantile treatment effects positive?
<!-- Start of picture text --> Is the distribution of treatment effects normal?<br><!-- End of picture text --> The distinction between uniform inference and corrections for multiple testing can be subsuch as the quantile process or the distribution function. Testing whether there was any effect on the outcome distribution pertains to uniform inference, because we would test the equality of two distribution functions. On the other hand, testing which quantile effects are significant is a problem of multiple testing, because we would perform a sequence of pointwise significance tests for each estimate. Statisticians and econometricians have developed various tests of popular distributional hypotheses, which are often based on scalar statistics. For example, consider the hypothesis of positive quantile effects. It is equivalent to first-order stochastic dominance, for which we have the Kolmogorov-Smirnov statistic (the largest absolute difference between the outcome distributions of treated and untreated participants). Chernozhukov, Fernandez-Val and width depends on the critical values of the Kolmogorov-Smirnov statistic. dence bands cover the entire function of interest with the correct probability, in the same way that confidence intervals cover point estimates. They have three advantages over scalar test statistics: they are easy to display on a graph; they allow researchers to test hypotheses for which no known statistics are available; and they allow readers to test hypotheses which dicate heterogeneity in individual responses. Such heterogeneity across subgroups is often interesting in itself. Tests are standard, but researchers should adjust critical values for MHT, which often takes a heavy toll on power. Crump et al. (2008) generalize this idea to accommodate discrete and continuous covariates in both parametric and non-parametric estimators of conditional means. They propose a scalar statistic, thereby avoiding MHT.

use the fact that the distributions of st with the bootstrap algorit m of S bsection 6. test is negative: , so the negative covariance would ffset the vari r ent . The analytical example in Se tion 3. illustrates thi t may the presence of heteroskedasticity, i.e., if the variances of the error terms differ between treatment and control. Heteroskedasticity may be a concern if randomFor some estimators, the literature provides asymptotic theory to construct confidence bands as well. For quantile treatment effects, see Chernozhukov, Fernandez-Val and Melly (2013) and Koenker and Xiao (2002).
ization was compromised. Conducting the test on residuals may address this concern and increase power, but it may also lead to false rejections if the model is misspecified. The idea in the previous paragraph generalizes to higher moments. Joint tests of multiple moments may achieve greater power. One can also use a test of equality of distributions, such as the Kolmogorov-Smirnov test, to assess whether the distributions of deviations from worried that the distributions differ for other reasons if randomization was compromised. from quantile effects or bounds. If treatment effects were constant, all quantile effects would be equal to the average treatment effect. Implementing this approach requires either corrections for multiple testing (Subsection 6.2.2) or uniform inference (Subsection 6.2.3). One may also be able to reject the null if the variance bounds do not include zero, as Section Power Calculations This section considers the choice of sample size for DIA in RCTs. We first adapt the analytic framework of List, Sadoff and Wagner (2011) to DIA. The complexities of DIA often render an analytic approach undesirable or unfeasible, so we close with some guidance on power calculations via simulation.

An important choice in experimental design is the sample size. Due to cost considerations, the po er ofa particular pointwise t st against the zero null hypothesis (ZNH). deter ne the opti mple size, _, and the optimal probability of assignment to treat. These par meters reflect the experimenter’s tolerance of statistiin s both treated and untreated participants. To derive the power of a test, eneedtea otic distribution of the estimator of interest. For concreteness, we focus on quantie treatment effects under the independence assumption of Section 3.2. Similar to ATTs, QTTs are asymptotically normal. Therefore, we only need to specify their asymptotic variance. Power calculations for Q Ts and ATTs only differ in the exact form of this We proceed in four steps. For a quantile Note that this test involves estimated parameters, which the bootstrap should account for. See Subsection 6.2.1. null hypotheses are possible with the necessary modifications. As a reminder, the significance level is the probability of a type | error in a two-sided test: rejecting the null hypothesis when it is true (a false positive). The power is the probability of rejecting the null hypothesis when the alternative is true. We assume that the data are IID. For a discussion of clustered data and stratified designs, see Section 4 of
Specify the quantile that this quantile ist e e undert Specify the quantile the alternative hypot e . Note that of potential outcomes Iter ov e of o n oe <!-- Start of picture text --> n tive hypothes s, the c variance of the estimator q<br>The optimal randomization r t<br>are the quan | e and fe) I dst ution. Note that sample<br>sizes a rger f quantiles in low-density regions, such as the tails of the outcome dist -<br>buti n, because e scarcity of observations reduces accuracy.<br>T ese same steps yield optimal sample sizes for other quantities. The formulas for<br>in step 4 go through as long as the estimator of interest is asymptotically no al<br>Adju t steps 1 and 2 according to the object of interest and its a y ariance<br><!-- End of picture text -->

potential outcomes instead of the quantile and the density, since The procedure above considers a particular quantile of inte s t applications, however, researchers compute quantile effects on a grid. Three di arise. The first consists of formulating distinct hypothesis tests for each quantile. It is often simpler o ms of the entire distribution. For example, a location model assumes that the _ is o h_byaconstant. Then all quan | _ ffects are equal to that constant and

and the randomizati n rate differ for each quantile. thou h_ = atthe formulas abov areinv li or extremal effects. It is advisable to avoid estimation of extremal effects — be ond the 5th and 95th percentiles, say. Lastly, grid estimation implies multiple hypothesis testing, invalidating the formulas above. There are two solutions to this problem. It is possible to adjust critical values for repeated testing (cf. Section 6.2). However, this approach involves complex algorithms and many unknown quantities, which limits its usefulness. Simulation offers a more feasible route, which we survey in the next It is also possible to use finite-sample distributions instead of the asymptotic normal approximation when they are available. See o 05) an Chernozhukov, Hansen and Jansson (2009) for quantile treatment For a location model,
<!-- Start of picture text --> subsection. The closed-form formulas provide a first guess of sample size to start the simu-<br>lation algorithm.<br>domization rate should also take MHT into account. Otherwise, significance levels will be<br>higher and/or test power will be lower than planned. Note that the number of observations<br>might dramatically increase with the number of tests. Splitting samples might help keep the<br>sample, we estimate the resulting model and test the significance of each variable, limiting<br>Simulation is an alternative strategy to determine the required sample size. Simulation for<br>DIA presents no particular challenge, so we only sketch a brief overview here. See<br>Similar to the previous subsection, the selection criteria are the significance and power<br>of a hypothesis test against the ZNH. Instead of computing these error probabilities with as-<br>ymptotic methods, however, we use simulation. To be precise, we simulate pseudo-samples<br>from the model of interest under the null and the alternative hypothesis. For each pseudo-<br>tions until empirical rejection rates satisfy the desired level of accuracy.<br>Simulation offers a number of advantages over asymptotic theory. Firstly, it accommo-<br>dates estimators with unknown or excessively complex asymptotic distribution (e.g., the IPW<br>estimator of QTTs), as well as difficult inference problems (e.g., uniform inference or multi-<br>ple hypotheses). Secondly, it reflects the finite-sample behavior of estimators which might<br>converge slowly to their limiting distribution. Thirdly, it is straightforward to incorporate<br>To conduct a simulation exercise, it is necessary to choose the number of pseudo-<br>samples. Unless the computational burden is high, it should be large: five thousand or ten<br>thousand are good figures. One also needs to specify the data generating process. Choosing<br>distributions of covariates and the error term is often challenging: they should reproduce<br>the conditions of the RCT, including dependence patterns in the error term (such as clusters<br>or serial correlation in panel data). Previous studies might provide some guidance. If an ex-<br>isting dataset contains the covariates of interest (or some of them), it is possible to draw<br>pseudo-samples from it to minimize distributional assumptions. Sensitivity analyses are al-<br>Applications<br>In this part, we present applications of the methods discussed above using data from two<br>RCTs. Section 7.1 revisits the study of financial education by Bruhn et al. (2016). In Section<br><!-- End of picture text -->
how to choose appropriate methods and how to assess their assumptions and address common concerns; and (3) to illustrate briefly the implementation of the methods and their Financial Education RCT in Brazil This section revisits the financial education program evaluated by Bruhn et al. (2016). Comaverage financial proficiency by a quarter of a standard deviation, which is large in compariHowever, policymakers may wish for more information than average effects. For instance, they may take particular interest in low values of financial proficiency, which might put consumers at risk of pyramid schemes and other predatory tactics. They may want to subgroups, which could help them target the intervention and understand why it works (or additional financial education for children from poor households. To explore these questions, we estimate several DIA parameters, mostly related to changes in the outcome distribution (Part 3) and conditional analysis (Part 5).

anced across treatment and control groups at each round of data collection. multiple-choice test, from which Bruhn et al. (2016) construct an index of financial 56.05 and the standard deviation is 14.81. Did the intervention change the frequency of very low test scores? Did it change dispersion in financial proficiency? Did it reshape the proficiency distribution? We focus on DIA and DlA-specific issues. The original papers discuss solutions to other common problems that affect both DIA and the estimation of average treatment effects. From an original list of 910 schools, nineteen did not participate for unknown reasons. As an exception, the authors find a small difference in the gender ratio, which is significant at the 10% level (these tests are unadjusted for multiple hypothesis testing).
<!-- Start of picture text --> 1 er i<br>§ 09 wee -<br>5 08 — ¢<br>= 07 wy?<br>S— o6| “oFwoe<br>3 “ot<br>= 0.5 Par,<br>% 0.4 o ,<br>£03 o o<br>2 02 Pa<br>5 wn o ? seeeeeees Control<br>o 0.1 oe<br>0 we” — — -Treatment<br>20 30 40 50 60 70 80 90 100<br>Financial proficiency score<br><!-- End of picture text -->

To answer these questions, we examine treatment effects on a broad set of parameters: the mean, the standard deviation and the ratio of the 75th to the 25th percentile; every percentile between the 5th and the 95th; and the CDF for every integer score between 33 and and treatment status (cf. Section 3.2). The intervention began halfway into the school year, so students could not change schools or classes before the first follow-up, limiting noncomplots the outcome distribution for each group. Table show estilevel using the step-down bootstrap algorithm of Romano and Wolf (2010), see Subsection 6.2.2. Thus, the probability of at least one false rejection across all significance tests is asymptotically smaller than five percent.

line with Bruhn et al. (2016). There was also a reduction in the proportion of very low scores. For example, the share of scores below 40 points is lower by 0.08, a 45.8% reduction from standard deviation is smaller by 0.46 points, a 3.12% reduction of the 14.81 points in the control group, and the ratio of the 75/25 quartiles is reduced by 7.5% from 1.53 in the control group. All of these effects are statistically significant. In summary, the intervention not Quantile effects provide further insight into these results. As Figure 6 shows, the intervention shifted the entire CDF to the right. Recall that quantile effects are the horizontal distance between these curves; hence, they are all positive (and statistically significant). Moreover, the change in lower quantiles is relatively larger. The first decile increased by 4.2 points or 11.5%. The first quartile is greater by 6.2 points (14.1%), whereas the difference in the ninth decile is 3.2 points (4.27%). A Kolmogorov-Smirnov test rejects equality of quantile The 5th percentile of control-group outcomes is 32.95. The 95th is 80.46. Figure 12 in Appendix 4 shows changes in the distribution function (the vertical distance between the CDFs in Figure 5). These effects mirror quantile effects.
Statistic. Effect estimate Standard deviation 75/25 perc. ratio 10th percentile 25th percentile 50th percentile 75th percentile 90th percentile tions) and clustered at the school level. The confidence region controls the FWER (probability of at least one false rejection across tests), following Romano and Wolf (2010). effects at the 5% level. This pattern reveals that the reduction in outcome inequality occurred despite increases of the upper percentiles (which are desirable in an educational program). It was a consequence of proportionally larger gains in the left tail. How would we estimate these treatment effects under the assumption of selection on observables? For illustration purposes, suppose that a small imbalance in the gender ratio and baseline scores generated suspicion of selection bias. Following Section 3.3, we can use inverse probability weighting to rebalance groups. We compute the weights with a logistic regression of treatment status on a quadratic polynomial of baseline scores, an indicator for Although the rebalanced estimates are smaller than their unweighted counterparts, the differences are not statistically significant, which suggests that endogenous selection did not 28 —alt i) a7 2 S 6 25 4 @ 23 = @<sup>2</sup> i = 4 a 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 Percentile of financial financial proficiency score <!-- Start of picture text --> 28 —alt<br>i)<br>a7<br>2<br>S 6<br>25<br>4<br>@<br>23<br>=<br>@ 2<br>i<br>= 4<br>a<br>0<br>5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95<br>Percentile of financial financial proficiency score<br><!-- End of picture text -->
Quantile an lysis helps us address a broad range of policy questions. N netheless, the possibility of m_ bility effects constrains our interpretation of quantile effect , in that the change in quantile may differ from the individual treatment effect at quantile It is tempti g to believe that low-scoring students benefited more tha average, since the change in ottom percentiles is relatively larger. As Part 2 and Section .5 argue, however, this reasoning hinges on an implicit assumption of rank invariance: the same students are in the bottom and the top of the distribution in both treated and untreated states. Despite this limitation, our estimates in Table 3 and Figure 6 have implications for indineous. Moreover, positive average and quantile effects imply that at least some participants benefited from the intervention. However, the methods of Part 3 do not allow us to quantify these qualitative results. Estimating features of the distribution of individual effects requires strong assumptions (cf. Section 4.3), which are hard to justify in this application. A native, Section 4.2 shows that we can construct bounds under minimal conditions a t exercise, we estimate the Makarov bounds on the share of positive effects, (cf. Section 4.3), which are hard to justify in this application. A native, Section 4.2 shows that we can construct bounds under minimal conditions a t exercise, we estimate the Makarov bounds on the share of positive effects, tributions. We find that the probability of positive individual tre e ffects lies between These bounds confirm that at least some participants benefited from the intervention; moreover, the data are compatible with no one being hurt, because the bounds include 100% positive effects. Next, we bound the standard deviation of individual effects. Assuming that the correlation between potential outcomes is positive, we obtain that it lies between 0.46 and 20.62 points. The lower limit is positive, suggesting that the dispersion in treatment effects is non-negligible. To summarize, these assumptions on the data.

The previous subsections showed that students’ responses to the financial education program were heterogeneous. This section investigates the relation between treatment impacts and pupils’ background characteristics. For that purpose, we compute conditional average effects (CATEs) and conditional QTTs for different subgroups of participants, following Section 5.2.

Bruhn et al. (2016) compute average treatment effects by linear regression. They include a quadratic polyresulting estimate is not statistically different from their baseline regression, which does not include controls. The 95% confidence interval is [0.07, 20.89] (Imbens and Manski, 2004). Under no assumptions on the correlation between potential outcomes, the upper bound is 29.15 and the confidence region is [0.07, 29.54]. We compute treatment effects as the difference in the relevant statistic between the treatment and control subsamples of each subgroup (cf. Section 5.2), as in Subsection 7.1.2.
~~—~~ <u>Effectestimate</u> The first four specifications include 848 schools. The last two include 851 schools. Standard errors and confidence region based on the bootstrap (five thousand replications) and clustered at the school level. The confidence region controls the FWER (probability of at least one false rejection across tests), followcy score (above or below median); (2) grade repetition; (3) gender; (4) family welfare status in Bruhn et al. (2016) reports summary statistics and balance tests by variable. displays our estimates of CATEs. Table -statistics for the difference in CATEs. We perform six equality tests in total; hence, we must adjust for multiple testing (MHT). In Subsection 7.1.2, we controlled the FWER (the probability of at least one false rejection). Although this error rate is the most stringent, all effects were significant at the fivepercent level. Evidence of conditional heterogeneity is less robust. Therefore, Table critical values for the 2-FWER (the probability of at least two false rejections) as well. We use the step-down bootstrap algorithm of Romano and Wolf (2010). (See Subsection 6.2.2.) of a third of our sample participate in Bolsa Familia, a welfare program for low-income households. On average, these students gained significantly less from the intervention (3.39 two false rejections in six tests. Other criteria do not yield significant effects, whether we A number of participants declined to answer the relevant questions in the surveys, depressing response rates to 64%. For rigorous policy evaluation, it would be necessary to assess robustness to missing observations.
Bold values indicate rejection of the null hypothesis. The first four specifications include 848 schools. The last two include 851 schools. Critical values based on bootstrapping -statistics (five thousand replications) and clustered at the school level. Columns 4, 5, 7 and 8 control the Figure 7 plots estimates of five conditional quantile effects for each subgroup. are similar to Figure 6, with effects peaking around the first quartile. The only clear exception is students whose baseline score was above the median, for which quantile effects are approaches to this question. On the one hand, we could test whether the difference between each quantile effect in each pair of subgroups is significant. However, this approach would require thirty hypothesis tests; adjusting for MHT would take a heavy toll on power. On the other hand, we could test whether all quantile effects are equal in each pair of subgroups. To do so, we could test the maximum absolute difference in quantile effects across quantiles, which only requires six tests. This second procedure reveals that quantile effects Note that these estimates are not informative about causal effects of the characteristics provide additional insights as we break down cells into smaller groups. However, the number of observations per cell would decrease just as the number of hypothesis tests increased, to the detriment of statistical power.

The quantiles are: the 10th, the 25th, the 50th, the 75th and the 90th. Table 10 in Appendix 3 reports point estimates and standard errors. This procedure tests for significant differences in at least one quantile effect for each pair of subgroups. It cannot detect all quantile effects for which there are significant differences.
<!-- Start of picture text --> Baseline score above median Student works<br>10 7<br>seefees No otem-+ No<br>vo 38 SN =-@--Yes ° weeta te a oennves<br>5s 6 *S, 4 a arn Tm n0<br>ws 4 vee msn, aacry ore<br>5 3 a te------ 4 ------# 2<br>5S 2<br>1<br>1<br>00<br>10 25 50 75 30 10 25 50 75 30<br>Has repeated the school grade Student earns income<br>7 8<br>a, seeees No stems No<br>=oo 65 awe* cee,iiawe eewee —@=-Yes 76 oeeSaati,en, ss tes<br>2 = 4 ae Bre ee 5 a we Tree oe.<br>7S e thee,hentems 4 ¢paert tee,Hemeekeel,<br>5 25 > ar SEeeeeeeTeeere<br>te} ie}<br>10 25 50 75 30 10 25 50 75 30<br>Is female Family is on welfare (Bolsa Familia)<br>7 8<br>wr sees No a seefees No<br>6 a aan ae == Yes ? a oo wot-Yes<br>g 2 ° or SA, el 5 a“ ores Mee<br>o 3 4 « sel “tee 4 went ssn re<br>Eg3 wee _---@ id a eaaanee<br>% § ~e--- 3 mae<br>5 25 3 O------W6<br>.o)<br>11<br>00<br>10 25 50 75 30 10 25 50 75 30<br>Percentile Percentile<br><!-- End of picture text --> highlights the importance of correcting for multiple tests (even at large sample sizes). Controlling the FWER inflates critical values by 30 to compared to three if we do not adjust. The choice of error rate is also consequential.

example, control of the 2-FWER leads to smaller critical values at the 95-percent level and probability of two or more false rejections across six pointwise tests at the five-percent level is 3.28%, which is lower than the nominal five-percent level. Therefore, critical values adjust downwards. At the 10-percent level, it is 11.43%. Hence, critical values adjust upwards.

This dataset includes 18,276 observations, which allowed us to obtain precise estimates of a rich parameter set. However, DIA is feasible with much smaller samples. This subsection
hypothesis detects a given treatment effect with size 5% and power 80%. See Subsection 6.3.1.

We follow the approach of Subsection 6.4.1. We compute the optimal sample size to de80%. We must also parametrize the asymptotic variance of our estimator of interest. For quantile effects, it depends on the density of outcomes, which we compute from the control In an application, researchers might obtain the density function based on a baseline sample, previous studies and parametric assumptions.

Small effects require large samples: detecting an increase of 1% in the 10th percentile requires a total of 67,920 observations. However, moderate samples suffice for modest effects.

Table 6 does not adjust sample sizes for multiple hypothesis tests or clustering, which would raise data requirements. The formula for the optimal sample size in Subsection 6.4.1 depends on the square of the critical values. In Subsection 7.1.2, we controlled the FWER across 143 tests, inflating cutoffs by up to 50%. Such a correction would increase the recommended sample size by a factor of 2.25. Estimating fewer parameters would reduce this parameters). Then, control of the FWER increases critical values by 30%, which implies multiplying the sample size by a factor of 1.7. As for clustering, the optimal sample size is proportional to the variance of the estimator of interest. In our application, accounting for clusCombining these rules of thumb, our estimates of each quantile effect (see Table 3 and Figure 6) and the formula for the optimal sample size (see Subsection 6.4.1), we find that the optimal sample sizes for this RCT would be 12,000 observations if we control the FWER across all 143 parameters and 9,000 observations if we only control the FWER across the mean and the deciles. Note that adjustments for multiple testing and clustering are unknown at the design stage. Simulations may help researchers incorporate these features into their choice of sample size. See Subsection 6.4.2.
<!-- Start of picture text --> average treatment effects. First, it is difficult to summarize the impact of an intervention<br>with any single statistic. In particular, average effects often hide significant heterogeneity.<br>Second, we gain important insights from the changes in the shape of the distribution. Quan-<br>tile analysis and other DIA methods allow us to address a broader range of policy concerns<br>than means alone under minimal assumptions. Third, conditional analysis is a complement<br>and not a substitute for unconditional approaches. Each method reveals a different dimen-<br>correcting for multiple testing.<br>In this section, we use the methods in this toolkit, focusing on those in Part 4, to re-analyze<br>500 grant to help cover costs associated with implementing new initia-<br>tives based on the training. To disentangle the impact of the WSDP training from the impact<br>without any additional training. In the third arm, 89 schools served as the control group and<br>test scores is approximately zero and statistically insignificant when looking at impacts sepa-<br>pact or with a program having heterogeneous impacts, benefitting some schools and hurting<br>others. In order to distinguish between these two cases we use the methods discussed in<br>Part 4 to estimate the variance of treatment effects and, relying on strong assumptions, the<br>entire distribution of treatment effects. The results of these exercises are important for poli-<br>cy: If there is treatment effect heterogeneity the program could become effective if better<br>targeted at those with positive treatment effects. This is obviously not possible if the treat-<br>The WSDP is a good application to demonstrate the methods discussed in Part 4 because<br>high quality panel data were collected for the evaluation. However, we do not find evidence<br>taset, based on the features of the data collected for this RCT, using the Simulation 1 data<br>generating process described in Appendix 1.<br><!-- End of picture text -->
Evans and Lahire (2016) demonstrate that schools’ baseline characteristics are balanced grade-year and restricting attention to WSDP and control group observations where 2 grades tions with 8 observations for each of 120 schools, including 61 schools in the WSDP treat(2016), who use student level data clustered by school-year for their cross-sectional analysis. We instead focus on a balanced panel of school level observations. As Table 7 shows, this the implementation of Mallows’ deconvolution algorithm, which we use in Subsection 7.2.2.

a similar small and statistically insignificant effect, using pooled OLS on the stacked crosssections. But does this null average effect mask treatment effect heterogeneity? Of course, because they think they will benefit participants, programs may have unintended negative consequences if, for example, they disrupt effective systems. If this is the case here, what proportion of schools benefitted from the WSDP? In this subsection, we answer these questions by studying features of the distribution of the WSDP’s effects beyond the mean. Unlike average effects and quantile treatment effects, features of the distribution of impacts beyond the mean are not identified by random variation induced by an RCT alone; additional assumptions are always required. The WSDP RCT is a good candidate for satisfying these assumptions because it includes four waves of data, and treated schools are observed both before and after the implementation of the WSDP.

tion 4.4.4. Conditional on school fixed effects, the variance of treatment effects is identified so long as the treatment effect is uncorrelated with all time-varying components of the error term, and these time varying components are not themselves too correlated over time. The full distribution of treatment effects is identified under the stronger assumption that treatment effects are independent of all time varying components of the error term. While these are not directly testable assumptions, they may be plausible. They would be violated, for ex-
ample, if the school management training is particularly effective in areas with the highest fication from Subsection 4.4.

ccountfo t ey he WSDP’s impact: Note two features of this pec tion: First, r r ool fixed effects, is only identified fort subp u_ tion of schools in the WSDP treatment rm, since iden fication relies on variati in . Control group schools ontribute toe mating the coefficients The first row of T ble 7 shows that the estimate of the average im act ac_ ss schools, 0.01, is quite similart th estimate assuming treatment effects are constant across schools. This estimate is just (A |r = 1). Standard errors are given by conventional formulas for method of moments _ timators. The second rows’ ws the estimate of the variance of treatment effects using the formula given in Arellano and Bonhomme’s (2012) equation (50). The estimate, half the size of the mean impact and is also statistically insignificant. The validity of this estimate relies on two assumptions. First, treatment effects are conditionally mean independ-

E(A |WSDP ) = E(A |WSDP ).

. The notation indicates standard errors may e dividual cova t ey component his latter assumption is that a sch ser are not correlate a_ ss periods. This can As discussed in Part 4, we can identify the entire distribution of treatment effects if we are willing to make even stronger assumptions. In many applications, these assumptions are unjustified. We include this analysis here primarily for the purpose of illustration. mean and variance estimates are sufficient statistics for the distribution. This distribution is shown with the dotted line in Figure 8. The distribution is tightly concentrated around zero. To be sure, we cannot rule out that the distribution is degenerate at zero, since neither the mean or variance is statistically significant.

Alternatively, we can recover the entire distribution using deconvolution if we are willing <!-- Start of picture text --> Estimate<br><!-- End of picture text --> N = 960 with 8 ervations on each of 120 schools. Mean and variance of impacts estimates ance estimators. See Section 4.4 for details.
<!-- Start of picture text --> servations. This cond ed in our sample; the variance of these partial residuals is<br>0.133 for treatment b rva = sand0.114 for control observations.<br>o<br>wT<br>><br>2<br>a<br>oOPan" _—<br>2 4 0_ 1 2<br>Delta-hat<br>——— Deconvolution ----- Estimates<br>cooee Normal<br><!-- End of picture text --> The estimates of | are very noisy because they are estimated using only 3 treatment observations and on_ control observation per treated school. Deconvoluti n attempts to disentangle the varia on due to the treatment effects from the variation ttributable to noise. The dashed line in Figure 8 shows the distribution of the unadjusted — estimates. In contrast, the solid line shows the distribution of treatment effects recovere from applying deconvolution via Mallows’ algorithm. As expected, the deconvolution estim e is much less dispersed than the unadjusted distribution. However, it is more dispersed and has heavier tails than the normal approximation to the distribution of treatment effects.

<!-- Start of picture text --> 2 Deconvolution Estimates<br>——— 95%Cl for QTT es<br>~|.aT 4So<br>/<br>“<br>2Nend<br>~ —_eso<br>oO<br>ae we<br>N‘ _ aei ee ae<br>0 2 < Decile 6 8 10<br><!-- End of picture text -->

With knowledge of the full distribution of treatment effects, we can calculate any feature of the distribution. For example, we can calculate any quantile of the treatment effect distribution or any moment of the distribution, like the mean, variance, skewness or kurtosis of the distribution. In Figure 9, we present deciles of the deconvolution estimates of the distri-
bution of treatment effects in gray. For comparison, we show quantile treatment effects at each decile in black. We calculate these quantile treatment effects using the empirical deciles of the partial residuals of a regression of average test scores in 2011 on grade level. While the distribution of treatment effects is calculated using the full panel, our panel functional form implies that treatment effects are the same in every year and should therefore be comparable to the quantile treatment effects calculated using only 2011 data. Pointwise confidence intervals for the QTTs are calculated using one thousand iterations of the pointwise bootstrap described in Subsection 6.2.1. In this case, deciles of the distribution of treatment impacts look quite similar to the quantile treatment effects at each decile. The deciles of the distribution of treatment impacts are increasing by construction, but the QTTs trend upwards at almost exactly the same timates suggests the WSDP did not cause much mobility across the potential outcome distributions. In other words, schools that ranked high in the control group distribution also ranked high in treatment group distribution. While none of the decile QTTs are significantly The analyses in this section so far amend Blimpo, Evans and Lahire’s (2016) finding of a zero than offsetting gains and losses. This demonstrates that DIA can provide further guidance for illustration of what else one can learn from DIA and such panel data and how failures of the assumptions can affect the results. To improve the illustration, we replicate our distributional cally, we draw a simulated panel of 8 observations for each of 120 “schools” from the following data generating process: With the following distrib tio al m os: ~N(O,1),( )=n([ | ~~_~~

|

<mark>~</mark> term to ex ore the i of violating the deconvoluti sumptions on the estimate particular, presents estimates o t e average and variance of treatment effe d Makarov Bounds on the proportion of sch_ Is who benefit from the program when is O so that the deconvolution assumptio are satisfied.

estimated is due to changing
Estimate N = 960 with servations on each of 120 schools. Based on simulated data with identical structure to WSDP sample. Average and variance of impacts estimates and standard errors calculated using Arellano and Bonhomme's (2012) mean group and robust variance estimators. See Section 4.4.4 for additional details. In this case, the estimated mean and variance of treatment effects are both statistically sigproportion of schools which were hurt range from 0.49 to 1, which is consistent with roughly half of schools being hurt/bene t by the program to everyone benefitting, but include the is 0.1 so that the deconvolution assumptions are violated, <!-- Start of picture text --> st<br>° j\\<br>} J‘<br>> f \<br>aa \<br>3 . \<br>\<br>\<br>— / \:<br>v/, \<br>\<br>=<br>o<br>-4 2 0 2 4 6<br>Treatment Effect<br>——-_ True Distribution p=0<br>——-— p01 p=0.8<br><!-- End of picture text --> but the correlation between the treatment effect and the omitted variable is relatively weak. Since the omitted variable is mean zero, the estimated average effect remains unbiased. observation is from an “older” grade (which are randomly selected since this coefficient does not enter the
<!-- Start of picture text --> °% 2 4 6 a 10<br>Decile<br><!-- End of picture text --> <!-- Start of picture text --> ° % 2 4 6 8 10<br>Decile<br><!-- End of picture text --> <!-- Start of picture text --> p=0.8<br>0 2 4 6 8 10<br>Decile<br>[—e— TrueDecies © — Deciles of F, |<br>QTT at Deciles<br><!-- End of picture text -->

Hower, the estimated variance of treatment effects increased slightly and as a result is this ca e, the estimated variance increases further and is significantly different from the true wider, but are qualitatively quite similar.

Figure 11 plots the true distribution of treatment effect and the deconvolution esti= 0 or 0.1, the estimated distribution is slig tly more dispersed but generally similar to the true distribution of treatment = 0.8, the estimated distribution is even more ver dispersed. The three gures in F gure 11 plot the deciles of the true distribution of treatment effects, the de onvolution esti ate of the distribution of treatment effects, and estimated , respectively. Similarly to Figure 10, the estimates are quite similar when . In both cases, the estimated deciles of the distribution of treatment effe tsa quite similar to the deciles from the true distribution. The correlation between the es ma ed and true deciles is about 0.92 in both cases. The deciles so that the deconvolution assumptions are violated, the deciles of the estimated di ribution of treatment effects are still quite similar to the true iles as the deconvolution estimates, whereas the QTTs were less correlated with the true
<!-- Start of picture text --> By going beyond the mean and estimating the distribution of treatment effects we were able<br>fact, the estimated variance implies that the standard deviation is about 70% of the estimat-<br>ed average effect and is also statistically insignificant. This finding of little treatment hetero-<br>geneity is corroborated by both the deconvolution estimate of the distribution and estimates<br>While relevant for policy, such a null result falls short of illustrating what we can learn<br>feature of the particular program being studied rather than a deficiency of the distributional<br>methods. Our estimates of both the variance and distribution of treatment effects are sta-<br>tistically indistinguishable from the truth even when the strong assumptions are violat-<br>ed. However, it is not clear how sensitive the methods are to violations of their assumptions<br>in general, and we do not mean to suggest that the robustness of the estimates in our appli-<br>cation generalizes.<br>(2005): “Estimating treatment effects for discrete outcomes when re-<br>sponses to treatment vary: An application to Norwegian vocational rehabilitation programs”,<br>(2002): “Bootstrap tests for distributional treatment effects in instrumental variable models”,<br>(2002): “Instrumental variables estimation of quantile treatment effects”,<br>“Econometric evaluation of social programs, part III: Distributional treatment<br>effects, dynamic treatment effects, dynamic discrete choice, and general equilibrium policy evaluation”,<br>“A three-step method for choosing the number of bootstrap repeti-<br>tions”,<br>(1994): “Identification and estimation of local average treatment effects”,<br>(2011): “Does affirmative action lead to mismatch? A new<br>(2012): “Identifying distributional characteristics in random coefficients panel<br>(2002): “Recursive partitioning for heterogeneous causal effects”, arXiv:1504.01132.<br>(2014): “Inference on treatment effects after selection among high-<br>(2006): “What mean impacts miss: Distributional effects of welfare re-<br>— (2014): “Can variation in subgroups’ average treatment effects explain treatment effect heterogeneity?<br>(2014): “Experimental evidence on distributional effects of Head Start”,<br>"Non-parametric confidence bands in deconvolution<br>density estimation”,<br><!-- End of picture text -->
<!-- Start of picture text --> tive than the services themselves? Experimental evidence from the UI system”,<br>(2015): “Parental human capital and effective school management: Evi-<br>(2010): “Generalized nonparameteric deconvolution with an application to earnings<br>cation: Evidence from a large-scale evaluation in Brazil”,<br>“A practitioner’s guide to cluster-robust inference”,<br>(2002): “Removing the veil of ignorance in assessing the distributional<br>— (2003): “Estimating distributions of treatment effects with an application to the returns to schooling and<br>, M. (2010): “Efficient semiparametric estimation of multi-valued treatment effects under ignorabil-<br>(2010): “Quantile and probability curves without crossing”,<br>(2013): “Inference on counterfactual distributions”,<br>(2004): “The impact of 401(k) participation on the wealth distribution: An in-<br>strumental quantile regression analysis”,<br>(2005): “An IV model of quantile treatment effects”,<br>(2013): “Quantile models with endogeneity”,<br>(2009): “Finite sample inference in econometric models via quantile<br>restrictions”,<br>(2016): “From LATE to MTE: Alternative methods for the<br>evaluation of policy interventions”,<br>— (2009): “Dealing with limited overlap in estimation of average treatment effects”,<br>(2010): “Estimating the technology of cognitive and non-cognitive skill<br>formation”,<br>(1996): “Labor market institutions and the distribution of wages, 1973-<br>(2014): “Estimation and inference for distribution functions and quantile functions in<br>(2014): “Testing the unconfoundedness assumption via inverse probability<br>weighted estimators of (L)ATT”,<br>(2010): “Sharp bounds on the distribution of treatment effects and their statistical infer-<br>(2017). “Optimal inference after model selection”, arXiv:1410.2597v4.<br>(2007): “Efficient semiparametric estimation of quantile treatment effects”,<br><!-- End of picture text -->
<!-- Start of picture text --> (2009): “Unconditional quantile regressions”,<br>(2015): “Identification and estimation of distributional impacts of interventions using<br>(2008). “Bounds on functionals of the distribution of treatment effects”,<br>(2007): “Propensity score matching without conditional independence assumption—with an<br>application to the gender gap in the United Kingdom”,<br>(2013a): “Identification of treatment effects on the treated with one-sided non-<br>—_ (2013b): “Unconditional quantile treatment effects under endogeneity”,<br>(1997): “Making the most out of programme evaluations and social ex-<br>periments: Accounting for heterogeneity in programme impacts”,<br>(1999): “Local instrumental variables and latent variable models for identifying<br>(2005): “Structural equations, treatment effects, and econometric policy evaluation”,<br>(2007): “Econometric evaluation of social programs, part Il: Using the marginal treatment effect to or-<br>ganize alternative econometric estimators to evaluate social programs, and to forecast their effects in new<br>(2003): “Efficient estimation of average treatment effects using the esti-<br>(2013): “Estimating treatment effect heterogeneity in randomized program evalua-<br>tion”,<br>“Matching methods in practice: Three Examples”,<br>“Confidence intervals for partially identified parameters”,<br>(1997): “Estimating outcome distributions for compliers in instrumental variables<br>(2009): “Recent developments in the econometrics of program evaluation”,<br>, E. (1957): “Information theory and statistical mechanics”,<br>(2015): “Evaluating public programs with close substitutes: The case of Head Start”,<br>(1978): “Regression quantiles”,<br>(2002): “Inference on the quantile regression process”,<br>, |. (1967): “On characterizing the gamma and the normal distribution”,<br>, M. (1999): “Earnings and employment effects of continuous off-the-job training in East Germany after<br>unification”,<br>(2014): “Multiple testing and heterogeneous treatment effects: Re-evaluating the effect<br><!-- End of picture text -->
<!-- Start of picture text --> (2016): “Multiple hypothesis testing in experimental economics”,<br>(1982): “Estimates for the distribution function of a sum of two random variables when the mar-<br>ginal distributions are fixed”,<br>(2007): “Deconvolution by simulation”, in R. Liu, W. Strawderman and C.H. Zhang (eds.),<br>(2004): “Statistical treatment rules for heterogeneous populations”,<br>M. (2015): “Going beyond simple sample size calculations: A practitioner’s<br>1994): “Large Sample Estimation and Hypothesis Testing.” In<br>(2014): “Generalizing from unrepresentative experiments: a stratified<br>(forthcoming): “Bootstrap inference of matching estimators for average treatment effects”,<br>C. (2012): “Partial distributional policy effects”,<br>S. (2013): “Convolution without independence”,<br>J. (2015): “The important role of heterogeneity in social and biological models”, presentation at the<br>J. (2009): “More on confidence intervals for partially identified parameters”,<br>(2015): “Estimation and inference of heterogeneous treatment effects using random<br>(2009): “High-dimensional variable selection”,<br>(2006): “Information-theoretic deconvolution approximation of treatment effect distri-<br>bution”, unpublished manuscript. College Station, TX: Texas A&M University.<br>, Ph.D. dissertation. Durham, NC: Duke University. Availa-<br>Appendix 1. Simulation Details<br>Throughout the toolkit, we use results from simulation exercises for illustration. In this ap-<br>pendix, we describe the data generating processes used for each of these simulations.<br>Simulation 1 is our workhorse simulation, because it c n be used to generate cross-sectional<br>In this simulation, individual<br><!-- End of picture text -->
( )-w("}° _ )).

. We req exa tly half of observations to be randomly selected for treatSimulation 2 draws data from two data generating processes with identical average treatassume the following data-generating ro : require exactly half of observations be electe f nt variable for each individual and selecting the t off s niform random ian value across In Simulation 3, potential outcomes a e: ()-n(P ED)0 In ddi on, we ssum We do not condition on i q ere is tted variable which is positively correla d eatment effects when andn ly correlated Appendix 2. Estimating ditional Probabilities
To construct the IPW and IV weights, we need conditional probabilities for stratification, for example, the randomiza on rules provide the stratum pecific treatment probabilities. Otherwise, we need to esti ate and predict grist and Imbens (2002) recommend sorting t e data into cells and using the proportion of treated observations within each cell. This me hod is nonparametric and efficient. For continuous covariates, Hirano, Imbens and Ridder (2003) suggest a flexible logistic specification, including polynomials of covariates and interaction terms. The authors give conditions under which the resulting estimator is nonparametric. The appendix of Imbens (2015) presents an references. Alternative semiparametric or nonparametric strategies are available. Because Both the IV and the IPW estimators are sensitive to observations in the tails of Unfortunately, t is difficult to predict this conditional probability with much accuracy n ar its boundaries. Crump et al. (2009) suggest trimming the sample and ignoring observations is close to zero or one when estimating quantiles. If the rate of trimming decreases with ample size, the resulting estimator remains consistent. The authors suggest using only obse vations with predicted probability between 0.1 and 0.9 as a rule of thumb. \=2| ( )4 |=2c 4 ( eC )E(e ). ca_ be estimated from the treatme roup residuals and ea E(e ) ca e fro the control group residuals. The e, this is a system of ind penden qu nsand tions for the first moment of the treatmente ect istributio E(A )=E[( I= ( )e@ EC ).

These moments can be estimated recursively u t e sample counterparts to each term in the above equation. Specifi ~~—~~ Vv (e) ~~=—~~ (YY 8B)
<!-- Start of picture text --> treatm nt and control group observations, respecti ely.<br><!-- End of picture text -->
Appendix 4. Additional Results from Applications <!-- Start of picture text --> 0.00<br>-0.02<br>-0.04<br>rat -0.06<br>Sd<br><= -0.08<br>Es<br>59 0.10<br>oO<br>=<br>QO -0,12<br>-0.14<br>-0.16<br>-0.18<br>33 35 37 39 41 43 45 47 49 51 53.55 57 59 61 63 65 67 69 71 73 75 77 79 81<br>Financial proficiency score<br><!-- End of picture text --> Effect estimate Statistic.

Standard deviation 75/25 perc. ratio 10th percentile 25th percentile 50th percentile 75th percentile 90th percentile Effects based on inverse probability weighting (Firpo and Pinto, 2016). Weights based on logistic regression of treatment on gender indicator and quadratic polynomial of baseline scores. Standard errors and confidence region based on the bootstrap (five thousand replications) and false rejection across tests), following Romano and Wolf (2010).
Percentile The first four specifications include 848 schools. The last two include 851 schools. Bootstrap standard errors in parentheses, based on five thousand replications and clustered at the school level.