metadata: authors: 0: author: Essama-Nssah, B.
count: World
docna: 0: docna: Propensity score matching and policy impact
            analysis - a demonstration in EViews
docty: Policy Research Working Paper
subtopic: Skills Development and Labor Force Training,State Owned Enterprise Reform,Public Sector Administrative & Civil
            Service Reform,International Trade and Trade Rules,Privatization,Energy Privatization,Inequality,Democratic Government,Economics and Finance of Public
            Institution Development,Public Sector Administrative and Civil
            Service Reform,De Facto Governments
teratopic: Science and Technology Development,Education,Poverty Reduction
abstracts: cdata!: Effective development policymaking
            creates a need for reliable methods of assessing
            effectiveness. There should be, therefore, an intimate
            relationship between effective policymaking and impact
            analysis. The goal of a development intervention defines the
            metric by which to assess its impact, while impact
            evaluation can produce reliable information on which
            policymakers may base decisions to modify or cancel
            ineffective programs and thus make the most of limited
            resources. This paper reviews the logic of propensity score
            matching (PSM) and, using data on the National Support Work
            Demonstration, compares that approach with other evaluation
            methods such as double difference, instrumental variable,
            and Heckman's method of selection bias correction. In
            addition, it demonstrates how to implement nearest-neighbor
            and kernel-based methods, and plot program incidence curves
            in E-Views. In the end, the plausibility of an evaluation
            method hinges critically on the correctness of the
            socioeconomic model underlying program design and
            implementation, and on the quality and quantity of available
            data. In any case, PSM can act as an effective adjuvant to
            other methods.
repnme: repnme: Propensity score matching and policy impact
            analysis - a demonstration in EViews
display_title: Propensity score matching and policy
            impact analysis - a demonstration in EViews
listing_relative_url: /research/2006/04/6708043/propensity-score-matching-policy-impact-analysis-demonstration-eviews
topicv3: Poverty m1326687 695,Trade m1326924 698,Privatization m1327236 1283,Training Programs m1504140 2817
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Science and Technology Development,Education,Poverty Reduction
no_of_pages: 59
model_extractions: WPS 3877

# **_PROPENSITY SCORE MATCHING AND POLICY IMPACT ANALYSIS A Demonstration in EViews_**

B. Essama-Nssah<sup>*</sup> Poverty Reduction Group (PRMPR) The World Bank Washington, D.C.

# **_Abstract_**

Effective development policymaking creates a need for reliable methods of assessing effectiveness. There should be, therefore, an intimate relationship between effective policymaking and impact analysis. The goal of a development intervention defines the metric by which to assess its impact, while impact evaluation can produce reliable information on which policymakers may base decisions to modify or cancel ineffective programs and thus make the most of limited resources. This paper reviews the logic of _propensity score matching_ (PSM) and, using data on the National Support Work Demonstration, compares that approach with other evaluation methods such as double difference, instrumental variable and Heckman’s method of selection bias correction. In addition, it demonstrates how to implement _nearest-neighbor_ and _kernel-based methods_ , and plot program incidence curves in EViews. In the end, the plausibility of an evaluation method hinges critically on the correctness of the socioeconomic model underlying program design and implementation, and on the quality and quantity of available data. In any case, PSM can act as an effective adjuvant to other methods.

Keywords: EViews, Double Difference, Impact Analysis, Instrumental Variables, Kernel Function, Matching, Propensity Score, Sample Selection.

JEL: C14, C81, C87, I38.

World Bank Policy Research Working Paper 3877, April 2006 _The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the view of the World Bank, its Executive Directors, or the countries they represent. Policy Research Working Papers are available online at_ _<u>http://econ.worldbank.org</u>_ <u>.</u>

> * The author would like to thank Glenn Sueyoshi (Quantitative Micro Software) and Jian Zhang (University of California at Davis) for very helpful suggestions.  He is also grateful to Pedro Olinto (LCSPP) and Markus Goldstein (PRMPR) for insightful comments on an earlier draft of the paper.  Finally, thanks are due to Martin Ravallion (DECRG) and Vijdan Korman (PRMPR) for information sharing and useful discussions.
# **TABLE OF CONTENTS**

|1. Introduction.................................................................................................................... 2|
|---|
|2. Matching Methods ......................................................................................................... 5|
|The Principle of Matching .............................................................................................. 5|
|Propensity Score Matching Algorithms.......................................................................... 9|
|Nearest-Neighbor Matching...................................................................................... 12|
|Kernel Matching ....................................................................................................... 14|
|A Brief Comparison with other Evaluation Methods ................................................... 16|
|3. Numerical Implementation .......................................................................................... 20|
|Estimating the Propensity Score................................................................................... 23|
|Underlying Model..................................................................................................... 23|
|Maximum Likelihood Estimation............................................................................. 24|
|Estimates of Program Impact........................................................................................ 27|
|Nearest-Neighbor.......................................................................................................... 31|
|Kernel Matching ........................................................................................................... 32|
|4. Concluding Remarks.................................................................................................... 33<br>**Tables**|
|Table 3.1. Descriptive Statistics for the Underlying Data ............................................... 21|
|Table 3.2. Output of the Binary Command....................................................................... 26|
|Table 3.3. Matched Estimates of the Treatment Impact ................................................... 27|
|Table 3.4. Parametric Estimates of the Treatment Impact................................................ 28|

# **Figures**

|Figure 3.1. Distribution of Relative Impact Based on Gaussian Kernel PSM................. 29<br>Figure 3.2. Relative Program Incidence Based on Gaussian Kernel PSM....................... 30<br>**Boxes**|
|---|
|Box 3.1 EViews Code for Nearest-Neighbor Matching................................................... 31|
|Box 3.2 EViews Code for Gaussian Kernel Matching ..................................................... 32|
|Box 3.3 EViews Code for Epanechnikov Kernel Matching............................................. 33|

# **Appendices**

|A. Propensity Score Estimation with the Log Likelihood Object .................................... 35|
|---|
|B. Computer Code for Propensity Score Matching .......................................................... 41|
|C. Coping with Unobservable Heterogeneity................................................................... 45|
|References......................................................................................................................... 54|

1
# **_1.  Introduction_**

Effective development policymaking creates a need for reliable methods of assessing whether an intervention had (or is having) the intended effect. _There should be therefore, an intimate relationship between effective policymaking and impact analysis_ . The goal of an intervention defines the metric by which to assess its effectiveness. Effective methods of evaluation produce reliable information on what works and why, and policymakers may use such information to modify or cancel ineffective programs and thus make the most of limited resources (Grossman 1994).

The assessment of the impact of a program (or a development intervention) requires _a model of causal inference_ . Holland (1986) specifies such a statistical model. He starts from the fundamental observation that the effect of a cause can be understood only in relation to another cause. This is the same idea underlying the economic principle of assessing the return to a resource employed in one activity relative to its opportunity cost (i.e. what it would have earned in the next best alternative use). Thus we can assess the effect of a development intervention only if we know what would have happened without such an intervention.

Consider a simple situation involving only two causes: program participation versus nonparticipation. A statistical causal inference model applicable to such a case involves the following elements: (1) a population of units upon which causes or interventions may act (e.g. individuals, households, districts, firms or regions), (2) an assumption that each unit is potentially exposable to the causes, (3) an observable variable<sup>1</sup> **d** , indicating the cause to which a given unit is exposed (e.g. **d=1** for exposure, and zero otherwise), (4) a set of variables representing pre-exposure attributes for each unit (some attributes may be observable, call them **x** , and some not, call these ε ); and (5) a variable, **y(d)** , representing the potential response of unit to exposure. In fact, **y** represents two variables standing for two potential responses: **y1** under exposure, and **y0** if no exposure.

> 1 Recall that a variable in this context may be viewed as a real-valued function defined over a unit.  The value taken by such a variable for a given unit is an outcome of a measurement process applied to the unit (Holland 1986).

2
Within the above framework, the effect of exposure on unit **i** is measured relative to non-exposure on the basis of the response variable **y** . If we call this effect, **gi** , then _g i_ = ( _y_ 1 _i_ − _y_ 0 _i_ ) . It is impossible to observe the value of the response variable for the same individual under two mutually exclusive states of nature (exposure and nonexposure). This _Fundamental Problem of Causal Inference_ (Holland 1986) makes it impossible to observe the effect of exposure on unit **i** . This is why evaluation methods are considered as ways of dealing with this _missing data_ problem. If the intervention is limited to a subset of the population, many of the methods suggest turning to nonexposed units (non-participants) in search of the missing information. They also specify circumstances under which the use of such information yields reliable estimates of the relevant effect.

The assumption of _unit homogeneity_ (Holland 1986) characterizes a benchmark case where the effect on individual **i** could be reliably estimated. An individual response is a function of participation, observable and unobservable characteristics. Suppose we can find among non-participants an individual **j** with the same pre-exposure (observable and non-observable) attributes as participant **i** . Thus, under unit homogeneity, the outcome of this non-participant is a proxy for what would have happened to **i** hadn’t she received the intervention. Hence, the effect of the intervention on **i** can be estimated as: _g i_ = ( _y_ 1 _i_ − _y_ 0 _j_ ) . The assumption of unit homogeneity is thus analogous to the _ceteris paribus_ assumption used in scientific enquiry. The assumption serves as a benchmark case against which to assess the implications of heterogeneity. In non-exposure state, one would generally expect response heterogeneity for participants and non-participants, particularly when eligible candidates are given the choice to participate or not<sup>2</sup> .

The most common impact indicator of interest is the mean impact of treatment on the treated. It is also known as the _average treatment effect on the treated_ (ATET). Here treatment means exposure to a cause or participation in a social program. Let _g_ = ( _y_ 1 − _y_ 0 ) , then the mean impact on the treated can be written as a conditional mean:

> 2 Heckman and Smith (1995) cite the case where those who choose to join a social program do so because of the poor alternative they face outside the program.  In such a case, non-participants would have better outcomes than participants had the latter not elected to participate.  This response heterogeneity is also known as _selection bias_ .

3
_ATET_ = _E_ ( _g_ | _x_ , _d_ = 1 ) = _E_ ( _y_ 1 | _x_ , _d_ = 1 ) − _E_ ( _y_ 0 | _x_ , _d_ = 1 ) . The missing data here relates to the counterfactual mean _E_ ( _y_ 0 | _x_ , _d_ = 1 ) . One might be tempted to use the mean outcome for nonparticipants _E_ ( _y_ 0 | _x_ , _d_ = 0 ) as a proxy for the above counterfactual mean. However, Heckman and Smith (1995) caution that subtracting the mean response for nonparticipants from the mean outcome of participants yields an estimate<sup>3</sup> which is equal to the average treatment effect on the treated (the parameter of interest) plus _selection bias_ . Selection bias stems from the failure of the assumption of unit homogeneity. In general, nonparticipants differ from participants in the nonparticipation state. This heterogeneity may be due to observable or unobservable characteristics.

There are both experimental and nonexperimental ways of dealing with selection bias. In the case of social _experiments_ , treatment is assigned randomly so that participation is statistically independent of potential outcomes. Thus, the control group is composed of individuals who would have participated but were denied access randomly. Heckman and Smith (1995) explain that the mean outcome of the control group provides an acceptable estimate of the counterfactual mean if randomization does not alter the pool of participants or their behavior, and if no close substitutes for the experimental program are readily available<sup>4</sup> . These authors further explain that randomization does not eliminate selection bias, but rather balances it between the two samples (participants and nonparticipants) so that it cancels out when computing the mean impact. This balancing act can be understood on the basis of the following consideration. Random assignment of treatment ensures that every eligible candidate has the same chance ex ante of being treated. _Therefore the distribution of both observed and unobserved characteristics prior to treatment is the same for both the treated and the control group_ .

In the context of observational (or non-experimental) studies for causal effects, there is no direct estimate of the counterfactual mean analogous to the one based on randomization. This paper focuses on a popular class of impact estimators. These estimators rely on _propensity score matching_ (PSM) which originated with Rosenbaum

> 3 _E_ ( _y_ 1 | _x_ , _d_ = 1 ) − _E_ ( _y_ 0 | _x_ , _d_ = 0 ) = _ATET_ + [ _E_ ( _y_ 0 | _x_ , _d_ = 1 ) − _E_ ( _y_ 0 | _x_ , _d_ = 0 )] 4 There would be randomization bias if those who participate in an experiment differ from those who would have participated in the absence of randomization. Furthermore, substitution bias would occur if members of the control group can easily obtain elsewhere close substitutes for the treatment (Heckman and Smith 1995).

4
and Rubin (1983). The purpose of the paper is to review the logic of this matching method, and illustrate its implementation and the computation of related impact indicators in EViews. PSM is an algorithm that matches treated and nonparticipants on the basis of the conditional probability of participation (the propensity score), given the observable characteristics. If outcomes are independent of participation, conditional on observables, then the use of the matched comparison group would yield an unbiased estimate of the mean impact of treatment.

The paper is organized as follows. Section 2 reviews the structure of two types of matching algorithms. The nearest-neighbor method pairs a given participant with the member of the comparison group with the propensity score that is closest to that of the given participant. Kernel-based methods associate with the outcome of participant **i** a matched outcome computed as a kernel-weighted average of the outcomes of all nonparticipants. This section also provides a brief comparison of PSM with evaluation methods such as double difference (DD), instrumental variable (IV) and Heckman’s method of selection bias correction. A more detailed description of these other methods is presented in Appendix C. Section 3 uses well-known and publicly available data sets to illustrate how to implement in EViews the methods described in section 2. Data on the treated come from the National Supported Work (NSW) Demonstration. The comparison group is drawn from the Population Survey of Income Dynamics (PSID) [Dehejia and Wahba 2002, 1999; Becker and Ichino 2002]. Concluding remarks are made in section 4. Appendix A shows how to estimate the propensity score using the log likelihood object (LOGL) while Appendix B provides the entire computer code for PSM.

# **_2.  Matching Methods_**

# **_The Principle of Matching_**

Matching methods can be framed within the context of _nonparametric estimation_ of the relation between an outcome variable for unit **i** ( **yi** ), a dummy variable indicating participation in the program ( **di** ), and set of other characteristics ( **xi** ). These characteristics are also referred to as covariates and are assumed exogenous in the sense that they are not affected by the intervention. Such a relation can be stated as (Moffit 5
2004): _yi_ = _f_ ( _d i_ , _xi_ ) . In the context of observational studies, the key assumptions underlying matching methods seek to mimic conditions similar to an experiment so that the assessment of the impact of the program can be based on a comparison of outcomes for a group of participants (i.e. those with **di=1** ) with those drawn from a comparison group of non-participants ( **di=0** ).

To yield consistent estimates of program impact, matching methods rely on a fundamental assumption known as “ _conditional independence_ ” or “ _selection on_ ”<sup>5</sup> _observables_ . This assumption can be formally stated as<sup>6</sup> : The above expression states that potential outcomes are orthogonal to treatment status, given the observable covariates. In other terms, conditional on observable characteristics, participation is independent of potential outcomes. Assuming that there are no unobservable differences between the two groups after conditioning on **xi** , any systematic differences in outcomes between participants and nonparticipants are due to participation If one is only interested in the mean impact for the treated, then the assumption of unconfoundedness can be weakened by focusing on potential outcomes in the nonparticipation state (Imbens 2004). This weaker version can be stated as follows.

In other terms, the outcome in the counterfactual state is independent of participation, given the observable characteristics. Thus, conditional on the observables, outcomes for the non-treated (the comparison group) represent what the participants would have experienced had they not participated in the program. Obviously, this makes sense in the particular situation where selection into the program is based entirely on

> 5 This assumption is also known as the _exogeneity_ or _unconfoundedness_ assumption or _ignorable treatment assignment_ (Imbens 2004)

> 6 The symbol ⊥ represents orthogonality between two variables.  Thus conditional independence may also be referred to as conditional orthogonality.

6
observable characteristics<sup>7</sup> . In order to solve the fundamental missing data problem, all we have to do is to find for each participant, one or more nonparticipants with the same values of observables. This is where matching comes in. In general, matching estimators of program effect impute the missing potential outcomes using only the outcomes of the matched individuals from the comparison group.

For matching to be feasible, there must be individuals in the comparison group with the same values of the covariates as the participant of interest. This requires an _overlap_ in the distribution of observables between the treated and the comparison groups. The overlap assumption is usually stated as: A weaker version of the overlap assumption requires only the following (Imbens 2004).

This implies the possible existence of a nonparticipant analogue for each participant. This is all that is required for the estimation of the mean impact on the treated (Smith and Todd 2005a). When this condition is not met, then it would be impossible to find matches for a fraction of program participants.

To fully appreciate the point of the overlap assumption, consider situations where, for some values of **x** , we have either _p_ ( _x_ ) = 0 or _p_ ( _x_ ) = 1. Individuals with such covariates are such that either they never receive treatment or they always receive it. If they always receive treatment, then they have no counterparts in the comparison group. On the other hand, if they never receive treatment, then they have no counterparts in the treated group. Thus, it would be impossible to use matching methods on such cases (Heckman, Ichimura and Todd 1998). In these circumstances, it is recommended to restrict matching and hence the estimation of the treatment effect on the region of common support. This implies using only nonparticipants whose propensity scores overlap with those of the participants.

> 7 Hence the name “selection on observables” for this orthogonality assumption which implies that unobservables play no role in determining participation (Dehejia and Wahba 2002).

7
Assuming selection on observables, proper matching requires that we select from the sample of non-participants a comparison group in which the distribution of observed characteristics is as similar as possible to the distribution among the participants. In the case of an exact match, the only difference between a participant and her match is that the former received treatment while the latter did not. Hence we may refer to the unconfoundedness assumption as the assumption of _conditional homogeneity_ and the overlap assumption as the _feasibility_ assumption.

Imbens (2004) makes the following observations about the _plausibility_ of the assumption of selection on observables in economic settings. The evaluation of any program ultimately entails the comparison of outcomes for participants and nonparticipants. The key issue then becomes the identification of units that best represent the treated unit had they not participated in the program. Matching analysis based on unconfoundedness is a useful initial step in any serious investigation of program effectiveness. Even in situations where agents do choose their treatment optimally, the assumption of selection on observables may still be valid if the difference in their behavior is driven by unobservables that are uncorrelated to the relevant outcomes. In particular, this might be the case if the objective of the decision maker is distinct from the outcome under consideration.

Diaz and Handa (2004) justify selection on observables in the context of PROGRESA<sup>8</sup> on the basis of the following features of the program. The inclusion of poor households in the program is based only on observable characteristics of households and the locality in which they reside. The program is mandatory and the rate of noncompliance with treatment is very low. Thus self-selection is not a major concern in this case and, matching provides a reliable approach to assess the impact of this program.

In practice matching may become more and more difficult, the larger the set of observable characteristics underpinning the matching exercise. Rosenbaum and Rubin (1983) show that the _dimensionality_ of the matching problem can be significantly reduced by using the propensity score (the conditional probability of participation given the observed covariates). Thus instead of conditioning on an n-dimensional variable, units

> 8 _Programa de Educacion, Salud y Alimentatcion_ or the Education, Health and Nutrition Program of Mexico also known as _Oportunidades_ .

8
are matched on a scalar variable. This simplification is due to the fact that conditional independence remains valid if we use the propensity score **p(x)** instead of the covariates **x** . Thus weak conditional independence [equation (2.2)] can be now expressed as: The rest of this section focuses on propensity score matching.

# **_Propensity Score Matching Algorithms_**

Propensity score matching (PSM) pairs observations on the basis of the conditional probability of participation. Wooldridge (2002) motivates PSM with the following thought experiment analogous to unit homogeneity. Select a propensity score **p(x)** at random. Find two units from the population at large with the same score. Let one participate in the program and the other one not. We can use the outcome of the nonparticipant as a proxy for the outcome she would have experienced had she not joined the program.

Sianesi (2001) explains the basic steps involved in implementing PSM. Assume that we have data on the following: (1) a binary dummy variable identifying participants and non-participants, (2) the outcome to be evaluated, and (3) a set of covariates. First, estimate propensity scores on the covariates using _probit_ or _logit_ and retrieve their predicted values. Second, pair each participant **i** with some group of comparable nonparticipants (on the basis of propensity scores). Finally, estimate the counterfactual outcome of participant **i** as the weighted outcomes of her neighbors in the comparison group.

Formally, let **c(pi)** be the set of the neighbors of **i** in the comparison group, then the matched outcome is defined by the following expression.

9
This matched outcome is our best guess of what participant **i** would have experienced had she not joined the program.

The specification of a matching algorithm is based on two key considerations. Each method requires the definition of _a measure of proximity_ in order to identify nonparticipants who are acceptably close (in terms of the propensity score) to any given participant. This is the criterion that determines **c(pi),** the set of the neighbors of **i** in the comparison group. Finally, we must select _a weighing function_ that determines the weight to be assigned to each member of a neighborhood in the computation of the matched outcome according to (2.6). Here we focus on two classes of algorithms, _nearest-neighbor_ and _kernel_ matching. The nearest-neighbor method assigns a weight of one to the nearest nonparticipant and zero to others. If there are more than one individual in the neighborhood then the method assigns equal weight to each and a zero weight to people outside the neighborhood. The kernel method uses all the members of the comparison group within the common support, and the further away a comparison unit is from the treated one, the lower the weight it receives in the computation of the counterfactual outcome.

In general, matching estimators of the mean impact of treatment on the treated take the following form.

Where **T** stands for the set of treated, and ω **i** can be interpreted more broadly as the evaluative weight assigned to participant **i** . Indeed, any evaluation entails the following four basic dimensions: (1) identification of the objects of value on the basis of the goal of the intervention; (2) the valuation of such objects through a measurement process (e.g. a survey instrument); (3) an overall (or aggregate) characterization of the social state in which these objects of value are observed; and (4) the ranking of alternative states. Evaluative weights underpin the overall characterization of a social state. The mean impact defined by (2.7) is an example of such a characterization. In standard applications, ω **i** is taken to be the sampling weight associated with observation **i** .

10
There may be situations where one is interested in the distribution of the program impact. In such situations, the mean impact indicator is not that helpful. One can factor in distributional concerns in the evaluation by considering the incidence of the gains (Ravallion 2003, 2005). In the context of anti-poverty programs for instance, one may be interested in the incidence of welfare gains. This requires knowledge about the welfare impact conditional on pre-intervention welfare. Pre-intervention welfare can be estimated by subtracting welfare gains<sup>9</sup> ( **gi** ) from post-intervention welfare for all participants. One could then compare both distributions of welfare.

Such a comparison could be made on the basis of a device analogous to the growth incidence curve (or GIC)<sup>10</sup> . The construction of the device is based on the fact that the distribution of outcomes is fully characterized by the mean outcome and the Lorenz representation of relative inequality. Let **y1(p)** stand for the post intervention out come at percentile **p** , **y0(p)** for the matched outcome in the non-treatment state. Let **L1(p)** and **L0(p)** stand for the Lorenz curves representing relative inequality in both states. The program would have a positive impact at percentile **p** , if the following is true.

Where μ **s** and **Ls’(p)** stand respectively for the mean of the distribution and the first order derivative of the Lorenz function in state **s=0, 1** . Since the logarithm is a monotonic transformation, the above condition is equivalent to the following.

> 9 Ravallion (2001) illustrates the importance of accounting for opportunity cost in computing the gains for program participation.  Using the example of cash transfer program designed to keep poor children in school, he explains that forgone income should be netted out of the cash transfer in order to avoid overestimating the income gains for the program.  Indeed, children have to be in school in order to receive the cash transfer.  Thus children who are working prior to joining the program would have to forgo their earnings.

> 10 In the context of pro-poor growth analysis, the _growth incidence curve_ depicts the rate of change in the welfare indicator at each percentile of the distribution due to economic growth (Ravallion and Chen 2003).

11
By analogy to the GIC, **g(p)** may be called _Program Incidence Curve_ (PIC). Thus program incidence at percentile **p** is equal to the rate of change in average outcome between the two states plus a distribution adjustment factor equal to the rate of change of the slope of the Lorenz curve between the two states. The program is considered to have an unambiguous positive social impact if **g(p)** ≥ **0** for all **p** . Expression (2.9) also reveals that program incidence at percentile **p** is equal to the average treatment effect on the treated adjusted by a distributional factor based on the slope of the Lorenz curve.

If one is not interested in this decomposition, then a simpler way to proceed is to plot the ratio **y1/y0** as a function of **p** , the cumulative distribution of the participants ranked in increasing order of the counterfactual outcome. The program would have a positive impact at each percentile where this ratio is greater than one. We may call such a plot _Relative Program Incidence Curve_ (RPIC). We show an example in section 3.

# _Nearest-Neighbor Matching_

For each participant **i** , this method searches for the nonparticipant **j** with the closest propensity score. Based on this concept, the relevant neighborhood is defined by the following expression.

This formula can be used in matching with or without replacement. Matching with replacement creates the possibility of matching a given non-participant to more than one participant. With respect to the trade-off between bias and variance, replacement improves the quality of matches on average while increasing the variance of the impact estimator (Smith and Todd 2005a). In the case of matching without replacement, once a nonparticipant has found his match he drops out of consideration. Matching without replacement can lead to many poor matches in situations where there are many participants with high values of the propensity score and few nonparticipants with such values. This would lead such participants to be matched with nonparticipants who have quite different observable characteristics. Finally, the quality of the impact estimate 12
based on nearest-neighbor matching without replacement depends on the order in which the observations come in the process.

One can try to avoid poor matches by implementing a variant of the nearestneighbor approach known as _caliper matching_ . This method selects the nearest neighbor within a caliper of width δ . The approach imposes a tolerance level on the distance between the propensity score of participant **i** and that of nonparticipant **j** . Formally, the corresponding neighborhood can be stated as follows (Sianesi 2001).

If there is no member of the comparison group within the caliper for the treated unit **i** , then the treated unit is left unmatched and dropped from the analysis. Thus caliper is a way of imposing the common support restriction. Naturally, there is uncertainty about the choice of a tolerance level.

A variant of caliper matching is known as _radius matching_ . In this case, an estimate of the counterfactual is based on the outcomes of all members of the comparison group within the radius **r** , rather than the outcomes of the nearest neighbors within the radius (as in the case of caliper matching). The corresponding neighborhood is: The nearest neighbor mean impact estimator can be written as: Where **nt** is the total number of treated units.

13
# _Kernel Matching_

The idea behind kernel-based matching is to associate the outcome of participant **i** with a matched outcome computed as a kernel-weighted average of the outcomes of all non-participants. The weight assigned to non-participant **j** is in proportion to how close she is to participant **i** . These weights are computed as follows: where **h** stands for the bandwidth. Two kernel functions are commonly used in applied work: _Gaussian_ and _Epanechnikov_ . The former uses information on all non-participants and is defined by the following expression.

The Epanechnikov kernel is defined as follows.

where **I(.)** is the indicator function that takes the value of 1 if its argument is true, and zero otherwise. In other terms, this kernel uses a moving window within the group of non-participants selecting only those whose propensity score is within a fixed _bandwidth_ of size **h** from **pi** , i.e. those for whom | **|pi-pj||<h** .

Other specifications of the kernel function include the _biweight_ or quartic, the _triweigh_ t and the _cosinus_ functions. The quartic kernel is defined as follows.

14
Similarly, the triweight kernel is given by the following expression.

The cosinus kernel is equal to: Based on (2.14), kernel matching can be interpreted as a locally weighted regression of the outcome on a constant, where the weights are determined by some kernel function (Smith and Todd 2005a). This regression analogy can be extended as follows. Assume that expected outcome of participant **i** in non-participation state is a linear approximation written as: Kernel-Weighted OLS minimizes the following weighted sum of squares.

Hence, the outcome participant **i** would have achieved had she not participated in the program is equal to 15
Generally speaking, kernel matching entails the following basic steps (Loader 2003): (i) Choose a particular participant and get her propensity score **pi** ; (ii) Assign weights to observations on non-participants according to their distance from **pi** ; (iii) Specify a local model of the expected outcome (e.g. 2.20); (iv) Estimate the underlying parameters according to (2.21); (v) Use the estimate of the intercept as an estimate of the expected outcome for participant **i** in non-participation state; (vi) Repeat for each participant. In fact, all matching algorithms discussed here can be viewed as extensions of the idea of a _moving average_ . The basic idea behind the moving average involves sliding a “ _window_ ” across the data and taking the average of the response variable (or outcome) for all observations in the window. This is essentially a matching algorithm.

# **_A Brief Comparison with other Evaluation Methods_**

As noted earlier, the outcome of interest is essentially a function of observable and unobservable characteristics of the unit under consideration, and whether or not the unit participated in the program. To assess the effect of an intervention on the outcome, we need to control for all observable and unobservable influences except participation. Failure to control for any of these factors will bias the results. Evaluation methods can therefore be characterized in terms of how they deal with these potential sources of bias.

Randomization balances selection bias between participants and nonparticipants so that subtracting the average outcome of nonparticipants from that of participants yields an unbiased estimate of average program impact. In effect, randomization ensures that participants have the same distribution of pre-intervention attributes (both observable and unobservable) as nonparticipants. In non-experimental situations, PSM attempts to create conditions similar to an experiment by assuming conditional independence and controlling observable heterogeneity through matching. In effect, PSM is a nonparametric method that assumes away unobserved heterogeneity.

There is a parametric analogue of PSM based on the switching regression model where the switching mechanism is assumed exogenous. As we show in Appendix C, the outcome equation is traditionally written as follows.

16
where **usi** stands for unobserved characteristics in state **s=0, 1** (for nonparticipation and participation respectively). Note that the specification of (2.23) reflects the assumption that program effect for a unit with characteristics **xi** and the coefficients β defining the relationship between observables and the outcome are invariant to participation. The invariance of the treatment effect is known as the _common effect_ assumption (Ravallion 2005) or the _homogeneous treatment effects_ assumption (Blundell and Costa-Dias 2000). Conditional independence implies that the expected value the last term in brackets is equal to zero. Thus the application of OLS to the above equation would yield an ^ unbiased and consistent estimate of program impact, θ . The basic difference between this regression analysis and PSM stems from the fact that regression analysis requires a specification of the relation between the outcome, the participation indicator and observed attributes, while PSM requires no such thing.

Suppose there is longitudinal or repeated cross-section information on outcomes and their determinants, and that unobserved influences enter the outcome equation additively and separately in the form of an individual-specific fixed effect, a common macroeconomic effect (the same for all individuals), and a temporal-individual-specific effect (Blundell and Costa-Dias 2000). Assume that the individual-specific and the macroeconomic components affect participation while the temporal-individual-specific effect is independent of participation and observed characteristics. PSM would not be valid in this case, as the conditional independence assumption no longer holds. The Double Difference (DD) or Difference in Differences (DiD) method offers a way to get rid of these troublesome unobservable characteristics. A DD estimate of program impact can be obtained in two steps. First, for each participant and comparison unit, take the difference in outcome before and after the intervention, then compute the difference between the average change for participants and nonparticipants. The first difference removes the offending unobserved heterogeneity and restores conditional independence, while the second produces the impact estimate.

Note that, once conditional independence has been established through firstdifferencing, one can use either regression analysis or PSM to control for observed heterogeneity. In fact, it has been observed that failure to make comparisons in the region of common support can contribute significant bias in DD estimates. Based on 17
expression (2.7) the average treatment effect on the treated over the common support is now given by the following expression which combines PSM with DD to yield matched double difference (MDD) estimates.

Combining PSM with DD is not necessary if there is no observable heterogeneity left after differencing. In general, this combination tends to reduce the bias associated with other evaluation methods (Ravallion 2003).

When participation and outcomes are jointly determined, one can specify a selection model including one participation and one outcome equation. Then one can resort to the instrumental variable approach to try to sort out that part of program impact attributable to exogenous variation in participation. This parametric approach too relies on a sort of conditional independence assumption known as _the exclusion restriction_ . This requires the instrumental variable to be independent of outcomes given participation (Ravallion 2003). Essentially, instrumental variable estimation (IVE) is a two-stage procedure. First estimate the participation equation as a nonlinear binary response model using probit or logit, just as in the first stage of PSM. Then use the predicted value for this stage as an instrument for the participation indicator in the outcome equation (2.23) and run OLS to estimate program impact. A robust identification strategy is to rely both on the nonlinearity of the first-stage estimation process and on the exclusion restriction.

This two-stage procedure suggests a regression-adjusted matching estimator that tends to produce asymptotically efficient estimates (Monteiro 2004). The regression is based on the equation for outcome in the matched comparison group, _y_ 0 _i_ = _xi_ β 0 + _u_ 0 _i_ . In this case, the following restriction is analogous to the conditional independence assumption: _u_ ^0 _i_ = ⎛ _y_ 0 _i_ − _xi_ β ^0 ⎞ ⊥ _d i_ | _p_ ( _zi_ ) , where **zi** is the set of participation ⎜⎝ ⎟⎠

determinants, at least one which must be excluded from the outcome equation. In this case, the impact estimator is 18
^ Where β 0 is the OLS estimate of the regression coefficients in outcome equation for the comparison group<sup>11</sup> .

Finally, one can resort to the standard Heckman’s selection-correction method to cope with heterogeneity bias (see Appendix C for details). Suppose we estimate the participation equation based on the probit model. The results of this probit analysis can be used to compute the following consistent estimates of the inverse Mills ratios above, a consistent two-stage estimate of θ can be obtained by running OLS regression of ^ ^ **yi** on _xi_ , _d i_ , σ _u_ ε [ _d i_ λ 1 _i_ + 1( − _d i_ ) λ 0 _i_ using all of the observations. In other terms, the estimating equation is (Lalonde 1986): Essentially, this two-stage estimator treats unobservable heterogeneity as a problem of an omitted variable, and solves this problem by including an estimate of the omitted variable as a regressor in the outcome equation along with the participation dummy and individual characteristics.

This comparison reveals how each method controls for observable and nonobservable determinants of outcome besides participation. Randomization ensures that, in the pre-intervention state, the distribution of these determinants is the same for both participants and nonparticipants. PSM assumes conditional independence to get rid of unobservable heterogeneity and controls for observed heterogeneity through matching on the propensity score. The conventional regression method is a parametric analogue of PSM. The DD method is a two-step procedure that relies on differencing to control for unobservable heterogeneity stemming from fixed effects, and on averaging to control for observed heterogeneity. The IV method relies on regression analysis to control for observables and uses an instrumental variable to recover conditional independence. The Heckman approach is analogous to the IV method except that it interprets unobservable heterogeneity as an omitted variable problem. Thus instead of using an instrumental

> 11 This approach extents easily to the case of PSM combined with DiD.  See Monteiro (2004) for details.

19
variable for the endogenous dummy variable in the outcome equation, it adds an estimate of the omitted variable in the equation. It turns out that PSM can act as an effective adjuvant to all these methods.

With respect to the choice among various evaluation methods, this comparison suggests situations that are best suited to each method. Hence, no single method can be considered ideal in all circumstances, and one should consider a flexible application of available methods. In the end, the plausibility of an evaluation method hinges critically on the correctness of the socioeconomic model underlying program design and implementation. It also depends on the quality and quantity of data available. The specification of the underlying socioeconomic model must be grounded on a sound understanding of political and socioeconomic determinants of participation, and all relevant factors that influence outcome besides participation.

# **_3.  Numerical Implementation_**

This section illustrates how some of the algorithms and estimators described above might be implemented in EViews. We first describe the data used. Next, we explain the estimation of the propensity score. We also discuss the computer code and output for nearest-neighbor and kernel matching respectively. Finally, in the true spirit of impact evaluation, we present matched estimates along results from alternative methods.

# **_Data_**

Our numerical implementation is based on two data sets from Dehejia and Wahba (1999)<sup>12</sup> . These authors explain that the available data sets are constructed from the data underlying Lalonde (1986). The data for the treated is contained in the file NSWRE74_TREATED.TXT. This is the male sub-sample of a 185 observations from the National Supported Work (NSW) Demonstration. This was a temporary employment program seeking to help disadvantaged workers<sup>13</sup> acquire basic job skills through work

> 12 The actual data sets were downloaded from http://www.columbia.edu/%7Erd247/nswdata.html 13 The program targeted women receiving aid for dependent children (AFDC), ex-drug addicts, ex-criminal offenders and high school drop outs of both sexes.  It was run by the Manpower Demonstration Research

20
experience and counseling in a protected environment. Qualified applicants were assigned to either training or control group _randomly_ . Members of the treatment group received job training for 9 to 18 months depending on their profile and the site. The nonexperimental data for the comparison group are contained in the file PSID_CONTROLS.TXT. This file holds relevant information on 2490 individuals drawn from the Population Survey of Income Dynamics (PSID). This sample consists of all male heads of household less than 55 years old who were continuously observed in the PSID, and not classified as retired in 1975.

Table 3.1. Descriptive Statistics for the Underlying Data

|**Variable**|**Treated**|**Comparison**|
|---|---|---|
|Age|25.82 (7.16)|34.85 (10.44)|
|Education|10.35 (2.01)|12.12 (3.08)|
|Black|0.84 (0.36)|0.25 (0.43)|
|Hispanic|0.06 (0.24)|0.03 (0.18)|
|Married|0.19 (0.39)|0.87 (0.34)|
|No High School Degree|0.71 (0.46)|0.31 (0.46)|
|Real Earnings in 1974|2095.57 (4886.62)|19428.75 (13406.88)|
|Real Earnings in 1975|1532.06 (3219.25)|19063.34 (13596.95)|
|Real Earnings in 1978|6349.14 (7867.40)|21553.92 (15555.35)|
|Zero Earnings in 1974|0.71 (0.46)|0.09 (0.28)|
|Zero Earnings in 1975|0.60 (0.49)|0.10 (0.30)|
|Sample Size|185|2490|

Source: Author’s calculations For both groups, the variables used in the analysis include the following: (1) outcome of interest, represented by real earnings in 1978 (RE78); (2) participation indicator (TREAT =1 if participant, and 0 otherwise); (3) a set of pretreatment covariates including age (AGE), education (EDU), marital status (MARRIED=1 if married and 0 Corporation on ten sites across the United States including Atlanta, Chicago, Hartford, Jersey City, Newark, New York, Oakland, Philadelphia, San Francisco, and Wisconsin (Lalonde 1986).

21
otherwise), indicator of black race (BLACK=1 if black and 0 otherwise), indicator of Hispanic origin (HISP=1 if Hispanic and zero otherwise), indicator of high school degree holder (NODEGREE=1 if no high school degree, 0 otherwise), real earnings in 1975 (RE75), and real earnings in 1974 (RE74).

Table 3.1 contains summary statistics (mean and standard deviation, the latter is between parentheses) of variables in the data set and for both the treated and the comparison group. Focusing on the outcome variable, these descriptive statistics reveal that pretreatment mean earning levels are much lower for the treated than for the comparison group. In 1974, for instance, average earnings stand at about 2,096 for the treated compared to 19,429 for the comparison group. The variable “Zero Earnings in 1974” is a dummy variable that indicates observations for which real earnings in 1974 is zero. There is a similar variable for 1975. The mean of these dummy variables represents the proportion of the sample with zero earnings for the year in question. About 71 percent of the treated had no earnings in 1974 compared to 9 percent for the comparison group. Finally 71 percent of the treated have no high school degree compared to 31 percent for the comparison group. These descriptive statistics suggest that the program did indeed focus on disadvantaged workers.

The estimation of the propensity score requires that data for both participants and nonparticipants reside in the same workfile page. In preparing the data prior to estimation, we use the PAGEAPPEND command to combine information for both groups. The procedure works as follows. Both the source and the destination workfiles must be open. In our case, we bring the information on the treated into the file for the nonparticipants. The file NSW_TREATED is therefore the source file and PSID_CONTROLS is the destination file. We select the destination workfile as the default then invoke the PAGEAPPEND command according to the following syntax<sup>14</sup> : PAGEAPPEND SOURCEFILENAME. Specifically, with PSID_CONTROLS as default we use the command: PAGEAPPEND NSW_TREATED. Before execution, EViews issues the following warning:

> 14 The general syntax of this command is: PAGEAPPEND(OPTIONS) WFNAME[\PAGENAME] [OBJECT_LIST WFNAME stands for the name of the source file.  Optionally, one may specify the name of a page within the source file and the list of objects to be brought from the source.  Other options include a sample restriction specifying which observations from the source page to be appended (see EViews 5.1 User’s Guide for more details).  The result of the command is to append the relevant content of the source page to the active page within the destination workfile.

22
“ _Append will add 185 observations to the workfile and remove any structure. Do you want to continue? Yes or no_ ?” We choose “yes”, and rename the new workfile page: “COMBINED”. This page now holds a total of 2675 observations. It resides in the overall wokfile called “NSWDATA”.

# **_Estimating the Propensity Score_**

# _Underlying Model_

The first step in propensity score matching is to estimate the probability of participation (receiving treatment) conditional on some covariates **x** . This can be based on a model of the probability that **d=1** given **x** , **Pr{d=1|z}** . This probability is also equal to the conditional expectation of the dummy variable **d** . Let π **(z)** stand for this conditional expectation, then we can write:

By analogy to standard regression analysis, we can write a _participation model_ as the sum of the conditional expectation and a random disturbance term, **u** . In other terms, It can be shown that **E(** ε **i)=0** , and **var(** ε **i)=** π **(zi)[1-** π **(zi)]=var(di)** .

To further specify the model, we assume that **di=1** only when the underlying function, **h(zi,** ε **i)** , of observable and unobservable characteristics is greater than zero, otherwise **di=0** . Let _h_ ( _zi_ , ε _i_ ) = _zi_ γ + ε _i_ , and **F()** stand for the cumulative distribution of ε **i** , then we have the following expression.

23
It is commonly assumed that the distribution function **F()** is symmetric as in the case of the normal or the logistic distributions. Thus, Pr{ ε _i_ > − _zi_ γ } = Pr{ ε _i_ < _zi_ γ } . This implies that **F(-zi** γ **)=1-F(zi** γ **)** , and π ( _zi_ ) = _F_ ( _zi_ γ ) . We maintain this assumption of symmetry throughout. This expression reveals that, the marginal effect of covariate **zk** on the propensity score is equal to the following.

where **f()** stands for the density function associated with distribution **F()** .

# _Maximum Likelihood Estimation_

The relevant parameters of this model can be estimated by the maximum likelihood method. For a sample on size **n** , the likelihood function may be written as follows.

The corresponding _log likelihood function_ is The first order conditions for maximizing the log likelihood with respect to the parameters may be written as: 24
The first order derivative of the log likelihood function with respect to the constant term among the explanatory variables is known as the _generalized residual_ (Greene 2000). It is defined as follows.

Thus, the first order conditions for maximization stated in (3.7) may be regarded as an _orthogonality condition_ between the generalized residuals and the explanatory variables in **z** . The condition associated with the constant term (among the explanatory variables) implies that the sum of generalized residuals is equal to zero, which in turn implies that the sample average of estimated propensity scores must equal the proportion of observations for which **di=1** .

To proceed with estimation of the relevant parameters, we must further specify the distribution function **F()** . EViews supports the following specifications: (1) _Logit_ , based on the logistic distribution; (2) _Probit_ , based on the cumulative distribution of the standard normal; and (3) _Gompit_ , based on the Type-I extreme value distribution. We implement the logit method. In the case of binary dependent variable models (such as the one considered here), there are two ways of implementing maximum likelihood estimation in EViews. The simplest (which we use here) is to invoke the BINARY command. The more general approach is to use the log likelihood (LOGL) object. We explain this approach in Appendix A.

25
Table 3.2. Output of the Binary Command

|Variable|Coefficient|Std. Error|z-Statistic|Prob.|
|---|---|---|---|---|
|C|-7.474743|2.443511|-3.059018|0.0022|
|AGE|0.331690|0.120330|2.756509|0.0058|
|AGE2|-0.006367|0.001855|-3.431530|0.0006|
|EDU|0.849268|0.347706|2.442490|0.0146|
|EDU2|-0.050620|0.017249|-2.934625|0.0033|
|MARRIED|-1.885542|0.299331|-6.299189|0.0000|
|BLACK|1.135972|0.351785|3.229161|0.0012|
|HISP|1.969020|0.566859|3.473560|0.0005|
|RE74|-0.000106|3.53E-05|-3.003993|0.0027|
|RE75|-0.000217|4.14E-05|-5.235083|0.0000|
|RE742|2.39E-09|6.43E-10|3.716073|0.0002|
|RE752|1.36E-10|6.65E-10|0.204285|0.8381|
|BLACKU74|2.144130|0.426815|5.023557|0.0000|
|Mean dependent var|0.069159|S.D. depen|dent var|0.253772|
|S.E. of regression|0.146679|Akaike info|criterion|0.162972|
|Sum squared resid|57.27243|Schwarz cr|iterion|0.191605|
|Log likelihood|-204.9754|Hannan-Qu|inn criter.|0.173332|
|Restr. log likelihood|-672.6495|Avg. log li|kelihood|-0.076626|
|LR statistic (12 df)|935.3484|McFadden|R-squared|0.695272|
|Probability(LR stat)|0.000000||||

Source: Author’s calculations.

The basic syntax of the binary command is EQ_NAME.BINARY(OPTIONS) Y X1 [X2 X3...]. EQ_NAME is the name of an equation object. The key applicable options involve the specification of the likelihood, the choice of the maximization algorithm, the method of computation of standard errors, the setting of the maximum number of iterations, the choice of a convergence criterion, and the setting of starting values for the coefficients. The normal distribution is the default likelihood function and quadratic hill climbing is the default maximization algorithm<sup>15</sup> .

The specific command we use on our data is: EQUATION LOGITEQ.BINARY ( D=L, M=1000, C=1E-10, SHOWOPTS) TREAT ZGRP. The first option (D=L) selects the logistic model, and the second sets the maximum number of iterations to one thousand. The third option sets the convergence criterion, while the last asks EViews to show these options with the results. The dependent variable is the treatment indicator (TREAT) while the

> 15 Other algorithms include Newton-Raphson and Berndt-Hall-Hall-Hausman.

26
explanatory variables are contained in a group object called XGRP. Following the specification of Becker and Ichino (2002), these variables are: the constant term (C), AGE, age squared (AGE2), education (EDU), education squared (EDU2), MARRIED, BLACK, HISP, real earnings in 1974 (RE74), real earnings in 1975 (RE75), square of real earnings in 1974 (RE742), square of real earnings in 1975 (RE752) and an indicator for black participants who were not employed in 1974 (BLACKU74). The estimation results are presented in table 3.2.

These results show, for instance, that Blacks and Hispanics were more likely to participate in the NSW program, while marital status has quite a negative influence on the probability of participation. Based on the estimated propensity score, we determine that the region of common support is [0.00061066, 0.97552547].

# **_Estimates of Program Impact_**

Table 3.3. Matched Estimates of the Treatment Impact

||PSM|MDD|RPSML RPSMD|
|---|---|---|---|
|Nearest Neighbor|1667.64|1262.04|1103.88<br>283.63|
|Gaussian Kernel|1537.95|1933.70|1746.25<br>1033.28|
|Epanechnikov Kernel|1370.43|1480.26|1469.80<br>748.71|

Source: Author’s calculations Table 3.3 presents matched estimates of average treatment effect on the treated based on different versions of propensity score matching. Column PSM contains results from standard propensity score matching, column MDD is based on matched double difference, while the last two combine regression analysis and propensity score matching. They differ only in the specification of the outcome equation in the pre-treatment state. Column RPSML is based on a modified specification proposed by Lalonde (1986). The included explanatory variables are: AGE, AGE2, EDU, BLACK, HISP, RE74, and RE75. The specification underlying the last column, RPSMD, comes from Dehejia and Wahba (2002). The relevant variable are: AGE, EDU, BLACK, HISP, NODEGREE, MARRIED, RE74, RE75, U74 and U75.

27
To make some sense of these results, we note from table (3.1) that average real earnings in 1978 were US$ 6,346.14 for the treated. We also have data on the experimental control group. Their average real earnings stood at US$ 4,554.80 in 1978. Thus ATET based on randomization is equal to US$ 1,794.34. Using this as a metric, we first note that all non-experimental methods represented in table 3.1 show a positive program impact. The results also clearly show that impact estimates vary with both matching algorithms and adjuvant tools. Given the data available, the nearest neighbor algorithm seems to perform best in the case of simple PSM. The Gaussian kernel algorithm seems to outperform the other algorithms across variants. In the particular case of regression adjusted matching based on the modified Lalonde (1986) specification, matching using the Gaussian kernel produces an estimate that is very close to the benchmark. A straight application of the DD method yields an estimate of average impact of US$ 2, 326.50. Combining kernel matching with DD seems to improve this estimate, particularly when using the Gaussian kernel. Also, the difference between the regression-adjusted results demonstrates the sensitivity of the results to the specification of the outcome equation. It is thus important to base such specification on a sound understanding of the theory and facts driving possible outcomes.

Table 3.4. Parametric Estimates of the Treatment Impact

||REGML|REGDW|
|---|---|---|
|OLS|217.94|4.16|
|IVE|881.44|1335.50|
|Heckman-1|989.91|796.74|
|Heckman-2|631.81|847.23|

Source: Author’s calculations Table 3.4 presents parametric estimates of treatment impact based on regression analysis under a variety of assumptions about the correlation between participation and outcomes, and on effect invariance. Column REGML is based on the modified Lalonde (1986) specification while Dehejia and Wahba (2002) specification underlies column REGDW. The first three rows assume common effect, while the last does not. Row OLS (ordinary least squares) assumes conditional independence, while the last three do not. Row IVE (instrumental variable) corrects for unobserved heterogeneity using the propensity score as an instrument for the endogenous participation dummy. Heckman adds an estimate of 28
the inverse Mills ratio to the relevant outcome equation to correct for unobserved heterogeneity. These parametric estimates show positive impact just like the nonparametric and semi-parametric methods of table 3.3. Given the experimental benchmark, these results show that, the nonparametric and semi-parametric methods generally outperform the parametric ones.

<u>Figure 3.1. Distribution of Relative Impact Based on Gaussian Kernel PSM</u> <!-- Start of picture text --> .35<br>.30<br>.25<br>.20<br>.15<br>.10<br>.05<br>.00<br>0 5 10 15 20 25 30 35<br>Relative Gains<br>Density<br><!-- End of picture text --> To look beyond average impact, we focus on the distribution of relative impact based on Gaussian kernel matching (the corresponding average is US$ 1,537.95 in table 3.3). This distribution is presented in figure 3.1. The figure is essentially a smoothed histogram of the sample distribution of the ratio **y1/y0** among the participants. The smoothing is based on the Epanechnikov kernel function using Silverman’s method to determine the bandwidth. The EViews command to accomplish this is the following: SERIES_NAME.KDENSITY(K=E, S, 100, O=ARG)<sup>16</sup> . The data underlying figure 3.1 indicate

> 16 The options have the following meanings: K=E for Epanechnikov, S for Silverman’s formula h=0.9 α n (-

> 1/5) , where α =min{standard deviation, (interquartile range)/1.34}.  The interquartile range is the difference

29
that relative impact varies from zero (for those participants who earned no income in 1978) to a maximum of 37.92, with an average of 2.27 and a standard deviation of 4.12.

<u>Figure 3.2. Relative Program Incidence Based on Gaussian Kernel PSM</u> <!-- Start of picture text --> 40<br>30<br>20<br>10<br>0<br>0 10 20 30 40 50 60 70 80 90 100<br>Percentiles of Counterfactual Outcome<br>Relative Gains<br><!-- End of picture text --> To identify who might have gained most out of the program, we look at the same distribution of impact in the form of the _relative program incidence curve_ presented in figure 3.2. This curve is obtained as follows. Rank the participants in ascending order of the estimated counterfactual outcome. Compute the ratio **y1/y0** (relative impact), and the relative rank **p** of each participant. Finally, plot the relative impact as a function of **p** . The relative incidence curve shows that, in general, the people who gain most from the program are among participants who would have been at the lower end of the counterfactual distribution. Indeed, the underlying data indicate that the mean relative impact is 3.85 for the 47 percent “poorest” participants, and only 0.87 for the 53 percent “richest”.

between the 75<sup>th</sup> and 25<sup>th</sup> percentiles. The integer 100 specifies the number of points at which to evaluate the density function. Finally O=ARG specifies the matrix to contain the kernel density computation.

30
We now move to the implementation of the matching algorithms in EViews. The computer code for the entire program (PSCOREMATCH.PRG) is presented in Appendix B. It has three major components. The first estimates the propensity score and determines the region of common support. The second component relies on three subroutines to compute matched outcomes for the three algorithms of interest. The last component computes the average treatment effects on the treated as reported in table 3.2. We obtain the same results as those reported by Becker and Ichino (2002). These are reported in table 3.3 , column PSM.

Note that all the three subroutines we consider next are local as opposed to global. In EViews, global subroutines have the ability to create or alter global objects. Such objects stay in the workfile after the routine has run. All objects created by a local subroutine are local in the sense that they are meaningful only within the subroutine and disappear from the workfile once the routine has run. Thus, a subroutine may not use or update global objects directly from within the subroutine. However global objects corresponding to arguments may be used and updated by referring to the arguments. Such objects must be created outside the local subroutine and passed on to the subroutine as arguments. The use of local subroutines minimizes workfile clutter.

# **_Nearest-Neighbor_**

Box 3.1 EViews Code for Nearest-Neighbor Matching SUBROUTINE LOCAL NEIGHBOR(VECTOR VP, SERIES PS, SERIES Y, VECTOR MO) SMPL @ALL !NT=@ROWS(VP) FOR !K=1 TO !NT SERIES U{!K}= ABS(VP(!K) - PS) SCALAR M{!K}=@MIN(U{!K}) SERIES NDIJ{!K}=(U{!K}=M{!K}) SCALAR DNO{!K}=@SUM(NDIJ{!K}) SERIES YNWIJ{!K} 'To hold matched outcomes IF DNO{!K} THEN YNWIJ{!K}=(NDIJ{!K}/DNO{!K})*Y ENDIF MO(!K)=@SUM(YNWIJ{!K}) NEXT ENDSUB 31
Box 3.1 presents the code for a local subroutine called NEIGHBOR, designed to perform nearest-neighbor matching. The subroutine has four arguments referring to global objects: (1) the vector of propensity scores for the treated, (2) the series containing the propensity scores for the non-treated, (3) the series of outcomes for the non-treated, and (4) the vector to keep the matched outcomes. The core loop defining this subroutine works as follows. For each participant, define a series containing the distances between her propensity score and those of all nonparticipants. Find the minimum value. Create a dummy variable that is equal to one for each observation for which the distance is equal to the minimum and zero otherwise. Finally use this dummy variable to construct the relevant weight and apply equation (2.6) for the matched outcomes.

# **_Kernel Matching_**

Box 3.2 EViews Code for Gaussian Kernel Matching SUBROUTINE LOCAL GAUSS(SERIES CS, VECTOR VP, SERIES PS, SERIES Y, VECTOR MO) !BW=0.06 . !NT=@ROWS(VP) SMPL @ALL IF CS FOR !K=1 TO !NT SERIES U{!K}= ABS(VP(!K) - PS)/!BW SERIES KIJ{!K}=@DNORM(U{!K}) SERIES YWIJ{!K}=(KIJ{!K}/@SUM(KIJ{!K}))*Y MO(!K)=@SUM(YWIJ{!K}) NEXT ENDSUB Both boxes 3.2 and 3.3 present the code for the implementation of kernel matching. As suggested by the names, GAUSS implements matching using the Gaussian kernel while EPAN uses the Epanechnikov kernel. In addition to the four arguments of NEIGHBOR, these two subroutines have an extra argument to enforce the common support condition. Again, the fundamental logic is the same in these cases as the one underlying NEIGHBOR. Essentially, these two subroutines implement expression (2.6) where the weights are defined according to (2.14) and the kernel are defined respectively by (2.15) 32
and (2.16). Note that in the case of the Epanechnikov kernel, there is no need to include the constant term since it cancels out in the definition of weights.

Box 3.3 EViews Code for Epanechnikov Kernel Matching SUBROUTINE LOCAL EPAN(SERIES CS, VECTOR VP, SERIES PS, SERIES Y, VECTOR MO) !BW=0.06 !NT=@ROWS(VP) SMPL @ALL IF CS FOR !K=1 TO !NT SERIES U{!K}= ABS(VP(!K) - PS)/!BW SERIES ED{!K}= U{!K}<=1 SERIES EKIJ{!K}=(1-U{!K}^2)*ED{!K} SCALAR DNO{!K}=@SUM(EKIJ{!K}) SERIES YEWIJ{!K} IF DNO{!K} THEN YEWIJ{!K}=(EKIJ{!K}/DNO{!K})*Y ENDIF MO(!K)=@SUM(YEWIJ{!K}) NEXT ENDSUB

# **_4.  Concluding Remarks_**

Effective development policymaking creates a need for reliable methods for assessing whether an intervention had (or is having) the intended effect. Such an assessment would be impossible without an estimate of what would have happened in the absence of the intervention. Evaluation methods generally rely on either a control or a comparison group to estimate this counterfactual. This paper reviews the logic of the propensity score matching method, compares matching to other methods and demonstrates numerical implementation in EViews using NSW data.

In general, individual characteristics (observable and unobservable) can confound any assessment of program impact. Failure to account for such heterogeneity will bias evaluation results. Evaluation methods can therefore be characterized in terms of how they control for these confounding effects in order to isolate the impact of the intervention. Randomization ensures that both participants and nonparticipants have the same distribution of these characteristics so that the comparison of average outcome 33
between the two groups yields an unbiased estimate of program impact. Propensity score matching (PSM) attempts to create conditions similar to an experiment by assuming that unobservable heterogeneity plays no role in participation (conditional independence). PSM accounts for observable heterogeneity by pairing participants with nonparticipants on the basis of the conditional probability of participation, given observable characteristics. The feasibility of propensity score matching depends on the extent to which the distribution of propensity scores within the treatment group overlaps with that of the comparison group.

The specification of a matching algorithm hinges on two basic factors. The first involves the definition of a measure of proximity (in the space of propensity scores) in order to identify nonparticipants who may be considered similar enough to any given participant. The second entails a weighing function that determines the weight to be assigned to each member of a neighborhood in the computation of the counterfactual outcome. The choice among matching algorithms implies a trade-off between _bias_ and _precision_ in estimation.

The traditional regression analysis based on a switching model of the outcome is a parametric analogue of PSM when the switching mechanism is assumed to be exogenous. When this assumption fails and longitudinal data are available for both participants and nonparticipants, then one can use the DD method to control heterogeneity assuming that the unobservable part stems from a fixed effect. Alternatively, one can resort to IV or to Heckman’s selection bias correction method to handle unobservable heterogeneity.

A comparison of all these methods suggests no one method fits all circumstances. Thus one should consider a flexible application of available evaluation methods. The numerical implementation of these methods on NSW data reveals that nonparametric methods tend to produce impact estimates that are closer to the experimental benchmark than the parametric approach. In the end, the plausibility of an evaluation method hinges critically on the correctness of the socioeconomic model underlying program design and implementation. It also depends on the quality and quantity of data available. The specification of the underlying socioeconomic model must be grounded on a sound understanding of political and socioeconomic determinants of participation, and all relevant factors that influence outcome besides participation.

34