metadata: authors: 0: author: Brown,Caitlin Susan
1: author: Ravallion,Martin
2: author: Van De Walle,Dominique
count: Africa
docna: 0: docna: A poor means test ? econometric targeting in Africa
docty: Policy Research Working Paper
subtopic: Inequality,Access of Poor to Social Services,Economic Assistance,Services & Transfers to Poor,Disability,Social Protections & Assistance,Poverty Diagnostics,Poverty Assessment,Poverty Lines,Poverty Monitoring & Analysis,Small Area Estimation Poverty Mapping,Poverty Impact Evaluation,Employment and Shared Growth
teratopic: Macroeconomics and Economic Growth,Social Protections and Labor,Poverty Reduction
abstracts: cdata!: Proxy-means testing is a popular method
            of poverty targeting with imperfect information. In a now
            widely-used version, a regression for log consumption
            calibrates a proxy-means test score based on chosen
            covariates, which is then implemented for targeting
            out-of-sample. In this paper, the performance of various
            proxy-means testing methods is assessed using data for nine
            African countries. Standard proxy-means testing helps filter
            out the nonpoor, but excludes many poor people, thus
            diminishing the impact on poverty. Some methodological
            changes perform better, with a poverty-quantile method
            dominating in most cases. Even so, either a basic-income
            scheme or transfers using a simple demographic scorecard are
            found to do as well, or almost as well, in reducing poverty.
            However, even with a budget sufficient to eliminate poverty
            with full information, none of these targeting methods
            brings the poverty rate below about three-quarters of its
            initial value. The prevailing methods are particularly
            deficient in reaching the poorest.
repnme: repnme: A poor means test ? econometric targeting
            in Africa
display_title: A poor means test econometric
            targeting in Africa
listing_relative_url: /research/2016/12/27018191/poor-means-test-econometric-targeting-africa
topicv3: Poverty m1326687 695,Poverty Measurement and Analysis m1326941 707,Social Protection and Labor m1327194 697,Cash Transfers m1327437 1608,Transfer Payments m1504098 2777
publishtoextweb_dt: 2016-12-13T00:00:00Z
historic_topic: Macroeconomics and Economic Growth,Social Protections and Labor,Poverty Reduction
no_of_pages: 55
author_keys: 0: author_key: 1423178
1: author_key: 99002
2: author_key: 99093
model_extractions: WPS7915 Policy Research Working Paper 7915 A Poor Means Test? Econometric Targeting in Africa _Caitlin Brown Martin Ravallion Dominique van de Walle_ Development Research Group Human Development and Public Services Team December 2016
Policy Research Working Paper 7915

# **Abstract**

Proxy-means testing is a popular method of poverty targeting with imperfect information. In a now widely-used version, a regression for log consumption calibrates a proxymeans test score based on chosen covariates, which is then implemented for targeting out-of-sample. In this paper, the performance of various proxy-means testing methods is assessed using data for nine African countries. Standard proxy-means testing helps filter out the nonpoor, but excludes many poor people, thus diminishing the impact on poverty. Some methodological changes perform better, with a poverty-quantile method dominating in most cases. Even so, either a basic-income scheme or transfers using a simple demographic scorecard are found to do as well, or almost as well, in reducing poverty. However, even with a budget sufficient to eliminate poverty with full information, none of these targeting methods brings the poverty rate below about three-quarters of its initial value. The prevailing methods are particularly deficient in reaching the poorest.

This paper is a product of the Human Development and Public Services Team, Development Research Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at dvandewalle@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
A Poor Means Test? Econometric Targeting in Africa Caitlin Brown, Martin Ravallion and Dominique van de Walle<sup>1</sup>

**Keywords:** Poverty, cash transfers, means test, targeting, Africa

**JEL:** I32, I38

> 1 Brown and Ravallion are with the Department of Economics, Georgetown University, and van de Walle is with the Development Research Group, World Bank. For their comments the authors thank Arthur Alik-Lagrange, <mark>Kathleen Beegle, Mary Ann Bronson, Raphael Calel, Phillippe Leite, M</mark> ead Over, Lucian Pop, Adam Wagstaff, anonymous reviewers from the World Bank’s Africa region and seminar participants at Georgetown University. The authors are grateful to the World Bank’s Strategic Research Program for funding assistance.
# **1. Introduction**

While universal social programs—whereby everyone is covered—are excellent at reaching the poorest, the beneficiaries can include many people who do not need this form of public help. Governments have tried many ways of assuring better “targeting,” with the explicit aim of concentrating the benefits of a social policy on poor people. The means used vary in their data requirements, methodological sophistication and costs (both administrative and broader social costs).

Readily measurable proxies for consumption or income are often used in efforts to reduce poverty in settings in which the means-testing of benefits is not an administratively feasible option, as in most low-income countries (and many middle-income countries). Efficiency considerations point to the need for indicators that are not easily manipulated by actual or potential beneficiaries. Proxy variables, such as gender and education, family size and housing conditions, have been common.<sup>2</sup> A score based on these variables is used in validating other targeting methods, such as those based on community-level subjective assessments of who is “poor.” The scores are also entering many social-protection registries—national data bases that are used in various ways including to flag ineligible households in future schemes.

The main challenge has been in setting the score’s weights. Various “poverty scorecards” or “basic needs indicators” have been used. Some versions use _ad hoc_ weights, such as taking a simple average of the scores across components.<sup>3</sup> Practitioners have turned to more sophisticated statistical methods in an effort to further improve targeting accuracy. These methods have come to be known as _proxy means testing_ (PMT).<sup>4</sup> This paper assesses an increasingly popular solution in which the weights in the PMT are identified from regression coefficients for household consumption or income as a function of readily observed covariates. The regression is calibrated to survey data and then used to make the out-of-sample predictions for the relevant population. This has the intuitive attraction that the dependent variable is a

> 2 This idea appears to have emerged in social policy making in Chile in the 1980s (Grosh, 1994, Ch.5). Grosh et al. (2008) provides a useful overview of PMT and other targeting methods found in practice in developing countries, with details on many examples.

> 3 A popular example of the poverty scorecard was proposed by Schreiner (2010); the <u>Progress out of Poverty Index uses</u> Schreiner’s (2015) method. The scorecard includes 10 easily measured correlates of poverty which are used to form a composite index. Diamond et al. (2016) argue that the predictive ability of such scorecards can be improved by calibrating the variables and their weights to local (sub-national) conditions, for which purpose they advocate econometric methods. 4 This term appears to be due to Grosh and Baker (1995, p. ix), who define PMT as “a situation where information on household or individual characteristics correlated with welfare levels is used in a formal algorithm to proxy household income, welfare or need.”

2
well-established measure of household economic welfare and, indeed, the same variable is typically used in measuring poverty.<sup>5</sup> To distinguish it from other methods of means testing, we will use the term “econometric targeting” to refer to any PMT based on a regression model. An influential early contribution by Grosh (1994) compared numerous social programs in Latin America and concluded that this class of methods produced the best targeting outcomes, measured in terms of reducing inclusion errors, whereby a nonpoor person is counted as poor. Various versions of econometric targeting have since been proposed, and the method has been widely implemented in developing countries.<sup>6</sup> Econometric targeting has also been criticized for its seemingly poor predictions about who is poor and who is not. For example, Kidd and Wylde (2011, p.ii) refer to the method’s “considerable inaccuracy at low levels of coverage.” Transparency has also been a concern. Sometimes the score variables and weights are deliberately kept secret for incentive reasons. In other cases, the method and formula are too complicated, or too poorly explained, for public consumption. Either way, observers on the ground do not always understand why some people are selected and some are not based on these targeting methods. With reference to a conditional cash transfer scheme in Nicaragua using PMT, field work by Adato and Roopnaraine (2004, p.15) led them to write that: “...the targeting process as a whole is poorly understood at the community level in both geographicaland household-targeted communities. When asked why some households were beneficiaries and others not, informants offered a range of explanations, from divine intervention to a random lottery. For example, one informant from a geographically-targeted community noted: ‘ _Well, some people wonder why they weren’t targeted even though they live in this same area. So we tell them that the Bible says that many are called but few are chosen._ ’” In the context of a PMT in Indonesia, Cameron and Shah (2014) argue that considerable local social unrest was generated by this lack of transparency in why some people were deemed beneficiaries and some not. This came with an erosion of local social capital and greater distrust of local administrators.

Another critique relates to the goals of social protection policies, which can be thought of as involving both protection from uninsured risks as well as promotion from poverty over the longer-term.<sup>7</sup> Some observers have questioned the effectiveness of PMT in responding to shocks or targeting insurance. Instead, it is argued that, because it is largely based on long-term assets, PMT is suitable “...for identifying the chronic poor and determining eligibility for programs that provide long-term support” (Del Ninno and Mills, p.22). Nonetheless, PMT is widely used in implementing policies that

> 5 For a critical review of the methods used see Ravallion (2016, Part 2). 6 Useful overviews can be found in Mills et al. (2015) and USAID’s website on <u>Poverty Assessment Tools.</u> 7 On this distinction and the implications for assessing social protection policies see Ravallion et al. (1995).

3
offer short-lived benefits and are seen to be justified by their provision of insurance or emergency relief such as public works and cash transfer schemes.

The criticisms of econometric targeting could reflect either methodological inadequacies or informational/data limitations. On the former, standard regression-based calibration of the PMT score will tend to work less well toward the extremes of the distribution of household consumption. By its design, a standard regression line passes through the means of the data. The residuals will be positively correlated with the dependent variable (more so the higher the variance of the residuals given exogenous regressors).<sup>8</sup> One can expect the method to have a tendency to overestimate living standards for the poorest and underestimate them for the richest, though the degree to which this is problematic for targeting accuracy is unclear. Indeed, it is theoretically possible that the PMT method predicts that nobody lives below a poverty line for which even a sizable share of the population is deemed to be poor based on observed consumptions. Another possibility is that the variables used are not sufficiently good proxies for household consumption. In other words, that there is an information problem.

The paper aims to provide a systematic assessment of the reliability of econometric targeting as a tool for social policies aiming to reduce poverty. We assess what appears to be the most common form of what we call “Basic PMT,” as well as some alternative methods using extra covariates and methods that are arguably more appropriate when it is recognized explicitly that the PMT is for antipoverty policy making. A natural counterfactual for assessing any form of PMT is a uniform allocation—the same for everyone. Various other counterfactuals of interest to policy makers are examined. The study also considers less finely-targeted options to econometric targeting, which are uniform only within stipulated categories.

While Latin America has attracted the bulk of the past research on PMT, we study the method using survey data for the world’s poorest region, Sub-Saharan Africa (SSA). This is also the region where existing social spending has been least effective in reaching the poorest.<sup>9</sup> The specific countries studied are Burkina Faso, Ethiopia, Ghana, Malawi, Mali, Niger, Nigeria, Tanzania and Uganda, being all those countries in SSA with recent and reasonably comparable surveys in the World Bank’s Living Standards Measurement Study (LSMS).<sup>10</sup> For a subset of these countries we also have panel data.

> 8 If the regression model is _y_   _x_   with _Cov_ ( _x_ ,  )  0 then _Cov_ ( _y_ ,  )ˆ  _Var_ (  )ˆ  0 (in obvious notation). 9 For evidence on the point see Ravallion (2016, Chapter 10).

> 10 Existing government safety net programs in Ghana, Malawi, Nigeria, and Tanzania, and World Bank projects in Burkina Faso, Ethiopia, Ghana, Malawi and Niger use PMT, often in combination with geographic and community-based targeting. At the time of writing, PMT is also being considered for Mali.

4
In advocating and assessing PMT, social policy making in developing countries has often emphasized the need to avoid the “leakage” of benefits to the non-poor, and to assure broad coverage of the poor. Following the literature, one can term failures with regard to these two aspects of targeting as the aforementioned “errors of inclusion” (i.e., counting someone as poor who is not) and “errors of exclusion” (i.e., counting someone as non-poor who is in fact poor).<sup>11</sup> The difference is important when deciding how much to spend on a program. Inclusion errors are generally costly to the public budget while exclusion errors save public money. Governments and international financial institutions concerned about the fiscal cost of social policies have thus put greater emphasis on avoiding inclusion errors as a means of cutting the cost to the government without hurting poor people.<sup>12</sup> Some observers have questioned this prioritization, arguing that exclusion errors should get higher weight when the policy objective is to minimize poverty.<sup>13</sup> In this paper we consider various measures of both targeting performance and impacts on poverty.

Some assessments of econometric targeting are already available in the literature.<sup>14</sup> The methods appear to vary considerably across the studies to date, such as in how many variables are used in the PMT, how targeting performance is assessed, and what poverty cutoff point is used. However, documentation is rarely ideal, often leaving the reader to guess what has been done. This makes it difficult to compare results. We provide similar tests on a consistent basis.

We go further than past work in a number of other respects. We consider alternative econometric methods for calibrating the PMT scores. These include methods that recognize explicitly that the goal of PMT is poverty reduction rather than obtaining unbiased estimates of conditional means. We also simulate stylized policies to see how well econometric targeting works. Here we consider simpler alternatives to PMT that have a long history, going back to the state-contingent transfers that were introduced under England’s Poor Laws, and the various proposals that have been made over the last 200 years for a “basic-income scheme.”<sup>15</sup> Additionally, we compare PMT to optimally differentiated

> 11 The distinction between these two targeting errors goes back to Weisbrod (1970) who called them “vertical” and “horizontal targeting efficiency.” Smolensky et al. (1995) called them “errors of inclusion” and “errors of exclusion.” Some authors refer to exclusion as “under-coverage” and refer to inclusion as “leakage.”  In development contexts, influential early contributions were made by Cornia and Stewart (1995) and Grosh and Baker (1995).

> 12 This emphasis on reducing inclusion errors appears to have emerged during macroeconomic adjustment efforts, notably in Latin America in the 1980s (Smolensky et al., 1995).

> 13 See Cornia and Stewart (1995), Smolensky et al. (1995) and Ravallion (2009).

> 14 See Grosh and Baker (1995) (Jamaica, Bolivia, Peru), Ahmed and Bouis (2002) (Arab Republic of Egypt), Narayan and Yoshida (2005) (Sri Lanka), Sharif (2009) (Bangladesh), Stoeffler et al. (2015) (Cameroon), Pop (2015) (Ghana) and Cnobloch et al. (2015) (Malawi).

> 15 On the history of these policy options see Ravallion (2016, Part 1).

5
transfers based on the same information set. In considering these options, we focus directly on the impacts on poverty rather than looking solely at measures of targeting performance. Here we take the view that “better targeting” should not be seen as an end in itself but rather as a possible means of assuring a greater impact on poverty.

The panel data that are available for a subset of countries help us address concerns about measurement errors in consumption. To some degree, what are called “targeting errors” are likely to be measurement errors (Ravallion 2008). By using the panel data to calculate time-mean consumption we can at least partly reduce the effect of measurement error, as a robustness test of our main findings.

Another departure from past work is that we allow for likely lags in implementation; past assessments have ignored the fact that PMT invariably entails such lags, given that the score must be set in advance of implementation. There are lags between the survey year and the release of the PMT formula, and further lags to implementation.<sup>16</sup> We can expect a degree of churning, with households moving in and out of poverty.<sup>17</sup> So implementation lags are likely to constrain the performance of econometric targeting in identifying the currently poor. We exploit the panel nature of our data for a subset of countries to explicitly introduce lags.

There are a number of issues that we do not take up. One of these is whether household consumption obtained from a survey is an adequate welfare indicator. The methods of econometric targeting studied here make that assumption, and we accept it for the purpose of evaluating the performance of these methods. Another issue not taken up here is how well a low level of household consumption identifies deprived individuals; Brown et al. (2016) take up this issue in the context of attempts to reach undernourished women and children. While we do address the performance of econometric targeting for stylized cash transfer programs we do not consider alternatives such as selftargeting using work requirements (“workfare”) or community-based targeting in which local communities are engaged directly in deciding who is poor and who is not.<sup>18</sup> Nor do we consider the (economic, social and political) costs of targeting, which have received some attention in the literature.<sup>19</sup>

> 16 For example, even for a relatively simple PMT such as the <u>Progress out of Poverty Index, we find that across the 59</u> countries for which the index is currently available, the number of years between the survey year and the release date of the index ranges from 1 to 9, with a mean of 3.9 years and a median of 3.5.

> 17 The implications of such churning for assessing the performance of social protection policies are examined further in Ravallion et al. (1995).

> 18 On workfare see, for example, Murgai et al. (2016) and on community-based targeting see Alatas et al. (2012), Karlan and Thuysbaert (2013) and Stoeffler et al. (2016). Barrientos (2013) provides a useful overview of the whole class of social assistance policies in developing countries.

> 19 See the discussions in van de Walle (1998), Gelbach and Pritchett (2000) and Ravallion (2016, Ch. 10).

6
For example, we do not discuss behavioral responses, social stigmas, or implications for social cohesion and political support for poverty programs.<sup>20</sup> A few caveats are also in order. The paper uses existing household consumption surveys to simulate the targeting performance of various methods and the likely impacts on poverty of using the methods to allocate a fixed budget. In a sense, we are creating a “laboratory.” While this is inevitably somewhat artificial, we believe it to be essential for properly testing the performance of the various targeting methods found in practice. It must be acknowledged, however, that things happen in the real world that cannot be predicted in our laboratory setting. None of the methods considered here could be implemented flawlessly. While we do allow for lags in implementation, idiosyncratic mistakes will no doubt also be made. Most likely the methods will perform less well than our calculations suggest. We present and compare the best case results for the various methods reviewed, unaffected by potentially differential costs, ease of implementation, and susceptibility to manipulation and corruption.

In addition, there are potential benefits of each method in practice that we cannot identify in our laboratory setting. Our purpose here is to judge the methods solely on how accurately they reach the poor and by how much transfers targeted in these ways would reduce poverty. These are undeniably important as goals for targeted programs and are usually explicit in the justifications provided by donors and governments. Nonetheless, there may be other objectives in practice such as capacity building or building systems. Donors, governments and NGOs implementing these methods do sometimes make considerable effort to include communities in the selection of beneficiaries, build robust grievance and redress procedures, and continuously monitor and evaluate the performance of the targeting systems. Such efforts could well be important in building administrative capacity for social protection over the longer term and also improve targeting. It is not obvious how an accounting for such potential benefits would alter the relative assessments of alternative methods based on our simulations.

The paper finds that when the counterfactual is a uniform allocation of the same budget, even our Basic PMT allows a substantial reduction in the rate of inclusion errors; in this setting it should be possible to roughly halve the rate of inclusion errors using econometric targeting. When judged against a fixed poverty line, this success at avoiding leakage to the nonpoor comes with seemingly weak coverage of poor people—a high rate of exclusion errors. In other words, the method helps exclude the poor as well as the non-poor. The paper finds that econometric targeting typically provides at most

> 20 Smolensky et al. (1995) conclude that none of these issues is likely to be decisive for or against targeting. Atkinson (1995) argues that broader objectives of social policy (including social solidarity) warn against targeting.

7
modest gains in the poverty impacts over other policy-relevant alternatives. Indeed, in a number of cases and depending on the country and the nature of its poverty profile, simpler state-contingent targeting methods or even a “basic-income scheme” (in which everyone is covered) dominate in certain policyrelevant cases, such as when one allows for plausible lags in PMT implementation. However, none of these methods can be considered to perform especially well. Prevailing methods do not reliably reach the poorest. The costs of each method in practice may then be decisive in the choice.

The following section describes the PMT method that we assess, while Section 3 describes the measures we use in assessing econometric targeting. Section 4 studies the basic version of PMT, while Section 5 turns to various extensions and revisions to that version. For stylized transfer programs, Section 6 compares the poverty impacts of econometric targeting to those of less methodologically sophisticated methods, including un-targeted (universal) transfers and simple demographic “scorecard” methods. Section 7 presents our results for (informationally-feasible) differentiated transfers, including optimal transfer schemes for poverty reduction with a given budget but limited information. Section 8 uses the panel surveys to introduce lags in implementation. Section 9 offers some cross-country comparisons of the performance of econometric targeting; here we ask how much the impacts of PMT on poverty for a given budget are explicable in terms of the alternative targeting measures and the predictive ability of the PMT regressions. Section 10 concludes.

# **2. Econometric targeting**

Quite generally, we can think of any PMT as some weighted function of a vector of covariates _xijt_ . The specific form of this function that has become popular and that we focus on uses householdconsumption regression coefficients as the weights. We can write the following empirical regression function for the consumption of household _i_ in country _j_ at date _t_ on a vector of covariates _xijt_ using a survey sample of size _N jt_ : The PMT is then based on: The most common method in practice for estimating  _jt_ and  _jt_ in (1) appears to be Ordinary Least Squares (OLS) using log consumption per capita as the dependent variable. As usual, OLS 8
chooses the parameter estimates to minimize the sum of squared errors with no difference in the weights attached to poor versus non-poor households (i.e., choosing  ˆ _jt_ and <sup>ˆ</sup> _jt_ to minimize   ˆ _ijt_ 2 for each _j, t_ ).

We also considered the option of using a binary indicator for whether a household’s actual consumption falls below the poverty line as the dependent variable (equal to one if a household is poor, and zero otherwise). We tried this for both OLS (giving a linear probability model) and a Probit. However, we found that targeting errors were substantially higher with a binary dependent variable in all cases. So we confine attention to the continuous dependent variable in the rest of this paper.

Another option to OLS in estimating equation (1) is to try to better tailor the estimator to the specific policy problem, in this case poverty reduction. Two ways of doing this can be suggested. The first is the quantile regression method of Koenker and Bassett (1978). This is more robust to outliers than OLS, and (importantly) the method can be tailored to the problem at hand in that the quantile can be set at the overall poverty rate.<sup>21</sup> In other words, we calibrate the PMT score to how that specific quantile in the distribution of log consumption, given the covariates, changes with those covariates. The second method entails placing higher weight on the squared errors of poorer people, giving “povertyweighted least-squares” (PLS). Among the various weighting schemes that might be used, we choose the method proposed by Mapa and Albis (2013), which weights equally all observations below the poverty line but gives zero weight to those above the line. In other words, we run the regression on poor households only. We extend this method by including households somewhat above the line. Once we have the PLS parameter estimates we calculate the revised PMT scores using the actual values of _xijt_ .

Any PMT method is likely to be quite constrained in practice in the choice of covariates. Practitioners are restricted to using _xijt_ variables that are considered easy to observe or verify in the field. There are feasibility constraints associated with the number and nature of the variables used in practice; administrative costs almost certainly rise with the number of variables. There are also incentive constraints, stemming from the scope for manipulation by local agents when there are many variables in the PMT (Niehaus et al. 2013).

The variables used in practice typically cover readily observed living conditions of the household, such as basic consumer durables or assets, demographic variables (size and composition) and attributes of the head.<sup>22</sup> Two important exclusions are notable. First, prices are rarely used and assets are

> 21 For example, this is one of the methods used by USAID (2011) in calibrating a PMT for Peru. This method is also discussed in Mills et al. (2015).

> 22 See, for example, the various studies in the compilation by Del Ninno and Mills (2015).

9
identified in broad categories; clearly, two households can each own a “fridge” but in one case it is 30 years old and works poorly while in the other case it is a fancy new model. Second, an important exclusion is that one cannot use fine geographic effects, such as at the level of the village, since one is constrained to estimating on a sample survey that will typically only cover a sample of villages (typically determined by the first stage of a two-stage sampling design). One does not know the geographic effect for the population, as required for implementing the PMT.<sup>23</sup> However, in one version we include community-level variables that go some way toward addressing this concern.

There is a degree of judgement required in selecting covariates. Here we consider various options, starting with a “Basic PMT” that seems to capture well the set of variables found in practice. We also consider “Extended PMT” methods that include variables that have extra explanatory power; while this provides a useful indication of the gains from more data, it is acknowledged that this version may not be easily implemented in the field. The <u>Statistical Addendum</u> provides descriptive statistics.

# **3** . **Measures of targeting and poverty**

An early strand of the literature formulated the targeting problem as that of choosing a schedule of transfer payments across types of households to minimize a measure of poverty subject to a budget constraint.<sup>24</sup> The subsequent literature has instead emphasized “targeting efficiency,” defined in terms of reducing targeting errors as defined below. Here we shall study both types of measures. We start with targeting measures.

**_Measures of targeting performance:_** The relevant counts corresponding to the joint distribution of _yijt_ and _y_ ˆ _ijt_ are shown in Table 1, which helps clarify our notation and some of the properties of our measures.

We focus on three main measures of targeting performance. The first is the Inclusion Error Rate ( _IER_ ), defined by the proportion of those identified as poor who are not. This can be written as:<sup>25</sup>

> 23 The same limitation is shared by small-area estimation methods (“poverty mapping”) as in Elbers et al. (2003).

> 24 The idea was developed in theoretical terms by Kanbur (1987) and the problem was formulated and solved numerically in Ravallion and Chao (1989) for the squared poverty gap index of Foster et al. (1984). Glewwe (1992) generalized this approach to allow for continuous variables.

> 25 The indicator function 1(.) takes the value unity when the condition in parentheses is true and zero otherwise.

10
Here the poverty line (in consumption space) is _z jt_ and the sample size is _N jt_ with households indexed _i_ =1,..., _N jt_ and _wijt_ denotes the appropriate sample weights (to deal with differences in household size and sample design);  _iN_  1 _jt w ijt_  1 .

Inclusion errors have received much attention in efforts to reduce the budgetary cost of social policies aiming to use transfer payments (in cash or kind) to reduce poverty. Inclusion errors imply a fiscal cost without any direct impact on poverty. For a uniform transfer paid to all those who are deemed to be poor, the _IER_ gives the share of the transfers going to the non-poor.<sup>26</sup> If everyone is deemed “poor,” so the transfer payment is universal, then _IER_ is simply one minus the poverty rate.

The _IER_ is often normalized by the poverty rate when the latter varies, which we will also do in some cases. The resulting measure has been used extensively—clearly more than any other targeting measure—in comparing the targeting performance of social programs across developing countries.<sup>27</sup> Critics of the focus on reducing inclusion errors have pointed to a number of issues, including measurement errors and the need for more inclusive policies in the interest of social coherence/stability.<sup>28</sup> The second measure is the Exclusion Error Rate ( _EER_ ), given by the proportion of the poor who are not identified as poor. (Sometimes the term “coverage rate” is used instead, which is simply one minus the _EER_ .) For a social program providing a uniform transfer payment to all—variously called a “basic income guarantee” or “citizenship income”—the _EER_ is of course zero, since everyone is covered. One might expect measures based on the _EER_ to be better predictors of a social program’s impact on poverty.<sup>29</sup> While that is intuitive—the more the poor are covered, the greater their expected gain—it does not necessarily hold as it will depend on the measure of poverty used, the distribution of coverage and the budget.<sup>30</sup> The Exclusion Error Rate can be written as:

> 26 This is what Weisbrod (1970) dubbed “vertical efficiency.”

> 27 This normalized share of transfers going to the poor was used by Coady et al. (2004a, b) to compare 85 programs across many countries.

> 28 Weisbrod (1970) raised concerns about focusing solely on reducing inclusion errors (vertical efficiency in his terms). On measurement errors in targeting see the discussion in Ravallion (2008).

> 29 See Ravallion (2009) who finds supportive evidence using data for a large cash transfer program in China.

> 30 For example, for the headcount index of poverty one focuses on whether there is exclusion at the poverty line.

11
To better understand the properties of these measures it helps to also think of _IER_ and _EER_ in probabilistic terms as: ˆ ˆ ˆ Also note that Pr( _y ijt_  _z jt_ )  Pr( _y ijt_  _z jt_ ) implies Pr( _yijt_  _z jt_ , _yijt_  _z jt_ )  Pr( _yijt_  _z jt_ , _yijt_  _z jt_ ). Then, from (5.1) and (5.2), we see that _IER jt_  _EER jt_ . Thus the two error rates are equalized (though ˆ not at zero unless all levels are predicted correctly) when the poverty rates are equal ( _H jt_  _H jt_ ), i.e., ˆ Pr( _y ijt_  _z jt_ )  Pr( _y ijt_  _z jt_ ) implies that _IER jt_  _EER jt_ . Intuitively, this is because each time a person who is in fact poor (based on the survey-based consumption) is incorrectly identified as non-poor, that person has to be replaced by someone who is in fact non-poor, so as to keep the total count of the poor constant. In other words, every exclusion error must generate an inclusion error once the poverty rate is identical when comparing actual and predicted values. Of course, we do not expect the actual and predicted poverty rates to be equal in general. However, in the methodology of PMT there is the option of fixing the poverty rate for predicted values according to the survey-based measure using actuals. For example, if the survey indicates that 20% of the population is poor then one targets the poorest 20% based on the PMT scores. When the poverty rate is fixed this way we will simply refer to the “Targeting <u>Error Rate” (</u> _TER_ ).

The third measure is the Normalized Targeting Differential ( _NTD_ ). In the context of a transfer program, the (ordinary) Targeting Differential ( _TD_ ) is defined as the mean transfer made to the poor less that made to the non-poor.<sup>31</sup> For a uniform transfer paid to all those who are deemed eligible, the _TD_ becomes the difference <mark>between the proportion of the poor who are predicted to be poor and the proportion of the non-poor who are predicted to be poor.</mark> (In the case of a specific antipoverty program it is the difference between the program’s coverage rate for the poor and that for the non-poor.) The _NTD_ divides this measure by the mean transfer receipt, to make the resulting measure more comparable

> 31 This measure was proposed by Ravallion (2000).  Also see Galasso and Ravallion (2005) and Ravallion (2009) on the properties of this measure and the discussions in Stifel and Alderman (2005) and Stoeffler et al. (2016).

12
across countries and programs. For a basic income guarantee, _NTD_ =0. When only the poor get help from the program and all of them are covered, the _NTD_ reaches its upper bound of 1; when only the non-poor get the program and all of them do, the _NTD_ is at its lower bound of -1. For a uniform transfer to all recipients in the amount  _jt_ we have: where _H jt_ is the headcount index of poverty (or poverty rate), defined as the proportion of the relevant population living in households with consumption per person below the poverty line, and _H_<sup>ˆ</sup> _jt_ is the headcount index obtained based on predicted consumptions.

Another concept of targeting errors occasionally found in the literature makes the distinction between “Type 1” ( _T_ 1) and “Type 2” ( _T_ 2) errors of targeting (borrowing the terms from statistics).<sup>32</sup> The former is defined as the proportion of the (ineligible) non-poor who are assigned a program targeted to the poor; thus, in this context:<sup>33</sup> ˆ When the poverty rate is fixed ( _H jt_  _H jt_  _H_ for all ( _j_ , _t_ )), _T_ 1 _jt_ is directly proportional to _EER jt_ ; specifically, _T_ 1 _jt_  _EER jt H_ /(1  _H_ ) . On the other hand, the Type 2 error rate is _T_ 2 _jt_  _EER jt_ . This yields another interpretation of the _NTD_ as (one minus) the aggregate of Type 1 and 2 errors: _NTD jt_  1  ( _T_ 1 _jt_  _T_ 2 _jt_ ) (8) When the poverty rate is fixed the _NTD_ is also a simple linear transform of the exclusion rate; i.e. _NTD jt_  1  _EER jt_ /(1  _H_ ) ). We will not use _T_ 1 and _T_ 2 given that they are so closely related to _EER_ and _NTD_ .

> 32 The designation of which is Type 1 and which Type 2 is arbitrary, and usage has varied. For example, Wodon (1997) and Ravallion (2009) define them our way but Grosh and Baker (1995) and Barrientos (2013) swap the two labels while Van Domelen (2007) has both usages. Appeals to statistics (whereby a Type 1 error is the incorrect rejection of a true null hypothesis while Type 2 is the failure to reject a false null) cannot resolve the matter since one can define the relevant null hypotheses consistently with either interpretation. (For our interpretation the hypothesis being tested is that a specific person is poor; the null is that she is not poor.) Readers are free to swap the labels and nothing substantive changes in our argument.

> 33 Note that  _iN_  1 _jt wijt_ (1 _y_ ˆ _ijt_  _z jt y ijt_  _z jt_ )  _H jt N jt_   _iN_  1 _jt wijt_ (1 _y_ ˆ _ijt_  _z jt y ijt_  _z jt_ )  _H jt N jt_ 1(  _EER jt_ ) .

13
**_Poverty measures:_** Given that poverty reduction is typically the primary (or even sole) objective of this class of policies it is appropriate that we also study impacts on poverty measures. The first measure we use is the popular headcount index, defined already. We denote the empirical cumulative distribution function (CDF) of consumption as _p_  _F jt_ ( _y_ )  [01,] , which gives the proportion of the population of country (or group) _j_ at date _t_ consuming less than the amount _y_  [ _y_ min , _y_ max ] . Then the headcount index can be written as:

The calculated poverty rate when based on the empirical distribution of _y_ ˆ _ijt_ ( _i_ =1,..., _N jt_ ) is _H_<sup>ˆ</sup> _jt_ .

While _H_ is (by far) the most popular measure in practice, its limitations are widely appreciated, notably that the measure does not reflect changes in living standards below the poverty line. We also consider two “higher-order” measures. The first is the poverty gap index, as given by the mean distance below the poverty line as a proportion of the line where the mean is taken over the whole population, counting those above the line as having zero gap.<sup>34</sup> The poverty gap index can be written as: We also make use of a distribution-sensitive measure, namely the Watts index proposed by Watts (1968) given by the mean proportionate poverty gap (counting the non-poor as having zero gap). This measure penalizes inequality among the poor, by putting higher weight on poorer people.<sup>35</sup> The Watts index can be written as:

**_Optimal transfers for a given budget:_** It appears to be a near-universal practice to provide a uniform transfer payment to all those who are identified as poor by the PMT. Transfer size may vary according to the number or age of children in the household (as in some conditional cash transfer schemes) but not with respect to predicted poverty levels based on the PMT. The popularity of such uniform transfers to those predicted to be poor can be thought of as a feasibility constraint on PMT; in the field it is likely to be difficult to make finely differentiated transfers. However, it is still of interest to see how much this constraint is limiting the impact on poverty.

> 34 The Statistical Addendum gives selected results for the squared poverty gap index of Foster et al. (1984).

> 35 The Watts index is known to have a number of other desirable theoretical properties, as detailed in Zheng (1993).

14
We explore the effect of this constraint in two ways. The first is to vary the size of transfers based on the PMT scores. The second is to reformulate the problem as one of optimizing the transfers as a function of the variables going into the PMT. Quite generally one can think of the informationallyfeasible transfers as a function of _m_ observed _x_ ’s. The policy maker only observes the covariates _x_ for each person; it is not known who is poor and who is not. However, the policy maker has a survey with much more information available for a sample. The problem is to choose the parameters of a score for assigning the real-world transfers based on the _x_ ’s, as given by: 0 0 _k_ (Here _xijt_  1 so that  _jt_ is the intercept—the transfer received by someone with _xijt_  0 for _k_ =1,..., _m_ .) In one version transfers are linear in the _x_ ’s, i.e.,  1. We call this the linear optimization. We also estimate a nonlinear version with  2, which introduces squared terms and interaction effects among the _x’_ s.<sup>36</sup> The choice of the score parameters  _kjt_ is made to minimize the Watts index in the sample survey data: The choice is constrained by the budget: We solve this problem numerically.<sup>37</sup> One start value we use for the optimization is the uniform 0 _k_ case obtained by setting  _jt_  _B jt_ and  _ijt_  0 for _k_ =1,..., _m._ Other start values are tested. When there are multiple local optima the solution for the parameters  _kjt_ that gives the lowest value of the poverty measure is chosen.

**_Data:_** The data for implementing these measures come from the World Bank’s well-known LSMS.<sup>38</sup> Table 2 lists the countries, years of survey, and numbers of households surveyed. In keeping

> 36 Glewwe (1992) recommended this in his formulation of the optimal targeting problem.

> 37 <mark>We use the “fmincon” program in Matlab.</mark> 38 The LSMS has designed and implemented household surveys across many countries since the 1980s. These are nationally representative multi-purpose surveys spanning a quite wide range of topics. Further information can be found at the <u>LSMS website. All surveys except for Ghana are LSMS-ISA surveys.</u>

15
with the bulk of the literature, our dependent variable is log total consumption per capita.<sup>39</sup> Consumption is measured in local currency units. Spatially deflated consumption values are available for all countries except Burkina Faso.<sup>40</sup> We use two poverty lines, corresponding to _H jt_  2.0 and 0.4 for all ( _j_ , _t_ ). The 40% figure coincides fairly closely with the overall poverty rate found for the Africa region using the World Bank’s international line.<sup>41</sup> The 20% rate allows us to focus on how well the method does at identifying those who can be considered extremely poor. When comparing the actual values and the PMT scores one can chose to either fix the poverty rate (at 0.2 or 0.4) or fix the poverty line in the  1  1 consumption space (i.e., fixing _z jt_  _F jt_ (02.) or _F jt_ (4.0) ). This choice makes a difference and practice varies so we present results for both options.

# **4. Results for Basic PMT**

The “Basic PMT” follows our reading of prevailing practice. Variables used include the type of toilet a household has; floor, wall and roofing material; type of fuel used for cooking; certain characteristics of the head, including gender, education and occupation; the household’s religion and demographic size and composition. All regressions have dummy variables for categories of household size, age of head, month of survey and region of residence; the latter is measured at an aggregate level (typically a state or province) for which the surveys can be considered representative. The <u>Statistical Addendum</u> gives the OLS regression results for the Basic PMT. The simple average R<sup>2</sup> is 0.53, with a range from 0.32 (for Ethiopia) to 0.64 (Burkina Faso); Table 3 provides summary statistics for the Basic PMT (as well as for the extended version discussed below). This explanatory power is typical of past studies.<sup>42</sup> We did not try to prune this model, by either _ad hoc_ or more systematic methods (such as stepwise regression). This would reduce the number of predictors but (of course) also reduce R<sup>2</sup> and probably increase targeting errors. However, we do consider stepwise regression as an option in Section 5 when using a much larger set of explanatory variables.

> 39 We also considered the option of using log consumption per equivalent single adult using the scales provided by the LSMS. We focus on the “per capita” case in this paper although the Addendum also gives regressions and key results using scales.

> 40 We use nominal consumption for Burkina Faso.

> 41 Using the World Bank’s international line of $1.90 a day at 2011 purchasing power parity, 43% of the population of SubSaharan Africa are found to be poor in 2013 (based on <u>PovcalNet).</u>

> 42 A seemingly representative set of studies is Grosh and Baker (1995) (R2 from 0.3 to 0.4), Ahmed and Bouis (2002) (R<sup>2</sup> =0.43), Narayan and Yoshida (2005) (R<sup>2</sup> =0.59), Sharif (2009) (R<sup>2</sup> =0.57), Stoeffler et al. (2015) (R<sup>2</sup> =0.62), Pop (2015) (R<sup>2</sup> =0.54) and Cnobloch et al. (2015) (R<sup>2</sup> =0.5 to 0.7). The simple average is 0.52.

16
Using the lower poverty line, fixed across the comparisons between distributions of _yijt_ and _y_ ˆ _ijt_ , the Basic PMT substantially under-predicts the poverty rate in all countries. This is not unexpected given the properties of OLS, as noted in the Introduction. The extent of the problem can be seen in Table 4. While the poverty rate based on the data is 20%, that based on the predicted values ranges from 0 (for Mali) to 12% (Nigeria); the simple (population weighted) average is 8%. This improves considerably when one switches to the higher poverty line ( _F jt_  1(4.0) ), for which the average _H_<sup>ˆ</sup> _jt_ is 37% with a range from 29% to 44%.

**_OLS results for Basic PMT:_** We turn now to the targeting measures. Table 5 gives the results. Let us focus first on the fixed poverty line case with _H_ =0.2. On average, the rate of inclusion errors implies that 48% of those identified as poor by the Basic PMT method are in fact non-poor, i.e., just over half of those identified as poor using Basic PMT are in the poorest 20% when measured using the survey-based consumption.  For a poverty rate of 20% and a fixed line, the PMT method has nearly halved the rate of inclusion errors that would be obtained with a uniform transfer payment. However, this has come at the expense of exclusion. The average exclusion error is sizeable, with 81% of those who are in the poorest 20% in terms of survey-based consumption being incorrectly identified as nonpoor by the PMT method.

There is considerable variation across countries, with _IER_ ranging from 33% to 100%, and _EER_ from 55% to 100%. In the country with the lowest coverage rate of the poor implied by PMT, Mali, all poor families are incorrectly identified as non-poor. Unsurprisingly, we also find a tendency for the PMT to do better at correctly identifying poor households when the R<sup>2</sup> in the PMT regression is higher (comparing Tables 5 and 3). However, the proportion of households correctly included is less than half of those who are poor under a poverty line corresponding to _H=_ 0.2.

Both inclusion and exclusion errors are lower for _H=_ 0.4. Taking a (population weighted) average of our estimates of _IER_ and _EER_ for _H=_ 0.4, we find that 36% of those who are poor are excluded on average, while 31% of those who are deemed poor are actually not poor. So we again find that econometric targeting halves the inclusion error rate of 0.6 that would be implied by uniform transfers. There is also less spread in the values across countries with _IER_ ranging from 25% to 40% and _EER_ from 24% to 56%.

The finding that the errors tend to be higher using the lower poverty line again suggests that econometric targeting may have difficulty in identifying those who are very poor. This is supported by Figure 1, which plots actual consumption against predicted consumption. The lines represent the poverty 17
lines at _H=_ 0.2. The bottom left quadrant represents households that are correctly identified as poor by the Basic PMT. The top left quadrant is the inclusion error, and the bottom right quadrant is exclusion. In one case, Mali, there are no data points in the bottom left quadrant; only one household is (incorrectly) predicted to be poor (thus giving the result that _IER_ = _EER_ =1.0 from Table 5). While Mali is exceptional in this respect, the point remains that PMT is missing many of the poorest households in all countries. Figure 2 gives the implied residuals. As expected, these tend to be lower (more negative) for poor people, but it is notable just how much the PMT regression is over-estimating the living standards of the poorest. For the poorest 20% in terms of actual consumption, the mean residual ranges from -0.73 to -0.37, implying that the PMT regressions yield predicted consumptions for the poor between 50% and 100% above their actual consumption.<sup>43</sup> (The fact that consumptions of the poor are overestimated by the PMT regressions at the poverty line, as is evident in Figure 2, echoes our finding above that Basic PMT underestimates the poverty rate.) Looking at Figures 1 and 2, one can understand why many of those accepted or rejected might be tempted to believe that econometric targeting is something like a random lottery, or maybe even divine intervention (with reference to the quote from Adato and Roopnaraine, 2004, in the Introduction). At a given level of consumption, the predicted values generated from the PMT can vary considerably – see, for example, Ethiopia. A more encouraging finding is that households who are incorrectly included do not seem to be among the wealthiest households, that is, many of these households have actual consumption values that are relatively close to the poverty line.

So far we have focused on PMT using a fixed poverty line in consumption space. As we have seen, this tends to predict far fewer households as poor than the actual poverty rate, particularly when the poverty line corresponds to _H=_ 0.2 (Table 4). Table 5 also provides the results for the case where we instead fix the poverty rate. For example, we calculate the mean targeting error for the poorest 20% in the distribution of predicted consumption to be 51%, falling to 32% using _H=_ 0.4. Note that fixing the poverty rate instead of the poverty line will typically increase the number of predicted poor households thus resulting in higher _IER_ and lower _EER_ .

As noted in the introduction, “targeting errors” may reflect to some extent time-varying measurement errors in the cross-sectional data. For those countries with panel data we can address this problem by assessing targeting performance using the time-mean consumption instead of current

> 43 <mark>The mean residuals for the poorest 20% by country are -0.371 (Burkina Faso), -0.711 (Ethiopia), -0.497 (Ghana), -0.564 (Malawi), -0.725 (Mali), -0.402 (Niger), -0.401 (Nigeria), -0.555 (Tanzania), and -0.543 (Uganda).</mark>

18
consumption. This will reduce, though probably not eliminate, any bias due to time-varying measurement errors. The lower panel of Table 5 gives the results. In the majority of cases, the measures of targeting performance improve, although this is less evident for exclusion errors than inclusion errors when using a fixed line. (Ethiopia accounts for about half of the exceptions.) Overall, the results are broadly consistent with the view that measurement errors are playing some role, but the panel data do not overturn our main conclusions about PMT. (Section 8 returns to the panel data, as a means of allowing for implementation lags in PMT.)

**_Results using “poverty-focused” estimation methods:_** As noted, the OLS method used for the Basic PMT chooses the parameter estimates to minimize the unweighted sum of squared errors. Recall that we consider two “poverty-focused” options to OLS. The first is a quantile regression using the poverty rate as the quantile. For this estimator, Table 6 gives the analogous results to Table 5. This method allows a substantial reduction in the exclusion error rate using a fixed poverty line. This comes at the cost of higher inclusion errors, especially when using the lower poverty line (anchored to a quantile of 0.2). Targeting errors are similar to those for Basic PMT when using a fixed poverty rate instead.

Table 7 reports the targeting errors using our PLS method when a fixed poverty line is used to classify predicted poor households, as well as the results when a fixed poverty rate is used instead. (The <u>Statistical Addendum</u> gives the coefficients for our PLS regression with the Basic PMT variables.) In both cases, the weighted regressions correctly include almost all poor households. However, as with the poverty-quantile regression, inclusion errors are also high. The PMT using PLS regression is better at covering the poor but predicts that too many households are poor.

An alternative is to include some households who are above the poverty line in the PLS regression. We did this by also including in the sample all households at or below the poverty line, plus the next 20% of households, as ranked by their consumption. For example, at the poverty line for _H=_ 0.2, the bottom 40% of households is used in the regression. For _H=_ 0.4, the bottom 60% is used. Table 8 provides the inclusion and exclusion errors for this version. There is a decrease in the _IER_ relative to Table 7, but with higher _EER_ (though still lower than for the OLS).

**_Targeting subgroups within a national population:_** So far we have used a basic PMT calibrated to national populations. However, when using PMT to target programs meant for a specific group it will typically be better to calibrate the PMT to that group. We tested this by estimating the PMT model on the samples restricted to two groups of households, namely those containing elderly and/or disabled

19
members, and those households with children under 5. Next we compared the targeting measures based on the predicted values for each group with those predicted for the same household subgroups using a nationally calibrated PMT. We found that there is a modest improvement in targeting performance when using the sub-group-specific PMT. For example, to focus on the elderly and disabled subgroup case: for a fixed poverty rate of 0.2, average targeting errors go down from 0.50 to 0.47. Both inclusion and exclusion errors also fall using a fixed line, from 0.40 to 0.37 and 0.74 to 0.70, respectively. But again the gains are small (see Statistical Addendum for full results).

# **5. Extended PMT**

We now test an extended specification with far more data, including the household’s water source; more detailed information about housing materials; the number of household members per room; whether the household has a separate room for cooking; whether the household has electricity; household assets; and more details on the characteristics of the household. Regression results for the extended PMT are shown in the <u>Addendum. The values of R</u><sup>2</sup> are higher but in most cases the gains are relatively small; although the number of explanatory variables has almost doubled there are clearly some strong correlations between the extra variables and those in the core set used for the Basic PMT.

As expected, the Extended PMT does better than Basic PMT with respect to targeting errors (Table 9). However, the improvement would have to be judged as modest (comparing Tables 9 and 5). For example, many more than half of the poorest 20% are still misidentified as non-poor. Table 4 also gives the average proportion of households that are predicted as poor using the poverty line methodology under the extended PMT method. At _H=_ 0.2, we see a slight improvement over the Basic PMT, with 11% of the sample predicted to be poor using the poverty line corresponding to _H_ =0.2 (as compared to 8% using Basic PMT). The results are more similar between Basic PMT and Extended PMT for a line corresponding to _H_ =0.4.

We also reran the poverty-weighted PMT regressions as in Section 3 for the extended PMT model. The <u>Addendum gives the regression results when the extended PMT model is estimated on the</u> bottom 20<sup>th</sup> , 40<sup>th</sup> and 60<sup>th</sup> percentiles. The Addendum also gives the targeting errors for the poverty line and poverty rate method when the PMT is fitted using poor households only as well as the results when the poor plus the next 20 percent of the distribution are used. The key findings for Basic PMT using the poverty-weighted regression (Table 6) were confirmed using the Extended PMT.

20
The field implementation of a PMT formula with many variables is expensive and difficult, so some practitioners have opted for stepwise regression to obtain a more parsimonious PMT. We tested a backwards stepwise regression on the extended model to identify the key variables in the PMT. We used a cut-off of _p_ = 0.01. The targeting errors for the more parsimonious regressions are given in the <u>Addendum. We see a modest increase in the targeting errors, which are now back to approximately the</u> same values we found for Basic PMT.

A further methodological change we considered is to include variables that are not as readily available as those in our Extended PMT regressions, but are likely to have extra explanatory power. In one case we used extra data on households’ food security as well as on any shocks the household may have experienced. (Note that these variables are only available for four countries.) We augment the extended PMT model with these food security and shock variables. (The Addendum lists the variables, their means and the regression results for this model.) The R<sup>2</sup> increases slightly for all countries (Table 3). However, this version produced negligible improvement in targeting (Addendum). In another variation on the Extended PMT we included a range of community-level variables; again this was not possible for all countries. And (again) there was only a modest reduction in targeting errors, as can be seen in the Addendum.

We also tried other versions of PMT that might be of interest. In one case we used quantile regression at the median (in both the Basic and Extended PMTs). In another we used log consumption per equivalent single adult as the dependent variable. The <u>Addendum gives the results. There was little</u> improvement in the targeting performance of the PMT.

So far we have focused solely on the inclusion and exclusion rates as the measures of targeting performance. These appear to be the most popular measures in the literature, though others have been proposed and used in some studies. Probably the most promising example of the latter when the policy objective is poverty reduction is the targeting differential (Ravallion, 2000, 2009). Recall that the normalized _TD_ is in the range [-1, 1], with zero corresponding to a uniform (un-targeted) transfer.

Table 10 gives summary statistics on the normalized targeting differential using both the Basic and Extended PMTs. The mean _NTD_ for Basic PMT is 0.21, meaning that if program participation was based on the PMT scores the participation rate for the poor would be 21% points higher than that for the non-poor. Using Basic PMT, in four of the five countries with panel data, the _NTD_ is higher using timemean consumption; this rises to five out of five using the Extended PMT. Returning to the cross-section surveys, the poverty-quantile method yields the highest _NTD_ , at around 0.49 on average for Basic PMT, 21
rising to 0.53 for the Extended PMT. For all nine countries, the poverty-quantile regression method comes out best. The poverty-weighted method does almost as well provided that the 20% of households above the line are included.

# **6. Poverty impacts of stylized transfer schemes using various targeting methods**

PMT is typically used to identify eligible recipients of a specific transfer scheme with the aim of reducing poverty. So we now study the poverty impacts of stylized transfers that are allocated according to various PMT specifications and selected counterfactuals. Our comparisons are all budget neutral with the budget for each stylized scheme set at the aggregate poverty gap ( _PG jtz jt N jt_ ) for that country.<sup>44</sup> We assume a poverty line corresponding to _H_ =0.2. (The Addendum gives the average transfer amounts by country.) If the PMT worked perfectly— so that predicted consumption equaled actual consumption—then the transfers differentiated to exactly fill the poverty gaps would eliminate poverty. In this section we confine attention to uniform transfers among those deemed eligible, as is common in practice; in the next section we consider more finely differentiated transfers.

A natural benchmark is a universal (“basic income”) scheme in which every person (whatever their characteristics) receives the same transfer payment. We then calculate the impacts on poverty of transfers using the various versions of PMT discussed above. We measure the impact of a uniform transfer per capita given to all households who are predicted to be below the line according to the PMT. The total transfer amount for a given country (as given by the country’s aggregate poverty gap) is divided by the total number of individuals who reside in designated poor households, and distributed to households according to their size. (For example, if a poor household has two members, the transfer will be two times the per capita amount.) We also consider counterfactual policies that use categorical targeting rather than PMT. These policies make uniform transfers within a specified category of people, as defined by a “poverty scorecard.” Here we consider an especially simple form of demographic scorecard.<sup>45</sup> The first category is the set of persons 65 years or older. The second is any person who is a (female) widow, disabled (where disabled is defined as an illness or condition that significantly impairs a person over the age of

> 44 We make no claim about this being a realistic budget but we do consider it to be an obvious reference point.

> 45 Indeed, our method is even simpler than the “Simple Poverty Scorecard” developed by Schreiner (2010, 2015) and used for the Progress out of Poverty Index.

22
14 and their ability to work or study), or orphaned (defined as any child 14 or younger whose parents have both died or whose whereabouts are unknown). The third is a combination of the first two: a transfer to the elderly, widowed, disabled or orphaned. Note that if a person fits two categories, the score and (hence) transfer is doubled. The fourth transfer is a payment to households with children – whereby up to three children are each alloted a transfer. Finally, the last scheme combines all previous schemes, where children, the elderly, widowed, disabled or orphaned are eligible. (Recall that all stylized schemes considered have the same aggregate budget.) Table 11 shows the implied headcount index for each case. (Recall that the baseline headcount index across all countries is 20%.) Most methods bring the poverty rate down to around 16%, well short of eliminating poverty; indeed, more than three-quarters of the poor remain poor. On average, Basic PMT does only slightly better than the universal basic income with the same budget, and Basic PMT does not do as well as the universal transfer in one third of the countries. Using the time-mean consumptions for the countries with panel data makes little difference on average. The quantile regression method does noticeably better on average, bringing the poverty rate down to one percentage point below the level attainable with the Basic PMT. Extended PMT does slightly better. However, it is notable how well categorical targeting does in many cases. On average, categorical targeting to households with elderly, widows, disabled and children does as well as Basic PMT. Nevertheless, categorical targeting never does as well as the poverty quantile regression method, which typically has the greatest impact on poverty. While categorical targeting does not have quite as much impact on poverty as the Basic PMT, it clearly comes close and is simpler and more transparent.

Tables 12 and 13 give the corresponding results for the poverty gap index and Watts index respectively; the pre-transfer poverty measures are shown in the first row. Aggregating across countries, the Basic PMT methods reduce the poverty gap by around 27% and the Watts index by 28%. As for the headcount index, the Extended PMT gives a larger reduction, namely 35% and 39% respectively. Simply giving a uniform transfer based on household size does as well as Basic PMT on average for both PG and the Watts index.

# **7. Allowing differentiated transfers**

So far we have focused on the standard practice of giving the same transfer payment to all those predicted to be poor using PMT. While this is the most relevant case in practice, differentiating the transfers could be expected to work better if the predicted poverty gaps are quite accurate. However, we 23
have already seen that this is not the case—that PMT works poorly in predicting the levels of living of the poorest. So it is unclear on _a priori_ grounds whether differentiated transfers will have larger impacts on poverty.

How much better can PMT do using the same information if the transfers are differentiated, with more going to those who appear to be poorer? To put the question another way: how much does the constraint of relying on uniform transfers to the “predicted poor” limit the effectiveness of PMT? We address these questions in two ways. First, we simply fill the predicted poverty gaps, scaling up (or down) to attain the same budget. That is, each household predicted as poor receives the difference between the poverty line and its predicted consumption value, scaled such that the sum of all transfers equals the aggregate poverty gap. “PMT Gap” refers to this first method.

The allocation of transfers obtained this way need not be optimal in the sense of minimizing an agreed poverty index for a given budget. Following Ravallion and Chao (1989) and Glewwe (1992), we also devised a program for calculating the optimal allocation based on the set of PMT covariates. We chose the Watts index as the objective function given its desirable properties as a poverty measure (Section 2), which also provides suitable curvature to the objective function. Multiple solutions were common but we also found that the objective function tended to be quite flat in the sub-set of the parameter space corresponding to the various solutions found. Indeed, for all nine countries the minimum value of the Watts index was the same up to two decimal places whatever start value we used (though the parameter estimates themselves often differed for a given country).

Table 14 gives the results for the Watts index for _H_ =0.2 using differentiated transfers that are determined by both the PMT gaps and optimization. (The <u>Addendum gives those for the headcount</u> index, though note that the solutions are only optimal for the Watts index.) Overall, filling the predicted gaps does little to reduce the poverty measure. As expected, the non-linear specification in the optimization routine (  2 in equation 12) does better than the linear one in reducing poverty, and the nonlinear version does as well on average as the PMT.

# **8. Allowing for lags in PMT implementation**

Lags in the implementation of a PMT are almost certainly universal. It takes some time to set up the data and the administrative apparatus for implementation. Yet there is undoubtedly some “churning” 24
in living standards over time, even when using consumption as the welfare indicator. So the lags in implementation have bearing on the performance of PMT in reducing current poverty.

We have panel data for a subset of our study countries, namely Ethiopia, Malawi, Nigeria, Tanzania and Uganda. By exploiting the panel data, we can introduce a 1 to 2 year lag in the implementation of PMT. The precise lags are one year for Uganda, and two years for the other countries.<sup>46</sup> In other words, we develop the PMT on the Round 1 survey data and then apply it in Round 2. If anything, our lags appear to be less than found in practice.<sup>47</sup> We consider two types of lags. In the first (Method 1), we take the regression parameters from Round 1, but use the covariates from the Round 2 data. Here there is no lag in the observations of the covariates; the lag is only due to the need to estimate the PMT scores. In the second (Method 2), we simply use the PMT score from Round 1, which we then compare to the survey data on consumptions in Round 2. The lag then applies to all aspects of the PMT method (both parameter estimates and covariate values).

The targeting errors obtained using Methods 1 and 2 are found in Tables 15 and 16, respectively. Comparing the results in Table 15 (top panel) with Table 5 we see that allowing for lags increases the targeting errors on average.<sup>48</sup> For the lower line, we now find that, on average, about half of those predicted to be poor are not in fact poor based on the survey data (a mean IER of 0.553, as compared to 0.481 from Table 5). Exclusion errors are also affected, though these errors rose less markedly. Using the Extended PMT we also find a substantial increase in the targeting errors, especially for inclusion, when we allow for lags using Method 1. A similar pattern is found for Method 2.

In Table 17 we give the targeting differentials. Allowing for lags, we find mixed results. Ethiopia and Nigeria have better pro-poor targeting with lags, while Malawi, Tanzania and Uganda see increases in targeting of the non-poor. Malawi in particular has negative TD’s, indicating that the correction for lags now means that a uniform (un-targeted) policy would do better.

The post-transfer poverty rates allowing for lags are provided in Table 18. PMT still brings the poverty measures down, but by about two percentage points less when allowing for lags. For example, allowing for lags achieves an average post-transfer headcount index of 19% instead of 17%. The stylized categorical targeting schemes now attain similar or somewhat lower post-transfer poverty rates. On

> 46 <mark>The survey years are as follows: Ethiopia 2011/12 and 2013/14; Malawi 2010/11 and 2013; Nigeria 2010/11 and 2012/13; Tanzania 2010/11 and 2012/13; Uganda 2010/11 and 2011/12.</mark>

> 47 For example, recall that the mean lag between the survey year and the release date of the Progress out of Poverty Index is 3.9 years (Introduction).

> 48 Switching to the panel samples changes the targeting measures somewhat but the following observations still hold.

25
average, the simple demographic scorecards bring the poverty rate down by an extra one and in some cases two percentage points once one allows for plausible lags in PMT implementation. The <u>Addendum</u> gives results for other poverty measures, which follow a similar pattern.

# **9. Predictors of poverty impacts**

We can bring a number of the results from previous sections together to quantify the relative importance of targeting errors and estimator fit to the impacts of PMT on poverty. With only nine country observations, cross-country comparisons should be treated with caution. Nonetheless, some strong patterns emerge even with so few degrees of freedom.

As noted in the Introduction, inclusion errors have tended to receive more attention in the policy community although it has been argued by some that exclusion errors may well be more important to the impacts on poverty. There is a clear pattern in our results whereby the exclusion error rate is generally a better predictor of the poverty impact of PMT than the inclusion error rate. This can be seen in Table 19 which gives regressions of the final “post-PMT” poverty measure on the two error rates (with controls for the initial poverty measure, which is of course a constant in the case of the headcount index). In all instances, the _EER_ is the stronger predictor of impacts on poverty, and it is a strong (statistically significant) predictor in all but one case (the exception being for the method of filling the “PMT gap” described in section 7).

Another plausible predictor of both the targeting performance of PMT and its impact on poverty is the R<sup>2</sup> of the original regression used to calibrate the PMT. Indeed, it appears that this is often the main parameter that practitioners focus on in PMT applications. Table 20 gives the regressions across countries for both Basic and Extended PMT. The value of R<sup>2</sup> has only weak predictive power for the main measure used in practice, the headcount index. The R<sup>2</sup> does emerge as a strong predictor of the poverty impacts of standard PMT for the higher-order poverty measures ( _PG_ and the Watts index). This is not the case when we allow for differentiated transfers using either of the methods from the last section, although (as noted) the practical relevance of differentiated transfers is a moot point. For the more relevant case of uniform transfers across those predicted to be poor, R<sup>2</sup> can only be considered a useful predictor of impacts on poverty for the higher-order measures.

# **10. Conclusions**

26
Highly imperfect information and limited administrative capabilities create challenges for implementing effective antipoverty programs in most developing-countries. Practitioners have often turned to some form of proxy means test. While these methods have an _a priori_ appeal, users should have realistic expectations of what the methods can deliver.

Our results point to both strengths and weaknesses of standard econometric targeting methods. While these methods can substantially reduce inclusion errors in an antipoverty program—in most cases studied here the inclusion error rate can be at least halved—this comes at the cost of substantial exclusion errors when judged against the data on household consumption used to calibrate the test scores. Standard methods found in practice may look fine when the sole aim is to reduce inclusion errors—to prevent non-poor people receiving benefits when judged against a fixed poverty line. However, if poverty-reduction relative to a fixed line is the objective then policy makers with a given budget should be more worried about exclusion errors than inclusion errors. When attention switches to the problem of assuring broad coverage of the poor (reducing exclusion errors), better methods can be proposed, which give higher weight to performance in predicting the living standards of poor people. The method we find to generally perform best from the point of view of reducing exclusion errors is a “poverty-quantile regression.” This method generates more inclusion errors than prevailing PMT methods, though still less than un-targeted transfers.

When judged in terms of the impact on poverty for a given budget (set equal to the aggregate poverty gap), we find that what appears to be the most widely-used form of PMT in practice does only slightly better on average than a universal basic income, in which everyone gets the same transfer, whatever their characteristics. One can achieve somewhat larger impacts on poverty using other PMT methods considered here, with either a richer data set or using the poverty-quantile regression method. However, even under seemingly ideal conditions, the “high-tech” solutions to the targeting problem with imperfect information do not do much better than age-old methods using state-contingent transfers or even simpler basic income schemes. We find that an especially simple demographic “scorecard” method can do almost as well as econometric targeting in terms of the impacts on poverty. Indeed, on allowing for likely lags in implementing PMT, the simpler categorical targeting methods perform better on average in bringing down the current poverty rate. This conclusion would undoubtedly be strengthened once the full costs of fine targeting are taken into account.

We were surprised that econometric targeting only allowed such small (or even negative) gains in reaching poor people compared to simpler methods. For practitioners deciding on targeting methods 27
going forward, we suspect that other criteria besides targeting accuracy should take precedence in the choice, such as the specifics of the poverty profile, administrative capabilities and cost, the need for transparency, and the scope for fine targeting to undermine political support for social policies.

Not everything relevant to performance on the ground can be quantified using the kind of “laboratory experiment” done here. Field implementation may introduce idiosyncratic mistakes or allow ad hoc efforts to improve beneficiary selection on the ground. Nor have we allowed for the pursuit of other objectives besides poverty reduction such as social cohesion or administrative capacity building.

That said, looking at our findings as a whole, it would be fair to say that none of the examined methods performs particularly well when one is striving to reduce poverty. When the budget required for a set of transfer payments that would eliminate poverty (ignoring behavioral responses) is allocated by any of these methods, about three-quarters of the original (pre-intervention) count of poor people remain poor.

28