metadata: authors: 0: author: Ravallion,Martin
count: World
docna: 0: docna: How relevant is targeting to the success of
            an antipoverty program ?
docty: Policy Research Working Paper
subtopic: Social Analysis,Quality of Life & Leisure,National Governance,Government Policies,Disability,Public Sector Economics,Poverty Impact Evaluation,Poverty Monitoring & Analysis,Services & Transfers to Poor,Youth and Governance,Economic Assistance,Access of Poor to Social Services,Public Financial Management,Public Finance Decentralization and
            Poverty Reduction,Inequality,Poverty Assessment,Poverty Diagnostics,Poverty Lines,Small Area Estimation Poverty Mapping
teratopic: Health, Nutrition and Population,Poverty Reduction
abstracts: cdata!: Policy-oriented discussions often assume
            that "better targeting" implies larger impacts on
            poverty or more cost-effective interventions. The literature
            on the economics of targeting warns against that assumption,
            but evidence has been scarce. The paper begins with a
            critical review of the strengths and weaknesses of the
            targeting measures found in practice. It then exploits an
            unusually large micro data set for China to estimate
            aggregate and local-level poverty impacts of the
            country's main urban antipoverty program. Standard
            measures of targeting are found to be uninformative, or even
            deceptive, about impacts on poverty and cost-effectiveness
            in reducing poverty. In program design and evaluation, it
            would be better to focus directly on the program's
            outcomes for poor people than to rely on prevailing measures
            of targeting.
repnme: repnme: How relevant is targeting to the success of
            an antipoverty program ?
display_title: How relevant is targeting to the
            success of an antipoverty program
listing_relative_url: /research/2007/11/8665762/relevant-targeting-success-antipoverty-program
topicv3: Political Economy of Reform m1326263 836,Public Finance Management m1326316 716,Poverty m1326687 695,Poverty Measurement and Analysis m1326941 707,Social Assistance m1327201 1466
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Health, Nutrition and Population,Poverty Reduction
no_of_pages: 34
author_keys: 0: author_key: 99002
model_extractions: WPS4385 Policy ReseaRch WoRking PaPeR 4385 How Relevant is Targeting to the Success of an Antipoverty Program?

_Martin Ravallion_ The World Bank Office of the Director Development Research Group November 2007
Policy ReseaRch WoRking PaPeR 4385

# **Abstract**

Policy-oriented discussions often assume that “better targeting” implies larger impacts on poverty or more costeffective interventions. The literature on the economics of targeting warns against that assumption, but evidence has been scarce. The paper begins with a critical review of the strengths and weaknesses of the targeting measures found in practice. It then exploits an unusually large micro data set for China to estimate aggregate and local-level poverty impacts of the country’s main urban antipoverty program. Standard measures of targeting are found to be uninformative, or even deceptive, about impacts on poverty and cost-effectiveness in reducing poverty. In program design and evaluation, it would be better to focus directly on the program’s outcomes for poor people than to rely on prevailing measures of targeting.

This paper—a product of the Director's office, Development Research Group—is part of a larger effort in the department to assess the reliability of the methods used in practice for guiding policy making. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at mravallion@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# How Relevant is Targeting to the Success of an Antipoverty Program?

## Martin Ravallion<sup>1</sup>

_Development Research Group, World Bank, 181 H Street NW, Washington DC, USA_

**Keywords:** Poverty, cash transfers, errors of targeting, China

**JEL:** I32, I38, O15

> 1 For helpful discussions on this topic and help with the data used here the author is grateful to Shaohua Chen, Jean-Yves Duclos, Emanuela Galasso, Garance Genicot, Margaret Grosh, Pilar Garcia Martinez, Philip O’Keefe, Adam Wagstaff, Dominique van de Walle, Youjuan Wang, Xiaoqing Yu, and seminar participants at Beijing University and at the Ministry of Finance-World Bank Roundtable on Public Finance, 2006. The findings, interpretations and conclusions of this paper are those of the author, and should not be attributed to the World Bank.
Various measures of the “targeting performance” of antipoverty programs have been widely used to inform policy discussions. These measures are typically interpreted by both analysts and policy makers as indicators of a program’s performance in “...directing benefits toward poorer members of the population” (Coady, Grosh and Hoddinott, 2004a, p.81). Comparisons of such measures across different programs have informed public choices on which programs should be scaled up and which should be dropped.<sup>2</sup> As in any situation in which measurement is used to inform policy, “the indicators need to be related to the overall policy problem, with an explicit formulation of the objective and constraints” (Atkinson, 1995, p.31). It is widely agreed that the objective of this class of public programs is to reduce poverty, subject to the relevant constraints, including those related to the information available and the behavior of relevant agents, as well as resources. Better targeting is not seen as desirable in its own right, but rather as an instrument for reducing poverty.

Do the measures of targeting used in policy discussions provide useful indicators for this policy problem? The most widely used measures quantify some aspect of how well a given program concentrates its benefits on the poor, which is essentially what “targeting” has come to mean. An example is the share of transfers going to the poor. Cornia and Stewart (1995) have been influential in arguing that measurement practices and policy discussions have put too high a weight on avoiding one type of error—the “Type 1 error” of having (ineligible) non-poor participants—relative to the “Type 2 error” of incomplete coverage of the poor.<sup>3</sup> Cornia and Stewart did not present data linking these aspects of targeting performance to poverty outcomes (though they do point to this as an important direction for further research). However, the literature warns us against assuming that better targeting, as assessed by standard measures, will necessarily enhance a program’s total impact on poverty.<sup>4</sup> A number of factors cloud the relationship between targeting performance and total impact on poverty, including

> 2 Early empirical studies by Mateus (1983) and Grosh (1992) were influential in arguing the case for finer targeting.  The meta studies of Grosh (1994) and Coady et al. (2004a,b)  have provided the most comprehensive comparative data on program performance based on targeting measures.

> 3 The distinction between these two errors goes back to Weisbrod (1970) who called them “vertical-” and “horizontal target efficiency.” Cornia and Stewart (1995) used the terms “E-mistakes” and “F-mistakes;” Smolensky et al., (1995) called them “errors of inclusion” and “errors of exclusion.” The literature on social welfare policy in developed countries has also suggested that coverage of the poor is given little weight by standard measures of targeting.  See, for example, the results of Duclos (1995) on the implications of incomplete take up of Britain’s welfare benefits for measures of targeting.  However, the relationship between these problems and overall impacts on poverty has received little attention.

> 4 For an overview of the arguments and evidence see van de Walle (1998).

2
aspects of program design, implementation and the context in which a program operates. Incentive issues have been a theme of one strand of the literature, pointing to the possibility that fine targeting will impose high marginal tax rates on recipients, possibly creating poverty traps.<sup>5</sup> The literature has also warned that fine targeting can undermine political support for an antipoverty program; concentrating gains on the poor may induce a lower overall transfer to the poor, with benefits spread too thin, or covering too few people.<sup>6</sup> It is also unclear how useful these measures are as indicators of cost-effectiveness, as an input to scaling-up decisions. Here it is not the total impact on poverty that one is focusing on, but rather the impact per unit of the resources devoted to a given program. (The total impact then depends on the allocation of resources across programs, weighted by their cost-effectiveness ratios.) Intuitively, the impact on poverty will depend on both the share of transfers going to the poor and the total transfer. Plainly a large uniform transfer (received by everyone, whether poor or not) can have more impact on poverty than a small well-targeted transfer. But will the latter type of program, with low leakage to the non-poor, necessarily be more cost-effective? The answer is far from obvious on _a priori_ grounds. The factors noted above that cloud the relationship between targeting performance and a program’s total impact on poverty will not, in general, vanish when total impact is normalized by total spending.

For example, finer targeting typically entails administrative costs, which are debits against the total budget in determining the government’s total transfer payment. Then the share of transfers going to the poor does not even identify the transfer to the poor per unit public spending. Less obviously, but no less importantly, targeting can generate “hidden” costs to participants, notably when there are conditionalities, such as work requirements, behavioral condionalities or sources of social stigma. Given the costs of targeting, it is not difficult to imagine cases in which the better targeted program (with the higher share of transfers going to the poor) is less cost effective in reducing poverty, and the literature already contains examples.<sup>7</sup> In short, avoiding leakage to the non-poor can reduce the amount actually going to the poor, with theoretically ambiguous implications for poverty and cost-effectiveness in fighting poverty.

> 5 Besley and Kanbur (1993) pointed to this problem and other issues raised by targeting. Also see Smolensky et al. (1995). Kanbur et al. (1995) study the incentive issues in fine targeting, including characterizing an optimal scheme for poverty reduction, taking account of labor supply responses.

> 6 For theoretical analyses see De Donder and Hindriks (1998) and Gelbach and Pritchett (2000).

> 7 Ravallion and Datt (1995) and Murgai and Ravallion (2005) provide examples for workfare programs in India.

3
Whether better targeting, as measured in practice, implies a greater impact on poverty, or a more cost-effective intervention, is ultimately an empirical question. Yet, beyond a few suggestive examples, we really know rather little about how well these popular targeting measures perform in practice.

This paper tries to help fill this gap in knowledge using a detailed case study of one program, namely China’s “Minimum Livelihood Guarantee Scheme,” popularly known in China as _Di Bao_ (DB). This has been the government’s main response to the new challenges of social protection in urban areas. A number of factors—the decentralized nature of the program, its scale and the availability of a large data set representative at local level—combine to make this an unusual opportunity to put targeting measures to the test. The program’s targeting performance and impacts on poverty are estimated under standard assumptions across each of the 35 major municipalities of China. The most popular targeting measures found in policy-oriented discussions are thus tested as indicators of program performance in reducing poverty.

# **Measures**

In principle, one can measure “targeting performance” by a program’s impact on poverty relative to an explicit counterfactual, such as an un-targeted allocation of the same budget (as in Ravallion and Chao, 1989). Then the interpretation for poverty is unambiguous. That is not, however, the approach that has dominated the literature and practice. This discussion will focus on the main measures of targeting performance found in practice, and on which much of our current knowledge about “what works and what doesn’t” is based. More precise definitions of the measures can be found in Table 1.<sup>8</sup>

## _Targeting measures_

I focus on four main measures, the first three of which are based on the concentration <u>curve,</u> _C_ ( _p_ ), giving the cumulative share of transfers going to the poorest _p_ % of the population ranked by (say) household income per person (Figure 1).

The first measure is the share of transfers going to the poorest _<u>H</u>_ <u>%, such as the poorest</u> 40%. This is demoted _S_ = _C_ ( _H_ ). In the empirical work discussed later, it will be natural to identify the poorest _H_ % as the target group, i.e., the set of people deemed to be have incomes

> 8 Table 1 relates only to the measures used in this study. For a more comprehensive discussion of these and other measures, including their analytic properties, see the excellent volume by Lambert (2001).

4
below the municipal _Di Bao_ poverty line; more precisely, we can set _H_ = _H_ 0 , which is the preintervention headcount index of poverty—the proportion of the population living in households with pre-transfer income per person less than the poverty line. (The post-transfer headcount index is _H_ 1 .) For much of the present discussion we can just take the poorest _H_ % to be some reference group of poor or relatively poor people, without presuming that it is the precise target population for the program in question.

The popularity of _S_ is evident in the fact that the meta-studies by Grosh (1994, 1995) and Coady, Grosh and Hoddinott (2004a,b) found that this was the most readily available measure in their primary sources.<sup>9</sup> The measure’s popularity may well stem from its ease of interpretation. Against this advantage, the measure has some obvious drawbacks. For one thing, it tells us nothing about how transfers are distributed amongst the poor; two programs can have the same share of transfers going to the poor, but in one case the gains are heavily concentrated amongst the poorest, while in the other they case they only reach those just below the poverty line. Another concern is that this measure does not directly reflect the overall size of the transfer program, which will clearly matter to impacts on poverty, as discussed in the introduction.<sup>10</sup> The second measure is the normalized share, _NS_ , obtained by dividing _S_ by _H_ (Figure 1). Coady et al. (2004a, b) preferred to use _NS_ as their measure of targeting performance, arguing that this was more comparable than _S_ because it measures performance relative to a “... common reference outcome...that would result from neutral (as opposed to progressive or regressive) targeting” (p.69).<sup>11</sup> By “neutral targeting” they mean that everyone gets the same transfer amount (whether poor or not), i.e., a “uniform transfer.”<sup>12</sup> If the transfer is uniform then clearly _NS_ =1. However, finding a value of _NS_ close to unity does not imply that the allocation is “close” to being uniform. There are many ways one could get a value for _NS_ of unity (or nearly so), with rather different interpretations. Similarly to _S_ , the _NS_ measure is insensitive to how transfers are

> 9 Coady et al. provide the shares going to the poorest 10%, 20% and 40% for 85 of the antipoverty programs in their study (though with missing data in some cases).

> 10 The literature has pointed to the possibility that the share going to the poor can vary with the scale of a program, though the political economy of program capture; see Lanjouw and Ravallion (1999).

> 11 Coady et al. used _H_ =40% when it was available, which was the case for about half the programs in their study, and the next lowest available number (20% or 10%) when the value for _H_ =40% was not available.  In the earlier comparative study of targeting performance by Grosh (1994), the value of _H_ is set at 40% in all programs studied, in which case the first two measures will (of course) rank

> 12 This is sometimes called an “un-targeted transfer” in the literature, although it is not clear the absence of any effort at targeting would yield a uniform transfer.

5
distributed amongst the poor. The poor can receive _H_ % of the transfers, but different people amongst the poor receive very different amounts; for example, the money could all go to either the poorest person or the least poor person; either way _NS_ =1. _NS_ also approaches unity as _H_ approaches 100%, no matter how the money is distributed. When the reference outcome is this ambiguous, the usefulness of the measure becomes theoretically questionable.

The third measure is the concentration index, _CI_ , which is a widely used in studies of fiscal incidence. This can be thought of as a “generalized _S_ ” in that, instead of focusing on one point on the concentration curve, _CI_ measures the area between the curve and the diagonal (along which the transfer is uniform); in Figure 1, _CI_ is just twice the area marked A.<sup>13</sup> The index is bounded above by 1 (at which point the poorest person receives all payments) and below by -1 (the richest person receives all). This measure has the attraction that it reflects distribution amongst the poor, and (indeed) over the whole range of incomes. A disadvantage is that it is not as easy to interpret as _S_ or _NS_ . And, as with the previous measures, it tells us nothing directly about the scale of transfers.

Although these measures are all based on the concentration curve, they can give quite different results. Of course, _S_ and _NS_ will always be in the same ratio to each other when the same value of _H_ is used for all programs. However, these two measures can rank programs differently when _H_ varies, as in the case study presented later in this paper, and would presumably be the case in many applications.

To illustrate, consider a transfer scheme operating in two cities and giving all participants the same amount. In city A all the transfers go to the poorest 20% and the overall poverty rate is 50% while in city B the transfers go to the poorest 40% and the poverty rate is 10%. A far higher share of the transfers goes to the poor in A ( _S_ =100% versus 25% in B). City A also has the higher concentration index ( _CI_ =0.8 in A versus 0.6 in B). By contrast, it is in city B where the scheme is deemed to be better targeted according to the normalized share ( _NS_ =2.5 for B versus 2 for A). More generally, the concentration curve for program A could lie everywhere above that for program B and yet _NS_ is higher for B, given its lower _H_ .

The fourth measure is the “targeting differential,” _TD_ , which is the difference between the participation rate for the poor—which I will call the coverage rate ( _CR_ )—and that for the

> 13 To assure that all measures go in the same direction, I multiply the usual definition of _CI_ by -1.

6
non-poor (Table 1).<sup>14</sup> Alternatively, one can normalize the targeting differential by the mean transfer over all recipients; call this _TD_<sup>*</sup> . (When all recipients get the same transfer, _TD_ = _TD_<sup>*</sup> .) However, it turns out later that the choice between _TD_ and _TD_<sup>*</sup> makes little difference in the case study. Since _TD_ is easier to interpret I shall focus on this measure.

To interpret the targeting differential, note that when only the poor get help from the program and all of them are covered, _TD_ = 1, which is the measure’s upper bound; when only the non-poor get the program and all of then do, _TD_ = -1, its lower bound. (In the “two cities” example above, _TD_ =0.67 for city B and 0.4 for A.) This measure is easy to interpret, and it automatically reflects both leakage to the non-poor and coverage of the poor.

How are these measures related to the incidence of Type 1 and Type 2 errors? A Type 1 <u>error</u> can be defined as incorrectly classifying a person as poor, while a Type 2 error is incorrectly classifying a person as not poor. A Type 1 error entails a leakage of transfers to the non-poor, while a Type 2 error implies lower coverage of the poor. Let the proportions of Type 1 and Type 2 errors in the populations of the non-poor and poor (respectively) be _T_ 1 and _T_ 2, as defined more precisely in Table 1.<sup>15</sup> (Note that _T2_ =1- _CR_ .) Consider _S_ . This can be written as a function of _T_ 1, namely _S_ = 1 − _T_ 1(1 − _H_ /) _P_ where _P_ is the overall program participation rate. (Alternatively _S_ = 1 − _T_ 1* where _T_ 1* is the proportion of participants who are Type 1 errors.) But one can equally well write _S_ as a function of Type 2 errors, namely _S_ = 1( − _T_ 2) _H_ / _P_ . (Or _S_ = ( _H_ / _P_ ) − _T_ 2 * .) Nor is _P_ likely to be independent of _T_ 1 and _T_ 2; for example, higher coverage of the poor (lower _T_ 2) may tend to come with larger programs. Thus _S_ can be taken to depend on both _T_ 1 and _T_ 2. (The corresponding formulae are more complex for _CI_ , and are omitted.) For the targeting differential, however, the relationship is very clear: _TD_ automatically gives equal weight to both errors; more precisely: _TD_ = 1 − ( _T_ 1 + _T_ 2) .

Thus standard targeting measures depend on the incidence of both types of errors. For the measures based on the concentration curve it should not be presumed that they will be largely unaffected by Type 2 errors. Is an empirical question what weights are attached to these two “errors of targeting.”

> 14 This measure was proposed by Ravallion (2000).  Also see Galasso and Ravallion (2005) on the properties of this measure and the discussion in Stifel and Alderman (2005).

> 15 One might prefer to normalize by population size; similar formulae for this case are easily derived, but the essential point remains.

7
# _Poverty impacts_

In testing the relevance of these targeting measures to a program’s poverty impacts I use three poverty measures: the headcount index, the poverty gap index ( _PG_ ), and the squared poverty gap index ( _SPG_ ) (introduced by Foster et al., 1984). The measures are defined in Table 1. The pros and cons of each are well-documented; for a review see Ravallion (1994). Briefly, _H_ is the easiest to interpret, and is the most popular measure, but is unaffected by income gains or losses to the poor unless they cross the poverty line. _PG_ reflects mean income of the poor, but not inequality amongst the poor, which is the main advantage of _SPG_ .

Impacts are measured by pre-transfer less post-transfer poverty measures ( _H_ 0 − _H_ 1 and similarly for _PG_ ). Impacts on these measures are estimated on the same data, and under the same assumptions about how the scheme works (including behavioral responses), as used in measuring targeting. In particular, I shall assume that income in the absence of the program is observed income less payments received under DB. This assumes that there is no displacement of other income sources through behavioral responses, such as reduced work effort or lower private transfer receipts. This is the most common assumption in the literature on measuring targeting performance; indeed, it appears that virtually all of the primary studies used by Coady et al. (2004a,b) made this assumption. The assumption is questionable, however; I offer some tests that, while not conclusive, suggest that the data are at least consistent with the assumption.

In assessing cost effectiveness I will normalize the poverty impacts by the cost of the program, though a more flexible econometric method of controlling for total spending will also be used. Given the costs of targeting, it is not difficult to imagine cases in which the better targeted program by any of the above measures is less cost-effective against poverty. Consider again the example of cities A and B above in which the program in city A is better targeted according to both _S_ and _CI_ (but not _NS_ or _TD_ ). Suppose that the total cost to the government is the same, but that the finer targeting of city A’s program (for which it will be recalled that all of the transfers go to the poorest 20%, versus 40% in city B) entails extra costs to both the government and participants such that only 25% of participants in city A escape poverty, while in B all poor participants are able to do so. The headcount index falls by 5% points in A, but 10% points in B. B’s program has higher impact on poverty and is more cost-effective.

There is a special case in which one of these measures, namely _S_ , is a perfect indicator of cost-effectiveness for _PG_ . That special case is when the program has no impact on _H_ and there 8
are no fiscal costs besides the transfers. Then it can be readily shown that the impact on _PG_ per unit public spending is simply _S_ . Of course, this special case is unlikely to be of much practical interest, given that people in a neighborhood of the poverty line will presumably be transfer recipients and there will undoubtedly be other costs.

Under the same assumptions, it can be readily shown that the normalized share, _NS_ , is a perfect indicator of cost-effectiveness in reducing the income-gap ratio ( _I_ ) (Table 1). This is (implicitly) the poverty measure relevant to comparisons of program performance based on the normalized share. However, as a poverty measure, the income-gap ratio is known to have a number of undesirable properties; for example, if a poor person living above the mean for the poor escapes poverty then this measure perversely suggests higher poverty. ( _PG_ does not have this property.) It should be noted that these measures of poverty impacts and cost-effectiveness can all be calculated from the same data required for the various measures of targeting performance described above. Of course, if one knows the impacts on poverty—which we agree to be the objective—then one does not need the targeting measures. However, since these targeting measures are widely used in assessing antipoverty programs and in comparative work, it is of interest to test their value as indicators for that policy problem.

# **Program and data**

While economic reforms and structural changes in the Chinese economy have meant high rates of economic growth, it is believed that certain sub-groups have been adversely affected or have been unable to participate in the new economic opportunities due to their lack of skills, long-term illness or disability. The collapse of the old safety-net provided by guaranteed employment has clearly left some households vulnerable. Some of the “left behind” households started poor and some became poor, even though aggregate poverty rates have tended to fall over time. Urban areas have figured prominently in these concerns about the “new poor.” On paper, the _Di Bao_ program provides a transfer to all urban households with incomes below a DB line sufficient to bring them up to that line. The scheme became a national policy in 1999 and expanded rapidly; by 2003 participation had leveled off at 22 million people, representing 6% of urban residents. Municipal authorities have considerable power over the program, including setting the DB lines, funding (the center provides partial co-financing) and 9
implementation. China’s cities vary in ways that could well be relevant to the outcomes of _DB_ ; for example, across the 35 largest urban areas studied in the paper, the highest mean household income per person (the city of Shenzhen) is over four times that of the lowest (Chongqing). The proportion living below the DB poverty line varies from 2% (in Fuzhou) to 19% (Haikou).

The analysis uses China’s Urban Household Short Survey (UHSS) for 2003/04, as discussed in Chen et al. (2006). The UHSS was done by the Urban Household Survey Division of the National Bureau of Statistics (NBS). I use the UHSS sample for the 35 largest cities, giving a total sample of 76,000, varying from 450 (in Shenzhen) to 12,000 (in Beijing). For these 35 cities, the definitions of geographic areas in the UHSS coincide with those for the DB lines and the entire data set has been cleaned by NBS staff and made available for this research.

While the UHSS is a relatively short survey, it allows us to measure a fairly wide range of household characteristics. The survey also included a question on household income and questions were added on DB participation and income received from DB.

As noted in the last section, in measuring targeting and poverty impacts I assume that income in the absence of the program is observed income less payments received under DB. While this is a common assumption, it is clearly questionable. Testing the assumption is difficult without panel data (and even then there can be severe identification problems). With only a single cross-sectional survey it is hard to be confident in the results, given the likelihood of omitted variables correlated with both program placement and the behaviors of interest.

However, I can offer some observations that are at least consistent with this assumption. The design of DB intends that the benefits received will decrease as income rises, implying that participants face a positive marginal tax rate. Indeed, if the program works the way it is supposed to then it exactly fills the gap between current non-DB income and the DB line. Then participants will have no incentive to work (under the usual assumptions that leisure is a normal good and work yields no direct utility). Earned income net of DB will fall to zero. The program will have created a poverty trap, whereby participants do not face an incentive to raise their own incomes, because of the loss of benefits under DB.

The extent to which this is a real problem in practice is unclear. Benefits are unlikely to be withdrawn quickly. There are reports that local authorities allow DB benefits to continue for some period after the participant finds a job (O’Keefe, 2004). Observations from field work also indicate that a notion of “imputed income” was used in a number of provinces. This was a 10
notional level of income that reflected the potential income given the household labor force; this was apparently done with the aim of minimizing work disincentives.<sup>16</sup> Figure 2 plots DB payment (per capita) against the DB gap, given by the difference between the relevant DB line and income net of DB (both per capita). If the program exactly filled these gaps (when positive) then DB payments would rise with a slope of unity, but would be zero for those with income above the DB line. We see a marked tendency for mean DB payments (conditional on the DB gap) to rise with the DB gap, though the conditional expected value (as measured by a non-parametric regression) has a slope appreciably less than unity. The regression line starts to be noticeably positive at per capita incomes that are about 2,000 Yuan above the DB line and peaks at a mean of around 300 Yuan per capita, at a DB gap of around 4,000 Yuan. (The conditional mean is, of course, positive throughout, but very close to zero below 2,000 Yuan.) Thus Figure 2 suggests that the average benefit withdrawal rate (BWR) — the amount by which mean DB payments change with an extra Yuan of income — is around - 0.05; on average, a 100 Yuan increase in income entails a drop of only 5 Yuan in DB payments. An alternative method of estimating the average benefit withdrawal rate is to regress the per capita DB payment received on income per person less DB receipts, with a complete set of dummy variables for municipalities (to capture the differences in the generosity of the program). The implied BWR is very low, at -0.0012 (t-ratio=-17.51, n=76,808). This does not allow for the censoring that is evident in Figure 2. Using a Tobit regression, the estimate is -0.004 (t=-76.23). Estimating the Tobits separately for each municipality, I obtained statistically significant BWRs in all cases, but all were very low, with none higher (in absolute value) than -0.001.

There is almost certainly attenuation bias in these estimates, due to income measurement errors. There is the usual source of measurement error in asking incomes using only one question, plus the fact that income net of DB payments will probably underestimate income in the absence of DB if there are behavioral responses. To address this concern, I tried an Instrumental Variables Estimator (IVE), in which a set of household-level characteristics (including demographics, education attainments, occupation, housing conditions) are used as instrumental variables for income in estimating the BWR; Chen et al. (2006) provide details on the variables used in the first-stage regressions. Note that this only works for the unconditional

> 16 This is based on a personal communication with Philip O’Keefe at the World Bank, drawing on his field-work discussions with local administrators.

11
regression coefficient of DB payments on pre-DB income, so the instrumental variables are automatically excluded from the main regression of interest; the conditional BWR is unidentified. The IV estimate of the unconditional BWR is -0.0021 (t=-28.33), again very low. I also repeated these calculations separately for each municipality, using the IVE for the full sample in each municipality. The estimates were significantly negative for all municipalities and ranged from -0.0102 to -0.0001.

While each of these tests requires an assumption that can be questioned, they all suggest that the benefit withdrawal rate for _Di Bao_ is very small. It would thus appear unlikely that the program would provide any serious disincentive for earning income, thus supporting our assumption that income in the absence of DB is simply observed income minus DB payments received. However, at the same time, such a low BWR raises concerns about how well the program reaches the poorest and how well it adapts to changes in household needs. The BWR for the program is almost certainly too low; Kanbur et al. (1995) find that an optimal BWR around one half is consistent with evidence on the relevant income elasticity of labor supply.

# **Targeting performance and poverty impacts**

On calculating all these measures on the same data set and under the same assumptions, one can test the assumption commonly made in policy discussions that better targeting allows a greater impact on poverty and/or a more cost effective antipoverty program. One can also revisit some of the findings from past research on the factors relevant to targeting success. I begin with the aggregate results and then turn to the city-level analysis.

## _Aggregate results_

I find that 7.7% of the total population of the 35 cities had a net income (observed income _minus_ DB receipts) below the DB line (Table 2). The program’s total participation is equivalent to about half of the eligible population by this definition. About 40% of DB recipients are ineligible according to these data (0.43=1.69/3.91). The proportion of these Type 1 errors amongst the non-poor is clearly very low at 0.018 (=1.69/92.29). But there is a high proportion 12
of Type 2 errors, with almost three-quarters of those who are eligible not being covered by the program (0.71=5.48/7.71, i.e., _CR_ =0.29).<sup>17</sup> Nonetheless, targeting performance appears to be excellent by international standards, with _S_ =64%, _NS_ =8.3 and _CI_ =0.78.<sup>18</sup> Coady et al. (2004a,b) provide estimates of _NS_ for 85 programs. Argentina’s _Trabajar_ program has a _NS_ = 4.0, making it the best performer by this measure amongst all programs surveyed by Coady et al.<sup>19</sup> The median _NS_ is 1.25. By this measure, _Di Bao_ is a clear outlier in targeting performance internationally.

Turning to the fourth measure of targeting performance I find that while 29% of the poor receive DB, this is only true of about 2% of the non-poor. Thus I find that _TD_ =0.27. The mean DB payment across all those with _Y_ < _Z_ is 87.61 Yuan per person per year, while the corresponding mean for those with _Y_ ≥ _Z_ is 4.15. The overall mean DB payment across all recipients is 270.33, so _TD_<sup>_*_</sup> =0.31.

These calculations indicate that, while the program is good at concentrating benefits on the poor, it still falls well short of perfect targeting ( _TD_ =1) in which all of the poor and only the poor are covered, as would be implied by the program’s design (as was plain from Figure 2). Another way to see this is to calculate the total receipts for those with net income below the DB line. One then finds that only 12.1% of the aggregate DB gap is filled by the program. DB is a long way off reaching its own aim of bringing everyone up to the DB line.

The weak coverage of the program — in terms of both coverage of those living below the DB line and coverage of the DB gap — is naturally limiting its impact on poverty, despite excellent targeting in the sense of avoiding leakage to the non-poor. Table 3 gives various poverty measures before and after DB transfers. To test robustness to the location of the DB lines, Figure 3 gives the empirical cumulative distribution functions of income — with

> 17 Ravallion (2007) proposes a method of testing robustness of such calculations to income measurement errors and implements the method for the same program.  The results indicate that, if anything, the survey-based incomes probably understate the coverage of the eligible population.

> 18 To calculate _CI_ from the micro data I used the regression-based method of Jenkins (1988).

> 19 _Trabajar_ is a combination of a workfare program and social fund, whereby participants are offered low-wage work to do things of value to poor communities; see Jalan and Ravallion (2003). Coady et al., calculate the normalized share from _Trabajar_ from Jalan and Ravallion (2003) who estimate that 80% of _Trabajar_ participants come from the poorest 20% of the Argentine population ranked by income net of _Trabajar_ receipts (which are roughly constant across recipients).  The corresponding normalized share for the poorest decile is much higher, at about 6.0, though still less that for DB.

13
households ranked by income normalized by the relevant DB line — with and without DB receipts for both participants and the full (35-city) sample.

The program is having a sizeable impact on poverty amongst the participants (Table 3). The proportion of the participant population falling below the DB line is 45% with DB transfers, but it would have been 57% without them. However, the impact on poverty in the population as a whole is much less. The proportion falling below the DB lines falls from 7.7% to 7.3% after DB transfers. Proportionate impacts are slightly higher for _PG_ than for the headcount index (and slightly higher again for the squared poverty gap); this indicates that the program has increased the mean income of those below the DB line and reduced inequality amongst them. _Targeting and poverty across cities of China_ Given the scheme’s decentralized financing and implementation and the differences observed across China’s cities, heterogeneity in outcomes across municipalities is to be expected. There will, of course, be differences in local resources and administrative capabilities, but there will also be (less obvious) differences in the local political economy. Here I only aim to describe the differences in DB performance across municipalities, and to use these differences to assess how well prevailing targeting measures perform in predicting impacts on poverty.

There is considerable variation in targeting performance across municipalities. (The Appendix gives results by city.) _S_ varies from 31% to 98%; _NS_ varies from 2.8 to 18.8, and _CI_ varies from 0.64 to 0.93, while _TD_ varies from 0.06 to 0.53. (All cities except one, Kunming, have _NS_ higher than the best performing program surveyed by Coady et al., 2004.) Recall that _TD_ automatically gives equal weights to the Type 1 and Type 2 error proportions. The weights for the three measures based on the concentration curve depend on the analytic properties of these measures and how the design features and setting influence the overall program participation rates (as discussed above). A simple way of summarizing this (potentially complex) relationship, is to regress each measure of targeting performance on _T_ 1 and _T_ 2 (as defined in Table 1). Table 4 gives the regressions. _T_ 2 has only a small and statistically insignificant effect on _S_ , _NS_ or _CI_ ; by contrast, _T_ 1 has a strong and significant effect for _NS_ and _S_ ; the coefficient is also significant at the 3% level for _S_ if one drops _T_ 2.<sup>20</sup> However, the normalizations used in defining _T_ 1 and _T_ 2 have bearing on the weights attached to Type 1 and Type 2 errors. If instead one normalizes by total populations of the

> 20 The regression coefficient on Type 2 errors is then -1.361 (t=-2.28; prob=0.03).

14
municipality the results change noticeably; thus the regressors become _H_ 0<sup>_T_</sup> 1 and 1( − _H_ 0 )<sup>_T_</sup> 2 . As can be see in Table 4, the three measures based on the concentration curve all attach negative weights to Type 1 errors per capita, but now we find that _NS_ also puts a negative weight on Type 2 errors (as does _CI_ , although it is not significant). And we find that _S_ puts a positive weight on Type 2 errors, while _TD_ puts a positive weight on Type 1 errors. Only for _CI_ do we find that the weighs attached to the two errors of targeting are robust to the normalization. The generalizations found in the literature about the relative importance of Type 1 and Type 2 errors to standard measures of targeting are questionable.

However, the most important point for the present analysis is that none of the measures based on the concentration index have strong correlations with the coverage rate of the poor. The simple correlation coefficients with _CR_ are -0.28, -0.30 and -0.40 for _S_ , _NS_ and _CI_ respectively. By contrast, _TD_ has a correlation of 0.98 with the coverage rate.

There are some clear covariates of the heterogeneity in targeting performance, echoing some past findings in the literature. There is a high correlation between DB spending and targeting performance as measured by _TD_ (r=0.73), though the correlation with targeting performance is appreciably weaker for the other measures.<sup>21</sup> Figure 4 shows the relationship for _TD_ . This pattern in the data is consistent with evidence for antipoverty programs in other settings indicating that _TD_ tends to improve as programs expand, and to deteriorate in fiscal contractions (Ravallion, 2004). It appears that the early benefits (at a low level of spending) tend to be captured more by the non-poor while the poor benefit more when the program expands and are the first to bear the costs of contractions. As can be seen from Figure 5, the differences in program scale, as measured by participation rates ( _P_ ), are also highly positively correlated with coverage rates for the poor (r=0.80). As we will see, this ‘scale effect’ will be key to understanding why measures of targeting performance based on the concentration curve perform so poorly as predictors of poverty impacts.

The impact of higher initial poverty on targeting performance has also been discussed in the literature.<sup>22</sup> I can confirm the Coady et al. (2004a,b) finding that the normalized share is higher in richer cities. The correlation coefficient between the normalized share and the pre-DB

> 21 By contrast, the other targeting measures tend to be negatively correlated with DB spending, though only significantly so for _CI_ (r = -0.52)

> 22 For a theoretical analysis (using _TD_ as the measure of targeting) see Ravallion (1999).

15
headcount index is -0.81. However, this is entirely due to the normalization; if one uses the ordinary share, _S_ , the correlation is positive and significant (r=0.55).<sup>23</sup> (One wonders whether the Coady et al results would be robust to using _S_ , or _NS_ with a uniform _H_ across all programs.) Impacts on poverty also vary across cities (see the Appendix for details). Subtracting the post-DB poverty rate from the pre-DB rate, the impact on the headcount index varies from 0.0% to 1.5% points. Table 5 provides the correlation coefficients between the targeting measures and the program’s impacts on both the headcount index and poverty gap index. The results for _SPG_ were very similar to _PG_ and are omitted for brevity. Correlations are given for both the levels impact (pre-DB poverty measure _less_ post-DB measure) and the proportionate impact (normalized by the pre-DB poverty measure).

Amongst the four main measures of targeting performance, the strongest indicator (by far) of the impact on poverty is the targeting differential, _TD_ . Strikingly, I find no sign of a <u>positive</u> correlation between the impacts on poverty and any of the three most popular measures, _S_ , _NS_ and _CI_ . For _S_ the correlation coefficient with the impacts on the level of the headcount index is only 0.03, while for _CI_ and _NS_ the correlation coefficients with poverty impacts turn out to be negative. This switches for proportionate impacts, which are negatively correlated with _S_ but virtually orthogonal to _NS_ . Figures 6 and 7 plot the impacts on the level of the headcount index against the _NS_ and _TD_ measures respectively. We see clearly that municipalities with a higher normalized share going to the poor tended to have lower impacts on poverty (Figure 6). The targeting differential does not have this perverse property (Figure 7).

In Table 5, the correlations are pair-wise. Instead, Table 6 gives regressions of the poverty impacts (columns 1 and 3) on all four measures jointly. (The table also includes regressions that control for spending, which I return to.) _TD_ remains the strongest predictor of poverty impacts. _S_ now emerges as a positive predictor, though still not significant at the 1% level and _NS_ is no longer a significant negative indicator, at given values of the other targeting measures. _CI_ remains a negative predictor of poverty impacts.

Table 5 also gives the correlation coefficients for the incidence of Type 1 and Type 2 errors. Strikingly, one finds that the proportion of Type 1 errors ( _T_ 1) is positively correlated with poverty impacts; cities that did a better job excluding the non-poor tended to do less well in

> 23 Using mean income instead one obtains r=0.57 for _NS_ and -0.34 for _S_ .

16
reducing poverty. It is only _T_ 2 that is negatively correlated with impacts.<sup>24</sup> I found the same pattern in the partial correlations, by regressing poverty impacts on both the proportions of Type 1 and Type 2 errors; the regression coefficients on the Type 1 (Type 2) error proportions were significantly positive (negative) at the 1% level for the impacts on both _H_ and _PG_ .

It is now evident that the main reason why the three targeting measures based on the concentration curve perform so badly as indicators of poverty impact is that they are not even positively correlated with the program’s coverage of the poor, which is highly correlated with poverty impacts (as can be seen from the correlation coefficients for Type 2 errors in Table 5).

What about cost-effectiveness? Cities with higher DB spending tend to have higher impacts on poverty; the correlation coefficients between DB payments per capita (of the population) and the impacts of _H_ and _PG_ are 0.80 and 0.86 respectively. The simplest way to test how well the targeting measures predict poverty impacts at given levels of spending is by normalizing poverty impacts by DB spending, to give the cost-effectiveness ratio.<sup>25</sup> Table 5 gives the correlation coefficients.

The share going to the poor now emerges as having a positive correlation with costeffectiveness and statistically significantly so for _PG_ . Figure 8 plots the data points in this case; a positive relationship is evident, with a regression coefficient of 0.033 (t=5.61, based on a White standard error). Even so, R<sup>2</sup> =0.42, so that the majority of the variance in cost-effectiveness in reducing PG is left unexplained.

This is the exception though. None of the measures show significant correlations with cost-effectiveness in reducing the headcount index. The normalized share still has the perverse negative correlation found for the total poverty impacts.<sup>26</sup> The reason why the normalized share, _NS_ , emerges as a perverse indicator (even though _S_ is uncorrelated with impact, and positively correlated with cost-effectiveness) is that there is a positive correlation between the (pre-

> 24 Expressing the incidence of targeting errors on a per capita basis gives virtually identical correlations for Type 1 errors. However, the per capita incidence of Type 2 errors is uncorrelated with poverty impacts. This stems from the strong positive correlation between (pre-DB) _H_ and poverty impacts (and positive correlation between _H_ and Type 2 errors per capita); using a regression to control for _H_ a significant negative correlation remerges between the poverty impacts and Type 2 errors.

> 25 Data were not available on administrative costs of the program at municipal level, so the spending variable is solely based on transfer payments. The correlation coefficients will only be unaffected if the administrative cost share is the same across different cities.

> 26 The normalized share is significantly correlated with cost-effectiveness in reducing the incomegap ratio (r=0.53), although recall that this is a flawed measure of poverty.

17
program) headcount index and the program’s impact on poverty; the correlation coefficients are 0.36 and 0.51 for _H_ and _PG_ , respectively. The incidence of the two types of errors shows little or no correlation with the cost-effectiveness ratios.

A more flexible way of seeing whether the targeting measures reveal poverty impacts at given spending is to use a regression of poverty impacts on the targeting measures with controls for spending. To allow for nonlinearity in a reasonably flexible way, I used a cubic function of DB spending per capita. Testing each targeting measure one by one, only _S_ turns out to be a significant (positive) predictor of impacts on poverty at given program spending; the correlation was significant at the 4% level for _H_ and 1% level for _PG_ . The results using the four measures together are given in Table 6 (columns 2 and 4; for PG the higher order terms can be dropped, giving column 5). Again, one finds that only _S_ is a significant predictor of impacts on poverty at given program spending, though now this holds for for _H_ at the 2% level as well as _PG_ .

# **Conclusions**

The three most popular measures of targeting performance found in practice are the share of transfers going to the poorest _H_ %, the share normalized by _H_ , and the concentration index. Each has it strengths and weaknesses. However, I find that none of these measures reveal much about the success of China’s _Di Bao_ program in achieving its objective of eliminating extreme urban poverty. Only one of the measures studied here, namely the targeting differential, has a statistically significant positive correlation with the program’s poverty impacts. The cities of China that are better at targeting this program are generally not the ones where the scheme came closest to attaining its objective.

Possibly more surprising, I find that these measures are not very informative about the program’s cost-effectiveness, i.e., poverty impact at given program spending. The one exception is that the share going to the poor is a statistically significant predictor of cost effectiveness in reducing the poverty gap index. But, even then, about 60% of the variance in the costeffectiveness ratio is left unexplained. All other measures perform poorly, or even perversely.

These findings echo some of the warnings in the literature against relying on standard measures of targeting performance for informing policy choices concerning antipoverty programs. The paper’s findings also cast doubt on the generalizations found in the literature about what type of program “works best,” and so should be scaled up, based on cross-program 18
comparisons of targeting measures. The external validity of these programmatic comparisons is clearly questionable if the targeting measures have such a poor fit with poverty impacts. It is also unlikely that past findings on the socio-economic factors influencing targeting performance at country level are robust to seemingly arbitrary differences in the measures used.

One question is left begging: Why have the literature’s warnings carried so little weight in practice? Possibly the more “theoretical” objections to these targeting measures have fallen on deaf ears for lack of clear evidence on how the measures perform in practice. The results of this case study will then help. One can also conjecture that the preference for targeting measures that put a high weight on avoiding leakage to the non-poor stems from fiscal pressures, given that reducing leakage helps cut public spending, while expanding coverage does the opposite. While I do not doubt that such thinking has had influence at times, it is surely misguided. For if the problem was to minimize public spending (unconditionally) then why would governments bother with such programs in the first place? Evidently there is a demand for these policies, as part of a comprehensive antipoverty strategy. A more credible characterization of the policy problem would then give positive weight to both avoiding leakage and expanding coverage of the poor.

From that perspective, measures of targeting performance that penalize both errors of targeting make more sense—again echoing recommendations found in the literature. However, that conclusion would still miss the point. Even the targeting measure studied here that is found to be the best predictor of poverty impacts is a long way from being a perfect indicator. If there is a single message from this study it is that analysts and policy makers would be better advised to focus on the estimable outcome measures most directly relevant to their policy problem. In the present context, impacts on poverty can be assessed with the same data and under the same assumptions as required by prevailing measures of targeting performance.

19