metadata: authors: 0: author: Schnitzer,Pascale
1: author: Stoeffler,Quentin
count: Africa
docna: 0: docna: Targeting for Social Safety Nets : Evidence
            from Nine Programs in the Sahel
docty: Policy Research Working Paper
subtopic: Inequality,Economic Assistance,Disability,Services & Transfers to Poor,Access of Poor to Social Services,Nutrition,Food Security,International Trade and Trade Rules,Social Protections & Assistance
teratopic: Macroeconomics and Economic Growth,Agriculture,Social Protections and Labor,International Economics and Trade,Health, Nutrition and Population,Poverty Reduction
abstracts: cdata!: Proxy-Means Testing (PMT) and
            Community-Based Targeting (CBT) are among the most widely
            used methods for poverty targeting in low-income settings.
            This paper analyzes household data from nine programs
            implemented in the Sahel region using a harmonized approach
            to compare PMT and CBT selection as conducted in practice,
            once geographical targeting has been applied. The results
            show that the targeting performance measured depends
            critically on the definition of the targeting objectives,
            share of beneficiaries selected, and indexes used to
            evaluate targeting. While PMT performs better in reaching
            the poorest households based on per capita consumption, it
            differs little from CBT, or a random or universal allocation
            of benefits when distances to poverty lines are considered.
            When aiming to identify food insecure households, most PMT
            and CBT targeting schemes perform no better than a random
            allocation of benefits. On the other hand, targeting costs
            represent only a small share of budgets. Overall, the
            results emphasize the need to study programs as implemented
            in practice instead of relying on simulations of targeting
            performance, as widely used by practitioners and academics.
            Taken together, the findings suggest that while there may be
            a need to select households resulting from budget
            constraints, PMT and CBT contribute little to poverty or
            food insecurity reduction efforts in poor and homogeneous settings.
repnme: repnme: Targeting for Social Safety Nets : Evidence
            from Nine Programs in the Sahel
display_title: Targeting for Social Safety Nets :
            Evidence from Nine Programs in the Sahel
listing_relative_url: /research/2021/10/33538405/targeting-social-safety-nets-evidence-nine-programs-sahel
topicv3: Food Security m1326439 199,Poverty m1326687 695,Trade m1326924 698,Social Protection and Labor m1327194 697,Cash Transfers m1327437 1608
publishtoextweb_dt: 2021-10-26T00:00:00Z
dois: 10.1596/1813-9450-9816
historic_topic: Macroeconomics and Economic Growth,Agriculture,Social Protections and Labor,International Economics and Trade,Health, Nutrition and Population,Poverty Reduction
origu: Social Protection & Labor ECA (HECSP)
no_of_pages: 42
author_keys: 0: author_key: 957122
1: author_key: 1162816
model_extractions: Policy Research Working Paper 9816 Targeting for Social Safety Nets Evidence from Nine Programs in the Sahel _Pascale Schnitzer Quentin Stoeffler_ Social Protection and Jobs Global Practice October 2021
Policy Research Working Paper 9816

# **Abstract**

Proxy-Means Testing (PMT) and Community-Based Targeting (CBT) are among the most widely used methods for poverty targeting in low-income settings. This paper analyzes household data from nine programs implemented in the Sahel region using a harmonized approach to compare PMT and CBT selection as conducted in practice, once geographical targeting has been applied. The results show that the targeting performance measured depends critically on the definition of the targeting objectives, share of beneficiaries selected, and indexes used to evaluate targeting. While PMT performs better in reaching the poorest households based on per capita consumption, it differs little from CBT, or a random or universal allocation of benefits when distances to poverty lines are considered. When aiming to identify food insecure households, most PMT and CBT targeting schemes perform no better than a random allocation of benefits. On the other hand, targeting costs represent only a small share of budgets. Overall, the results emphasize the need to study programs as implemented in practice instead of relying on simulations of targeting performance, as widely used by practitioners and academics. Taken together, the findings suggest that while there may be a need to select households resulting from budget constraints, PMT and CBT contribute little to poverty or food insecurity reduction efforts in poor and homogeneous settings.

This paper is a product of the Social Protection and Jobs Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at pschnitzer@worldbank.org and quentin.stoeffler@um6p.ma.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
Targeting for Social Safety Nets: Evidence from Nine Programs in the Sahel Pascale Schnitzer<sup>*</sup> & Quentin Stoeffler<sup>†</sup><sup>**_‡_**</sup>

**Keywords** : targeting; proxy means testing; community-based targeting; social protection; poverty; Africa

**JEL Codes** : D63, I32, I38, O12, O15, R23, R28

> * World Bank.

> † University Mohammed VI Polytechnic. Corresponding author: quentin.stoeffler@um6p.ma.

> ‡ This study was supported by the World Bank Sahel Adaptive Social Protection Program, funded by the United Kingdom, France, Germany and the Wellspring Foundation. We are very grateful to Diana Cheung, Mame Fatou Diagne, Yves Kameli, Papa Sosthène Konate, Patrick Premand, Solene Rougeaux and Kalilou Sylla for providing us with the data analyzed in this paper. We are also grateful to Arthur Alik-Lagrange, Christian Bodewig, Caitlin Brown, Margaret Grosh, and Patrick Premand for their comments.
# **1 Introduction**

The question of how to allocate limited resources is central to economic thinking. This issue is also currently driving a heated policy and academic debate on targeting, that is, the manner of selecting beneficiaries of social transfers. The debate has been fueled in recent years by the rapid proliferation of cash transfer programs, requiring decisions on how to select beneficiaries. But despite the increasing number of studies on the topic, questions remain on the best method for targeting households in development or humanitarian contexts. Cross-country comparisons of targeting methods employed in practice emphasize that there is a wide variation in targeting efficiency for each method (Coady et al., 2004; Devereux et al., 2017). Similarly, within-country comparisons of different targeting methods also show that targeting errors are important and that there are often numerous trade-offs across targeting objectives (Alatas et al., 2012; Schnitzer, 2019; Stoeffler et al., 2016). Moreover, the very idea of targeting beneficiaries, rather than implementing “universal” programs, is questioned by some scholars (Banerjee et al., 2019; Brown et al., 2018; Devereux, 2021; Kidd et al., 2017; Mkandawire, 2005).

Two important aspects have often hindered the ability to make informed decisions on targeting (or universal) choices. First, the concepts and methods used to discuss targeting performance are often misleading and can hinder a consensus on some aspects of the issue – or even a clear exposition of the specific trade-offs and sources of disagreement. Second, apart from individual project-case studies, empirical evidence on targeting outcomes is still scarce. Comparison across these studies is difficult, given the differences in assessment methodologies employed. The few studies that compare targeting performance across several programs are based either on simulated scenarios (such as in Brown et. al 2018) or on actual programs, but without fully harmonized features (such as in Coady et al., 2004 and Devereux et al., 2017).

Relying on harmonized household-level data from nine programs implemented in the Sahel region, this paper contributes to the academic and policy discussion on targeting by (i) providing comparable evidence on the performance of targeting methods implemented in practice; and (ii) showing the role of different measurement choices in targeting performance. We analyze targeting schemes in six countries (Burkina Faso, Mali, Niger, Senegal and Chad, and Northern Cameroon) that were implemented between 2014 and 2018.<sup>1</sup> Our study focuses on low-income countries where evidence is scarce and where targeting raises specific questions in contexts that are characterized by high poverty rates, low budgets relative to needs, and low administrative capacities. We analyze data sets that include information on proxy means testing

> 1 Each targeting scheme was evaluated under a different study, currently part of the gray literature (Table A1). A review of these reports and papers was conducted by the authors and is available upon request. Taken together, these studies represent a wide range of methodologies for measuring performance, making comparisons difficult.

2
(PMT) and community-based targeting (CBT), two targeting methods that are commonly employed, and compare their performance as implemented in practice, within and across countries. Since in low-income settings, PMT and CBT are almost always applied after geographical targeting, our analysis allows us to understand the actual contributions of PMT and CBT once _geographical_ targeting has been applied. In addition, we explore the role of measurement choices across multiple dimensions, including three key ones. First, we consider both per-capita consumption and food security as well-being metrics, the latter being a major policy objective in humanitarian interventions. Second, we explore the role of program coverage (or the share of beneficiaries selected) in the performance of targeting. Third, we rely on multiple measures used in the literature to evaluate targeting, with a preference for distribution-sensitive indices (i.e. measures that consider distances to poverty lines). To provide additional insights, we conduct simple regressions to identify whether distinct types of households are selected or excluded by different targeting methods. Finally, we collect cost data, an area where evidence is particularly limited, although the cost of targeting instruments is often cited as an argument against targeting.

Several important results emerge from this study. We show that measurement choices play a decisive role in the performance of targeting. At first sight, PMT seems to perform well in reaching the poorest households.<sup>2</sup> The median PMT-targeted program provides 20 percent more resources to the poorest than would random allocations. This is only 6 percent for CBT, which performs systematically worse than PMT based on consumption per capita across all programs studied. Nonetheless, when considering distributionsensitive measures of performance, none of the methods seems to perform significantly differently from each other and from random or budget-neutral universal delivery of benefits. This result is driven by the high poverty rates and the significant homogeneity present in the program areas studied. A different picture emerges when relying on food insecurity as a well-being metric: only four targeting schemes perform better than a random allocation of benefits, and there are no systematic differences between PMT and CBT. Again, when relying on distribution-sensitive metrics, no targeting scheme makes a significant contribution toward reaching the most food-insecure households.

While differences between targeting methods are relatively small, we also find that for a given method, the program coverage (i.e. the share of beneficiaries selected) plays a crucial role in explaining targeting performance. Higher program coverage usually implies a lower contribution that targeting makes relative to a random allocation of benefits. As such, comparisons of targeting performance that do not account for this parameter can be misleading, and the adjustments that we make in our primary data analysis by harmonizing program coverage rates are not trivial. In addition, our results highlight the critical role of

> 2 Where ‘poorest’ is defined based on program coverage rates. For instance, if a program covers 30 percent of the population, the poorest are then defined as those in the bottom 30 percent of the consumption per capita distribution.

3
clearly defining objectives (e.g. selecting the poorest, or any poor, or the food-insecure). We find that when measured consistently in comparable contexts, the variation in the performance of methods is relatively small. This result is different from Coady et al. (2004) and Devereux et al. (2017) whose reviews include countries that differ greatly in several ways. Finally, our results highlight how targeting studies based on simulations can be misleading. First, simulations are conducted on nationally representative surveys while most interventions are implemented on largely different samples, after geographical targeting has been applied. Second, program coverage rates often differ from the simulations conducted, having dramatic implications on the performance of targeting. The actual targeting performance is also affected by implementation issues.

Overall, our findings suggest that after geographical targeting is applied, the other targeting method employed is not likely to make a major difference in reducing poverty or food insecurity in low-income settings. That being said, targeting costs are also similar across methods and represent a small share of the total amount of funds distributed. Our findings imply that when making choices on targeting methods (or whether to target or not), other parameters – not considered in our paper because of data limitations - may be more important than targeting errors and costs. For instance, the legitimacy of targeting schemes, the program spillovers that targeting may generate on non-beneficiaries, or how targeting choices affect the program impact on final outcomes deserve greater attention.

Our study contributes in several ways to the literature. As far as we know, this is the first cross-country comparison of targeting performance based on a primary data analysis from actual programs. As such, it differs from cross-country analyses based on simulations (e.g. Brown et al., 2018), which focus on PMT performance but cannot assess CBT or measure actual performance in practice. Our study also differs from cross-country reviews that are based on actual programs, but not based on primary, micro-level data analysis (Coady et al., 2004; Devereux et al., 2017). Indeed, our methodology allows us to rely on multiple measures of performance that are comparable, and to benchmark our results against potential alternatives (random, geographic or universal targeting). Finally, we provide data on targeting costs, and discuss policy and methodological implications for the design and study of targeted programs in the future.

There are several caveats to our analysis. Our data do not allow us to consider the legitimacy of different targeting methods, or how they affect program impacts on household outcomes. Besides, we do not consider political economy and other issues (Sen, 1993; Devereux, 2016; Duchoslav et al., 2021). All these areas call for further research.

The next section discusses key measurement considerations in analyzing targeting methods. Section 3 describes the Sahel context and reviews the existing literature on targeting methods. Section 4 presents the 4
data and the methodology used to compare targeting methods. Section 5 shows the results, and section 6 concludes with methodological and policy recommendations.

# **2 Measuring targeting performance**

Analyzing targeting outcomes is challenging because of the different conceptions of what targeting should achieve and the different ways to assess its performance. We discuss various concepts and methodologies in order to outline our framework for approaching targeting assessment.

## _2.1 Objectives: Why targeting and what to target?_

In ultra-poor contexts, budgets are often insufficient to address needs, and decisions must be made on who will benefit from social transfers (“targeting”).<sup>3</sup> While there may be equity reasons (redistribution to the poor) or political motives (providing assistance to the “deserving poor” only) for targeting in other settings, the rationale for targeting in low-income environments is most frequently the limited budget available for social or humanitarian programs.<sup>4</sup> For instance, in the six low-income countries that we focus on, between 44 and 50 percent of the population lives in extreme poverty ($1.9 PPP), and almost the entire population lives with less than $5.5 PPP per day.

Paradoxically, this context also suggests that “universal” transfers, delivered to the entire population, may be more appropriate.<sup>5</sup> Targeting may be illegitimate if local populations perceive that “we are all poor here” (Ellis, 2012). Besides, high poverty rates mean that the risk of making transfers to the “wrong,” wealthy households in the rural Sahel is very low. However, delivering transfers to the full population over several years would be a significant financial challenge for countries with limited resources in the form of tax

> 3 As noticed by Sen (1993), “targeting” is not a good word to characterize the selection of beneficiaries of social programs because it implies aggressive reach of passive beneficiaries. Also, “targeting” suggests that it is possible to perfectly reach the selection objective, which is misleading. Finally, “targeting” has several distinct meanings even within development economics, which generates confusion. However, we continue using this word to be in dialogue with the large existing literature on targeting, and for the lack of a better word (since “selection” also implies a competition; and “identification” of the poor is also vague and polysemous).

> 4 For recent examples of this justification for targeting, see for instance Kaul et al. (2019) for a review of WFP humanitarian programs in the Sahel, Cuevas et al. (2019) for the targeting analysis of the largest humanitarian program in the world. Verme & Gigliarano (2019) analyze the implications of this existing budget constraint for optimal targeting.

> 5 For important arguments in favor of universalism related to the political economy, human rights and/or targeting errors and costs, see for instance Gelbach & Pritchett (2002), Mkandawire (2005), or Kidd et al. (2017). However, ”universalism” is often employed in confusing manners to qualify categorical targeting (e.g. all elderly individuals receive transfers) as it implies excluding all individuals who do not belong to this category. When limited to a given area (all individuals in a given area receive transfers), ”universalism” is also a confusing term as it implies excluding all individuals from different areas. To clarify these approaches, we consider both universal selection and geogrpahic targeting in our analyses.

5
revenues (Ikegami et al., 2017; Hanna and Olken, 2018; Banerjee et al., 2019). In Sahelian countries, even programs targeting poor households do not manage to cover the entire national territory, and social programs reach a relatively small proportion of the population (Monchuk, 2013; Beegle et al., 2018).<sup>6</sup> In this regard, social-protection programs do not differ from other development interventions that target specific geographic areas and populations.<sup>7</sup> Because targeting is likely to remain the norm _de facto_ in lowincome environments (despite potential aspirations to universalism), studying targeting remains a relevant economic question (Banerjee et al., 2019).

That being said, if we assume a fixed budget constraint for the allocation of social protection resources, the debate on “universalism” points to important trade-offs for targeting transfers. The first trade-off regards transfer amounts: one can choose to include more people (potentially everyone) with lower benefits or to target fewer people with higher levels of benefits. The second trade-off regards the geographic areas: one can decide to include more people (or everyone) in fewer geographic areas (e.g., villages) _vs._ to fewer people in more geographic areas. All programs implicitly face these trade-offs, and some analyses have attempted to measure their implications on targeting performance (e.g. Brown et al., 2018; Schnitzer, 2019). But several dimensions have been left aside, such as how these trade-offs affect spillover effects on nonbeneficiaries, the legitimacy of the program, and the impacts of the transfer (e.g. does providing lower transfer amounts dilute the effects of the transfers?). We consider these trade-offs when we discuss the critical importance of program coverage.

Finally, targeting is ultimately a policy issue, and it is not possible to make optimal choices in terms of targeting strategy without defining the goals and the criteria for assessing its performance. “What is the target?” is thus an important public policy question, but it is rarely clearly defined _ex-ante_ (Stoeffler et al., 2020; Schnitzer, 2019). Cash transfers in particular address a wide variety of distinct, competing objectives (e.g., improving nutrition, health, and education for children, and generating investments in productive assets and agriculture). In the absence of clearly defined goals, targeting studies do test performance along several welfare metrics such as food security, per-capita consumption, an asset index, a health index, or poverty self-assessment, (Alatas et al. 2012; Schnitzer 2018; Premand & Schnitzer 2020; Stoeffler et al. 2016; Çubukçu & Stoeffler 2019). Several of these studies (although not all) have found that one targeting

> 6 An additional challenge is a logistical one in low administrative capacity environments. Providing cash transfers in the Sahel is non-trivial in terms of uniquely identifying and registering beneficiaries, delivering transfers, and administering the program (among others) and requires a strong infrastructure in addition to technical capacities. Both universal and targeted programs would share this difficulty if they were implemented without prior geographic targeting (see below).

> 7 For instance, Briggs (2017b) and Briggs (2018) show that foreign aid tends to target wealthier regions and households in Africa. Burgess et al. (2015) show that road construction in Kenya is politically targeted based on the leader’s ethnicity during non-democratic periods, which is costly in terms of economic growth.

6
method works better for achieving some objectives, and another method works better for achieving other objectives. Under these circumstances, discussing which method is more adapted to a given context is easier when public policy objectives are clearly defined.<sup>8</sup>

# _2.2 Assessing targeting_

The most common way to assess targeting performance has been measuring its ability to reach the intended population. In doing so, three challenges have often hindered the ability to make informed policy decisions. First, indicators used in the literature have largely relied on binary classifications of whether someone is poor or not.<sup>9</sup> The exclusion and inclusion errors, the CGH index and the Targeting Differential are some examples of these measures. While easy to interpret, these measures have important limitations that can bias the policy discussion. Classifying households as “poor” and “non-poor” in a binary distinction provides limited information, which has long been acknowledged (Foster et al., 1984). This binary classification overstates the actual level of targeting errors. For instance, a household just above the poverty threshold will be counted as _wrongly included_ if it receives benefits, even though its consumption is still very low. The issue is even more severe in low-income settings, where the binary distinction of “poor” and “nonpoor” is often meaningless (Ellis, 2012) and where welfare is often measured noisily and imperfectly (Deaton & Zaidi, 2002). In addition, the binary distinction between the correct and incorrect inclusion can bias the policy discussion by suggesting high rates of “non-deserving” individuals that receive transfers – even when those are actually poor based on some other, also-relevant poverty lines. In sum, it is important for targeting measures to consider the full distribution of the well-being metric of interest, rather than classifying households in two categories.<sup>10</sup> Second, selection thresholds (share of population selected by a program), and eligibility thresholds (share of population deemed eligible) critically affect the performance of targeting methods (see section 4). Nonetheless, these are rarely acknowledged or considered when measuring targeting performance. As a consequence, one can arbitrarily obtain different performance results by changing the selection or eligibility threshold. Eligibility and selection thresholds should be set in a policy-relevant manner. For example, if the objective is to reach anyone below the national poverty line, the corresponding threshold should be used. If, instead, the objective is to reach the _poorest_ households, the eligibility threshold could be set equal to

> 8 This is leaving aside the equity-efficiency trade-off that some programs may face in terms of targeting. See, for instance Pan & Christiaensen (2012) for an analysis of targeting that includes these dimensions, and Fafchamps et al. (2020) for an analysis of targeting performance focused on the efficiency criteria.

> 9 The Addendum, available upon request, describes how key indicators used in the literature are measured.

> 10 Such a targeting measure can be the simulated effect of transfers using the targeting method on the poverty gap, or on the poverty severity, which puts more weight on the selection of the poorest households. On the other hand, the simulated impact of the headcount ratio can be misleading, as it may indicate a greater performance for methods that target “less poor” households (close to the poverty line) as opposed to methods that reach the poorest of the poor.

7
the selection threshold.<sup>11</sup> Similarly, selection thresholds should try to mimic as much as possible actual program coverage. In practice, researchers can rely on various eligibility thresholds to present potential trade-offs and allow policy makers to decide based on key objectives. It would also make comparisons across studies easier and more relevant.

Third, studies on targeting often lack relevant benchmarks or counterfactuals, which hinders the ability to understand the extent to which different targeting options may help in a given context, if at all. Where possible, it is important to provide the counterfactual of no targeting, for example by relying on what would happen if we would distribute benefits randomly or universally, keeping budgets constant.

Beyond the ability to reach the intended population, there are other aspects that should be considered when assessing targeting performance. Three of these aspects are legitimacy (generally speaking, the satisfaction generated by a targeting scheme), spillover-effects (how the targeting method and related program affect non-beneficiaries positively or negatively), and impacts (how a targeting method affects the effect of the targeted program). Given the research design and data requirements, few targeting studies have addressed these issues – which is also a limitation of our current study.

Generating targeting methods that bring _satisfaction and legitimacy_ to local communities can, arguably, be as or more important as generating methods that reach the poorest of the poor. This is especially relevant in ultra-poor settings where social conflict and cohesion often represent important barriers to development. Only a few studies have compared the satisfaction and legitimacy generated by PMT and CBT and found opposite results, which suggests that context or other parameters are important (Alatas et al., 2012; Premand and Schnitzer, 2020).<sup>12</sup> _Spillovers_ on non-beneficiaries is also an important issue: for instance, targeting errors are more problematic if they generate negative spillover on excluded households. There is a growing literature pointing to both positive and negative spillover effects of cash transfer programs on non-beneficiaries, whether they are economic, social or psychological (Gertler et al., 2021; Della Guardia et al., 2021; Haushofer et al., 2019; Filmer et al., 2021). The net effect of positive and negative spillovers can therefore be a key parameter to consider when deciding to target or not, and how.

The final important dimension to measure targeting performance is the _impact_ that using a targeting method makes, as targeting is not an objective in itself. Very few studies have measured the differential impact of

> 11 For instance, when comparing CBT and PMT, if 35% of the population is selected by CBT, then 35% of the population with the lowest per capita consumption is defined as “poor” (or as the target), and the PMT threshold is adjusted so that 35% of the households with the lowest PMT scores are selected.

> 12 Rigorous methods to test satisfaction and legitimacy need to be further explored, since obtaining meaningful answers about subjective issues from survey respondents is challenging.

8
targeting methods (Premand and Schnitzer, 2020). Targeting impact should not be the only element considered, because social programs also have equity objectives. However, for instance, we may prefer among equally legitimate targeting schemes the one which generates the largest positive improvements related to the policy objectives in local communities.

Although limited by the data at hand, we consider this framework in our heuristic discussion of the targeting performance of various targeting alternatives in nine programs implemented in the Sahel.

# **3 Context, targeting methods and existing evidence**

This section presents the context of social protection programs implemented in the Sahel region and reviews the existing evidence on targeting methods employed.

# _3.1 Context: Poverty and social safety nets in the Sahel_

This study focuses on the Sahel region, one of the poorest in the world that has suffered recurrent economic shocks.<sup>13</sup> Four of the countries studied have among the 10 worse human development outcomes in the world and this situation has been aggravated by the Covid-19 pandemic.<sup>14</sup> Given tremendous needs, unless budgets would substantially increase, choices need to be made on how to identify beneficiaries, even among the poor. For example, coverage rates of cash transfer programs for Chad, Burkina Faso, Niger, and Mauritania ranged between 0.4% and 1.6%. These coverage rates contrast with the high poverty rates in these countries, ranging from 38% to 45%, and with the high frequency of food crises. The limited coverage relative to needs has led, in part, to a heated policy debate on how to select beneficiaries.

Despite these facts, Sahelian countries tend to be understudied, resulting in a scarcity of evidence, which is at odds with the humanitarian and development needs of the Sahel.<sup>15</sup> This context, however, has favored the emergence of social safety nets as key elements of the poverty-alleviation strategy, with the aim of integrating emergency and development efforts and promoting long-term well-being (Bowen et al. 2020).

> 13 While climatic shocks were already pervasive in the Sahel, a study by Hallegatte et al. (2015) shows that climate change may increase dramatically the number of poor people by 2030 and affects diverse outcomes such as agricultural production, food security and health.

> 14 In the 2018 HDI ranking, Niger is #189 (last), Chad #186, Burkina Faso #183 and Mali #182. Senegal (#164) and Mauritia (#159) have only a slightly higher HDI. Among the countries reviewed in this paper, only Cameroon has a “medium” HDI (#151, last in the group) but the project studied is located in the poorest area of the country where chronic poverty reaches 72%. In addition, the Sub-Saharan Africa region expected to be home to about a third of the people who are newly impoverished by COVID-19 (World Bank 2020).

> 15 For instance, a recent review of safety nets in Africa (Ralston et al., 2017) includes only one study from Burkina Faso, while the global review on targeting by Devereux et al. (2017) and Coady et al (2014) does not include any of the countries we study. Briggs (2017a) and Porteous (2020) also show that more generally, the Sahel region tends to be under-studied based on academic publications.

9
In the last ten years, all countries in the Sahel region have launched some kind of pilot social safety-net projects, and some have already reached the first scale-up phase (Beegle et al., 2018). Cash transfer programs constitute one of the main components of the social safety-net systems implemented in the Sahelian countries. These cash transfer programs have been shown to generate various improvements in household consumption levels, food security, human capital, productive activities and resilience (Akresh et al., 2016; Stoeffler et al., 2019, Kandpal et al., 2021; Premand & Stoeffler, 2020).

However, there is a heated policy debate on the best targeting method to use in a Sahel environment, typically opposing PMT and CBT (Schnitzer, 2019). Besides, there is also a policy disagreement on the well-being metric that should be used for targeting households, namely food security or per-capita consumption. This calls for a clarification of the concepts and objectives employed in the targeting policy discussion and related analyses.

# _3.2 Targeting methods and existing evidence_

Proxy means testing (PMT) and community-based targeting (CBT) are the two most common methods employed for selecting beneficiaries of social safety-net programs in the Sahel and in other developing countries (Monchuk, 2013).<sup>16</sup> In the Sahel as well as in low-income settings where the coverage of safety net interventions is small relative to needs, PMT and CBT methods are usually applied only after geographical targeting is conducted (focusing on the poorest areas of the country in principle).

The PMT method relies on predicting household income, consumption or poverty status from a limited set of observable household characteristics (Grosh & Baker, 1995; Stoeffler et al., 2015; Brown et al., 2018). In contexts in which the means-testing of benefits is not an administratively feasible option, PMT provides the advantage of relying on information that can be measured relatively quickly and transparently. While the implementation of PMT does not vary as much (relative to CBT), Brown et al. (2018) show that the different ways to develop PMT formulas can significantly affect targeting performance.

On the other hand, CBT relies on communities to identify beneficiaries (Conning & Kevane, 2002; Beaugé et al., 2018). CBT has the advantage of leveraging community knowledge and involvement for targeting, which has the potential to improve both accuracy and legitimacy. However, CBT in some contexts is plagued by elite capture (Pan & Christiaensen, 2012; Alatas et al., 2019; Basurto et al., 2020; Stoeffler et al., 2020). There is a wide range of ways in which CBT is implemented. Among others, the degree and type of guidance provided to communities on how they should conduct the process vary greatly. In the Sahel region in particular, the Household Economy Analysis (HEA) represents a widely used type of CBT,

> 16 For good descriptions of the various targeting methods employed, see Coady et al. (2004), Del Ninno & Mills (2015) and Devereux et al. (2017).

10
especially by humanitarian agencies responding to food crises (Schnitzer, 2019). While different variations of this method also exist, it usually categorizes households into wealth groups based on specified criteria that are agreed by the community and guided by a previous, nationwide qualitative assessment of household livelihoods and food security – the ‘HEA baseline.’ Three of the CBT schemes that we assess rely on such type of CBT.

While targeting is not a recent topic in the economics literature,<sup>17</sup> empirical studies based on household survey data have emerged in recent years with the increasing popularity of social safety-net programs. A few studies have compared PMT and CBT efficiency directly and found that PMT is better at selecting households with low per-capita consumption in Indonesia, Cameroon, and Niger (Alatas et al., 2012; Stoeffler et al., 2016; Premand & Schnitzer, 2020). However, in Indonesia, CBT can generate greater community satisfaction, while the opposite result was found in Niger. Two review studies have measured the performance of PMT, CBT and other targeting methods worldwide (Coady et al., 2004; Devereux et al., 2017). Both found important variations within methods and suggest that this result is explained by important variations in implementation. These studies understate the role of widely diverse levels of coverage rates in explaining targeting performance.<sup>18</sup> While the evidence on PMT and CBT methods in the Sahel region was scarce, several studies were conducted on the subject in the past years as safety nets interventions became increasingly important. We reviewed the existing evidence based on the reports from these studies (review available upon request). Most of these studies cannot be directly compared, given variations in performance measurement, but the overall findings are largely consistent. The following conclusions emerged: (i) PMT selects households with lower per-capita consumption than CBT, but no method makes a large difference; (ii) targeting transparency and appropriation is low regardless of the method employed; (iii) hybrid methods do not work well in comparison to each individual method; (iv) little is known about the legitimacy of different methods and about the role of targeting in the overall program effectiveness.

Our study contributes to the scarce literature that both compares different methods and makes comparisons across countries in a harmonized manner, by directly analyzing household data collected from various programs implemented in the Sahel.

> 17 See for instance Besley & Kanbur (1991), Sen (1992) or Cornia & Stewart (1993) for earlier studies.

> 18 By focusing on the Sahel region and adjusting selection threshold rates when possible, we control for differences in context and coverage rates – and do not find large within-targeting method variations.

11
# **4 Data and methodology**

This section describes the nine data sets that we analyze, and the methodology that we use to measure targeting performance, informed by the framework laid out in section 2.

# _4.1 Data_

We analyze nine data sets from six countries, presented in Table 1. Two main criteria were used in choosing these data sets. First, we looked for data sets from Sahelian environments, to compare across poor, homogeneous settings. Second, the data had to include information on implemented targeting schemes of social safety nets programs. Third, data had to have been collected prior to the intervention. The data sets obtained included information on programs that employed CBT, PMT or both. The reports from the original studies for which the data were collected (Table A1) indicate several issues in the implementation of the targeting operations (e.g., CBT thresholds deviating from the original target in Cameroon). These implementation gaps are inherent to targeting and to CBT selection in particular (de Sardan & Piccoli, 2018). Since we study actual targeting outcomes, the results reflect these implementation issues. On the other hand, studies suggest that data quality is high.

In all data sets, information was collected on households in the areas of intervention, after a geographical targeting process. This geographic selection process varied across data sets: it was very narrow in Cameroon (15 villages in the poorest _commune_ of the country) but not as focused on the poorest areas in Senegal 2 and Burkina Faso 2.<sup>19</sup> Most data sets include information on per-capita consumption and on the food consumption score (FCS). These well-being measures have the advantage of being quasi-continuous. However, the Household Dietary Diversity Score (HDDS) is available in two data sets only, and in Chad, no household consumption is available (Table 1). CBT selection thresholds vary from 21% to 68%, reflecting the wide variation in coverage rates within program areas. As discussed, this calls for considering the selection threshold when comparing targeting outcomes, and for harmonizing selection thresholds within and (when possible) across countries when conducting targeting analyses.

Our data also include information on household characteristics, although the variables collected vary across data sets. We use a limited number of household demographic and housing characteristics (e.g., household size, roof material) when exploring the determinants of beneficiary selection across methods.

_4.2 Methodology: Measures and metrics_

> 19 There are two data sets each for Burkina Faso, Niger and Senegal, which we differentiate by a number (1 or 2). There is only one data set each for Cameroon, Chad and Mali.

12
Informed by our discussion in section 2, the methodology employed here focuses on making relevant comparisons across countries and on targeting methods along several well-being metrics. For that purpose, we focus on three main targeting measures. First, we compute inclusion/exclusion error rates, which is an imperfect, but simple measure widely used in the literature. Given the threshold adjustment that we impose, exclusion and inclusion error rates are equal (one excluded “poor” household means one included “nonpoor” household; see below), and we call these targeting error rates (as in Brown et al. 2018). Second, we present CGH indices, which show the contributions of targeting schemes relative to a random allocation of benefits.<sup>20</sup> Third, we look at measures that are sensitive to the distribution of the well-being metric. We show the simulated FGT poverty-rates reduction, focusing on the gap and severity. As discussed in section 2, this third measure goes beyond classifying households in binary categories (poor or non-poor), and accounts for _how_ poor people may be. In addition, we rely on the Distribution Characteristic Index (DCI) as a robustness check. This measure also considers the full distribution of the well-being metric, without fixing a poverty line – but is less straightforward to interpret, especially across countries. The results on DCI are broadly consistent with those obtained from our three main measures (see Addendum, available upon request).

The first well-being metric we focus on is per-capita consumption, which is widely employed for analyzing targeting performance. It allows comparability with other studies and contexts. Moreover, per-capita consumption is a broadly accepted indicator that aggregates over several dimensions of well-being. As consumption is continuous, we can fully adjust the well-being threshold (or poverty line) according to the targeting threshold used.<sup>21</sup> Our second well-being metric is an indicator of food security. In a Sahelian context characterized by chronic and recurring acute food insecurity, this critical indicator is largely used by humanitarian organizations. We focus on the Food Consumption Score (FCS), a widely used measure of food security that incorporates both the quantity and the variety of food items consumed, in cases where the FCS is available in our data set (Vaitla et al., 2015). Moreover, FCS values are quasi-continuous in our sample, which allows for adjusting the FCS threshold to various selection thresholds. However, in the data sets where the FCS could not be constructed, we used the Household Diet Diversity Score (HDDS), which has only a few discrete values (from 1 to 12).

While most targeting schemes rely on different selection rates, we are able to adjust selection rates for PMT schemes, given that PMT scores provide a full welfare ranking of households. When comparing PMT and CBT selections, we adjust PMT selection rates for each database to match CBT selection rates (as indicated

> 20 Specifically, CGH = b p p , where b is the share of the transfers received by the poor population, and p p is the share of the population that is poor (see Addendum, available upon request). 21 bb bb Although all the countries of our sample use a similar currency, we convert the consumption per capita into USD SS /SS SS SS PPP to make it comparable to national figures (see Table 2).

13
in Table 1). This means that if CBT selects 20% of the households in a given database, we adjust the PMT threshold to also select 20% of households. In addition, we consider the target as being 20% of the households. In other words, we define 20% of the households as “poor” or “food insecure.”<sup>22</sup> While the above-mentioned approach allows us to make meaningful comparisons between PMT and CBT _within_ databases, comparisons across databases are confounded by varying selection thresholds. For this reason, we also present harmonized selection thresholds for PMT schemes and set them all equal to 35%.<sup>23</sup> Thus, we can compare the performance of PMT schemes _between_ databases, net of thresholds effects. In addition, to better explore the role of selection thresholds in targeting performance, we compute targeting measures for each selection rate between 5% and 100% for PMT selection with per-capita consumption as a welfare metric. We reproduce the same analysis with FCS as a welfare metric as well. This also means that we can compare PMT across databases for a range of different thresholds.

Our simulations of poverty reduction are based on transfers of 15,000 CFA per capita per year approximatively ($0.2 PPP per day), which represents about 15% of the median consumption level in our sample (and about 10 to 20% of the median per capita consumption in each data set).<sup>24</sup> The simulations consist in adding the transfer amount to the per capita consumption of each household selected by a given method. A recent review of cash transfer impacts in Sub-Saharan Africa found an average marginal propensity to consume out of cash transfers of 0.67 (Beegle et al., 2018), but increasing or decreasing the amount received by each household is unlikely to affect the results qualitatively. Indeed, we compare methods with equal budgets, based on the poverty gap and severity (not the headcount ratio), and ignore dynamic behavioral and investment responses. Moreover, the objective of these simulations is to assess targeting performance in a distribution-sensitive manner, not to realistically predict cash transfer impacts.

While simulations of FGT indices reduction are usually based on per-capita consumption in the targeting literature, we extend them to food security by simulating an increase in household FCS for selected households. We assume an increase of the FCS of 7 points after transfers, which is also about 15% of the median FCS, and consistent with findings from impact evaluations of cash transfer programs in the region.

> 22 Because some food security measures are lumpy (such as HDDS), the CBT threshold does not always correspond to a food security threshold. This happened to be the case only in Mali, where 66% of the households were selected by CBT, but 55% of the households had a HDDS of 6 and 76% and HDDS of 7. In that case, the CBT threshold is still used, but the households with HDDS between 6 and 7 would be randomly allocated in the “target group”. Only in Niger 2 is the CBT threshold fully adjustable because households were ranked by the community (see Premand & Schnitzer, 2020, and for a similar community ranking, Alatas et al., 2012).

> 23 Such adjustments are possible because per capita consumption and the FCS are continuous (or largely continuous) variables. We use 35% because HDDS measures, which are less continuous, also marked a discontinuity at this threshold, for example jumping from 5 to 6 in Burkina Faso 2.

> 24 Given the household size in our data sets (7.25 on average), the amount transferred is approximatively 10,000 CFA per month per household – a typical transfer amount in the programs that we study.

14
Again, our results are not sensitive to the use of different assumptions regarding the actual FCS increase generated by social protection programs since our focus here is on targeting accuracy.

Finally, we simulate alternative selection mechanisms: random targeting and universal selection. Random targeting will consist in selecting randomly X% of the households. The threshold is adjusted to provide a relevant comparison with PMT or CBT selection in each case. Universal selection on the other hand will include all households but will deliver the same total amount of benefits as with other methods to keep program budget constant. This means that each household will receive a smaller amount of benefit compared to PMT or CBT, but everyone will receive benefits.

# _4.3 Initial conditions and descriptive statistics_

Table 2 shows initial FGT poverty indices in our nine data sets, confirming that most data sets have remarkably high levels of poverty, and poverty indices much higher than national poverty rates. This is likely, in part, a result of geographical targeting, although comparisons based on different consumption aggregates may be difficult. These initial conditions are important to keep in mind, as they may contribute to explain targeting results. Indeed, it is more difficult to select the poorest where “everyone is poor” (Ellis, 2012). The data sets are relatively homogeneous in terms of consumption and food insecurity levels<sup>25</sup> . However, Senegal 2 households are better off, followed by Burkinabe 2 households. In these two data sets, geographical targeting is not as narrow as in other data sets, which focus on smaller, poorer geographic areas.<sup>26</sup> The FCS is not as straightforward to compare across countries, but its highest values are also in Senegal 2 and Chad, followed by Niger 2. Cameroon is the country with the lowest average per-capita consumption, but food security levels are even lower in Mali (HDDS).

# **5 Results**

# _5.1 Targeting performance_

# _Performance in reaching the poorest households_

We start by presenting the performance of PMT relative to CBT targeting schemes to reach the poorest households, based on CBT selection thresholds in each database. For example, if a CBT targeting scheme selected 20 percent of the population, we adjust the PMT threshold so that 20 percent of the population is

> 25 See Figure 1 for the per-capita consumption cumulative distribution by data set. Additional descriptive statistics by data set for our key variables are presented in Table A2 in the Addendum, available upon request. 26 The Senegal 2 study was conducted to assess the targeting performance of the Unique National Registry (RNU), which was conducted nationally. The sample comes from 4 regions. Likewise, the health insurance project studied in Burkina Faso 2 was rolled out in several regions, and the sample comes from 2 distinct regions.

15
selected by PMT targeting in the same database as well, and we assess the performance of these CBT and PMT schemes to reach the poorest 20 percent of the population. As mentioned earlier, this approach enables us to make meaningful comparisons, net of threshold effects, between PMT and CBT _within_ databases.

When we rely on targeting error rates to assess targeting performance, PMT systematically selects individuals with the lowest per-capita consumption relative to CBT for all targeting schemes (Table 3). This is consistent with the literature and with the design of PMT formulas, but it is particularly striking across our nine data sets. The median targeting error for CBT is 50 percent, while that of PMT is 39 percent. This means that 50 and 39 percent of households selected by CBT and PMT targeting scheme respectively were incorrectly identified. PMT still delivers a significant improvement over a random allocation: the CGH index is 20 for PMT and 6 for CBT. In other words, the median PMT targeting scheme provides 20 percent more resources to the poor than a random allocation would, versus 6 percent for the median CBT targeting scheme.

While the median difference in the performance between PMT and CBT is large, an important variation exists across databases. This variation depends largely on the selection threshold employed in each database. The difference in the CGH index between PMT and CBT is lowest in Mali (i.e. with a difference of 7 percentage points), where the selection threshold is among the highest (at 66%). On the other hand, the difference in the CGH index between PMT (2.18) and CBT (1.47) is highest in Burkina Faso 2, where the selection threshold is lowest (21%). In sum, when the selection threshold is higher (i.e. more people selected), there is a mechanical trend due to the targeting measures used to see lower targeting errors rates (i.e. more exclusion/inclusion) and lower CGH index (i.e. less budget directed towards the poor).

When relying on distribution-sensitive measures for performance based on poverty, the differences between PMT and CBT are relatively small (Table 4). The CBT simulated poverty-gap reduction of the median targeting scheme is 3.3 versus 3.7 for PMT. The small difference in poverty-gap reductions is partly mechanical, given that an important share of households is well below the poverty line in most databases. Nevertheless, results are similar when measuring the simulated reduction in poverty severity, which values more transfers made to the poorest of the poor. The CBT poverty-severity reduction of the median targeting scheme is 3.3 versus 3.6 for PMT.

Perhaps more striking is how close the FGT poverty-reduction targeting measures are when random or universal selection is compared to CBT and PMT. When looking at the poverty gap, CBT and universal targeting are virtually identical, except in Senegal 2 where CBT is slightly above. But when looking at the poverty severity, universal targeting outperforms CBT in Cameroon and Niger 2 – and is almost identical in other countries except the two data sets with the lowest selection rates (Senegal 1 and Burkina Faso 2).

16
This is because all the poorest receive transfers with universal targeting, while some of them do not with CBT (although those who do receive higher amounts). Finally, PMT is only slightly above universal targeting in terms of poverty-severity reduction – by 0.23 to 0.73 points.<sup>27</sup>

# _Performance in reaching the most food-insecure households_

When considering food insecurity as a well-being metric using measures which are not distributionsensitive, the median PMT and CBT targeting schemes perform similarly to a random allocation of benefits (Table 3). The median CBT and PMT programs provide, respectively, 5% and 2% more resources to the most food insecure households than would a random allocation. There are however notable exceptions in two databases, where both PMT and CBT targeting schemes make a significant difference. These are the two data sets with the lowest CBT selection rates. In Burkina Faso 2, the median CBT and PMT program provide 83% and 38% more resources to the food insecure, respectively. The opposite result is found in Senegal 1, where the CBT and PMT programs provide 13% and 27% more resources to the food insecure, respectively. This opposite result may be a result of several factors, including the scope of geographical targeting and the inequality levels.

Turning to distribution-sensitive measures based on the FCS, the difference between CBT and PMT appears negligeable (Table 5). PMT targeting schemes outperform CBT schemes in all cases but one, according to the simulated reduction in the gap and the severity of food insecurity, but magnitudes are very small. Moreover, in a majority of cases (Chad, Niger 1, Niger 2, Senegal 2), a universal allocation of benefits performs better than alternative targeting schemes based on both the simulated reduction of the gap and the severity of food insecurity. For example, in Niger 2, where the difference is the largest, the FCS severity decrease for PMT is 1.08 versus 1.50 for a universal allocation of benefits.

# _PMT performance and program coverage_

In this sub-section, we assess the performance of PMT targeting schemes based on the same selection thresholds (or program coverage) across databases with two objectives in mind. The first objective is to compare the performance across PMT schemes, net of threshold effects. The second objective is to systematically explore the role of thresholds in the performance of targeting. We achieve these objectives by comparing the performance of targeting methods based on a range of harmonized selection thresholds.

> 27 In the Addendum, available upon request, we also compare PMT selection with geographic selection, i.e. selecting all households in villages with the lowest, average PMT scores (Figure A2). Geographic selection performs relatively well in some data sets and for some selection rates, and achieves error rates that range generally between PMT and random selection error rates. However, given that the programs studied are generally implemented in homogenous areas _after_ some geographical targeting has occurred, the potential of geographical targeting tested here is relatively low.

17
A downside of this harmonization exercise is that it cannot be applied to CBT, given the lack of a welfarebased ranking for all data sets except Niger 2.

The performance of PMT under a harmonized threshold tells a different story, compared to the analysis based on different thresholds (i.e. using CBT selection rates for comparison between PMT and CBT). Threshold adjustment affects both the levels of performance and the variations in performance across data sets (Table 6). The median PMT scheme delivers 38% more resources to the poorest households than a random allocation would, compared to 20% for the median PMT scheme under CBT selection rates (see Table 3). The median targeting error rates based on per-capita consumption also increases from 38.6% (see Table 3) to 51.9%. Both results are a mechanical effect of the lower threshold (35%) used in Table 6.

Figure 2, which computes targeting error rates for each selection threshold between 5% and 100% of the population, confirms the decisive role played by the selection thresholds in targeting performance. In fact, the figure illustrates that the performance of targeting varies more across selection thresholds than across targeting PMT schemes. Targeting errors decrease at a quasi-constant rate with the increase of the percentage selection in all data sets. But decreasing targeting errors do not mean an increasing contribution of targeting relative to a random allocation (as illustrated by the 45-degree line). After a selection threshold of around 30 percent, the larger the selection threshold, the smaller the contribution targeting can make relative to a random allocation.<sup>28</sup> A similar result is obtained when measuring PMT performance based on the FCS, although in most data sets targeting errors are indistinguishable from errors obtained from random selection (Figure 5).<sup>29</sup> Using a harmonized selection threshold of 35% (Table 6) confirms that the differences in performance across PMT methods exist, but these are substantially smaller than the observed differences under actual (unharmonized, CBT-based) selection thresholds. Moreover, performance systematically varies, depending on whether methods were applied in areas undergoing narrower or broader geographical selection of poor areas. For instance, where geographical targeting was relatively broader (in Senegal 1, Senegal 2, and

> 28 Figures 3 and 4 show that the poverty gap and poverty severity decrease at quasi-constant rate with the selection threshold for most countries. For datasets with lower initial poverty levels (e.g. Senegal 2), PMT performs slightly better than random or universal transfers at addressing poverty severity, but for the poorest datasets (e.g. Cameroon), universal and random selection perform just as well as PMT.

> 29 We compute the DCI index for each data set in Figure A4 in the Addendum, available upon request. Mechanically, the index has a larger value for low selection rates, because the difference between selected and non-selected households is greater when few households are selected. The DCI values depend critically on the initial distribution in each data set: it is higher in countries with more heterogeneity (Burkina Faso 2 and Mali). For other countries, DCI values are closer, and relatively flat within a data set for selection rates from 20% to 100%. For selected countries, the DCI from PMT targeting is compared with perfect targeting (only the poorest households are rightly selected), random targeting and universal selection (Figure A5 in the Addendum, available upon request). PMT targeting results in a DCI that is substantially larger than random and universal transfers (which, by definition, treat all households equally) in Burkina Faso 2 but in Mali, PMT, random and universal targeting achieve very similar results.

18
Burkina Faso 2), targeting errors are lower and the CGH indices higher, ranging from 1.58 to 1.69. On the other hand, where geographical targeting was narrower (in Niger 1, Niger 2, Mali, and Cameroon), targeting errors are higher and the CGH indices lower, ranging from 1.21 to 1.33.

Previous multi-country studies on targeting have found large variations in targeting performance across programs employing the same method, concluding that implementation plays an important role (Coady et al., 2004; Devereux et al., 2017). While these studies have included a larger and more diverse set of countries, the different programs considered also employ a wide range of selection thresholds. Our results may contrast with these two studies because we study programs in a similar environment (the Sahel region) and because we account for selection thresholds.

# _Reaching the poor (and not the poorest)_

Up to now, we have assessed the performance based on how well we reach the _poorest_ or _most foodinsecure_ households, defined by the selection threshold. That is, if a method selects 35 percent of households, we measure its performance in reaching the worst-off 35 percent of households (the poorest or most food-insecure). Nonetheless, governments may also have the objective to reach _any_ poor or foodinsecure household, especially in low-income settings presenting homogeneous populations and imprecise or costly information. If this is the case, the relevant eligibility thresholds may be the national poverty or food insecurity line. While this adjustment may seem trivial, it changes the levels of inclusion and exclusion errors drastically: in our sample of poor households, it will mechanically decrease levels of inclusion errors (there are few non-poor) and increase exclusion errors (programs do not have the resources to include all the poor).

Results show that when it comes to exclusion errors under a fixed poverty line, what matters the most by far are budgets (Figure A2 in the Addendum, available upon request). Indeed, applying a PMT performs nearly as a random selection in excluding households, and exclusion errors decrease at a constant rate as budgets increase. For inclusion errors, initial levels of poverty are a more important determinant of performance. Overall, given the high prevalence of poverty in our samples, inclusion errors remain very low, and PMT plays a greater role at low selection thresholds in avoiding these inclusion errors.

# _Insights from studying targeting methods in practice_

Program managers often simulate the performance of different PMT models to make decisions on how to select program beneficiaries. The academic literature is also largely based on the simulation of PMT methods, which themselves are drawn from nationally representative surveys, to assess their performance. Nonetheless, there are several reasons why simulation results may differ from those obtained in practice 19
when programs are implemented (Devereux et al., 2017). While implementation flaws (including implementation lags) have been mentioned as caveats emerging from basic simulations (see for example Brown et al., 2018), other factors such as geographical targeting and the arbitrary choice of selection thresholds have been largely ignored. In contexts such as the Sahel, PMT or CBT methods are almost always applied after geographical targeting has been applied to focus the program on the neediest areas. Because of this, PMT targeting is implemented in places that significantly differ from those that are used to simulate their performance – usually based on national or regional samples. If geographical targeting selects areas that are poorer and more homogenous, then simulations will likely overstate the contributions that PMT targeting or other household-level selection methods can make. Besides, a series of implementation issues affect the actual targeting outcome. This is well-documented in the literature (Devereux et al., 2017; de Sardan & Piccoli, 2018; Stoeffler et al., 2020) and in the reports we reviewed (see Table A1). Finally, simulations often rely on selection thresholds that differ from practice. Given the significant role of selection thresholds in the performance of targeting, this is not trivial.

As an illustration, we compute targeting errors for selection thresholds set at 20% and 40% and provide a comparison with results from Brown et al. (2018) who use these thresholds in Burkina Faso, Mali and Niger (Table A3 and Table A4). As expected, the level of targeting errors is much larger in our data sets than in their study based on nationally representative data sets: our targeting error rates are approximatively 50% larger, based on the same selection thresholds in the same country. In sum, our results suggest that the role of geographical targeting and other implementation issues need to be carefully considered when conducting simulations, and that results obtained based on program data are largely different from those obtained from nationally representative surveys. For future studies based on simulations, our results imply that samples should aim at mimicking areas where PMT will be applied, and to use selection thresholds similar to those employed in practice by social programs (or a wide range of thresholds).<sup>30</sup>

# _5.2 Mechanisms: Determinants of CBT and PMT selection_

To explore what drives the differences between CBT and PMT results, we explore the determinants of the selection of households with each selection method. Table 7 shows simple probit models where the dependent variable is the selection by CBT or PMT in each data set. A few independent variables that were found in most data sets and discussed in the targeting literature are included. These probit regressions are conducted for the seven data sets where we have information on both CBT and PMT selection, and PMT selection thresholds are adjusted to match CBT selection rates (see section 4).

> 30 Yet, it is not always possible to conduct simulations based on the narrow region where PMT will be applied based on nationally representative surveys due to sample size issues.

20
Results show some clear patterns suggesting that definitions of poverty by communities across countries may have similarities. Households with female heads are significantly more likely to be selected by CBT in six cases. This is not generally true for PMT, which can either discriminate towards female heads, male heads, or none. The age of the household head is also positively correlated with CBT selection, which is generally not true for PMT. Regarding the size of the households, CBT is less consistent, but always puts a lower weight on the size of the households than does PMT – which always favors the inclusion of larger households. Indeed, household size usually has a high weight in the PMT formula, as it correlates strongly with low per-capita consumption. Schooling is not significant for CBT selection (except in Senegal 1), whereas it is significant and negative in PMT selection in all six data sets for which this information is available. Finally, CBT targeting is not consistent across data sets in the weights it allows to low-quality roofs, which may or may not be correlated with selection. Again, this variable enters PMT formulas in most cases and drives PMT selection in six data sets out of seven.<sup>31</sup> Overall, the results suggest that CBT tends to select smaller, vulnerable households, putting more weights on these determinants than on per-capita consumption (compared to PMT). These findings are consistent with the targeting literature. However, as far as we know, previous studies had not measured these determinants quantitatively in a consistent manner across several data sets from different countries.

# _5.3 Targeting cost and time_

The cost of targeting is often one of the main subjects of criticism formulated against targeting and one of the arguments for promoting universal programs. However, there is little information in the literature regarding actual targeting costs, especially in Africa. Although CBT is often discussed as a cheaper alternative than PMT, CBT was found to generate approximatively the same cost as PMT in Indonesia (Alatas et al., 2012). CBT costs were also found to be relatively large in Burkina Faso (Beaugé et al., 2018).<sup>32</sup> This point should be considered when designing targeting systems. It contributes, for instance, to make hybrid targeting methods unlikely to be cost-effective. But besides this general finding, the literature offers little guidance regarding whether targeting represents a large share of program budgets, and whether a best use of these funds could be, for instance, to redistribute them directly to all households in a universal manner – possibly after applying only geographic targeting.

> 31 Pooling all data sets together for CBT and controlling for per capita consumption, FCS, or both does not affect these results (Table A6). Households which are female headed, have an older household head, are smaller and do not have a solid roof are significantly more likely to be selected by CBT, while schooling of the household head is not significantly associated with selection.

> 32 The financial cost per individual was $5.73, while the economic cost (taking time into account) was $11.83.

21
To answer this question, we collected the available administrative data for all the programs studied and from other similar programs in the region.<sup>33</sup> Table 8 shows the targeting costs per screened households (e.g., for each interview, for PMT, regardless of whether the household will be selected). It also shows the targeting cost per final beneficiary. The cost per beneficiary is higher, since not all households screened will become beneficiaries. In addition, mechanically, programs with a lower selection rate (including a smaller share of total, screened households) will have higher targeting costs per beneficiary as a share of program budget. This would be an additional argument for high selection thresholds. The last column of Table 8 indicates the targeting costs as a share of total transfers made to households by the program.

The results from each program must be taken with caution since costs are not recorded systematically and in a consistent manner. Besides, targeting costs are not always easily separable from other program costs (e.g., staff time). Finally, targeting operations often serve other purposes, such as registration of households for future payments (which needs to be conducted even for non-targeted transfers). This means that eliminating targeting is unlikely to eliminate the full amount indicated in Table 8. Despite these caveats, Table 8 shows consistent figures and findings.

The cost to screen each household is found to be similar across methods: around $5-$7 per household, whether PMT, HEA or CBT is employed. It is higher in Chad ($9.5) and lower in Senegal 2 ($3.2), and also lower when self-targeting is employed. This indicates that a reasonable, conservative rule of thumb in the Sahelian context, consistent with Beaugé et al. (2018), would be $5 to $10 per screen household. Given that selection rates vary greatly across programs, these costs result in a cost per beneficiary household ranging from $13.5 to $38.8. However, these costs remain minor compared to the total amount transferred to households: they represent 1.5% to 5.5% of total transfers.<sup>34</sup> These numbers are consistent with those from the studies reviewed by Devereux et al. (2017). Given that program costs include additional administrative costs and that targeting operations serve other purposes such as registration, targeting costs do not seem to affect in a substantive manner the amounts delivered to households. Compared to the issue of targeting performance, targeting costs remain, at best, a minor argument against targeting households.

> 33 The estimates include variable costs associated with collecting the information necessary to identify beneficiaries and exclude fixed costs that are linked to multiple program aspects other than targeting, such as government administrative costs.

> 34 These numbers do not include programs with self-targeting. Self-targeting consists in individuals applying to the program in order to receive benefits, which reduces targeting costs because (i) there is no door-to-door survey (program applicants must go to a central office), and (ii) there is usually a much smaller number of households to be screened.

22
# **6 Targeting: The way forward**

Because direct transfers to households are increasingly used as emergency or long-term development instruments, the selection of beneficiary households – targeting – is also increasingly discussed in policy and academic settings. Recurrent shocks related to climate change, forced displacement and the Covid-19 pandemic have highlighted the need to deliver benefits rapidly, cost-efficiently and at scale with limited information and infrastructure to identify vulnerable households. This paper aimed to make conceptual and empirical contributions to this discussion by focusing on the Sahel, one of the poorest and most shockprone regions globally. Consistent with previous reviews, our study emphasizes how difficult it is to compare targeting performance across countries. However, by conducting a harmonized primary analysis across nine data sets from six countries, we are able to contribute to the heated policy debate on targeting, and to make some general remarks and recommendations regarding the design and study of targeting mechanisms.

By relying on multiple performance indicators, we show that the definition of the well-being metric, the share of beneficiaries selected, and the indices used to evaluate targeting play a decisive role in targeting performance. Our analysis suggests that a much greater importance needs to be paid, explicitly, to defining these parameters in a clear and policy-relevant manner.

Even though PMT is more successful at selecting households with the lowest per capita consumption based on a binary classification of households as “poor” and “non-poor,” PMT does not generate results that differ greatly from budget-neutral alternatives such as CBT, or random or universal delivery of benefits, when considering distribution-sensitive measures of performance. The high levels of poverty and relatively low levels of inequality, combined with the geographical targeting applied before household-level targeting, may be driving these results. Also, our analysis shows that while simulations of PMT schemes are widely used by practitioners and academics, ignoring geographical targeting and arbitrarily defining selection thresholds could potentially lead to large biases in the simulated targeting performance.

Our analyses also allow some conclusions on CBT, which are often ignored by studies based on simulated targeting mechanisms using existing household data sets, since CBT is not as easily simulated as PMT selection. We show that CBT selects households that are systematically different from PMT, and that the performance of CBT is lower than PMT’s when assessed based on performance measures related to percapita consumption.

Per-capita consumption is only one of the potential welfare metrics used to define targeting objectives. Reaching food-insecure households is actually a key objective of humanitarian interventions through which hundreds of millions of dollars are spent yearly in the Sahel. Nonetheless, little is known about ways in 23
which food-insecure populations can be best reached. Our results do not suggest that PMT and CBT perform well in terms of reaching the most food-insecure households in this environment. While our analysis does not allow us to distinguish between potential explanations, this result may be due to the fact that food insecurity is largely experienced depending on geography rather than particular household characteristics; that PMT is not designed to reach food-insecure households; that per-capita consumption does not correlate well with food insecurity; that food insecurity is more variable over time and more difficult to measure; and that CBT does not weigh food insecurity as a main determinant of selection, as opposed to other factors.

Taken together, while there may be a need to select households resulting from budget constraints, in poor and homogeneous settings, household-level targeting employing PMT or CBT contributes little to poverty or food-insecurity reduction efforts. Indeed, in the areas where PMT and CBT are applied – after geographical targeting – it is unlikely that PMT and CBT will select a large proportion of households who are not in need.

While not the focus of our study, more consideration must be paid to performance measures that go beyond the question of whether we are reaching the intended population. This is especially the case in contexts such as the Sahel where, as we show, targeting measures do not make a significant difference in terms of reaching the intended population or costs. Specifically, greater attention should be paid to satisfaction and legitimacy; final development impacts; and spillover effects on non-beneficiaries. A caveat of our study is that our data do not allow us to explore these dimensions. Additional evidence on these aspects would contribute to inform policy by emphasizing the existing trade-offs between methods and targeting features, and the potential alternatives available. We hope that this paper constitutes a motivation for such studies to be conducted and to make important contributions to the policy discussion on targeting.

24