metadata: authors: 0: author: Etang Ndip,Alvin
1: author: Ervin,Paul Alan
2: author: Namara,Suleiman
count: Sudan
docna: 0: docna: A Proxy Means Test for Targeted Social
            Protection Programs in Sudan
docty: Policy Research Working Paper
abstracts: cdata!: This study seeks to support the efforts
            of Sudan in the targeting of the Sudan Emergency Safety Nets
            Project (SESNP) and any other future safety net programs in
            Sudan to those most in need. Sudan has faced numerous
            challenges for several years. Since its separation from
            South Sudan in 2011, the economy has experienced
            macroeconomic imbalances that have lasted for over a decade.
            In addition, the country has faced political instability,
            internal conflicts, and challenging climatic conditions. All
            of these challenges pose direct consequences to the poor.
            More recently, the onset of the COVID-19 pandemic and the
            Russia-Ukraine crisis have only exacerbated the economic and
            social situation in the country. In this context, the number
            of poor and vulnerable is expected to have increased
            considerably since the last reported official poverty rate,
            which accounted for 61.1% of the population in 2015. In
            light of these events, there is a growing consensus on the
            need of social safety net programs in the country. The
            proposed program, the SESNP, is expected to provide
            unconditional cash and food transfers to nearly 2 million
            Sudanese people (i.e., about 5% of the population). To
            support this program in targeting beneficiaries to improve
            the poverty impact of the program, we develop a Proxy Mean
            Tests (PMT) for Sudan based on the National Household Budget
            and Poverty Survey (NHBPS) 2014/2015. The results indicate
            that the use of a PMT could considerably improve the program
            in reaching those most in need, while reducing expenditure
            towards those with adequate resources. This could improve
            both the poverty impact, as well as the sustainability of
            the program.
display_title: A Proxy Means Test for Targeted
            Social Protection Programs in Sudan
listing_relative_url: /research/2022/11/33947500/proxy-means-test-targeted-social-protection-programs-sudan
topicv3: Poverty m1326687 695,Proxy Means Tests for Targeting Social
            Programs m1582566 4416
publishtoextweb_dt: 2022-11-30T00:00:00Z
dois: 10.1596/1813-9450-10241
author_keys: 0: author_key: 1193617
1: author_key: 1590894
2: author_key: 675196
model_extractions: Policy Research Working Paper 10241 A Proxy Means Test for Targeted Social Protection Programs in Sudan _Alvin Etang Paul Ervin Suleiman Namara_ Poverty and Equity Global Practice November 2022
Policy Research Working Paper 10241

# **Abstract**

This study seeks to support the efforts of Sudan in the targeting of the Sudan Emergency Safety Nets Project (SESNP) and any other future safety net programs in Sudan to those most in need. Sudan has faced numerous challenges for several years. Since its separation from South Sudan in 2011, the economy has experienced macroeconomic imbalances that have lasted for over a decade. In addition, the country has faced political instability, internal conflicts, and challenging climatic conditions. All of these challenges pose direct consequences to the poor. More recently, the onset of the COVID-19 pandemic and the Russia-Ukraine crisis have only exacerbated the economic and social situation in the country. In this context, the number of poor and vulnerable is expected to have increased considerably since the last reported official poverty rate, which accounted for 61.1% of the population in 2015. In light of these events, there is a growing consensus on the need of social safety net programs in the country. The proposed program, the SESNP, is expected to provide unconditional cash and food transfers to nearly 2 million Sudanese people (i.e., about 5% of the population). To support this program in targeting beneficiaries to improve the poverty impact of the program, we develop a Proxy Mean Tests (PMT) for Sudan based on the National Household Budget and Poverty Survey (NHBPS) 2014/2015. The results indicate that the use of a PMT could considerably improve the program in reaching those most in need, while reducing expenditure towards those with adequate resources. This could improve both the poverty impact, as well as the sustainability of the program.

This paper is a product of the Poverty and Equity Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at aetangndip@worldbank.org; paervin@gmail.com; snamara@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# A Proxy Means Test for Targeted Social Protection Programs in Sudan

Alvin Etang, Paul Ervin, and Suleiman Namara<sup>1</sup> World Bank<sup>2</sup> Keywords : Proxy means test; poverty; consumption expenditure; welfare; targeting.

JEL : I32, I38, H53.

1 Alvin Etang is a Senior Economist in the Poverty and Equity Global Practice of the World Bank; Suleiman Namara is a Lead Social Protection Specialist in the Social Protection & Jobs Global Practice of the World Bank. Paul Ervin is a Hazard Risk Economist at North Carolina Emergency Management. This paper was prepared as part of the Sudan Poverty Analytics ASA (P174168) under the overall guidance of Pierella Paci (Practice Manager, EAEPV). The authors are grateful to Franck Adoho (Senior Economist, EAWPV) for feedback and suggestions on an earlier draft. All remaining errors and omissions are the responsibility of the authors. The authors may be contacted at aetangndip@worldbank.org, <u>paervin@gmail.com, and snamara@worldbank.org.</u> 2 On October 25, 2021 the World Bank paused disbursements in all of its existing operations in Sudan and stopped processing any new operations due to the developments in the country. These measures were adopted in accordance with the World Bank Operational Policy 7.30 “Dealing with De Facto Governments”’. The present paper is a Bank-financed product and was concluded before October 25, 2021. However, consultations with the client were not feasible due to the aforementioned pause.
# 1. Introduction

Sudan has been experiencing multiple macroeconomic shocks, such as low growth, a wide trade deficit, and the emergence of multiple exchange rates since its separation from South Sudan in 2011. The loss of three-quarters of its oil reserves, which were located in South Sudan territory, led to a shortage of fuels and food staples, and the country became a net oil importer. Given the high fuel prices, the government implemented subsidies to protect the poor and vulnerable. However, the subsidies contributed to a further deterioration of the fiscal accounts and an increase in inflation, which has sustained macroeconomic imbalances over the last decade. The social, economic, and political discontent led to the removal of President Omar al-Bashir in April 2019 and to a transitional civilian government in September of 2019. They latter implemented an ambitious agenda of reforms, including the removal of various subsidies and unification of the exchange rate, in an effort to reverse years of structural imbalances.

In this context, the civilian government deemed social safety net programs crucial to protect the poor and vulnerable. Indeed, the transitional government implemented a quasi-universal cash transfer program – the Sudan's Family Support Program (SFSP), parallel to its agenda of economic reforms. The SFSP program was specifically designed to mitigate the expected short-term adverse effects of the economic reforms on the population and to help generate political and economic space for the reforms. The SFSP –a quasiUniversal Basic Income (UBI) cash transfer program, was expected to reach 80% of the population. It consisted of delivering cash transfers to the Sudanese families in the recipient's territory expected to be affected by economic reforms and other short-term shocks. The SFSP sought to exclude the top 20% using administrative data and other sets of exclusion criteria.<sup>3</sup> The SFSP social safety net program was paused after a military takeover in October 2021. With the onset of the COVID-19 pandemic, and the adverse economic and social effects that it brought to the country, political discontent increased in Sudan. This ultimately resulted in a military takeover in October 2021. The military government in office since October 25, 2021 has continued the efforts to achieve fiscal

> 3 For the SFSP, beneficiary eligibility determination and enrollment modality may differ in urban and rural areas. In urban areas, program eligibility was determined through exclusion of 20 percent of households using administrative data to verify income and asset ownership. A simple set of exclusion criteria were most likely to include (i) voluntary self-exclusion; (ii) mobile phone usage of voice services above certain threshold; and (iii) holding a public service position (with the possibility for waiving this criterion for low-level public servants). In rural areas, with higher concentrations of poverty, the program considered using a geographic targeting approach to identify areas that may be universally included. In such localities, families were registered and automatically enrolled without the application of exclusion criteria, given targets of reaching 80 percent of the population.

2
consolidation and to fight inflation, however, the effects of these reforms are not expected to be seen in the short-term.

Since 2019, Sudan has been experiencing soaring rates of inflation, which were partially driven by the higher energy and staple food prices (Figure 1). More recently, the Russia-Ukraine crisis that started in March 2022, has further exacerbated the country’s social and economic problems, through Sudan’s higher import costs of wheat,<sup>4</sup> given the high dependency of the country on imported wheat and food and the country’s trade connections to Russia and Ukraine.<sup>5</sup> In particular, Sudan is expected to face significant negative food security implications as food prices reach historic peaks due to the Russian invasion of Ukraine. All of the economic and social problems that have negatively impacted Sudan are expected to have increased the number of poor in the country, which represented about 61.1% of the population based on the latest available household survey in 2014/2015.

_Figure 1 – Evolution of inflation rate (%), 2019 - 2022_ <!-- Start of picture text --> 450<br>400<br>350<br>300<br>250<br>200<br>150<br>100<br>50<br>0<br>Inflation rate (%)<br>Percent<br>January-19 March-19 May-19 July-19 September-19 November-19 January-20 March-20 May-20 July-20 September-20 November-20 January-21 March-21 May-21 July-21 September-21 November-21 January-22<br><!-- End of picture text --> _Source:_ Authors’ elaboration based on data obtained from the Central Bureau of Statistics (CBS) of Sudan, and World Bank Currently, there are efforts in Sudan to rollout a new Social Safety Net Program, which seeks to assist the most food insecure households in the most affected states and localities, by providing unconditional cash and food transfers to beneficiary families in 11 states of Sudan. The Sudan Emergency Safety Nets Project

> 4 Sudan Monthly Update, March 2022, World Bank, Macroeconomics, Trade and Investment (MTI) Global Practice

> 5 Sudan imports 50 percent of its wheat from Russia and 4 percent from Ukraine.

3
(SESNP), for which the World Food Program is both a recipient of the grant and an implementing agency, involves two components: i) Safety Net Transfers to provide unconditional cash transfers, and ii) Delivery Systems, Monitoring and Evaluation, and Learning and Project Management. The estimated total beneficiaries are expected to be about two million people in Sudan (which is around 4.6 percent of the total population).<sup>6</sup> Under the first component, the project will provide cash or food transfers. The cash transfers of US$7 per person per month will be provided for a period of four months to targeted vulnerable households in localities with functioning food markets. The food transfers equivalent to US$7 per person per month will be provided to households for a period of four months in localities with poorly functioning food markets. Given the limited number of beneficiaries, in this context, an efficient targeting tool will be crucial for the selection of beneficiaries of the program.

The objective of this study is to support the efforts of Sudan in the targeting the Sudan Emergency Safety Nets Project (SESNP) to those most in need and to improve the implementation of the SESNP to reach its intended beneficiaries. To meet this objective, we highlight the feasibility of using a Proxy Means Test (PMT) for targeting social programs in Sudan by developing a PMT and presenting how a PMT can be used to improve program targeting to reach those most as need.

A PMT is a regression-based index of observable and verifiable household characteristics that serves as a proxy for household welfare, such as consumption expenditure or income. The resulting PMT score is commonly used to determine household eligibility for public assistance programs.

We focus solely on analyzing the PMT method. We note that a number of alternative targeting approaches exist, such as targeting based on self-selection (Coady and Parker 2009), nutritional risk (Reber et al. 2019), social worker evaluation (Rossell et al. 2018), multidimensional poverty (Alkire and Seth 2013), and Poverty Scorecards (Skoufias et al. 2020), as well as community-based targeting (Alatas et al. 2012; Sabates‐ Wheeler et al. 2015) and geographical targeting (Van Domelen 2007), to name a few. However, PMTs have been widely used as a targeting system in several countries, due to their efficacy at accurately targeting the poor, as well as their relatively low cost and ease of implementation compared to alternative targeting methods.<sup>7</sup> Furthermore, a PMT can be combined with other targeting methods based on program goals, for example nutritional risks, environmental or hazard risks, or educational risk, to identify individuals who

> 6 As of 2020, Sudan’s population is estimated to be 43.8 million according to statistics from the World Development Indicators (WDI). Link here.

> 7 We direct the reader to Grosh (1994), Coady, Grosh, and Hoddinott (2004), Grosh et al. (2008), Kidd and Wylde (2011), and del Ninno and Mills (2015) for additional reviews of various program targeting approaches and the pros and cons of the PMT.

4
are most in need both in terms of resources (e.g. have a low PMT score) and program support (e.g. are at a higher risk of not meeting some program objective, such as having adequate nutritional intake).

We develop a PMT for Sudan based on the latest household survey, the National Baseline Household Budget and Poverty Survey (NHBPS) 2014/2015. We first identify a set of candidate variables that best suit the country’s situation and that we believe serve as good proxies for welfare in Sudan. Based on this set of variables, we then perform statistical analyses to determine the final candidate variables to be included in the PMT. This step includes the estimation of different model specifications. Later, we evaluate the out-ofsample performance of the model across different PMT cutoff scores and use several evaluation metrics to evaluate the targeting performance of the PMT. Given the recent and rapid changes of the Sudan economy, this PMT methodology can readily be applied to new household survey data when it becomes available.

Compared to social program targeting based on a uniform allocation, where everyone has the same chance of being included, the results show that a PMT should be considered for targeting social protection programs in Sudan.<sup>8</sup> The PMT would assist social programs to reach a higher share of the poor and vulnerable, while reducing expenditure going towards households that are less in need, improving both the poverty impact, as well as the sustainability of the program. We explore various specifications of the PMT and find that our proposed Sudan national PMT performs as well as a rural and urban specific PMT, making it easier to implement and manage in practice. Furthermore, costly and time-consuming employment information does not improve the accuracy of the PMT in a meaningful way. Finally, our proposed Sudan PMT relies on about 20 variables that could be collected by a short survey in the field on a large portion of the population.

The rest of the study proceeds as follows. Section 2 briefly describes Proxy Means Tests and provides an overview of program targeting considerations. Section 3 presents the data used in the study and the methodology followed. Section 4 presents the main results of the study, while Section 5 presents robustness analyses. Section 6 concludes.

# 2. Proxy Means Tests (PMT) and Program Targeting

A PMT is an index of observable and verifiable household characteristics that serves as a proxy for household welfare, such as consumption expenditure or income. In general, a PMT can be any weighted

> 8 Uniform allocation is a natural counterfactual for assessing a PMT and has been used as a baseline of comparison in other PMT studies (Brown, Ravallion, and Van de Walle 2018). We discuss this further in the methodology section.

5
function of household characteristics, such as asset ownership, housing quality, education levels, etc., that produces a score used to rank households by their estimated level of welfare, when verifiable consumption or income data are not readily available and costly to obtain. Various methods exist for identifying and aggregating household characteristics to produce a PMT score (Coady, Grosh, and Hoddinott 2002; Chen and Schreiner 2009). However, it has become common to use statistical, regression-based methods to identify weights in the PMT from regression coefficients of household consumption or income using household survey data (Brown, Ravallion, and Van de Walle 2018). The PMT weights are then applied to information collected from program applicants to construct a household specific PMT score, which is an out-of-sample prediction of consumption expenditure or income, depending on the welfare variable the National Statistics Office measures. The resulting PMT score is thus an estimate of the household welfare variable, which national poverty and welfare measures are typically based. The PMT score is then used to determine household eligibility for social programs based on a predetermined cutoff (i.e., the PMT score cutoff, similar to a poverty line) and may also be used to determine the level of benefit.

While universal social protection programs that cover the entire population may be desired, they are often not feasible given limited budgets, particularly in developing countries. The use of a PMT to target social programs has been shown to improve targeting incidence and reduce the inclusion of people unintended for public program support, improving program efficiency (Grosh, 1994). Furthermore, because the PMT ranks households by their estimated welfare level, the same tool can be used for targeting various welfare programs. A common targeting approach for the various social programs, based on a single PMT score, can improve consistency across programs, reduce overlap and duplication of benefits, and promote future harmonization of public programs (Sebastian et al. 2018). Due to these benefits, PMTs are now widely used in administering various social programs in numerous countries throughout the world (Grosh, 1994; Mills, Del Ninno, and Leite 2015; Brown, Ravallion, and Van de Walle 2018; Sebastian et al. 2018).

Program eligibility determined by a PMT, however, is susceptible to inaccuracy, because it is an estimate or prediction of a household’s welfare. Errors may arise both by including ineligible applicants whose PMT score predicts them to be poor when in fact they are not (i.e., inclusion errors), and excluding eligible applicants whose estimated PMT score predicts them to be not poor when in fact they are (i.e., exclusion errors). These inaccuracies are due to prediction error arising primarily from unmeasured and costly to collect socioeconomic factors and idiosyncratic behavior, as well as model selection and potential measurement error in the data (Gazeaud, 2020).

6
It is generally the case that verifiable information on program applicants’ level of consumption or income will not be available due to the difficulty and cost of collecting such information. Careful data collection and statistical modeling are required to ensure an accurate prediction of household welfare. Nevertheless, due to these inaccuracies, it is important for programs to include a grievance or appeals process available to excluded program applicants who may like to provide additional information in support of their application or have their case reviewed for potential errors (Sebastian et al. 2018).

Even with careful data collection and statistical modeling, the PMT score is still a prediction of a households’ actual welfare and subject to inaccuracies. The magnitude of inclusion and exclusion errors depends on the program eligibility threshold (the PMT score cutoff) and the true level of poverty rate in a country. Considering a fixed poverty rate in a country, at one extreme, a program that does not include any beneficiaries will have zero inclusion error, but an exclusion error equal to the poverty rate. Likewise, at the other extreme, a program that includes everyone will have zero exclusion error, but an inclusion error equal to percentage of the population not living in poverty. In practice, the PMT score cutoff can be selected to balance these errors based on policy and program objectives.

# 3. Data and Methods

The development of the Sudan PMT proceeds as follows: First, we identify a set of candidate variables from household survey data that we believe could serve as a good proxy for a household’s welfare—in the case of Sudan this is household consumption expenditure. Second, we perform statistical analyses to determine a subset of variables to be used in the final PMT. In this step, we also explore different model specifications. Third, we evaluate candidate models’ out-of-sample predictions across different PMT cutoff scores using several evaluation metrics. Finally, we explore the performance of hypothetical targeting methods and transfer schemes using the final PMT.

## _Data and Variables_

That data used to develop the Sudan PMT comes from the National Household Budget and Poverty Survey (NHBPS 2014/2015) conducted by the Sudan Central Bureau of Statistics.<sup>9</sup> The survey includes modules on consumption, household characteristics, education, employment, housing, and asset ownership. The survey, representative at the national level, as well as states and rural and urban areas, includes 11,953 households.

> 9 This is the most recent household survey data available at the time of this study.

7
Candidate variables were selected from the NHBPS 2014/2015 based on a review of the existing literature, hypothesized correlations to consumption, and the following criteria important for the field implementation of the PMT (Coady, Grosh, and Hoddinott 2004; Brown, Ravallion, and Van de Walle 2018; Sebastian et al. 2018):

1. It should be feasible and cost-effective to collect the required data on a variable for a significant share of the population in a timely manner.

2. Each variable in the PMT should be easy to measure and observe.

3. Each variable in the PMT should be difficult for the household to manipulate.

While it may be the case that surveyed household consumption is a more accurate measure of welfare, it is often difficult and costly to obtain such information from a large number of program applicants.<sup>10</sup> Additionally, if variables in the PMT are not observable and verifiable, households may have an incentive to hide or understate their assets and wealth in order to qualify for public assistance.

Based on these criteria, a broad range of candidate variables were selected from the household survey covering household head characteristics, household population characteristics, asset ownership, and employment. Table A1.1 in Annex 1 provides a list of the candidate variables included in this analysis along with their definitions. Summary statistics of for all candidate variables are presented in Table A1.2 in Annex 1.<sup>11</sup> In this paper, we propose a PMT and show how the use of a PMT could improve program targeting. The final design of the PMT targeting system and final administration, including the final choice of variables and survey forms, should be developed in consultation with the government agency tasked with managing the targeting system. Certain variables may be more easily manipulatable or difficult to verify in different country contexts.

Guided by the expected program design in Sudan and robustness analyses, we exclude state and area (e.g., rural and urban) indicator variables, as well as employment variables, since welfare programs in Sudan will be administered at the state level,<sup>12</sup> and employment information is often difficult to obtain and potentially manipulatable by households. The impact of these decisions on the PMT are presented in Section (5).

> 10 For example, the purchases and consumption module in the Sudan NHBPS 2014/2015 is approximately 20 pages long and consists of hundreds of questions.

> 11 A small percentage of observations were missing for several variables. We explore model results for dropping missing observations and imputation. We found minor changes and high parameter stability between the two methods (Annex 2, Table A2.1). Therefore, all missing observations are imputed based on modes and means, where applicable.

> 12 Geographic targeting will be a first tool to identify areas with a high prevalence of potential beneficiaries, e.g., areas with high poverty rates or nutritional risks, and allocate program budget shares based on program goals.

8
Furthermore, we do not include candidate variables that are applicable to only one segment of the population, such as farming assets and crop production, that only apply to household farmers. Although we do include categorical variables on the sources of livelihood (e.g., farming, aid, wages, etc.). Finally, we avoid variables that capture a temporary state of the household, such as a recent sickness or other shock to the household. However, a PMT could be designed for specific use cases or combined with additional information to target such households, e.g., households impacted by recent hazardous weather and that have a low PMT score.

A limitation of this analysis is that the 2014/15 household survey is outdated, particularly with the multiple shocks Sudan has experienced in recent years. With support from the World Bank, the Sudan Central Bureau of Statistics has started preparing a new household budget and poverty survey to update poverty and other socio-economic statistics. The survey was due to be fielded in 2022, but has been paused following the military takeover in October 2021. Survey preparations should resume once things return to normalcy. The PMT methodology presented in this paper can readily be applied to new household survey data when it becomes available.

# _Methodology_

In an effort to avoid overfitting on specific in-sample characteristics of the data, we split the household survey data into a training and a testing set using a stratified random sample based on the outcome variable of log per capita household consumption expenditure (Kuhn and Johnson 2019, Chapter 3).<sup>13</sup> All modeling decisions are based on the training data, before arriving at a small number of potential models that are applied to the testing data to explore model performance on unseen, out-of-sample information. Statistical analyses are performed using the statistical programming language R and the Tidymodels package (R Core Team 2022; Kuhn et al. 2020).

To select a subset of variables from all candidate variables, we use the Least Absolute Shrinkage and Selection Operator (LASSO) estimator (Tibshirani 1996). This is done to develop a parsimonious PMT formula that can be more easily implemented in the field. The LASSO estimator selects a subset of candidate variables by minimizing the sum of squared errors conditional on a tuning parameter that penalizes the number of parameters in the model. The tuning parameter is chosen to minimize the mean squared prediction error using split sample cross validation (CV) to reduce overfitting.<sup>14</sup> This is a primary advantage

> 13 We use a 75% and 25% split of household survey data for the training and testing set, respectively.

> 14 Julia Silge provides an excellent example of developing a LASSO model and tuning the penalty parameter using CV on her blog at <u>https://juliasilge.com/blog/lasso-the-office/</u>

9
to the LASSO method over common stepwise variable selection methods, which may overfit on in-sample information.

We then use Ordinary Least Squares (OLS) regression of the log of per capita household consumption on the subset of variables identified by the LASSO estimator to develop the PMT. This OLS post-LASSO estimator performs at least as well as the LASSO estimator, but with less bias (Belloni and Chernozhukov 2013) and has been applied in analyses of other topics, such as wage regressions (Böheim and Stöllinger 2021). We have chosen this method, because OLS is common in the PMT literature due to its ease of interpretation and implementation (Brown, Ravallion, and Van de Walle 2018; Sebastian et al. 2018). While additional statistical and machine learning methods may be able to improve prediction accuracy (McBride and Nichols 2018), many of these models are often difficult to implement and interpret, especially as their complexity grows (Petch and Nelson 2021). Furthermore, it is important that the determination of beneficiaries is defensible and transparent or there is a potential for social unrest and erosion of credibility in public administration (Cameron and Shah 2014; Brown, Ravallion, and Van de Walle 2018).<sup>15</sup> The performance of the resulting PMT models is evaluated based on a number of accuracy criteria for PMT cutoff scores set to include the bottom 20%, 40%, and 60% of the observed household consumption distribution. These PMT cutoff scores were chosen to evaluate the PMT’s predictive ability to identify households living in a range of poverty—from approximately extreme poverty to total poverty.<sup>16</sup> The PMT evaluation criteria include total accuracy, poverty accuracy, undercoverage, and leakage commonly found in the PMT literature (McBride and Nichols 2018; Brown, Ravallion, and Van de Walle 2018). Table 1 below presents a confusion matrix for the possible poverty prediction outcomes based on the PMT. These include true positives (TP), where the PMT correctly identifies households living in poverty; false negatives (FN), where the PMT predicts a household to be non-poor when in fact they are; false positives (FP), where households are predicted to be in poverty when in fact they are not; and true negatives (TN), where households are correctly identified as non-poor.

Table 2 presents a list of the PMT evaluation criteria and their definitions. Total accuracy (TA) is the sum of correctly predicted poor (TP) and the correctly predicted non-poor (TN) as a percentage of all households in the sample (TP+FP+FN+TN). Poverty accuracy (PA) is the correctly predicted poor (TP) as a percentage

> 15 The PMT formula and weights may intentionally be kept secret from the public to reduce applicants’ incentives to inaccurately report household information in order to improve their chances of receiving public assistance. In this case, transparency remains important so that independent agencies may audit and verify social program targeting methods.

> 16 The World Bank’s Poverty & Equity Brief (Oct. 2021) reports 2014 poverty rates in Sudan at 12.2% under a $1.90 (2011 PPP) per day per capita poverty line, 44% under a $3.20 (2011 PPP) per day per capita poverty line, and 79.3% under a $5.50 (2011 PPP) per day per capita poverty line (World Bank, 2021).

10
of the true observed poor (TP+FN). Undercoverage (UC) is the incorrectly predicted non-poor (FN) as a percentage of the true observed poor (TP+FP). Lastly, leakage (LE) is the incorrectly predicted poor (FP) as a percentage of the true observed non-poor (FP+TN).

Table 1. PMT Poverty Prediction Outcomes

|dicted|Poor|Ob<br>Poor<br>True Positive (TP)|served<br>Non-Poor<br>False Positive (FP)|
|---|---|---|---|
|Pre|Non-Poor|False Negative (FN)|True Negative (TN)|

Table 2. PMT Evaluation Metrics

|Evaluation Criteria Metric|Definition|
|---|---|
|Total Accuracy (TA)|TA = (TP+TN)/(TP+FP+FN+TN)|
|Poverty Accuracy (PA)|PA = TP/(TP+FN)|
|Undercoverage (UC)|UC = FN/(TP+FN)|
|Leakage (LE)|LE = FP/(FP+TN)|

Additional metrics, for example inclusion error and exclusion error rates, as well as metrics that apply (normative) penalties to poverty accuracy based on leakage and undercoverage rates, such as the Balanced Poverty Accuracy Criterion (IRIS Center 2005); are common in the PMT literature (McBride and Nichols 2018; Brown, Ravallion, and Van de Walle 2018). We do not present these metrics, because they are very similar to our chosen metrics in Table 2. Finally, we note that leakage is defined in various ways in the PMT literature (McBride and Nichols 2018; Brown, Ravallion, and Van de Walle 2018). We have adopted a definition of leakage known as the false positive rate in the statistics literature. This allows to understand the relative contribution of poverty accuracy (PA) and leakage (LE) to our sample of identified beneficiaries (B), where B=N*[PA*p+LE*(1-p)], given a population (N) and a poverty rate (p).

For our baseline of comparison, we compare our PMT results to a uniform allocation targeting strategy, where each household has the same probability of being targeted. Uniform allocation has been used as a baseline of comparison in other PMT studies (Brown, Ravallion, and Van de Walle 2018). To assess targeting performance under different model specifications, we compare targeting performance metrics across models. It was not possible to compare PMT targeting performance to the previous SFSP social safety net 11
allocation method, because the targeting strategy was being refined up until the program was stopped in 2021.

The final PMT was chosen based on the above evaluation criteria and program considerations, namely a tradeoff between model accuracy and complexity. We find that, in general, more complex models (i.e., including more variables) only marginally improve targeting accuracy. However, the inclusion of more information raises the cost of obtaining information and administering the program. Therefore, we propose a parsimonious model (i.e., one that uses less variables) that provides high targeting accuracy.

# 4. Results

To arrive at a final model for the Sudan PMT, we split the data into a training and testing set using a 75% training and 25% testing split. In an effort avoid overfitting on in-sample characteristics of the sample data, all modeling decisions, including any variable transformations or groupings, were based on the training data before arriving at a small number of potential models that were applied to the testing data to explore model performance on unseen, out-of-sample observations. This is done in an effort mimic how the PMT would be applied in practice and to understand out-of-sample predictive performance of the PMT. After exploring model performance on out-of-sample data, the top performing models were selected and estimated using the complete sample of data.

In an effort to improve predictive accuracy of the Sudan PMT, we analyzed the raw household survey data and selected and transformed variables to better capture the relationship with per capital household consumption and the country context.<sup>17</sup> For all categorical variables, we explored each category for intuitive groupings that were related to household income using cross-tabulations and boxplots. For continuous variables, we explored non-linear relationships and, if non-linear relationships were present, we applied transformations, so that the relationship between continuous predictors and the outcome variable of log per capita household consumption was approximately linear. Examples of these analyses are found in Annex 4.

Table 3 below presents the main results of the Sudan national PMT. During the modeling process, we found that a more parsimonious model, i.e., one with less variables and parameters, performed similar to the complete model, which included all candidate variables. Thus, the final model presented below retains the

> 17 For example, the level of schooling exhibited a stronger relationship with household consumption than years of schooling, due to grade repetitions increasing years of schooling, but not the level earned.

12
most important variables identified during the modeling process (See Table A3.1 in Annex 3 for an evaluation of the two models).

Table 3. National PMT OLS Regression of Log Per Capita Household Consumption

|Nati|onal|
|---|---|
|Variables<br>Estimate|Std. Error|
|Household size: 2 members<br>-0.432***|(0.037)|
|Household size: 3 members<br>-0.700**|(0.036)|
|Household size: 4 members<br>-0.902***|(0.036)|
|Household size: 5 members<br>-1.061***|(0.037)|
|Household size: 6 members<br>-1.181***|(0.037)|
|Household size: 7+ members<br>-1.379***|(0.037)|
|Dependency ratio<br>-0.090***|(0.015)|
|Max. school level: less than Secondary 4<br>-0.027***|(0.009)|
|Max. school level: less than Primary<br>-0.042***|(0.01)|
|Livelihood source: farming, aid, other<br>-0.028***|(0.007)|
|Head: female<br>-0.077***|(0.009)|
|Rooms per household member<br>0.055***|(0.007)|
|House material: mud, wood, straw, tent, or incomplete<br>-0.093***|(0.009)|
|Floor material: earth<br>-0.066***|(0.013)|
|Lighting source: grass, firewood, or none<br>-0.063***|(0.008)|
|Drinking water source: not common network<br>-0.029***|(0.009)|
|Cooking fuel: cow dung, grass, or does not cook<br>-0.134***|(0.009)|
|No motorcycle<br>-0.055***|(0.017)|
|No motor vehicle<br>-0.221***|(0.016)|
|No radio<br>-0.055***|(0.008)|
|No computer<br>-0.174***|(0.022)|
|No refrigerator<br>-0.073***|(0.012)|
|No television<br>-0.051***|(0.011)|
|No AC<br>-0.055***|(0.016)|
|No mobile phone<br>-0.147***|(0.008)|
|Intercept<br>10.727***|(0.046)|
|Obs.<br>11,947||
|Adj. R-Squared<br>0.579||
|_Notes:_Variable definitions and reference categories for categorical variables are pre<br>Table A1.1. in Annex 1. A small percentage of missing categorical and continuous ob<br>were imputed by modes and means, respectively. *** p<0.01, ** p<0.05, * p<0.10.|sented in<br>servations|

Furthermore, state and area (e.g., rural and urban) indicator variables, as well as employment variables, were excluded from the final model, since welfare programs in Sudan will be administered at the state 13
level, and employment information is often difficult to obtain and potentially manipulatable by households. The impact of the decision to exclude this information is explored further in Section (5).

The final Sudan national PMT presented in Table 3 has an adjusted R-Squared just above 0.57, which is in the range of 0.40 to 0.65 commonly reported in other PMT studies (Brown, Ravallion, and Van de Walle 2018; Sebastian et al. 2018). The number of household members is the largest contributor to the PMT score, with larger households having a lower score. This is typical of PMT models because welfare is defined in per capita terms and, thus, household size tends to be an important predictor of welfare (Sebastian et al. 2018). Additional socioeconomic and demographic characteristics of household members contribute to the PMT score. For example, the dependency ratio, measured as the ratio of the sum of household members younger than 15 and 65-years-old and older to total household members, contributes negatively to the PMT score. Lower schooling levels among household members are also associated with a lower household PMT score and households that rely on external aid or farming have a lower PMT score. Households headed by a female also have a lower PMT score.

Housing and asset ownership contribute to the household’s PMT score in expected ways. Houses with more rooms per household member have higher PMT scores, while houses made of mud, wood, and straw, as well as with dirt floors, have lower PMT scores. Lack of access to household services, such as lighting, drinking water, and improved cooking fuels, was also associated with lower PMT scores. Additionally, not having assets, such as a motorcycle, motor vehicle, radio, computer, refrigerator, television, air conditioner (AC), or mobile phone, was also associated with a lower PMT score.

To evaluate the performance of the national PMT in identifying the poor for inclusion in welfare programs, we explore the PMT’s performance in identifying the bottom 20%, bottom 40% and the bottom 60% of the population. To do this, we set the PMT cutoff score equal to the observed per capita household consumption that identifies this group in the bottom of the consumption distribution, although in practice this PMT cutoff score could be set to any value based on program size and program objectives.

For comparison purposes, we evaluate model performance relative to randomly drawing beneficiaries from the sample of households surveyed in the NHBPS 2014/2015.<sup>18</sup> Table 4 below presents targeting results based on a random sample of household beneficiaries. In this case each household has a probability of being included in the social program equal to the household poverty

> 18 It may be the case that in practice a program not using a PMT could do better than randomly admitting applicants from the population, but a program could also do worse by admitting wealthy applicants not intended for the program.

14
rate. Therefore, poverty accuracy and leakage will approximately be equal to the household poverty rate used to determine beneficiaries.

Table 4. Evaluation of a National Random Sample (Uniform Allocation Strategy for Baseline of Comparison)

||**Poverty Line/ PMT**<br>|**Cutoff**<br>||
|---|---|---|---|
|**Evaluation Criteria**|**Bottom 20%**|**Bottom 40%**|**Bottom 60%**|
|Povertyheadcount(observed)|20.00%|40.02%|60.00%|
|Householdpovertyrate(observed)|19.08%|37.20%|55.50%|
|Household poverty rate (predicted)|19.42%|37.31%|55.53%|
|Total accuracy|68.64%|53.35%|50.62%|
|Povertyaccuracy|18.69%|37.44%|55.54%|
|Undercoverage|81.31%|62.56%|44.46%|
|Leakage|19.59%|37.23%|55.52%|

_Notes:_ The predicted household poverty rate is determined by randomly selecting a proportion of households equal to the observed household poverty rate.

Table 5 presents targeting results based on the Sudan national PMT. Here program beneficiaries are determined by the PMT and a PMT cutoff score based on the observed household consumption that identifies the bottom percent of the distribution. We see that choosing this poverty line often underpredicts true poverty at low levels and overpredicts at higher levels. However, compared to randomly accepting beneficiaries, we see that poverty accuracy and leakage are significantly improved. For example, poverty accuracy with the PMT increases by over 15 to 30 percentage points depending on the targeted population living in poverty, with larger increases in performance associated with a larger population living in poverty and/or eligible for the social program. Likewise, leakage falls by a similar magnitude, again depending on the targeted population living in poverty. However, in this case, leakage increases with stricter PMT score cutoffs. It is also worth noting that in a random sample the total accuracy is much lower (50% to 68%) to the total accuracy of the PMT, which ranges between 78% and 83% over various PMT cutoff scores.

Table 6 presents the results of a rural and urban PMT. The same modeling process as the national PMT was used to create individual rural and urban PMT models. The results are similar for both rural and urban areas. The urban model has a higher adjusted R-squared (0.65) than the rural model (0.52), although parameter estimates between the two models do not vary much where they coincide. The main difference between the rural and urban models is that several variables, conditional on the other variables, were estimated to not be statically significantly related to the PMT score in one of the areas.

15
Table 5. Evaluation of the National PMT (OLS Regression)

||PovertyLine/PM<br>|T Cutoff<br>||
|---|---|---|---|
|Evaluation Criteria|Bottom 20%|Bottom 40%|Bottom 60%|
|Poverty headcount (observed)|20.00%|40.02%|60.00%|
|Household poverty rate (observed)|19.08%|37.20%|55.50%|
|Household poverty rate (predicted)|11.38%|37.01%|58.92%|
|Total accuracy|83.19%|78.29%|80.25%|
|Poverty accuracy|35.76%|70.57%|85.29%|
|Undercoverage|64.24%|29.43%|14.71%|
|Leakage|5.63%|17.14%|26.03%|

_Notes_ : The poverty line and PMT cutoff score are set at the observed consumption expenditure that identifies the bottom 20%, bottom 40%, and bottom 60% of the population. The PMT cutoffs are the log of these values.

For example, in urban areas, the gender of the household head, waste disposal, motorcycle ownership, and television ownership did not exhibit a strong relationship with per capita household income. While in rural areas, the age and marital status of the household head, floor material, and ownership of an air conditioner or satellite dish was not strongly related to per capita household income. Therefore, these variables were omitted from their respective PMTs.

To explore if an area specific PMT improves targeting performance, we evaluate the targeting performance of rural and urban specific models compared to the national model applied to rural and urban areas. Table 6 presents the rural and urban targeting evaluation by model. The total accuracy, poverty accuracy, and leakage of the models are nearly identical. The main difference in the models appears to be in the improved poverty accuracy of the urban model in identifying the bottom 20%, with a poverty accuracy of 47% compared to 40%. However, because the target population (the bottom 20%) is relatively small, the model performance is minimal in absolute number terms. For example, in a sample of 100 households, if 20 are poor (20%), then the urban model would identify 9 people (20*0.47) as poor versus 8 people (20*0.40) using the national model. Likewise, the slightly larger leakage of the urban model would result in including nearly an extra non-poor in the program per 100 people in the population, because the non-poor comprises a large portion of the sample (the top 80%).

16
Table 6. PMT OLS Regression of the Log of Per Capita Household Consumption by Area

||**Ru**|**ral**|**Ur**|**ban**|
|---|---|---|---|---|
|**Variables**|**Estimate**|**Std. Error**|**Estimate**|**Std. Error**|
|Household size: 2 members|-0.442***|(0.041)|-0.491***|(0.082)|
|Household size: 3 members|-0.721***|(0.041)|-0.727***|(0.081)|
|Household size: 4 members|-0.93***|(0.041)|-0.908***|(0.081)|
|Household size: 5 members|-1.08***|(0.042)|-1.086***|(0.081)|
|Household size: 6 members|-1.203***|(0.042)|-1.198***|(0.081)|
|Household size: 7+ members|-1.395***|(0.042)|-1.397***|(0.081)|
|Dependency ratio|-0.092***|(0.019)|-0.107***|(0.025)|
|Max. school level: less than Secondary 4|-0.028**|(0.012)|-0.022*|(0.014)|
|Max. school level: less than Primary|-0.041***|(0.012)|-0.046***|(0.016)|
|Livelihood source: farming, aid, other|-0.03***|(0.009)|-0.027*|(0.015)|
|Head: female|-0.093***|(0.012)|||
|Head: age|||-0.001***|(0.000)|
|Head: not married|||-0.067***|(0.016)|
|Rooms per household member|0.033***|(0.009)|0.121***|(0.012)|
|House material: mud, wood, straw, tent, or incomplete|-0.079***|(0.013)|-0.099***|(0.012)|
|Floor material: earth|||-0.06***|(0.015)|
|Lighting source: grass, firewood, or none|-0.075***|(0.01)|-0.088***|(0.017)|
|Drinking water source: not common network|-0.032**|(0.013)|-0.027**|(0.013)|
|Cooking fuel: cow dung, grass, or does not cook|-0.155***|(0.012)|-0.098***|(0.016)|
|Waste disposal: heap, burning, other|-0.067***|(0.017)|||
|No motorcycle|-0.064***|(0.023)|||
|No motor vehicle|-0.255***|(0.023)|-0.184***|(0.022)|
|No radio|-0.07***|(0.01)|-0.027**|(0.012)|
|No computer|-0.152***|(0.049)|-0.178***|(0.023)|
|No refrigerator|-0.054***|(0.018)|-0.096***|(0.015)|
|No television|-0.039***|(0.015)|||
|No AC|||-0.074***|(0.018)|
|No mobile phone|-0.145***|(0.01)|-0.139***|(0.019)|
|No satellite dish|||-0.052***|(0.015)|
|Intercept|10.697***|(0.07)|10.779***|(0.09)|
|Obs.|8,357||3,590||
|Adj. R-Squared|0.523||0.652||

_Notes:_ Variable definitions and reference categories for categorical variables are presented in Table A1.1. in Annex 1. A small percentage of missing categorical and continuous observations imputed by modes and means, respectively. *** p<0.01, ** p<0.05, * p<0.10.

17
Table 7. Evaluation of Different Models by Area (OLS Regression)

## **Performance by Area using the National Model**

||**Rural**<br>**Poverty Lin**<br>|**e/ PMT Cutoff**<br>|<br>|**Urban**<br>**Poverty Lin**<br>|**e/ PMT Cutoff**<br>|<br>|
|---|---|---|---|---|---|---|
|**Evaluation Criteria**|**Bottom**<br>**20%**|**Bottom**<br>**40%**|**Bottom**<br>**60%**|**Bottom**<br>**20%**|**Bottom**<br>**40%**|**Bottom**<br>**60%**|
|Poverty headcount (observed)|20.00%|39.97%|59.99%|20.04%|40.02%|59.95%|
|Household poverty rate (observed)|18.89%|37.11%|55.14%|16.77%|34.96%|53.82%|
|Household poverty rate (predicted)|8.50%|37.56%|59.51%|9.58%|32.79%|56.52%|
|Total accuracy|82.18%|76.33%|77.83%|87.19%|80.84%|81.20%|
|Poverty accuracy|25.33%|68.72%|83.85%|40.37%|69.48%|85.04%|
|Undercoverage|74.67%|31.28%|16.15%|59.63%|30.52%|14.96%|
|Leakage|4.57%|19.18%|29.58%|3.38%|13.06%|23.28%|

## **Model Performance by Area using Rural and Urban Models**

||**Rural**<br>**Poverty Line**<br>|**/ PMT Cutoff**<br>|<br>|**Urban**<br>**Poverty Lin**<br>|**e/ PMT Cutoff**<br>|<br>|
|---|---|---|---|---|---|---|
|**Evaluation Criteria**|**Bottom**<br>**20%**|**Bottom**<br>**40%**|**Bottom**<br>**60%**|**Bottom**<br>**20%**|**Bottom**<br>**40%**|**Bottom**<br>**60%**|
|Poverty headcount (observed)|20.00%|39.97%|59.99%|20.04%|40.02%|59.95%|
|Household poverty rate (observed)|18.89%|37.11%|55.14%|16.77%|34.96%|53.82%|
|Household poverty rate (predicted)|7.78%|37.12%|59.90%|11.75%|34.09%|56.49%|
|Total accuracy|82.23%|76.63%|78.08%|87.35%|81.03%|81.84%|
|Poverty accuracy|23.56%|68.53%|84.44%|47.34%|71.63%|85.61%|
|Undercoverage|76.44%|31.47%|15.56%|52.66%|28.37%|14.39%|
|Leakage|4.10%|18.59%|29.74%|4.59%|13.92%|22.56%|

_Notes:_ The poverty line and PMT cutoff score are set at the observed consumption expenditure that identifies the bottom 20%, bottom 40%, and bottom 60% of the population. These values are 3,181, 4,173, 5,278 SDG in rural areas and 3,958, 5,259, 6,734 SDG in urban areas. The PMT cutoffs are the log of these values. The national model is presented in Table 3 and the rural and urban models are presented in Table 6.

The results show that a PMT could help social programs in Sudan reach a higher share of the poor and vulnerable, while reducing expenditure going towards households that are less in need. The national PMT, which relies on 19 variables, performs nearly as well as a rural and urban specific PMT, making it easier to implement and manage in practice. Additionally, due to requiring less information than individual rural and urban models, the information needed for the national PMT could more easily and rapidly be collected by a short survey in the field on a large portion of the population.

# 5. Robustness

To develop the PMT a number of decisions have to be made on the statistical model to be used and the variables to be included. In this section, we explore how different models and variables effect the targeting 18
performance of the PMT. We explore a binary dependent variable model, which models the probability of a discrete outcome, in this case household poverty status. Then we explore, several specifications that add a complete set of state and rural and urban indicator variables (i.e., dummy variables), as well as employment information, in the form of the proportion of number of household members who have ever worked for income to the total household members.

Table 7 presents the evaluation metrics of the PMT using a binary dependent variable logistic model compared to the OLS regression model used in Section 4.<sup>19</sup> For comparison purposes, the PMT cutoff score for the logistic model was set so that the predicted household poverty rate was similar under both models. The results for total accuracy, poverty accuracy, undercoverage, and leakage, are very similar and always within 2 percentage points. Therefore, it does not appear to make much difference whether OLS or logistic regression is used to develop the PMT on targeting performance. However, in practice, OLS is easier to implement and interpret due to its linear functional form.

Table 8. Evaluation of a Binary Dependent Variable Model (Logistic Regression)

|**Binary Dependent Variable (Poverty**|**Status) Model**<br>**Logistic Mod**<br>**Poverty Line**<br>|**versus OLS Re**<br>**el**<br> **/ PMT Cutoff**<br>|**gression of Ho**<br>|**usehold Per C**<br>**OLS Model**<br>**Poverty Line**<br>|**apita Consum**<br>**/ PMT Cutoff**<br>|**ption**<br> <br>|
|---|---|---|---|---|---|---|
|**Evaluation Criteria**|**Bottom**<br>**20%**|**Bottom**<br>**40%**|**Bottom**<br>**60%**|**Bottom**<br>**20%**|**Bottom**<br>**40%**|**Bottom**<br>**60%**|
|Povertyheadcount(observed)|19.96%|39.95%|59.89%|19.96%|39.95%|59.89%|
|Household poverty rate (observed)|19.36%|37.59%|54.92%|19.36%|37.59%|54.92%|
|Household poverty rate (predicted)|12.88%|39.40%|59.40%|12.81%|38.76%|59.60%|
|Total accuracy|83.41%|78.39%|80.00%|83.01%|77.89%|79.26%|
|Povertyaccuracy|40.41%|73.67%|85.87%|39.21%|72.15%|85.38%|
|Undercoverage|59.59%|26.33%|14.13%|60.79%|27.85%|14.62%|
|Leakage|6.26%|18.76%|27.15%|6.47%|18.65%|28.19%|
|_Notes:_The logistic model is estimated<br>poverty line. For comparisons purpos<br>National Post-LASSO OLS model. Both<br>out-of-sample prediction on a random|using observe<br>es, the PMT cut<br>models includ<br>sample of 25|d poverty statu<br>off was selecte<br>e the same vari<br>% of household|s in the NHBPS<br>d to identify a<br>ables presente<br>s.|2014/2015 ba<br>similar share o<br>d in Table 3. T|sed on the na<br>f households<br>he results are|tional<br>as the<br>based on|

Table 9 presents the results of the Sudan national PMT with the inclusion of a complete set of state and rural and urban indicator variables. The inclusion of state and area indicator variables improves model accuracy by less than 2% with the largest gains observed when targeting the bottom of the consumption distribution. Similarly, poverty accuracy, undercoverage, and leakage all show modest improvements with

> 19 The regression results are available upon request.

19
the inclusion of location indicator variables. This improvement leads to one additional correctly identified poor household and the exclusion of one additional correctly identified non-poor households per 100 program applicants at the national level when those in the bottom 20% are targeted. However, in practice, the PMT will likely not be used to administer funds to states, rather the states will administer a program given a pre-determined program budget. Therefore, within a state, the ranking of households based on the PMT score will not change with or without state indicator variables and may not be necessary to include in the PMT.

Table 9. Evaluation of the National PMT with Additional State and Area Variables (OLS Regression)

**Evaluation of Including a Complete Set of Location Indicators in the PMT**

||**OLS Model w**<br>**Poverty Line**<br>|**ith Location In**<br> **/ PMT Cutoff**<br>|**dicators**<br>|**OLS Model w**<br>**Poverty Line**<br>|**ithout Locatio**<br>**/ PMT Cutoff**<br>|**n Indicators**<br> <br>|
|---|---|---|---|---|---|---|
|**Evaluation Criteria**|**Bottom**<br>**20%**|**Bottom**<br>**40%**|**Bottom**<br>**60%**|**Bottom**<br>**20%**|**Bottom**<br>**40%**|**Bottom**<br>**60%**|
|Poverty headcount (observed)|19.96%|39.95%|59.89%|19.96%|39.95%|59.89%|
|Householdpovertyrate(observed)|19.36%|37.59%|54.92%|19.36%|37.59%|54.92%|
|Household poverty rate (predicted)|12.71%|38.76%|58.73%|12.81%|38.76%|59.60%|
|Total accuracy|85.32%|79.63%|80.40%|83.01%|77.89%|79.26%|
|Poverty accuracy|44.91%|74.47%|85.63%|39.21%|72.15%|85.38%|
|Undercoverage|55.09%|25.53%|14.37%|60.79%|27.85%|14.62%|
|Leakage|4.98%|17.26%|25.96%|6.47%|18.65%|28.19%|

_Notes_ : This model includes all variables in Table 3, as well as location indicators for states and rural and urban areas where noted. The results are based on out-of-sample prediction on a random sample of 25% of households.

Table 10 presents the results of the Sudan national PMT with the inclusion of employment information in the form of the proportion of household members that have ever worked for income to total household members. Although this information is strongly and statistically significantly associated with per capita household consumption. Including this information does not impact the targeting performance of the PMT in a meaningful way. Employment information is generally difficult and time consuming to collect in household surveys, consisting of a series of questions to capture the employment status of each household member of working age and the type of employment, for example, salaried or non-paid, family workers.

20
Table 10. Evaluation of the National PMT with Employment Information (OLS Regression)

|**Evaluation of Including Employment Information in the PMT**|
|---|

||**OLS Model w**<br>**Information**<br>**Poverty Line**<br>|**ith Employme**<br> **/ PMT Cutoff**<br>|**nt**<br>|**OLS Model w**<br>**Information**<br>**Poverty Line**<br>|**ithout Emplo**<br>**/ PMT Cutoff**<br>|**yment**<br> <br>|
|---|---|---|---|---|---|---|
|**Evaluation Criteria**|**Bottom**<br>**20%**|**Bottom**<br>**40%**|**Bottom**<br>**60%**|**Bottom**<br>**20%**|**Bottom**<br>**40%**|**Bottom**<br>**60%**|
|Poverty headcount (observed)|19.96%|39.95%|59.89%|19.96%|39.95%|59.89%|
|Householdpovertyrate(observed)|19.36%|37.59%|54.92%|19.36%|37.59%|54.92%|
|Household poverty rate (predicted)|13.04%|39.16%|59.70%|12.81%|38.76%|59.60%|
|Total accuracy|83.18%|77.96%|79.43%|83.01%|77.89%|79.26%|
|Povertyaccuracy|40.24%|72.78%|85.63%|39.21%|72.15%|85.38%|
|Undercoverage|59.76%|27.22%|14.37%|60.79%|27.85%|14.62%|
|Leakage|6.51%|18.92%|28.12%|6.47%|18.65%|28.19%|

_Notes:_ This model includes all variables in Table 3, as well as the proportion of household members who have worked for income to the total household members where noted. The results are based on out-of-sample prediction on a random sample of 25% of households.

In this section, we explored the impact of different model specifications and the inclusion of location and employment variables on the targeting performance of the PMT. The results of a binary dependent logistic regression of the poverty status of the household were nearly identical to the results of the PMT based on OLS regression. Employment information did not impact the performance of the PMT in a meaningful way. And the inclusion of a complete set of location indicator variables marginally improved targeting performance of the PMT. However, in practice, social programs are likely to be administered at the state level with pre-determined budgets, rather than by using the PMT to determine the allocation of beneficiaries across states. Due to this, along with the minor improvements of the performance of the PMT with the inclusion of location variables, the Sudan national PMT excluding location indicators appears to perform well to meet program objectives.

# 6. Conclusion

In this paper we develop and propose a PMT to support the efforts of Sudan in the targeting of the Sudan Emergency Safety Nets Project (SESNP) to those most in need to improve the poverty impact and sustainability of the program.<sup>20</sup> Currently, there are efforts in Sudan to implement a new social safety net

> 20 Given that targeting can be politically sensitive (Del Ninno and Mills, 2015) and, particularly in FCV environments (Fragility, Conflict, and Violence) such as Sudan, political economy considerations and the costs of adopting a PMT should be examined. In this regard, when operationalizing the PMT, it will be important to coordinate with the Central Bureau of Statistics of Sudan and other government stakeholders on this work.

21
program, which seeks to assist the most food insecure households in the most affected localities and states, by providing unconditional cash transfers to beneficiary families in all 18 states of Sudan.

Sudan has been experiencing multiple macroeconomic shocks, such as low growth, a wide trade deficit, and the emergence of multiple exchange rates since its separation from South Sudan in 2011. The country has implemented an ambitious agenda of reforms, which included the removal of various subsidies, in an effort to reverse years of structural imbalances and high inflation. Parallel to such reforms was the introduction of a quasi-Universal Basic Income cash transfer program (SFSP) that sought to reach the bottom 80% of the population. The SFSP program was eventually discontinued, as, with the onset of COVID19, political discontent ultimately resulted in a military takeover and a renewed effort by the military-led government to achieve fiscal consolidation.

More recently, the Russia-Ukraine crisis that started in March 2022, has further exacerbated the country’s social and economic problems. In particular, Sudan is expected to face significant negative food security implications as food prices reach historic peaks due to the Russian invasion of Ukraine. All of the economic and social problems that have negatively impacted Sudan are expected to have increased the number of poor in the country.

The worsening situation in Sudan has led to renewed efforts to develop a new social safety net program. To support the efforts of Sudan in the targeting of the Sudan Emergency Safety Nets Project (SESNP) to those most in need, we develop and propose a PMT for Sudan based on the latest household survey, the National Baseline Household Budget and Poverty Survey (NHBPS) 2014/2015.

We find that a PMT should be considered for targeting social protection programs in Sudan. The PMT would assist social programs to reach a higher share of the poor and vulnerable, while reducing expenditure going towards households that are less in need, improving both the poverty impact, as well as the sustainability of the program. We explore various specifications of the PMT and find that our proposed Sudan national PMT performs as well as a rural and urban specific PMT, making it easier to implement and manage in practice. Furthermore, costly and time-consuming employment information does not improve the accuracy of the PMT in a meaningful way. Finally, our proposed Sudan PMT relies on 19 variables that could be collected by a short survey in the field on a large portion of the population.

22