metadata: authors: 0: author: Lucchetti,Leonardo Ramiro
1: author: Corral Rodas,Paul Andres
2: author: Ham,Andres
3: author: Garriga,Santiago
count: Peru
docna: 0: docna: Lassoing Welfare Dynamics with
            Cross-Sectional Data
docty: Policy Research Working Paper
subtopic: Inequality,Poverty Lines,Poverty Monitoring & Analysis,Small Area Estimation Poverty Mapping,Poverty Diagnostics,Poverty Assessment,Poverty Impact Evaluation,Labor & Employment Law,Educational Sciences,Gender and Development
teratopic: Law and Development,Education,Poverty Reduction,Gender
abstracts: cdata!: This paper introduces, validates, and
            applies a Least Absolute Shrinkage and Selection Operator
            with multiple imputation by Predictive Mean Matching
            (LASSO-PMM) method to estimate intra-generational welfare
            dynamics using cross-sectional data. Compared to previous
            welfare dynamic prediction methods, the LASSO-PMM makes
            fewer and less restrictive assumptions and allows estimating
            poverty transitions and income changes. We validate the
            method using 36 harmonized panel data sets in four Latin
            American countries and then apply it to cross-section data
            from 43 countries across the world. To the best of our
            knowledge, this is the first paper that uses these many
            datasets to validate and estimate welfare and mobility
            predictions. Validation results indicate that LASSO-PMM
            predictions are in general statistically indistinguishable
            from actual household poverty rates, mobility indicators,
            and income or consumption changes; results which are further
            supported by a series of sensitivity tests and robustness
            checks. These findings are sufficiently encouraging to
            suggest that estimating economic mobility using a LASSO-PMM
            approach may accurately approximate actual welfare dynamics
            in settings where panel data are unavailable. This
            application provides useful policy information on the
            dynamics of individual welfare in countries where two or
            more rounds of cross-section data are available.
geo_reg_key: 119228,517191
repnme: repnme: Lassoing Welfare Dynamics with
            Cross-Sectional Data
display_title: Lassoing Welfare Dynamics with
            Cross-Sectional Data
listing_relative_url: /research/2018/08/30335457/lassoing-welfare-dynamics-cross-sectional-data
topicv3: Economic and Social Mobility m1326259 1152,Poverty m1326687 695,Education m1327048 470,Gender m1327621 615,Household Surveys m1504071 2761
publishtoextweb_dt: 2018-08-08T00:00:00Z
historic_topic: Law and Development,Education,Poverty Reduction,Gender
no_of_pages: 63
author_keys: 0: author_key: 763344
1: author_key: 1204461
2: author_key: 1241317
model_extractions: Policy Research Working Paper 8545 Lassoing Welfare Dynamics with Cross-Sectional Data

# _Leonardo Lucchetti_

_Paul Corral Andrés Ham Santiago Garriga_ Poverty and Equity Global Practice August 2018
Policy Research Working Paper 8545

# **Abstract**

This paper introduces, validates, and applies a Least Absolute Shrinkage and Selection Operator with multiple imputation by Predictive Mean Matching (LASSO-PMM) method to estimate intra-generational welfare dynamics using cross-sectional data. Compared to previous welfare dynamic prediction methods, the LASSO-PMM makes fewer and less restrictive assumptions and allows estimating poverty transitions and income changes. We validate the method using 36 harmonized panel data sets in four Latin American countries and then apply it to cross-section data from 43 countries across the world. To the best of our knowledge, this is the first paper that uses these many datasets to validate and estimate welfare and mobility predictions. Validation results indicate that LASSO-PMM predictions are in general statistically indistinguishable from actual household poverty rates, mobility indicators, and income or consumption changes; results which are further supported by a series of sensitivity tests and robustness checks. These findings are sufficiently encouraging to suggest that estimating economic mobility using a LASSO-PMM approach may accurately approximate actual welfare dynamics in settings where panel data are unavailable. This application provides useful policy information on the dynamics of individual welfare in countries where two or more rounds of cross-section data are available.

This paper is a product of the Poverty and Equity Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/research. The authors may be contacted at llucchetti@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Lassoing welfare dynamics with cross-sectional data***

Leonardo Lucchetti Paul Corral The World Bank The World Bank Andrés Ham Santiago Garriga Universidad de los Andes Paris School of Economics _Updated December 16, 2020_ _Keywords_ : Poverty; Poverty transitions; LASSO; Machine learning; Multiple Imputation; Synthetic panels; Welfare dynamics.

_JEL classification_ : C53, D31, O15, I32. _Sector Board_ : POV Leonardo Lucchetti (llucchetti@worldbank.org) is a Senior Economist with the Poverty and Equity Global Practice, World Bank. Paul Corral (pcorralrodas@worldbank.org) is a Senior Economist with the HD Chief Economist Office, World Bank. Andres Ham (a.ham@uniandes.edu.co) is an Assistant Professor in the School of Government at Universidad de los Andes. Santiago Garriga (garrigasantiago@gmail.com) is a Ph.D candidate at the Paris School of Economics. We thank Aziz Atamanov, Eduard Bukin, Oscar Calvo, Andres Castañeda, Francis Addeah Darko, Carolina Diaz-Bonilla, Reno Dewina, Leonardo Gasparini, Gabriela Inchauste, Jose Montes, Laura Liliana Moreno, David Newhouse, Minh Cong Nguyen, Obert Pimhidzai, Carolina Sanchez-Paramo, Carlos Silva-Jauregui, Liliana D. Sousa, Shinya Takamatsu, Mariana Viollaz, Judy Yang, Nobuo Yoshida, Matthew Wai-Poi, Bagus Arya Wirapati, and Salman Zaidi. We also thank participants of the LACEA-LAMES 2018 meeting. Carlos Santiago GuzmánGutiérrez provided outstanding research assistance. Pablo Gluzmann provided outstanding support in the construction of all panel datasets. All remaining errors are ours.
# **1. Introduction**

Many countries now gather survey microdata at the household and individual level. The rapid expansion of household surveys at frequent intervals that are comparable over time and across countries has facilitated poverty monitoring in the developing world: coverage increased from 13 countries in the 1990s to over 60 countries in 2011 (Serajuddin _et al_ . 2015). One of the main uses for these surveys is to study poverty and welfare within countries and over time. However, most of the available micro data are cross-sectional (i.e., they do not track individuals and households over time and therefore only provide aggregate welfare trends). Panel datasets that follow individuals over several periods of time are less readily available, which limits the understanding of the factors that explain movements out of poverty, into poverty, and the duration of poverty (Dang et al. 2019).

This paper proposes a machine learning approach with a multiple imputation technique to overcome this empirical challenge. We propose using a Least Absolute Shrinkage and Selection Operator method with multiple imputation by Predictive Mean Matching (LASSO-PMM) to estimate intra-generational economic mobility using cross-sectional data.<sup>1</sup> The LASSO-PMM approach we propose estimates a model of household-level income<sup>2</sup> using a set of time-invariant and deterministic harmonized variables as controls to provide estimates of welfare dynamics. Compared to previous methods in the literature, LASSO-PMM makes fewer assumptions and estimates not only poverty transitions, but also income changes. We present, validate, and apply the LASSO-PMM method on harmonized microdata. Specifically, we validate the method using 36 panels in four Latin American countries (Argentina, Chile, Nicaragua, and Peru) and apply the

> 1 We developed a Stata command to implement the LASSO-PMM, which can be accessed here. The command makes use of Wilbur Townsend’s lassoregress Stata command.

> 2 For simplicity we refer to income as the welfare measure throughout this paper.

2
procedure to cross-section data from 43 countries across the world. To the best of our knowledge, this is the first paper that uses this many datasets to validate and estimate welfare and mobility predictions.

Some previous studies have proposed methods that harness cross-section data to infer about dynamic welfare changes. “Synthetic Panels”, developed by Dang, Lanjouw, Luoto, and McKenzie (2014), is the most recent.<sup>3</sup> The authors estimate a (log) income model in both the first and second rounds of cross-sectional data, including time-invariant, deterministic, and retrospective regressors. Parameters estimated in the first round are then used to predict unobserved first round income for all households interviewed in the second round. Depending on the assumptions on the correlation between error terms in the underlying regressions in both rounds, this “non-parametric” approach provides upper and lower bound estimates of poverty and mobility. The methodology has been validated using seven panels in Chile, Nicaragua, and Peru by Cruces et al. (2015), while Ferreira et al. (2013) applied it to predict intra-generational poverty mobility in 18 countries in Latin America and the Caribbean (LAC) using harmonized crosssectional micro data.

Because this synthetic panel approach only provides bounds, Dang and Lanjouw (2013) proposed a method that obtains a point estimate of intra-generational poverty mobility by assuming that the error terms in the underlying regressions follow a binormal distribution and using the agecohort correlation of residuals from cross-sections. This “parametric” method was validated by the authors on panel data from five countries and has been applied by Dang and Ianchovichina (2018) to study welfare dynamics in six Arab countries; by Dang and Dabalen (2018) to analyze whether growth has been pro-poor in 21 countries in Africa; and by Vakis, Rigolini, and Lucchetti (2016)

> 3 The synthetic panel method builds on the poverty mapping technique developed by Elbers, Lanjouw, and Lanjouw (2003).

3
to analyze chronic poverty in 17 LAC countries for which harmonized cross-sectional micro data are available.

The LASSO-PMM approach introduced in this paper is more flexible and entails fewer assumptions compared to these two methods. As such, it does not require estimating any error correlation terms or assuming a binormal distribution of residuals in the underlying regressions as in Dang and Lanjouw (2013). In addition, the LASSO-PMM method not only provides transition probabilities, but also predicts income growth for every observation between two rounds of crosssectional data as if we had actual panels. Finally, unlike Dang et al. (2014), the LASSO-PMM method obtains point estimates instead of bounds of welfare dynamics.

We use 36 panels from four countries to validate our estimates of mobility using harmonized variables frequently used in many regional and global studies, which facilitates implementation by researchers and policymakers in developing countries. The LASSO-PMM approach for predicting poverty and mobility approximates observed welfare changes. In general, validation results show that LASSO-PMM predictions are statistically indistinguishable from actual poverty rates, transitions, and income changes calculated using panel data. A series of validation exercises support the method’s predictive performance. We divide these exercises into: i) _sensitivity tests_ , in which the actual poverty rates and transitions do not change; and ii) _robustness exercises_ , when we change the estimation procedure, sample, or other attributes that change the actual welfare estimates. Both of these validation exercises show that the LASSOPMM approach for predicting poverty and mobility accurately approximates actual welfare changes. The two main conclusions from this validation are: i) changing parameters to make the method more computationally intensive provides only marginal changes in estimates; and ii) model specification matters. All these findings are sufficiently encouraging in general to suggest 4
that estimating economic mobility using LASSO-PMM may approximate actual welfare indicators in settings where cross-sections are routinely collected, but where panel data are unavailable.

Given these validation results, we apply the LASSO-PMM method to harmonized crosssection data from 43 countries across the world in circa 2010 and 2015. Compared to previous methods, the models we estimate have greater explanatory power, which increases our predictive capabilities. In general, aggregate poverty rates are accurately approximated when using crosssections across a heterogeneous group of countries. Several additional messages emerge from the dynamic analysis. First, there has been considerable mobility in recent years; approximately 22% of individuals changed their economic status in all countries under analysis. Second, even in a period of generalized poverty reduction, about 9% of the population fell into poverty. Third, a large share of the population is immobile; about 24% of all individuals remained in chronic poverty. Fourth, initial average income for the chronic poor was slightly lower but grew significantly less than for those who moved out of poverty. Lastly, those who were not poor but fell into poverty had an income substantially lower and decreased more than those who were not poor and remained out of poverty.

The remainder of this paper is organized as follows. Section 2 summarizes existing crosssection welfare dynamic predicting methods and presents the LASSO-PMM approach. Section 3 describes the harmonized data sources used to validate and apply the method. Section 4 presents the results of estimating the LASSO-PMM approach and performs a wide array of validation tests to determine its sensitivity and robustness with 36 panel data sets. Section 5 applies the LASSOPMM approach to cross-section data from 43 countries and discusses the findings. Section 6 concludes.

5
# **2. Welfare dynamic prediction methods**<sup>**4**</sup>

# _2.1. Non-parametric synthetic panels_

Assume two rounds of cross-sectional micro data are available: round 1 and round 2. These two rounds of survey are random samples of an underlying population and consist of a sample of _N1_ and _N2_ households, respectively. Let us define i as household’s _i_ log per capita income in moment _t_ , _xit_ as a vector of household characteristics for household yy _i_ in round _t_ , and _z_ as the poverty line. Variables included in _xi2_ refer to characteristics of household _i_ in round 2 that are also observed (for different households) in round 1. These characteristics often include: i) timeinvariant variables such as gender of the head of the household if his/her identity remains constant between the rounds of data; ii) deterministic variables such as age; and iii) retrospective variables such as whether a household surveyed in the second round had an asset in the first round (Cruces et al. 2015; Dang and Lanjouw 2018). The relationship between income and these characteristics can be expressed in regression form as: where ε _it_ is an error term and _xit_ is a vector of _K_ regressors whose first element is equal to one so that the first element of is the intercept of the model. ii We introduce superscripts to refer to observations surveyed at different times. Our ββ objective is to calculate, for a household _i_ interviewed in round 2, the income change across the two rounds of data: 2 2 2 , where 2 and 2 are the first and second round incomes of ii1 ii2 ii1 ii1 ii2 household _i_ surveyed in round 2, respectively. Therefore, because we observe the household in ∆yy = yy −yy yy yy round 2, we need to estimate its income in round 1. Consequently, we can also estimate the

> 4 This section draws from Dang and Lanjouw (2013), Dang et al. (2014), Cruces et al. (2015), Vakis et al. (2016), and Lucchetti (2017).

6
household’s poverty dynamics: the joint probability that household _i_ surveyed in round 2 escapes 2 2 2 poverty between round 1 and round 2 ( ), remains poor ( < 2 2 ii1 2 ii2 ii12 ), becomes poor ( Pr(yy < zz a aa yy), and remains non-poor (> zz) Pr(yy > ii22 5,6 ii1 ii2 ii1 zz a aa yy < zz)). Pr(yy > zz a aa yy < zz) Pr(yy ii2 zz a aa yyThese estimates can be easily calculated with panel data, since all households are > zz) interviewed in both rounds (i.e., 2 is known for every household _i_ interviewed in round 2). ii1 However, these datasets are rarely available in developing countries and costly to collect. yy Alternatively, synthetic panels allow predicting the first round “unobserved” incomes of households surveyed in the second round by multiplying their time-invariant characteristics and the first-round Ordinary Least Squares (OLS) estimates of parameters 1O that solve the OO following optimization problem: ββ̂ where 1 is the first-round log income of household _i_ surveyed in round 1, _N1_ indexes the number of observations in round 1, and yyii1 R 11 refers to the residual sum of squares. The two non-parametric synthetic panel approaches differ in the treatment given to the RR correlation between the error terms in the first and second round of cross-sectional data, which is likely to be non-negative according to Dang et al. (2014). Upper bound estimates assume no correlation between the first and second round error terms. The authors propose estimating the first-round incomes of households interviewed in the second round of data by drawing randomly

> 5 For ease of exposition, we will only focus on the probability of escaping poverty in this section. However, the results generalize to the other three mobility situations (remaining poor, becoming poor, and remaining non-poor).

> 6 Note that it is possible to carry out the exact inverse prediction (i.e., predict income of round 2 for those households interviewed in round 1). We will return to this point in the results section.

7
y =y y Pr(9 >z) é 6 ¢6 a. . Pr(v >z) —y —§
9 =9 +|a-ne ~~a~~

Pr(p
literature, the quality of the predictions might not be linked to the quality of the underlying welfare model used. For instance, Dang and Lanjouw (2013) obtain predictions that are statistically indistinguishable from actual household poverty transitions with a relatively low adjusted _R_<sup>_2_</sup> =0.08 in the underlying welfare model.<sup>8</sup> The intuition is that a worse fit of the model makes the upper and lower bound estimates get far away from each other. However, the point estimate ultimately depends on the value of _ρ_ selected and is unaffected by the value of the upper and lower bound predictions.<sup>9</sup> Second, Herault and Jenkins (2019) show that the estimates of _ρ_ are quite sensitive to the choice of the cohort, which (connected to the previous point) could generate inadequate estimates of _ρ_ and subsequently the dynamics of poverty. Third, the method does not allow predicting income changes (i.e., 2 2 2 ); it only obtains poverty transitions like the one shown in ii1 ii2 ii1 equation (6). Finally, the assumption of normality of the error terms is rejected in Vietnam and ∆yy = yy −yy Indonesia by Dang et al. (2014). In summary, obtaining a point estimate for welfare dynamics is feasible, but requires several restrictive assumptions and additional estimations.

_2.3. A machine learning and multiple imputation approach (LASSO-PMM)_ This paper proposes using a Least Absolute Shrinkage and Selection Operator complemented with multiple imputation by Predictive Mean Matching (LASSO-PMM) to estimate intra-generational poverty mobility and household-level income growth using cross-sectional data. The Least Absolute Shrinkage and Selection Operator (LASSO) procedure is one of the most popular

> 8 Herault and Jenkins (2019) also obtain good results with a relatively low adjusted R2 in the range of 0.098 to 0.226, while Dang and Ianchovichina (2018) obtain good predictions with the following adjusted R<sup>2</sup> coefficients: 0.16 for Jordan, 0.09 for Palestine, 0.05 for Syria, and 0.13 for Yemen.

> 9 Lucchetti (2017) found a similar result in non-parametric estimates. Lower values of the adjusted R2 make the lower and upper bounds get far away from each other, despite the fact that the non-parametric approach that uses a weighted average of the residuals to get a point estimate of mobility remains largely unaffected by the quality of the underlying welfare model.

10
machine learning methods among economists and consists of minimizing a quadratic loss function plus the sum of the absolute value of the coefficients (Mullainathan and Jann Spiess 2017). We propose estimating parameters in the first round of cross-sectional data by solving the following optimization problem: (7) = argmin 11 KK | OOLLO OO 1 �R � 1λλ ss1 ββ̂ ββ RR + λλ�|ββ ss=1 The estimates depend on the value of the “shrinkage” factor . Whenever , the objective function becomes the OLS objective function in (2) and λλ 1O . The LASSO λλ→0 OOLLO OO OO 1λλ estimate will deviate from the OLS estimate for positive values of . Finally, ββ̂ →ββ̂ will be shrunk OOLLO OO 1λλ to zero as . Therefore, for values , the LASSO attenuates towards zero if compared λλ ββ̂ with OLS. λλ→∞ λλ> 0 The factor is introduced for two reasons. First, the shrinkage penalty ∑ | in LASSO KK ss=1 ss1 provides corner solutions, which implies that some coefficients are forced to be zero. Therefore, λλ |ββ the LASSO works well for model selection when the number of candidate variables _K_ is large. Second, for appropriate values of , the bias introduced is compensated by a reduction of variance. Given that the main objective is to predict welfare, and not estimate causal effects of the λλ explanatory variables, minimizing the bias-variance trade-off is essential for accurate predictions. To avoid arbitrariness, the shrinkage factor is selected by means of a 10-fold crossvalidation algorithm,<sup>10</sup> which is a method to test the out of sample fit of the income model.λλ<sup>11</sup> The algorithm randomly divides the first-round of data into 10 equal sized folds. By leaving one fold out (the test fold), the model is fit in the other 9 folds (the training folds). Once the income model

> 10 This is the first stage of the cross-validation process we employ in our validation exercise. A second stage is explained in section 3.

> 11 Variables are standarized to have a mean of zero and standard deviation of one.

11
is estimated, the withheld fold is used to predict the model. This is repeated 10 times until all folds have been left out and all observations have a predicted value 1 . The value of is selected so 1 ii11 1 2 that it minimizes the mean squared error (MSE) defined as ∑ yy� ) _/N1_ . λλ NN ii=1 ii1 ii1 Like in the case of the other synthetic panels presented in previous sections, we can proceed (yy −yy� to compute two linear fits. The LASSO linear fit of the first-round incomes for all households surveyed in the first round is On the other hand, the LASSO first-round linear fit for all households surveyed in the second round is Like in the upper and lower bound non-parametric synthetic panels, the full welfare measure is completed by the addition of the residuals. The data generating process for the residuals is unknown under LASSO. This is one of the reasons why we simulated these unobservable using a predictive mean matching approach (PMM). This method is common in the multiple imputation literature (see Van Buuren, 2018), and always finds a value which has been observed in the data to produce a simulated value.

Once the linear fits in the first and second rounds are obtained using LASSO (equations (8) and (9)), they are compared with the aim of finding neighbors. For every observation in the second round of data, a set of neighbors (in terms of linear fit) are found in the first-round data by looking for observations with the smallest absolute difference between the two linear fits. Among the set of neighbors (the number of neighbors is determined by the researcher), one neighbor is selected at random. The observed welfare measure from that neighbor is then imputed to the corresponding observation in the second round, which represents the first-round income for that particular household surveyed in the second round. This imputation needs to be done for every 12
|y<br>y<br>y<br>i=1,2|
|---|

<!-- Start of picture text --> II<br><!-- End of picture text -->
- Find nearest neighbors and compute income dynamics for the rest of the observations in the second round.

- Repeat all steps _R_ times.

It is important to note that estimating LASSO with a multiple imputation approach has at least three advantages with respect to previous methods in the dynamic welfare prediction literature. First, the approach described in this paper does not require additional estimation of unknown parameters, such as the age-cohort correlation of residuals from cross-sections as in the parametric approach. Second, unlike the parametric approach, the method allows researchers to calculate household-level income changes and not just probabilities of poverty mobility.<sup>13</sup> Finally, given that the objective is prediction and not causal inference, the method takes advantage of machine learning techniques that minimize MSE (both bias and variance), to produce accurate welfare predictions outside the estimation sample. While previous studies minimize their discussion of model specification and explanatory power, the LASSO-PMM procedure we apply here selects the optimal set of variables in the underlying method by minimizing the MSE of the dependent variable (log of per capita household income) out of sample and with minimal

> 12 As such, instead of randomly allocating first-round residuals as in the upper bound non-parametric synthetic panel, the LASSO-PMM allocates actual incomes from the first-round nearest neighbords, which is similar to allocating residuals based on the minimization of the distance of linear fits.

> 13 Bourguignon and Moreno (2018) introduce in a working paper a modification of the parametric Synthetic panel approach that avoids its most arbitrary assumptions. Their method also allows to estimate income changes. However, their’s method is more difficult to implement in our context; their work involves the calibration (instead of an estimation) of Synthetic panels within the scope of AR(1) processes, which makes its implementation in several countries at once a difficult task because the error formation process needs to be modeled.

14
assumptions. This is important because the quality of the predictions is directly linked to the quality of the underlying welfare model used under this approach.

# **3. Data and empirical approach**

This paper uses harmonized microdata for several countries to validate and estimate economic mobility using LASSO-PMM. We first employ 36 panels from four selected countries to validate the method, performing a battery of sensitivity and robustness exercises in a controlled environment where we can observe household income in both periods. Then, we proceed to apply the method to harmonized cross-section data from 43 countries around the world, located in six broad regions.

# _3.1 Validation with harmonized panel data_

For the validation of the LASSO-PMM method, we employ panel data from four Latin American countries: Argentina, Chile, Peru, and Nicaragua. These data are part of the SEDLAC and LABLAC projects.<sup>14</sup> The selection of these countries depends mainly on the availability of comparable panel data over time, but we highlight that these sources contain information for small and large economies; lower-middle, upper-middle, and high income Latin-American countries, and encompass short and long panels.

We construct 36 different panels for the four selected countries, which vary in the length of time in which households are followed. There are 23 one-year panels, 7 two-year panels, 5 three-year panels, and one four-year panel. To the best of our knowledge, this is the first paper that

> 14 The SEDLAC project consists of more than 400 household surveys in more than 25 LAC countries to provide statistics on poverty and other distributional and social variables. The LABLAC project is a complementary source to SEDLAC, which monitors trends in labor market variables with the highest frequency information available in 13 LAC countries. Both projects are a joint venture between the World Bank and and the Center for Distributive, Labor, and Social Studies (CEDLAS, for its acronym in Spanish) at the _Universidad Nacional de La Plata_ in Argentina. See Bourguignon (2015) for a detailed description of the SEDLAC project. Data for Argentina, Peru, and Nicaragua are part of the SEDLAC project, while data for Chile are obtained from LABLAC project.

15
uses this many panel data sets from multiple countries to validate cross-section poverty dynamics prediction methods.<sup>15</sup> Variation in poverty rates, aggregate conditions, and panel length provide an added value that allows investigating the performance, sensitivity, and robustness of the LASSO-PMM method. This knowledge is pivotal to understand whether this method can be applied more generally to monitor and predict household welfare dynamics using data sources from other regions and countries.

All 36 panels are harmonized to maximize comparability across countries and over time. The harmonization procedure follows the criteria established by SEDLAC (2018), which consists in using identical definitions in each data set. The advantage of this procedure lies in ensuring comparability across relevant dimensions, so that any observed differences across countries are due to specific changes in poverty dynamics, aggregate conditions, and panel length. The harmonization procedure requires cleaning and processing each panel to ensure that all samples, definitions, and variables are identical in each country and year. The full list of available covariates for each country and panel in the validation data are shown in Appendix Table A.1.

The decision to use harmonized panels is taken in order to test how the method would perform on cross-sectional data that applies some form of harmonization, such as SEDLAC, LABLAC, and other worldwide repositories commonly used to measure and monitor poverty, much like we use in our application in section 4. Satisfactory validation results would provide confidence to implement LASSO-PMM in environments where cross-sectional harmonized microdata exist to predict welfare dynamics. Moreover, the need to use harmonized data incorporates an additional restriction to the validation exercise, apart from the existing constraint

> 15 For instance, Herault and Jenkins (2019) use 31 panels from two countries (Australia and the UK) in their validations; Dang and Lanjouw (2013) use 11 panels from 5 countries; Dang et al. (2014) use 8 panels from 6 countries; and Bourguignon and Moreno (2018) use one panel from Mexico.

16
of time invariance of the predictors used in the models. Therefore, the objective of the first part of this paper is to validate the LASSO-PMM method when we have comparable data across countries, weighting generalizability over specificity.

# _3.2 Application to harmonized cross-section data_

After performing the validation on panel data, we apply the LASSO-PMM method to harmonized cross-section data. We employ the Global Monitoring Database (GMD) which gathers income and/or expenditure household surveys across the world. Raw surveys routinely conducted by national statistical offices in each country are compiled by the GMD project that is hosted and maintained by the World Bank. Most of the value added of this exercise, besides the compilation of a large number of household surveys, is the additional data processing and cleaning within the GMD initiative that harmonizes data sources to be comparable not only across countries but also within countries over time. The GMD database is the main input for the well-known _PovcalNet_ online tool<sup>16</sup> to monitor global poverty and inequality.<sup>17</sup> The benefits of using the GMD data in terms of population coverage are undeniable, since they contain more than 1,000 surveys in 156 countries.<sup>18</sup> However, with greater coverage, some specificity is lost. The number of surveys and variables that can be harmonized in order to be plausibly compared across and within countries is limited. Therefore, a set of dimensions have been selected and harmonized by the GMD project. These dimensions are: welfare (nominal, real, and in PPP terms); demographics (household size, gender, relation to household head, and marital status); education (school attendance, education level, and literacy); location (either subnational regions or rural-urban areas); some basic assets (landline phone, cellular phone, and computer);

> 16 <u>http://iresearch.worldbank.org/PovcalNet/povOnDemand.aspx</u> 17 At the time of writing, our results are drawn from the July 2020 version of the GMD database. 18 In this paper we use surveys in 43 countries for which two round of cross-sectional data exist.

17
and access to water, sanitation, and electricity. The data also contain household and individual identifiers, as well as sampling weights. These dimensions contain most of the required timeinvariant and deterministic variables necessary to apply the LASSO-PMM method for predicting welfare dynamics and poverty mobility.

This paper uses household surveys for 43 countries around the world. The selection of countries is based on the availability of at least two comparable data points and on the availability of the necessary variables to conduct the estimation. We use the most recent surveys for each country at the time or writing, trying to cover a period of five years between both rounds of crosssectional data (i.e., circa 2010-2015). Our sample includes countries in the six geographical regions defined by the World Bank: East Asia and the Pacific (EAP), Europe and Central Asia (ECA), Latin America and the Caribbean (LAC), Middle East and North Africa (MNA), South Asia (SAR) and Sub-Saharan Africa (SSA). While the selected surveys do not cover all countries in the world, they do encompass a large set of nations with different levels of development and welfare. Appendix Table A.2 summarizes the countries and years, as well as the specific household surveys in the GMD that we use in the analysis.

# _3.3 Empirical strategy_

Estimations are carried out for household-level income changes as well as for poverty dynamics defined as the proportion of individuals with a harmonized per capita income lower than a US$5.5 per person per day poverty line, both in 2011 purchasing power parity (PPP) per day.<sup>19</sup> The main difference between the validation and application of the LASSO-PMM method is that we can

> 19 The US$5.5 per person per day in 2011 PPP is the World Bank poverty line used for upper-middle-income countries (UMICs). This line is the median value of UMIC national poverty lines (Jolliffe and Prydz 2016, Ferrerira et al 2016, Castañeda et al. 2018). The only exception is Chile in the validation analysis where we define a relative poverty line equal to 0.5 of median per capita income, because the poverty rate using the US$5.5 threshold is too low in this particular case.

18
observe actual welfare and mobility for households in the former case, but not the latter. Following previous studies (Cruces et al. 2005; Dang et al. 2014; Bourguignon and Moreno 2018), the key time invariance assumption is maintained by considering only those households whose heads are between 25 and 65 years of age in all estimates so that life cycle events are avoided in general.

We follow previous literature that estimates poverty and welfare by including time invariant and deterministic regressors in the models. The harmonized data used in this paper allow estimating poverty transitions and welfare changes with variables frequently associated with wellbeing (e.g., Ferreira et al. 2012; Vakis et al. 2016), specifically harmonized variables commonly found in household surveys that are comparable within and between countries and over time. The models consider the log of per capita household income in 2011 PPP/day as the left-hand side variable in the underlying regressions and the following regressors: [1] the household head’s characteristics (age, age squared,<sup>20</sup> gender, and years of education), household size, and household size squared;<sup>21</sup> [2] regional fixed effects;<sup>22</sup> and [3] the interaction between the first and the second set of covariates. The selected specification follows previous studies that define a core set of timeinvariant variables and correlates of welfare (Cruces et al. 2005; Dang et al. 2014; Bourguignon and Moreno 2018).

All these covariates are natural candidates as time invariant and deterministic regressors. However, it is important to bear in mind that all variables considered vary to some extent over time. The age, gender, and education of the head of the household can change each time the head

> 20 Age is a deterministic function of time. Therefore, we subtract the number of years betwen rounds of data from the household head’s age when predicting first round incomes for those surveyed in the second round. On the other hand, we add the number of years betwen rounds to the household head’s age when changing the forecasting direction. This procedure is commonly applied in the Synthetic Panel’s literature. See Dang and Lanjouw (2013) and Dang et al. (2014) for specific details and a more comprehensive explanation.

> 21 In the validation, we also include area of residence (i.e., urban vs. rural) in Chile, since it is the only country for which this variable is available.

> 22 We consider all regions for which surveys are representative in each country. In the application, we use area of residence (i.e., urban vs. rural) when regional fixed effects are not available.

19
of the household changes between survey rounds (e.g., death, migration, and divorce, among others, can force changes in the leadership of the household). Household size can also change due to births, deaths, divorces, among others.<sup>23</sup> The area of residence and regional fixed effects can change between rounds due to migration. On the other hand, other variables are easily discarded from the specifications, since they are clearly time-variant (e.g., employment status, sector of employment, etc.).<sup>24</sup> The LASSO-PMM approach takes the included variables and selects the relevant covariates in each of the countries and years by minimizing the MSE. The penalty parameter in the LASSO, λ , is chosen using a 10-fold cross validation exercise. The PMM method selects the household’s nearest neighbor. This procedure is repeated 20 times. The results provide first round predicted welfare for each household surveyed in the second round. These predictions are then compared to observed household’s welfare in the second round.

In the particular case of the validation, we carry out several robustness and sensitivity analyses to determine the stability of the predictions from the LASSO-PMM method. For instance, we change model parameters, estimation procedure, the direction of the prediction, sample, included variables, weighting structures, consider different ages, and use different poverty lines in our validation exercises in the next section. Given that we have panel data in this exercise, we follow Cruces et al. (2015) and perform a second stage cross-validation that randomly splits the

> 23 Household size was not considered in any of the studies with the exception of Bourguignon and Moreno (2018). However, household size is an important variable to be included for at least two reasons. First, the dependent variable (total household income) of the underlying regressions is in per capita terms. If we take two households that are identical in terms of head of household’s characteristics and have the same total income, but the first one is only one individual while the second one is a married couple, the second one would be twice as poor as the first one. Household size will be capturing this difference. Second, we are using household level data but individual weights (household weights times household size), and therefore household size also affects results through weights.

> 24 Bourguignon and Moreno (2018) present a thorough discussion of what variables may be considered as timeinvariant in poverty prediction approaches.

20
panels into two subsamples and treats each of these subsamples as a cross-section. Therefore, the coefficients are estimated in one of these subsamples in the first round of data and applied to the second subsample in the second round. By treating each subsample of the panel as a cross-section, this second stage cross-validation avoids any bias that might arise from using the panel dataset to validate the method.

# **4. Validation results**

# _4.1 Predicted poverty rates_

We apply the LASSO-PMM approach to predict the log of per capita household income in 2011 PPP/day in each country at the household-level. Covariates are harmonized across countries and years, and include the demographic, educational, and regional variables described in the previous section. The number of regressors varies from 27 in Nicaragua to 103 in Chile (see Appendix Table A.1).<sup>25</sup> The objective is to predict first round per capita household incomes for every household we observe in the second round.<sup>26</sup> Based on the estimated LASSO-PMM predictions, a first step of the intra-generational mobility analysis is to compare actual poverty rates in round 1 with the predicted poverty headcounts that emerge when applying the LASSO-PMM approach to those households surveyed in the second round. Estimates for all 36 panels are shown in Figure 1 and for the last available panel in each country in Table 1. The table presents both point estimates and 95 percent confidence intervals for actual and predicted poverty headcount rates.

> 25 The panel for Peru covering 2009-2012 was excluded due to low number of observations.

> 26 We also test the robustness of predictions to changes in the forecasting direction. That is, we predict second round incomes for every household observed in round one. We find qualitatively similar results when changing this direction. The full set of results are omitted due to space, but are available upon request.

21
Overall, the LASSO-PMM method performs well, since the predicted poverty rate estimates are close to the actual poverty headcounts calculated from the panel (or in general lie within their 95% confidence intervals). The figure shows predictions on the y-axis and actual poverty headcounts on the x-axis. Perfect correspondence between both is represented by the 45degree line. Many of the predictions are either on the line or in close proximity. The table shows this overlap more clearly. For instance, in Argentina and Nicaragua, predicted and actual poverty rate estimates are identical. In Chile and Peru, while the point estimates are different, the 95% confidence intervals overlap. Therefore, predicted poverty rates from the LASSO-PMM are statistically indistinguishable from actual poverty rates calculated from panel data.

# _4.2 Predicted poverty transitions_

The LASSO-PMM method also allows estimating poverty mobility patterns (i.e., transitions into and out of poverty). Columns 2 to 5 in Table 1 show the point estimates and 95% confidence intervals for actual mobility from panel data and predictions obtained by means of the LASSOPMM approach using the latest panel available in each country. We report four joint probabilities: the probability that a household is poor in both rounds of data ( _Poor, poor_ ), escapes poverty ( _Poor, non-poor_ ), becomes poor ( _Non-poor, poor_ ), and the likelihood that a household remains non-poor in both periods ( _Non-poor, non-poor_ ).

Figure 2 summarizes the results for poverty transitions for all available years.<sup>27</sup> With few exceptions, many of the point estimates arising from the LASSO-PMM approach fall within the 95 percent confidence interval of actual poverty transitions. The confidence intervals in Table 1 show that in many countries and years, the LASSO-PMM approach accurately predicts mobility patterns. There is variation across panels, with some predictions being more optimistic, while

> 27 The results for Peru using 2- and 3-year panels are shown in Appendix Figure A.1.

22
others more pessimistic. For instance, the method underestimates persistent poverty and overestimates upward mobility in Argentina. Conversely, the method is pessimistic in other cases, since downward mobility is overestimated in Argentina and Peru. However, the results indicate that the LASSO-PMM predictions approximate actual poverty transitions with a reasonable degree of precision.

The results in Table 1 and Appendix Figure A.1 also suggest that the method performs well irrespective of the length of the prediction window, given that we have one-year, two-year, threeyear, and four-year panels. We find suggestive evidence that the method performs slightly better when the length of the panel is longer, perhaps due to less risk of idiosyncratic and unexpected aggregate shocks. This can be seen by comparing the results for Peru, where we can construct 1- year, 2-year, and 3-year panels. As the length of time between panels increases, mobility is better approximated by LASSO-PMM both in terms of the point estimates and their respective confidence intervals.

# _4.3 Changes in income and non-anonymous growth incidence curves_

One additional contribution of the LASSO-PMM procedure compared to the parametric synthetic panel is that it also predicts income changes apart from just poverty transitions. Figure 3 graphs estimated household per capita income growth for two population groups defined by: i) the dynamic poverty transitions (poor, poor; poor, non-poor; non-poor, poor; and non-poor, non-poor) and ii) the quintiles of the income distribution in the first round—i.e., the non-anonymous growth incidence curves (GIC).<sup>28</sup> The figure presents the point estimate and the 95 percent confidence interval for the last available panel in each country. All estimates from the LASSO-PMM approach

> 28 The anonymous GIC refers to quantile-level (or any othe percentile) growth of mean incomes by quantile (or any other percentile) of the income distribution (Ravallion and Chen 2003). On the other hand, the non-anonymous GIC refers to mean individual-level growth of incomes by quantile (or any other percentile) of the income distribution.

23
are compared with the actual income growth from panel data. Following the literature using GIC, we change the forecasting direction. That is, we predict second round household per capita incomes for every household interviewed in the first round of data.<sup>29</sup> We find encouraging results for the prediction of income changes from the LASSO-PMM method. The predictions tend to trace out the actual income changes across the distribution. Therefore, in comparison to the parametric synthetic panel prediction method, the LASSO-PMM provides additional information on income growth, that approximates actual changes quite well. With some exceptions, our predictions overlap with the 95% confidence intervals for all subgroups of poverty dynamics and quintiles of the income distribution. In some cases, the LASSOPMM approach is optimistic, predicting higher income growth for the bottom two quintiles in certain countries. Once again, there is suggestive evidence that results are better when the length of the panel is longer.

# _4.4 Validation: sensitivity and robustness tests_

These findings suggest that the LASSO-PMM approach is well-suited to predict aggregate poverty rates, transitions, and welfare changes with few assumptions and simple models that use harmonized variables. While our empirical approach is fairly general, we now proceed to test whether departures from the main assumptions and changes in how the method is applied affect the precision of the resulting welfare predictions. The main objective of these validations is to test the sensitivity and robustness of the LASSO-PMM approach to different scenarios that may be encountered in practice by other researchers that only have access to cross-section data.

> 29 We also test the robustness of predictions to changes in the forecasting direction and find similar results.

24
Similar to other cross-section poverty prediction methods, applying the LASSO-PMM approach requires making several empirical decisions. We test the extent to which these decisions affect the sensitivity and robustness of the LASSO-PMM approach to obtain welfare dynamic predictions. For simplicity, we divide these exercises into two groups: i) _sensitivity tests_ , in which the actual poverty rates and transitions do not change; and ii) _robustness exercises_ , when we change the estimation procedure, sample, or other attributes that may change the actual welfare estimates. We compare predictions from the LASSO-PMM approach with the actual estimate for the poverty headcount and mobility for the latest available panel in each country.<sup>30</sup> Tables 2 to 7 show the results for our validation exercises by country. The upper panel presents sensitivity analyses, and the lower panel shows robustness tests, with LASSO-PMM estimates from Table 1 in the first row.

We begin with sensitivity analyses. These simulations change parameters that affect the prediction procedure but do not affect the actual poverty rate or poverty transitions. First, we increase the number of neighbors used in the PMM matching procedure from one to five. Second, we increase the number of PMM repetitions from 20 to 40. Last, we reduce the number of PMM repetitions from 20 to 10. The results for all countries suggest that varying these parameters does not affect predicted poverty rates and confidence intervals significantly.

Results for mobility patterns paint a similar picture. The full results for each of the four possible poverty transitions are shown in the last columns of Tables 2 to 7. In general, the confidence intervals of the LASSO-PMM method overlap with actual mobility estimates. These results can be interpreted as evidence that the method is stable to variation in the chosen parameters.

> 30 Sensitivity and robustness results for all panels are omitted due to space restrictions but are available upon request.

25
We also test whether the accuracy of LASSO-PMM changes when considering different sets of variables. In particular, we begin with a model that includes household head gender, age, age squared, and years of schooling (model 1); then add household size and its square (model 2), and then include regional variables (model 3). The model tested so far is model 3 with interactions between the regional variables and all included covariates. Figure 4 shows predictions from these simplified models and compares the results to actual poverty transitions. Overall, we see an improvement in a model’s predictions when we include more time-invariant and deterministic covariates. The models that include household size and its square and region fixed effects are closer to actual poverty transitions than just including household head attributes. Together with the results from our preferred model, these findings suggest that model specification matters.

We now turn to robustness exercises. These simulations involve making changes to the sample, covariates, and weighting structure that changes actual poverty and transition estimates. The lower panels of Tables 2 to 7 present the results for the most recent panel in each country. First, we estimate household per capita income using Ordinary Least Squares (OLS) with PMM instead of LASSO. OLS sets , using all included variables instead of selecting those that minimize the mean squared error. The results between both estimation procedures yield λλ= 0 similar predictions within the confidence interval of true poverty rates and transitions. One plausible explanation is that, if the underlying model exhibits an adequate fit, OLS and LASSO find suitable neighbors throughout the original welfare distribution in a similar way. However, OLS and LASSO might arrive to different predictions in situations in which the underlying model does not have a good fit.

Second, we include household heads aged between 25 and 75 years old in the estimations, increasing the upper limit by ten years. This affects the actual poverty rate because we use a larger 26
sample. The findings suggest that the predictions remain accurate when including household heads older than 65 years of age.

Third, we consider what happens if the model had perfect foresight with respect to the included covariates. Given that we have panel data, we can observe all included covariates in both time periods. Instead of using cross-section controls, we make use of the actual panel structure to include actual variables for the period in which we predict poverty, in our case retrospective variables. Given that we estimate the model for households observed in round 2 and predict their income in round 1, this is feasible in our setup. As already mentioned, all variables are time-variant to some extent. Therefore, this exercise intends to correct for any changes in age, years of education, household size, and migration from households in each of the four countries. In general, using retrospective covariates does not affect performance substantially.

Last, we consider population changes by means of different weighting structures. As mentioned in Section 3, we weight the estimates using the survey weights in each of our data sources. This is common practice in household surveys to carry out inference on the entire population. There are two factors that may affect the performance of LASSO-PMM. On the one hand, statistics institutes in some countries vary their sampling structure from survey to survey. On the other hand, immigration shocks can dramatically alter the population in a country if many people enter or leave the country, which can affect welfare and poverty rates. If either of these events occur, they may affect welfare predictions from LASSO-PMM and other methods because they rely on the assumption that the population is relatively stable between periods.

The baseline LASSO-PMM approach estimates individual-level poverty using household level data and therefore multiplies household-level weights by household size. We try two different 27
alternatives. On one hand, we predict household-level poverty using household-level weights. On the other hand, we predict household-level poverty assuming equal weights across households.

The results in the tables show that changes in weighting structure do not significantly alter the predictions from the LASSO-PMM approach; in general, predictions are statistically indistinguishable.

In unreported results available upon request, we also test whether changing the direction of the prediction affects the estimates. Changing the direction so we predict second round "unobserved” income for households observed in period 1 leads to similar results across the board.

While not shown here, we also conduct these validation exercises on estimated growth incidence curves, finding that LASSO-PMM approximates actual income changes accurately. We also test whether the LASSO-PMM approach is sensitive to changes in the poverty line. We have used the $5.5 a day line throughout this paper, but also consider how the method fares with thresholds ranging from $5.5 to $13 a day.<sup>31</sup> Results show that irrespective of how we define the poverty threshold, poverty rates, transition, and income change predictions from the LASSO-PMM approach are in general statistically similar to the actual welfare estimates using panel data.

In addition to considering poor and non-poor individuals, we also test whether the method can predict welfare not only in two states, but across more than two. This extension is relevant when there is a fraction of “vulnerable” population that are more likely to enter poverty. Previous studies have identified these households as important to policy (Cruces et al. 2012), and we test whether the method is able to predict household defined as poor (income < $5.5 a day), vulnerable (income between $5.5-$11 a day), and middle class (income > $11 a day). The results, available upon request, indicate that expanding the number of population groups does not affect the

> 31 In Chile, we test lines from $6 a day to $14.

28
method’s overall performance. The confidence intervals for LASSO-PMM predictions and actual classifications of households into these categories overlap in general. These tests further support the accuracy of LASSO-PMM to predict welfare dynamics using cross-sections.

Overall, the sensitivity analyses and robustness exercises show that the LASSO-PMM approach for predicting poverty and mobility approximates actual welfare changes. The method performs well considering changes in parameters, estimation procedure, direction of the prediction, sample, covariates, and weighting schemes. The two main conclusions from this validation are: i) changing parameters to make the method more computationally intensive provides only marginal changes; and ii) models with greater explanatory power may improve LASSO-PMM’s predictions. While we focus on generalizability by using a reduced set of harmonized controls to ensure comparability across countries and years, the determinants of poverty vary across countries and over time. This provides researchers in specific countries more leeway since they can use other variables in the models, such as ethnicity, and other retrospective and time-invariant correlates of welfare. As previous studies have shown, machine learning methods tend to work best when the model has high predictive power. While we show that the LASSO-PMM approach works well with a core-set of harmonized explanatory variables, our results suggest that poverty and mobility predictions may improve if the model is chosen adequately. An underlying model with a better fit will produce more accurate neighbors along a larger proportion of the welfare distribution and, therefore, the original distribution will be better replicated. Given these results, we now apply the procedure to harmonized cross-section data to predict poverty, mobility, and income changes in 43 countries across the world.

29
# **5. Application to cross-section data**

We employ the basic LASSO-PMM specification used throughout the paper to estimate welfare dynamics using cross-section data in circa 2010 and 2015, considering one nearest neighbor and 20 PMM repetitions. We take all households in the first round cross-section survey and predict their income in the second round to obtain our main results. The welfare variable used in each of the 43 countries is shown in Appendix Table A.2. Given the number of countries and our harmonized approach to welfare prediction, we only include common variables in the model, which are described in Appendix Table A.3. We begin by predicting aggregate poverty rates, then study predicted mobility transitions, and finish by plotting predicted income changes across the distribution in growth incidence curves.

# _5.1 Predicted poverty rates across the world_

We plot the poverty rate (using a $5.5 a day poverty line) that results from the cross-section data in the second round and the poverty rate that arises using the predicted income in the second round for those households surveyed the first round (Figure 5). First, as expected from the exercise with panel data, predicted and actual poverty rates are similar, given their proximity to the 45-degree line. Second, there is large heterogeneity in poverty across the world. While some countries have low poverty rates, such as Poland, there are nations where most of the population is below the poverty line, as in Rwanda. Despite the large differences across countries, LASSO-PMM approximates aggregate poverty rates well.<sup>32</sup>

> 32 We originally had cross-sectional data for 49 countries. However, countries Bangladesh, India, Pakistan from SAR and Indonesia, Mongolia, and Phillipines from EAP were removed from the analysis because they presented substantial differences between actual and predicted poverty rates.

30
In the previous section, we highlighted that in order to predict well, the model matters. While we use a harmonized approach that includes a restricted set of variables, the models we fit have better predictive power than in previous studies. The second panel of Figure 5 presents the cumulative distribution of R-squared coefficients from the core papers in the welfare dynamics prediction literature and compares them with our estimations.<sup>33</sup> The figure shows that the Cumulative Distribution Function ( _cdf_ ) in our application is shifted to the right, which implies that the models we estimate have better predictive power. Roughly 80 percent of our estimations have an R-squared greater than 0.30. This means that despite using a restricted set of harmonized variables, the estimated models have good predictive power. In country-specific applications that allow the inclusion of additional predictors of welfare, the explanatory power of implementing LASSO-PMM may rise further, potentially leading to greater precision in welfare prediction.<sup>34</sup>

# _5.2 Predicted poverty transitions across the world_

We proceed to study predicted poverty transitions. We use the same four categories as in the previous section, but for simplicity refer to them as _chronic poor_ (Poor, poor), _downward mobile_ (Non-poor, poor), _upward mobile_ (Poor, non-poor), and _never poor_ (Non-poor, non-poor).

Figure 6 presents the predicted share of individuals in each of these categories by region. For our sample of 43 countries, the figure suggests that there has been considerable mobility; 22 percent of individuals changed their economic status. Moreover, the figure shows that, even in a period of poverty reduction (i.e., poverty decreased by 5 percentage points for this set of countries), about 9 percent of the population fell into poverty. Finally, the figure suggest that a large part of the population is immobile, about 24 percent of all individuals remained chronic poor.

> 33 The papers considered are: Hérault and Jenkins (2019), Dang and Dabalen (2018), Dang and Lanjouw (2013), Dang, Lanjouw, Luoto and McKenzie (2014), Dang and Ianchovichina (2018) and Lucchetti (2017).

> 34 We get similar results when predicting first round income for those surveyed in the second round. These results are available upon request.

31
This picture does not account for heterogeneity across and within regions. The results suggest that there are three types of regions. The first group has low chronic poverty levels and includes Europe and Central Asia (ECA), Latin America and the Caribbean (LAC), and East Asia and the Pacific (EAP). The second group, composed by South Asia (SAR) and Middle East and North Africa (MNA) lies on middle ground, where roughly one quarter to one third of the population seems to be chronically poor. The last group is composed by the Sub-Saharan Africa (SSA), where almost 80 percent of the population is chronically poor. Appendix Figure A.2 presents transition categories by country, sorted in increasing order by their chronic poverty level. Uruguay presents, for instance, one of the lowest chronic poverty levels, while on the other hand, Rwanda has one of the highest values. Interestingly, there are also large heterogeneities within regions. For instance, Kyrgyz Republic has a relatively large chronic poverty rate, probably comparable to African countries, but it is geographically surrounded by a group of countries with low chronic poverty.

These predictions suggest that a sizable fraction of individuals across the world remain trapped in chronic poverty, although this share varies by region. These predictions can help governments in different regions and countries to target policies that mitigate welfare losses for some households and that promote upward mobility at all levels.

_5.3 Changes in income and growth incidence curves across the world_ As mentioned throughout this paper, one of the advantages of the LASSO-PMM approach to welfare prediction is that it allows producing a point estimate of unobserved household income in any given period. This feature is particularly interesting because it allows looking at income changes at individual level (i.e., non-anonymized). We can therefore analyze what drives, or not, movements into the different categories: initial income or income changes.

32
Welfare changes depend on the initial level of welfare and the magnitude of changes in welfare. For instance, people can remain poor in both periods either because they start with an income far below the poverty line or because their income growth over time is not sufficient to lift them out of poverty.

To study whether predicted transitions are due to initial welfare or its respective changes, Figure 7 presents the initial and final welfare by region and the aggregate set of 43 countries. These welfare changes are presented for the same four groups: chronic poor, downward mobile, upward mobile, and never poor. Various findings are drawn from this figure. First, for the 43 countries, initial average income for the chronic poor ($2.8) was slightly lower than for the upward mobile ($3.8). Second, average income grew significantly for the upward mobile (to $11.0), while it remained almost constant for the chronic poor (final income was $2.9 on average for this group). Third, the never poor had an initial income substantially higher on average ($18.1) than those who were non-poor but move into poverty ($11.0). Finally, the average income of those who remained out of poverty almost did not change (final income was $17.2 on average for this group), while it decreased considerably for the downward mobile (final income was $3.9 on average for this group).

These magnitudes vary by region, although the overall message remains the same. For instance, the largest income losses for the downward mobile occur in Sub-Saharan Africa and Latin America and the Caribbean. Welfare gains are also highest in those two regions for the upward mobile. The remaining regions see lower changes, with losses for the downward mobile of around 55 percent and increases for the upward mobile close to 140 percent.<sup>35</sup>

> 35 Appendix Figure A.3 presents the same results at the country-level.

33
We also estimate a growth incidence curve for the 43 available countries to determine whether predicted welfare changes are pro-poor or not. One important advantage of the LASSOPMM is that, unlike traditional growth incidence curves developed by Ravallion and Chen (2003), we do not need to rely on the rank preservation assumption. On the contrary, for each household surveyed in the first round, we can estimate the annualized income growth at individual level, using the predicted income the second round. Afterwards, we construct centiles (deciles) at world (regional) level using the baseline income and we compute the average of the individual income growth for each group. Pooling all countries together, we observe in Figure 7 that annualized income growth is positive until the 68<sup>th</sup> percentile and becomes negative afterwards, suggesting that predicted income growth is pro-poor. Observing differences by region highlights some heterogeneity in these predictions. For example, income growth in the SAR region is always positive until the 8<sup>th</sup> decile, while in ECA the lower half of the distribution is predicted to grow, while the top half is not.

# **6. Conclusion**

This is the first paper, to the best of our knowledge, that uses supervised machine learning and multiple imputation techniques to estimate welfare dynamics in the absence of panel datasets. The LASSO-PMM approach proposed in this paper estimates household-level welfare using a set of deterministic and time-invariant harmonized variables as controls and employs Predictive Mean Matching (PMM) with a household’s nearest neighbor to provide estimates of poverty rates, transitions, and income changes. Compared to previous poverty prediction methods in the literature, LASSO-PMM makes fewer assumptions and estimates both poverty transitions and income changes. This research presents, validates, and applies the method using harmonized data 34
from 36 panels in four Latin American countries and cross-section data from 43 countries across the world.

In general, validation results indicate that LASSO-PMM predictions are statistically indistinguishable from actual household level poverty rates, transitions, and income changes. A series of validation exercises support the method’s overall performance. We divide these exercises into two groups: i) _sensitivity tests_ , in which the actual poverty rates and transitions do not change; and ii) _robustness exercises_ , when we change the estimation procedure, sample, or other attributes that may change the actual welfare estimates. The validation exercises show that the LASSO-PMM approach for predicting poverty and mobility approximates actual welfare changes. The two main conclusions from this validation are: i) changing parameters to make the method more computationally intensive provides only marginal changes; and ii) models with greater explanatory power improve LASSO-PMM’s predictions. These findings are sufficiently encouraging to suggest that estimating welfare dynamics using LASSO-PMM may accurately approximate actual welfare indicators in settings where panel data are unavailable. Interestingly, the robustness exercises show that OLS and LASSO yield similar results. A plausible explanation for this is that, if the underlying model exhibits an adequate fit, the two estimation methods find suitable neighbors throughout the original welfare distribution in a similar way. However, the two estimation methods might arrive to different predictions in situations in which the underlying model does not have a good fit.

We then apply the LASSO-PMM method to harmonized cross-section data. Aggregate poverty rates are accurately approximated when using cross-sections across a heterogeneous group of countries. Mobility probabilities are also estimated at the world, regional, and country-levels. Worldwide, we predict that 24 percent of all households are in chronic poverty, 9 percent are 35
downward mobile, 13 percent are upward mobile, and 53 percent are never poor. The method estimates that downward mobile households see their welfare fall by 65 percent on average, while upward mobile households escape poverty with a mean welfare increase of 190 percent. We also predict growth incidence curves and find evidence of a marked pro-poor growth.

This paper contributes to two strands of literature. First, it contributes to the methodological discussion on welfare dynamic prediction methods when panel data are unavailable. The LASSOPMM approach presented in this paper embraces machine learning methods whose primary goal is accurate prediction, makes less restrictive assumptions than previous synthetic panel methods, and provides a simple framework that is easier to replicate in practice. The paper also highlights that the quality of the predictions is directly linked to the quality of the underlying welfare model used. Previous papers present predictions that are statistically indistinguishable from actual household poverty transitions with a relatively low fit. This is not the case for predictions in this paper; the quality of the underlying welfare model is key. Although the paper is restricted by a limited set of harmonized variables, the models we estimate have greater explanatory power compared to previous methods. Second, we contribute to the discussion of how the results from welfare prediction methods can be used to guide policy. The application of LASSO-PMM is carried out on 43 cross section datasets. To our knowledge, no other paper in the literature has used this amount of microdata to estimate welfare dynamics using cross-sections. The results from applying LASSO-PMM to cross-section data yield not only predicted poverty rates in the aggregate, but also the share of individuals who are upward and downward mobile, a key population for targeting anti-poverty policies. The results also provide insight on the magnitude of income changes that drive individuals both into and out of poverty, which is also useful knowledge for development efforts.

36
The LASSO-PMM approach described in this paper presents several advantages compared to other methods. For instance, unlike the parametric synthetic panel, the LASSO-PMM allows to calculate household-level income changes instead of just probabilities of poverty transitions. In addition, unlike the parametric approach, the LASSO-PMM does not require additional estimation of unknown parameters, such as the age-cohort correlation of residuals from cross-sections. Finally, instead of allocating residuals from the first-round regression to estimate first-round incomes for households interviewed in the second round, as in the upper bound non-parametric synthetic panel method, the LASSO-PMM directly imputes the observed total income from a firstround near neighbor observations. Therefore, all first-round sources of incomes (e.g., labor income, public social transfers, private transfers, capital income, rent, etc.) from the same neighbors can also be imputed in the second round. As such, future research could then analyze the contribution of each source of income to poverty and welfare changes between two moments in time.

37