rafmacalaba/data-use-mentions-extended huggingface dataset:
# **4.1. Guidelines for practice**

1. _Highly autocorrelated outcomes:_ For outcome measures like anthropometric measures or test scores, for which the autocorrelation is high (e.g. ρ=0.6 to 0.8), always include at least one baseline. Difference-in-differences and ANCOVA have much greater power than POST in these cases. Moreover, Bruhn and McKenzie (2009) also show that the power improvements from stratified or matched randomization are highest when ρ is high. However, if the total sample size is fixed, a single baseline and follow-up will offer more power than multiple pre-treatment or post-treatment rounds when ANCOVA is used. If the treatment and control sizes are fixed, the gains from going from one to two post-treatment surveys can still be non-trivial in these cases, but there is little gain from more than say 3 post-treatment surveys.

_2. Outcomes with low autocorrelation:_ For outcome measures like business profits, incomes, or expenditure, for which the autocorrelation is typically low (e.g. ρ=0.20 to 0.40) , it can be optimal to have no baseline at all, and just do a single follow-up survey if the total sample size is limited. If the treatment group size is fixed, researchers can dramatically increase power by doing multiple post-treatment surveys. Even if a baseline is taken, researchers should not use difference-in-differences in such cases, since doing so has much lower power than either POST or ANCOVA analysis.

# **4.2. Other practical issues**

In practice there are several other factors that should guide researchers in choosing how many survey rounds to conduct. I note some of the key factors here, and their implications for choice of the number of pre-treatment and post-treatment waves:

1. _Over what horizon is a pooled treatment effect relevant and of interest?_ The analysis in this paper has assumed a common treatment effect γ. Of course if the treatment effects vary over individuals, researchers are usually content to
on the best techniques to use when measuring household or self-employment. This results in employment and earnings data that lack relevance to the country setting, and mismeasure the key features of employment (non-sampling error). Not surprisingly, in their detailed analysis of surveys from 23 countries, Guarcello et al (2010) found that the most consistently measured employment variable was wage employment.

The purpose of this paper is to _highlight the sensitivity of employment outcomes to questionnaire methodology and show the analytical difficulties in trying to use survey data from SSA countries (where questionnaire methodologies are inconsistently applied and non-sampling error appears to be large) to understand how employment and earnings patterns have changed over time, let alone forming of policies based on such analysis._ While it uses Ugandan data as an example<sup>3</sup> , the objective of the paper is not analyzing recent trends in employment outcomes in Uganda or any other country. Rather, this paper is about why employment data from SSA countries are unreliable and what issues analysts face in using these data to try to identify employment outcomes and relate them to government policy or household behavior, for the purposes of diagnosing past problems or establishing a baseline to track progress.

The paper mainly relies on Uganda data simply because the labor module in its surveys has significantly evolved over time and contains variations both within and across surveys that can be exploited

> 2 <u>http://www.worldbank.org/ida/results.html, accessed on August 15, 2012.  Shining a spotlight on an indicator maybe necessary but clearly not</u> sufficient to improve quality. Despite the attention paid to poverty reduction in the past twenty years, it could be argued that the collection of the consumption data for needed for calculating absolute poverty is plagued by similarly high non-sampling errors. See for example, Jerven (2012). 3 This paper is not on how to interpret Uganda’s employment trends over time; interested readers on this topic may wish to consult Fox and
6.7|-7.0|
|Income total, %|-1.6|-1.7|-1.8|-2.0|
|Absolute poverty rate of the formalized, p.p.|0.0|5.1|4.1|3.7|
|<br>Absolute poverty rate, p.p.|0.8|0.9|0.9|1.0|
|<br>Gini index, p.p.|0.0|0.0|0.0|0.1|
|Total budget balance, %of GDP|-0.9|-0.7|-0.5|0.0|

Source: Authors’ calculations using RUSMOD for 2017 policy year and 2016 RLMS-HSE data.

Figure 9 shows the results of reform 2 with the assumption that nobody formalizes by income deciles and socio-demographic groups. In this case the distributional impact of the tax maneuver completely depends on our assumptions about the economic incidence of SIC. If it falls on workers (scenario B), the benefit from SIC reduction is higher than the additional burden of higher VAT, and is beneficial for all the income deciles. The relative effect is of similar size across the distribution. Not surprisingly, households with fewer children and adult only households benefit the most. The situation reverts if the economic incidence of SIC reduction does not favor employees (scenarios A and C). In this case, households are assumed to carry the burden of higher VAT and no additional benefit due to lower SIC, which results in a negative effect for all the population subgroups.

24
third the length, and one enumerator did the work of a team of three with the monitoring tools, the QAS tools were roughly a third of the staff costs to implement as the diagnostic research tools (see appendix 2). Given the relative consistency between data collection firm and MoETE supervisors, who were randomly assigned, one important implication of our results is that existing staff may well be able to undertake QAS data collection; although there may be some opportunity cost in shifting their time allocation to the QAS, this may make implementation and eventual scale-up more feasible.

While ministry staff can be used for cost savings, other aspects of the QAS system may merit more investment. Longer observation times, for example, may lead to more consistent results. Ongoing efforts to validate and improve question and response design may make the QAS more informative. Increasing the length of training, as well as the quality of training are important areas that may improve the quality of the QAS data itself. Which of these improve consistency and accuracy most cost-effectively is an important question that merits further research. One challenge in Egypt was that we were unable to obtain local videos of teaching in classrooms, due to security and privacy concerns, so enumerators trained on international examples. Investing in local training materials may help improve the functioning of the QAS. The enumerators all did pass reliability quizzes (80% correct or above), but this may not be sufficient to ensure high reliability during fieldwork.

Collecting a full census of QAS tool data from all schools on a regular basis can also be valuable, but may be costly, and resulting data should be used with caution when informing quality enhancement actions. QAS tools should be low-stakes, not high-stakes, and treated as formative not summative. There can sometimes be positive potential information/incentive effects of quality information being released, as with school report cards (Aturupane et al. 2014; Andrabi, Das, and Khwaja 2017; Bassok, Dee, and Latham 2019). We recommend against using QAS results
Second, the project may have had its principal impact before the first survey in February 2016, in which case the difference between the two surveys might not capture project-induced improvements. We examine the Annual Performance Assessments (APAs) of the project to discern which components show the greatest increase in project measures of institutional performance. In Figure B1 we present the mean scores across project LGAs in each of the performance assessments. Our first round survey was implemented in early 2016, around the fourth APA, while our second round survey happened in 2018, close to the sixth APA. We find that the "Accountability" component of the APA shows a large increase between the 2nd and 3rd APAs, while the other components (Revenues, Infrastructure and Urban Planning) do not show a pattern of concentrated growth before our first survey round. (We discuss the available evidence in more detail in Appendix B.) It could be that had we undertaken a citizen survey in 2013, we may have found that the 18 project LGAs had much lower levels of citizen-survey-based Accountability measures than the 22 non-project LGAs in 2013? What if the project incentivized the 18 LGAs who were reluctant to publicly post their budget information in the absence of these project incentives; what if our 2016 survey measure, compared to this hypothetical 2013 survey measure, shows that Accountability significantly improved, as a result of the project incentivizing the LGAs to reach out to citizens? We cannot rule this out because we do not have survey data from 2013 across both project and non-project LGAs. We find no significant difference between project and non-project LGAs in the 2016 survey measures of citizen engagement targeted by the Accountability component, nor in changes in citizen engagement between 2016 and 2018.

We argue that our two rounds of surveys appear well positioned to capture something proxying a baseline for outcomes in February 2016, before the project was disbursing substantially, and improvements over a two year period as the project is actively implemented. The 2018 Quality Assurance Review (a report that reviews the project’s implementation
In the next section, we explain the rationale for the PMT as a tool for measuring poverty. In Section III we describe the data and general set‐up of the PMT formula. In Section IV we present results and discuss the selection of the best‐performing models. This section demonstrates that PMT simulations outperform the targeting approach used in the Samurdhi program. Section V presents results of the robustness checks that we conduct on our primary PMT model. The final section presents conclusions and suggested next steps.

# II. The Rationale for a Proxy Means Test

PMT‐based targeting has been adopted in many low‐ and middle‐income countries since the 1990s. Case studies on performance in terms of targeting incidence suggest that the PMT approach has been very successful and outperformed other targeting mechanisms (Grosh, 1994). For example, in Chile and Mexico approximately 90 percent of social assistance reached the bottom 40 percent of the population when a PMT was adopted (Castañeda and Lindert, 2005). Beyond Latin America, Armenia introduced a PMT in 1994 for targeting cash transfers (World Bank, 1999, 2003), Indonesia has adopted PMT for targeting subsidized rice rations and transfers following the removal of fuel subsidies (Sumarto et al., 2000), and Turkey introduced PMT‐based targeting in 2002 (Ayala, 2003). In the South Asia region, PMT‐based targeting systems have been employed in Pakistan and Bangladesh (Sharif, 2009; Hou, 2008).

The PMT determines eligibility for a program based on an index of easily observable household characteristics, such as the quality of the household’s dwelling, ownership of durable goods, demographic structure, and the education and employment status of adult household members (Grosh et al., 2008). Statistical analysis of data from household surveys is normally used to refine the set of indicators included in the index and to estimate their respective weights. Information collected from each program applicant is then used to construct a household‐specific ‘PMT score’, which
hard assets,’ and ‘real things’ that are the essentials of not just your life but the lives of everyone in the world.” Similarly Heap (2005), noted that China’s high rates of urbanization, industrialization, and capital for‐ mation set the stage for a super cycle, similar to the ones experienced during the late 1800s and early 1900s (driven mainly by growth in the United States) and the years fol‐ lowing the Second World War (prompted by reconstruction in Europe and the Japanese expansion.) <mark>Several papers focused on super cycles, especially for metals (Radetzki 2006; Cuddington and Jarrett 2008; Erten and Ocampo 2013).</mark> <mark>Concerns were heightened about energy during 2011‐13 when oil prices averaged above $100/bbl. References to the “peak oil” hypothesis intensified as oil prices peaked.</mark><sup>**10**</sup> <mark>Income growth in China and India was cited as the main factor behind the price increases of food commodities, especially following the 2007‐08 food price spike. Indeed, the June 2009 issue of the</mark> _<mark>National Geographic</mark>_ <mark>noted that “... as countries like China and</mark>

> **10** In May 2008, for example, “peak oil” appeared 100 times in the news—the crude oil price averaged $123/bbl that month (in January 2016, it appeared only 3 times and the crude oil price averaged $30/bbl). The average monthly appearance of “peak oil” dropped from 42 times during 2008‐11 to 3 times during 2017‐18 (based on Google Trends, “worldwide”, “all categories”, “news search”).

_— 33 —_
interactions at the commune-level, i.e., the F2F index. The F2F index is interacted with time fixed-effects under a difference-in-differences (DiD) framework to estimate the differential effect of COVID-19 across communes. The DiD model to estimate post-lockdown aggregate impact is as follows: Likewise, the corresponding model to estimate impact observed in each month is

> 19All of our results are robust to using the raw reported income from LFS.

8
, the downward trend became much steeper and the global equilibrium interest rate dropped from around 2.0 percent in 2000 to below 0.5 percent in 2003. It remained low in the following years and took another hit due to the global financial crisis. It plunged to negative territory in 2009 and declined further to a minimum of almost -1.0 percent in 2013. Since then it recovered only slightly and in 2019, the last year in the sample, it stood at -0.5 percent. The uncertainty around the estimates is not larger than for many other estimation strategies, despite this sample including many EMDEs. The

> 18 The average time-varying volatility across advanced economies and EMDEs is plotted in Appendix A3. 19 For a detailed discussion of how the global equilibrium rate is related to country-specific equilibrium and policy rates, refer to Rachel and Smith (2015).

> 20 GDP values in constant 2010 US$ come from the World Bank and OECD national accounts data. China is included in a robustness check in section 4.3.

9
referenced UNEP/GRID‐Europe storm track dataset is used to separately explore the effects from high winds and gusts from cyclones. As the damage from heavy rainfall associated with cyclones will already be captured by the localized and riverine flood indicators, these measures will capture the wind damage associated with cyclones.<sup>5</sup>

# _Rainfall maps_

To construct spatially disaggregated indicators of drought and excess rainfall, the weather station data must first be spatially interpolated and cut‐offs must be determined beyond which precipitation is considered too little or too much. Spatial interpolation of weather data is a discipline in itself which has developed an array of techniques that go from the most crude and obvious (such as the nearest neighbor techniques<sup>6</sup> ) to the more precise and resource intensive (such as the radial basis functions).<sup>7</sup> Nonetheless, as the complexity of the techniques increases, they also display rapidly decreasing returns in terms of precision given resources spent.<sup>8</sup> As the focus of the paper is

> 4 The frequently used CRU TS2.1 monthly rainfall database (Mitchell and Jones, 2005) only has a half degree resolution, which is around 55 kilometres at the equator.

> 5 See Thomas (2009) for a detailed description of the different data sources and a more elaborate description of the construction of the different natural hazard maps.

6 The nearest neighbour method assigns the value of the weather station nearest to the grid cell centre. 7 Radial basis functions are functions of distance used for exact interpolation of point data. They attempt to overlay a smooth surface on the point data, allowing for various degrees of bending in that surface through a smoothing parameter that is specified by the researcher. Common radial basis functions are thin‐plate spline and multiquadratic functions. See De Smith et al. (2007) for a comprehensive textbook that is reasonably accessible.

8 In a comparison of the performance of different interpolation techniques
So the main conclusion to be drawn from this research is that the precise definition of informality does matter. It cannot be assumed that alternative definitions are broadly consistent, and that the conclusions drawn from any detailed analysis or modeling of informality will be robust to definitional choice. This is particularly important given recent developments in the debate on the subject. Any particular choice of definition or measurement may be conditioned by a priori perspectives or hypotheses about the nature and role of the informal sector.

The remainder of the paper is structured as follows. Section 2 provides a more detailed discussion of the existing literature and in particular on the way in which informality has been defined and measured. Section 3 discusses the Brazilian data source used and the different measures of informality which are analyzed. Section 4 provides some descriptive analysis and discussion of these measures. Section 5 documents a multivariate analysis to investigate commonalities and differences in the way in which the different indicators of informality are associated with demographic, employment and household characteristics. Section 6 provides a concluding discussion.

# **2. Defining Informality**

The practice of defining informality in terms of the economic status of the individual (self-employed or own-account worker, domestic worker, small scale employer etc.) in conjunction with the characteristics of the employer (small scale, no collective bargaining etc.) is well-established in the literature. A substantial number of early analyses of informality simply 6
where F is the probability function that is determined by the structure of the errors.

The overall approach is to estimate a broadly standard specification augmented with some additional proxies and modified to better capture liquidity effects. We then introduce integration related variables and explore to what degree their effects are channeled through the labor market.

# **III.  Data**

<u>Migration data:</u> The 2000 census generates migration flows from a question that asks in what state the interviewee resided 5 years ago. Though this approach is standard in the literature, it has the drawback of obscuring migrants who may have left and returned in the five year period. Flows to the US are derived from a question asking whether during the last five years a member of the household has gone to the US.

<u>Moving Costs:</u> Following the literature (again, see Greenwood 1997), we approximate the costs of transportation as a quadratic function of distance.<sup>4</sup> This is a proxy for the costs of migration that consist of the moving costs themselves, the opportunity costs of moving which rise with the length of the journey, and rising communication costs with the family in the point of origin, including the increased costs of return visits. In general, the literature expects a negative impact on migration but with decreasing effect.

<u>Networks:</u> An emerging literature stresses the importance of existing networks as lowering transaction and information costs. The Mexico-US literature has particularly expanded on this point (see, Zabin and Hughes 1995, Massey and Espinosa 1997, Winters, de Janvry and Sadoulet 2000, McKenzie and Ropoport 2004 among others).

> 4 Though we assume that the indirect utility function is linear and the weight of each variable is similar in each region, this assumption can be easy relaxed to differentiate the origin (o) and destination (d) parameters.

5
19|Manufacture of furniture|2076|16|0.8|0.0|
|20|Water collection, treatment and supply|255|3|1.2|0.0|
||Total employees (in thousand)|233022|65645|22.0|100|

Source: ILOSTAT database, International Labor Organization Even the latest National Sample Survey (NSS) report on employment and unemployment in India (2013)<sup>14</sup> highlights that about 75 percent of female workers were engaged in the agricultural sector while only 44 percent of the male workers were engaged in the sector during this period. Similarly, 26 percent and 30 percent male workers were engaged in secondary and tertiary sectors and the corresponding proportions for female workers were 17 percent and 8 percent, respectively. Moreover, the industrial distribution of the workers in the urban areas was distinctly different from that of rural areas. In urban areas, the share of the tertiary sector was dominant followed by that of secondary sector while agricultural sector engaged only a small proportion of total workers for both male and females. In rural areas, nearly 59 percent of the male workers and 75 percent of the female workers were engaged in the agricultural sector while the share of urban workforce in agricultural sector was nearly 6 percent for male workers and 11 percent for female workers. In urban areas, about 59 percent of male workers and 55 percent of the female workers were engaged in the tertiary sector.

> 14 Key Indicators of Employment and Unemployment in India, 2011-12, NSS 68th Round, Ministry of Statistics and Programme Implementation, Government of India, June 2013.

15
p<0.1. We report standard errors in parenthesis. Variables which enter the propensity score include: age, number of siblings, household size, lives with mother, lives with father, health problems, whether currently working, household ownership of DVD, television, MP3 player, cellular, phone, computer, calculator, student teacher ratio, whether the school has a computer, whether the school has rooms with more than one grade, average teacher education levels and marginality index components. Community level variables include the margination index, proportion illiterate, proportion of households without piped water, proportion without electricity and proportion without drainage.

In summary, the results for sixth grade children enrolled in schools in medium to very high marginalized areas show significant increases in enrollment, effects which persist through high school. With respect to secondary schools attended the results indicate that the program overall increases the probability of attending a tele-secondary school and reduces the probability of attending a technical school. At the high school level, the program increases the probability of attending a general high school and reduces the probability of attending a technical high school. There is no effect of the program on enrollment in private schools. With respect to effects on achievement, the results are generally small and variable according to the estimator used. On balance, the results are statistically significant but not large in magnitude, although, due to selection in enrollment, the estimates are susceptible to downward bias.

Results for indigenous children show similar trends, but somewhat larger. Enrollment effects are on the order of 10 to 12 percentage points increases in the probability of attending secondary school and on the order of 8 percentage points for high school. Effects on achievement were larger in Spanish at about 0.1 standard deviation but generally not significant in the area of mathematics. Effects of 0.1 standard deviation however compare favorably to a recent survey of impacts of educational interventions around the world (see <u>https://www.povertyactionlab.org/research-resources/cost-effectiveness).