rafmacalaba/data-use-mentions-extended huggingface dataset:
Policy Research Working Paper 9980

# **Abstract**

This paper estimates the effectiveness of government support to the private sector during the COVID-19 pandemic in El Salvador and Georgia using firm-level data collected before and during the pandemic. The two countries are selected because eligibility criteria for support involved pre-pandemic features of firms, as opposed to more prevalent criteria directly linked to firms’ experiences during the pandemic and that greatly exacerbate concerns about selection bias in estimation. Four outcome variables are studied relating to firms’ workforce, hours of operations, and expectations. Matching and panel estimation techniques are used on full and restricted samples, with the latter aimed at reducing selection bias. Government support appears to have helped firms avoid a reduction in operations in El Salvador, mainly through cash transfers, which also helped in terms of permanent workers, with the latter effect counteracted by wage subsidies. Smaller firms in Georgia appear to have benefited more from government support, mostly through fiscal relief, which was partially counteracted by wage subsidies that benefited larger firms more. The finding that smaller firms have benefited more helps raise confidence in the analysis as strong negative selection bias is expected in this context. Manufacturers of textiles and garments in El Salvador and hotels and restaurants in Georgia appear to have benefited from government support, but the patterns in other sectors are mixed and country-specific, highlighting potential complexities of attempting to target sectors.

This paper is a product of the Global Indicators Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at nkaralashvili@worldbank.org and mtamkoc@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if
World Bank. 2006. Review of Selected Railway Concessions in Sub-Saharan Africa. Washington, DC: World Bank.

———. 2009. Global Economic Prospects: Commodities at Crossroads. January. Washington, DC: World Bank.

———. 2014a. “Transitional Policies to Assist the Poor While Phasing Out Inefficient Fossil Fuel Subsidies that Encourage Wasteful Consumption.” Contribution by the World Bank to G20 Finance Ministers and Central Bank Governors. Washington, DC: World Bank (September).

———. 2014b. MENA Economic Monitor: Corrosive Subsidies. October. Washington, DC: World Bank.

———. 2018. A New Economy for the Middle East and North Africa. MENA Economic Monitor. Washington, DC: World Bank.

———. 2019. Global Economic Prospects: Darkening Skies. January. Washington, DC: World Bank.

———. 2019b. Doing Business 2020: Sustaining the Pace of Reforms. Washington DC: World Bank.

World Trade Organization. 2000-2019. Trade Policy Review. World Trade Organization, Switzerland.

———. 2019. “Distortion.” WTO Glossary. Geneva, Switzerland: World Trade Organization, Geneva.

17
# **5. Conclusions**

Given that it is rare to evaluate development projects by repeated observation over a long period, this case study has provided an opportunity to study a less costly method, based on respondent recall using subjective-qualitative questions. Success for this method would open up many low-cost opportunities for learning about development effectiveness.

Neither the ―expensive‖ nor ―shoestring‖ double-difference estimates suggest that the poor-area development program studied here had a significant long-term impact on living standards in poor areas of rural China. But their agreement is not because the retrospective qualitative assessments provided good proxies for the changes in consumption derived from high-quality contemporaneous surveys. Indeed, the analysis suggests that long-term recall of the household’s overall standard of living contains only a weak and biased signal of changes in consumption. Controlling for the actual change in consumption, the recalled improvement in living standards tends to be higher for initially richer households. There are clear signs of telescoping in the recall responses, but the bulk of the benefits occurred in the earlier half of the recall period, which is given too little weight by respondents in treatment villages. Recall is clearly also affected by many idiosyncratic factors not accountable to consumption.

Furthermore, there is an indication that the shoestring method can be deceptive. By not being able to effectively address the problem of selection bias based on the unobserved factors that determined which villages got selected for the program, the recall method becomes vulnerable to spurious impact signals. In this particular case, the recall method suggests positive impacts after controlling for observed differences between treatment and comparison villages at the baseline. The paper has argued that a plausible interpretation of this finding is that the selection bias based on observables is working in the opposite direction to that based on unobserved factors. Thus, only reducing the former bias (by balancing the distribution of observables between treated and comparison units) makes matters worse.

So (alas) this case study does not offer much encouragement on the reliability of this ―shoestring approach.‖ Of course, this
adult literacy rate (18+), the proportion of adults with no education, and the proportion of adults with only primary education were comparable between 2015 and 2016 (Table 8). Similarly, cultural norms should be expected to remain stable, such as the prevalence of polygamy and the gender of the household head, both of which are again seemingly unchanged. Some types of infrastructure can provide good indicators if they are not susceptible to be destroyed in the fighting. Mobile

> 15 Replacement EAs were approved by the project manager. Replacement of households were approved by the supervisor after a total of three unsuccessful visits of the household.

> 16 At the very start of Wave 3 data collection year‐on‐year inflation was equal to almost 650 percent. The CPI between the start of Wave 1 and the end of Wave 3 had increased by almost 1,600 percent. Similarly, more than a third of the population was displaced by mid‐2018.

6
|**Variables**|**No Shock**|**Mean**|**Shock**|**Mean**|**Mean Difference**|
|---|---|---|---|---|---|
|Per capita other expenditure (in SDGs)|2,548|1.32|5,365|0.903|0.418***|
|Total per capital expenditure (in SDGs)|2,548|177.35|5,365|156.89|20.469***|
|Household asset index|2,548|42.16|5,365|36.64|5.516***|
|Household tropical livestock index|2,548|1.08|5,365|2.66|−1.575***|
|Household size|2,548|5.75|5,365|6.37|−0.626***|
|Male-headed household|2,548|0.83|5,365|0.82|0.01|
|Age of the household head|2,548|45.74|5,365|45.85|−0.11|
|Total income in previous month (in SDGs)|2,544|1,672.69|5,356|462.88|1,209.81|
|Total income in previous year (in SDGs)|2,544|7,483.45|5,356|6,524.2<br>5|959.20|

_Source_ : Authors’ calculation using 2009 NBHS data. *** p<0.01, ** p<0.05, * p<0.1 Table 2: Test of Means between Households Affected
varying reference periods of one week, month, or year to demonstrate sensitivity of employment indicators to this factor.

# 3. **Trends**

Figures 2 to 9 illustrate trends and patterns in male and female employment in South Asia. We used multiple surveys in the aggregate graphs to look at employment rate sensitivity to recall periods. For the disaggregated graphs, we only used the weekly recall period to determine employment status and restricted ourselves to surveys in which pertinent information was available. All estimates use individual sample weights.

# _Aggregate_

As shown in Figure 2, the female employment rate—the number of employed women as a share of the working-age population—remained stable since 2001 in Sri Lanka, Pakistan, and Bangladesh, regardless of survey instrument or recall period. Also, there is a persistent gap of approximately 42 and 55 percentage points between male and female employment rates in Sri Lanka and Pakistan, respectively.

In Nepal, India, and Bhutan, however, we observe greater sensitivity in employment indicators. As expected, yearly reference periods yield higher employment rates compared to when respondents are asked about activities only in the last week. For Nepal, a clear trend does not emerge; using the NLSS, the female employment rate declines based on yearly recall (95 to 75 percent between 2003 and 2010) but slightly increases when considering weekly recall. The category “extended economic activities” is not included in this estimate to remain consistent with employment definitions in other surveys. In India, we observe a consistent pattern of falling female employment rates across surveys.

In Bhutan, women are about 9 to 15 percentage points less likely to be employed than men (depending on the year) when respondents recalled activities in the past week rather than the last year. However, women’s employment clearly declined from 69-78 percent in 2003 to 50 percent in 2012, a sharper decline for women than for men.

In Afghanistan, the male employment rate is similar to other countries, and male employment also experienced a relative decline towards
to South Africa although they have followed different trajectories since 2010. Machinery exports declined by 40% in nominal terms between 2010 and 2016 and the sector lost relative importance, dropping from accounting for about 55% of exports to South Africa in 2010 to less than 15% in 2016. On the other hand, apparel exports increased five-fold in nominal terms during the same period and increased its share of exports from 20% to 55%.

Lesotho has not taken steps to exploit potential services export expansion. Despite the sector’s growing importance to GDP and employment, Lesotho’s tourism sector continues to underperform, and has seen a decline. In 2015, Lesotho was ranked 129 of 141 countries by the World Economic Forum’s Tourism Competitiveness Index (WEF 2015). Among comparator countries, Lesotho has the third lowest ratio of foreign to domestic spending on tourism, after two other landlocked countries, Eswatini and Malawi. Lesotho could differentiate itself from other Southern African countries, with mountain ranges, clear water and snow in winter. To capitalize on its tourism potential, Lesotho needs to take some steps to enhance its infrastructure and train its human resources. Of 82 accommodation enterprises surveyed by a group of World Bank economists (Yiu and Saner 2015), less than 40% were considered “market ready”.

# 4. Impacts of potential future external trade policy changes

## 4.1. Data, Modeling Framework and Scenarios

The CGE analysis aims to assess potential impacts based on specific trade-related scenarios, incorporating Lesotho in the GTAP database for the first time. The core specification of LINKAGE replicates largely a standard global dynamic CGE mode (van der Mensbrugghe, 2011 and 2013). Production is specified as a series of nested constant elasticity of substitution (CES) functions for the various inputs – unskilled and skilled labor, capital, land, natural resources (sector-specific), energy and other material inputs. LINKAGE uses a vintage structure of production that allows for putty-semi putty capital. In the labor market we assume full employment
**Chart 2: Price of Water by Type of Service Provider**

<!-- Start of picture text --> $11.00<br>$10.00<br>Public Privat e<br>$8.00<br>Piped Unit ized<br>$6.32<br>$6.00<br>$4.00<br>$3.60<br>Based on data from 47 countries<br>Average<br>and 93 locations based on the<br>literature review<br>St d. Deviat ion<br>$2.00<br>$0.79 $0.86 $0.76<br>$0.34 $0.45<br>$0.00 $0.02 $0.17<br>Public Utilities Private Netw orks Point-source Vendors Tanker Trucks Carters<br>Mobile Distributors<br><!-- End of picture text --> Chart 2 above indicates that price of water increases as economies of scale in production and sales decrease. Likewise, as unit size or volume of sales decrease, prices go up. Carters, which have the smallest sales volume, charge the highest unit price per m<sup>3</sup> of water— as much as $11.00/ m<sup>3</sup> or more. Thus, the ability of the service provider to amortize capital costs as well as operating inefficiencies (such as fuel, bribes and salaries) over larger volumes is the key to improving affordability.

On average, water from piped network operators (PNO) costs 1.5 times more than that from a formal utility, whereas, water from a point source (PS) costs, on average, up to 4.5 times the utility and, finally, water from the mobile distributors (MD) can costs up to 12 times more than the
specify our core identification strategy and discuss the interpretation of our preferred estimates. Finally, we discuss the potential consequences of sample attrition and discuss how we address these concerns.

## **3.1 Data**

We combine two detailed panel datasets. The first set of data is from the Nigeria General Household Survey (GHS), which is a product of the Nigeria Bureau of Statistics and the World Bank’s Living Standard Measurement Study (LSMS).<sup>23</sup> The GHS was conducted over four rounds in 2010–2011, 2012–2013, 2015–2016, and 2018–2019, and each survey round includes two data points for each household: one in the post-planting (rainy season) period and one in the post-harvest (dry season) period, regardless of whether the household engages in agricultural activities. Data collection in the post-planting period generally occurs in the later fall months, while fieldwork in the post-harvest season occurs within the first few months of the next year. Figure 2 illustrates the timing of GHS survey rounds and shows how these data collection periods correspond with the trend in herder-related violence in Nigeria.

The GHS is designed to include a nationally (and zonally) representative set of enumeration areas,

> 23These data (and all documentation) were accessed via the World Bank’s microdata catalog. Data were downloaded on November 8, 2019.

8
then select the model for which the test MSE is the smallest. To measure the accuracy of the model or the extent to which it gives the most similar results to the first dataset we would then calculate the bootstrap estimates, this time resampling from the second dataset.

# **5. Presentation of results from the statistical analysis with interpretation**

As noted in section 4, statistical learning can give information on how informative a model is when applied to a holdout sample. As this was an exploratory exercise, rather than presenting one methodology and set of results, the exercise begins with the q‐fold replication. One subset of the data is taken as the “training dataset” and then we assess the model fit in terms of the RMSE on the holdout sample or “testing dataset”. There are thus several different cuts of the data that have been applied to assess the information needed.

We began with the combined 2005/11 dataset, and conducted q‐fold (10 rep) cross‐validation on the shortlisted 4 models with a linear specification for the drought variable. To be specific, this meant that for the pooled dataset, we fit the model onto 9 tenths of the data, and assess the fit on the one remaining tenth. The exercise is repeated 10 times. As discussed in section 4, the models are assessed based on the average mean squared error (MSE) over all ten repetitions as defined in equation (5.1). The results are shown in table 6.1 below. There is little difference between the models, with model 4 having the lowest average MSE by a small margin.

24
. Moreover, with more women in the judiciary, the aggregate effects of in-group bias for women would be less severe.

As we show in the summary statistics section, below, there has been substantial progress towards gender parity in the judiciary–though there is still a long way to go to achieve equality. Inequalities in the judiciary are reflective of broader gender inequalities in Kenyan society. According to the 2020 United Nations Development Programme’s Gender Inequality Index, which scores countries based on gender gaps related to representation in government, educational attainment, and labor force participation, Kenya ranks 126th out of 189 countries. Notably, women hold only 23 percent of seats in parliament (UNDP 2020).

Inequalities are also widespread along ethnic lines, a fact which provided an impetus for the adoption of the new constitution and its measures to devolve power (Akech 2010). As we show below, some groups are underrepresented in the judiciary relative to their share of the population, whereas others are overrepresented. This finding showcases relative differences in access to justice and judicial representation across ethnicities.

# **3 Data**

## **3.1 Variable Construction**

The main data source used in our analysis is the Kenyan Judiciary’s publicly available database for court cases.<sup>8</sup> The database includes 159,645 cases, almost exclusively from the Superior Courts, over the period of 1976 to 2020. Kenya Law, an organization within the Kenya Judiciary, began uploading case information in 2006. They upload all cases that are sent to them from the individual courts. Judicial officers in Superior Courts have a mandate to send cases to Kenya Law. For cases prior to 2006, Kenya Law has made (and continues to make) efforts to gather and upload case information.

In order to build our dataset for analysis from this database, we scraped the metadata and full text decision associated with each case. In doing so, we were able to directly extract the following for most cases
). ILO Labour force estimates and projections: 1990‐2030 (methodological description).

ILO. (2018a). Conceptual framework for statistics on work relationship _s._ ILO. (2018b). Data collection guidelines for ICSE‐18.

ILO. (2018c). Decent work and the Sustainable Development Goals: a guidebook on SDG Labour Market Indicators. ILO.

ILO. (2018d). 19th ICLS implementation: National LFS practices and implementation plans. _Presentation prepared for the 20th ICLS._ ILO. (2018e). Survey methods to improve measurement of paid and unpaid work: Country practices in time‐use measurement.

ILO. (2018f). Women and men in the informal economy: A statistical picture.

Kalton, G., & Schuman, H. (1982). The effect of the question on survey responses: A review. _Journal of the Royal Statistical Society._ Series A (General), 42‐73.

Kan, M. Y., & Pudney, S. (2008). Measurement error in stylized and diary data on time use. _Sociological Methodology, 38_ (1), 101‐132.

Koolwal, G. (2019). Improving the measurement of rural women’s employment: global momentum and survey research priorities. _Working paper, World Bank_ .

Langsten, R., & Salen, R. (2008). Two approaches to measuring women's work in developing countries: A comparison of survey data from Egypt. _Population and Development Review, 34_ (2), 283‐305.

Lentz, E., Bezner Kerr, R., Patel, R., Dakishoni, L., & Lupafya, E. (2018). The invisible hand that rocks the cradle: On the limits of time use surveys. _Development and Change_ .

30
about the stability of certain estimated relationships over 23 years were deemed too strong. The consumption surveys which could be used to generate nationwide regional price indices, for instance, are so far apart (1975 and 1996), as to make sensible comparisons of regionally deflated data over the period we are concerned with in this paper hazardous. See, however, Ferreira et. al. (2003) for results when these adjustments are carried out for a single point in time.

> 8 This is the average nominal exchange rate for the survey reference month, September 2004.

> 9 These axioms are as follows: anonymity; the Pigou-Dalton transfer principle; scale invariance; population replication invariance; and decomposability (see Cowell, 1995).
# **Appendix A: Calculating Earnings**

Self-employment income was calculated in the weekly survey by summing the responses to the questions “How much of that money was for work you did over the past 7 days?” (the follow-up to the question “How much money have you received over the past 7 days?”) and “How much more do/did you expect to receive for the work you did over the past week? Please do not include money that you received over the past week.” Reported costs were then subtracted from these revenues, using the answer to the question “How much were the costs for only the past 7 days (if the costs are for a machine for example, only include the cost of using the machine for a week)”, which was the follow-up to the question “Did you have any costs for goods or equipment that were needed for your work over the past week?”.

Wage employment income was calculated by multiplying days of work by the respondent’s usual daily wage rate, as calculated from the baseline reports of their usual weekly pay “On average, how much do you earn (or, if in kind, what is the value of what you earn) in your primary job, in a week?” divided by their usual days of work, given by their response to the question “How many days do you work in your primary job during a normal week?” unless the respondent answered yes to the question “Since the last interview, did the way that you calculate the amount of work you get paid for change?” If so, they were asked for the earnings in this new job in a normal week, and this rate was multiplied by the number of days worked that week to the calculate earnings. Note that this calculation means that workers who are paid less frequently than on a weekly basis – 73 percent of wage workers – are assigned the fraction of their weekly pay that corresponds to the number
based on U.S. estimates, which are obtained from the 2018 Occupational Employment and Wage Statistics (OEWS) published by the U.S. Bureau of Labor Statistics.<sup>9</sup> The O*NET-SOC 2010 codes are linked to their corresponding four-digit ISCO-08 codes using the U.S. Bureau of Labor Statistics correspondence table between SOC 2010 Codes and ISCO-08 categories. This correspondence, as well as the employment estimates in the OEWS, is available at the six-digit level of the SOC. At the same time, the O*NET-SOC 2010 classification of green occupations is presented with an eightdigit level of detail. Since 12.7 percent of the 6-digit SOC occupations are associated with more than one 8-digit SOC, the total number of workers in a 6-digit SOC occupation was equally distributed among the 8-digit SOC codes associated with it.<sup>10</sup> As an example, Table 2 shows how the share of green occupations was constructed for a specific ISCO code, representing the probability that an individual with an occupation in this ISCO code has a green job.

**Table 2 – Example of the methodology used to estimate the probability for an ISCO-08 code to represent a green occupation.**

|ISCO-08<br>Code|O*NET-SOC 2010<br>Code|Green<br>Job?|2010 SOC<br>Code|Number of US<br>workers|“Simulated”<br>Number of US<br>workers|Share of Green<br>occupations|
|---|---|---|---|---|---|---|
|1234|17-2061.01|Yes|17-2061|10|5||
|1234|17-2061.02|No|17-2061||5|5%<br>|
|1234|17-2063.00|No|17-2063|40|40|=5/(5+5+