rafmacalaba/datause-encoder-data huggingface dataset:
# **2. Evaluation Questions and Strategy**

Despite the potential gains from including VFM analysis in impact evaluations, such integration seems to be the exception rather than the rule. To understand why, we used mixed methods to answer three evaluation questions:

1. How frequently is Value for Money analysis incorporated into published impact evaluations?

2. What are the existing incentives and barriers faced by producers and users, both as individuals and institutions, for VFM incorporation?

3. What are some options to overcome the challenges for the integration of Value for Money analysis into impact evaluations?

## **Approach**

We investigated these questions from the perspective of those who produce impact evaluation estimates and reports (IE producers), and those who use or who could use those estimates and reports in decision-making for international development policy work (IE consumers). This report defines IE producers as those in the production line of impact evaluations: They pay for, fund, carry out, or communicate evaluations and findings. IE consumers include those who use evaluation results to change behavior, make funding decisions, select programs, or legitimize predetermined behavior. IE consumers include impact evaluation beneficiaries as well as policymakers and decisionmakers.

### **Box 2.1. Defining Value for Money**

The definition of Value for Money is contested. Our working definition relates specifically to the use of VFM in impact evaluations. In practice, impact evaluations most often use efficiency analyses such as cost-benefit or cost-effectiveness analysis. We adapted a definition from that used by the Department for International Development (DFID) after an extensive review of VFM field methods. DFID defines VFM to include the four “E’s”: measurement of a program’s economy, efficiency, effectiveness, and equity, though in this paper most of the discussion centers around using the quantified attributable benefits identified in impact evaluations as the numerator to establish efficiency with costs as the denominator. The ratio of Impact to Cost yields an estimate of efficiency, recognizing that there are many ways to arrive at estimates of each of those broad constructs.

There are times, of course, when institutions may act as both producers and consumers of both impact evaluations and Cost analyses. For example, a large agency (such as USAID) or multilateral bank (such as the World Bank) could potentially use the evidence from evaluations of vaccine programs that it funded to determine the most effective way of distributing vaccines in resource constrained countries.

We worked through a simple framework of the process by which evaluative evidence is created, disseminated, and applied. That is, we solicited views from impact evaluation 4
# **1 Introduction**

Outliers – observations that “appear to deviate markedly from other members of the sample in which they occur” (Grubbs, 1969) – are omnipresent in real-world data sets, and are almost always a cause for concern. If the ‘abnormality’ of extreme values is the result of measurement error, rather than a genuine manifestation of the variability of the data, then including outliers in calculations can cause serious bias to many statistics of interest.

Given the stakes, it is no surprise that the task of dealing with outliers often takes up considerable time and effort on the part of the analyst (Mancini and Vecchi, 2022). Even the very foundation of any strategy to handle outliers – identifying which observations qualify as ‘extreme’ – is far from trivial. Underlying a number of popular outlier detection methods is a basic algorithm: the analyst defines some concept of distance from the bulk of the distribution, which then allows to flag as ‘extreme’ any observation for which this distance is larger than a certain threshold. This principle is as simple as it is elusive: in practice, it is declined in a great many different ways, and the criteria used are not always well defined. This poses a number of problems: ‘good practices’ fail to consolidate, and the steps taken to deal with outliers are often difficult to replicate, which is an issue for scientific rigor as well as for comparability of data analysis.

In this paper we present `outdetect` , a tool designed to help the analyst _i)_ identify extreme values in a univariate distribution based on a transparent procedure, and _ii)_ gauge their potential impact on selected statistics of interest.<sup>2</sup> While the focus is on distributional analysis ( _e.g._ inequality and poverty estimation – see Cowell and Flachaire, 2007, 2015), the use of `outdetect` naturally extends to any situation where the presence of ‘too large’ or ‘too small’ values in a distribution is an issue.

The algorithm employed by `outdetect` relies on the normalization of the target variable, and the imposition of cutoffs to define an outlier region. A popular version of this procedure is to log-transform the distribution, and flag observations that are more than two or three standard deviations from the mean ( _e.g._ , Deaton and Tarozzi, 2005). However, `outdetect` allows users to choose from a number of alternative (and more flexible) normalizing transformations, and to use a number of alternative and statistically robust measures of location and scale. The use of transformations other than the log allows the analyst to apply a consistent detection criterion to a wide range of variables, which is

> 2 The command `outdetect` can be installed in Stata by typing `ssc install outdetect.`

2
<u>Table 1 : Main Perceived Constraints to Reduce Poverty</u>

|**First action against poverty**<br> <br>|
|---|
|Create employment<br>45.51%|
|Roads construction<br>11.49%|
|Ease access to education<br>6.26%|
|_Source: Authors’calculation and ECAM II_|

Concerning dissatisfaction about roads, households quote remoteness (53.23%), difficulty of access (18.67%) and roads condition (10.98%) as the first ones, whatever the poverty status<sup>13</sup> .

The statistical analysis of our data confirms these subjective views. Among the 10 992 households surveyed 25.15% are poor. As it is in all developing countries, poverty in Cameroon proves to be a rural phenomenon since 34.7% of rural households are poor (against only 13.6% of the urban ones) and 75.6% of the poor live in rural areas. More precisely, poor households primarily live either in the rural Savannah or the High "Plateau" zones. Table 2 presents for each agro-ecological area the corresponding poverty rate, the average time and the average distance to reach the closest tarred road.

<u>Table 2 : Roads’ Access, Poverty and Activity across Agro-ecological Zones</u>

||||**Access to the close**|**st tarred road**|
|---|---|---|---|---|
|**Agro-ecological Zones**|**Poverty Rates**|**Activity Rates**|**Time (min)**|**Distance (km)**|
|Yaounde|7.9%|20.7%|4.16|0.57|
|Douala|8.6%|30.3%|5.39|0.67|
|Other Cities|17.8%|29.9%|8.6|1.96|
|Rural Forest|29.0%|35.0%|77.54|38.20|
|Rural High Plateau|33.6%|40.7%|53.77|14.54|
|Rural Savannah|40.6%|38.7%|68.88|29.22|
|Urban|13.6%|29.7%|6.89|1.36|
|Rural|34.7%|38.4%|65.32|25.92|
|_Source: Authors’calculation and ECAM II_|||||

Poverty and bad access to roads appear to be typically rural issues. Poverty rates in the rural areas range from 29% to 40.6% against only 13.6% for cities. The average time to reach the nearest tarred road varies from almost 7 minutes in urban areas to an hour for rural ones. The

> 13 A household is considered as poor if its per unit consumption expenditures per year is less than 232 547 CFAF (354 euros).

12
institutional change, it may be possible to obtain steady-state expressions for not just human and physical capital, but also institutions, and the interactions between these economic and political factors. Empirical opportunities include directly testing the role of governance in education with micro-level indicators of governance, such as teacher absenteeism rates or the pervasiveness of informal payments in schooling, using micro-level data on student performance.

# **References**

- Acemo ̆glu, K. Daron, Simon Johnson & James A. Robinson (2001). “The Colonial Origins of Comparative Development: An Empirical Investigation”. _American Economic Review_ 91(5) (December): 1369–1401

- Acemo ̆glu, K. Daron, Simon Johnson & James A. Robinson (2005). “Institutions as a Fundamental Cause of LongRun Economic Growth”. In Philippe M. Aghion & Steven N. Durlauf (editors), _Handbook of Economic Growth_ , volume 1, chapter 6, pp. 385–472. Amsterdam, The Netherlands: Elsevier

- Albouy, David Y. (2012). “The Colonial Origins of Comparative Development: An Empirical Investigation: Comment”. _American Economic Review_ 102(6) (October): 3059–3076

- Alesina, Alberto F., William R. Easterly, Arnaud Devleeschauwer, Sergio Kurlat & Romain T. Wacziarg (2003). “Fractionalization”. _Journal of Economic Growth_ 8(2) (June): 155–194

- Altinok, Nadir & Hatidje Murseli (2007). “International Database on Human Capital Quality”. _Economics Letters_ 96(2) (August): 237–244

- Angrist, Joshua D., Eric P. Bettinger, Erik Bloom, Elizabeth M. King & Michael R. Kremer (2002). “Vouchers for Private Schooling in Colombia: Evidence from a Randomized Natural Experiment”. _American Economic Review_ 92(5) (December): 1535–1558

- Arellano, Manuel & Stephen R. Bond (1991). “Some Tests of Specification for Panel Data: Monte Carlo Evidence and an Application to Employment Equations”. _Review of Economic Studies_ 58(2) (April): 277–97

- Arellano, Manuel & Olympia Bover (1995). “Another Look at the Instrumental Variable Estimation of ErrorComponents Models”. _Journal of Econometrics_ 68(1) (July): 29–51

- Barro, Robert J. (1991). “Economic Growth in a Cross Section of Countries”. _Quarterly Journal of Economics_ 106(2) (May): 407–443

- Barro, Robert J. (1996). “Democracy and Growth”. _Journal of Economic Growth_ 1(1) (March): 1–27

- Barro, Robert J. & Jong-Wha Lee (2001). “International Data on Educational Attainment: Updates and Implications”. _Oxford Economic Papers_ 53(3) (July): 541–563

- Behrman, Jere R. & Nancy Birdsall (1983). “The Quality of Schooling: Quantity Alone is Misleading”. _American Economic Review_ 73(5) (December): 928–946

- Benhabib, Jess & Mark M. Spiegel (1994). “The Role of Human Capital in Economic Development: Evidence from Aggregate Cross-Country Data”. _Journal of Monetary Economics_ 34(2) (October): 143–173

26
up prices by 5.8 percent and as a result consumption falls by 2.3 percent. Domestic production falls by 2 percent, primarily from the least productive areas, including the Upland and IVS ecologies. The higher price for domestic rice also makes imports more attractive, notwithstanding the 1 percent tariff and therefore imports increase by 5.1 percent and consequently the fiscal revenues from the tariff also increase by an equivalent percentage.

**Table 13: Results Wage Simulations**

|**Minimum Wage**<br>**Increase(%)**|**Baseline**|**0**|**10**|**10**|**10**|**10**|
|---|---|---|---|---|---|---|
|**Tariff Level**|0%|1%|1%|5%|10%|15%|
|Key policyindicators|||||||
|**11. CPS (US$ Mn)**|1,401.6|1,192.5|1,163.5|1,160.3|1,156.6|1,154.9|
|**12. Employment (No.)**|142,280|893,140|866,840|869,300|872,370|934,300|
|**13. Consumption (Mt)**|1,002,010.2|909,536.1|888,723.6|887,558.2|886,101.4|885,377.9|
|**14. Production (Mt)**|524,520|1,224,120|1,199,740|1,202,080.0|1,205,000.0|1,269,409.1|
|**15. Imports (Mt)**|861,521.5|115,736.8|121,674.2|115,847.2|108,563.4|-|
|**16. Exports (Mt)**|384,030|430,330|432,700|430,370|427,450|384,030|
|**17. Trade balance**<br>**(Mn$)**|(413.50)|96.70|93.70|93.59|93.88|155.15|
|**18. Fiscal Revenues ($)**|-|154,300|162,220|802,820|1,576,300|-|
|**19. Price ($/Mt)**|624.11|856.69|906.60|861.80|914.91|916.90|

Source: Model Simulation Increasing the minimum wage by 10 percent under a 10 percent tariff regime, would result in an overall 9 percent reduction in total employment in the rice sector. This is comprised of a 3.8 percent reduction in the use of household labor and a much larger, 15.8 percent reduction in the demand for temporary labor. This suggest that employment, and particularly temporary employment in the rice sector is extremely sensitive to the wage rate. As Table 13 shows, it would require a tariff level above 10 percent to fully mitigate the adverse employment effects of the 10 percent increase in the minimum wage.

In summary, the results of the simulation of the increase in the minimum wage suggest that employment in the rice sector is highly sensitive to the wage rate. Furthermore, hired-in labor (who are generally from the poorest groups) are likely to be most adversely affected by the unemployment that would likely result from the increase in the wage rate as suggested by the simulations.

# **Technical Experiments**

A central pillar of the government’s agricultural strategy is to increase productivity in the rice sector, by increasing yields through better agricultural practices and the use of improved seeds and increased use of fertilizer. Indeed, the government has taken steps, with the support of the World Bank to liberalize the seed and fertilizer markets to ensure better access to these inputs at more competitive prices.

A report of farmers' field trials with rice production in 1979 suggests that yields could be substantially increased with improved practices. Table 14 below shows the orders of magnitude of such increases.

24
likely that consumers pay a mark up over the producer price of rice (given the need to transport and market the locally produced rice), but it is also likely that some of the rice produced in Mali is exported to neighboring countries. Therefore, one can assume that about 40 percent to 50 percent of the rice consumed in the country is locally produced, which is the common perception in the country, and is also what one finds in the SAM data. The household survey suggests that rice is consumed as frequently in rural than in urban areas, although rural consumption of rice relies more on auto-consumption. However, the data also suggests that rice is consumed more intensively by urban and comparatively richer households than by the rural poor. In the top quintile of consumption per capita, rice consumption per household is almost five times higher than in the bottom quintile.

It is important to mention that beyond rice, other cereals also matter, but it is the price of rice that is likely to have the largest effect on households. The 2006 ELIM survey data suggest that millet and sorghum consumption is at about the same level on average as rice consumption, while the consumption of corn and wheat and bread is five times smaller. As for rice, more than nine out of ten households consume millet and sorghum, but the key difference is that most of the consumption of millet and sorghum is locally produced, and a very high share of that production is auto-consumed. The same is true for corn (by contrast, wheat and bread tend to be imported, but these goods are much smaller in terms of their share of total consumption). Thus, because millet and sorghum are essentially auto-consumed, changes in prices for these cereals are likely to have a much lower impact on the overall economy and household poverty than changes in rice prices.

In addition to the 2004 SAM, additional data used to calibrate the static model include behavioral elasticities for production, demand, and trade. Demand elasticities have been 12
Table 1 describes each relationship linearly. The three columns correspond to equations 1-3 outlined in Section 2. Column 1 is given by _K_ = _μ_ 3 1 + _E_ 3, column 2 by _C_ = _η_ 2+ _μ_ 2 _K_ + _E_ 2, and column 3 by _D_ = _η_ 1 + _μ_ 1 _C_ + _E_ 1. Column 1 describes the know gap, which is explained only by a constant because the target level of performance is unchanging. Columns 2 and 3 describe the know-can and can-do relationships, which are graphically represented by Graph 1 and Graph 4 in Figure 6.

Table 1: The know, know-can, and can-do gaps

||(1)<br>Know|(2)<br>Can|(3)<br>Do|
|---|---|---|---|
|Target|0<br>(.)|||
|Know||0.980<sup>_∗∗∗_</sup><br>(0.0300)||
|Can|||0.157<sup>_∗_</sup><br>(0.0850)|
|Constant|0.451<sup>_∗∗∗_</sup>|-0.0156|0.270<sup>_∗∗∗_</sup>|
||(0.0336)|(0.0157)|(0.0407)|
|Observations|831|831|831|
|Adjusted _R_<br>2|0.000|0.937|0.023|

Clustered standard errors in parentheses. _‡_ Target is a constant, so the regression produces only a constant: there cannot be a systematic relationship between knowledge and the target. _∗ p <_ 0 _._ 1,<sup>_∗∗_</sup> _p <_ 0 _._ 05,<sup>_∗∗∗_</sup> _p <_ 0 _._ 01 Ideally, these coefficients would estimate the variance in performance that resulted from exogenous variation in motivation or capacity from, for example, a randomized control trial3 This study, as a baseline, includes only cross-sectional variation, so we cannot make causal

> in hospitals work in teams so a clinician with low levels of knowledge might perform better when part of a team that includes more competent providers. Indeed, when we reanalyze the data focusing only on cases in which health workers worked as individuals, this area of the figure disappears. We retain this particular specification in the analysis because teams matter in this context.

> 3Variation in performance (not volume of services delivered) driven by  exogenous variation in capacity and motivation is seen in Basinga et al. (2011); Brock et al. (2016, 2018) and there are many studies of the link between performance and exogenous variation in training exposure (Jamtvedt et al., 2013).

17
# **I. Introduction**

In a single decade, from the mid 1990 to 2000, the share of the Brazilian metropolitan area work force unprotected by labor legislation and thereby classified as “informal” rose an astronomical 10 percentage points. This episode is of relevance for several reasons.

First, such movements have been relatively common over the last decade: Urban informality increased in Argentina from 1992 to 2003 by 10 percentage points; in Venezuela from 1995 to 2003 by 8 percentage points.<sup>1</sup> To the degree that such increases represent the progressive exposure of the work force to risk and loss of other benefits, they are intrinsically worrying.

Second, understanding the causes of these movements can contribute to our understanding of the drivers of informality more generally. Brazil offers several dramatic policy changes across the period that theory suggests could affect gross labor flows and their volatility and hence the steady state size of the formal sector: a far reaching trade reform, and the establishment of a new Constitution in 1988 that had substantial impacts on labor costs and flexibility.

Third, unlike Argentina or Venezuela, Brazil offers an excellent panel data set that, with perhaps the exception of Mexico, is one of the very few in the developing world to have a sufficient time dimension for us to study the shifts in magnitudes of gross labor flows associated with two complete business cycles and the secular recomposition of the labor force.

This paper first applies recent advances in the study of labor market dynamics over the business cycle, introduced by Shimer (2005b) and Hall (2005) and confirms for Brazil the patterns identified for the US and for Mexico by Bosch and Maloney (2006):

> 1 See Gasparini and Tornarolli (2006)

2
if we wished to ensure quality in the fieldwork. A simplified, and more feasible, design was drawn up that still allows us to make basic comparisons. These comparisons focus on two issues. The first is the quality of household usage information provided by informants versus a full enumeration. The second issue is data quality obtained using a product-based questionnaire versus an institution-based one.

Physically, three different questionnaires were fielded with the second and third questionnaires containing more than one treatment (see Table 1). Once the sample design was complete, households were randomly assigned to one of three groups. Each of these groups of households were administered a different questionnaire. This selection was made by GSS staff in headquarters. In households where one of the treatments was for a randomly selected adult to be interviewed, interviewers used Kish tables to make that selection in the field. Only individuals aged 15 or older were included in the full enumeration or were selected as random respondents.

The GLSS5 households included in the FSS were taken from the two interviewing cycles of the GLSS5 closest in time to the fielding of the experiment. This was done to minimize the chances that a household might have changed significantly between the time of the GLSS5 and the FSS: we rely on the GLSS5 data for non-financial household and individual information. The selected enumeration areas (and households) were distributed throughout the country (see Table 2).

The questionnaires were developed in close collaboration with the GSS. In addition to their own experiences in other surveys, the GSS called upon various experts and sources in the country to determine the comprehensive list of financial services that existed and the range of service providers. The GSS staff also worked to determine the 13
Table 2: Design of the insurance products

||Bangladesh|Burkina Faso|Gha|na|India|Kenya|Mali|
|---|---|---|---|---|---|---|---|
||||Year 1|Year 2||||
||(1)|(2)|(3)|(4)|(5)|(6)|(7)|
|Type of insurance|Hybrid|Area-yield|Rainfall|Ibid.|Rainfall|Rainfall<sup>a</sup>|Area-yield|
|Targeted crop|Not<br>crop-specifc<sup>b</sup>|Cotton|Maize|Ibid.|Not<br>crop-specifc|Four crops|Cotton|
|Sale price (in PPP USD)<sup>c</sup>|0.3 to 3.3 USD|12.6 to 50.5 USD|Free|1.1 to 15.3<br>USD|Free|Free<br>(conditional on<br>purchasing<br>improved<br>seeds)|15.8 to 31.6 USD|
|Actuarially fair price (in PPP<br>USD)<sup>c,d</sup>|3.6 USD|28.9 USD|38.8 USD|7.9 to 10.3<br>USD|18.0 USD|5.2 to 13.8 USD|31.6 USD|
|Purchase decision level|Individual|Cluster|Individual|Ibid.|Individual|Individual|Cluster|
|Unit|Stand-alone<br>policy<sup>b</sup>|1 ha|0.4 ha|Ibid.|Stand-alone<br>policy|0.4 ha|1 ha|
|Timing of payment|Up-front|After harvest|NA|Up-front|NA|Up-front|After harvest|
|Take-up<sup>e</sup>|87%|45%|100%|63%|100%|59%|29%|
|Payoff triggers|Dry spell<br>duration, low<br>average of area<br>yields|Low average of<br>farmer group<br>yields|Number of<br>monthly<br>dry/wet days|Ibid.|Cumulative<br>rainfall|Rainfall<br>excess/defcit|Low average of<br>cooperative<br>yields|
|Source of data|Weather station,<br>crop-cutting<br>exercise|Purchasing<br>company|Weather<br>station|Ibid.|Weather station|Weather station|Purchasing<br>company|
|Insurer|NGO|NGO|NGO|Ibid.|Insurance<br>company|Insurance<br>company|Insurance<br>company|

Source: Authors’ elaboration.

> a The insurance product also includes an indemnity component covering against other risks such as hail, frost, fire, windstorm, and uncontrollable pests and diseases. Indemnities are released after crop stand checks conducted by field inspectors.

> b While not explicitly tied to a particular crop, each policy was meant to cover revenue from 0.1 acre (0.04 ha) of land cultivated under transplanted aman rice. Households could purchase multiple units of insurance based on the amount of land they cultivate during the monsoon season. According to Hill et al. (2019), this should reduce incentives to view the insurance as a gamble.

> c Indexed to 2015 dollars. _Source:_ World Bank.

> d In some studies, the actuarially fair price varies depending on the community (Karlan et al., 2014) or crop (Bulte et al., 2019). For example, in Bulte et al.

(2019), the price per unit insured is 5.2 USD for sunflower, 10.9 USD for soya bean, 11.1 USD for sorghum, and 13.8 USD for maize.

> e Take-up is defined as the share of treated households that subscribe to at least one unit of insurance.

34