metadata: authors: 0: author: Vivalt,Eva
1: author: Coville,Aidan
2: author: KC,Sampada
count: World
docna: 0: docna: Local Knowledge, Formal Evidence, and Policy Decisions
docty: Policy Research Working Paper
abstracts: cdata!: How do policymakers value advice from
            local experts versus formal evidence from impact evaluations
            when making policy decisions? Using a discrete choice
            experiment conducted in collaboration with the World Bank
            and Inter-American Development Bank, we show that
            policymakers were willing to accept a program that had a 5.0
            percentage point smaller estimated effect on enrollment
            rates if it were recommended by a local expert. They also
            preferred programs supported by evidence from a different
            region over programs supported by local evaluations only if
            the former had a 5.8 percentage point higher estimated
            impact. These premiums are large, surpassing the effects of
            many programs aimed at improving enrollment rates. This
            highlights the substantial weight that policymakers place on
            local evidence.
display_title: Local Knowledge, Formal Evidence,
            and Policy Decisions
listing_relative_url: /research/2024/12/34433852/local-knowledge-formal-evidence-policy-decisions
topicv3: Strategic Policy Planning m1326752 1927,Public Sector Strategic Policy Planning,
            Monitoring and Evaluation m1503535 2907
publishtoextweb_dt: 2024-12-09T00:00:00Z
dois: 10.1596/1813-9450-10994
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
no_of_pages: 28
author_keys: 0: author_key: 1053635
1: author_key: 1063728
2: author_key: NA
model_extractions: Policy Research Working Paper 10994 Local Knowledge, Formal Evidence, and Policy Decisions _Eva Vivalt Aidan Coville Sampada KC_ Development Economics Development Impact Evaluation Group December 2024
Policy Research Working Paper 10994

# **Abstract**

How do policymakers value advice from local experts versus formal evidence from impact evaluations when making policy decisions? Using a discrete choice experiment conducted in collaboration with the World Bank and Inter-American Development Bank, we show that policymakers were willing to accept a program that had a 5.0 percentage point smaller estimated effect on enrollment rates if it were recommended by a local expert. They also preferred programs supported by evidence from a different region over programs supported by local evaluations only if the former had a 5.8 percentage point higher estimated impact. These premiums are large, surpassing the effects of many programs aimed at improving enrollment rates. This highlights the substantial weight that policymakers place on local evidence.

This paper is a product of the Development Impact Evaluation Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at acoville@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
Local Knowledge, Formal Evidence, and Policy Decisions Eva Vivalt<sup>∗</sup> Aidan Coville<sup>†</sup> Sampada KC<sup>‡</sup> _JEL Codes:_ D04, O21, O22.

_Keywords:_ Evidence-based policy; discrete choice experiment; external validity.

> ∗Department of Economics, University of Toronto, eva.vivalt@utoronto.ca

> †Development Impact (DIME), World Bank, acoville@worldbank.org

> ‡University of British Columbia, samkc.student@ubc.ca. We thank Arianna Legovini for support and helpful feedback and Oscar Mitnik, Sebastian Martinez, and Silvia Velez Caroco for enabling the surveys to be run. Antonio Ortiz-Mena and Young Ju Nam provided excellent research assistance. Computational reproducibility for the results presented in this paper was verified by DIME Analytics. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.
# **1 Introduction**

The significant increase in impact evaluations in recent years means that policymakers have more formal evidence than ever before to inform decisions on which policies to pursue. At the same time, policymakers often seem to rely on the recommendations of local experts and there is significant disagreement regarding how well local experts tend to forecast the effects of different programs (e.g., Bessone et al., 2021; Milkman et al., 2022; Iacovone et al., 2023). We consider how much policymakers weigh the recommendations of local experts relative to how much they weigh impact evaluation results and whether there are any features of impact evaluations that make them more likely to influence policymaker decisions, using a discrete choice experiment. We find that policymakers place significant weight on advice from local experts, but they also value evidence from impact evaluations in their country. This highlights the importance of generating evidence that is perceived as contextually relevant if impact evaluations are to inform policy decisions.

We conduct the discrete choice experiment at World Bank (WB) and InterAmerican Development Bank (IDB) impact evaluation workshops. The WB workshops are meant to serve as “matchmaking” events that connect government officials, who are interested in using evidence from impact evaluation to inform their projects, with researchers supporting the design of such evaluations. The IDB workshops are intended to provide training to improve policymaker and policy practitioners’ awareness and understanding of impact evaluations. Participants at both types of workshops are demonstrably interested in impact evaluation, while also being at the frontline of program and policy decisions, making them a particularly relevant target group for this study. We refer to this group of workshop participants, excepting researchers, as “policy professionals” and break this down further into government ( _policymakers_ ) and international organization staff ( _policy practitioners_ ). Individuals are identified and invited to the workshops based on the projects they are working on and their role within their respective organizations. The aim is to ensure the participation of teams with decision-making authority, program knowledge and day-to-day follow up capacity. Specifically, each project represented at the workshop includes five project team members on average: a high-level policymaker for decision power, a project lead from the government and an international organization project lead to ensure requisite knowledge about each project is available, and junior analysts or monitoring and 2
evaluation (M&E) officers to support follow-up activities. This team is paired with researchers for a week with the express objective of developing evaluations to inform project designs. This is a relevant sample because the typical WB/IDB-supported project is developed through the interactions of people in these types of roles.<sup>1</sup> A more detailed set of profiles for each type of policy professional and their role in project and policy decision-making can be found in the appendix.

We surveyed 190 policy professionals, providing them with descriptions of programs that are each associated with an impact evaluation result and asking them to choose between them. The impact evaluations attached to the programs differ by their identification strategy; location; impact; and the precision of the estimate. We compare how participants weigh these impact evaluation results relative to advice from a local expert.

We find that policymakers place a relatively high weight on contextual factors, such as whether a local expert recommended the program and whether the program was evaluated by an impact evaluation in their country. This may be due to concerns with how much rigorous evidence from one setting may be transportable to another (Pritchett and Sandefur, 2015; Vivalt, 2020), an important consideration when papers are still fairly geographically clustered (Leight, 2022). Policy practitioners also care about contextual factors but additionally pay attention to the internal validity of evidence from impact evaluations, preferring programs evaluated by RCTs and programs with results that were precisely estimated.

Our study builds on existing work exploring the decision-making processes of policymakers. Rogger and Somani (2023) find that bureaucrats in Ethiopia have beliefs about their constituencies that vary significantly from official statistics and that pro-

> 1Specifically, governments typically have priority targets or policy areas (such as increasing school enrollment) and will request donor support to finance activities that support achieving these outcomes. Initial project designs will typically resemble existing / past activities, and teams may draw on both tacit information and existing evidence from impact evaluation studies and other research (if it exists) on how effective these activities have been and make adjustments based on an iterative process, incorporating different inputs that people/organizations preparing the project bring to the discussion. This culminates in a draft “project document” that outlines the actual components that will be implemented to achieve the higher-level outcomes, which is then reviewed by senior management and adjusted to develop a final project plan for sign off. Throughout this process, decisions about which components will form part of the project and the nature of these project components are influenced by information available to the project preparation team who need to decide how to use this information to adapt the project plan. The sample included in this study was purposely selected based on their role in this particular process that guides how the pipeline of WB/IDB-supported projects are developed.

3
viding evidence briefings can help reduce this gap. Nellis et al. (2019) consider how policy practitioners weigh meta-analysis results compared to results from individual studies. Banuri et al. (2019) highlight the prevalence of behavioral biases among policymakers while Toma and Bell (2023) examine the impact of decision aids on policy choice. Hjort et al. (2019) consider Brazilian mayors’ willingness-to-pay for information from impact evaluations. Finally, in a companion paper, we consider which attributes policymakers, policy practitioners and researchers find attractive when searching for evidence (Vivalt et al., 2023).

In contrast to this past literature, we ask policy professionals to select _programs_ rather than _studies_ . Seeking out new evidence and weighing evidence when selecting a program are related but conceptually distinct. For example, policy professionals could seek information from an impact evaluation that found that a program had a negligible effect in order to investigate the reasons for the lack of impact. However, all else equal, they would not want to choose a program associated with negligible effects when weighing which program to select. While our experiment considers hypothetical choices, our focus on programs rather than studies is perhaps closer to the main item of interest, namely, how policy professionals make policy decisions. In making these decisions, policy professionals may still place some weight on study attributes, but they are not the only things they consider.

Our short paper is also the first to consider how policymakers weigh expert advice relative to impact evaluation results. This is relevant to the practical concerns of many working in development, as expert knowledge is a resource commonly used by policymakers (Morgan, 2014). More broadly, this paper relates to the concept of “tacit” vs. “explicit” knowledge, a distinction often made in the field of knowledge management. Tacit knowledge can be defined as knowledge held by an individual that is hard to formalize. It may be based on personal experiences and intuitions (Polanyi, 2009). In our experiment, impact evaluation results represent explicit knowledge - these results are formalized and can be communicated and understood by others. However, advice from a local expert could include elements of both tacit and explicit knowledge. For example, a local expert could draw on a number of formal sources of evidence in coming to their conclusions, and how they interpret past findings and integrate this information with their broader understanding and experience to form a recommendation involves their tacit knowledge. While tacit knowledge has been shown to be important in decision-making in a number of diverse fields (e.g., 4
Podg ́orski, 2010; Hanna et al., 2014; Meisch et al., 2022), it is relatively understudied in economics, and this is the first paper to examine how policymakers weigh this type of evidence.

Finally, to quantify these trade offs, we consider how much participants would be willing to give up in terms of estimated impact in exchange for a program being supported by a certain kind of information. Program impacts provide a natural unit of analysis for assessing trade-offs because they are analogous to a public budget: they are a real cost borne by the public that depends on the choices of the policymaker. Our experimental design allows us to say, for example, that policymakers would accept a conditional cash transfer program that was not recommended by a local expert, over one that was, only if such a program had at least a 5.0 percentage point higher estimated impact on enrollment rates; further, they would prefer a program evaluated in a different region over one evaluated in their country only if the program evaluated in a different region had at least a 5.8 percentage point higher estimated impact. These estimated impacts are very large compared to the typical effects of popular programs that improve enrollment rates.<sup>2</sup> Our results highlight the importance of local knowledge for policy decisions and suggest that researchers looking to maximize their impact leverage appropriate settings and communicate their research findings to those local experts from whom policymakers may seek advice.

The rest of the paper proceeds as follows. First, we discuss the data used in the experiment. Then we describe and present results from the discrete choice experiment. Finally, we discuss the implications of our results.

# **2 Data**

We surveyed 190 policy professionals at several World Bank and Inter-American Development Bank workshops, listed in Table 1. The context for this paper is that governments typically have priorities they wish to target and will request donor support to implement activities that support those outcomes. Teams working to develop

> 2For example, Vivalt (2020) finds that conditional cash transfers improve enrollment rates by 5.3 percentage points, and unconditional cash transfers by 3.5 percentage points. A more recent metaanalysis of unconditional cash transfer programs found an improvement of 4.1 percentage points (Crosta et al., 2024). This suggests that, in this experiment, policymakers would be willing to trade away much if not all of the estimated impact.

5
a project proposal may draw on a variety of sources of information, including expert advice and evidence from impact evaluations, if available. Throughout this process, decisions about project components are influenced by this information, but relatively little is known about how these factors are weighed by the different types of actors that engage in this policy process. The sample of workshop participants we focus on contains many of the types of actors that could plausibly use impact evaluations to inform policy decisions. Of the 190 policy professionals attending these workshops, we obtained responses from 156, representing a very high response rate of 82%. Eightyone policymakers and 75 policy practitioners participated in the survey. The following subsections describe each sample in more detail.

# **2.1 World Bank Sample**

We surveyed attendees at WB workshops organized in Athens (September 2019), Marrakesh (December 2019), Bangkok (July 2023), and Dar es Salaam (August 2023). Workshop attendees consisted of policymakers and policy practitioners.<sup>3</sup> Each workshop was approximately one week long and was meant to facilitate connections between government staff and researchers. Policymakers and policy practitioners were matched with researchers and worked together over the course of the workshop to design a prospective impact evaluation that could be used for their program.

Workshops were attended by participants from around the world and we observed high response rates across World Bank workshops (91%). We expect this is in part because of the data collection approach within the workshop format: surveys were conducted as part of the program agenda. The designated time slot was early in the workshops to mitigate the possibility of experimenter demand effects.

3 Participants are classified into policymakers and policy practitioners based on their survey responses. Specifically, participants self-identify as government employees, publicly elected or appointed officials, international development organization operational staff, NGO employees, private sector employees, researchers, or other. Government employees and publicly elected or appointed officials are classified as “policymakers”. International development organization operational staff as well as a few NGO and private sector partners collaborating on projects are classified as “policy professionals”. Too few researchers participated in these particular workshops to serve as a useful comparison group. 5 individuals who had selected “other” were classified into one of the substantive categories using the same decision rule as in Vivalt et al. (2023), and 6 others were re-classified based on an _ex post_ review of the data that identified a mismatch between self-reported employment and actual employment. Our findings do not substantively change if we use the original self-classification (results available upon request).

6
Table 1: Response Rate at Workshops

|Institution|Location|Year|Eligible<br>Attendees|Surveyed|Response<br>Rate|
|---|---|---|---|---|---|
|IDB|Washington, D.C.|2018|49|18 (18)|0.37 (0.37)|
|World Bank|Athens, Greece|2019|39|38|0.97|
|World Bank|Marrakesh, Morocco|2019|41|33|0.80|
|World Bank|Bangkok, Thailand|2023|30|27|0.90|
|World Bank|Dar es Salaam, Tanzania|2023|31|30|0.97|
|Total|||190|156|0.82|

The IDB rows include responses from the “pre” period and, in parentheses, the “post” period. The total is calculated using the total number of unique respondents across both rounds of the IDB survey (28). We exclude researchers from both the “eligible attendees” and “surveyed” counts, as too few researchers attended these workshops to be considered. Otherwise, all workshop participants are considered to have been eligible. Overall, response rates were very high.

## **2.2 IDB Sample**

Participants were also recruited from a workshop organized in May 2018 at the IDB headquarters in Washington, DC. Like the WB workshops, this workshop ran for approximately one week, was attended by policymakers and policy practitioners, and focused on impact evaluation methods. This workshop, however, did not include matching participants with researchers to design an impact evaluation as the World Bank workshops did.

Participants were emailed the survey link by the workshop organizers before the start of the workshop and again after the workshop (for a second, identical survey). We focus on the survey responses collected before the start of the workshop, as these may be more representative of the typical preferences held by policymakers and policy practitioners.<sup>4</sup>

# **3 Method**

A brief introduction informed participants that they would be asked questions that referred to conditional cash transfer programs. We also defined terms that some

> 4We do not have a sufficiently large sample to make before/after comparisons.

7
participants may not have been familiar with (“experimental”, “quasi-experimental” and “observational” - see Figure A1).

Participants were then asked to repeatedly choose which of two programs they would prefer, being asked to make this decision for a randomly-selected set (or block) of six questions.<sup>5</sup> 6 The programs were identified only as _Program A_ or _Program B_ , and each was associated with a study. These studies varied by method (experimental, quasi-experimental or observational); location (same country, different country in the same region, different country in a different region); the effect the study found (an increase in enrollment rates by 0, 5, or 10 percentage points); the precision of the estimate (a confidence interval of +/- 1 percentage point or +/- 10 percentage points); and whether a local expert recommended it. These attributes are summarized in Table 2.<sup>7</sup> Respondents saw one randomly-selected set (or block) of six questions each at the World Bank workshops in Athens, Marrakesh, Bangkok and Dar es Salaam, and two randomly-selected sets of six questions at the IDB. We used a fractional factorial design to select the levels of the attributes that were part of each choice option within each question and block to optimize power.

Our main results are analyzed using conditional logistic regression, clustering at the individual level, though we will also present results estimated using a mixed logit. Specifically, we estimate:

> 5E.g. “Now imagine that you need to provide a recommendation to a counterpart agency in your country on which of two programs to implement. A study was done on each program, with the results below. Please select which program you would recommend.” Appendix Figure A2 provides the introduction shown before the choice sets, and Figure A3 shows an example of a choice scenario participants might have faced. Participants at the IDB instead were asked to make this decision for two randomly-selected sets of six questions.

> 6By requiring respondents to make an active choice between each program, we reduce the possibility of acquiescence bias where respondents simply agree with research statement rather than provide a reflection of their own position

> 7We focus throughout on the effects of conditional cash transfers on children’s enrollment rates in school as that is a common outcome of interest among many low and middle-income countries and there have been extensive studies on these programs in many settings, increasing the chance that policy practitioners would be familiar with them. Of our sample, 54% was from lower-income or lower-middle income countries where enrollment rates are a common target of development interventions. In addition, 34 percent of the sample was from Latin America, where net enrollment rates for secondary school continue to be a challenge in the region despite many countries being classified in the upper-middle-income category. As an illustration, the national average for the net secondary enrollment rate in 2018 was 91 per cent in Argentina and 78 percent in Colombia (World Bank, 2018).

8
Table 2: Attributes and Levels in the Discrete Choice Experiment

|Attributes|Levels|
|---|---|
|Method|Experimental, Quasi-experimental, Observational|
|Location|Same country, Diferent country in the same region<br>Diferent region|
|Impact|0, `5, `10 percentage points|
|Confdence Interval|`/ ́1, `/ ́10 percentage points|
|Recommended|Yes, No|

This table shows the different attributes used in the discrete choice experiment and their levels.

where _U_ captures the utility of individual _i_ selecting alternative _j_ , _Impact_ is the estimated impact of the alternative, _Confidence Interval_ is a binary variable capturing whether the alternative had a small confidence interval, _Recommended_ is a binary variable reflecting whether or not that alternative was recommended by a local expert, and indicators for the method of the study (“experimental” or “quasi-experimental”) and location of the study (“same country” or “different country in the same region”) are included in _Method_ and _Location_ , respectively.<sup>89</sup> The questions asked were hypothetical. We expect that this might tend to bias our results towards zero, since individuals would have less motivation to consider each question carefully. Since the study aims to elicit participants’ unbiased beliefs and there are no clearly “correct” answers, we did not provide incentives that depended on participant responses. No incentives were provided for completing the survey, and the surveys were anonymous to reduce the potential for experimenter demand effects.

> 8 Standard errors are clustered at the individual level as this was the level of the randomization.

> 9While each of the study characteristics are orthogonal by design, it is possible that certain characteristics have unobserved connotations that are not captured - for example if knowing a study is from one’s same country, this might infer some connotation about quality as well, beyond the characteristics defined in the discrete choice experiment.

9
# **4 Results**

Table 3 presents results from the discrete choice experiment estimated using conditional logistic regression. Policymakers preferred programs with larger estimated treatment effects or programs that came recommended by a local expert, as well as those that had an impact evaluation from their country (Column 2). Policy practitioners preferred programs with larger, more precisely estimated impact evaluation results as well as results from the same country as the target program and results from RCTs (Column 3). They also preferred programs that came recommended by a local expert.<sup>10</sup> Table 4 translates these results into estimates of how much policymakers and policy practitioners would be willing to pay, in terms of estimated impact, for programs with these different attributes. Policymakers would prefer a program recommended by a local expert even if it had an approximately 5.0 percentage point lower estimated impact on enrollment rates (Column 2), which is very large relative to the range of effects that programs typically have on enrollment rates.

Table A1 further breaks down responses across the World Bank and IDB preworkshop samples and tests whether policymakers and policy practitioners put statistically significantly different weights on different attributes. Results appear largely comparable across these sub-samples, though policymakers put relatively less weight on large or precise estimates and do not emphasize programs with impact evaluations done in the same country as much as policy practitioners do.<sup>11</sup> To further explore the importance placed on precise estimates, we construct an indicator for whether the result shown was significant. While this variable was not explicitly shown to participants, it could be discerned from the provided estimated impact and confidence interval. When we include this variable in the regressions, the

> 10Note that in order to avoid framing a program as _not_ recommended, whether the program was recommended was displayed outside the table that contained study attributes; this also effectively clustered the study attributes together and kept the non-study attribute separate, to make the distinction between study-related information and non-study-related information easier for participants to parse. While we cannot rule out that something about the format affects the weight policy professionals place on advice from a local expert, we do not expect it to drive the observed preference for local evidence overall given that policymakers also place so much emphasis on impact evaluation results coming from their country, and location was displayed within the table of impact evaluation characteristics.

> 11This could partially be explained by their generally putting less weight on impact evaluation results compared to the recommendation of a local expert, although this is speculative.

10
Table 3: Weighing Different Kinds of Evidence

||Pooled<br>(1)|Policymaker<br>(2)|Policy<br>Practitioner<br>(3)|
|---|---|---|---|
|Impact|1.07***|1.05***|1.10***|
||(0.01)|(0.02)|(0.02)|
|Quasi-Experimental|1.11|0.97|1.31|
||(0.13)|(0.15)|(0.22)|
|Experimental|1.40***|1.30*|1.55**|
||(0.16)|(0.19)|(0.29)|
|Diferent country, same region|1.15|1.05|1.31*|
||(0.12)|(0.15)|(0.20)|
|Same country|1.59***|1.33**|2.02***|
||(0.15)|(0.17)|(0.26)|
|Recommended by local expert|1.24***|1.28**|1.21*|
||(0.10)|(0.15)|(0.12)|
|Small C.I.|1.29***|1.06|1.61***|
||(0.11)|(0.12)|(0.20)|
|Observations|818|407|411|

This table reports the results of conditional logit regressions on which program was selected. Odds ratios are reported. “Impact” refers to the estimate of the effect associated with the program; “Quasi-Experimental” indicates whether the study associated with the program was quasi-experimental; “Experimental” indicates whether the study associated with the program was an RCT; “Different country, same region” indicates whether the study associated with the program was described as done in a different country in the same region; “Same country” indicates whether the study associated with the program was described as done in the same country; “Recommended by local expert” indicates whether the program was recommended by a local expert; “Small C.I.” refers to the estimates having small confidence intervals. The omitted categories are “Observational”, “Different region”, and “Large C.I.”. The number of observations represents the total number of choices made across individuals. Standard errors are provided in parentheses. * pă0.10, ** pă0.05, *** pă0.01.

11
Table 4: Willingness to Pay (in Terms of Estimated Impact)

||Pooled<br>(1)|Policymaker<br>(2)|Policy<br>Practitioner<br>(3)|
|---|---|---|---|
|Quasi-Experimental|-1.55|0.66|-2.89|
||(1.61)|(3.10)|(1.84)|
|Experimental|-4.83***|-5.26|-4.70**|
||(1.84)|(3.42)|(2.18)|
|Diferent country, same region|-2.07|-1.03|-2.88*|
||(1.57)|(2.95)|(1.74)|
|Same country|-6.70***|-5.77*|-7.55***|
||(1.73)|(2.97)|(2.09)|
|Recommended by local expert|-3.11**|-4.99*|-2.01*|
||(1.23)|(2.84)|(1.12)|
|Small C.I.|-3.67***|-1.21|-5.13***|
||(1.24)|(2.21)|(1.49)|
|Observations|818|407|411|

This table reports the results of conditional logit regressions in terms of the implied willingness-to-pay for programs with certain attributes. “Impact” is implicitly included in each estimate. For example, policymakers would only be willing to accept a program that was not recommended over one that was if the program that was not recommended had a 5.0 percentage point higher estimated impact on enrollment rates. The number of observations represents the total number of choices made across individuals. Standard errors are provided in parentheses. Standard errors are provided in parentheses. * pă0.10, ** pă0.05, *** pă0.01.

12
preference towards programs with studies having larger and more precisely-estimated effects mostly goes away, suggesting these preferences may be driven by significance (Table A2). However, a limitation of these results is that the study was not set up to explicitly test this hypothesis and the levels of the attributes that were used to construct this variable only very crudely capture significance.<sup>12</sup> There are several potential explanations for policymakers’ preferring programs supported by local experts. For example, policymakers may prefer programs recommended by a local expert because they worry about the transportability of research findings, particularly when evidence is not available from their country, and the local expert’s knowledge could comprise an alternative form of local evidence; in other words, they could value the expert for their _local_ knowledge. They could also appreciate the tacit knowledge of a local expert, who may have synthesized a number of data sources, potentially including but not limited to impact evaluation results.<sup>13</sup> Ultimately, we cannot distinguish between these competing hypotheses. However, given that policy professionals appear to care about two forms of local evidence - the recommendations from local experts and having impact evaluation evidence from the same setting - it is natural to wonder whether they view these as substitutes. The main results suggest this is not the case, since both variables are independently significant (Table 3). However, to further explore this issue, we interact whether a program was recommended by a local expert with whether impact evaluation results from the same country were displayed. Results in Table 5 suggest that policymakers do not put any more or less weight on advice from a local expert depending on whether an impact evaluation done in the same country is available, though it is possible that this is a function of limited power and the direction of the estimate for the interaction term is consistent with policymakers considering the two forms of local evidence to be partial substitutes.<sup>14</sup>

> 12In the knife-edge case of a point estimate of 10 and a confidence interval of 10, we consider that confidence interval to (barely) include 0 and hence be “insignificant”.

> 13They could also plausibly trust specific experts, though we did not specify that the “local expert”was any particular individual in the experiment.

> 14Note that merely having impact evaluation results from the same country does not mean those impact evaluation results were promising. The average impact evaluation result was positive (representing an increase in enrollment rates of 5 percentage points), but the weight policymakers place on programs being recommended by local experts could in principle stem from those instances in which an impact evaluation showed no effect. The interaction term remains insignificant when all attribute levels are included, with the same caveat that the experiment was not designed to test for interactions (results available upon request).

13
Table 5: Having an Impact Evaluation from the Same Country is not Seen as a Substitute for Local Expertise

||Pooled<br>(1)|Policymaker<br>(2)|Policy<br>Practitioner<br>(3)|
|---|---|---|---|
|Impact|1.07***|1.05***|1.09***|
||(0.01)|(0.02)|(0.02)|
|Same country|1.61**<br>(0.32)|1.51<br>(0.40)|1.78*<br>(0.53)|
|Recommended by local expert|1.31**|1.44**|1.19|
||(0.17)|(0.27)|(0.23)|
|Same country * Recommended|0.80|0.75|0.84|
|by local expert|(0.28)|(0.35)|(0.46)|
|Observations|818|407|411|

This table reports the results of conditional logit regressions on which program was selected. Odds ratios are reported. The significance of the “Recommended by local expert” indicator and the insignificant interaction between “Same country” and “Recommended by local expert” in Column 2 suggests that policymakers do not put more or less weight on advice from a local expert depending on whether another form of local evidence - an impact evaluation done in the same country - is available. The number of observations represents the total number of choices made across individuals. Standard errors are provided in parentheses. * pă0.10, ** pă0.05, *** pă0.01.

14
While our limited sample size prevents most analyses of heterogeneity, it may be informative to consider how much policymaker and policy practitioner preferences overlap. To this end, we use a mixed logit to estimate the same specifications as Table 3 but allowing effects to vary by individual (a “random effects” model). This model also inherently accounts for the fact that we have more observations for some participants than for others, as not all participants completed the full set of questions, and thus provides a further robustness check to the earlier regressions. Results are presented in Table A3. We observe some interesting patterns: while policy practitioners tend to put relatively more weight on factors relating to the internal validity of an estimate (method and the precision of the estimate) and policymakers put more weight on the recommendation of a local expert, there is substantial overlap in the distribution of their preferences (Table A3). For example, if we were to use the estimated standard deviation to construct the interquartile range (IQR) for the distribution of odds ratios across individuals, the IQR among policymakers for programs supported by experimental results would range from 1.02 - 1.83, while the IQR among policy practitioners would be 0.96 - 3.20. Similarly, the IQR among policymakers would be 0.78 - 1.59 for programs supported by studies with small confidence intervals, compared to 1.29 - 2.70 among policy practitioners. Looking at preferences towards programs recommended by a local expert, the IQR among policymakers is 0.91 - 2.28 and 1.23 - 1.42 among policy practitioners. While the typical policymaker cares more about this attribute than the typical policy practitioner, there is significant heterogeneity in preferences.

Research results serve as only one input into decision-making processes. While fully unpacking the role of different sources of information in policy decisions is beyond the scope of this short paper, we make a simple distinction between two commonlyused sources of evidence and consider how much weight policy professionals place on them. Because of our experiment’s simplicity, there are necessary caveats. In particular, policy professionals in our setting choose between hypothetical programs and their decisions may differ under different circumstances. Further, the information we provide about these programs is necessarily limited in order to focus attention on the attributes of interest, and while the attribute levels were randomized, it remains possible that participants might have selected different options if other attributes had been shown. Nonetheless, we hope that this short paper highlights policymakers’ strong demand for local evidence.

15
# **5 Conclusion**

We find that policymakers place considerable weight on programs that are recommended by local experts as well as those with impact evaluations from their own country. The estimated policymaker preferences can have startling implications: according to our willingness-to-pay estimates, if a policymaker were considering a program aimed at increasing enrollment rates they would be willing to accept a program that had been shown to have a 5.0 percentage point lower impact if the program came recommended by a local expert. They would also be willing to accept a program shown to have a 5.8 percentage point lower impact if it had been evaluated in their own country. These trade-offs can be larger than the effect of many programs. These results suggest that unless research is seen by policymakers as valid in their target setting, policymakers are likely to choose alternative programs that may have lower estimated treatment effects but be from a more relevant-seeming context.

Given that individuals like those in our sample approve and implement many development programs, evidence on how they value the results that research provides can help us understand how to design studies such that they better feed into the evidence-to-policy pipeline. Our results imply that researchers aiming to improve policy should try to design studies to approximate the target context as closely as possible to maximize the chance that their results are taken up by policymakers.

16