**EVALUATION QUESTIONS AND STRATEGY**

Rather, this purposive sample was constructed to parsimoniously cover the range of roles of those engaged in the production chain of IEs. Ten of the selected interviewees worked at multilateral agencies, seven worked at international NGOs, six worked at bilateral agencies, four were from large foundations active in international development, and two apiece worked in a research institute or the Executive Office of the President. Finally, IEG interviewed one individual currently working in an academic institution.

The sample included eight individuals whose primary role was to design and develop analytical tools, guidelines, and templates for economic efficiency analysis such as costbenefit, cost-effectiveness, impact modeling, and Costing studies. Ten interviewees were primarily involved in designing and carrying out impact evaluations at a research institute, a multilateral agency or a non-governmental organization.

Ten interviewees primarily served in the capacity of a funder. Their decision-making often included determining the basis for awarding grants to impact evaluation teams, monitoring progress on the evaluations, reviewing impact evaluation results, and disseminating findings.

Five interviewees function primarily as policymakers. Two policymakers provide training, assistance, review, and support to agencies in fulfilling government-wide policies related to efficiency analysis and evidence standards. Two worked in large, highly influential research institutes or NGOs that help to define international standards of evidence for impact evaluations in international development.

Many of the individuals interviewed had a secondary role of influencing or setting their own institution’s policies. For example, all the funders interviewed also consume and use the results of the impact evaluations and Cost analyses to inform grant-making decisions and institutional policies. Likewise, many impact evaluators use evaluation results to recommend policies and programs, or to identify areas for further and future research.

# **POLICYMAKER’S WILLINGNESS TO PAY FOR COST AND IMPACT INFORMATION**

Our online survey of more than 400 individuals involved in the policymaking process in developing nations presented a hypothetical scenario in which an unspecified education program had generated test score improvements of 0–20 percent in other countries. Respondents were told that they had an opportunity to implement the program for up to 10,000 students in their country, but that they could also choose to do an impact evaluation to learn of the program’s true effects in their setting.

Respondents were asked how many of the 10,000 students they would be willing to have not receive the program in order to be able to pay for the impact evaluation. A similar question was then asked about the number of beneficiaries they would forgo in order to do a Costing study.

Out of concern for framing effects, the order of the first and second of these questions—on impacts and Costs alone—was randomized. Simple t-tests revealed that order did not have an effect on responses for willingness to pay for either Effectiveness or Cost studies.

12