# **EVALUATION QUESTIONS AND STRATEGY**

administrative cost, and other non-comparative discussions of Costs and Benefits as GEA. For example, we classified as GEA “Seaweed Farming in Indonesia” based, in part, on the following statement: “SEAplant’s cost has been high relative to benefits derived to-date. IFC committed roughly $1.9 million to the SEAplant program through 30 June 2006, and this investment has not yet yielded significant tangible returns in terms of either increased earnings for farmers or the introduction of value-added processing.” We documented the cost indicators and ingredients by type, e.g., opportunity costs, discount rates, timeframes, inflation, and exchange rates between local currency and the currency used in the analysis, for each IE reviewed in detail. Transparency was evaluated along six dimensions: 1) methods clarity (i.e. statement of a specific, identifiable method), 2) cost reporting (i.e. listing of ingredients and their value), 3) analysis (i.e. level of aggregation, unit cost per impact, unit cost per input, treatment of opportunity cost) , 4) Cost adjustment reporting (i.e. reporting discount rates where appropriate, inflation adjustments, and currency exchange rates), 5) rationale (i.e. explicit rationale for a VFM analysis), and 6) study limitations, (i.e. reporting partial analyses such as the exclusion of costs related to some benefits derived from the program, or the use of approximate costs, data quality issues, missing data, or focusing on only a short period for accounting benefits).

## **FREQUENCY OF VFM IN GLOBAL IMPACT EVALUATIONS**

To estimate the frequency of VFM analysis in global impact evaluations, we first developed a set of keywords to represent each of the VFM methods based on extensive review of the VFM literature. Keyword sets were developed to represent cost-effectiveness analysis, costbenefit analysis, cost-utility analysis, financial analysis, social return on investment, basic efficiency resource analysis, multi-criteria appraisal, and rank correlation of cost vs impact.

A text analytics algorithm was conducted over a full-text search of each of the 236 impact evaluation studies to generate tables of keyword frequencies. This method also was applied to a “benchmark sample” of 65 studies drawn from a database of 168 World Bank Group impact evaluations studies published between 2002 and 2011 (World Bank Group impact evaluation database).

The benchmark sample was formed by combining IEs from two groups of World Bank impact evaluations. The first group contains 35 IEs identified as including some kind of (unverified) efficiency analysis. The second set of studies was pulled randomly from the remaining (untagged) 131 studies contained in the near census of World Bank impact evaluations described above. The outcome variable for the untagged impact evaluations is “No VFM.” Since IEs in the first and second group do not have the same probability of being selected into the benchmark sample, we applied a sampling weight. The probability of selection into the benchmark sample is 1 for the 35 tagged studies. The probability of selection for the untagged studies is 30/131 = 0.229 for the 30 untagged studies. Hence the sampling weight, 10