One of the driving incentives for IE producers—whether in academia or (to a lesser degree) in international organizations—is the opportunity to publish in professional journals. From a researcher’s perspective, there is little incentive to take time out of the preferred activity— measuring benefits—to collect high quality cost data and work through tricky cost estimates and assumptions to achieve a VFM analysis with the same level of rigor as the IE if it is not rewarded.

Evidence from our journal editor survey indicates that demand for VFM from top journals is mixed at best. Perhaps most illustrative of the low esteem held for VFM by journals is the fact that while nearly all of the top economics and economic development journals that we investigated had statements on the standards of estimating benefits (often termed effects, outcomes, results, and so on), none addressed quality control criteria for Cost estimates, much less efficiency. Additionally, 13 of 14 journal editors surveyed reported neither formal nor informal editorial practices governing inclusion of (CEA, CBA, or other VFM analysis) in published impact evaluations. Twelve of the journal editors did not give special consideration to VFM analysis—an IE with VFM is generally no more likely to be published in the top academic journals than is an IE without VFM analysis.

Journal editors mainly thought that Value for Money is so infrequently incorporated into impact evaluations because so few studies can obtain high quality cost data. “While such calculations are helpful in ‘ball parking’ the magnitude of an intervention, and whether it is ‘worth it,’ I suspect most researchers don't believe that the estimated numbers are of sufficient precision or reliability to require people to put dollar figures on interventions.” Several journal editors noted methodological issues and a lack of standardization could cause the analysis to be rejected: “Because careful valuation of Costs and benefits involves difficult decisions on certain parameters, such as shadow prices, discount rates and welfare weights, which may be arguable, and thus expose authors to additional rejection risks from referees.” Two editors said they will ask authors to remove VFM analyses. Two others noted that CBA, if done carefully, could stand on its own and deserved to—rather than be crowded in with impact estimates.

Still, about half of the editors indicated that VFM should be included in IEs. Collectively the Journal Editors provided three stipulations for including VFM analysis: each case would need to be considered on its own merits; the analysis would have to be done well and convincingly; and the journal’s quality standards would have to be met. Editors indicated that VFM analysis with a high level of rigor could be reported independently, although not necessarily in a top journal—the currency by which many IE producers exercise influence and secure tenure.

Indeed, although journal editors raised the notion that CBA, if done properly, can potentially stand on its own, there are few if any examples of this in those editors’ own journals. Scrutiny for VFM quality in journals is de facto left to peer reviewers, who are almost certain to be inconsistent in the standards that they apply. That inconsistency adds another layer of uncertainty—and so risk—to a prospective author.

18