metadata: authors: 0: author: Bell,Clive L.G.
1: author: Squire,Lyn
count: World
docna: 0: docna: Providing policy makers with timely advice :
            the timeliness-rigor trade-off
docty: Policy Research Working Paper
subtopic: Health Care Services Industry,Labor Markets,Health Economics & Finance,Primary Education,Health Insurance,International Trade and Trade Rules
abstracts: cdata!: Policy makers bemoan the lack of
            research findings to guide urgent decisions, whereas
            researchers' professional code puts rigor first. This
            article argues that provisional assessments, produced early
            in the research cycle, can bridge the gap. Numerous case
            studies point to the importance of early interaction with
            policy makers and the delivery of brief, policy-focused
            papers; but preliminary analyses may be flawed and so
            increase the chances of a wrong decision. This article
            demonstrates analytically that a preliminary assessment,
            supported by the offer of more refined research, provides an
            option that is superior, on average, to the current practice
            of submitting a final report at the end of the research
            cycle. Where practical implementation is concerned, it calls
            for donor-funded subsidies to promote the use of provisional
            assessments and for a rapid, independent, professional
            review process to ensure their quality. While the
            research-policy exchange in developing countries is a
            complex, context-specific phenomenon, the proposal offered
            here holds out some promise of improving decisions in the
            public sphere under a wide range of circumstances.
repnme: repnme: Providing policy makers with timely advice
            : the timeliness-rigor trade-off
display_title: Providing policy makers with timely
            advice : the timeliness-rigor trade-off
listing_relative_url: /research/2016/03/26089560/providing-policy-makers-timely-advice-timeliness-rigor-trade-off
topicv3: Primary Education m1326139 480,Informal Sector m1326258 1168,Trade m1326924 698,Health Insurance m1327759 1287,Health m1504661 3480
publishtoextweb_dt: 2016-03-18T00:00:00Z
no_of_pages: 36
author_keys: 0: author_key: 430403
1: author_key: 88472
model_extractions: WPS7610 Policy Research Working Paper 7610 Providing Policy Makers with Timely Advice The Timeliness-Rigor Trade-off _Clive Bell Lyn Squire_ Development Economics Vice Presidency Operations and Strategy Team March 2016
Policy Research Working Paper 7610

# **Abstract**

Policy makers bemoan the lack of research findings to guide urgent decisions, whereas researchers’ professional code puts rigor first. This article argues that provisional assessments, produced early in the research cycle, can bridge the gap. Numerous case studies point to the importance of early interaction with policy makers and the delivery of brief, policy-focused papers; but preliminary analyses may be flawed and so increase the chances of a wrong decision. This article demonstrates analytically that a preliminary assessment, supported by the offer of more refined research, provides an option that is superior, on average, to the current practice of submitting a final report at the end of the research cycle. Where practical implementation is concerned, it calls for donor-funded subsidies to promote the use of provisional assessments and for a rapid, independent, professional review process to ensure their quality. While the research-policy exchange in developing countries is a complex, context-specific phenomenon, the proposal offered here holds out some promise of improving decisions in the public sphere under a wide range of circumstances.

This paper is a product of the Operations and Strategy Team, Development Economics Vice Presidency. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at clive.bell@urz.uni-heidelberg.de and lynsquire@yahoo.com.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Providing Policy Makers with Timely Advice:**

# **The Timeliness-Rigor Trade-off**

_Clive Bell and Lyn Squire_ If economists could manage to get themselves thought of as humble, competent people on a level with dentists, that would be splendid.

—J. M. Keynes, _Essays in Persuasion_ , 1931.

Keywords: Decision-making under uncertainty, developing countries, policy research, research-policy dialogue, rigor-timeliness trade-off JEL Codes: D61, H43, O20, O21, O22 Clive Bell is professor of economics, emeritus, at the University of Heidelberg and Adjunct Senior Researcher at Chr. Michelsen Institute (CMI), P.O. Box 6033, N-5892 Bergen, Norway; his email address is: clive.bell@urz.uni-heidelberg.de. Lyn Squire (corresponding author) is the former president of the Global Development Network, 4, Vasant Kunj Institutional Area, New Delhi-110070, India; his email address is <u>lynsquire@yahoo.com. We are much indebted to Pierre Jacquet for valuable and extensive comments on an earlier</u> draft. Subsequently, helpful suggestions from Fred Carden, Randy Filer, Stefan Klasen, Halvor Mehlum, Lant Pritchett, Indira Rajaraman, Alan Winters, participants in CMI's annual development seminar, and especially two anonymous referees led to improvements in various parts of the text. We bear sole responsibility for all remaining errors of analysis and opinion.
# I INTRODUCTION

The development community has embraced the notion of so-called evidencebased policy, which is generally understood to be public policy informed by rigorously established, objective evidence (see, e.g., Pawson 2006; Nutley et al. 2007; Carden 2009). This general desideratum is certainly not new, but it was popularized lately by the Blair Government in the United Kingdom (Cabinet Office 1999a, 1999b). Official pronouncements have their value, but the link between research and policy is often tenuous at best.

The focus of this article is the interaction between researchers and policy makers in developing countries. Descriptions of this process usually stress its complexity and context-specificity and point to a multitude of factors, other than research, that influence decisions (Court et al. 2005; Livny et al. 2006; Carden 2009). Policy makers have to worry about the interests of relevant constituents, pressure from external agencies, and personal career ambitions; they may have to contend with weak implementing agencies, corruption, and even violence; they may prefer to draw on their own experience and knowledge or that of their immediate advisors; and they may feel that local research is unreliable or based on data that are limited, inaccurate, or unavailable when required. In sum, even where research appears to play a role, it will only be one, and usually not the most important, in a complex array of factors influencing policy.

If the aim is to make research more influential, efforts to remove, or at least weaken, the impediments to a vigorous research-policy interaction merit serious consideration. The contribution of this article is to propose a way of overcoming one of these obstacles—the frequent unavailability of research when required for
urgent policy decisions. While research must be sufficiently rigorous that, if acted upon, there is a reasonable expectation it will improve the policy decision, it must also be timely, in the sense that it has to reach the policy maker before the decision is made. This article argues that timeliness of results has received less attention than research quality—and less than it merits.

Meeting accepted professional standards has been the watchword for the research community. In consequence, efforts to feed advice and guidance into the policy-making process are usually undertaken only at the end of the research cycle, after the research has been completed and vetted. While there is an obvious rationale for this—no one wants results based on badly flawed analysis or grossly inaccurate data reaching the policy-maker’s desk—the approach runs the risk that results arrive too late to be of value. Indeed, decision makers worldwide constantly voice frustration with the slowness of the research production process. The ODI/SciDev.Net International Survey on the Science-Development Policy Interface, for example, reports that “A general consensus from the expert interviews was that a major challenge is the narrow focus and long time-scales of scientific research compared with political priorities” (Jones et al. 2008, 20). Similarly, an ODI Briefing Paper notes that “Policymakers bemoan the inability of many researchers to make their findings accessible and digestible in time for policy decisions. Practitioners often just get on with things” (Overseas Development Institute 2004, 1).

A possible remedy is to encourage the submission of preliminary results early in the research cycle in order to increase the likelihood of their reaching the policy maker before the decision has been taken. It is this thought that leads to the main proposal of this article—the introduction of what we term provisional assessments. As explained more fully in section II, provisional assessments are short papers focused on a current policy issue and delivered to the policy maker
early in the research process. With timeliness at a premium, the preliminary analysis presented in these submissions should be based entirely on existing, easily accessible information and routine calculations, with evidence from other countries or extrapolation of historical trends being called upon to supplement whatever hard data are at hand. In consequence, a provisional assessment should also indicate whether its findings are sufficiently robust to warrant an immediate decision, and if not, what additional evidence and analysis are needed. Ideally, the latter conclusion should lead to a request from the policy maker for a more thorough investigation, accompanied by an indication of when results are required if they are to be useful.

If, as might first appear, provisional assessments simply provide an additional arrow in the researcher’s quiver to be used when and where needed in the effort to inform policy, then, apart from a few remarks on implementation, no further discussion would be required. Provisional assessments come, however, with a potential cost. The danger is that the policy maker acts immediately on receiving one. Had the researcher bided her time, completed her research as thoroughly as possible, and only then submitted a final report, its findings might have pointed to a quite different policy. Meanwhile, the policy maker, who is buffeted by the pressures of office, might have made his decision well before the final report reaches his desk. In short, there is a trade-off between timeliness and rigor. Section III addresses this trade-off analytically. It is one of the first attempts to our knowledge to go beyond the reliance, in the research-policy literature, on case studies by offering a rigorous analytical framework. It is demonstrated that a provisional assessment, backed by the promise of a more complete assessment to follow if the provisional one points to significant uncertainty about the policy’s social profitability, is superior to the current practice of submitting only a final report.

Section IV deals with implementation. It explores ways of promoting the use
of provisional submissions, currently a missing component in the research-policy chain, initially by means of inexpensive, donor-funded subsidies. It also proposes checks on provisional assessments in order to ensure quality and reduce the chances of an incorrect decision, even if that decision is taken immediately on receipt of the preliminary analysis. The section also discusses ways of increasing the likelihood that the initial approach to the policy maker does in fact lead to whatever further research is recommended in the provisional assessment. These practical measures buttress our claim that provisional assessments, if widely used, could have a significant and favorable impact on policy in many situations. Section V summarizes the argument and proposals.

# II PROVISIONAL ASSESSMENTS

# _Rigor Reigns_

The primacy of rigor over timeliness is not surprising given that the incentives prevailing within the economics profession promote rigor before all else. There are well-established mechanisms to ensure that professional standards are met through the widespread use of peer reviews; and researchers, who naturally seek professional advancement, are keen to submit their research to a thorough, independent vetting by other scholars. There is, in contrast, no such history of individual or collective action from within the profession to promote timeliness in relation to policy making.

As one illustration, consider the widely held view that randomized controlled experiments are superior to other techniques of impact evaluation such as multivariate regression and propensity score matching, even though these other options offer the prospect of yielding results more quickly. Thus, the website of the Abdul Latif Jameel Poverty Action Lab notes that “Randomized evaluations are
often deemed the gold standard of impact evaluation, because they consistently produce the most accurate results” (http://www.povertyactionlab.org/about-j-pal).<sup>1</sup> To compound matters, the instruments currently used to reach policy makers—the media, dedicated seminars, and policy briefs<sup>2</sup> —are not well designed to ensure timeliness, since they are almost invariably brought into play only after the research has been completed and reviewed. Thus, the main vehicles now used to strengthen the impact of research on policy reinforce, or at least coincide with, the researcher’s commitment to rigor, in that both imply an interaction with the policy-maker that follows research, and therefore raise the question of whether as much as possible is being done to ensure the timely delivery of results.

# _A Way Forward_

The experience of the International Development Research Centre (IDRC), perhaps the leading international organization as far as linking research and policy is concerned, as well as that of other institutions, suggests a way forward. In particular, Carden (2009, 45), drawing on a detailed review of 23 IDRC-funded research projects and other evidence, stresses two points: first, the importance of initiating a dialogue with the policy maker as early as possible in the research process; and second, the value of submitting short papers tightly focused on the policy question under consideration, as opposed to the more traditional, methodology-heavy accounts typically produced at the end of a research project. It is this combination of early interaction and policy-focused submissions that leads to the main proposal of this article—the introduction of what we term provisional

> 1. Similar statements abound: the mission statement of the Coalition for Evidence-Based Policy asserts that “evidence of effectiveness generally cannot be considered definitive without ultimate confirmation in wellconducted randomized controlled trials” (http://coalition4evidence.org/). The Report of the Evaluation Gap Working Group observes that “most questions about the impact of social programs require collecting data over years. Valid evidence of a program’s effectiveness often cannot be produced in less time” (Savedoff et al. 2006, 25) 2. For a useful overview of policy briefs, see Jones and Walsh (2006).
assessments.

The type of research for which provisional assessments are intended is typically country-specific, issue-specific, and employs techniques such as costbenefit analysis, cost-effectiveness analysis, and impact evaluations, possibly supported by econometrics or partial and even general equilibrium models.<sup>3</sup> The following case study provides an example of how a provisional assessment can be used to address the type of well-defined policy question we have in mind.

The issue considered by Annor-Amevor et al. (2012) is the low pass rate for the Basic Education Certificate Examination achieved by Ghanaian children in junior high schools, a matter of then current concern in Ghana.<sup>4</sup> In particular, the authors assess the merits of participatory remedial classes for those who have failed the examination, combined with an opportunity to re-sit without repeating previous school grades. Besides useful background information, the study provides information on four key magnitudes: • Size of the population likely to be affected. The authors estimate that 41,686 children failed the examination in the five regions under study.

• Costs. They compile a detailed estimate of expected costs, concluding that the cost per pupil will be $254.50.

• Impact. Based on experience elsewhere, they assume that the pass rate will increase from 62 to 77 percent as a result of the intervention.

• Benefits. Using current Ghanaian data on earnings, they project that a certificate-holder will earn 20 percent more than someone without the certificate.

Armed with this information, the authors conduct a cost-benefit analysis and

> 3. Another kind of research attempts to improve our understanding of development in the broad. The findings of such research typically enter the policy-making process indirectly and only gradually. They first appear in, and then percolate through, the pages of academic journals, those important devices for quality control. Only then do they enter policy fora and permeate policy discussions.

> 4. This is one of twelve country studies in a research project entitled “Strengthening Institutions to Improve Public Expenditure Accountability,” managed by the Global Development Network and Results for Development.
arrive at a positive net present value for the proposed intervention. They also undertake a sensitivity analysis, especially with respect to impact and benefits, which suggests that the initiative is worth undertaking for a wide range of estimates. The example highlights the three characteristics that are likely to be typical of provisional assessments: focus on a well-specified policy issue; use of standard techniques; and reliance on readily available information.

# _The Timeliness-Rigor Trade-off_

The Ghanaian example illustrates the trade-off between timeliness and rigor. All required information for this analysis could be obtained from existing sources because the analysts were willing to draw not only on “hard” evidence (coverage and costs) but also on the “soft” kind, in the form of international experience (impact) and extrapolation (benefits). This emphasis on ready availability should not be understood to rule out the possibility of collecting original data; but the key point is that the information underlying a provisional assessment, whatever the source, has to be marshalled quickly.<sup>5</sup> Even though this approach will usually entail reliance on data that are soft and potentially unreliable, professional integrity is in no way compromised. All that is required is that the provisional assessment states clearly which inputs into the analysis have been established fairly accurately, which have been based on historical or international experience,<sup>6</sup> and how use of the latter affects the robustness of the findings.

> 5. An example involving the use of specially collected data is provided by an analysis of the extension of the Nicaraguan Social Security Institute's health insurance program to informal sector workers. The easily ascertained fact that the proposed cost of the insurance was considerably more than current, out-of-pocket health expenditures clearly pointed to the likelihood of low take-up rates, an indication subsequently confirmed by a controlled randomization experiment (Thorton et al. 2010).

> 6. Two recent articles address issues surrounding the use of estimates of impacts drawn from other countries. Dhaliwal et al. (2012) provide guidance on how to compare results based on RCTs, using evaluations of educational programs in multiple countries as an example. Pritchett and Sandefur (2013) add the important point that context matters, arguing that nonexperimental results from the right context are currently a better guide to policy than experimental ones from a different context.
In the Ghanaian example, the robustness checks suggest that the downside risk of sacrificing full rigor is negligible, but this cannot hold in general. In evaluating the potential of provisional assessments, therefore, it would be desirable to have some sense of the size of this downside risk and, going a step further, whether the sacrifice of rigor involved in providing such assessments likely nullifies their advantage in terms of timeliness, a task to which we now turn.

# III ANALYZING THE TRADE-OFF

A project or program is under consideration. Both the exact timing of the simple “yes-no” decision of whether to adopt it and its social profitability are uncertain. There are two actors, a policy maker and a researcher, hereinafter denoted by D and R. As “insiders” in their respective fields, the former is better informed about timing, the latter about social profitability, but neither is perfectly so. An exchange of information, however incomplete or indirect, may improve the chances that the decision, when it does come, will be the right one, in the sense that the chosen course of action is welfare-superior when viewed _ex ante_ at the time of decision.

There is always an alternative to the action in question, and consequently a cost of choosing wrongly. The net social benefit of making the correct decision, when it occurs, is assumed to be given, and is therefore omitted from the analysis. In this connection, it should be recalled that otherwise identical projects or policies initiated at different dates are indeed different. In general, therefore, timing will affect the level of social profitability _ex ante_ as well as an outcome _ex post_ . Both parties are assumed to be aware of this fact, though R will be able to formulate it more exactly when assessing the probability that an affirmative decision at a particular time _t_ will be correct.

An assumption about how to treat uncertain outcomes in public decision-
making is also needed. Let the assumptions for the validity of the Arrow-Lind (1971) theorem hold, namely, that the project’s net returns be evenly spread over a large population and statistically independent of those of other projects. It is then valid to use expected values in making decisions, even though some outcomes are poor when viewed _ex post_ . For simplicity, we also omit the social cost of producing the advice from the formal analysis.<sup>7</sup> Let the variate _T_ denote the date on which the decision occurs. At the outset (at time _t_ = 0), D’s subjective prior is that _T_ has the continuous distribution function _H_ ( _t_ ), whose support is [ _t_ 1, _t_ 2 ]. In the absence of the specific advice which will be treated below, his own knowledge, understanding, predilections, and such other opinion—expert or otherwise—at his disposal combine to yield the probability<sup>_q_</sup> _t_<sup>that the decision, if it occurs on the date</sup><sup>_t_, will be the correct one.</sup> Along with political pressures, these various influences are themselves likely to fluctuate over time. This does not, however, imply that, for any given _t_ ,<sup>_q_</sup> _t_<sup>is</sup> “fuzzy.” In this, we follow Elga (2010), who argues that rational agents must have 1 2 sharp subjective prior probabilities: for each and every _t_  [ _t_ , _t_ ] , D has an exact _ex ante_<sup>_q_</sup> _t_<sup></sup> [0,1]. At the outset, therefore, D assesses the corresponding _ex ante_ probability that the decision will be correct to be Aware that the issue is on D’s agenda but initially uninformed about his _H_ ( _t_ ) and<sup>, R undertakes to analyze it.8 The longer she devotes herself to this task, the</sup><sup>_q_</sup> _t_ better founded the advice will be, and the more compelling will be the way in

> 7. To a fairly good approximation, this is the value of the researcher's time devoted to the task when priced at her shadow wage rate.

> 8. We assume that the interaction between the researcher and the policy maker is initiated by the former, this being, in our view, the more realistic representation. The analytical apparatus could, of course, be re-jigged to make the policy maker the instigator, a desirable extension that we discuss, among others, in section IV.
which she frames the evidence and arguments. The advice will therefore improve as time passes, especially in the sense of increasing the probability that the correct decision will be taken—provided the memorandum is actually on D’s desk when the decision point arrives. Timeliness, then, is essential if her efforts to make a persuasive case are not to be in vain. When weighing all these considerations, she has her own subjective priors at the outset, _F_ ( _t_ ) and<sup>_p_</sup> _t_<sup>, respectively.</sup> Even if the research results are delivered in time, whether they are used depends on D’s response. In particular, it is arguable that the gap between the memorandum’s arrival and the actual decision matters. Advice that arrives far in advance may suffer a loss of effectiveness simply because it appears a bit dated when that moment of decision does arrive. There is even the mundane possibility that the memorandum will get mislaid or overlooked, as D’s office struggles to deal with the stream of matters demanding his attention. Working against these hazards is the fact that a memorandum’s early arrival gives him the opportunity to reflect on it, which should make it more influential.

D and R face a common hazard, namely, that there will be a “changing of the guard” before the decision actually occurs.<sup>9</sup> The new regime may take the form of a new government or, if D is a senior bureaucrat, his replacement through regular rotation or otherwise. As D and R reflect on the possible timing of the decision and the associated probability that it will be correct, each must also form a prior probability of whether a changing of the guard will take place, and if so, when and with what consequences for timing and a happy outcome.

There are three broad possibilities, though D and R may well differ in their assessments of whether, when and the resulting consequences. First, there is a firm conviction that the decision will occur before any changing of the guard. Second, if 9. We are indebted to a referee for urging us to examine this possibility.
such a change could happen, its date can still be forecast without error, as might well be the case in a stable democracy with a fixed electoral cycle and any stable, comparatively independent bureaucracy. Then the forms of _H_ ( _t_ ) and<sup>_q_</sup> _t_<sup>will reflect</sup> D’s assessment of how the event will influence them, and the same applies to R’s _F_ ( _t_ ) and<sup>_p_</sup> _t_<sup>. Note that this includes the special case in which the said date precedes</sup> the prior estimate of the earliest possible date for a decision. The remainder of this section is formulated on the basis that one or other of these two broad possibilities holds.

Turning to the third possibility, what happens if a change of regime within the relevant time interval cannot be ruled out and its timing is uncertain? In principle, D must formulate a pair _H_ ( _t_ ) and<sup>_q_</sup> _t_<sup>for each date on which such a</sup> change can occur with positive probability, together with the associated prior probability that it will occur on the date in question. The resulting compound lottery will tax most people’s powers of analysis, so D searches for a plausible simplification.

_d_ 1 2 _Assumption_ . The event in question occurs on the date _t_  ( _t_ , _t_ ) with _d d_ probability  , or not at all. If it occurs, _H_ ( _t_ ) is unaltered up to _t_ = _t_ and is amended to _H d_ ( _t_ ) over the remaining, revised interval [ _t d_ , _t_ 2 _d_ ] ; and<sup>_q_</sup> _t_<sup>is likewise</sup> amended.

_d_ Note that the problem only arises when _t_ lies in the said interval; otherwise, either the first or second broad possibilities set out above will apply. The single possible date represents a sort of average, albeit one weighted by whatever information at his disposal. In a very uncertain setting, he might well choose the halfway point. Let the same assumption apply, _mutatis mutandis_ , to R’s view of things. The Appendix sets out how the basic elements _Q_ , and—see immediately below—R’s<sup>_p_</sup> _t_ (<sup></sup> )<sup>and</sup> _P_  ( ) must be reformulated to cover this third possibility,
whereby the extension to two or more possible dates is obvious.

With these considerations in mind, let R’s prior probability that a memorandum submitted at time  will result in a correct decision occurring at time _t_ be as follows: where the “influence function”  reduces 1 <sup>_p_</sup> _t_ , the probability of an erroneous decision. This function is assumed to be decreasing in its first argument (research quality), but it may be decreasing or increasing in the second (D’s response to the gap between the delivery of the advice and taking the decision). The assumption that a submission that arrives before or at the point of decision makes a correct decision more probable—but not certain—is plausible. For if R is competent, even her preliminary analysis is very likely, _on average_ , to improve on whatever process yields D’s unassisted<sup>_q_</sup> _t_<sup>. This assumption implies that</sup>  <1 if  > 0 , with  (0, _t_ ) =1. Observe also that the larger is<sup>_p_</sup> _t_<sup>, the smaller is the effect of a specific piece of</sup> advice,<sup>_p_</sup> _t_ (<sup></sup> )<sup>being at most unity.</sup>

# _A Single Submission_

Let R obtain no additional information about the timing of the policy decision in the course of her work, so that she gains nothing by postponing her decision about when to submit her memorandum, and she will hold fast to the plan  chosen at the very start. If she waits beyond _t_<sup>1</sup> , it is possible that she will submit her advice too late. This must be weighed against the improvement in its quality that would result from taking longer over it. The plan  yields her _ex ante_ probability that the policy decision will be the correct one:
The trade-off is described exactly by differentiating w.r.t.  : where<sup></sup> _i_<sup>denotes the partial derivative ofw.r.t. its</sup> _ith_ argument. The first expression on the right-hand side is the loss in terms of _P_ caused by spending a little more time on the memorandum, and so increasing the chances that it will arrive too late; the second expression is the corresponding gain arising from the improved analysis, if it arrives on time. As noted above, the assumption that advice becomes more influential as its quality improves implies that  1<sup>< 0</sup> . While  2 may take either sign, it is very plausible that, on balance, better analysis will reduce the chances that the wrong decision will follow a timely submission, that is, _d_  = (  1  2 ) _d_  < 0 .

Let denote R’s optimal plan. It is clear from equation (4) that waiting until the very last 2 minute (  = _t_ ) is not optimal. Thus, a researcher’s natural inclination to delay any policy input until all refinements have been incorporated, and all conceivable robustness checks undertaken, may very well defeat the purpose of influencing the decision. Indeed, the possibility that, at the other extreme, taking no chances at all is optimal, that is,  * = _t_ 1 , cannot be ruled out. Indeed, if  1  2  0 , then that is exactly what R should do.
# _A Provisional Submission_

The preceding subsection sets out the costs and benefits of delaying a single submission when R cannot acquire additional information about the timing of the policy decision. We now allow her to test the waters by submitting a provisional assessment, accompanied by the offer of a more definitive analysis to follow if required. This provisional assessment also goads D into playing an active role in the ‘game’ between them. We shall prove that the resulting option has a positive value, relative to a single submission, in the sense of increasing the probability that D will decide correctly.<sup>10</sup> 1 2 Let R submit a provisional assessment at time  [ _t_ , _t_ ] . If the decision has already been taken, she is so informed, and there is nothing more to be done. If, on the contrary, the decision is still due, let D respond in just one of three ways. First, with her memorandum now lying on his desk, he waits no longer and makes his decision, and R is informed accordingly. It is rather likely that D’s precipitate action is prompted by the memorandum telling him what he wanted to hear, but its professional content still raises the probability that the decision is correct; for the memorandum reflects R’s assessment of the project’s social profitability, albeit a provisional one. On the basis of the information available to her, the probability that the decision is correct is 1  (1  _p_  )  ( ,0) , with a corresponding improvement over _p_  of (1  _p_  )(1  ( ,0)) .

Second, D expresses a clear interest in a more definitive assessment and sets * 2 a deadline _t_  (  , _t_ ] , which both parties regard as fixed, and the final memorandum * is submitted at _t_ . This more measured response requires D to choose a deadline in the light of his subjective priors at _t_ =  concerning the timing of the decision and the probability that it will be correct. His decision problem is therefore analogous

> 10. The provisional assessment is most unlikely to pass muster with a refereed journal. We return to the issue of quality in section IV.
to R’s in the case of a single submission. Let there be no new information about timing, other than the fact that the decision is still outstanding, so that _H_ ( _t_ ) is truncated at _t_ =  . Since R will have worked on her assessment throughout the * period up to _t_ = _t_ , let its effect on D’s prior at _t_ =  that the decision, when it comes, will be correct take the form analogous to equation (2): which allows for the possibility that events will force a decision before the * deadline _t_ and so confine R’s influence on the proceedings to her provisional memorandum.

Corresponding to R’s _ex ante_ probability _P_ , D’s _Q_ at _t_ =  is given by

* * The trade-off confronting D when choosing _t_ so as to maximize _Q_ ( _t_ ;  ) is also analogous to that facing R in  the case of a single submission and need not be set *0 out in detail. Let _t_ denote the maximizer in question. Since D’s course of action necessarily involves _t_ *0 >  , it follows that it must yield an improvement not only over receiving no advice from R at any stage but also over just her provisional assessment at _t_ =  . R is privy only to D’s specified deadline. While she could infer something about D’s priors at _t_ =  , such speculation will serve little or no purpose in this round—though it might do in the future should they have further dealings. She therefore labours away until submitting her final assessment at _t_<sup>*0</sup> . Having done so, * she achieves a total improvement over her _ex ante_ prior *0 of (1  *0 )(1  ( _t_ ,0)) ,<sup>_p_</sup> _t_<sup>_p_</sup> _t_ as she sees things.

D’s third possible response to R’s provisional assessment is to send her a
polite, noncommittal acknowledgement, so that she gains only the information that the decision is yet to come. She must now decide how much more time to invest in producing a final memorandum, with no firm deadline. It is the remaining uncertainty in the event of such an acknowledgement at  that makes the problem of backward induction, and hence of determining the optimal plan for  at _t_ = 0, nontrivial—in principle, at least. The corresponding optimal timing of her final submission, denoted by  0 ( ) , must be found in order to establish the optimal choice of  at _t_ = 0.

She is now back in the setting of a single submission, with starting point _t_ =  . It is plausible that learning that the decision has yet to be made will have no effect on her priors other than truncating _F_ ( _t_ ) at _t_ =  . Let this be so. Then the effect of a provisional submission that draws a noncommittal response on the probability that the final decision will be correct will be given by the upper branch of equation (2), unless it is followed up by a timely final submission, in which event,  will replace  in  ( ) and thus represent the improvement in the quality of the advice.

Conditional on the event that such an acknowledgement is received at  , the 2 choice  [  , _t_ ] yields, analogously to equation (3), where The associated first-order condition for an interior maximum is analogous to equation (4):
A comparison of the two conditions establishes that  0 ( ) is indeed dependent on  when the latter is the date on which the provisional assessment is submitted. To 2 be more precise, consider the choice  =  < _t_ , which is effectively the same as making a single, final submission at  . The first term in brackets on the right-hand side of equation (8) vanishes, and the integral is negative, in virtue of  1  2<sup>< 0</sup> . 0 Hence,  ( ) >  , and the difference between the values yielded by submitting at  0 ( ) and  , respectively, yields the (positive) option value of a provisional 1 2 submission for any choice of  ( _t_ , _t_ ) , conditional on R receiving a noncommittal reply.

The size of the option value varies, in particular, with  and the behavior of  . Inspection of equation (7) reveals that the option value will be small if  is 2 sufficiently close to _t_ . For a delay in making the provisional submission reduces the time in which to make the most of the resulting option; and it lowers the probability that the option will arise in the first place. It is also seen that the option value is smaller, _cet. par._ , if  is decreasing in its second argument; for the interruption at the point  reduces the delay from _t_  to _t_  thereafter. If, conversely,  increases with the delay, the interruption will be advantageous.

Now R’s choice of  also arguably affects the probabilities of getting the responses listed above, conditional on the decision yet to be taken. That is to say, the act of making a provisional submission also affects the timing of the decision whose outcome it intends to influence. It is plausible that the closer  approaches 2 the upper limit _t_ , the greater is the chance that it will “trigger” the decision, an outcome whose probability is denoted by  1<sup>()</sup> . The opposite holds for the probability of getting a noncommittal response,  3<sup>()</sup> ; for the earlier the provisional submission is made, the sketchier it is likely to be—and to appear. At all events, R’s plan to submit a provisional assessment at  , with her rational
anticipation of choosing  0 ( ) in the event of receiving a noncommittal acknowledgement, involves the following possible outcomes: (i) the provisional memorandum arrives too late, with probability _F_ (  ) ; and (ii) it arrives before the decision, with probability 1  _F_ (  ) , and then elicits one of the three said responses There is, however, one further complication to be resolved before R chooses *0  at the outset, namely, that the contingent deadline _t_ has yet to be revealed. The latter certainly depends on  . R knows nothing about D’s priors at this stage, so it is natural for her to assume that this date, denoted by _t e_ , is uniformly distributed on 2 _e_ the interval [  , _t_ ] . Yet whatever be her prior distribution of _t_ in relation to  , R’s _ex ante_ prior probability that the decision, when it does come, will be correct is where _E_ is the expectation operator. Inspection of equation (9) reveals that, as R sees things, the probability of securing a correct decision with a provisional submission is necessarily superior to that with a single, final submission if  1<sup>() <1</sup> . For a single submission is equivalent to declining an invitation to submit a final assessment against a fixed deadline, or, in the event of receiving only a noncommittal reply, to choosing  =  whatever be the choice of  , and it has been proved above that neither course of action is ever optimal.

The choice of  itself is influenced by various considerations. Under the 2 above assumptions, a provisional submission before _t_ always yields, with strictly positive probability, the opportunity both to deliver the (constrained) best advice and to adjust the plan optimally when uncertainty about the timing of D’s decision
still remains. Against this, there is the drawback that a provisional submission will, with positive probability, precipitate an immediate decision. For its arrival may focus his attention on the matter in question, especially if there is a good chance that a leak might arouse the interest of other influential parties—and provide them with ammunition. If  ( )  is decreasing in its second argument, such an early triggering of the decision will decrease the probability that it will be the right one. For the delay _t_  between the provisional submission and the decision then has a favorable effect on  – whenever there is such a delay.

The above proof of the superiority of a provisional over a single, final submission implies that the trigger effect cannot, on average, outweigh the others.<sup>11</sup> To grasp why this is so, consider the limiting case in which the trigger effect occurs with certainty (  1 =1,  2 =  3 = 0) for all  ; for then there is effectively neither an option nor an invitation with a fixed deadline, and hence no advantage to a provisional submission. Indeed, all provisional submissions become final ones. There are, of course, consequences for the optimal choice of  , but that is another matter. In this connection, it should also be noted that in the case of a single submission, there is neither a trigger effect nor even the interim information as to whether the decision has been taken. In equation (9), the trigger effect is expressed by the term involving  ( ,0) , but the corresponding term in equation (3),  ( , _t_  ) , contains the delay _t_  and some delay is virtually certain. The two equation models are not therefore fully comparable, and the intuition afforded by the one cannot be applied wholesale to the other.

While the potential triggering effect of a provisional submission cannot overturn the main result of this section, it can greatly reduce the superiority of the 0 plan (  , ( )) over a single, final submission. Observe that the option value arises

> 11. In particular instances, the decision will indeed be triggered, with an attendant adverse effect; but the claim of superiority relates to a whole series of “trials.” The appeal to the Arrow-Lind theorem is important here.
from the second and third terms within braces in equation (9). Now suppose D’s behavior tends to be of the hair-trigger kind, so that the triggering effect approaches a near certainty even if R attempts to keep it at bay by making a 1 provisional submission soon after _t_ . Since the remaining terms within braces on the right-hand side of equation (9) approach zero as  1 approaches unity, the claim follows.

More generally, all of the above considerations duly appear in the first-order condition for _P_  ( , 0 ( )) to take a maximum, whose baroque details need not be laid out here.<sup>12</sup> Suffice it to say that the said condition, together with condition (8), 0 0 0 characterizes the optimal plan (  ,  ( )) .

Models of this kind are best thought of as a means of ordering one’s ideas through a private conversation, with the model as partner. No one in her right mind is going to set about calculating the optimal plan (  0,  0 ( )) on the lines set out above: researchers have better things to do with their time. Yet the analysis points to conclusions of practical importance. In order to exploit fully the benefits of a provisional submission, and to minimize the danger of incorrect decisions, researchers need an appropriate framework of incentives, policy makers (and donors) need an adequate quality-control mechanism, and both parties need some mechanism for encouraging communication. Institutional support in these areas would surely help all parties to allocate their time and energies more efficiently where promoting better decisions is concerned. These themes will be pursued in the next section.

# IV PROVISIONAL ASSESSMENTS IN PRACTICE

12. They are available from the authors on request.
The first part of section III demonstrated formally the rather obvious, albeit routinely ignored, point that delaying a single submission until the underlying research has been refined to the fullest possible extent is unlikely to be a wise strategy, since there are very good chances that the policy maker will have acted before the advice arrives. If this happens, his decision will not be informed by any specific guidance from the research community, and the resources devoted to the research will be wasted—at least from an immediate policy-making perspective. In these circumstances, an alternative approach, in the form of an early, provisional submission with the promise of a more refined analysis to follow, may well be preferable. This thought led to the analysis in the second part of section III and thence to the key result that the said alternative is in general superior to a single, final submission.

Several factors caution against concluding that this result points to a simple way of improving the impact of research on policy. First, as noted in the Introduction, the research-policy interaction is a messy business and decidedly context-specific. The analytical framework presented above necessarily abstracts from this complexity and captures only selected aspects of reality. Second, and also noted in the Introduction, the incentives currently confronting researchers deter provisional submissions and favor more polished outputs aimed at refereed journals. Third, the key result—that a provisional submission, coupled with the offer of a more thorough analysis, is superior to a single submission—holds only in a probabilistic sense: it is correct on average over a long run of trials, but not necessarily in each case. We address these three points in turn. We begin with some comments on the analytical framework’s value added and its applicability in a wider range of circumstances than may be apparent at first sight. We then discuss how incentives may be changed to promote greater reliance on provisional assessments. Finally, we outline some measures to increase the likelihood of a
positive outcome and counter the risk of an early and incorrect decision based on faulty analysis and evidence.

# _Value Added and Applicability_

The model in section III was designed specifically to illustrate the trade-off between timeliness and rigor and to allow an exploration of the merits of provisional assessments. The intention was to cut away the tangle of real-world complexity and context-specificity to expose the essential tension between meeting professional research standards and serving policy makers in a timely manner. This analytical approach departs from the reliance on case studies that has to date dominated research on the research-policy link. The strength of case studies lies in their ability to delve into the rich detail of specific examples, but they are not easily generalizable and usually lack analytical rigor. The model presented above builds on, and brings analytical confirmation to, the results of existing, casespecific research by demonstrating analytically the general merit of two findings that have emerged from observation in a variety of different circumstances: first, successful research-policy episodes are often characterized by early interaction between researcher and policy maker; and second, short, focused inputs are usually the most effective means of communication. The value added of the model lies primarily in its rigorous demonstration that these two elements, when combined in what we have called provisional assessments, do indeed yield outcomes superior on average to those yielded by the traditional approach of waiting until the research is complete in all respects before sharing results with policy makers.

The framework also offers an approach to thinking about the interaction between policy-makers and researchers in an orderly way. For it is flexible enough to be more widely applicable than its formal structure might at first suggest. For example, the initial exchange between researcher and policy maker could lead to
several rounds during the decision process rather than the single exchange assumed in the model. Likewise, the researcher-led version in section III could be reversed to allow for an opening approach from the policy maker. Or the provisional assessment could be initiated after the main research study has begun if there is still scope for adjusting the delivery date of its results.<sup>13</sup> There is a place for other agents, too. Thus, if the policy maker has little interest in research and ignores the provisional assessment, the researcher could make her preliminary results available to the media and NGOs in the hope of influencing the decision by arousing public opinion.

It should also be remarked that the nature of the decision to be taken, as formulated in section III, is not especially restrictive. Instead of a straightforward yes-no decision on a single project or program, the set of choices can be extended to an array of specific, competing alternatives, in addition to the unspecified alternative, in any cost-benefit analysis that corresponds to rejecting the single proposal if its net present value at shadow prices is negative. To give examples, the tendering process often yields different designs of the same project, be it a port or a primary school. Likewise, a policy intervention may allow different sequences _ex ante_ . The timing of the analysis can also vary. Thus, a program or policy that stalls during implementation may present the policy maker with a cancel-or-redesign decision that could benefit from a provisional assessment.<sup>14</sup> Similarly, but earlier in the project cycle, the analysis could be applied to a pilot project. Indeed, any specific decision that can be subjected to empirical analysis involving standard economic tools is a possible candidate.

> 13. In the event that the provisional assessment convincingly questions the merit of the program or policy, then the research study should be abandoned. This is a risk, but the common-sense message here is—undertake a provisional assessment before embarking on a major research exercise. 14. Variants of the project in question may suggest themselves, whether from experience or even institutionalized learning in the course of execution itself, as advocated vigorously by Pritchett et al. (2012).
# _Leveling the Playing Field_

As argued in the Introduction, the ruling incentives within the economics profession encourage researchers to refine and verify their analyses as fully as possible before sharing their results with policy makers. It follows that if the aim is now to promote greater use of provisional assessments, it will be necessary to modify researchers’ behavior by means of new incentives specifically designed to counter existing ones.

The obvious way of leveling the playing field is to subsidize provisional assessments. Yet it is unlikely that policy makers in developing countries would be willing to pay for these before a solid track-record has been established. Consequently, a third party—the obvious candidate is the international donor community, which is already an important funder of research in developing countries—has to be the source of the subsidy, at least initially. The payment per submission would have to motivate a significant number of researchers to participate and would therefore have to exceed the marginal cost of delivering the input. The cost of producing a preliminary assessment, given its nature, is likely to be small. Even with some additional premium to guarantee adequate participation, therefore, the total cost of the subsidy would remain modest. The appropriate size of this premium could be determined through experimentation, with donors adjusting the amount until sufficient numbers of provisional assessments are tendered.

To ensure that the subsidy does indeed result in an early submission to the policy maker, donors could undertake two specific actions. First, they could process requests for funding, and disburse funds, quickly. Second, the initial grant could be limited to the costs of the study, with the premium paid only on confirmation that the provisional assessment has been delivered to the policy maker within some reasonable, and precisely specified, time frame.
Additionally, and to encourage interaction between researcher and policymaker, donors could give priority to those proposals for which the researcher can produce a written endorsement from the relevant policy maker<sup>15</sup> or where the policy maker initiates the process himself. Such expressions of interest would demonstrate both that contact between the parties has been established and that the policy maker can be expected to act on the analysis when it arrives. In some cases, a policy analyst inside a ministry may undertake the preliminary research for a provisional assessment. This attractive practice could be encouraged by extending the subsidy to government analysts wherever legal rules and bureaucratic procedures allow.

Whatever the origin of a research-policy interaction, if policy makers share their timetable with researchers following receipt of a provisional assessment, then the likelihood that any ensuing final submissions will arrive on time should increase significantly. Donors should be especially interested in promoting this outcome, since it ensures that research output at a more refined stage is actually feeding into the policy-making process. This highly desirable result could be encouraged by declaring that requests to fund full-scale research projects will receive preferential treatment if they have been preceded by a provisional assessment. The exact advantage—speedier turn-around or more generous funding—could be decided by individual donors; but whatever its form, the knowledge that funds are more likely to be forthcoming for a fully fledged research proposal provided a provisional assessment has been completed, will serve as a further and powerful inducement for researchers to interact with policy makers early in the research cycle.

Since time is of the essence, the administrative procedures for funding both

> 15. The International Initiative for Impact Evaluation uses this approach. Researchers are asked to include in any application for funds an endorsement from the relevant policy maker.
provisional assessments and full proposals that have been preceded by a provisional assessment should be streamlined, with quick turn-around for the former and almost automatic funding for the latter. This departure from the thorough vetting of full proposals should be balanced by a systematic effort to monitor each stage of the research-policy interaction from start to finish. This would provide reassurance that, simplified procedures notwithstanding, the approach is yielding the desired result in terms of the frequency with which policy makers move from provisional assessments as a first step, to final submissions as a contingent second step, and then to policy action, or otherwise indicate the need for fine-tuning. An attractive side-benefit of this monitoring program is that the information so generated would create a unique data base capable of expanding and deepening the profession’s understanding of the research-policy interaction well beyond that afforded by sporadic case studies.

# _Reducing the Risk of Errors_

The introduction of provisional assessments would improve policy making in the large; but, in specific cases, the advice may increase the probability that the wrong decision is made. The initial analysis, precisely because it is a first cut, may be so faulty as to point the policy maker in the wrong direction, whereas the alternative—waiting for the full analysis—would provide better guidance (assuming it arrives in time to be of use). The quality of provisional assessments is, therefore, an issue to be addressed.

Confidence in quality can be increased by means of an independent, professional review of each provisional assessment prior to its delivery to the policy maker.<sup>16</sup> Thus, researchers receiving grants to conduct preliminary

> 16. This point is not confined to provisional assessments; it applies equally to all existing means of reaching the policy-maker (e.g., policy briefs), which are often delivered without review. In their effort to reach policy-makers,
assessments could be required to submit their assessments to a review covering such basic aspects of their analysis as specification of the policy question, use of data, choice of analytical tool, and robustness checks. The reviewer, who should be of another nationality and drawn from a panel of experts, could also offer more general advice—on alternative designs, for example—drawn from both the published and the so-called grey international literature dealing with whatever program or policy is being evaluated. Making the premium to be paid to the researcher conditional on such a review would reduce the chances of unsatisfactory advice reaching the policy maker and adversely affecting policy decisions.

The drawback with this suggestion is, of course, that reviews inevitably take time, and yet timeliness is essential if provisional assessments are to fulfill their intended purpose. To minimize any hold-up, reviewers could be paid a fee per review, provided it is delivered within a specified time. The academic community, drawing on its well-established expertise in controlling quality by means of double-blind refereeing prior to publication, is well placed to contribute to this more specialized review process. Furthermore, compared with submissions to professional journals, provisional assessments are short, focused pieces and should therefore admit of a speedy review.

It is noteworthy that there are journals in other fields, such as medicine and branches of the natural sciences, that enforce a very rapid review, with accepted articles appearing in a matter of weeks after submission.<sup>17</sup> Such journals devoted to policy questions would serve our present purposes admirably; but there are none, and there seems little prospect of any unaided births of this kind. Donors might wish to act as midwives, but editors would have to be sure of keeping them at many development specialists fail to subject research-based advice to careful scrutiny before delivering it, when in fact, such research should probably receive more scrutiny than research submitted to a professional journal. Weiss, in her foreword to Carden (2009), makes this point when she rightly queries the latter's apparent belief that “research is always good and right.” 17. We are indebted to a referee for drawing this alternative to our attention.
arm’s length subsequently.

Even if adequate quality is assured, the pitfalls do not end with the submission of a provisional assessment. A policy maker may simply ignore its analysis and instead use its arrival as an excuse to move forward with a longpreferred, but quite possibly incorrect, decision. This is the worst sort of trigger effect: the decision is based on personal experience, a yearning for advancement, political pressure, or other factors quite unconnected with the research findings in question. One effective way of reducing this risk is to distinguish a priori those policy makers who are most likely to use a provisional assessment on its merits from those who are not.

The best, and perhaps only sensible, solution to this problem is to leave the tactical details to local researchers, who are far better acquainted with local policy makers’ inclinations and behavior. These researchers will have the model and the option of submitting a provisional assessment in the back of their minds, but how they actually conduct the business will depend on the particular policy maker in question. Given her unique knowledge, the local researcher is the one best placed to identify those policy makers most likely to act prudently upon receipt of a provisional assessment.

Having done so, her knowledge will be valuable at the next stage, if a more refined analysis is needed. For in practice, she will not be in the corset of the highly asymmetrically held information posited in section III. Rather she will likely have a prior, however fuzzy, regarding the form of what is termed the policy  maker’s “influence function” (the  ( ) of section III) and even some feeling for his view of the timing and quality of the decision (the _H_ ( _t_ ) and<sup>_q_</sup> _t_<sup>functions). These,</sup> too, should improve the probability of a correct decision.

The requirement that a researcher seek the relevant policy-maker’s
endorsement before receiving the premium will also sharpen her incentives. In effect, such a requirement functions as a method of “informed targeting,” one that relies on local-researchers’ knowledge regarding which local policy makers to approach and how best to conduct business with them if asked. In like manner, only more so, interactions initiated by policy makers or policy analysts can be thought of as stemming from a form of self-selection that can be expected to lead more often than not to better decisions. It is difficult to know how effective these methods of targeting may be; but even if they fail on occasion, the researcher always has recourse to the media and public opinion to exert pressure on politicians who misuse or ignore the findings of research.

As noted in section III, the possibility of a changing of the guard will be on both parties’ minds, and it will surely crop up in any discussion between them. An attraction of our proposed scheme is that it provides some measure of protection against this particular hazard in the policy-making process. If they see eye to eye on the likelihood of this event and it is not on the distant horizon, there will be an additional incentive to deliver a provisional assessment in short order, in the hope of tying the hands of any new regime, however the assessment turns out. Yet this potential advantage comes with a drawback; for if R senses that her view of the social profitability of the project is likely to be rather different from D’s, she may delay matters, precisely in the hope of a changing of the guard. Such strategic behavior by both parties cannot be ruled out.

# V CONCLUDING REMARKS

Researchers, policy makers, and donors are constantly seeking ways to improve the impact of research on policy. This article offers a practical way of contributing to this goal by encouraging the delivery of more timely inputs to the decision maker in the form of provisional assessments. This proposal draws on two
observations emerging from the considerable case-study material now available on the interaction between research and policy. The evidence suggests that early interaction and short, policy-focused inputs are often associated with successful outcomes. Our call for a much greater reliance on provisional assessments builds on, and is fully consistent with, these observations. By establishing analytically the superiority, on average, of a provisional assessment, accompanied by the offer of a more refined analysis, over a single, final submission, this article strengthens the argument in favor of early and focused policy inputs. In this instance, the happy outcome is that both case-study evidence and analytical rigor point in the same direction.

The second, related contribution of the article is to address the chief difficulties of implementing such a scheme. Provisional assessments are not widely used at present, so there is little experience to go on. It is argued that prevailing incentives are the main cause, and that inexpensive donor-funded subsidies are the right remedy to get things started. Ensuring quality is a related concern. To deal with it, the paper calls for a fast but rigorous independent review of all provisional assessments and the tying of funds for more thorough research proposals to the timely submission of a related provisional assessment. The reshaping of incentives coupled with quality control can be thought of as supply-side measures that should encourage more reliance on provisional assessments, as a demand-side response. This, we have argued, will improve the quality of decisions in the public sphere.

# 6 APPENDIX

_d_ It is assumed that on date _t_ , and no other, there may be a changing of the _d_ guard, and then with probability  . Under this assumption, equation (1) in the text becomes
If D has a poor opinion of the likely new regime or his own successor,<sup>_q_</sup> _td_<sup>will be</sup> markedly smaller than<sup>_q_</sup> _t_<sup>.</sup> R revises her priors _F_ ( _t_ ) and<sup>_p_</sup> _t_<sup>in analogous fashion. Note that her priors</sup> _t r_ and  _r_ can also differ from D’s. If the event in question occurs, equation (2) in the text becomes Equeation (3) now takes the form, compactly expressed in terms of<sup>_p_</sup> _t_ (<sup></sup> )<sup>and</sup> _r_<sup>_p_</sup> _t_ (<sup></sup> )<sup>,</sup> _r_ Note that substitution for<sup>_p_</sup> _t_ (<sup></sup> )<sup>and</sup><sup>_p_</sup> _t_ (<sup></sup> )<sup>from equations (2) and (11), respectively,</sup> will yield _P_  ( ) in extensive form. Like D, R will have her own views about how a changing of the guard will affect the timing and correctness of the decision.

# REFERENCES

- Annor-Amevor, E. A., and C. E. Afudego. 2012. “Simulating Alternatives for Increasing Basic Education Certificate Examination Pass Rate in Junior High Schools in Ghana,” mimeo, Integrated Social Development Centre, Ghana.

- Arrow, K. J., and R.C. Lind. 1971. “Uncertainty and the Value of Public Investment Decisions,” _American Economic Review_ , 60: 364–78.

- Cabinet Office. 1999a. “Modernising Government,” The Stationery Office, London.

- ———. 1999b. “Professional Policy Making for the Twenty First Century,” Report by Strategic Policy Making Team, London.
- Carden, F. 2009. _Knowledge to Policy: Making the Most of Development Research_ , Sage Publications India Pvt. Ltd, New Delhi.

- Court, J., I. Hovland, and J. Young. 2005. _Bridging Research and Policy in Development: Evidence and the Change Process._ London: ITDG.

- ’

- Elga, A. 2010. “Subjective Probabilities Should be Sharp,” _Philosopher s Imprint_ , 10 (5): 1–11.

- Dhaliwal, I., E. Duflo, R. Glennerster, and C. Tulloch. 2012. “Comparative CostEffectiveness Analysis to Inform Policy in Developing Countries: A General Framework with Applications for Education,” Conference Paper, Abdul Latif Jameel Poverty Action Lab, MIT.

- Jones, A. M., L. Squire, and R. Thomas. 2010. “Evaluating Innovative Health Programs,” _Health Economics_ 19, Supplement 1, September.

- Jones, N., and C. Walsh. 2006. “Policy Briefs as a Communication for Development Research,” Overseas Development Institute, London.

- Jones, N., H. Jones, and C. Walsh. 2008. “Political Science? Strengthening Science-Policy Dialogue in Developing Countries,” Overseas Development Institute, London.

- Livny, E., A. Mehendale, and A. Vanags. 2006. “Bridging the Research Policy Divide in Developing and Transition Countries: Analytical Lessons and Proposals for Action,” Global Development Network, mimeograph, November.

- Nutley, S, M., I. Walter, and H. T. O. Davies. 2007. _Using Evidence: How Research Can Inform Public Services_ , The Policy Press, Bristol.

- Overseas Development Institute. 2004. “Bridging Research and Policy in International Development,” ODI Briefing Paper, London.

- Pawson, R., 2006. _Evidence-Based Policy: A Realist Perspective_ , Sage Publications Ltd, London.

- Pritchett, L., S. Samji, and J. Hammer. 2012. “It’s All About MeE: Using Structured Experiential Learning (‘e’) to Crawl the Design Space,” Working Paper No. 2012/104, UNU-Wider Working Paper No. 2012/104, Helsinki.

- Pritchett, L., and J. Sandefur. 2013. “Context Matters for Size: Why External Validity Claims and Development Practice Don’t Mix,” Working Paper 336, Center For Global Development, Washington, DC.

- Savedoff, W. D., R. Levine, and N. Birdsall. 2006. “When Will We Ever Learn? Improving Lives Through Impact Evaluation,” Center For Global
Development, Washington, DC.

Thornton, R. L., L. E. Hatt, E. M. Field, M. Islam, F. S. Dia, and M. A. Gonzalez. 2010. “Social security health insurance for the informal sector in Nicaragua: A randomized evaluation,” _Health Economics_ 19, Supplement 1, September.