B.5 appropriate, do the authors suggest what additional data could be collected and/or analysis undertaken to better answer the question posed or to answer additional questions the study raised? C Reviewer’s summary comments (use as much as needed) Guidelines for Rating Policy Research Reports Question Weak Strong A General A.1 Is the issue well-defined and the case for its policy importance effectively made? Hard to identify the issue under discussion, possibly because it is confused with others; or issue is stated but there is no attempt to explain why it merits public policy attention. Issue crisply and clearly defined and a cogent case for its policy importance and timeliness is made. A.2 Is the issue defined or structured in such a way that a clear hypothesis or researchable question is stated? Difficult-to-impossible to understand the specific question or hypothesis that is the research subject. The basic policy issue is expressed in a way that makes addressing it empirically straightforward and accessible to the reader. A.3 Are all the relevant aspects of the issue included for analysis? Author leaves out key point, e.g., the distribution of benefits or subsidies or the efficiency with which they are employed, All relevant elements are (It is not necessary that they all be covered in the paper butenough information should be
while focusing only on the total amount provided to fully understand the situation.) A.4 Are relevant previous studies on the in the country cited and built on?b No prior studies are cited. There is a good review of the prior studies and the advances that the current research makes over the prior is clearly articulated. A.5 Do the authors show knowledge of the relevant international studies on this topic?b Such studies are not mentioned. This study exhibits knowledge of the relevant literature and states or implies its influence on the current study. A.6 Has the right type of information and data been assembled to address the issue? If not, what was omitted that should have been included? Where sample data are employed, is the sample correctly drawn to be representative? Is it sufficiently large for the necessary tests? The selection of data seems arbitrary and not well-suited to the study. Where survey data are used, insufficient information is provided to judge its quality, or the information provided makes problems with the sample clear. The data employed are ideal for the study. Where survey data are used, the sample is and clearly appropriate for the task at hand. A.7 Are the methods employed appropriate? Are statistical used where needed? The authors do not employ the relevant statistical tests but rather just describe qualitatively the patterns in the data. Relevant statistical tests are used throughout. The author interprets the results of the tests effectively. A.8 Is the report well-organized and clearly and succinctly written? The report is very poorly structured, with little logic to the sequencing of the presentation. The writing style is very wordy or otherwise makes it hard for the reader to understand the argument being made and the information presented. Tables are poorly constructed and hard to understand without referring to the text. The report is well-organized and tightly written. The flow of language makes it easy to read. There are few extra words. The author exercises good judgment in allocating material to annexes. Tables are thoughtfully constructed and can be understood without referring to the text. B Conclusions B.1 Are the conclusions based squarely on the paper’s findings? (Or do the authors There is little relation between the analysis and the conclusion. For example, the author brings The conclusions are firmly based on the analysis. The findings’
go beyond the findings, in effect expressing personal views or political opinions?) in political considerations, e.g., income distribution, when this is not at all the subject of the analysis. Personal opinions are expressed. implications are carefully and fully drawn out. B.2 If the conclusions call for action through government programs, is the cost realistically estimated? Is the administrative feasibility and complexity of the program considered? Cost and administrative considerations are not covered. The author provides defensible estimates of the cost involved and realistically discusses the administrative issues involved. (The extent of detail necessary will vary with the objective of the study.) B.3 Do the authors consider various options for addressing the issue and the merits of each, or focus exclusively on a single approach? The authors focus on a single approach with little or no justification for its selection. Other options are not even acknowledged to exist. Relevant options are presented and criteria by which they should be judged are explicitly stated. The criteria are applied to the options and the superior one selected for recommendation. B.4 Are the authors careful not to make proposals that go beyond the study’s findings? The proposals are far too broad, extensive or otherwise beyond what the study’s results can defend. The recommendations are consistent with the specific findings of the If more general statements are made, they are fully labeled as not being based specifically on the study’s findings. B.5 Where appropriate, do the authors suggest what additional data could be collected and/or analysis undertaken to better answer the question posed or to answer additional questions the study raised? There is no treatment of these topics. It is either not appropriate to make such suggestions or the authors lay out how the data deficiencies they encountered could be remedied in the future.
C.2 Is the dimension of the problem, e.g., the share of children not attending classes, well-developed and based on credible sources and analysis? Essentially no are presented. There seems to be an assumption that everyone agrees that this is a problem and no further explanation is needed. are succinctly and effectively marshaled to document the problem. C.3 Are “next steps” defined, i.e., now that the problem is defined, who should take responsibility for addressing it in some way? Essentially no recommendations are made or if they are they are not based on concrete information. The “next steps” are clearly stated and the sequencing makes sense. Sufficient information about them is presented for the reader to believe he is making an informed opinion. C.4 Is the recommendation (C.3) sensible and well-justified? They are missing or appear to be merely the authors’ opinion. The steps are appropriate to the problem’s degree of urgency. D All Briefs D.1 Is the presentation engaging, i.e., is the reader motivated to continue to read? Not at all. The document is poorly written and organized and it is a to read. The draws the reader’s attention from the outset and maintains it. The language is lively and the layout engaging. D.2 Is the of presentation suitable for the policymaker or “intelligent layman”? No. The discussion is too technical. Statistical analysis is discussed that is unlikely to be understood by many. There is a lot of jargon. The document’s language and presentation is excellent. Jargon is minimized and all terms are explained. Graphics and tables are excellent. D.3 Is the presentation succinct, closely reasoned, and of the appropriate length? (1-10) The brief is too long, too wordy and presents more information than is needed to make the argument. Reading it is difficult and time consuming. The brief takes no more space that is really needed. The argument is tight and the right amount of information is provided. “A pleasure to read.”