Project Narrative The project narrative (recommended length: no more than 25 pages) for an Efficacy/Replication project application must include four sections: Significance, Research Plan, Personnel, and Resources. Significance – The purpose of this section is to explain why it is important to test the impact of the intervention on student education outcomes under the proposed conditions and sample. Requirements: In order to be responsive and sent forward for scientific peer review, applications under the Efficacy/Replication goal must describe The intervention to be evaluated and For a Replication study or a Follow-up study, the evidence from the original Efficacy study. Recommendations for a Strong Application: In order to address the above requirements, the Institute recommends that you include the following in your Significance section to provide a compelling rationale for the proposed Efficacy/Replication work. Note the type of study proposed (Efficacy, Replication, Follow-Up, or Retrospective) early in the Significance section. Include the following in your description of the fully developed intervention that you propose to evaluate: The intervention’s components; Processes and materials (e.g., manuals, websites, training, coaching) that will be used to support implementation of the intervention; and Evidence that the intervention is fully developed and ready for implementation in authentic education settings (e.g., all materials and implementation supports such as professional development are available). Applications to evaluate newly developed and non-widely used interventions often require more of this type of evidence than those evaluating widely used interventions. If the intervention you wish to test and/or its implementation processes and materials are not yet fully developed, you should apply under Development/Innovation to complete it. Describe the intervention’s context: Identify the target population and where implementation will take place. Identify who the end users of the intervention are and describe how implementation will be carried out by them. Describe the ideal or routine conditions under which the intervention will be implemented. Ideal conditions provide a more controlled setting under which the intervention may be more likely to have beneficial impacts. For example, ideal conditions could include more implementation support than would
the design and how conclusions from the research will be tempered in light of these threats. Because quasi-experimental designs can meet the WWC’s standards for evidence with reservations only, it is also important to detail how you will ensure that the study will meet these standards (e.g., by establishing baseline equivalence between treatment and comparison groups and preventing high and/or attrition). Describe and justify the counterfactual. In evaluations of education interventions, individuals in the comparison group typically receive some kind of treatment. It may be a well-defined alternative treatment or a less well-defined standard or frequent practice across the district or region. A clear description of the intervention and the counterfactual helps reviewers decide whether the intervention is sufficiently different from what the comparison group receives to produce different student education outcomes. Describe strategies or existing conditions that will reduce potential contamination between treatment and comparison groups. Discuss how your study, if well implemented, will meet WWC evidence standards (with or without reservations). Power Analysis: Discuss the statistical power of the research design to detect a reasonably expected and minimally important effect of the intervention on the focal student education outcomes and consider how the clustering of participants (e.g., students in classrooms and/or schools) will affect statistical power. Identify the minimum effect of the program or policy that you will be able to detect, justify why this level of effect would be expected, and explain why this would be a practically important effect. Detail the procedure used to calculate either the power for detecting the minimum effect or the minimum detectable effect size. Include the following: The statistical formula you used; The parameters with known values used in the formula (e.g., number of clusters, number of participants within the clusters); The parameters whose values are estimated and how those estimates were made (e.g., intraclass correlations, role of covariates); Other aspects of the design and how they may affect power (e.g., stratified sampling/blocking, repeated observations); and Predicted attrition and how it was addressed in the analysis. Provide a similar discussion regarding power for any causal analyses to be done using subgroups of the proposed sample and any tests of mediation or moderation, even if those analyses are considered exploratory/secondary.
Discuss how exclusion from testing and missing data will be handled in your analysis. If you intend to link multiple data sets, provide sufficient detail for reviewers to judge the feasibility of the linking plan. Cost Analysis: Include a description of your plan to conduct a cost analysis. The cost analysis should help schools and districts understand the monetary of implementing the intervention (e.g., expenditures for personnel, facilities, equipment, materials, training, and other relevant inputs). Annual costs should be assessed to adequately reflect expenditures across the lifespan of the program (e.g., start-up costs and maintenance costs). Intervention costs can be contrasted with the costs of comparison group practice to reflect the difference between them. The Institute is not asking for an economic evaluation of the program (e.g., cost-benefit, cost-utility, or cost-effectiveness analyses), although such analyses can be proposed. In your plan, you should include information about the following: how you will identify all potential expenditures; how you will compute per-unit costs for each expenditure; how you will separate start-up costs from annual maintenance costs and how you will estimate the total cost of each; and the degree to which your cost analysis, based on your study’s sample, will generalize to other schools and districts. Retrospective studies and follow-up studies of students who received an intervention may, but are not required to, include a plan to conduct a cost analysis. If information about implementation cost is available, the inclusion of a plan to analyze those costs strengthens the application. Timeline: Provide a timeline for each step in your evaluation, including such actions as sample selection and assignment, baseline data collection, intervention implementation, ongoing data collections, fidelity of implementation and comparison group practice study, impact analysis, and dissemination. Indicate procedures to guard against bias entering into the data collection process (e.g., pretests occurring after the intervention has been implemented or differential timing of assessments for treatment and control groups). Timelines may be placed in either the Project Narrative or but may only be discussed in the Project Narrative. Personnel – The purpose of this section is to describe the relevant expertise of your research team, the responsibilities of each team member, and each team member’s time commitments. Requirements: In order to be responsive and sent forward for scientific peer review, applications under the Efficacy/Replication goal must describe The research team.