According to OMB’s 2013 memorandum on the Next Steps in the Evidence and Innovation Agenda …evidence includes evaluation results, performance measures, and other relevant data analytics and research studies, with a preference for high-quality experimental and quasi-experimental studies. Since 2010, OMB has used its annual budget guidance to encourage federal agencies to use evidence to consider the effectiveness of their programs and to institutionalize the use of evidence to foster innovation rooted in research and rigorous evaluation. In a 2013 memorandum, OMB identified areas in which agencies could improve their use of evidence (see sidebar), including (1) strengthening evaluation capacity and proposing new evaluations; (2) developing high-quality, low-cost evaluations and rapid, iterative experimentation; (3) using innovative outcome-focused grant design; and (4) increasing agency capacity to use evidence. In 2015, OMB directed agencies to provide an overview of evidence-building strategies and identify related priorities with their fiscal year 2017 budget submissions. To incorporate evidence into grantmaking, the White House and OMB have encouraged agencies to use tiered evidence grants, which link funds to an evidence framework. When applying for grants, grantees demonstrate the strength of evidence on their service models and agencies award funding based on the level of evidence (see figure 1). Smaller awards are used to test new and innovative service models; larger awards are used to scale service models with strong evidence. Grantees generally evaluate their service model during the grant period. As a result, tiered-evidence grant programs have a goal of identifying evidence-based service models that can be replicated. According to an OMB memorandum on evidence and innovation, the goal is that, over time, service models move up tiers as evidence becomes stronger. OMB staff said that other goals of tiered evidence grants include devoting more resources to service models with stronger evidence, allowing for testing of innovations, and encouraging rigorous evaluation. From fiscal year 2010 to 2015, agencies have awarded approximately $4.1 billion to the tiered evidence grants we reviewed. As shown in figure 2, each agency uses its own names and descriptions for its evidence tiers. While Education, DOL, and CNCS use a three-tier model, HHS uses a two-tiered model, where the agency makes a larger overall investment in replicating evidence-based practices by awarding more grants in its high tier. For example, HHS awards 75 percent of funds to replicating service models with moderate or strong evidence for the Teen Pregnancy Prevention Program, and awards 25 percent of funds to rigorously evaluate promising approaches. While there is substantial variation among grant types, competitively awarded federal grants generally follow a life cycle of pre-award (announcement and application), award, implementation, and closeout. To implement tiered evidence grants, the four agencies in our review add evidence and evaluation requirements throughout the federal grant life cycle. These added requirements include: (1) assessing the evidence base and identifying evidence-based approaches, (2) implementing evidence-based approaches with fidelity, (3) conducting independent evaluations, and (4) disseminating evaluation results, as shown in figure 3. The four federal agencies in our review had varied approaches to identify evidence-based approaches for tiered evidence grants. Education, DOL, and CNCS first developed definitions of each evidence tier that grantees must meet to qualify for a grant award as well as the program priorities for the grant competition. Evidence definitions included requirements, such as the type and number of studies on the service models, as well as the scale of service models studied. The rigor and scale of the evidence requirements increased from tier to tier. For example, Preliminary evidence tier. CNCS’s Social Innovation Fund’s preliminary evidence tier was designed for service models that have preliminary evidence based on a reasonable hypothesis supported by credible research findings. Moderate evidence tier. Education’s Investing in Innovation Fund’s moderate evidence tier required at least one experimental study of effectiveness of the grantees’ chosen service model, or one quasi- experimental study that was conducted on a large and multi-site sample. Grantees could validate their service model by testing the service model in a new location, testing it on a new or bigger population, or tweaking an aspect of their service to see if it still works on their target population. Strong evidence tier. DOL’s Workforce Innovation Fund’s strong evidence tier was designed to support significant expansion of service models that had previously demonstrated strong evidence of positive results and will be evaluated using a randomized controlled trial evaluation. For HHS, the evidence tiers for the Teen Pregnancy Prevention Program are defined in statute. For example, in the appropriation providing funding for the Teen Pregnancy Prevention Program for fiscal year 2016, 75 percent of amounts provided for the grant were to be used for a strong evidence tier to replicate programs that have been proven effective through rigorous evaluation to reduce teenage pregnancy, and behavioral risk factors underlying teenage pregnancy. The other 25 percent were to be used for a preliminary evidence tier to conduct research and demonstration projects to develop and test additional models and strategies. After the four agencies in our review defined the evidence tiers, they used the standards to assess the available evidence on the service models. The agencies varied in terms of when they assessed the evidence: Before announcement of grant opportunity. For the Teen Pregnancy Prevention Program and the Federal Home Visiting Program, HHS reviewed the available evidence on teen pregnancy prevention and home visiting models before it announced the grant opportunities and identified a list of approved evidence-based models. Tier one (strong evidence tier) grantees that replicated evidence-based models selected an evidence- based model from the approved list that met the needs of their target populations when applying for the grant. As of July 2016, the Teen Pregnancy Prevention Program included 44 service models and the Federal Home Visiting Program included 17 service models. HHS periodically updates the approved models. During review of grant applications. Grantees of DOL’s Workforce Innovation Fund and Education’s Investing in Innovation Fund identified in their grant applications the evidence-based service models they planned to use. DOL and Education officials assessed the evidence that supported the service models when they reviewed the grant applications and make award decisions. For example, Education’s Institute of Educational Sciences’ What Works Clearinghouse—the entity within Education that is focused on research and evaluation—reviewed the citations of the studies that supported Investing in Innovation Fund grant applications to determine whether they met the evidence requirements. After grant award is made. For the Social Innovation Fund, CNCS made awards to intermediary organizations, which then made awards to subgrantees to implement service models. CNCS provided the intermediary organizations with guidance on the levels of evidence, which they used to assess the evidence for their subawards. CNCS reviewed and confirmed the intermediaries’ decisions before they made the final awards. Grantees were responsible for delivering their evidence-based service models with fidelity—in a manner consistent with how the program was designed. For example, in its guidance to Teen Pregnancy Prevention Program grantees, HHS defined fidelity as the degree to which a program is implemented with adherence to its core components – the key ingredients related to achieving the outcomes associated with the program model. The core components included a model’s content and how it is delivered. Fidelity is important because delivering a service model as it was designed increases the likelihood that the participants served will experience similar positive outcomes to those found in the original evaluation study. The five grant programs we reviewed had evaluation requirements, but they generally had two different evaluation approaches. For DOL, Education, and CNCS tiered evidence grants, every grantee was required to contract with a third party evaluator to conduct an independent evaluation of its service model and submit it to the respective federal grantmaking agency at the end of the grant period. DOL also commissioned a crosscutting analysis of the findings from, and experiences with, the evaluations of the projects under the Workforce Innovation Fund. For HHS, the Federal Home Visiting and Teen and Pregnancy Prevention Programs used different evaluation approaches for each tier that included both grantee and federal government-led evaluations. For the preliminary evidence tier to test promising approaches of both grant programs, grantees were required to conduct an independent evaluation. For the strong evidence tier of the Federal Home Visiting Program, grantees receiving competitive awards or tribal grants were also required to conduct grantee-led evaluations.Also, state formula grantees are encouraged to conduct grantee-led evaluations. For the strong evidence tier to replicate evidence-based service models of the Teen Pregnancy Prevention Program, all grantees receiving a grant award of greater than $1 million were required to conduct an independent evaluation. HHS also is conducting programwide evaluations of both grant programs (see sidebar). When conducting independent evaluations, grantees we reviewed hired an independent evaluator to conduct evaluations. This work included developing an evaluation plan, collecting and analyzing data, and completing an evaluation report to communicate the service model results. The four federal agencies we reviewed contracted with a separate external national evaluation coordinator to provide technical assistance to the independent evaluators and grantees on their evaluations. The national evaluation coordinators developed guidance on evaluations, reviewed evaluation plans, provided ongoing assistance on conducting the evaluations, and provided feedback on the final reports. The four agencies we reviewed encouraged grantees to conduct rigorous evaluations, such as randomized controlled trials or studies using quasi-experimental designs. We have previously reported that when developing an evaluation design, an evaluator should select appropriate measures and comparisons that will permit valid conclusions. The evaluator should explore a variety of options available to collecting and analyzing information, and choose alternatives that will best address the evaluation objectives within available resources. During and after the grant closeout stage, the four agencies we reviewed disseminated the evaluation results to various audiences. For example, Education and CNCS posted links to completed evaluations from the Investing in Innovation Fund and the Social Innovation Fund, respectively, online.