Federal agencies are increasingly expected to demonstrate effectiveness in achieving agency or governmentwide goals. The Government Performance and Results Act of 1993 (GPRA) requires federal agencies to report annually on their progress in achieving agency and program goals. The President’s Budget and Performance Integration initiative extends GPRA’s efforts to improve government performance and accountability by bringing performance information more directly into the budgeting process. In developing the fiscal year 2004 budget, OMB (1) asked agencies to more directly link expected performance with requested program activity funding levels and (2) prepared effectiveness ratings, with a newly devised Program Assessment Rating Tool (PART), for about one-fifth of federal programs. The PART consists of a standard set of questions that OMB and agency staff complete together, drawing on available performance and evaluation information. The PART questions assess the clarity of program design and strategic planning and rate agency management and program performance. The PART asks, for example, whether program long-term goals are specific, ambitious, and focused on outcomes, and whether annual goals demonstrate progress toward achieving long-term goals. It also asks whether the program has achieved its annual performance goals and demonstrated progress toward its long-term goals. Ratings are designed to be evidence-based, drawing on a wide array of information, including authorizing legislation, GPRA strategic plans and performance plans and reports, financial statements, Inspector General and our reports, and independent program evaluations. Almost a decade after GPRA was enacted, the accuracy and quality of evaluation information necessary to make the judgments called for in rating programs is highly uneven across the federal government. GPRA expanded the supply of results-oriented performance information generated by federal agencies. However, in the 2004 budget, OMB rated 50 percent of the programs evaluated as “Results Not Demonstrated” because they did not have adequate performance goals or had not collected data to produce evidence of results. We have noted that agencies have had difficulty assessing (1) many program outcomes that are not quickly achieved or readily observed and (2) contributions to outcomes that are only partly influenced by federal funds. To help explain the linkages between program activities, outputs and outcomes, a program evaluation—depending on its focus—may review aspects of program operations or factors in the program environment. In impact evaluation, scientific research methods are used to establish a causal connection between program activities and outcomes and to isolate the program’s contributions to them. Our previous work raised concerns about the capacity of federal agencies to produce evaluations of program effectiveness. Few deployed the rigorous research methods required to attribute changes in underlying outcomes to program activities. Yet, we have also seen how some agencies have profitably drawn on systematic program evaluations to explain the reasons for program performance and identify strategies for improvement. To identify ways that agencies can improve evaluation capacity, we conducted case studies of how five agencies had built evaluation capacity over time. To select the cases, we reviewed departmental and agency performance plans and reports, as well as evaluation reports, for examples of how agency performance reports had incorporated evaluation results. To obtain a broadly applicable set of strategies, we selected cases to reflect a diversity of federal program purposes. Because program purpose is central to considering how to evaluate effectiveness or worth, the type of evaluation an agency conducts might shape the key elements of the agency’s evaluation capacity. For this review, we selected cases based on a classification of program purposes employed in our previous studydemonstration, regulation, research, and service delivery. The first three classifications are represented in our case selection of ACF, NHTSA, and NSF. For service delivery, we chose one agency that delivers services directly to the public (the Coast Guard), and another that provides services through third parties (HUD). Although we selected cases to capture a diversity of federal program experiences, the cases should not be considered to represent all the challenges faced or strategies used. We describe all five cases in the next section. For each agency, to identify the key elements of evaluation capacity and strategies used to build capacity, we reviewed agency and program materials and interviewed agency officials. Our findings are limited to the examples reviewed and do not necessarily reflect the full scope of each agency’s evaluation activities. For example, we did not review all HUD evaluations, only evaluations of flexible grant programs. We conducted our work between June 2002 and March 2003 in accordance with generally accepted government auditing standards. We requested comments on a draft of this report from the heads of the agencies responsible for the five cases. The Departments of Health and Human Services and Housing and Urban Development provided technical comments that we incorporated where appropriate throughout the report. We describe the program structures, major activities, and evaluation approaches for the five cases in this section. ACF, in the Department of Health and Human Services (HHS), oversees and helps finance programs to promote the economic and social well- being of families, individuals, and communities. Through the Temporary Assistance for Needy Families (TANF) program, ACF provides block grants to states so that they can develop programs of financial and other assistance. These programs help needy families find employment and economic self-sufficiency. In 1996, TANF replaced Aid to Families with Dependent Children (AFDC), commonly referred to as welfare, and the Job Opportunities and Basic Skills Training (JOBS) programs. Under the AFDC program, states conducted demonstrations, for three decades, to test out alternative approaches for moving recipients off welfare and into work. As part of a broad array of studies of poverty populations and programs, ACF and the Office of the Assistant Secretary for Planning and Evaluation (ASPE) continue to support evaluations of state welfare-to- work experiments, including implementation and process studies, as well as impact studies based on experimental evaluation methods. In the Department of Transportation (DOT), the Coast Guard provides diverse customer services to ensure safe and efficient marine transportation, protect national borders, enforce maritime laws and treaties, and protect natural resources. The Coast Guard’s mission includes enhancing mobility, by providing aids to navigation, icebreaking services, bridge administration, and vessel traffic management activities; security, through law enforcement and border control activities; and safety, through programs for accident prevention, response, and investigation. The agency monitors numerous indicators to assess allocation of resources to and performance in achieving service goals. The Coast Guard has initiated an effort to evaluate its direct services and resource-building efforts through a Readiness Management System, which covers people, equipment, and stations. In addition, special studies of the success of specific initiatives may be contracted out. The HUD Office of Community Planning and Development (CPD) provides financial and technical assistance to states and localities in order to promote community-based efforts to develop housing and economic opportunities. CPD’s largest program, the Community Development Block Grant program (CDBG) has, for the past two decades, provided formula grants to cities, urban counties, and states to foster decent, affordable housing, and expanded economic opportunities for low- and moderate- income people. Communities may use funds for a wide range of activities directed toward neighborhood revitalization, economic development, and improved community facilities and services. CPD also administers the HOME Investment Partnerships Program (HOME), a block grant to state and local governments, to create decent, affordable housing for low- income families. First funded in 1992, HOME has more specific goals than CDBG: (1) to help build, buy, or rehabilitate affordable housing for rent or home ownership or (2) to provide direct tenant-based rental assistance. In addition to maintaining information on housing need, market conditions, and programs across the department, HUD’s Office of Policy Development and Research (PD&R) supports studies of the use and benefits of the CDBG and HOME grants. To promote highway safety, DOT’s NHTSA develops regulations and provides financial and technical assistance to states and local communities. These communities, in turn, conduct highway safety programs that respond to local needs. To identify the most effective and efficient means to bring about safety improvements, NHTSA also conducts research and development in vehicle design and driver behavior. To assess the effectiveness of its regulatory and safety promotion efforts, NHTSA reviews outcomes, such as reduction of alcohol-related fatalities or increase in helmet or safety belt use. To illuminate the causes and outcomes of crashes and evaluate safety standards and initiatives, NHTSA analyzes state and specially created national databases, for example, the Fatality Analysis Reporting System (FARS). NSF funds education programs and a broad array of research projects in the physical, geological, biological, and social sciences; mathematics; computing; and engineering; which are expected to lead to innovative discoveries. NSF provides support for investigator-initiated research proposals that are competitively selected, based on merit reviews. The agency has a long-standing review infrastructure in place: for each individual research program, panels of outside experts rank proposals on merit. NSF also convenes panels of independent experts as external advisers—a Committee of Visitors (COV)to peer review the technical and managerial stewardship of a specific program or cluster of programs periodically, compare plans with progress made, and evaluate outcomes to determine whether the research contributes to NSF mission and goals. Each COV, based on an academic peer review model, usually consists of 5 to 20 external experts, who represent academia, industry, government, and the public sector. These reviews serve as a means of quality assurance for NSF management. About a third of the 220 NSF programs are evaluated each year so that a complete assessment of programs can be accomplished over a 3-year period. Four main elements of evaluation capacity were apparent across the diverse array of agencies we reviewed, although they took varied forms. These elements include an evaluation culture, data quality, analytic expertise, and collaborative partnerships. (See figure 1.) Agencies demonstrated an evaluation culture through commitment to self- examination and learning through experimentation. Data quality and analytic expertise were key to ensuring the credibility of evaluation results and conclusions.