Through its development and use of PART, OMB has more explicitly infused performance information into the budget formulation process; increased the attention paid to evaluation and to performance information; and ultimately, we hope, increased the value of this information to decision makers and other stakeholders. By linking performance information to the budget process, OMB has provided agencies with a powerful incentive for improving both the quality and availability of performance information. The level of effort and involvement by senior OMB officials and staff clearly signals the importance of this strategy in meeting the priorities outlined in the PMA. OMB should be credited with opening up for scrutiny—and potential criticism—its review of key areas of federal program performance and then making its assessments available to a potentially wider audience through its Web site. As OMB and others recognize, performance is not the only factor in funding decisions. Determining priorities—including funding priorities—is a function of competing values and interests. Accordingly, we found that while PART scores were generally positively related to proposed funding changes in discretionary programs, the scores did not automatically determine funding changes. That is, for some programs rated “effective” or “moderately effective” OMB recommended funding decreases, while for several programs judged to be “ineffective” OMB recommended additional funding in the President’s budget request with which to implement changes. In fact, the more important role of PART was not its use in making resource decisions, but in its support for recommendations to improve program design, assessment, and management. As shown in figure 1, we found that 82 percent of PART’s recommendations addressed program assessment, design, and management issues; only 18 percent of the recommendations had a direct link to funding matters. OMB’s ability to use PART to identify and address future program improvements and measure progress—a major purpose of PART—depends on its ability to oversee the implementation of PART recommendations. As OMB has recognized, following through on these recommendations is essential for improving program performance and ensuring accountability. Currently, OMB plans to assess an additional 20 percent of all federal programs annually. As the number of recommendations from previous years’ evaluations grows, a system for monitoring their implementation will become more critical. However, OMB does not have a centralized system to oversee the implementation of such recommendations or evaluate their effectiveness. The goal of PART is to evaluate programs systematically, consistently, and transparently. OMB went to great lengths to encourage consistent application of PART in the evaluation of government programs, including pilot testing the instrument, issuing detailed guidance, and conducting consistency reviews. Although there is undoubtedly room for continued improvement, any tool is inherently limited in providing a single performance answer or judgment on complex federal programs with multiple goals. Performance measurement challenges in evaluating complex federal programs make it difficult to meaningfully interpret a bottom-line rating. OMB published both a single, bottom-line rating for PART results and individual section scores. It is these latter scores that are potentially more useful for identifying information gaps and program weaknesses. For example, one program that was rated “adequate” overall got high scores for purpose (80 percent) and planning (100 percent), but poor scores in being able to show results (39 percent) and in program management (46 percent). In a case like this, the individual section ratings provided a better understanding of areas needing improvement than the overall rating alone. In addition, bottom-line ratings may force raters to choose among several important, but disparate goals and encourage a determination of program effectiveness even when performance data are unavailable, the quality of those data is uneven, or they convey a mixed message on performance. Any tool that is sophisticated enough to take into account the complexity of the U.S. government will always require some interpretation and judgment. Therefore it is not surprising that OMB staff were not fully consistent in interpreting complex questions about agency goals and results. In addition, the limited availability of credible evidence on program results also constrained OMB’s ability to use PART to rate programs’ effectiveness. Many PART questions contain subjective terms that are open to interpretation. Examples include terminology such as “ambitious” in describing sought-after performance measures. Because the appropriateness of a performance measure depends on the program’s purpose, and because program purposes can vary immensely, an ambitious goal for one program might be unrealistic for a similar but more narrowly defined program. Without further guidance, it is unclear how OMB staff can be expected to be consistent. We found inconsistencies in how the definition of acceptable performance measures was applied. Our review surfaced several instances in which OMB staff inconsistently defined appropriate measures—outcome versus output—for programs. Agency officials also told us that OMB staff used different standards to define measures as outcome-oriented. Outputs are the products and services delivered by the program whereas outcomes refer to the results of outputs. For example, in the employment and training area, OMB accepted short-term outcomes, such as obtaining high school diplomas or employment, as a proxy for long-term goals for the Department of Health and Human Services’ Refugee Assistance program, which aims to help refugees attain economic self-sufficiency as soon as possible. However, OMB did not accept the same employment rate measure as a proxy for long-term goals for the Department of Education’s Vocational Rehabilitation program because it had not set long-term targets beyond a couple of years. In other words, although neither program contained long- term outcomes, such as participants gaining economic self-sufficiency, OMB accepted short-term outcomes in one instance but not the other. The yes/no format employed throughout most of the PART questionnaire resulted in oversimplified answers to some questions. Although OMB believes it helped standardization, the yes/no format was particularly troublesome for questions containing multiple criteria for a “yes” answer. Agency officials have commented that the yes/no format is a crude reflection of reality, in which progress in planning, management, or results is more likely to resemble a continuum than an on/off switch. We found several instances in which some OMB staff gave a “yes” answer for successfully achieving some but not all of the multiple criteria, while others gave a “no” answer when presented with a similar situation. For example, OMB judged the Department of the Interior’s (DOI) Water Reuse and Recycling program “no” on whether a program has a limited number of ambitious, long-term performance goals, noting that although DOI set a long-term goal of 500,000 acre-feet per year of reclaimed water, it failed to establish a time frame for when it would reach the target. However, OMB judged the Department of Agriculture’s and DOI’s Wildland Fire programs “yes” on this question even though the programs’ long-term goals of improved conditions in high-priority forest acres are not accompanied by specific time frames. The lack of program performance information also creates challenges in effectively measuring program performance. According to OMB, about half of the programs assessed for fiscal year 2004 lacked “specific, ambitious long-term performance goals that focus on outcomes” and nearly 40 percent lacked sufficient “independent, quality evaluations.” Nearly 50 percent of programs assessed for fiscal year 2004 received ratings of “results not demonstrated” because OMB decided that program performance information, performance goals, or both were insufficient or inadequate. While the validity of these assessments may be subject to interpretation and debate, our previous work has raised concerns about the capacity of federal agencies to produce evaluations of program effectiveness as well as credible data. PART was designed for and is used in the executive branch budget preparation and review process. As a result, the goals and measures used in PART must meet OMB’s needs. By comparison, GPRA—the current statutory framework for strategic planning and reporting—is a broader process involving the development of strategic and performance goals and objectives to be reported in strategic and annual plans and reports. OMB said that GPRA plans were organized at too high a level to be meaningful for program-level budget analysis and management review. OMB acknowledges that GPRA was the starting point for PART, but as I will explain, it appears that OMB’s emphasis is shifting such that over time the performance measures developed for PART and used in the budget process may also come to drive agencies’ strategic planning processes. The fiscal year 2004 PART process came to be a parallel competing structure to the GPRA framework as a result of OMB’s desire to collect performance data that better align with budget decision units. OMB’s most recent Circular A-11 guidance clearly requires both that each agency submit a performance budget for fiscal year 2005 and that this should replace the annual GPRA performance plan. These performance budgets are to include information from the PART assessments, where available, including all performance goals used in the assessment of program performance done under the PART process. Until all programs have been assessed using PART, the performance budget will also include performance goals for agency programs that have not yet been assessed. OMB’s movement from GPRA to PART is further evident in the fiscal year 2005 PART guidance stating that while existing GPRA performance goals may be a starting point during the development of PART performance goals, the GPRA goals in agency GPRA documents are to be revised, as needed, to reflect OMB’s instructions for developing the PART performance goals. Lastly, this same guidance states that GPRA plans should be revised to include any new performance measures used in PART and that unnecessary measures should be deleted from GPRA plans. Although there is potential for complementary approaches to GPRA and PART, the following examples clearly illustrate the importance of carefully considering the implications of selecting a unit of analysis, including its impact on the availability of performance data. They also reveal some of the unresolved tensions between the President’s budget and performance initiative—a detailed budget perspective—and GPRA—a more strategic planning view.