GPRAMA requires OMB to coordinate with agencies to develop long- term, outcome-oriented federal government priority goals for a limited number of crosscutting policy areas and management improvement areas every 4 years. Furthermore, with the submission of the fiscal year 2013 budget, GPRAMA required OMB to identify a set of interim priority goals.interim CAP goals, 9 of which were related to crosscutting policy areas and 5 of which were management improvement goals. The President’s 2013 budget submission included a list of 14 The CAP Goal Leader. As required by GPRAMA, each of the interim CAP goals had a goal leader responsible for coordinating efforts to achieve each goal. CAP goal leaders were given flexibility in how to manage these efforts, and were encouraged by OMB to engage officials from contributing agencies by leveraging existing inter-agency working groups, policy committees, and councils. For information on the position of the goal leader and the interagency groups used to engage officials from agencies contributing to each interim CAP goal, see figure 1. For more information on the interagency groups used to engage agency officials in efforts related to each goal, see appendix III. According to OMB and PIC staff, because CAP goal leaders were responsible for managing efforts related to the achievement of the goals as part of a larger portfolio of responsibilities, staff from the PIC, OMB, and—in some cases—from agencies with project management responsibilities, provided additional capacity for coordinating interagency efforts and overseeing the process of collecting, analyzing, and reporting data. Specifically, PIC staff provided logistical support, assisting with the regular collection of data, updates to Performance.gov, and the development of CAP goal governance structures and working groups. They also provided support in the area of performance measurement and analysis. For example, PIC staff supported the Exports goal leader by informing discussions of how to measure the success and impact of export promotion efforts, providing expertise in the development and selection of appropriate performance measures, and assisting in the collection and analysis of relevant data. Progress Reviews. GPRAMA also requires that the Director of OMB, with the support of the PIC, review progress towards each CAP goal with the appropriate lead government official at least quarterly. Specifically, the law requires that these should include a review of progress during the most recent quarter, overall trends, and the likelihood of meeting the planned level of performance. As part of these reviews, OMB is to assess whether relevant agencies, organizations, program activities, regulations, tax expenditures, policies, and other activities are contributing as planned to the goal. The law also requires that OMB categorize the goals by risk of not achieving the planned level of performance and, for those at greatest risk of not meeting the planned level of performance, identify strategies for performance improvement. In an earlier evaluation of the implementation of quarterly performance reviews at the agency level, we found that regular, in-person review meetings provide a critical opportunity for leaders to use current data and information to analyze performance, provide feedback to managers and staff, follow up on previous decisions or commitments, learn from efforts to improve performance, and identify and solve performance problems. As part of this work we also identified nine leading practices that can be used to promote successful performance reviews at the federal level. To identify these practices, we conducted a review of relevant academic and policy literature, including our previous reports. We refined these practices with additional information obtained from practitioners at the local, state, and federal level who shared their experiences and lessons learned. Nine Leading Practices That Can Be Used to Promote Successful Performance Reviews Leaders use data-driven reviews as a leadership strategy to drive performance improvement. Key players attend reviews to facilitate problem solving. Reviews ensure alignment between goals, program activities, and resources. Leaders hold managers accountable for diagnosing performance problems and identifying strategies for improvement. There is capacity to collect accurate, useful, and timely performance data. Staff have skills to analyze and clearly communicate complex data for decision making. Rigorous preparations enable meaningful performance discussions. Reviews are conducted on a frequent and regularly scheduled basis. Participants engage in rigorous and sustained follow-up on issues identified during reviews. Reporting Requirements. In addition to requiring quarterly reviews, GPRAMA requires that OMB make information available on “a single website” (now known as Performance.gov) for each CAP goal on the results achieved during the most recent quarter, and overall trend data compared to the planned level of performance. In addition, information on Performance.gov is to include an assessment of whether relevant federal organizations, programs, and activities are contributing as planned, and, for those CAP goals at risk of not achieving the planned level of performance, information on strategies for performance improvement. New CAP Goals. As required by GPRAMA, in March 2014, OMB announced the creation of a new set of CAP goals in the fiscal year 2015 budget. It then identified 15 CAP goals with 4-year time frames on Performance.gov—7 mission-oriented goals and 8 management-focused goals. Five goal areas—Cybersecurity; Open Data; Science, Technology, Engineering, and Mathematics (STEM) Education; Strategic Sourcing; and Sustainability (renamed Climate Change (Federal Actions))—were carried over from the set of interim CAP goals, while the other 10 are new goal areas. OMB stated on Performance.gov that more detailed action plans for each of the goals, including specific metrics and milestones that will be used to gauge progress, will subsequently be released. The new CAP goals will also have co-leaders; one from an office within the Executive Office of the President (EOP) and one or more from federal agencies. According to OMB staff, this change was made to ensure that CAP goal leaders can leverage the convening authority of officials from the EOP while also drawing upon expertise and resources from the agency level. GPRAMA Requirements for Establishing Planned Performance for CAP Goals GPRAMA requires the Director of OMB to establish, in the annual federal government performance plan, a planned level of performance for each CAP goal for the year in which the plan is submitted and the next fiscal year, as well as quarterly performance targets for the goals. GPRAMA Requirements for Reporting CAP Goal Performance Information GPRAMA requires the Director of OMB to publish on Performance.gov information about the results achieved during the most recent quarter and trend data compared to the planned level of performance for each CAP goal. OMB released the federal government performance plan on Performance.gov concurrently with the fiscal year 2013 budget submission that identified the 14 interim CAP goals. The information on Performance.gov included a goal statement for each of the interim goals that established an overall planned level of performance. During the two- year interim goal period, OMB addressed the requirement to report on results achieved during the most recent quarter for each of the CAP goals by publishing 5 sets of quarterly updates to the interim CAP goals on Performance.gov. The first set of updates, for the fourth quarter of fiscal year 2012, was published in December 2012 and the final set of updates, for the fourth quarter of fiscal year 2013, was published in February 2014. These documents described general accomplishments made to date, specific actions completed, or both. The updates to the Broadband CAP goal, for instance, included short descriptions of general progress made towards each of the five strategies identified for achieving the goal, as well as specific milestones accomplished. The quarterly updates did not, however, consistently identify required interim planned levels of performance and data necessary to indicate progress being made toward the CAP goals. Updates to eight of the goals included quarterly, biannual, or annual data that indicated performance achieved to date toward the target identified in the goal statement. Three of the eight goals (Cybersecurity, Energy Efficiency, and Strategic Sourcing) also contained the required annual or quarterly targets that defined planned levels of performance, which allowed for an assessment of interim progress. For example, the Cybersecurity goal’s updates stated that the goal would not be met within its established time frame, and provided quarterly performance data compared to quarterly targets for the entirety of the goal period to support the statement. In contrast, the updates for the other five goals did not contain annual or quarterly targets, which made it difficult to determine whether interim progress towards the goals’ overall planned levels of performance was being made. For example, updates to the Exports goal included data on the total amount of U.S. exports by quarter for calendar years 2012 and 2013 but did not include a target level of performance for those years or quarters. Therefore, it was unclear whether the goal’s overall planned level of performance of doubling U.S. exports by the end of 2014 is on track to be met. Furthermore, updates to six interim CAP goals did not include trend data to indicate progress being made towards the goals’ overall planned levels of performance. Figure 2 below identifies the frequency with which data on CAP goal performance were reported, as well as the overall performance CAP goal leaders reported making compared to the goal’s planned level of performance through the fourth quarter of fiscal year 2013. Through our review of information on Performance.gov and interviews with managers of the six interim CAP goals that did not report any data on progress towards the stated goal, we identified reasons that included: Lack of quantitative planned level of performance (targets). The Entrepreneurship and Small Business CAP goal lacked a quantitative performance target. The quarterly updates to the goal explained that efforts were focused on the goal’s 10 sub-goals. Most of these sub- goals, however, also lacked quantitative performance targets. The deputy goal leader told us that some of the sub-goals did not have quantitative targets by design, as goal managers thought it more appropriate to use qualitative milestones to track progress towards them. The quarterly updates to the “Streamline immigration pathways for immigrant entrepreneurs” sub-goal lacked a quantitative target but had a range of qualitative milestones. For example, the Department of Homeland Security and the Department of State established a milestone to identify reforms needed to ease the application and adjudication processes for visas available to certain immigrant entrepreneurs. Unavailable data. Some CAP goal managers told us that the data needed to assess and report progress toward their goals’ performance targets were unavailable or not yet being collected. For example, a manager of the Job Training CAP goal told us that staff had not established a baseline number of participants served by federal job training programs against which progress towards the goal could be tracked. In addition, managers of the Real Property CAP goal told us that they did not have data available for tracking progress toward the goal of holding the federal real property footprint at its fiscal year 2012 baseline level. Where key data were not reported, some goal managers took actions to obtain previously unavailable data or developed an alternative approach for assessing progress. Job Training CAP Goal. The first quarterly update for the Job Training CAP goal, published on Performance.gov in December 2012, stated that federal agencies were surveyed to compile a list of all job training programs in the federal government, including the number of participants served by those programs, and that a working group was developing a baseline for measuring progress towards the goal of preparing 2 million workers with skills training by 2015. A goal manager told us that the deputy goal leader and staff from the PIC gathered baseline information for most of the programs within the scope of the CAP goal, but that they were unable to complete the efforts by the end of the goal period. Real Property CAP Goal. Managers of the Real Property CAP goal told us that they worked to establish a baseline and metrics for measuring future performance and would be able to report on progress after the goal period ended. Closing Skills Gaps CAP Goal. A manager of the Closing Skills Gaps goal told us that the goal’s managers decided early on that it did not make sense for each of the goal’s identified mission-critical occupations to have the same skills gaps reduction target. Instead, managers of the goal’s sub-goals identified efforts to reduce skills gaps in their specific occupations. They identified an individual targeted level of performance for that effort and collected and reported data on progress made towards the target. For instance, managers of the Acquisitions sub-goal established a target for increasing the certification rate of GS-1102 contract specialists to 80 percent. The final quarterly status update to the Closing Skills Gaps CAP goal reported that the target was met and the certification rate increased to 81 percent. Veteran Career Readiness CAP Goal. The leader of the Veteran Career Readiness CAP goal told us that efforts were made to collect data to assess the veteran employment situation. For instance, she said that an interagency data-gathering working group reviewed sources of available data, integrated those data – such as the unemployment rate for various sub-populations of veterans – into dashboards for senior leadership review, and made proposals to improve data availability. In addition, the Army led a working group to develop a more complete picture of veterans receiving unemployment compensation. She said that these and other efforts led to a concerted effort to improve the availability of data, and to develop and implement metrics measuring career readiness and attendance in a veteran career transition assistance program. However, no data to track progress towards the overall goal were reported during the interim goal period. As we have previously reported, no picture of what the federal government is accomplishing can be complete without adequate performance information. However, OMB and CAP goal leaders did not identify interim planned levels of performance or targets for most of the interim CAP goals. Furthermore, they established a number of CAP goals for which data necessary to indicate progress towards the goal could not be reported. In so doing, they limited their ability to demonstrate progress being made towards most of the CAP goals and ensure accountability for results from those who helped to manage the goals. GPRAMA Requirement for Establishing Milestones GPRAMA requires the Director of OMB to establish, in the federal government performance plan, clearly defined quarterly milestones for the CAP goals. GPRAMA Requirement for Reporting on Contributions towards Cross-Agency Priority Goals GPRAMA requires that OMB identify the agencies, organizations, program activities, regulations, tax expenditures, policies, and other activities that contribute to each CAP goal on Performance.gov. It also requires OMB to make available on the website an assessment of whether relevant agencies, organizations, program activities, regulations, tax expenditures, policies, and other activities are contributing as planned. In the status updates that were published on Performance.gov, managers of each of the CAP goals reported the general approaches, strategies, or specific initiatives being employed to make progress towards the achievement of the goal, as well as the departments, agencies, and programs that were expected to contribute to goal achievement. For example, the leader of the Science, Technology, Engineering, and Mathematics (STEM) Education CAP goal identified a number of general strategies for making progress towards the achievement of its goal of increasing the number of graduates in STEM subjects by 1 million over the next 10 years, such as “Address the mathematics preparation gap that students face when they arrive at college” and “Identifying and supporting the role of technology and innovation in higher education.” In addition, the goal leader identified a number of programs and goals within four departments and agencies that were likely to contribute in part or in whole to the goal. Figure 3 below illustrates how this information was presented in the update to the STEM Education CAP goal for the fourth quarter of fiscal year 2013. In a May 2012 report on our work related to the CAP goals, we noted that information on Performance.gov indicated additional programs with the potential to contribute to each of the CAP goals may be identified over time. We then recommended that OMB review and consider adding to the list of CAP goal contributors the additional departments, agencies, and programs that we identified, as appropriate. OMB agreed with the recommendation, and in the quarterly updates to the CAP goals published in December 2012 and March 2013, OMB added some of the departments, agencies, and programs we identified in our work to some CAP goals’ lists of contributors. For example, we had noted that 12 member agencies of the Trade Promotion Coordinating Committee had not been identified as contributors to the Exports CAP goal. OMB added additional information about contributors to the Exports goal in the update published in December 2012. During our review, in some cases CAP goal managers told us about additional organizations and program types that contributed to their goals, but which were not identified on Performance.gov or in our previous report. For example, the leader of the STEM Education CAP goal told us that representatives from the Smithsonian Institution led an interagency working group that contributed to key efforts towards achieving the goal. Although the CAP goal updates indicate that the Smithsonian Institution is involved in federal STEM education efforts, it was not identified in a dedicated list of contributors to the goal. We have previously found that federal STEM education programs are fragmented across a number of agencies. We continue to believe that the federal government’s efforts to ensure STEM education programs are effectively coordinated must include all relevant efforts. Furthermore, the leader of the Broadband CAP goal told us that he is aware that tax deductions available to businesses making capital investments contributed to the goal by incentivizing investments in broadband. We have long referred to such deductions, along with other reductions in a taxpayer’s liability that result from special exemptions and exclusions from taxation, credits, deferrals of tax liability, or preferential tax rates, as tax expenditures. As we have previously reported, as with spending programs, tax expenditures represent a substantial federal commitment to a wide range of mission areas. We have recommended greater scrutiny of tax expenditures. Periodic reviews could help determine how well specific tax expenditures work to achieve their goals and how their benefits and costs compare to those of programs with similar goals. As previously mentioned, GPRAMA also requires OMB to identify tax expenditures that contribute to CAP goals. However, tax expenditures were not reported as contributors to the Broadband CAP goal in the quarterly status updates published on Performance.gov. Leading practices state that a clear connection between goals and day-to- day activities can help organizations better articulate how they plan to accomplish their goals. In addition, a clear connection between goals and the programs that contribute to them helps to reinforce accountability and ensure that managers keep in mind the results their organizations are striving to achieve. Milestones—scheduled events signifying the completion of a major deliverable or a set of related deliverables or a phase of work—can help organizations demonstrate the connection between their goals and day-to-day activities and that they are tracking progress to accomplish their goals. Organizations, by describing the strategies to be used to achieve results, including clearly defined milestones, can provide information that would help key stakeholders better understand the relationship between resources and results. GAO-13-174; GAO-13-228; and GAO, Managing for Results: Critical Issues for Improving Federal Agencies’ Strategic Plans, GAO/GGD-97-180 (Washington, D.C.: Sept. 16, 1997). actions, however, lacked clear time frames for completion. Figure 4 below illustrates the “next steps” identified for the Strategic Sourcing CAP goal in the update for the third quarter of fiscal year 2013. Completion status: The Real Property CAP goal update for the second quarter of fiscal year 2013 identified two planned actions as “next steps.” “After agencies submit their Revised Cost Savings and Innovation Plans to OMB, OMB will evaluate agency plans to maintain their square footage baselines, while balancing mission requirements,” and “Updates on agency square footage baselines and projects are forthcoming and will be posted on Performance.gov.” These two actions were again identified as “next steps” in the update for the third quarter of fiscal year 2013, but no update was provided on the status of the actions. By establishing planned activities that, in many of the CAP goal updates, did not have information about their alignment with the strategies they supported, their time frames for completion, or their completion status, CAP goal leaders did not fully demonstrate that they had effectively planned to support goal achievement or were tracking progress toward the goal or identified milestones. OMB did not issue formal guidance to CAP goal leaders on the types of information that were to be included in the CAP goal updates, including information about contributors and milestones. Standards for internal control in the federal government emphasize the importance of documenting policies and procedures to provide a reasonable assurance that activities comply with applicable laws and regulations, and that managers review performance and compare actual performance to planned or expected results and analyze significant differences.staff told us they provided an implementation plan template to goal leaders, which outlined the data elements to be reported in the quarterly status updates. The template was also used to collect information for internal and public reporting. Some CAP goal managers told us that OMB or PIC staff, in their role supporting the collection, analysis, and presentation of data on CAP goal performance, occasionally provided feedback on the information that the individuals submitted in draft updates that OMB reviewed before they were published on Performance.gov. For example, one CAP goal manager told us that during a review of an update submission PIC staff told him that he should develop additional milestones to be completed during a specific future fiscal year quarter. This is in contrast to the detailed guidance that OMB issued on the types of information that agencies must provide for the updates for agency priority goals (APG), which are also published quarterly on Performance.gov. The APG guidance includes explicit instructions for agencies to identify, as appropriate, the organizations, regulations, tax expenditures, policies, and other activities within and external to the agency that contribute to each APG, as well as key milestones with planned completion dates for the remainder of the goal period. Because guidance for the types of information that should have been included in the CAP goal updates was never formally established, CAP goal leaders were at a heightened risk of failing to take into account important contributors to the goal and providing incomplete information about milestones that could help demonstrate progress being made. GPRAMA Requirement for OMB Progress Reviews GPRAMA requires that, not less than quarterly, the Director of OMB, with the support of the PIC, shall review progress on the CAP goals, including progress during the most recent quarter, overall trends, and the likelihood of meeting the planned level of performance. GPRAMA also requires that, as part of these reviews, OMB categorize goals by their risk of not achieving the planned level of performance and, for those goals most at risk of not meeting the planned level of performance, identify strategies for performance improvement. As required by GPRAMA, OMB reviewed progress on CAP goals each quarter, beginning with the quarter ending June 30, 2012. This review process consisted of the collection of updated information for each CAP goal by OMB or PIC staff, and the development of a memorandum for the Director of OMB with information on the status of the CAP goals. To develop these memorandums, OMB staff told us that approximately 6 weeks after the end of each quarter, OMB and PIC staff worked with CAP goal leaders to collect updated data and information on goal metrics and milestones, and to update the narratives supporting the data. CAP goal leaders, or staff assisting leaders with the management of efforts related to the goal, would provide this information to OMB using a template for the status updates ultimately published on Performance.gov. In addition to the memorandums developed for the Director of OMB, OMB published more detailed information through the quarterly status updates available on Performance.gov. OMB and PIC staff told us that to support OMB’s quarterly review efforts, PIC staff were to conduct assessments rating the overall health of implementation efforts and goal leader engagement. They were also to assess the execution status of each goal, including the quality and trend of performance indicators. One purpose of these assessments was to identify areas where risks, such as goal leader turnover, could affect the ability to achieve the planned level of performance. Consistent with this intent, several of the quarterly OMB review memorandums we examined highlighted turnover in goal leader or deputy goal leader positions as risks, and suggested the need to find or approve replacements. Although PIC staff have been tasked with assessing these elements of CAP goal implementation, and said that there was a shared understanding between involved staff as to how these assessments would be carried out, the PIC has not documented its procedures or criteria for conducting these assessments. Standards for internal control in the federal government emphasize the importance of documenting procedures, including those for assessing performance. Without clearly established criteria and procedures, PIC staff lack a means to: consistently assess implementation efforts and execution across all goals; bring any deficiencies, risks, and recommended improvements identified to the attention of leadership; and ensure consistent application of criteria over time. While these quarterly review memorandums identified one goal as being at risk of not achieving the planned level of performance, and identified other instances where progress on goals had been slower than planned, the memorandums did not consistently outline the strategies that were being used to improve performance or address identified risks. For example, the Cybersecurity CAP goal was the one goal specifically described as being at risk of not achieving the planned level of performance, both in these memorandums and in the status updates on Performance.gov. Specifically, the memorandum for the third quarter of fiscal year 2012 identified the risk of not achieving the planned level of performance, and outlined seven specific risks facing the goal and the steps being taken to mitigate them. Similarly, the memorandum for the second quarter of fiscal year 2013 also acknowledged that some agencies were at risk of not meeting their Cybersecurity CAP goal targets. However, in contrast to the earlier memorandums, no information was included about the specific steps that were being taken to mitigate these risks, although information on planned and ongoing actions to improve government-wide implementation was included in the milestones section of the status update for that quarter on Performance.gov. The memorandum for the fourth quarter of fiscal year 2012 also acknowledged that the pace of progress on the STEM Education and Closing Skills Gaps goals had been slower than expected. While the memorandum stated that additional OMB attention was needed to support implementation and assure sufficient progress, no information on the specific strategies being employed to improve performance was mentioned. According to OMB staff, however, these memorandums were used to inform subsequent conversations with OMB leadership, which would build on the information presented in the memorandums. Furthermore, because the data necessary to track progress for some goals were unavailable, the Director of OMB would not have been able to consistently review progress for all CAP goals, or make a determination about whether some CAP goals were at risk of meeting their planned levels of performance. This fact was acknowledged in the quarterly review memorandums for quarters one and two of fiscal year 2013, which acknowledged that progress on three goals (Entrepreneurship and Small Business, Job Training, and STEM Education) was difficult to track, and that additional work was needed on data collection. However, no information on the specific steps that were being taken to address these shortcomings was included. A lack of specific information about the steps being taken to mitigate identified risk areas and improve performance could hinder the ability of OMB leadership—and others—to adequately track the status of efforts to address identified deficiencies or risks and to hold officials accountable for taking necessary actions. GPRAMA Requirement for Goal Leader and Agency Involvement in Progress Reviews As part of the quarterly review process, GPRAMA requires that the Director of OMB review each priority goal with the appropriate lead government official, and include in these reviews officials from the agencies, organizations, and program activities that contribute to the achievement of the goal. According to OMB staff, to encourage goal leaders and contributing agencies to take ownership of efforts to achieve the goals, OMB gave goal leaders flexibility to use different approaches to engage agency officials and review progress at the CAP-goal level. While guidance released by OMB in August 2012 encouraged goal leaders to leverage existing interagency working groups, committees, and councils in the management of the goals as much as practicable, it did not include information on the purpose of reviews, expectations for how reviews should be conducted to maximize their effectiveness as a tool for performance management and accountability, or the roles that CAP goal leaders and agency officials should play in the review process. Again, standards for internal control in the federal government emphasize the importance of documenting procedures for reviewing performance against established goals and objectives. This is in contrast to the detailed guidance that OMB released for agency priority goal and agency strategic objective reviews, which outlined the specific purposes of the reviews, how frequently they should be conducted, the roles and responsibilities of agency leaders involved in the review process, and how the reviews should be conducted. We also found that this guidance for reviews at the agency level was broadly consistent with the leading practices for performance reviews that we previously identified. While no official guidance was published to guide how reviews involving goal leaders and staff from contributing agencies could be conducted for the CAP goals, OMB staff said the principles of the guidance released for agency reviews, which reflected many of the leading practices, was referenced in conversations with CAP goal leaders and teams. OMB has emphasized that flexibility is needed to ensure that goal leaders can use review processes that are appropriate given the scope of interagency efforts, the number of people involved, and the maturity of existing reporting and review processes. The guidance for agency reviews gave agencies flexibility to design their performance review processes in a way that would fit the agency’s mission, leadership preferences, organizational structure, culture, and existing decision- making processes. In our previous work, we detailed how several federal agencies had implemented quarterly performance reviews in a manner consistent with leading practices, but which were also tailored to the structures, processes, and needs of each agency. In this way, flexible implementation of review processes is possible within a framework that encourages the application of leading practices. A lack of clear expectations for how progress should be reviewed at the CAP-goal level resulted in a number of different approaches being used by goal leaders to engage officials from contributing agencies to review progress on identified goals and milestones, ranging from regular in- person review meetings led by the CAP goal leader to the review of written updates provided to the goal leader by officials from contributing agencies. See appendix IV for more detailed information on the various processes used by goal leaders to collect data on, and review progress towards, identified goals. Instituting review processes consistent with the leading practices we previously identified can help ensure that reviews include meaningful performance discussions, provide opportunities for oversight and accountability, and drive performance improvement. Taken together, these leading practices emphasize the importance of leadership involvement in the review process, data collection and review meeting preparation, participation by key officials, and rigorous follow-up. Through our evaluation of how goal leaders and contributing agency officials reviewed progress towards the interim goals, we identified two CAP goals—Cybersecurity and Closing Skills Gaps—and one sub-goal— the Entrepreneurship and Small Business sub-goal on improving access to government services and information (BusinessUSA sub-goal)—where goal managers instituted in-person review processes with officials from contributing agencies that were broadly consistent with the full range of leading practices for reviews, which we have summarized in four categories below. The processes used by other CAP goal leaders to engage agency officials in the review of progress did not reflect the full range of leading practices. Leadership Involvement. Leading practices indicate that leaders should use frequent and regular progress reviews as a leadership strategy to drive performance improvement and as an opportunity to hold people accountable for diagnosing performance problems and identifying strategies for improvement. The direct and visible engagement of leadership is vital to the success of such reviews. Leadership involvement helps ensure that participants take the review process seriously and that decisions and commitments can be made. The goal leaders managing the Cybersecurity and Closing Skills Gaps goals, as well as the BusinessUSA sub-goal, were directly involved in leading in-person reviews for these goals, and in using them as opportunities to review progress, identify and address performance problems, and hold agency officials accountable for progress on identified goals and milestones, as detailed in table 1. Data Collection and Review Meeting Preparation. Leading practices also indicate that those managing review processes should have the capacity to collect, analyze, and communicate accurate, useful, and timely performance data, and should rigorously prepare for reviews to enable meaningful performance discussions. The collection of current, reliable data on the status of activities and progress towards goals and milestones is critical so that those involved can determine whether performance is improving, identify performance problems, ensure accountability for fulfilling commitments, and learn from efforts to improve performance. The ability to assess data to identify key trends and areas of strong or weak performance, and to communicate this to managers and staff effectively through materials prepared for reviews, is also critical. As detailed in table 2, those supporting the Cybersecurity and Skills Gap goals, and the BusinessUSA sub-goal, instituted processes to regularly collect and analyze data on progress towards identified goals and milestones, and to ensure these data would be communicated through materials prepared for review meetings. Participation by Key Officials. Leading practices indicate that key players involved in efforts to achieve a goal should attend reviews to facilitate problem solving. This is critical as their participation enables those involved to break down information silos, and to use the forum provided by the review to communicate with each other, identify improvement strategies, and agree on specific next steps. Reviews for both the Cybersecurity and Closing Skills Gaps CAP goals, and the BusinessUSA sub-goal, were structured so that relevant agency officials playing a key role in efforts to carry out the goal were included, as detailed in table 3. Review Follow-Up. Leading practices indicate that participants should engage in sustained follow-up on issues identified during reviews, which is critical to ensure the success of the reviews as a performance improvement tool. Important follow-up activities include identifying and documenting specific follow-up actions stemming from reviews, those responsible for each action item, as well as who will be responsible for monitoring and follow-up. Follow-up actions should also be included as agenda items for subsequent reviews to hold responsible officials accountable for addressing issues raised and communicating what was done. Goal managers for the Cybersecurity and Closing Skills Gap CAP goals, as well as the BusinessUSA sub-goal, took steps to follow up on action items identified in these meetings, and to ensure that steps were taken towards their completion, as detailed in table 4. Review Effects. Goal leaders and managers we interviewed said that these review processes were valuable in driving improved performance, establishing a greater sense of accountability for progress on the part of contributors, and in providing a forum for interagency communication and collaboration. For example, according to DHS staff involved in the management of the Cybersecurity CAP goal, implementation of Personal Identity Verification (PIV) requirements across the federal government had been stagnant for several years prior to the introduction of cybersecurity as a CAP goal. The review process was used to hold agencies accountable for improved PIV implementation, which helped bring an increased focus on the issue and drive recent progress. Since the reviews were instituted in 2012, DHS has reported improved PIV adoption in civilian agencies, which has increased from 1.24 percent in fiscal year 2010, to 7.45 percent in fiscal year 2012, to 19.67 percent in the fourth quarter of fiscal year 2013. According to data from DHS, while still falling short of the target, this has contributed to the overall increase in PIV adoption across the federal government—including both civilian agencies and the Department of Defense—from 57.26 percent in fiscal DHS year 2012 to 66.61 percent in fourth quarter of fiscal year 2013.staff also added that agencies generally had not previously collaborated on cybersecurity issues or worked to identify best practices. According to DHS staff, the reviews have created an important point of collaboration between DHS, OMB, National Security Staff, and agencies, and provided an opportunity to inform agencies of best practices and connect them with other agencies that are meeting their targets to learn from them. Similarly, OPM officials and sub-goal leaders involved in the management of the Closing Skills Gap CAP goal said that the quarterly review meetings were a critical means to ensure sub-goal leaders and staff were demonstrating progress. Having sub-goal leaders report out on progress, and hear about the progress made in other sub-goal areas, provided additional pressure for continuous improvement and the need to remain focused on driving progress towards their goals. Having the goal leader lead the review was also a way to demonstrate leadership commitment to the achievement of each sub-goal. According to OPM officials and sub- goal leaders, the review meetings also served as an important forum for discussing innovative approaches being taken to address skills gaps in different areas, opportunities for collaboration to address challenges shared by different sub-goals, and how leaders could leverage the efforts of other sub-goals to drive progress on their own. The BusinessUSA sub-goal leader said that having it as the basis for a CAP sub-goal elevated the cross cutting nature of the initiative. In addition to reviewing performance information and the status of deliverables, discussions at inter-agency Steering Committee meetings were used to discuss how contributors could work together to meet the initiative’s performance goals. This communication and coordination led to connections between agencies and to discussions about how programs could be working in a more integrated way. For example, these discussions were used to identify ways that programs could more effectively integrate program information on the BusinessUSA website to increase customer satisfaction. We found that the processes used by other CAP goal leaders to engage agency officials in the review of progress, which are summarized in appendix IV, did not reflect the full range of leading practices. For example, the process for reviewing progress on the Job Training CAP goal involved staff from the PIC collecting updates on recent milestones from agencies, which were then compiled in the quarterly status update and reviewed by the goal leader. This approach was used by the goal leaders for the Broadband and STEM Education CAP goals to review progress as well. While goal leaders and managers for these goals indicated that they used the collection and review of information as an opportunity to communicate with officials from contributing agencies, this approach contrasts with OMB guidance for reviews of agency priority goals, which states explicitly that performance reviews should not be conducted solely through the sharing of written communications. As OMB noted in its guidance, in-person engagement of leaders in performance reviews greatly accelerates learning and performance improvement, and personal engagement can demonstrate commitment to improvement, ensure coordination across agency silos, and enable rapid decision making. While not employing the full range, goal leaders for a number of goals did use processes that reflected one or more leading practices. For example, many CAP goal leaders led or participated in interagency meetings with representatives of contributing agencies. While these were used to facilitate interagency communication and collaboration on the development of plans and policies, it was unclear whether many of these meetings were consistently used to review progress on identified CAP goals and milestones. The goal leader for the Strategic Sourcing CAP goal used processes that reflected leadership involvement, participation by key officials, and the collection and analysis of relevant data. Specifically, according to goal managers, the goal leader led regular meetings of the Strategic Sourcing Leadership Council (SSLC), which were attended by senior procurement officials from eight agencies that combine to make up almost all of the federal government’s total procurement spending. To prepare for each SSLC meeting, staff from OMB’s Office of Federal Procurement Policy (OFPP) held a meeting for supporting staff from each agency, who would then prepare the SSLC member from their agency for the issues to be discussed in the SSLC meeting. OFPP also established a regular data collection process where each agency would report on its adoption and spending rates for two strategic sourcing options, which would then be used for the purposes of reporting on the CAP goal. However, it was unclear how regularly, if at all, SSLC meetings were used to engage agency officials in the review of data on agency adoption of, and spending on, strategic sourcing options, or how regularly meetings were used to review progress that was being made towards the CAP goal. It was also unclear what mechanisms, if any, were used to ensure rigorous follow-up on issues raised in these meetings, a key leading practice, as there were no official meeting minutes maintained. The lack of an official record could hinder follow-up and accountability for any identified actions that need to be taken. Representatives of some goals stated that it was difficult to isolate the impact of the CAP goal designation, and its associated reporting and review requirements, on performance and collaboration. According to some goal managers, because their interim goals were based on initiatives that had been previously established in executive orders or Presidential memorandums, much of the interagency activity supporting their efforts would have happened without the CAP goal designation and its reporting and review requirements. For example, a manager for the Data Center Consolidation CAP goal told us that the previously established Federal Data Center Consolidation Initiative was used to drive progress and that the CAP goal designation and quarterly reporting and review requirements had little impact. Similarly, Job Training CAP goal managers said that interagency collaboration on job training issues had been established prior to the creation of the CAP goal, that the goal’s reporting and review requirements were incidental to the contributors’ ongoing work, and that it did not add an additional level of accountability for the completion of job training initiatives. However, this is a goal where no data were reported to demonstrate its impact on federal job training programs, and which was identified in multiple OMB reviews as having slower than anticipated progress due, in part, to extended periods of time in which there was no deputy CAP goal leader to provide support necessary to improve coordination and collaboration. While many CAP goal leaders and staff we interviewed noted the progress they had made with their existing interagency meetings and approaches, a lack of clear expectations or guidance for how review processes at the CAP goal level should be carried out can lead to a situation where reviews are implemented in a manner that is not informed by, or fully consistent with, leading practices. This could result in missed opportunities to realize the positive effects on performance and accountability that can stem from the implementation of review processes that regularly and consistently involve leaders and agency officials in the analysis of performance data to identify and address performance deficiencies, and use rigorous follow-up to ensure accountability for commitments. Many of the meaningful results that the federal government seeks to achieve require the coordinated efforts of more than one federal agency. GPRAMA’s requirement that OMB establish CAP goals offers a unique opportunity to coordinate cross-agency efforts to drive progress in priority areas. That opportunity will not be realized, however, if the CAP goal reporting and review requirements and leading review practices are not followed. The reporting and review requirements for the CAP goals, and leading practices for the reviews, are designed to ensure that relevant performance information is used to improve performance and results, and that OMB and goal leaders actively lead efforts to engage all relevant participants in collaborative performance improvement initiatives and hold them accountable for progress on identified goals and milestones. OMB reported performance information in the quarterly CAP goal status updates it published on Performance.gov. While updates for most goals reported data on performance towards the identified planned level of performance, the information in the updates did not always present a complete picture of progress towards identified goals and milestones. For example, while updates for 8 of the 14 goals included data that indicated performance towards the identified overall planned level of performance, only 3 also contained annual or quarterly targets that allowed for an assessment of interim progress. Updates for the other 6 of the 14 goals did not report on performance towards the goal’s primary performance target because the goal was established without a quantitative target or because goal managers were unable to collect the data needed to track performance. In other cases, planned activities that were identified as contributing to the goal were sometimes missing important elements, including alignment with the strategies for goal achievement they supported, a time frame for completion, or information on their implementation status. The incomplete picture of progress that many of the updates gave limited the ability of goal leaders and others to ensure accountability for the achievement of targets and milestones. Holding regular progress reviews that are consistent with GPRAMA requirements and the full range of leading practices can produce positive effects on performance and collaboration. Engaging contributors in regular reviews of data on performance can help ensure interagency efforts are informed by information on progress towards identified goals and milestones, which can be used to identify and address areas where goal or milestone achievement is at risk. Reviews can also be used to reinforce agency and collective accountability for the achievement of individual and shared outcomes, helping to ensure that efforts to improve performance or address identified risks are implemented. Lastly, reviews can be used to foster greater collaboration, ensuring opportunities for communication and coordination between officials involved in efforts to achieve shared outcomes. While OMB and CAP goal leaders instituted processes for reviewing progress on the interim CAP goals, if GPRAMA requirements and leading practices for reviews are not consistently followed, it may result in missed opportunities to improve performance, hold officials accountable for achieving identified goals and milestones, and ensure agency officials are coordinating their activities in a way that is directed towards the achievement of shared goals and milestones. We recommend that the Director of OMB take the following three actions: Include the following in the quarterly reviews of CAP goal progress, as required by GPRAMA: a consistent set of information on progress made during the most recent quarter, overall trends, and the likelihood of meeting the planned level of performance; goals at risk of not achieving the planned level of performance; and the strategies being employed to improve performance. Work with the PIC to establish and document procedures and criteria to assess CAP goal implementation efforts and the status of goal execution, to ensure that the PIC can conduct these assessments consistently across all goals and over time. Develop guidance similar to what exists for agency priority goal and strategic objective reviews, outlining the purposes of CAP goal progress reviews, expectations for how the reviews should be carried out, and the roles and responsibilities of CAP goal leaders, agency officials, and OMB and PIC staff in the review process. To ensure that OMB and CAP goal leaders include all key contributors and can track and report fully on progress being made towards CAP goals overall and each quarter, we recommend that the Director of OMB direct CAP goal leaders to take the following four actions: Identify all key contributors to the achievement of their goals; Identify annual planned levels of performance and quarterly targets for each CAP goal; Develop plans to identify, collect, and report data necessary to demonstrate progress being made towards each CAP goal or develop an alternative approach for tracking and reporting on progress quarterly; and Report the time frames for the completion of milestones; the status of milestones; and how milestones are aligned with strategies or initiatives that support the achievement of the goal. We provided a draft of this report for review and comment to the Director of OMB, the Secretaries of Commerce and Homeland Security, the Director of the Office of Personnel Management, the Administrator of the Small Business Administration, as well as the officials we interviewed to collect information on the interim CAP goals from the Council on Environmental Quality, Department of Education, Department of Labor, Department of Veterans Affairs, National Science Foundation, and the Office of Science and Technology Policy. OMB and PIC staff provided oral comments on the draft, and we made technical changes as appropriate. OMB staff generally agreed to consider our recommendations. For example, while they said that OMB and PIC staff will continue to work directly with CAP goal leaders to convey suggested practices for reviewing performance, they will consider referencing principles and practices for data-driven performance reviews in future Circular A-11 guidance related to the management of CAP goals. Furthermore, while they noted that quantitative performance data for some key measures may not available on a quarterly basis, they said that they will continue to work to develop more robust quarterly targets. Officials or staff from the Departments of Commerce and Veterans Affairs, and the Office of Science and Technology Policy provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the Director of OMB as well as appropriate congressional committees and other interested parties. The report is also available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-6806 or mihmj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix VI. This report is part of our response to a mandate that we evaluate the implementation of the federal government priority goals under the GPRA Modernization Act of 2010 (GPRAMA). Due to the timing of our work, we focused on the implementation of the reporting and review requirements for the 14 interim cross-agency priority (CAP) goals established in February 2012.about progress made towards the interim CAP goals; and (2) how, if at all, quarterly progress reflected GPRAMA requirements and leading practices for data-driven reviews, as well as how they contributed to improved cross-agency performance and collaboration. Specifically, this report assesses (1) what is known To address these objectives, we interviewed representatives of 13 of the 14 interim goals. For 8 of the 13 goals we spoke directly with the goal leader or deputy goal leader, along with, in some cases, staff from Office of Management and Budget (OMB) and agencies involved in supporting efforts related to the goals. For the other five goals (Closing Skills Gaps, Cybersecurity, Data Center Consolidation, Exports, and Job Training) we met with agency officials or OMB staff playing a key role in the management of interagency efforts related to the CAP goal. During these interviews, we asked officials questions concerning how the goal leader and officials from contributing agencies reviewed progress on the goal; the interagency groups used to engage agency officials and manage efforts related to the goal; the role that staff from OMB and the Performance Improvement Council (PIC) played in the review process; and any impact the CAP goal designation and review processes had on performance, collaboration, and accountability. We also participated in interviews with the goal leaders of 11 agency priority goals that were aligned with, or identified as a contributor to, a CAP goal. To further address the first objective, and assess what is known about progress made toward the interim CAP goals, we analyzed information on identified performance metrics and milestones included in the quarterly status updates for each CAP goal published on Performance.gov. We also analyzed relevant information collected through our interviews with CAP goal leaders, deputies, and supporting staff. We compared the data and information made available through the quarterly status updates with requirements in GPRAMA that Performance.gov include information for each goal on results achieved during the most recent quarter and overall trend data. To assess the reliability of performance data and information available through Performance.gov we collected information from OMB and PIC staff, and CAP goal representatives, about data quality control procedures. We determined that the data and information were sufficiently reliable for our analysis of what was reported on Performance.gov about progress towards identified goals and milestones. To address the second objective, we reviewed quarterly review memorandums developed for OMB leadership for five quarters, from the third quarter of fiscal year 2012 to the third quarter of fiscal year 2013.We compared the contents of these review memorandums with requirements for the OMB quarterly reviews established in GPRAMA. We also interviewed staff from OMB and the PIC to discuss the various approaches being used to review progress at the CAP-goal level, the data collection and review process, and the role of the PIC in supporting the quarterly review process. To further address the second objective we reviewed (where available) documents created for interagency meetings, such as meeting agendas, presentation materials, meeting notes, and attendee lists. We also observed one quarterly review meeting held for the Closing Skills Gap goals, and conducted interviews with sub-goal leaders from the Closing Skills Gaps and Entrepreneurship and Small Business CAP goals. These interviews were used to learn more about the involvement of officials from contributing agencies in the quarterly review process for each CAP goal, the processes that had been established to review progress at the sub- goal level, and to gain a more complete picture of participating agency officials’ perceptions of the impact of the CAP goals and review processes. We selected these sub-goals through a two-part process. Of the eight CAP goals for which we had completed interviews through the end of 2013, the team selected one goal for which the goal leader held quarterly meetings dedicated to reviewing progress toward the CAP goal with the goal’s contributors (Closing Skills Gaps). The team also selected a second goal for which the goal leader used a review process that did not rely on quarterly meetings between the goal leader and contributing agencies (Entrepreneurship and Small Business). To ensure that the team would have at least one goal representing each type of goal, the team also ensured that one goal would be an outcome-oriented policy goal and one goal would be a management goal. For both the Closing Skills Gaps and Entrepreneurship and Small Business CAP goals the team then selected four sub-goals for interviews. For the Closing Skills Gaps CAP goal the team interviewed the sub-goal leaders for the Economist; Information Technology/Cybersecurity; Science, Technology, Engineering, and Mathematics (STEM) Education; and Human Resources sub-goals. For the Entrepreneurship and Small Business CAP goal the team held interviews with the sub-goal leaders for the sub-goals to “Accelerate commercialization of Federal research grants,” “Advance federal small business procurement goals,” “Improve access to government services and information,” and “Streamline immigration pathways for immigrant entrepreneurs.” These were selected to ensure that the team would capture sub-goals in which a range of approaches for measuring and reviewing progress were being used. Specifically, sub- goals were selected to ensure the team would have some that did, and did not, hold regular meetings, and some that did, and did not, track quantitative measures of performance or milestones with time frames. Our selection of these sub-goals was nonstatistical and therefore our findings from these interviews are not generalizable to the other CAP goals. We compared what we learned about review processes at the CAP goal and sub-goal levels, through interviews and the collection of documentation, used by leaders from each goal against leading practices for performance reviews previously identified by GAO. Because the scope of our review was to examine the implementation of quarterly progress reviews, we did not evaluate whether these goals were appropriate indicators of performance, sufficiently ambitious, or met other dimensions of quality. We conducted our work from May 2013 to June 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Goal Statement As part of expanding all broadband capabilities, ensure 4G wireless broadband coverage for 98 percent of Americans by 2016. Close critical skills gaps in the federal workforce to improve mission performance. By September 30, 2013, close the skills gaps by 50 percent for three to five critical federal government occupations or competencies, and close additional agency-specific high risk occupation and competency gaps. Executive branch departments and agencies will achieve 95 percent implementation of the administration’s priority cybersecurity capabilities by the end of FY 2014. These capabilities include strong authentication, trusted Internet connections, and continuous monitoring. Improve information technology service delivery, reduce waste, and save $3 billion in taxpayer dollars by closing at least 2,500 data centers by fiscal year 2015. Increase energy productivity (amount of real gross domestic product in dollars/energy demand) 50 percent by 2030. Increase federal services to entrepreneurs and small businesses with an emphasis on 1) startups and growing firms and 2) underserved markets. Double U.S. exports by the end of 2014. The federal government will achieve a payment accuracy rate of 97 percent by the end of 2016. Ensure our country has one of the most skilled workforces in the world by preparing 2 million workers with skills training by 2015 and improving the coordination and delivery of job training services. The federal government will maintain the fiscal year 2012 square footage baseline of its office and warehouse inventory. In support of the president’s goal that the U.S. have the highest proportion of college graduates in the world by 2020, the federal government will work with education partners to improve the quality of STEM education at all levels to help increase the number of well- prepared graduates with STEM degrees by one-third over the next 10 years, resulting in an additional 1 million graduates with degrees in STEM subjects. Reduce the costs of acquiring common products and services by agencies’ strategic sourcing of at least two new commodities or services in both 2013 and 2014, that yield at least a 10 percent savings. In addition, agencies must increase their use of Federal Strategic Sourcing Initiative vehicles by at least 10 percent in both fiscal years 2013 and 2014. By 2020, the federal government will reduce its direct greenhouse gas emissions by 28 percent and will reduce its indirect greenhouse gas emissions by 13 percent by 2020 (from 2008 baseline). By September 30, 2013, increase the percent of eligible service members who will be served by career readiness and preparedness programs from 50 percent to 90 percent in order to improve their competitiveness in the job market. Goal leaders for 13 of 14 cross-agency priority (CAP) goals leveraged interagency groups for the purposes of coordinating efforts designed to contribute to progress on the cross-agency priority goal. This appendix includes information on the membership of these interagency groups, the frequency with which they met, and the purposes of those meetings. Membership Fourteen agencies with federal property management or transportation funding responsibilities, and broadband or other related expertise. To discuss best practices on broadband-related land management issues, and actions to implement an executive order on accelerating broadband infrastructure deployment. Senior officials from agencies considered major spectrum stakeholders and users of spectrum, including the Departments of Defense, Justice, Homeland Security (DHS), Commerce, and the National Aeronautics and Space Administration (NASA). To provide advice on spectrum policy and strategic plans, discuss commercial transfer of federal agency spectrum, and resolve issues affecting federal/non-federal users. To review progress on performance metrics and actions taken to close skills gaps in each of the six sub- goal areas. Officials from National Institute of Standards and Technology, General Services Administration (GSA), DHS, National Security Staff, Office of Management and Budget (OMB) and Performance Improvement Council. Twice each quarter Beginning in 2013, a meeting was held each quarter prior to the collection of data on agency progress on cybersecurity metrics. Another was held after data had been collected and analyzed to review and discuss agency progress. Data center consolidation program managers from 24 federal agencies. To identify and disseminate key information about solutions and processes to help agencies make progress towards data center consolidation goals. Interagency group No interagency groups were used to manage efforts related to this goal. Interagency groups were used to manage efforts at the sub-goal level. Senior-level representatives from 24 participating agencies. To oversee strategy, resources and timetables for the development of the BusinessUSA website, resolve interagency issues and ensure department/agency viewpoints are represented. Mid-to-senior level program, technology and customer service managers from 24 participating agencies. To assist the BusinessUSA program management office coordinate the design, development, and operation of the BusinessUSA website, and to track and monitor performance metrics on customer service and outcomes. SBIR/STTR program managers from11 agencies, and coordinating officials from Small Business Administration (SBA) and Office of Science and Technology Policy (OSTP) To discuss the development of SBIR/STTR program policy directives, the implementation of requirements, outreach and access to the programs, and program best practices. To provide updates on relevant agency activities and identify opportunities for interagency collaboration. To share best practices for expanding contracting to small and disadvantaged businesses, and reviewing progress on agency simplified-acquisition threshold goals. To provide officials from the White House, SBA, Commerce, and OMB with an opportunity to meet with senior agency leaders and discuss the steps agencies are taking to increase small business contracting. Membership Principals (cabinet secretaries and deputies) and staff from 20 agencies involved in export policy, service, finance, and oversight. To review progress on deliverables supporting the National Export Strategy, communications, and the status of individual export promotion initiatives. Bi-weekly to monthly To review the status of agency implementation of Do Not Pay requirements and milestones, and guidance for implementation. Officials from agencies with “high-priority” programs, as designated by OMB. To discuss the government- wide improper payment initiative and overall strategy. To discuss expanding access to job training performance data, and opportunities to promote its use at the local, state, and federal levels. Among other policy discussions, to discuss the development of agency “Freeze the Footprint” plans. To discuss policy to guide the federal government on sustainability issues, and to discuss sustainability goals. Every 4-6 weeks, during the development of the 5-year strategic plan. To develop a 5-year strategic plan for federal support for STEM education. Representatives from Departments of Defense, Energy, and Veterans Affairs (VA), DHS, HHS, GSA, NASA, and SBA. To discuss the development and adoption of strategic sourcing options. To review ongoing policy initiatives and opportunities for collaboration between agencies. To develop and implement a redesigned veterans transition program. meetings with officials from each agency. According to OMB staff, during these reviews participants reviewed metrics from across the agency’s information technology portfolio, which included, in some cases, those related to data center consolidation. Each quarter staff supporting the goal leader would collect updated information on contributing agency priority goals for the purposes of updating the quarterly status update. Each quarter the deputy goal leader would collect updated information on goals and milestones from the leaders of each of 10 sub-goals for the purposes of developing the quarterly status update. The deputy goal leader would follow-up with sub-goal leaders or agency officials, as necessary, to address issues or questions about the status of efforts. The goal leader would then review and approve the quarterly status update. Some sub-goal leaders would hold in-person meetings with officials from contributing agencies to, among other things, review progress on identified goals and milestones. See appendix III for information on interagency groups that were used to manage efforts for four of the sub-goals. Each quarter the goal leader, with the assistance of staff from Commerce and the PIC, would collect updated information on relevant agency metrics and activities for the purposes of updating the quarterly status update. Periodic meetings of the Export Promotion Cabinet/Trade Promotion Coordinating Committee, and its Small Business and other working groups, were also used to discuss the status of export promotion efforts and progress on specific deliverables. Each year OMB would collect and report data on agency improper payment rates. Staff from the OMB Office of Federal Financial Management led monthly meetings with agency representatives to discuss the implementation of the Do Not Pay initiative, which was designed to contribute to the reduction of improper payments. The Department of Treasury, as the agency leading implementation of the Do Not Pay initiative, would track agency progress on implementation milestones. Each quarter staff from the PIC would collect updated information on progress towards agency milestones, and work with the goal leader on the development of the quarterly status update. After this goal was revised in the second quarter of 2013, a new review process to track agency adherence to the goal was under development by OMB. Twice a year the Council on Environmental Quality (CEQ) would collect and review quantitative and qualitative data on agency progress towards established sustainability goals, including the reduction of agency greenhouse gas emissions. Following the collection of these data, the goal leader hosted meetings of the Steering Committee on Federal Sustainability, which were used to discuss federal sustainability policy and progress on sustainability goals. According to CEQ staff, the goal leader and CEQ staff would meet with representatives from agencies about sustainability issues on an ad hoc basis. In instances where there was a gap between an agency’s actual performance and the target established in that area, the goal leader, or other staff from CEQ, would meet with officials from that agency to discuss ways to address the performance gap. Each quarter the goal leader would collect updated information on agency milestones for inclusion in the quarterly status updates. Progress on some identified strategies to achieve the goal, such as the National Science Foundation’s efforts to improve undergraduate STEM education, were reviewed at the agency level. After progress was reviewed at the agency level the information was passed onto the goal leader and reported publicly in the quarterly status update. Each quarter the General Services Administration would collect data on agency adoption and spending rates for the Federal Strategic Sourcing Initiative (FSSI) solutions for domestic delivery and office supplies. The Strategic Sourcing Leadership Council met bi-monthly to guide the creation and adoption of new FSSI options, and, as part of that effort, might review quarterly data on agency adoption and spending rates. According to the goal leader, each month staff from the Departments of Defense and Veterans Affairs, and the PIC, would provide data for “one-pagers” and other status update documents with key pieces of relevant information, such as the veterans’ unemployment rate and the number of active employers on the Veteran’s Job Bank. These one-pagers would be used to inform regular Interagency Policy Council (IPC) discussions, along with more specific briefing memorandums, which were used to cover the latest issues, keep stakeholders focused on overall outcomes, and to inform discussion around specific outliers. Some of the data in these one-pagers would also be incorporated into the quarterly status updates. More frequently, issue papers and data analysis were provided to Veterans Employment Initiative (VEI) Task Force and IPC members as needed to address topical issues. Ongoing milestone reviews held by the VEI Task Force and its associated working groups on Education, Employment, Transition, and Entrepreneurship, provided an opportunity to discuss strategies being employed to improve performance. This appendix includes the print version of the text and rollover graphics contained in interactive figure 2. Overall Planned Level of Performance …achieve 95 percent implementation of the Administration’s priority cybersecurity capabilities by the end of fiscal year 2014. Data reported for primary performance goal …save $3 billion in taxpayer dollars by closing at least 2500 data centers by fiscal year 2015. “Agencies have already closed 640 data centers…” Overall Planned Level of Performance Double U.S. exports by the end of 2014. Frequency of Data Reporting for Overall Goal Quarterly …agencies’ strategic sourcing of at least two new commodities or services in both 2013 and 2014, that yield at least a 10 percent savings… In addition, agencies must increase their use of Federal Strategic Sourcing Initiative vehicles by at least 10 percent in both fiscal years 2013 and 2014. …ensure 4G wireless broadband coverage for 98 percent of Americans by 2016. Overall Planned Level of Performance …achieve a payment accuracy rate of 97 percent by the end of 2016. “Data Not Reported” Increase federal services to entrepreneurs and small businesses with an emphasis on 1) startups and growing firms and 2) underserved markets. “Data Not Reported” “Data Not Reported” The Federal Government will maintain the fiscal year 2012 square footage baseline of its office and warehouse inventory. “Data Not Reported” …increase the number of well-prepared graduates with STEM degrees by one-third over the next 10 years, resulting in an additional 1 million graduates with degrees in STEM subjects. Frequency of Data Reporting for Overall Goal “Data Not Reported” “Data Not Reported” In addition to the contact named above, Elizabeth Curda (Assistant Director) and Adam Miles supervised the development of this report. Virginia Chanley, Jehan Chase, Steven Putansu, Stacy Ann Spence, and Dan Webb made significant contributions to this report. Deirdre Duffy and Robert Robinson also made key contributions.

Summary: The federal government faces complex, high-risk challenges, such as protecting our nation's critical information systems. Effectively managing these challenges is essential for national and economic security and public health and safety. However, responsibility for addressing these challenges often rests with multiple agencies. To effectively address them, shared goals and cross-agency collaboration are fundamental. This report responds to GAO's mandate to evaluate the implementation of GPRAMA. It assesses (1) what is known about progress made towards the interim CAP goals; and (2) how, if at all, quarterly progress reviews reflected GPRAMA requirements and leading practices for reviews, as well as how reviews contributed to improved cross-agency performance and collaboration. To address these objectives, GAO analyzed CAP goal status updates and other documents from OMB and CAP goal progress-review meetings, and interviewed OMB staff and CAP goal representatives. GAO compared this information to GPRAMA requirements and to leading practices for performance reviews previously reported on by GAO. CAP Goal Progress. The GPRA Modernization Act of 2010 (GPRAMA) requires the Office of Management and Budget (OMB) to coordinate with agencies to: (1) establish outcome-oriented, federal government priority goals (known as cross-agency priority, or CAP, goals) with annual and quarterly performance targets and milestones; and (2) report quarterly on a single website now known as Performance.gov the results achieved for each CAP goal compared to the targets. In February 2012, OMB identified 14 interim CAP goals and subsequently published five quarterly updates on the status of the interim CAP goals on Performance.gov. While updates for eight of the goals included data that indicated performance towards an overall planned level of performance, only three also contained annual or quarterly targets that allowed for an assessment of interim progress. Updates for the other six goals did not report on progress towards a planned level of performance because the goals lacked either a quantitative target or the data needed to track progress. The updates on Performance.gov also listed planned activities and milestones contributing to each goal, but some did not include relevant information, including time frames for the completion of specific actions and the status of ongoing efforts. The incomplete information in the updates provided a limited basis for ensuring accountability for the achievement of targets and milestones. OMB Quarterly Progress Reviews. GPRAMA also requires that OMB—with the support of the Performance Improvement Council (PIC)—review CAP goal progress quarterly with goal leaders. OMB instituted processes for reviewing progress on the goals each quarter, which involved the collection of data from goal leaders and the development of a memorandum for the OMB Director. However, the information included in these memorandums was not fully consistent with GPRAMA requirements. For example, GPRAMA requires OMB to identify strategies for improving the performance of goals at risk of not being met, but this was not consistently done. Without this information, OMB leadership and others may not be able to adequately track whether corrective actions are being taken, thereby limiting their ability to hold officials accountable for addressing identified risks and improving performance. Leading Practices for Reviews. At the CAP-goal level, goal leaders for two CAP goals and one sub-goal instituted in-person progress reviews with officials from contributing agencies that were broadly consistent with the full range of leading practices for reviews, such as leadership involvement in reviews of progress on identified goals and milestones, and rigorous follow-up on issues identified through these reviews. In these cases, goal managers reported there were positive effects on performance, accountability, and collaboration. In contrast, review processes used by other goal leaders did not consistently reflect the full range of leading practices. Effective review processes consistently engage leaders and agency officials in efforts to identify and address performance deficiencies, and to ensure accountability for commitments. Thus, not using them may result in missed opportunities to hold meaningful performance discussions, ensure accountability and oversight, and drive performance improvement. GAO is making seven recommendations to OMB to improve the reporting of performance information for CAP goals and ensure that CAP goal progress reviews meet GPRAMA requirements and reflect leading practices. OMB staff generally agreed to consider GAO's recommendations.