GPRAMA requires agencies to publicly report on how they are ensuring the accuracy and reliability of the performance information they use to measure progress towards APGs and performance goals. Priority Goals and Performance Measures: GPRAMA requires agencies to identify their highest priority performance goals as APGs and have ambitious targets for these APGs that can be achieved within 2 years. measures to track the progress they are making on achieving their APGs or identify alternative ways of measuring progress, such as milestones for completing major deliverables for the APG (for more information on the measures and milestones the selected agencies identified, see appendix I). 31 U.S.C. § 1120(b). GPRAMA uses the term “measured values” instead of performance data. We define verification as the assessment of completeness, accuracy, consistency, timeliness, and related quality control practices for performance information. We define validation as the assessment of whether performance information is appropriate for the performance measure. For more information, see, GAO, Performance Plans: Selected Approaches for Verification and Validation of Agency Performance Information, GAO/GGD-99-139 (Washington, D.C.: July 30, 1999). the level of accuracy required for the intended use of the data; any limitations to the data at the required level of accuracy; and how the agency will compensate for such limitations (if needed) to reach the required level of accuracy. GPRAMA requires agencies to provide information to OMB that addresses all five requirements for each of their APGs for publication on a website (Performance.gov). Agencies also must address all five requirements for performance goals in their performance plans and reports. GPRAMA states that Performance.gov shall consolidate information about each APG, thereby making this information readily accessible to the public, members of Congress, and congressional committees. GPRAMA makes OMB responsible for Performance.gov and requires agencies to provide OMB with quarterly updates on their APGs, including how they are ensuring the quality of performance information, for publication on Performance.gov. Further, GPRAMA continues transparency requirements set in GPRA that require agencies to publish annual performance plans and reports (see text box). While GPRAMA requires certain information to be reported in performance plans and certain other information to be reported in performance reports, the Reports Consolidation Act of 2000 authorizes agencies–with the concurrence of OMB–to consolidate performance plans and reports into a single publication that covers past actual and future planned performance. GPRAMA’s Requirements for Agencies’ Annual Performance Plans and Reports Performance plans should identify the planned level of performance for the current fiscal year and the next fiscal year, explain how the agency will ensure the accuracy and reliability of its performance information, identify the agency’s priority goals (APGs), and be published every February, concurrent with the President’s Budget. summarize the actual level of performance agencies achieved during the previous five fiscal years, explain how the agency ensures the accuracy and reliability of its performance information, and be published every February. Guidance and Information Sharing on Implementing GPRAMA: OMB provides guidance to agencies in Circular A-11 on how to implement GPRAMA. OMB updates A-11 annually, and the most recent update was published in June 2015. GPRAMA also established in law an interagency council–the PIC–chaired by OMB and composed of agency PIOs to facilitate the exchange of useful practices to strengthen agency performance management, such as through cross-agency working groups. The selected 23 APGs we reviewed are intended to drive progress in important and complex areas, such as assisting veterans, addressing climate change, and protecting workers. Given the significance and complexity of many APGs, congressional and public understanding regarding how federal agencies are measuring and assessing progress toward these goals is important. GPRAMA requires agencies to publicly report on how they are ensuring the accuracy and reliability of the performance information they use to measure progress towards these APGs. However, our review found that overall, it would be challenging for Congress and the public to understand how the selected agencies are ensuring that the performance information they report for their 23 APGs is accurate and reliable–that is, suitable for making judgments about agency progress or decisions for different courses of action. We found limited information on Performance.gov on the quality of performance information used to assess progress on the six selected agencies’ 23 APGs. While each agency has a section dedicated to its priority goals on Performance.gov, there is no place on the website that is set aside to discuss the quality of performance information for each APG. The six agencies we reviewed used various sections of Performance.gov to discuss some of the performance information quality requirements for APGs. But, none of the agencies addressed all five GPRAMA requirements for their individual APGs. Moreover, while we found hyperlinks from Performance.gov to the selected agencies’ performance plans and reports, there was no explanation on Performance.gov of where to find performance information quality discussions in these plans and reports. We discussed our preliminary findings with OMB staff in January 2015. In response, OMB updated its A-11 guidance in June 2015 to direct agencies to either provide information for publication on Performance.gov of how they are ensuring the quality of performance information for their APGs, or provide a hyperlink from Performance.gov to an appendix in their performance report that discusses the quality of their performance information. OMB staff stated that this information will likely not be available until agencies start reporting on the next set of APGs (for fiscal years 2016 and 2017). This is because OMB will need to update a template that agencies complete for their Performance.gov updates. OMB staff confirmed in July 2015 that they are still using a version of this template that they provided to us in January 2015 that has not yet been updated to reflect this change. Five of the six agencies’ performance plans and reports we reviewed did not describe how they ensured the quality of performance information for their individual APGs. On the other hand, all six agencies did describe how they ensured the quality of their performance information overall. Of the 23 APGs in our sample, we could only find performance information quality discussions that addressed all five of the GPRAMA requirements for 3 APGs, which belonged to the Department of Homeland Security. The U.S. Department of Agriculture (USDA) Provided Some Information for How Performance Information Quality Is Ensured For One of Three APGs, but Did Not Address All Requirements USDA identified its APGs and briefly summarized the results achieved for each APG in its performance report for fiscal year 2014. Further, USDA’s performance report provided data quality discussion for each of the performance measures presented in the 2014 report, which included the two performance measures USDA used to measure progress on its Reduce the Number of Foodborne Salmonella Illnesses APG. However, USDA did not address all of the GPRAMA requirements for this APG or its two other APGs, as shown by table 1. For the Reduce the Number of Foodborne Salmonella Illnesses APG, USDA addressed two of the five GPRAMA requirements. For example, USDA identified the Centers for Disease Control and Prevention (CDC) as the source of the data measuring the number of illnesses from products USDA’s Food Safety and Inspection Service regulates. USDA also noted that CDC receives information from state and local health agencies concerning outbreaks of illnesses. USDA acknowledges that the quality of the data can vary by reporting agency, which is an example of identifying a potential limitation. While the fiscal year 2014 performance report did not explain how USDA and CDC are compensating for this limitation, USDA did provide a hyperlink in its performance report to a CDC web page that provided more detailed information about tracking and reporting of foodborne illnesses. Further, while USDA described a number of steps it is taking to reduce illnesses and detect contamination in food products, USDA did not explain to external audiences what level of accuracy it requires to make decisions related to this APG. Our prior work has identified improving oversight of food safety as a high-risk area, emphasizing the need to improve planning and collaboration among USDA and other federal food safety agencies. This makes it important for USDA to expand its performance information quality discussion and address all requirements for this APG. USDA’s performance plans and report did not address any of the five GPRAMA requirements regarding the quality of the performance information for its two other APGs: Create New Economic Opportunities and Improve the Health of Our Nation’s Soils. The lack of information related to the two other APGs makes it more challenging for Congress and the public to understand how USDA is ensuring the quality of performance information, including potential limitations. For example, USDA’s Inspector General identified the need to develop effective performance measures as a management challenge facing the agency and raised concerns about the accuracy of some performance information. But, USDA’s performance plans and report do not address this issue with regard to two of its APGs.officials in USDA’s Office of Budget and Program Analysis, and they acknowledged their agency could improve its public reporting on the quality of its performance information for APGs. We shared our analysis with USDA’s performance report for fiscal year 2014 contained an explanation on how the agency ensures the quality of its performance information overall, which is reproduced below (see text box). This helps external audiences understand that USDA’s methodology for collecting performance information has been vetted by scientists and policymakers. U.S. Department of Agriculture’s Statement on Performance Information Quality–2014 Annual Performance Report The data used by the Department to measure performance are collected using a standardized methodology. This methodology has been vetted by federally employed scientists and policymakers, and, ultimately, the Under Secretaries of the respective mission areas. All attest to the completeness, reliability, and quality of the data. The Department of Defense (DOD) Highlighted Progress for All APGs, but Did Not Explain How Performance Information Quality Is Ensured DOD included performance discussions for all of its APGs and stated whether the agency had met its interim or final APG targets in its performance reports for fiscal years 2013 and 2014; however, it did not address the performance information quality requirements for each APG, as shown in table 2. While DOD’s fiscal year 2013 and 2014 performance reports do not describe agency-wide guidance for ensuring performance information quality, we found an explanation of DOD’s guidance regarding performance information quality in DOD’s “2014 Performance Plan Update.” DOD officials said this document was intended to serve as the agency’s performance plan for fiscal year 2015. As reflected in table 2, the document provided a description of DOD’s APGs for fiscal years 2014 and 2015 and contained a brief statement addressing agency-wide data verification and validation practices. It states that, “at the beginning of the fiscal year, goal leaders provide action plans and verification and validation forms on each performance goal listed in the .” Officials in the Office of the Deputy Chief Management Officer said that DOD’s fiscal year 2014 performance plan update constituted their agency’s fiscal year 2015 performance plan, although this purpose is not clearly explained in the document. DOD officials told us their agency would publish its 2016 performance plan as part of its agency strategic plan, which was intended for publication by the end of summer 2015. The Secretary of Defense signed this plan covering fiscal years 2015-2018, which is dated July 31, 2015. Our review of the plan indicated that it described DOD’s performance management process. However, it did not explain how DOD will address all of the performance information quality requirements for each of the agency’s APGs. In addition, the plan did not make clear which sections of the plan were intended to address the requirement to publish a performance plan establishing performance goals for 2016. Department of Defense, Annual Energy Management Report: Fiscal Year 2014 (May 2015). related to the GPRAMA requirements, such as on their verification and validation processes. Improving the public reporting of performance information quality is important because DOD’s Reform the DOD Acquisition Process and DOD Financial Statement Audit Readiness APGs address areas that we have identified as high risk. For the acquisition APG, we reported in 2014 that DOD expects to invest $1.5 trillion in its portfolio of major defense acquisition programs, making it particularly important that DOD explains how it is ensuring the quality of performance information for this APG. Similarly, given that DOD is responsible for more than half of the federal government’s discretionary spending, we reported that it is particularly important DOD has accurate, timely, and useful financial information. The reliability of DOD’s financial information and ability to maintain effective accountability for its resources will be increasingly important to the federal government’s ability to make sound resource-allocation decisions. The Department of Homeland Security (DHS) Addressed GPRAMA Requirements in Explaining How Performance Information Quality Is Ensured for All APGs DHS presented information about performance information quality for all three of its APGs in its performance plans and reports, which included detailed discussion for 10 of the 14 performance measures used to measure progress on these APGs. Specifically, DHS published an appendix to its performance plans and reports with detailed performance information quality discussion for these measures. For each measure, DHS’s appendix describes the related program, the scope of the data, the source and collection methodology for the data, and an assessment of data reliability. For example, for the number of convicted criminal aliens that Immigration and Customs Enforcement removes from the country for the Enforce and Administer Our Immigration Laws APG, DHS addresses the requirements for explaining verification and validation. It explains how headquarters staff looks for unusual patterns in data field offices have entered into a database tracking removals. DHS also states it conducts additional checks by cross-referencing data on removals reported by detention facilities and field offices, and says a statistical tracking unit does further checks. Related to the intended use of performance information, DHS explains for each performance measure discussed in this appendix how it uses the measure for decision making. For example, DHS explains that its measure of the number of convicted criminal aliens removed from the country reflects the “full impact” of its program activities in this area. This helps a reader understand that DHS will need a high level of accuracy to ensure that its programs are achieving its goals for this area. We found examples of DHS acknowledging potential limitations for some APG performance measures. For example, DHS acknowledged that the average number of days to process inquiries from individuals experiencing difficulties with travel screening for its Aviation Security APG does not include the time DHS is waiting for the traveler to submit all required documents. This helps a reader understand that the performance measure may not reflect the total number of days it takes to resolve issues impeding an individual from traveling. DHS addresses the final requirement–how the agency will compensate for limitations to reach the required level of accuracy, if needed–by stating that each APG performance measure discussed in the appendix is reliable. Further, DHS also provided more specific explanation of corrective actions for some performance measures. For the Ensure Resilience to Disasters APG, DHS acknowledges that there is some variation in how states and territories assess their capabilities for dealing with disasters. But, it also explains that federal officials provide technical assistance and review the submissions from state and territorial officials to ensure that they align with guidance the Federal Emergency Management Administration provides to states and territories for how to assess their capabilities. In addition, DHS’s performance plans and reports explained how the agency ensures the quality of its performance information overall. DHS states it has an agency-wide performance management framework, which includes a process for verifying and validating its performance information. For example, DHS explains that one of the steps it takes is to have an independent review team assess the completeness and reliability of its performance measurement data. By addressing the five GPRAMA requirements for its APGs, DHS’s performance plans and reports helped external audiences better understand how it ensures the accuracy and reliability of performance information for the agency’s highest priority performance goals. The Department of the Interior (Interior) Described How Performance Information Quality is Ensured Overall, but Did Not Address GPRAMA Requirements for its APGs Interior’s performance plan and report covering fiscal years 2014 through 2016 provided an overall explanation of how it verified and validated its performance information, which is reproduced in figure 1. Interior’s plan and report states that it requires component agencies to have verification and validation processes, and referred to a more detailed document on the website of Interior’s Office of Policy, Management, and Budget “Data Verification and Validation Standards.” These standards provide direction to component agencies on how to verify and validate performance information and other aspects of performance information quality. For example, the standards explain that component agencies should document data sources, describe the accuracy limits of data, and identify data limitations. Interior shared these standards in June 2015 with other agencies participating in a Performance Improvement Council cross- agency working group on data quality. Interior’s performance plans and reports did not explain how performance information quality was ensured for its individual APGs. As shown above, Interior’s statement on verification and validation is written at a high level and does not explain the specific steps component agencies took to ensure that performance information for each APG was accurate and reliable. Interior’s Deputy Performance Improvement Officer noted that the performance plans and reports discussed the agency’s performance in mission areas that relate to the APGs, and thereby provided contextual information that would allow the public to understand the quality of performance information for these APGs. Thus, they provided information on performance targets and past performance for some of the performance measures related to APGs. The available contextual information in the performance plans and reports did not address all of the GPRAMA performance information quality requirements for each APG. For example, related to its Water Conservation APG, which aims to increase the available water supply in the western states, Interior identifies the number of people and farmers the Bureau of Reclamation delivers water to, and reports on the acre feet of water conservation capability enabled through Reclamation’s programs. While this helps external audiences understand the importance of this APG and the related mission area, Interior does not explain how it is ensuring that it is accurately measuring the water conservation capability enabled through Reclamation’s programs. Interior explains that operating regulations require inspection of leases which produce high volumes of oil or natural gas and those leases that have a history of noncompliance at least once a year. These inspections help ensure that hydrocarbon production on federally managed lands are properly accounted for and results in accurate royalty payments to the public and Indian owners of such minerals. federal leases. Also, this revenue is one of the federal government’s largest nontax source of revenue. The Department of Labor (Labor) Described the Overall Quality of its Performance Information, but Not on How Quality Is Ensured for its Individual APGs Labor’s performance reports contained limited discussion of performance information quality for its APGs, as table 5 shows. The reports referred readers to its Summary of Performance and Financial Information, which includes an attestation statement from the Secretary of Labor as to the reliability and completeness of the agency’s performance information (we reproduce these statements in the text boxes below). These statements provide the reader with the Secretary’s assurance of the quality of Labor’s performance information. However, these statements do not describe what practices are in place to ensure that the agency is using accurate and reliable performance information to measure and report on progress for its individual APGs. Department of Labor’s Data Completeness and Reliability Statement–Fiscal Year 2014 Annual Performance Report The Fiscal Year 2014 Summary of Performance and Financial Information includes an assessment by the Secretary of the reliability and completeness of DOL (Labor’s) performance data reported under the GPRAMA. The Department satisfies this requirement with Agency Head-level attestations that the data do not contain significant limitations that can lead to inaccurate assessments of results. The Secretary of Labor’s Data Quality Attestation Statement–Fiscal Year 2014 Summary of Performance and Financial Information Secretary’s Attestation Statement I attest that the summarized financial and performance data included in this document as well as the data in the Agency Financial Report and the Annual Performance Report are complete and reliable in accordance with Federal requirements. When we shared our analysis with Labor officials, they said that they were not fully aware of the performance information quality requirements for APGs. However, they emphasized that their agency places considerable emphasis on ensuring the quality of its performance information, and on conducting program evaluations to assess the effectiveness of its programs. Labor officials referred us to other agency publications–The Department of Labor FY 2014-2018 Strategic Plan and The Department of Labor FY 2016 Congressional Budget Justification–for explanation to the public of the activities they said their agency was taking to ensure the quality of its performance information. We confirmed that these publications do describe the agency’s research and evaluation agenda. For example, Labor states in its strategic plan that it is committed to improving the quality of performance information by conducting future evaluations to ensure the outcome data it reports are accurate. While Labor’s strategic plan does not explain the extent to which these data quality studies will focus on issues related to its APGs, Labor does provide valuable information and important context on how it plans to ensure the quality of its performance information. Nevertheless, GPRAMA’s performance information quality requirements are important because our previous work on fragmentation, overlap, and duplication shows that Labor needs to report more transparently on program performance for its veterans’ employment and training programs. This relates to one of Labor’s APGs on improving employment outcomes for veterans. Specifically, we reported in 2012 that Labor provided Congress with an annual veterans’ program report that provided certain performance information, such as the number of disabled and recently separated veterans who received intensive services. But, we found that Labor was not reporting these results relative to the performance goals it had set. We recommended in our prior work that Labor report both performance goals and associated performance outcomes for its veterans’ employment and training programs. Labor agreed with our recommendation and has made some progress in addressing it. For example, Labor has reported on how the results achieved for the performance measure it uses to measure progress on one of its APGs (percent of veterans receiving intensive services served by Disabled Veterans Outreach Program specialists) compare to the targets it set. Labor provided this information for this performance measure in both its performance report and on Performance.gov. The National Aeronautics and Space Administration (NASA) Described Its Overall Approach for Ensuring Performance Information Quality, but Does Not Explain How Performance Information Quality for its APGs Is Ensured NASA’s performance plans and reports characterize the agency as a performance-based organization committed to managing toward specific, measurable goals, and to using performance information to continually improve operations. As shown in table 6, these performance plans and reports explain how the agency ensures the quality of its performance information overall. For example, NASA stated in its performance plans and reports that it held internal reviews for its projects, determined technology readiness levels, and required mission directorates and mission support offices to submit evidence supporting all performance measure ratings.such as scientific review committees and aeronautics technical evaluation bodies, to help it validate program performance. These statements provide valuable insight into how NASA measures its performance and uses evidence. Further, NASA stated that it used external entities, However, NASA’s performance plans and reports did not explain how the agency ensured the quality of performance information for individual APGs. NASA presented concise summaries of each APG, progress updates, and next steps. However, there was little explanation provided for external audiences which described how NASA took the approach it has outlined for ensuring the overall quality of its performance information, and applied this approach to individual APGs. In our discussions with NASA officials, they emphasized that they do collect information related to all of the GPRAMA requirements. To illustrate this, NASA officials demonstrated their internal Performance Measure Manager system to us in April 2015. NASA officials told us that the system functions as a warehouse for agency performance information and they upload information from the system to Performance.gov for quarterly APG updates and to help develop their performance plans and reports. They showed us that this system collects information on the quality of performance information for a range of performance measures, including for APGs. For example, the internal database has a field for verification and validation materials for the James Webb Space Telescope APG, and identifies a data limitation for it. However, NASA does not publicly report all of the information the system collects on how it ensures the quality of performance information for its APGs. NASA officials expressed concern about how well the GPRAMA performance information quality requirements can be applied to their agency’s performance reporting. NASA officials said that they use what GPRAMA and Office of Management and Budget Circular A-11 refer to as an alternative form of performance measurement that, among other things, allows an agency to use milestones for completing major deliverables for the APG instead of performance measures. They further explained that they use numerous milestones to measure progress on their APGs. They added that NASA only reports on key quarterly milestones in its performance plans and reports and on Performance.gov. Unlike other agencies, they noted they often do not have quantitative data sets for their performance information. However, the GPRAMA performance information quality requirements apply to all APGs, even if the agency is using milestones. As we noted in our 2015 High Risk update, NASA plans to invest billions of dollars in the coming years to explore space, understand Earth’s environment, and conduct aeronautics research. We designated NASA’s acquisition management as high risk in 1990 in view of NASA’s history of persistent cost growth and schedule slippage in the majority of its major projects. Going forward, we noted in our February 2015 high risk update that it will be critical for NASA to ensure adequate and ongoing assessments of risks related to two of its APGs for developing new systems for exploring deep space and the James Webb Space Telescope. However, as our review shows, NASA has not explained to external audiences how it is ensuring the quality of performance information for these APGs related to these high-risk areas. Without such information, it will be more difficult for Congress and the public to understand whether NASA is effectively measuring progress toward these APGs, and whether the billions of dollars being spent to accomplish these important efforts are being used effectively. In 2015, OMB and the Performance Improvement Council (PIC) established the Data Quality Cross-Agency Working Group. This group met for the first time in February 2015; and four other meetings were held in April, May, June, and July 2015. As of June 2015, PIC staff reported that a total of 12 agencies were participating, which is more than half of the agencies with APGs. Three of the six agencies we selected for review–the Departments of Defense and Homeland Security (DHS) and NASA–are participating. DHS and NASA officials told us that they have made or plan to make presentations at these meetings on their agencies’ performance information quality processes. In addition, Interior’s Deputy Performance Improvement Officer shared his agency’s verification and validation standards with the group. An additional nine agencies–the Departments of Commerce, Education, Health and Human Services, Justice, Treasury, and Veterans Affairs, and the Environmental Protection Agency, Small Business Administration, and Social Security Administration–are also participating in the group. According to meeting notes for the May 2015 meeting provided by OMB and PIC staff, the group had identified several goals: improve the reliability and quality of performance information and of the reporting process; set standards and develop consistency across agencies; and highlight good performance measures and accurate and appropriate performance information. The May 2015 meeting notes state the group’s end product will be to identify solutions agencies have used to solve a data quality problem. In June 2015, the PIC’s Executive Director explained that while the group is still working on defining this end product, it wants to develop a collection of useful and leading practices that can be shared with agency officials. The PIC’s Executive Director and her staff also noted that this end product could include providing recommendations to OMB on changes that could be made to the A-11 guidance on how to address GPRAMA’s performance information quality requirements. OMB staff also indicated to us that they would like to get input from the group on additional changes that could be made to A-11. PIC staff told us that the participating agencies met with OMB staff in July 2015 to discuss additional changes that should be made to A-11. For more than two decades, agencies have been required to publicly report on the quality of their performance information in annual performance plans. More recently, agencies have also been required to report on Performance.gov on how they will ensure the accuracy and reliability of the performance information used to measure progress on each of their highest priority performance goals, the APGs. However, insufficient progress has been made. While OMB for several years has directed agencies to discuss the quality of APG performance information in their annual performance plans and reports, the selected agencies’ plans and reports often did not. OMB recently changed its guidance to require agencies to provide this information for publication on Performance.gov. The next key step is to build upon this recent guidance to implement the change and make this important information readily accessible to the public and Congress. This overall lack of transparency means that members of Congress, citizens, journalists, and researchers seeking information about agency performance related to priority goals have to search in multiple places, and often end up finding no explanation of the quality of performance information for APGs. For agencies to maintain the confidence of Congress and the public that they are indeed achieving their priority goals for the challenging and complex results they seek to achieve, agencies will need to provide more transparent explanations of how they are ensuring the accuracy and reliability of performance information for their APGs. More broadly, our review shows that five agencies continue to provide limited information in their annual performance plans and reports concerning the quality of performance information for their APGs. The same is true for all six agencies on Performance.gov. In some cases, the needed context and information may be available within the agency for the agency’s use. However, this information is not consistently provided to external audiences. The Performance Improvement Council’s (PIC) Data Quality Cross-Agency Working Group provides a potential forum for agencies to collaborate and share information on this topic, and the group is defining its intended end product. Given the shortcomings our review identified at the majority of the six agencies reviewed, the working group could help agencies identify practices that will help them more clearly explain to Congress and the public how they are ensuring that the performance information for their highest priority performance goals is accurate and reliable. OMB could also work with this PIC working group to continue updating its guidance to agencies to ensure that this information is readily accessible on Performance.gov. To improve the public reporting about how agencies are ensuring the quality of performance information used to measure progress towards their priority goals, we recommend the following actions: The Secretaries of Agriculture, Defense, Homeland Security, Interior, and Labor, and the Administrator of NASA should more fully address GPRAMA requirements and OMB guidance by working with OMB to describe on Performance.gov how they are ensuring the quality of performance information used to measure progress towards their APGs. The Secretaries of Agriculture, Defense, Interior, and Labor, and the Administrator of NASA should more fully address GPRAMA requirements and OMB guidance by describing in their agencies’ annual performance plans and reports how they are ensuring the quality of performance information used to measure progress towards their APGs. To help participating agencies improve their public reporting, we recommend that the Director of OMB, working with the PIC Executive Director, should: Identify additional changes that need to be made in OMB’s guidance to agencies related to ensuring the quality of performance information for APGs on Performance.gov. Identify practices participating agencies can use to improve their public reporting in their performance plans and reports of how they are ensuring the quality of performance information used to measure progress towards APGs. We provided a draft of this report to the Director of OMB and the Secretaries of Agriculture, Defense, Homeland Security, Interior, and Labor, and the Administrator of NASA. The Department of the Interior and NASA concurred with the recommendations directed to them, and discussed specific actions they plan to take to address these recommendations. Interior and NASA’s written responses are reproduced in appendixes II and III. In its response, NASA also shared a concern about how we portrayed its high-risk reporting in the draft report. It stated that our draft report suggested that the select milestones identified as part of its performance reporting are the sole mechanisms NASA uses to assess risks and measure progress towards launching the James Webb Space Telescope, and developing new systems for human exploration of deep space. NASA further stated that to comply with reporting requirements related to APGs, it has opted to provide information on key quarterly milestones that the public can easily understand. To address NASA’s concern, we revised the report to recognize that NASA officials told us they use numerous milestones to measure progress on their APGs. NASA also said that it only reports on key quarterly milestones in its performance plans and reports, and on Performance.gov. NASA also provided technical clarifications, which we incorporated as appropriate. The Department of Homeland Security (DHS) also concurred with the recommendation directed to it. However, DHS stated that it has already taken action to implement our recommendation to work with OMB to describe on Performance.gov how DHS is ensuring the quality of performance information used to measure progress towards its APGs. Thus, DHS regards the recommendation as resolved and closed. DHS stated that on July 1, 2015, agency officials in its Office of Program Analysis and Evaluation provided OMB with several specific suggestions to consider as possible enhancements to the internal system that OMB uses to gather agency data for public posting on Performance.gov. This will allow agencies to include more comprehensive data quality information on this public website. DHS’s efforts are an important step toward addressing our recommendation. However, as our review found, and DHS recognizes in its response letter, more will need to be done to make DHS’s explanations of performance information quality for its APGs accessible to external audiences on Performance.gov. For example, in our report we noted that OMB’s updated A-11 guidance in June 2015 gives agencies the option of providing a hyperlink from Performance.gov to an appendix in their performance reports containing their performance information quality discussion, which DHS could do. We will continue to monitor DHS’s efforts to work with OMB to fully implement the recommendation. DHS’s written comments are reproduced in appendix IV. The Department of Defense (DOD) partially concurred with the recommendations directed to it. DOD stated that it has ongoing actions to improve the quality of performance information, and to make better use of that information in management. However, DOD stated that it did not agree that making discussion of the process of managing the quality of performance information a part of either the agency strategic plan or annual reporting has any major management value. We disagree. First, GPRAMA does not require, nor did we recommend, that DOD provide information on the quality of performance information for its agency priority goals in its agency strategic plan. Second, GPRAMA does require that this information be provided in agency performance plans and reports and on Performance.gov. We continue to believe it is important for DOD to fully address these GPRAMA requirements because, as described in our report, two of DOD’s APGs address areas we have identified as high risk. Also, DOD is responsible for more than half of the federal government’s discretionary spending. DOD’s written comments are reproduced in appendix V. The Departments of Agriculture (USDA) and Labor did not comment on the recommendations directed to them. However, they both discussed specific actions they plan to take to improve the quality of their publicly- reported performance information for their agency priority goals. In comments relayed to us in an August 14, 2015, e-mail from the Associate Director of USDA’s Office of Budget and Program Analysis, who is also the agency’s Performance Improvement Officer (PIO), he stated the agency would ensure that a description of the quality of performance information be added for each performance measure included in its APGs for fiscal years 2016 and 2017. He also stated that USDA will work with OMB to put this information on Performance.gov or in its annual performance plan and report with a reference to that information on Performance.gov. He also stated a reference to this information would be provided in USDA’s annual performance plan. He also provided us with a technical clarification, which we incorporated. In its response, Labor raised a concern about statements in our draft report regarding information on several of its programs that serve veterans, which were drawn from our prior work. Labor stated that the report incorrectly asserts that it does not report on the number of veterans receiving intensive services relative to performance goals. Rather, Labor stated that it has used the Veterans’ Employment and Training Services measure for the percent of veterans being served by the Disabled Veterans’ Outreach Program as an APG for 4 years, and has included how results relate to performance goals. Additionally, Labor asserted that outcomes for the Veterans Workforce Investment Program have been included in the annual report to Congress in fiscal years 2013 and 2014. We revised the report to reflect Labor’s updated actions as appropriate. Labor’s written comments are reproduced in appendix VI. In an August 26, 2015, e-mail from OMB’s liaison to GAO, OMB did not comment on the recommendations, but provided technical clarifications, which we incorporated as appropriate. We are sending copies of this report to the Director of OMB and the heads of the agencies we reviewed as well as appropriate congressional committees and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-6806 or mihmj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. J. Christopher Mihm, (202) 512-6806 or mihmj@gao.gov. In addition to the contact named above, Sarah E. Veale, Assistant Director, and Michael O’Neill, Analyst-in-Charge, supervised the development of this report. Virginia Chanley, Emily Christoff, Erik Kjeldgaard, and A.J. Stephens made significant contributions to all aspects of this report. Deirdre Duffy and Robert Robinson provided additional assistance.

Summary: Federal agencies have not always clearly and transparently explained to Congress and the public how they ensure the quality of their performance information. GPRAMA requires agencies to publicly explain how they ensure the accuracy and reliability of their performance information used to assess progress for their APGs. This is one of a series of GAO reports examining the implementation of GPRAMA, as required by the act. This report assesses how well selected agencies publicly reported on the quality of performance information used to measure progress on APGs. GAO selected six agencies–the Departments of Agriculture, Defense, Interior, and Labor, and NASA and DHS– based on GAO's 2013 federal managers survey on their agency's use of performance information. GAO reviewed information concerning these agencies' APGs published on Performance.gov and in their annual performance plans and reports. The six agencies GAO reviewed generally did not publicly report on how they ensured the accuracy and reliability of performance information used to measure progress on their highest priority performance goals, referred to as agency priority goals (APGs). The GPRA Modernization Act of 2010 (GPRAMA) requires agencies to identify the following when publicly reporting on their APGs: 1) how performance information was verified and validated; 2) data sources; 3) level of accuracy required for intended use; 4) any limitations at the required level of accuracy; and 5) how the agency will compensate for such limitations (if needed) to reach the required level of accuracy. GPRAMA requires agencies to provide this information to the Office of Management and Budget (OMB) for publication on Performance.gov. GPRAMA also directs agencies to provide this information for performance goals, which include APGs, in their annual performance plans and reports. While all six agencies described how they ensured the quality of their performance information overall, GAO found discussions about performance information quality addressing all five GPRAMA requirements in only the Department of Homeland Security's (DHS) performance plans and reports. Source: GAO analysis of selected agencies' performance plans and reports. | GAO-15-788 OMB and the Performance Improvement Council (PIC)–a cross-agency council of agency performance improvement officers–established the Data Quality Cross-Agency Working Group in February 2015. The group has identified several goals, such as improving the reliability and quality of performance information, and could serve as a vehicle for disseminating good practices in public reporting on data quality. GAO recommends that all six of the agencies work with OMB to describe on Performance.gov how they are ensuring the quality of their APGs's performance information, and that all agencies, except for DHS, also describe this information in their annual performance plans and reports. GAO also recommends that OMB, working with the PIC, focus on ways the PIC's data quality working group can improve public reporting for APGs. OMB did not comment on the recommendations, but the six agencies generally concurred or identified actions they planned to take to implement them.