NQF is a nonprofit organization established in 1999 in order to foster agreement, or consensus, on national standards for measuring and public reporting of health care performance data.more than 400 organizations that represent multiple sectors of the health care system, including providers, consumers, and researchers. NQF’s mission focuses on three core areas: (1) building consensus on national priorities and goals for performance improvement and working in partnership to achieve them, (2) endorsing national consensus standards for measuring and publicly reporting on performance, and (3) promoting the attainment of national goals through education and outreach programs. Its membership includes Prior to its contract with HHS, NQF established a consensus development process (CDP) to evaluate available health care quality measures to determine which ones are qualified to be endorsed—that is, recognized— as national standards. Under this process, organizations that develop quality measures submit them to NQF for consideration, in response to specific solicitations by NQF. NQF forms a committee of experts from its member organizations as well as other organizations and agencies to conduct an objective and transparent review of these quality measures against four standardized criteria established by NQF, such as whether the measures are scientifically acceptable. After this committee evaluates the measures against these criteria, NQF’s process allows for a period during which its member organizations and the public may comment on the committee’s recommendation for each measure. The process also provides for a period for its member organizations to vote on whether the measures should be endorsed by NQF as a national standard. Ultimately, NQF’s board of directors makes a final decision on whether NQF should formally endorse the measures. As of October 2011, NQF has endorsed over 600 health care quality measures in 27 areas, such as cancer and diabetes. HHS uses NQF- endorsed measures in its programs and initiatives to promote quality measurement, and NQF continues to endorse quality measures separate from its contract with HHS. NQF’s work under the contract includes endorsement of quality measures and other activities that are expected to support HHS’s quality measurement efforts, such as through value-based purchasing programs. Specifically, NQF’s work under the contract consists of various projects under the nine contract activities related to health care quality measurement. The work plans developed annually to respond to MIPPA and NQF’s technical proposal to respond to PPACA delineate the projects NQF is required to conduct under the nine contract activities, as well as expected time frames and cost estimates for the projects for each year. Table 1 provides more detailed information on the nine contract activities. Some of these activities are required by either MIPPA or PPACA, while others are quality measurement activities established by HHS or administrative activities. To help determine the activities and the projects under the nine contract activities that NQF is expected to perform during each contract year, HHS has established an interagency workgroup that comprises officials from multiple divisions within HHS, including the Agency for Healthcare Research and Quality, the Centers for Medicare & Medicaid Services (CMS), and the Office of the National Coordinator for Health Information Technology. The workgroup is responsible for prioritizing and selecting the activities and projects under each activity that NQF is expected to perform during each contract year. HHS officials told us that the representatives from these various HHS agencies provide input on the work NQF is expected to perform, including determining quality measures requested from NQF for their respective programs. The activities and projects selected by the interagency workgroup become part of NQF’s scope of work under the contract. Some of the projects under the contract activities that NQF is expected to perform during the year will be ongoing from the previous contract year while new work will be incorporated into the work plan as necessary. For the NQF contract, HHS selected a cost-plus-fixed-fee contract, under which HHS reimburses NQF for actual costs incurred under the contract in addition to a fixed fee that is unrelated to costs. Cost-plus-fixed-fee contracts are used for efforts such as research, design, or study efforts where costs and technical uncertainties exist and it is desirable to retain as much flexibility as possible in order to accommodate change. However, this type of contract provides only a minimum incentive to the contractor to control costs. As we reported in 2009, these contracts are suitable when the cost of work to be done is difficult to estimate and the level of effort required is unknown. This cost-plus-fixed-fee contract is NQF’s first cost-reimbursement contract. For cost-reimbursement contracts, the Federal Acquisition Regulation (FAR) requires appropriate government surveillance during performance to provide reasonable assurance that efficient methods and effective cost controls are used. Under the FAR, contracts are to contain a provision for agency approval of a contractor’s subcontracts. HHS’s contract with NQF contains this provision and also requires the approval of consultants engaged under the contract. The review and approval of NQF’s use of subcontractors and consultants require appropriate support documentation provided by NQF to HHS, including a description of the services, the proposed price, and a negotiation memo that reflects the principal elements of the price negotiations between NQF and the subcontractor or consultant. Under its contract with HHS, NQF has utilized 31 subcontractors and 16 consultants since January 14, 2010, to provide support to NQF on many of the contract activities and associated projects. Two HHS components are principally responsible for administering the NQF contract: the office of the Assistant Secretary for Planning and Evaluation (ASPE) and CMS—an agency within HHS. Specifically, the project officer for the NQF contract is a representative of ASPE. This individual is responsible for program management and works with the contracting officer to oversee the contract. The contracting officer for the NQF contract, responsible for administering the contract, is a representative of CMS.contracting officer is required to perform, such as conducting an annual evaluation of the contractor’s performance. From January 14, 2010, through August 31, 2011, NQF made progress on projects under its contract activities. However, our review of NQF documents found that NQF had not met or did not expect to meet time frames on more than half of the projects, and it exceeded its cost estimates for projects under three of the contract activities. HHS did not use all tools for monitoring that are required under the contract. From January 14, 2010, through August 31, 2011, NQF has made progress on 60 of the 63 projects under the activities required under its contract with HHS. Specifically, NQF had completed 26 projects and was (App. III provides the continuing to work on the remaining 34 projects.status of all contract activities and the projects under each activity NQF was expected to perform during our reporting period.) Examples of projects under the contract activities include both completed and continuing projects: Endorsement of Measures Activity. NQF endorsed 101 measures since the beginning of the second contract year by conducting work on endorsement projects on different topic areas. Specifically, NQF completed two projects to endorse 38 outcome measures related to 20 high-priority conditions identified by CMS that account for the majority of Medicare’s costs, and mental health and child health conditions; and 21 performance measures for chronic and postacute care nursing facilities. NQF also worked on two projects related to child health quality and patient safety. As of August 2011, NQF endorsed 41 child health quality measures and 1 patient safety measure under these projects. NQF expected to complete the child health quality project in September 2011 and the patient safety project in December 2011. In addition, NQF completed a contractually required review of its endorsement process, subcontracting with Mathematica Policy Research, Inc. (Mathematica). The review focused on the timeliness and effectiveness of the endorsement process; identified inefficiencies, including those that may contribute to delays; and recommended, among other steps, that NQF create a schedule for its endorsement process for measure developers and develop feasible time lines that include clear goals for each endorsement project. HHS officials stated that Mathematica’s recommendations were valuable because much of the work under the NQF contract needs to be completed in an accelerated timeline to help fill critical measurement gaps associated with HHS’s health care quality programs and initiatives. For more information about this review, see appendix IV. Maintenance of Endorsed Quality Measures Activity. NQF maintained—that is, updated or retired—124 measures under the contract since the beginning of the second contract year. These included 41 measures reviewed under NQF’s 3-year review cycle related to diabetes, mental health, and musculoskeletal conditions. In addition, 83 measures were maintained under NQF’s other maintenance review processes. NQF was also continuing to work on maintenance projects it initiated in 2010 for measures related to cardiovascular and surgery measures. As of August 2011, the two projects were expected to be completed by December 2011 and January 2012, respectively. Promotion of the Development and Use of Electronic Health Records Activity. NQF has made progress on three projects related to retooling—that is, converting previously endorsed quality measures to an electronic format that is compatible with electronic health records (EHR). First, NQF completed initial retooling of 113 measures. This work is intended to allow data from EHRs to be used for quality measurement, which is a part of HHS’s long-term goal to use health information technology to exchange information and improve quality of care. Second, as of August 2011, NQF convened an expert review panel to review the retooled measures to ensure that each retooled measure is properly formatted, the logic is correctly stated, and the intent of the measures is maintained in the electronic format that will use data obtained from EHRs, instead of from claims as originally formatted. Third, as of August 2011, NQF was expected to complete another project to provide an updated list of the 113 retooled measures to HHS by December 2011, which would incorporate any revisions identified by the expert review panel and others involved in the retooling process. After these updated measures are completed, HHS officials told us that they will contract with other entities to conduct testing of some of the 113 retooled measures to assess the feasibility of implementing the measures in the electronic format. Although NQF’s endorsement process requires that measure developers submit data on validity and reliability testing of measures they submit for endorsement, this testing does not include feasibility testing for implementing the measures in an electronic format for performance measurement. As of December 2011, HHS officials did not provide an expected date of completion for this feasibility testing but told us that they have awarded two contracts that include this in their scope of work. In addition to the retooling projects, NQF is developing a software tool—the Measure Authoring Tool—to allow measure developers to create standardized electronic measures that help capture information in EHRs so that less retooling would be needed in the future. As of August 2011, NQF was completing final testing of the beta, or initial, version of this tool. NQF expected to complete testing and publish an updated version for public use by January 2012. Multistakeholder Input into HHS’s National Quality Strategy Activity. NQF convened the National Priorities Partnership (NPP), a multistakeholder group expected to provide annual input on national priorities, among other things, to be considered in the National Quality Strategy. As of August 2011, the NPP was completing a report on this input, which was then published in September 2011. The report noted the need for a national comprehensive strategy that identifies core sets of standardized measures to meet each of the national priorities HHS identified in the 2011 National Quality Strategy, among other things. The NPP noted in the report that a common data platform, core measure set, and public reporting mechanism are key components of the infrastructure for performance measurement. It also highlighted that a strategic plan, road map, and timeline for establishing an infrastructure should be accelerated to allow for rapid implementation over the next 5 years. Additionally, the NPP reported that it was critical that all federal programs drive toward the establishment of a common platform for measurement and reporting. Multistakeholder Input on the Selection of Quality Measures Activity. NQF has convened the Measure Applications Partnership (MAP). The MAP is a multistakeholder group that is expected to conduct work in two areas. First, the MAP is expected to provide input to the Secretary of HHS on the selection of quality measures for use in payment programs and value-based purchasing programs required by PPACA, among others. The MAP will review a list of measures published by the Secretary of HHS on December 1 of each year, and develop a report that contains a framework to help guide measure selection. The MAP will provide its annual input beginning February 1, 2012, for measures used in the following 11 programs: hospice, hospital inpatient, hospital outpatient, physician offices, cancer hospitals, end-stage renal disease (ESRD) facilities, inpatient rehabilitation facilities, long-term care hospitals, hospital value-based purchasing, psychiatric hospitals, and home health care. Second, the MAP is expected to publish reports that provide input on the selection of measures for use in various quality reporting programs, including those for physicians. As of August 2011, the MAP had held meetings and initiated its work for reports due October 1, 2011. Other Health Care Quality Measurement Activity. NQF completed a project to endorse six imaging efficiency measures. NQF was also continuing to work on a project to identify existing quality measures and gap areas related to measurement of regionalized emergency care services. Our review of NQF documents found that NQF had not met or did not expect to meet time frames on more than half of the projects under the contract activities that were completed or ongoing, as of August 2011. Specifically, our review of documents found that NQF had not met expected time frames on 18 of the 26 projects it completed under the nine contract activities. Further, NQF did not expect to meet time frames on 14 of the 34 projects on which it was continuing to work. The delays of these projects under the contract activities varied in time from about 1 to 12 months. HHS officials told us they approved all changes to the time frames, which were established by HHS and NQF in NQF’s 2010 and 2011 final annual MIPPA work plans and the PPACA technical proposal. Appendix III provides the status for all projects related to each of the nine contract activities, including information on their expected and actual time frames for completion during our reporting period. Examples of projects under the contract activities for which NQF did not meet or did not expect to meet expected time frames include the following: Endorsement of Measures Activity. NQF did not meet or did not expect to meet time frames for all five endorsement projects under the endorsement contract activity. (See app. III for details on the five projects.) For example, NQF was expected to complete an endorsement project for nursing home quality measures in July 2010; however, the measures were not endorsed until February 2011. (See fig. 1 for estimated time frames and actual completion dates for all projects related to the endorsement contract activity.) NQF officials stated that several factors contributed to NQF exceeding the expected time frames for the five endorsement projects, including the high volume of measures submitted for review, the amount of time it took to harmonize measures between measure developers, and a need for additional technical expertise on review panels. Promotion of the Development and Use of Electronic Health Records Activity. NQF did not meet or did not expect to meet expected time frames for five out of eight projects related to the EHR contract activity. For example, NQF was expected to complete its initial retooling of 113 endorsed quality measures into electronic formats by September 2010, but this effort was not completed until December 2010. (See fig. 2 for estimated time frames and actual completion dates for all projects related to the EHR contract activity.) In addition, NQF was expected to complete the project to convene an expert panel to review the 113 retooled measures by January 2011. However, the panel did not complete its review of the 113 measures until June 2011. According to HHS and NQF officials, several factors contributed to NQF exceeding expected time frames for the retooling project under the EHR contract activity. HHS officials stated that the first set of 44 retooled measures submitted had errors that required correction. For example, HHS officials stated that they found errors in the electronic coding of these 44 retooled measures requiring NQF and its subcontractors who retooled the measures to make corrections. In addition, HHS and NQF officials stated that after starting the retooling project, they quickly learned that the estimated time frames for the retooling project, as well as other projects related to the EHR activity, were overly ambitious, given the scope and complexity of the work. For example, HHS officials noted that retooling of quality measures into electronic format had never been attempted before and the technical complexity and labor required to complete the project were greater than anticipated. NQF officials also told us that HHS’s requests to modify the scope of work for this project often required changing the time frame for completing the retooled measures. These factors resulted in an extension of the project that delayed the final delivery of the 113 retooled measures as well as contributed to the need for additional staff at NQF. Other Health Care Quality Measurement Activity. NQF was expected to complete two projects under the other health care quality measurement activity related to efficiency and resource use—one white paper on resource use and another on geographic-level efficiency by July 2010. These white papers were intended to provide information for an endorsement project on resource-use measures that began in January 2011. However, as of August 2011, the resource-use paper was still under review by HHS, and NQF officials stated they expected to receive comments in September 2011. The geographic-level efficiency paper was canceled in June 2010 at the request of HHS. NQF initially intended to subcontract the work on these two projects, but officials told us that they were unable to identify a subcontractor at the level of funding approved for this project. As a result, HHS approved NQF’s proposal to complete this work internally. HHS officials stated that the drafts NQF submitted on both topics were poor in quality and did not meet its needs, resulting in HHS requesting additional revisions for the resource-use white paper that delayed its completion, and requesting the cancellation of the geographic-level efficiency white paper. Administrative Activity. NQF did not meet the expected time frames for completing one of the required projects under the administrative activity—finalizing its annual work plan. Specifically, the NQF contract requires NQF to develop an annual work plan and to receive final approval from HHS within the first 4 weeks of each contract year; however, NQF did not meet this requirement in 2010 or 2011. For example, the final 2011 MIPPA annual work plan was not developed by NQF and approved by HHS until April 1, 2011. According to NQF and HHS officials, the 2011 MIPPA work plan was not developed and approved on time due to extended discussions on the scope and cost estimates of NQF’s EHR activities. HHS officials told us that the primary reason for the extended discussions was that they expected the costs to reflect all the work needed to complete the Measure Authoring Tool (MAT) by the end of the second contract year. However, they said that NQF only submitted a beta version of the tool by the end of the second contract year, which was not the version expected by HHS. NQF officials told us that the version was never intended to be final but rather a beta version, consistent with their understanding of HHS’s expectations. As a result, HHS and NQF officials needed to evaluate the scope of work and cost estimates for this and other projects. Further, NQF officials told us the delay in completing the 2011 MIPPA annual work plan resulted in the interruption of NQF’s ongoing work related to the MAT under the EHR contract activity. The delay also delayed its receipt of funding for some new or ongoing work under the contract. In some instances, NQF chose to start new or continue ongoing work with its own funding. For example, NQF officials stated that NQF began work related to the MAP using its own funds until HHS authorized the work. In addition, the delay in completing the 2011 MIPPA work plan resulted in the need to set the start date for fall 2011 rather than earlier in the contract year for some of the projects under the maintenance activity. NQF also exceeded its cost estimates for projects under three of the contract activities. HHS officials told us they approved the changes to the cost estimates and in some cases modified NQF’s scope of work to help ensure that NQF’s costs did not exceed the amount HHS had obligated for the contract activities. NQF officials stated that in certain cases, not meeting expected time frames contributed to NQF exceeding these cost estimates. For example, the delays in projects related to the EHR contract activity, including expanding the scope of the retooling project, contributed to NQF exceeding its cost estimate of about $3.8 million for the entire EHR contract activity by about $560,000 in the second contract year. In another example, the delays in finalizing the 2010 and 2011 MIPPA work plans contributed in part to NQF exceeding its cost estimate for developing and finalizing these plans, which is a project under the administrative contract activity. Specifically, while NQF estimated that completion of the annual work plan would cost approximately $77,000, NQF reported an actual cost of $176,590. In addition, NQF also exceeded its cost estimate for the endorsement contract activity during the second contract year for various reasons, including a need for additional technical experts for review panels. Specifically, NQF exceeded estimated costs of about $3.1 million for the entire endorsement activity by about $146,000 While HHS officials told us they approved in the second contract year.all changes to the cost estimates, in certain cases they reduced the scope of NQF’s work in 2011 to ensure that total available funding for the contract year was not exceeded and that sufficient funding was available for ongoing projects. For example, HHS officials told us that they had hoped to start several new endorsement projects beginning in 2011; however, these projects were not included in the 2011 final annual MIPPA work plan so that funding would be available for NQF to complete its ongoing projects, including work that was delayed under the EHR contract activity. In addition, HHS requested that NQF discontinue its work on one project related to the development of a public website for 2011, which is associated with the administrative contract activity. HHS officials told us that to help monitor NQF’s performance on the projects under the contract activities, they rely on NQF to report any issues, including those related to time frames or cost estimates, in the monthly progress reports that NQF is required to submit to HHS or during phone calls held at least monthly. While HHS monitored NQF’s progress and approved changes to the time frames and cost estimates for the projects under the contract activities, HHS did not use available tools for monitoring that are required under NQF’s contract. These tools could have helped to provide an opportunity for HHS to make any appropriate changes to NQF’s projects. For example, HHS did not conduct an annual performance evaluation required by the contract that would assess timeliness and cost control issues, among other things, for the previous contract year. The results of such an evaluation could help HHS officials to consider potential timeliness and cost issues when determining NQF’s scope of work for the next year. Further, while monthly progress reports and invoices include information on NQF’s costs, these documents do not compare reported costs to initial cost estimates. HHS officials told us that, prior to August 2011, they had not enforced a contractual requirement for NQF to submit—nor had it received from NQF—a financial graph in its monthly progress reports that provides information comparing NQF’s monthly incurred costs for each of the contract activities with initial cost estimates. Instead, HHS officials informally requested that NQF provide them with the financial status of the contract activities in midyear 2010, which helped them to plan for NQF’s work under the contract for 2011. Having a financial graph in the monthly progress report could have helped HHS officials to identify instances where any contract activity was approaching or exceeding NQF’s initial cost estimates prior to HHS’s midyear review. This, in turn, could have provided HHS and NQF an opportunity to adjust estimates of future costs for these or related activities earlier in the contract year. HHS officials had asked NQF to begin to include such a financial graph in its monthly progress reports beginning in August 2011. From January 14, 2010, through August 31, 2011, NQF reported a total of approximately $22.4 million in costs and fixed fees on monthly invoices submitted to HHS for projects under activities conducted in response to MIPPA and PPACA. Specifically, NQF reported about $12.8 million in total costs and fixed fees for the contract activities it performed during the second contract year—January 14, 2010, through January 13, 2011. From January 14, 2011, through August 31, 2011, part of the third contract year, NQF reported an additional $9.6 million in total costs and fixed fees. During the second contract year, the majority of NQF’s reported costs were related to the promotion of the development and use of EHRs (36 percent, or $4.6 million) and endorsement of health care quality measures (26 percent, or $3.3 million). Figure 3 illustrates the costs and fixed fees NQF reported for eight of the nine contract activities we reviewed that occurred during the second contract year. The ninth contract activity relates to multistakeholder input on the selection of quality and efficiency measures, as directed by PPACA. This contract activity did not begin until after January 14, 2011, which is the start of the third contract year. For the part of the third contract year covered in our review—January 14, 2011, through August 31, 2011—almost one-half of NQF’s reported costs were for the activity to promote the development and use of EHRs and for the activity to provide multistakeholder input on the selection of quality and efficiency measures. Each of these activities accounted for 22 percent, or about $2.1 million of NQF’s reported costs. Other costs reported by NQF include those for the activity related to providing multistakeholder input into HHS’s annual National Quality Strategy ($1.55 million, or 16 percent) and those for the activity related to the maintenance of endorsed quality measures activity (13 percent, or $1.29 million). Figure 4 illustrates the costs and fixed fees NQF reported for the part of the third contract year covered in our review for each of the nine contract activities we reviewed. According to HHS, as of August 2011, about $55.2 million remains available for the NQF contract. About $15.1 million in MIPPA funding remains available for work to be conducted through January 13, 2013. In addition, HHS plans to obligate approximately $40.1 million of its PPACA funding through 2014 for NQF’s activities related to health care quality measurement in response to PPACA. For its various programs or initiatives, HHS has used or planned to use about one-half of the quality measures that NQF has endorsed, maintained, or retooled under the contract, as of August 31, 2011, and HHS officials expect to evaluate if and how the remaining measures will be used. However, HHS has not comprehensively determined how it will use NQF’s work under the contract to implement PPACA requirements related to quality measures. According to HHS officials, HHS has used or planned to use about one- half (164) of the 344 health care quality measures it has received from NQF through various endorsement, maintenance, and retooling projects under the contract, as of August 31, 2011. For example, of the 164 measures used or planned for use, 44 were used in CMS’s Medicare and Medicaid EHR Incentive Program after being retooled—that is, converted to an electronic format that is compatible with EHRs—under the NQF contract. Although these 44 retooled measures were used in the EHR Incentive Program, HHS officials stated that NQF and HHS detected coding and other errors in the versions of the 44 retooled measures that were published in the program’s final rule in July 2010 that required NQF to make corrections to them after publication of the final rule. NQF did not submit the revised versions of the 44 retooled measures published in the final rule to HHS until December 2010. HHS officials stated that because the final rule had already been published prior to receiving the final formatted measures, CMS listed general guidance on its website to address the errors. HHS officials told us that these 44 measures are being used but have not yet been tested to assess the feasibility of implementing them in the electronic format. Until the testing is complete, HHS runs the risk that some of these measures may not work as intended when implemented in electronic format for performance measurement. As a result, the agency does not have reasonable assurance that the retooled versions of the measures will correctly capture information from EHRs. In addition to the 44 retooled measures used in the EHR Incentive Program, HHS also has used or planned to use 120 measures that it received from endorsement and maintenance projects under the NQF contract for various HHS programs and initiatives. (See table 2 for details on specific programs in which HHS has used or planned to use health care quality measures received from NQF under the contract.) HHS officials told us that they expect to evaluate if and how they could use all of the remaining 180 of the 344 quality measures that were endorsed, maintained, or retooled under the NQF contract that are not currently in use or planned for use in HHS programs or initiatives. According to HHS officials, any measure developer can submit a measure to be considered for NQF endorsement. Therefore, all the measures received under the contract may not be applicable to a particular HHS health care quality program or initiative. HHS officials told us that they will review the remaining 180 measures to determine if they are applicable to their health care quality programs or initiatives. The officials expect that many of these measures will be used in HHS programs or initiatives required by PPACA. For example, HHS officials told us that they will consider implementation of most of the retooled measures in future stages of the EHR Incentive Program. In addition, PPACA directed HHS to establish a hospital value-based purchasing program, as well as to make plans or begin pilot programs for value-based purchasing in other settings of care. The hospital value-based purchasing program will use various quality measures and depend on the information collected on them to determine payments to providers. PPACA also required the development of no less than 10 provider-level outcome measures for hospitals and physicians by March 2012. Further, PPACA directed HHS to identify quality measures that could be used to evaluate hospice programs and publish these measures by October 1, 2012. HHS officials told us that they are in the process of determining whether or to what extent the remaining 180 measures HHS has received under the NQF contract can be used to address the new measurement needs and priorities established by PPACA. HHS officials told us that they prefer to use NQF-endorsed measures to meet HHS’s measurement needs because these quality measures are nationally recognized standards and in some cases HHS is required to use them. Although HHS has taken steps to determine how it can use the measures received under the contract with NQF, the agency does not have a comprehensive plan for determining how it will use the remainder of the work conducted under NQF’s contract to implement PPACA requirements, including plans for additional quality measures that need to be endorsed during the remaining contract years. HHS officials told us that HHS determines on an annual basis which activities—including work on quality measures—NQF is to perform under the contract through the interagency workgroup. The workgroup is comprised of representatives from various HHS agencies and allows them to provide input on their needs, including quality measures that need endorsement from NQF, for their respective programs. However, HHS officials told us that each HHS program assesses its quality measurement needs separately and provides varying levels of detail about its needs. Therefore, it is unclear the extent to which all programs consistently incorporate PPACA’s quality measurement requirements and deadlines into these assessments. The NPP’s September 2011 report noted the importance of greater alignment of national quality measurement efforts, including the establishment of a comprehensive measurement strategy that identifies core measure sets, among other things. In addition, the report noted that all federal programs should work toward the establishment of a common platform for measurement and reporting. Without a comprehensive plan that delineates HHS’s quality measurement needs, and given that each program assesses its quality measurement needs separately, the interagency workgroup may not be able to systematically ensure that all of HHS’s quality measurement needs that implement PPACA requirements align with the selection and prioritization of activities for NQF to complete under the contract. While HHS has begun various efforts to assess its quality measurement needs, the lack of a plan that comprehensively determines the impact of PPACA on its needs could affect the agency’s progress on its quality measurement efforts as well as how it selects and prioritizes NQF’s contract activities. Officials told us that prior to PPACA’s enactment, CMS maintained a 5-year plan that listed its measurement needs based on agency priorities and the priorities established by the NPP for some of its programs.to reflect the requirements related to quality measurement and time frames established by PPACA. In March 2011, HHS published the National Quality Strategy as required by PPACA, which included six priority areas of focus. The report was required by PPACA to include agency-specific plans, goals, benchmarks, and standardized quality metrics for each priority area, but did not do so. HHS officials stated that this document describes HHS’s initial plan for these elements and that they may be included in future versions of the strategy. In June 2011, HHS officials told us that they plan to convene a Quality Measurement Task Force within CMS with a goal to comprehensively align, coordinate, and approve the development, maintenance, and implementation of health care quality measures for use in various CMS programs. As of August 2011, the task force was in an early stage of development, and therefore it is too early to determine whether it will accomplish its goal. Although these various HHS efforts are key steps toward helping the agency meet its quality measurement needs, they are not guided by a comprehensive plan that synthesizes key priority areas identified in various sources, such as those reported by the NPP or in the National Quality Strategy, for which measures may be needed. Without such a plan, HHS may be limited in its efforts to prioritize which specific measures it needs to develop and have endorsed by NQF for its health care quality programs and initiatives established by PPACA. As a result, HHS may be unable to ensure that the agency receives the quality measures needed to meet PPACA requirements and specified time frames related to quality measurement. Health care quality measures are increasingly important to HHS as it uses and will continue to use them in its existing and forthcoming programs and initiatives to evaluate health care delivery. For example, HHS’s value- based purchasing programs are pay-for-performance programs that will require providers to collect and report information on health care quality measures and adjust payment levels based on providers’ performance against the measures. PPACA has increased HHS’s quality measurement needs, and the time frames specified in the law have also increased the urgency of obtaining endorsed quality measures—which are nationally recognized standards and in some cases are required by statute—to meet these needs. Given that NQF is the entity in the United States with lead responsibility for endorsing health care quality measures, NQF’s endorsement activities under the contract are of key importance to help meet HHS’s quality measurement needs. However, NQF’s endorsement process takes time. For more than half of the projects, including all five projects in the endorsement activity, NQF did not meet or did not expect to meet the initial time frames approved by HHS. In addition, projects under three of the contract activities have exceeded initial cost estimates, which resulted in HHS’s modification of NQF’s scope of work in some instances to help ensure that NQF’s costs did not exceed the funding allocated for the contract activities. While HHS received information in monthly progress reports to help monitor NQF’s performance under the contract, the agency did not use all of the monitoring tools required under the contract to help address issues related to time lines and cost estimates. These monitoring tools included an annual performance evaluation that could help HHS officials consider potential issues related to NQF’s time frames and cost estimates when planning work for the next year and a financial graph to be included in NQF’s monthly progress reports. The graph would have compared reported costs to initial cost estimates, which is something that monthly progress reports do not do. Although HHS officials reported that they recently began in August 2011 to enforce the contractual requirement for NQF to submit the graph, they have not implemented the required annual performance evaluation. By not taking advantage of these tools, HHS runs the risk of not having detailed and timely information that could help identify instances in which NQF might be at risk of not meeting time frames or exceeding estimated costs. Identifying such instances could provide an opportunity for HHS to make any appropriate changes to NQF’s scope of work, including setting priorities to ensure that HHS receives the quality measures it needs in a timely manner. With the time remaining under the contract, HHS has an opportunity to ensure that the work performed under NQF’s contract better meets the agency’s needs for its programs and initiatives. However, HHS has not developed a plan that comprehensively identifies its quality measurement needs for its programs and initiatives in light of PPACA’s requirements or determines how it will use the work conducted during the remaining years of the NQF contract to help it meet these needs. In addition, critical tasks may need to be completed outside of the NQF contract. For example, HHS requested that NQF retool 113 measures under the contract and used 44 of the 113 measures that included errors in its EHR Incentive Program. As of November 2011, feasibility testing related to implementation of the retooled measures had not been completed, and HHS expected to perform this work outside of the NQF contract. Until the testing is completed, HHS runs the risk that some of the retooled measures may not work as intended when implemented in electronic format for performance measurement, which is a concern because use of these measures is an important component of HHS’s long-term goal for providers to use health information technology (IT) to exchange information and improve the quality of care. Without a comprehensive plan, HHS lacks assurance that its selection of the work to be performed by NQF—and the approximately $55.2 million that the agency expects to spend for remaining work under the NQF contract—will be prioritized in the most effective way possible. Given that PPACA includes time frames for the implementation of quality measurement programs, NQF’s pace in completing some of the work under the contract—particularly the endorsement activity—raises concerns. If the endorsement projects continue to require extended completion times, HHS runs the risk of not having all the endorsed measures it needs for implementing its programs and initiatives. Should this occur, HHS may need to select nonendorsed measures for its programs and initiatives that have not undergone an objective and transparent review by NQF. To help ensure that HHS receives the quality measures it needs to effectively implement its quality measurement programs and initiatives within required time frames, we recommend that the Secretary of HHS take the following three actions: use monitoring tools required under the NQF contract to obtain detailed and timely information on NQF’s performance and use that information to inform any appropriate changes to time frames, projects, and cost estimates for the remaining contract years; ensure that testing of the electronic versions of the measures retooled by NQF that are being used or are planned for use in the Medicare and Medicaid EHR Incentive programs is completed in a timely manner to help identify potential errors and address issues of implementation; and develop a comprehensive plan that identifies the quality measurement needs of HHS programs and initiatives, including PPACA requirements, and provides a strategy for using the work NQF performs under the contract to help meet these needs. We provided a draft of this report to HHS and NQF for review and comment. HHS neither agreed nor disagreed with our recommendations and provided general comments. NQF concurred with many of the findings in the report and provided clarification and additional context on the findings and recommendations. HHS and NQF’s letters conveying their comments are reproduced in appendixes V and VI, respectively. In addition to the overall comments discussed below, we received technical comments from HHS and NQF, which we incorporated into our report as appropriate. HHS’s comments included separate general comments from CMS and ASPE that provided context on aspects of our findings and recommendations. CMS’s comments stated that the draft report suggests that CMS must use all of the measures endorsed by NQF, and noted that not all NQF-endorsed measures are suitable for HHS quality reporting and public reporting programs. Although our draft report did not state that CMS must use all of the measures endorsed by NQF, we modified it to note specifically, among other things, that all measures received under the contract may not be applicable to a particular HHS health care quality program or initiative. CMS also stated that the report suggests that CMS has not developed measurement plans for various provisions of PPACA related to quality reporting, public reporting, and value-based purchasing programs. CMS provided additional context for current planning efforts to address these requirements, including its Quality Measurement Task Force. The draft report acknowledged this and other CMS planning efforts to address the health care quality requirements contained in PPACA and noted that, as of August 2011, this initiative was just beginning. Further, while various efforts are underway and CMS’s comments state that it has documented how quality measures will be used to address all relevant provisions of PPACA, CMS has not provided documentation of comprehensive plans to address PPACA requirements that include alignment across programs, detailed time frames to meet PPACA deadlines, or how it will use the NQF contract to help ensure that it receives the endorsed measures it needs to meet these requirements. ASPE’s comments noted, with respect to our first recommendation, that HHS used all except two of the monitoring tools called for in the contract. As noted in the draft report, HHS began receiving the monthly financial graph—one of the two monitoring tools—from NQF in August 2011. Also, ASPE noted its plans to update its performance evaluation system with NQF performance information for the first 2 contract years—the period January 14, 2009, through January 13, 2011—and to complete a final performance evaluation at the end of the contract in January 2013, which is the end of the fourth contract year. It did not indicate any plans to conduct the annual performance evaluation for the third contract year— January 14, 2011, through January 13, 2012—which would be consistent with the contract’s requirements. With respect to our second recommendation, ASPE provided technical comments and also told us that CMS issued a contract solicitation to test the retooled measures, but CMS did not receive any bids. Instead, ASPE noted in its comments that two of CMS’s current contractors will conduct feasibility testing on 69 of the 113 retooled measures that are planned for use in HHS’s EHR Incentive programs. CMS does not plan to issue a solicitation for a new contract to test the feasibility of the remaining 44 retooled measures, which are currently being used in HHS’s EHR Incentive Program. We noted these comments in the report. Regarding our third recommendation, ASPE stated that the measures that are not currently in “use” are being evaluated by HHS and that any conclusions that they will not be used are not accurate. Our draft report provided information on which measures were used or planned for use as of August 2011, and indicated that the remaining measures may be used in the future. Specifically, the report noted that HHS officials expect that many of these measures will be used in HHS programs or initiatives, and that HHS officials told us that they will review all the measures received under the contract to determine if they are applicable to their health care quality programs or initiatives. ASPE’s comments also noted that our draft report did not include information on all NQF-endorsed measures used by the various agencies within HHS. As noted in the draft report, we relied on HHS to identify programs and initiatives across HHS that use or plan to use these health care quality measures and recognize that those included in our report may not represent a comprehensive list of all health care quality programs and initiatives. As we recommended in our report, having a comprehensive plan could help HHS identify programs or initiatives that use or plan to use health care quality measures, including those endorsed by NQF. NQF’s comments state that it is providing its services to HHS under a cost reimbursement contract, which is used in circumstances where aspects of performance, such as time frames, cost estimates, and scope of work, cannot be reasonably estimated, and therefore, should not be expected. As noted in the draft report, the contract type used for this work is used for efforts such as research, design, or study efforts where costs and technical uncertainties exist and it is desirable to retain as much flexibility as possible in order to accommodate change. However, the draft report also noted that this type of contract provides only a minimum incentive to the contractor to control costs. Given the risk associated with this type of contract, the fact that NQF has not met expected time frames on about half of its projects as of August 2011, and that NQF exceeded its initial cost estimates for some of its projects under its contract activities, it is especially important that HHS obtain detailed and timely information on NQF’s performance and use that information to inform any appropriate changes to time frames, projects, and cost estimates for the remaining contract years, as noted in our recommendations. NQF’s comments also state that time frames and costs for the work performed under the contract were initial estimates based on an early understanding of the work, that HHS and NQF understood that there would likely be changes to them as a result of the complexity and novelty of the work, and that they have worked collaboratively throughout the contract period to address these and other factors. As noted in the draft report, the final work plans, the technical proposal, and other documents that we reviewed included initial time frames for all projects and costs for the work performed during the contract year that were approved by HHS in collaboration with NQF. The draft report also notes several examples of reasons why the time frames and costs were modified over time. Contributing factors include the high volume of measures submitted, changes to the scope of work, and the novelty and complexity of the work. We are sending copies of this report to the Secretary of Health and Human Services and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at kohnl@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs can be found on the last page of this report. Other major contributors to this report are listed in appendix VII. Figure 5 illustrates a health care quality measurement framework of the various stages that a quality measure will go through, as described by the Department of Health and Human Services (HHS) and National Quality Forum (NQF) officials and others. These stages include measure development, endorsement, selection, and use, among others. This framework also shows examples of which entities, including HHS and NQF, are involved in each of the stages. As an example of actions taken during the second stage of this quality measurement framework, HHS officials described two different processes used for planning and identifying gap areas. The Centers for Medicare & Medicaid Services (CMS) Office of Clinical Standards and Quality has developed a standardized approach to identify quality measures that it uses in its health quality initiatives and programs using CMS’s Measures Management System. The Measures Management System requires the convening of a technical expert panel in the initial planning stage. Once convened, the technical expert panel is expected to work with measure developers who will gather information that will help the panel determine whether measures need to be developed for a program or initiative. During this stage, measure developers may conduct environmental scans or literature reviews, to determine the existence of measures that could be used for a program or initiative. If a measure does not exist, then the developer will work with CMS to develop the needed measures for the program or initiative, including measure testing. Upon development of the measures, the technical expert panel will evaluate them based on (1) importance to making significant gains in health care quality and improving health outcomes, (2) scientific acceptability of the measure properties including tests of reliability and validity, (3) usability, and (4) feasibility. Measures recommended by the panel are generally submitted for NQF endorsement. In contrast, CMS’s Center for Medicaid, Children’s Health Insurance Program (CHIP), and Survey & Certification Office—the CMS center which implements CHIP—uses a measure identification process that relies on existing measures rather than development of new measures, according to officials. This office worked with a technical advisory group, the Subcommittee on Children’s Healthcare Quality Measures for Medicaid and CHIP, to recommend an initial core set of measures for the CHIP. With assistance from CMS, the subcommittee evaluated measures based on importance, validity, and feasibility. CMS officials told us that they considered existing NQF-endorsed and non-NQF-endorsed measures based on the measurement needs of the program, and relied on measure testing conducted by the measure developers. Officials stated that they have also relied on the subcommittee to evaluate candidate measures for Medicaid child health programs. Officials said that they are not required to submit measures that will be used for Medicaid programs for NQF endorsement. From January 14, 2010, through August 31, 2011, the National Quality Forum’s (NQF) contract with the Department of Health and Human Services (HHS) included 16 tasks that NQF is required to perform. For purposes of our work, we categorized these tasks into nine contract activities. Specifically, in certain cases, we grouped activities that covered related areas of work into a single contract activity. For example, we consolidated the six administrative activities NQF is required to perform into a single contract activity. (See table 3 that shows how we consolidated these contract activities.) NQF was required to perform specific projects under the nine contract activities we identified. For example, under the endorsement contract activity, NQF was required to complete an endorsement project related to patient outcome measures. For purposes of our work, we identified and reviewed 63 projects NQF is required to perform under the nine contract activities, as shown in appendix III. The tables below provide a status update on the projects that the National Quality Forum (NQF) is required to complete under the nine contract activities we identified (see app. II). The contract activities and the projects under the activities NQF is expected to perform are determined on an annual basis by the Department of Health and Human Services (HHS) and NQF. As a result, the number of projects under the contract activities varies by contract year. For our reporting period—January 2010 through August 2011—we determined that NQF was required to conduct work on 63 projects under the contract activities we reviewed. To determine initial time frames for each project, we calculated the approximate time between expected start and end dates established in NQF’s 2009, 2010, and 2011 final annual Medicare Improvements for Patients and Providers Act of 2008 (MIPPA) work plans, the 2011 Patient Protection and Affordable Care Act (PPACA) technical proposal, and other NQF documents. Actual time frames were determined by calculating the approximate time between the actual start date and the actual date of completion. For projects that were not yet complete as of August 2011, we included an expected time frame based on the approximate difference between the actual start date and the expected date of completion. NQF and HHS officials stated that any changes to the initial time frames were approved by HHS. As part of a project under its contract with the Department of Health and Human Services (HHS), NQF was required to review its endorsement process. To complete this project, the National Quality Forum (NQF) subcontracted with Mathematica Policy Research, Inc. (Mathematica), to conduct a review of NQF’s endorsement process, as requested by HHS. HHS officials stated that, given the importance of the endorsement process as part of the health care quality measurement framework, they requested that an objective and thorough review of NQF’s endorsement process that focused on timeliness, efficiency, and effectiveness should be conducted. For example, they stated that they were interested in whether there were any efficiencies that could be implemented to shorten the process while maintaining an objective review of the health care quality measures that were evaluated under the process. Mathematica initiated its review of NQF’s endorsement process in October 2009 and completed the work in December 2010. In December 2010, Mathematica submitted a final report to NQF and recommended eight areas where improvements could be made and inefficiencies could be addressed in the endorsement process. In the final report, Mathematica noted that the current process is lengthy and the timeliness of the endorsement projects varies substantially. The report further noted that the length of the endorsement process affects the availability of endorsed measures for end users, such as HHS. To help reduce the time required to complete projects, Mathematica recommended that NQF create a schedule for its endorsement process for measure developers and develop feasible time lines that include clear goals for each endorsement project. As of May 2011, NQF officials stated that NQF has taken steps or plans to take steps in its future projects to address the eight areas for improvement Mathematica identified. For example, as of May 2011, NQF has solicited measures earlier based on a tentative annual project schedule to reduce the time lines of its endorsement process and reduced the period for voting by NQF member organizations from 30 to 15 days. NQF officials stated that they believe their efforts to implement the recommendations will shorten the time lines for the endorsement projects by 3 to 4 months without compromising the integrity of the endorsement process and measures to be evaluated under the process. HHS officials stated Mathematica’s recommendations were valuable because much of the work under the NQF contract needs to be completed in an accelerated time line to help fill critical measurement gaps associated with HHS’s health care quality programs and initiatives. They noted that it is too soon to tell the effects of these changes on the endorsement process, but they plan to monitor implementation of the changes in NQF’s 2011 endorsement projects under the contract. In addition, as of September 2011, HHS approved a new project under the contract to identify how the endorsement process can best align with HHS’s time frame for needed measures. As part of this project, NQF is expected to work with a consulting group to identify key performance metrics and define milestones and time lines to help streamline its endorsement process. In addition to the contact named above, Will Simerl, Assistant Director; La Sherri Bush; Krister Friday; Amy Leone; Carla Lewis; John Lopez; Elizabeth Martinez; Lisa Motley; Teresa Tucker; Carla Willis; and William T. Woods made key contributions to this report.

Summary: The Medicare Improvements for Patients and Providers Act of 2008 (MIPPA) directed the Department of Health and Human Services (HHS) to enter into a 4-year contract with an entity to perform various activities related to health care quality measurement. In January 2009, HHS awarded a contract to the National Quality Forum (NQF), a nonprofit organization that endorses health care quality measuresthat is, recognizes certain ones as national standards. In 2010, the Patient Protection and Affordable Care Act (PPACA) established additional duties for NQF. This is the second of two reports MIPPA required GAO to submit on NQFs contract with HHS. In this reportwhich covers NQFs performance under the contract from January 14, 2010, through August 31, 2011GAO examines (1) the status of projects under NQFs required contract activities and (2) the extent to which HHS used or planned to use the measures it has received from NQF under the contract to meet its quality measurement needs, as of August 2011. GAO interviewed NQF and HHS officials, reviewed relevant laws, and reviewed HHS and NQF documents. NQF has made progress on projects under its contract activities, as of August 2011. Specifically, NQF has completed or made progress on 60 of 63 projects. For example, NQF has completed projects to endorse measures related to various topics, including nursing homes. However, for more than half of the projects, NQF did not meet or did not expect to meet the initial time frames approved by HHS. For example, NQF completed one project to retool measuresthat is, convert previously endorsed quality measures to an electronic format. While the retooling project was expected to be completed by September 2010, its completion was delayed by 3 months. NQF and HHS officials identified various reasons that contributed to this delay, including an expansion of the projects scope and complexity. As a result of the delay, HHS did not have all the retooled measures it expected to include in its Electronic Health Records (EHR) Incentive Program. The delay of this project was also a contributing factor to NQF exceeding its estimated cost for its entire contract activity related to EHR by about $560,000 in the second contract year January 14, 2010, through January 13, 2011. While HHS monitored NQFs progress through monthly progress reports and approved changes to time frames and costs, HHS did not use all of the tools for monitoring that are required under the contract. Specifically, HHS did not conduct an annual performance evaluation to assess timeliness and cost issues that could have helped to inform NQFs future scope of work. Until August 2011, HHS did not enforce the provision for NQF to submit a financial graph to compare monthly costs for each contract activity with cost estimates, which is information not included in monthly progress reports. These tools could have provided additional, more detailed information to help identify instances in which NQF might have been at risk of not meeting time frames or exceeding cost estimates, which could have provided HHS an opportunity to make any appropriate changes to NQFs activities. HHS had used or planned to use about half of the measures164 of 344that it received from NQF under the contract, as of August 2011. For example, HHS used 44 measures that NQF retooled under the contract in its EHR Incentive Program. HHS officials stated that the 44 measures used in the program contained errors, which required corrections. HHS officials also have not yet tested the retooled measures to assess the feasibility of implementing them in the electronic format; therefore, HHS runs the risk that some of these measures may not work as intended when implemented. HHS officials told GAO they expect to evaluate if and how they could use all of the remaining measures HHS received under the contract. However, HHS has not determined how PPACA requirements for quality measurement may have changed its needs for endorsed quality measures. As a result, HHS has not established a comprehensive plan that identifies its measurement needs and time frames for obtaining endorsed measures and that accounts for relevant PPACA requirements. Without such a plan, HHS may be limited in its efforts to prioritize which specific measures it needs to develop and to have endorsed by NQF during the remainder of the NQF contract. As a result, HHS may be unable to ensure that the agency receives the quality measures needed to meet PPACA requirements, including time frames for implementing quality measurement programs. GAO recommends HHS: (1) use all monitoring tools required under the contract to help address NQFs performance, (2) complete testing of retooled measures, and (3) comprehensively plan for its quality measurement needs. HHS neither agreed nor disagreed with these recommendations. NQF concurred with many of the findings in the report and provided additional context.