PPACA requires HHS to perform several duties related to CER, including disseminating, training, and building data capacity for research. (See table 1.) Although PPACA did not direct HHS to complete these duties by a specified deadline, it appropriated funds to the Patient-Centered Outcomes Research Trust Fund (PCORTF) through fiscal year 2019 to enable HHS and PCORI to implement their respective requirements. PPACA specified that 20 percent of the amounts appropriated or credited to PCORTF be transferred to the Secretary of HHS in each of fiscal years 2011 through 2019. In total, HHS estimates that about $731 million will be transferred to AHRQ (16 percent of the PCORTF) and about $190 million will be transferred to ASPE (4 percent of the PCORTF). With the exception of the amounts transferred to HHS, PPACA designates the remaining PCORTF funds for PCORI’s CER work—an estimated $3.5 billion from fiscal year 2010 through fiscal year 2019. AHRQ has taken some steps to disseminate CER as required under PPACA, including the creation of systematic reviews to develop CER findings, tools to disseminate CER, plans for a website to list and provide links to research databases that include CER, and plans for receiving feedback from stakeholders to whom information is disseminated. However, AHRQ has yet to take other actions that would help it address all PPACA dissemination requirements. AHRQ has taken some steps to implement the law’s key requirements for disseminating federally funded CER, that is to (1) broadly disseminate— develop and distribute—CER in consultation with NIH, (2) create tools that organize and disseminate research findings to certain targeted stakeholder groups, (3) develop a publicly available database, and (4) establish a process for receiving feedback from entities to which information is disseminated. From fiscal year 2012 through 2013, AHRQ has obligated about $37 million of the estimated $731 million it expects to receive through 2019 from the PCORTF on its dissemination activities. Development and distribution of CER findings. AHRQ contributes to the dissemination of CER in various ways, including through the development of systematic reviews, technical briefs, and research summaries that explore the benefits and harms of treatments. In particular, a key method to disseminate CER is through systematic reviews—syntheses of existing research that compare the effectiveness and harms of different healthcare interventions. A systematic review is an assessment and evaluation of all research studies that address a particular clinical issue. Researchers use an organized method of locating, assembling, and evaluating a body of literature on a particular topic. Systematic reviews typically include a description of the findings from the research studies. AHRQ identifies topics for systematic review of CER, such as cardiovascular disease and arthritis, by evaluating topics nominated by individuals or groups against program selection criteria, in order to determine if the topic is appropriate or not appropriate for review. In addition to using its own criteria to identify CER topics for systematic reviews and dissemination, AHRQ documentation states that the agency will consult with experts, such as those from NIH, and review literature to determine whether any similar systematic reviews of relevant studies have already been conducted by other agencies or research organizations in order to reduce potential duplication. Topics selected for a systematic review are further refined with input from key stakeholder groups, technical experts, and patients to develop focused research questions. According to AHRQ officials, research funded by PCORI is not yet included in these systematic reviews because PCORI research is not yet complete. For each systematic review AHRQ synthesizes CER findings from existing research, and the agency disseminates these findings to various targeted stakeholder groups. From June 2012 to June 2014, AHRQ synthesized CER findings through 74 systematic reviews. (See appendix I for a listing of the 74 systematic reviews for which AHRQ disseminated CER findings.) Once a systematic review is complete, AHRQ follows procedures included in its dissemination guidance materials to develop a marketing plan that identifies key messages and targeted stakeholder groups, as well as the types of dissemination mechanisms it will use to conduct outreach. AHRQ officials told us they distribute CER results generally by using the same mechanisms as we previously reported in 2012. These mechanisms include social media, as well as AHRQ’s website and AHRQ’s Effective Healthcare Program website. According to AHRQ officials, the agency determines which specific mechanisms will be used to disseminate CER results by considering the unique characteristics of the research, such as its type, potential impact, and stakeholder groups most likely to use its findings. For example, CER identified as being of particular interest to specific specialties may be disseminated to certain clinical professional associations. Tools to organize and disseminate CER. AHRQ’s marketing plans include various informational tools to disseminate CER. Informational tools include (1) patient decision aids that walk patients through options and choices that patients should consider in working with their clinicians to make informed health care decisions; (2) continuing education and medical education modules to help clinicians understand and use CER findings; (3) slide sets to assist clinicians, researchers and other health professionals with education and training needs; and (4) short, plain- language research summaries that communicate research findings to clinicians, consumers, caregivers, and policymakers. For example, the marketing plan for the systematic review titled, Childhood Exposure to Trauma: Comparative Effectiveness of Interventions Addressing Maltreatment, was developed for a systematic review that examines evidence about interventions for maltreated children. The marketing plan included the specific informational tools to be used to disseminate this project’s findings, such as research summaries for clinicians, a summary of treatments for parents and caregivers, a continuing education module for health care providers, and a slide presentation on the topic. They on all CER that has been conducted. National Library of Medicine officials told us that they have informally consulted with AHRQ on its plans and agree with this approach. In November 2014, AHRQ officials told us that they were sharing their planned approach with senior HHS officials for review and approval. Feedback and evaluation process. As required by PPACA, AHRQ officials told us they receive feedback on dissemination efforts and materials from stakeholders, both formally and informally. For example, officials said that for some of their projects, AHRQ convenes focus groups and advisory panels to assess the needs of stakeholder groups and determine how best to disseminate materials. Some stakeholders we spoke to told us that they have provided feedback to AHRQ on materials the agency has disseminated; however, they were uncertain about the extent to which their feedback was incorporated into AHRQ’s dissemination efforts. AHRQ conducted a feedback assessment and issued a March 2012 feedback report that highlighted stakeholders perspectives about the agency’s disseminated materials. In this report, AHRQ noted that although there is a growing awareness about its disseminated materials, clinicians raised concerns about the timeliness of the information included in the materials, among other things. Officials told us that the agency may conduct future feedback assessments, but they do not know when these will occur and which targeted stakeholder groups will be included. AHRQ also has funded an evaluation to assess its CER dissemination activities and materials supported by the Recovery Act. In September 2013, IMPAQ International—the contractor that conducted the evaluation—issued presentation slides as its final report. The evaluation indicated that stakeholders’ exposure to AHRQ’s CER information, such as the number of website visits and dissemination materials requested, increased over time with AHRQ’s dissemination efforts. The final report also included feedback from certain stakeholder groups through focus groups and surveys. For example, clinicians who participated in focus groups indicated that they typically had little to no experience with the CER information that AHRQ disseminates to clinicians, and suggested that AHRQ more visibly promote the benefits and credibility of this information and then integrate the results and products into existing, easy-to-access sources of medical information focused on point-of-care decision-making. AHRQ officials told us that they plan to award a contract to evaluate the CER dissemination mechanisms—along with the materials they use to share CER findings—that they continued under PPACA. This evaluation project, according to officials, is under development as staff and senior leadership determine the objectives and methods for the study. Although AHRQ staff have not documented their plans as of November 2014, they told us that the evaluation is likely to measure progress on process and intermediate outcome goals of dissemination activities—similar to the last CER evaluation conducted for Recovery Act investments where the agency assessed the level of awareness, understanding, use, and perceived benefits of CER. Officials said the evaluation will also address longer term goals, such as improving health care practice. AHRQ has not taken other actions to help it fully address requirements for disseminating CER in PPACA. Specifically, AHRQ has not taken actions to help it fully address (1) the time frames for disseminating CER, (2) how it will disseminate to all targeted stakeholder groups, (3) its implementation plans for the publicly available database, and (4) how it will coordinate with NIH. Time frames for certain aspects of the dissemination process have not been identified and documented. Although AHRQ has outlined its dissemination process in various documents, it has not clearly identified and documented time frames for one of its key dissemination activities— to implement marketing plans and distribute associated informational tools. According to GAO’s Standards for Internal Control in the Federal Government, significant events need to be clearly documented to ensure management goals are carried out.which together describe the key activities of its dissemination process, including the steps the agency takes to identify key CER findings from systematic reviews, draft and finalize its marketing plans, and distribute its informational tools to the public. While certain AHRQ documents highlight time frames associated with key dissemination activities, we did not identify any documents that specify time frames for when the marketing plans are to be implemented and associated informational tools are to be distributed to stakeholder groups. Once the marketing plans are finalized, the informational tools are to be distributed to targeted stakeholder groups after results of the research have been posted online, such as publication in a major journal. AHRQ officials said they would expect to distribute the informational tools as soon as the results of the AHRQ has several documents research have been posted; however, the dissemination guidance materials we reviewed did not specify time frames for the completion of the implementation of the marketing plans and distribution of informational tools. Without identifying and documenting time frames for these key activities, AHRQ cannot ensure that CER findings are disseminated in a timely manner or that the dissemination process is consistently implemented by all parties. Setting time frames is especially important for dissemination given the length of time and uncertainty inherent in applying CER findings; the large volume of CER research expected from PCORI in the near future, which will increase AHRQ’s dissemination responsibilities; and the need to maximize the investment of PCORTF appropriations made through fiscal year 2019. Dissemination plan for some stakeholders identified in PPACA has not been clearly defined. Additionally, AHRQ has not determined how it will disseminate information to certain stakeholder groups identified in law, and its dissemination to some of these groups has been limited. While AHRQ’s marketing plans include informational tools aimed at most of the targeted stakeholder groups— physicians, health care providers, patients, and appropriate professional associations—federal and private health plans, and vendors of health information technology focused on clinical decision support are not included.dissemination to all of the targeted stakeholder groups, AHRQ may be missing opportunities to reach the key stakeholder groups identified in the law. Although as of October 2014 there were no specific marketing plans that identified private or federal health plans to receive disseminated CER information, AHRQ officials told us they have conducted outreach to these groups. For example, we spoke to a representative at a private health plan who confirmed receipt and use of AHRQ disseminated CER materials. For federal health plans, AHRQ officials said that they worked with the Office of Personnel Management, which manages the Federal Employees Health Benefits Program, and this program encouraged health plans to use an AHRQ report on the comparative effectiveness of autism treatments when determining coverage decisions. AHRQ officials noted that some health plans told them that CER information without a corresponding cost analysis is insufficient in informing coverage Without a defined plan for decisions. Officials also told us that AHRQ found challenges translating CER findings into clinical decision support applications; plans are underway to determine next steps. Implementation plans for addressing the requirement to create a publicly available database have not been documented. As of November 2014, AHRQ officials also have not developed and documented a specific implementation plan to create a publicly available database for CER. GAO’s Standards for Internal Control in the Federal Government state that management should compare actual performance to plans, and as previously noted, should document significant events. The agency formerly acknowledged its plan to address the PPACA requirement to build a publicly available database during our prior work in 2012,existing databases has not been documented and is in the process of being fully vetted with senior leadership. Additionally, while AHRQ officials told us that their instructions on how to search databases for CER will be aimed at the general public, they have not yet determined how effective these tactics will be to meet the needs of various user groups, such as non-researchers who may be unfamiliar with research databases. For example, officials have not determined if or how they may seek feedback from potential users or test the instructions or search terms to see if they meet potential users’ needs. Additionally, AHRQ officials told us they have not determined how to address potential limitations with this new approach. Without taking steps to develop and document an implementation approach that includes time frames and strategies to address potential limitations and AHRQ’s plans to assess whether its tactics meet the needs of various users, the agency does not have reasonable assurance that it will implement the PPACA requirement in a timely or effective manner. but AHRQ has since modified this plan, and the new plan to use NIH’s consultation role regarding AHRQ’s dissemination efforts is unclear. AHRQ is required by law to consult with NIH regarding dissemination efforts, and agency officials told us they meet informally with NIH staff. NIH officials concurred. AHRQ officials said that they have had interactions with NIH on specific dissemination projects of interest to specific NIH institutes or centers, such as the National Cancer Institute. AHRQ and NIH have not determined what role NIH should take in the dissemination process, or which NIH officials should be involved. Previous GAO work has identified key practices that can help federal agencies collaborate effectively when they work together to achieve goals. This work highlighted, for example, the importance of agreeing on roles and responsibilities and establishing compatible policies, procedures, and other means to operate across organizational boundaries. While coordination between the two entities has been informal and limited to specific NIH institutes or centers at this time, AHRQ officials told us that there is a designated AHRQ official that serves as a liaison to NIH to work on this effort. Additionally, AHRQ officials told us that the agency’s senior management is currently working with NIH to determine how best to more formally coordinate on AHRQ’s dissemination activities, but the officials did not state when this effort will be complete. Without specific plans on how it will collaborate, AHRQ officials lack reasonable assurance that they have buy in from NIH regarding dissemination activities or that their independent efforts are not unnecessarily duplicative. As required by PPACA, AHRQ has implemented a training program aimed at individual researchers and academic institutions that is designed to increase the supply and expertise of CER investigators. Through this program, AHRQ awards grants to support graduate training on CER, career enhancement of beginning and midcareer investigators who utilize CER methods, and institutional CER teaching programs. (See table 2.) AHRQ provides grants to individuals it selects and also to institutions that can select a number of individuals to train on CER. During the planning stages for AHRQ’s training program, AHRQ officials told us they consulted with NIH staff members with expertise on the design and management of training grants. An AHRQ official told us that funding will continue for the existing grants awarded to date through 2018. For example, there are currently some training awards that AHRQ will continue to fund through 2018. However, because AHRQ’s allocation from the PCORTF is scheduled to end in 2019, AHRQ officials told us that they do not expect to create or initiate additional individual grants. Additionally, AHRQ does not expect that additional funding announcements will be made for the institutional grants, since these grants are on a 5-year cycle with current grants running through 2018. For any grant on a 2-year cycle, there will likely be new awards made, but only up until 2018. In order to monitor the various training grant awards funded since 2012, AHRQ collects progress reports from training grantees on an annual basis. AHRQ officials told us that participants learn about CER methods and apply what they learn to conduct research projects as part of their training. AHRQ requires that grantees annually submit progress reports to assess their performance on these activities. These reports include performance information, such as (1) a description of career development and research-related activities undertaken; (2) a list of accomplishments including publications, scientific presentations, dissemination activities, new collaborations, inventions, or project-generated resources made; (3) any methodological changes implemented; (4) key preliminary findings from research; and (5) an annual evaluation statement of the award recipient’s progress by the mentor. AHRQ officials told us that they are considering an interim evaluation of the training grant program for fiscal year 2016 and an overall evaluation after the program is complete in fiscal year 2019. Officials stated that they expect to document specific details about their plans before the evaluations occur, which would be consistent with findings in our prior work that a plan for data collection and evaluation is a key attribute of effective training and development programs and can guide an agency in a systematic approach to assessing effectiveness and efficiency. AHRQ officials emphasized that the training program is ongoing and grantees are not yet expected to have outcomes. For these evaluations, they have collected baseline data from progress reports and they plan to collect additional data once the grant program ends to help inform their evaluations, such as a recipient’s promotion and tenure status to measure academic progress. ASPE has coordinated among various agencies to fund projects intended to build data capacity for CER. However, its approach to building data capacity for CER lacks key elements, such as defined objectives, milestones, and time frames, that are necessary to ensure effectiveness. ASPE officials have coordinated and funded projects that they say will help build data capacity for CER. According to ASPE officials, building CER data capacity involves improving data infrastructure, such as facilitating the creation of new health data sets or the sharing of existing health data via the creation of needed standards, services, policies, federal data, and governance structures. ASPE officials say the agency intends these projects to enable interoperable data networks that could support the efficient collection, linkage, and analysis of data for CER from multiple sources. ASPE officials told us that the agency’s goal is to identify a number of investment opportunities through fiscal year 2019 for enabling the development of a CER data infrastructure using funds from the PCORTF. Beginning in fiscal year 2013, ASPE officials worked with the Office of the National Coordinator for Health Information Technology (ONC) to develop a strategic road map to guide both the identification and selection of ASPE’s PCORTF projects beginning in fiscal year 2014 through fiscal year 2019. The strategic framework for the road map, completed in January 2014, specified five component types—standards, services, policies, federal data, and governance structures—necessary to build CER data capacity. As of October 2014, ASPE has funded a total of 10 projects. (See appendix II for descriptions and funding amounts for the 10 ASPE projects.) ASPE has obligated about $23 million of the total estimated $190 million it expects to receive through FY 2019 from the PCORTF. Prior to the development of the road map, ASPE worked with HHS’s Leadership Council, responsible for overseeing ASPE’s PCORTF investment process, to identify and fund new projects that utilized the expertise of an HHS agency. Some projects extended the work of existing Recovery Act projects, with the initial projects beginning in 2011. These projects focused on developing new or enhancing existing data resources, such as expanding administrative and clinical data sets for CER and establishing health information technology standards to leverage electronic health records for CER. For example, ASPE funded a new project conducted by ONC known as the Structured Data Capture initiative. For this project, ONC identifies standards for common data elements that consist of structured data definitions and electronic case report forms, to capture patient data from electronic health records for CER studies. ASPE’s approach to building data capacity for CER through investments in data infrastructure lacks key elements necessary to ensure its effectiveness. Specifically, ASPE updated the strategic framework for the road map in February 2014, but did not define specific objectives linked with performance metrics or establish milestones and time frames that could be used to gauge its progress toward the goal of coordinating relevant federal health programs to build data capacity, as required by PPACA. Without these key elements, ASPE may be unable to gauge its progress towards meeting the requirements of the law. Standard practices for project management call for agencies to conceptualize, define, and document specific goals and objectives in the planning process, along with the appropriate steps, milestones, time frames, and resources needed to achieve those results. Although the updated February 2014 strategic framework for the road map highlighted a purpose—to identify a set of investment opportunities for developing CER data infrastructure to build CER data capacity—and included guiding principles and objectives, it did not clearly define those objectives, nor did it include other elements such as milestones or time frames that would help allow for monitoring and reporting on progress. Specifically, ASPE identified several guiding principles, such as ensuring that data infrastructure projects are “non-duplicative of other related federal and non-federal investments” and “achieve synergy with PCORI and AHRQ.” It also included priority objectives, such as further enabling the collection of standardized clinical data, but many of the objectives were broad and not clearly defined—and did not specify milestones or time frames—as would be consistent with effective project management. Although ASPE identified and considered related, ongoing federal and non-federal data infrastructure investments in an attempt to identify needs or gaps, opportunities where contributions could be made, and ways to avoid duplication, its strategic road map was unclear on the timing and level of coordination necessary for its investments to work together with existing projects—such as PCORI’s PCORnet initiative—to improve data capacity. For example, ASPE officials were not clear on how precisely the standards for common data elements resulting from the ONC Standard Data Capture initiative could be incorporated into PCORnet or other existing publicly funded data networks, although ASPE does plan to make them available for use, and officials told us that they will work with other HHS agencies and PCORI to determine adoption strategies. Furthermore, the ONC Standard Data Capture initiative is not expected to be completed until 2016, which occurs after PCORI’s common data model for the PCORnet initiative is expected to be used for conducting research, beginning in September 2015. Having more clearly defined objectives and establishing milestones and time frames can also help ASPE assess how it expects the results of its CER investments to build data capacity, and how they will be coordinated in a timeline with many other entities’ existing and planned efforts. Moreover, this information can help ASPE officials understand the extent to which their efforts are not duplicative and align with other federal efforts. ASPE officials told us that as of October 2014, they are planning to award a contract for developing an evaluation framework that will be used to assess the effectiveness of their CER data infrastructure projects. They also told us that they monitor and assess the 10 individual projects by collecting quarterly reports and assessing progress against the statements of work that were developed for each project. However, it is unclear from ASPE’s strategic road map whether these efforts will be sufficiently timely and coordinated with other federal and non-federal efforts to result in improvements to CER data capacity. Comparative clinical effectiveness research can give health care providers information to help decide which treatments may be most beneficial for a given patient, and it also can inform decisions by patients and caregivers. However, this information is often incomplete or unavailable. While HHS has multiple, ongoing efforts to meet its requirements under PPACA related to CER, it has not determined how it will fully address some of these requirements, particularly those related to dissemination and data capacity building. Disseminating CER in a timely manner is particularly challenging given the length of time and uncertainty inherent in applying research findings to help improve health care practice. AHRQ, for instance, has taken steps to disseminate CER and documented these processes, including time frames for some, but not all, of its key dissemination activities. Such time frames may become especially important due to the large volume of CER research expected from PCORI in the near future, which will increase AHRQ’s dissemination responsibilities, and the need to maximize the investment of PCORTF appropriations made through fiscal year 2019. Additionally, effective dissemination of research findings involves multiple stakeholders, some of which are specified in PPACA. Without clear plans to target each of these stakeholder groups, including federal and private health plans and vendors of health information technology focused on clinical decision support, it is unclear whether pertinent CER findings are being directed to key targeted stakeholders identified in PPACA and presented in a meaningful way to those groups. Other aspects of AHRQ’s dissemination process, such as its plans for a publicly available database of CER—including whether AHRQ’s instructions and CER search terms will be effective to meet the needs of various potential users in the general public—and its collaboration with NIH on dissemination activities, have not been fully defined. HHS’s plan to build data capacity involves identifying projects that would enhance existing data resources for CER. While HHS has a strategic road map with information on projects that it is funding to build the capacity for CER data, the road map does not include key elements, such as clearly defined objectives, milestones, and time frames needed to assess the agency’s progress toward the goal of building data capacity for CER, as would be consistent with practices for effective project management. Without defining these key elements, for example, it is unclear to what extent ASPE’s projects will build on or contribute to other similar federal or non-federal activities, rather than being duplicative. ASPE officials, for instance, could use more defined objectives and time frames to help them better assess the extent to which the CER projects they choose to fund will be useful and timely for other relevant federal and non-federal work, such as PCORI’s PCORnet initiative. To help ensure that HHS fully addresses its dissemination requirements under PPACA, we recommend that the Secretary of Health and Human Services direct AHRQ to take the following four actions: 1. identify and document time frames for the implementation and distribution of marketing plans and informational tools; 2. expand dissemination efforts to federal and private health plans and vendors of health information technology focused on clinical decision support; 3. document and complete plans to develop a publicly available database, including plans to meet the needs of various potential users in the general public; and 4. develop specific plans on how it will collaborate with NIH on its dissemination activities. In addition, to ensure that HHS fully addresses the PPACA requirements to build data capacity for CER, the Secretary should direct ASPE to include clearly defined objectives, milestones, and time frames, or other indicators of performance, in its strategic road map that is used to identify its PCORTF projects. We provided a draft of this report to HHS, and HHS provided written comments, which are reprinted in appendix III. HHS concurred with all five of our recommendations and provided additional information about its work to build data capacity for CER. Additionally, HHS provided technical comments, which we incorporated as appropriate. Specifically, for the first four recommendations, HHS—including AHRQ— stated that it would ensure that starting and ending time frames for the implementation and distribution of patient-centered outcomes research findings are clearly specified and documented. continue and expand dissemination activities that target federal and private health insurance plans, as well as vendors of health information technology focused on clinical decision support. HHS stated that it recently issued a funding opportunity announcement focused on the use of clinical decision support to disseminate and implement patient-centered outcomes research findings. document and complete its plans to ensure that multiple potential users, including the general public, have access to patient-centered outcomes research studies and their findings. As noted in our findings, these plans include creating a web page to list and provide users with links to existing publicly available databases that could be used to search for these studies. Complete plans would include time frames, strategies to address potential limitations, and whether the needs of various users are being met. continue to collaborate with NIH institutes and centers, and develop and document specific collaborations around patient-centered outcomes research dissemination activities. HHS stated that AHRQ has begun regular meetings with NIH—through its Office of Science Policy and the NIH Deputy Director for Science, Outreach, and Policy—to discuss how NIH’s and AHRQ’s activities can best complement one another. Regarding our last recommendation, HHS stated that it intends, through ASPE, to further develop the road map by specifying milestones with corresponding time frames. HHS will also develop specific performance indicators for its portfolio of data capacity investments. Consistent with our findings and conclusions, HHS’s comments also stated that its data capacity investments need to coincide with other key HHS policy initiatives and be responsive to the needs of CER data networks, including PCORI’s PCORnet. We are sending copies of this report to the Secretary of Health and Human Services, the Director of AHRQ, the Assistant Secretary for ASPE, and other interested parties. In addition, the report is also available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or kohnl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs are on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV. According to the Agency for Healthcare Research and Quality (AHRQ), it conducted 74 systematic reviews—syntheses of existing research—that were related to comparative clinical effectiveness research (CER) and resulted in findings disseminated between June 2012 and June 2014. Table 3 lists each systematic review with dates for each processing step leading up to posting the results of the review on AHRQ’s website for the public. Based on GAO’s analysis of the 74 systematic reviews, the time frame from when a systematic review began to when the findings were disseminated, including posting via AHRQ’s website, ranged from 1 year to more than 4 years. Appendix II: ASPE’s Comparative Effectiveness Infrastructure Projects (FY 2011-2014) Amount obligated (dollars in millions) $1.7 a centralized inventory of CER studies to serve as the foundation for a publicly accessible database of current publicly and privately funded CER projects, and related published policy and scientific literature. algorithms to accurately identify and classify CER studies. an improved web-based tool to provide a better understanding of the landscape of current CER activity to users. a mechanism and plan to pilot test the tool prior to making it publicly accessible. Enhancements to the existing database will combine claims data from public and private sources, matching patient information as appropriate, as is necessary for cross-payer and longitudinal analysis. pursue options to test the value of secure distributed data networks for research applications like CER. Ongoing pre-existing project to support CER through a research database that provides researchers with Medicare and Medicaid beneficiary claims and assessment data linked by beneficiary across the continuum of care. Funded enhancements include expanding the amount of Medicaid data available and security enhancements. Collaboration between the Office of the National Coordinator for Health Information Technology and the National Library of Medicine to integrate clinical information and research information within a “template” that can be utilized by researchers. Expand the amount of data collected by a nationwide network of 19 community health centers and five research organizations in 10 states, which together collect CER-related data about patients in underserved communities. Build upon previous Centers for Disease Control and Prevention efforts by augmenting a publicly available dataset for CER with additional longitudinal follow-up data on disease recurrence and vital status for colon, rectum, and breast cancer cases. Enhance software tools and methodology for management and consolidation of electronic data reported on a real-time basis from electronic health records to registries. Identify concrete, strategic opportunities to contribute long term to building data infrastructure for CER, and help maximize the impact of the Patient- Centered Outcomes Research Trust Fund investments. Assess the current landscape of data infrastructure for CER, identify gaps, and opportunities. Develop, select, and validate standards for common data elements for use in CER and a template to collect data from electronic health records for research purposes. Allow providers to access data in their own electronic health records in a standardized way to support CER. Allow researchers outside of the organization who have remote access authorization to access an organization’s electronic health record data for the purpose of CER. These projects have two initiatives under the same project description. In addition to the contact named above, Will Simerl, Assistant Director; Jennie Apter; La Sherri Bush; Christine Davis; Ashley Dixon; Colbie Holderness; Andrea Richardson; and Jennifer Whitworth made key contributions to this report.

Summary: PPACA imposed new requirements on HHS related to CER—research that evaluates and compares health outcomes and the clinical effectiveness, risks, and benefits of two or more medical treatments or services. Among other things, PPACA required AHRQ to broadly disseminate findings from federally funded CER and the Secretary of HHS (who, by delegation, charged ASPE) to coordinate federal programs to build data capacity for CER. PPACA also mandated that GAO review HHS's CER activities. This report examines (1) AHRQ's activities to disseminate the results of federally funded CER and (2) ASPE's activities to coordinate federal programs to support CER by building the capacity to collect, link, and analyze data, among other objectives. GAO reviewed relevant legal requirements and HHS documentation; interviewed HHS officials; and obtained information from five stakeholder groups that AHRQ targeted to receive disseminated information or were otherwise involved in AHRQ's dissemination efforts. The Agency for Healthcare Research and Quality (AHRQ), an agency within the Department of Health and Human Services (HHS), has taken some steps to disseminate comparative clinical effectiveness research (CER), as required under the Patient Protection and Affordable Care Act (PPACA), but has not taken other actions to help it fully address its dissemination requirements. The steps it has taken include the creation of tools that organize and disseminate research findings to certain targeted stakeholder groups and the development of plans for a publicly available database that includes CER. For example, AHRQ's marketing plans—customized plans to help convey key messages about AHRQ's research—include various informational tools to disseminate CER, such as research summaries that communicate research findings to clinicians, consumers, caregivers, and policymakers. However, the agency has not clearly defined how to disseminate information to certain stakeholder groups specified in the law, nor has it identified and documented time frames to implement the marketing plans and distribute the associated informational tools, as would be consistent with federal internal control standards, which state that significant events need to be clearly documented to ensure management goals are carried out. Additionally, in order to implement PPACA's requirement for developing a publicly available database that contains CER evidence, AHRQ officials told GAO that they plan to create a web page to list and provide users with links to existing publicly available databases that could be used to search for CER, but they have not documented a specific implementation plan that includes time frames and strategies to address known potential limitations, such as difficulties that certain users may face in searching the databases for CER results. HHS's Assistant Secretary for Planning and Evaluation (ASPE) has coordinated among various agencies to fund projects intended to build data capacity for CER, but its approach lacks key elements needed to ensure its effectiveness. For example, these projects include an effort to better standardize data that could be used in multiple research projects. However, HHS's approach to building data capacity for CER lacks key elements, such as defined objectives, milestones, and time frames, that are necessary to ensure effectiveness. ASPE officials worked with the Office of the National Coordinator for Health Information Technology to develop a strategic road map to guide both the identification and selection of ASPE's projects beginning in fiscal year 2014 through fiscal year 2019. Although the February 2014 strategic framework for the road map highlighted several priority objectives, such as enabling the collection of standardized clinical data, these objectives were broad and not clearly defined. For example, although ASPE identified and considered related, ongoing federal and non-federal data infrastructure projects in an attempt to identify needs or gaps, among other things, its strategic road map is unclear on the timing and level of coordination that would be necessary for its projects to work together with these related projects to improve data capacity. Standard practices for project management call for agencies to conceptualize, define, and document specific goals and objectives in the planning process, along with the appropriate steps, milestones, time frames, and resources needed to achieve those results. GAO recommends that HHS direct (1) AHRQ to take several actions related to its dissemination efforts, including identifying and documenting time frames for the implementation and distribution of marketing plans and informational tools, and (2) ASPE to include clearly defined objectives, milestones, and time frames, or other indicators of performance, in its strategic road map used to identify its CER-funded projects. HHS concurred with the recommendations.