According to FDA, the goal of its regulatory science initiative is to develop and apply the best available scientific data, knowledge, methods, and tools to reduce uncertainty and make regulatory decisions more efficient and consistent. In doing so, the agency seeks to ensure public access to products that are manufactured or processed in a high quality manner and monitored to ensure safety and quality during real-world use. FDA’s 2011 strategic plan identified eight priority areas for regulatory science, of which seven related to medical products where the agency determined that new or enhanced engagement is essential to the continued success of the agency’s public health and regulatory mission. The agency then added an additional priority area related to global product safety in 2013. (See table 1.) FDA conducts work to advance regulatory science through intramural research and extramural collaborations, such as collaboration with other government agencies, academia, industry, patient organizations, professional associations, and other stakeholders. Targeted funding for regulatory science at FDA comes from a number of centers and offices, including the Center for Biologics Evaluation and Research (CBER), the Center for Drug Evaluation and Research (CDER), and the Center for Devices and Radiologic Health (CDRH), all within FDA’s Office of Medical Products and Tobacco. These centers are responsible for approving medical products (biologics, drugs, and devices, respectively) and monitoring their ongoing safety once approved. In addition, other offices support the regulatory mission of FDA, such as the Office of International Programs (OIP) in the Office of Global Regulatory Operations and Policy, and those under the Office of the Chief Scientist—whose mission is to provide strategic leadership, coordination and expertise—including the National Center for Toxicological Research (NCTR), the Office of Counterterrorism and Emerging Threats (OCET), the Office of Minority Health (OMH), the Office of Regulatory Science and Innovation (ORSI), and the Office of Women’s Health (OWH). (See fig. 1.) Additional broad regulatory science efforts may occur within these centers and offices as well as others at FDA. FDA does not have measurable goals, such as targets and time frames, to assess its progress in advancing regulatory science. Further, FDA does not consistently track information on center and office funding decisions for each FDA priority area, leaving it without important information needed to conduct strategic planning on the agency’s regulatory science priorities. FDA’s 2011 and 2013 strategic planning documents do not identify measurable goals, such as targets and time frames, for assessing progress in regulatory science. For example, the 2013 strategic planning document states that FDA will measure progress by enumerating and describing product-specific advisory committees and staff training opportunities related to the priority areas. However, the document neither sets any targets or time frames, nor establishes further outcome-based measures for what FDA hopes to achieve for either a given priority area or regulatory science in general. Likewise, FDA’s 2011 strategic planning document presents strategies for addressing the priority areas, but the document does not include measurable goals within these strategies. For example, it indicates that addressing the toxicology priority area would involve developing human and animal models to predict adverse responses, but it does not provide the number of models FDA intends to develop or the adverse responses for which the models are intended. FDA’s lack of specific measurable goals is reflected in the progress report that it completed for FDASIA. In that report, FDA includes examples of achievements but it does not specify overall results, such as the number of tests and technologies developed for the efficacy and safety of medical products, which the 2013 planning document indicated would help the manufacturing and quality priority area. In addition, FDA generally did not link the categories that it used to measure the adoption of regulatory science to a specific priority area. Specifically, in the progress report, only four of the eleven categories are linked to the FDA priority areas that they address. For example, while FDA reported one or more related FDA priority area for each guidance document included in the progress report, it did not provide the same information for the reported training examples or the projects included in the Drug Development Tool Qualification programs. According to our work on leading practices for strategic planning, an agency’s strategic goals should explain what results are expected from the agency and when to expect those results. It is also critical that the strategic planning documents describing these goals include specific actions and implementation schedules for how the agency is to achieve these goals. Without measurable goals, clear targets, and implementation time frames, FDA cannot provide a complete assessment of progress made in the regulatory science areas it has designated as priorities. According to FDA officials, it is difficult to measure the progress made in the priority areas because the priority areas are very broad and the underlying science is continually changing. Furthermore, the adoption of discoveries from regulatory science research can take years. Because the priority areas are managed across multiple organizational units, they are not being overseen by a specific center or office. However, we have previously recognized that while measuring the performance of science- related projects can be difficult, science-related agencies like FDA should still have clearly established goals. One of the centers that targets funding at regulatory science has a plan to track the progress of its regulatory science activities, but this plan does not include measurable goals that could be used to assess progress. Specifically, CDRH officials told us the center has drafted a logic model to identify and track the short- and long-term outcomes of funds it spends on regulatory science research. The center expects the model to be finalized in 2017. However, the current draft does not include measurable goals to assess progress. In addition, officials from most of the other centers and offices that obligated funds targeted at regulatory science activities in fiscal years 2010 through 2014 told us they are not developing such a model. One office also stated that the CDRH model may not be applicable to their work and that they may not have the resources needed to customize and implement such a model. In another effort, FDA officials indicated that the agency recently initiated internal discussions primarily to share information, and if feasible and desired, harmonize best practices within the agency. FDA officials also said that this group would discuss evaluation strategies and processes and perhaps eventually arrive at a common FDA approach to thinking about research outcomes and impact. This group first met in October 2015 to discuss current efforts, and FDA officials reported in April 2016 that it is still in the early stages of sharing information. FDA does not have the information necessary to track funding and conduct strategic planning agency-wide for its regulatory science priorities because most of the centers and offices did not collect information on the FDA priority areas that were addressed by the projects they funded. As a result, information on funding for each priority area is generally not readily available and therefore the majority of these centers and offices had to retrospectively assign FDA priority areas to each project at the time of our review. Six centers and offices—CDRH, NCTR, OCET, OIP, OMH, and OWH—fund regulatory science projects that address multiple FDA priority areas, but generally did not collect information about how funds are distributed across those priority areas. All six had to retrospectively identify associated FDA priority areas for our review. For example, while CDRH and NCTR collect some information about research priorities for the projects they fund, the information they collect does not fully align with the FDA priority areas. Specifically, CDRH collects information on its own set of priorities and NCTR collects information on FDA goals. While this information is partially aligned with the FDA priority areas, there is not a one-to-one correspondence, thus information on FDA priority areas is not readily available. According to NCTR officials, they began capturing information on the FDA priority areas in 2016. Further, OMH asks researchers to identify the unmet regulatory science need that their proposed research addresses, but the researchers are not asked to identify specific FDA priority areas. In addition, OCET and OIP officials told us that all of their funded projects are related to one FDA priority area, and therefore they do not collect information about other priority areas. However, there is no guarantee they will not fund projects related to other priority areas in the future. Further, while OIP told us they focus on just the FDA priority area global product safety, the data they provided for our review also identified an additional priority area, information sciences, as an FDA priority area being addressed by some of their projects. Three offices, CBER, CDER, and ORSI, collect some information on the FDA priority areas addressed by their targeted projects. CBER and CDER have not collected this information at the time of funding. Rather, CBER asks researchers to provide it as part of required annual reports and CDER officials indicated that CDER has made a similar request of researchers since 2012. Officials from both CBER and CDER said that in the near future their centers plan to track the FDA priority areas at the time projects are proposed for funding. ORSI asks researchers to identify FDA priority areas for its broad agency announcements, which are competitive funding announcements for extramural research programs and accounted for 21 percent of ORSI’s funding for the projects that we reviewed. ORSI does not ask this of researchers applying for its other regulatory science funding programs, although the announcements for the intramural grants state that the proposals should align with one or more of the eight original priority areas. For these other funding programs, ORSI confirms that a project proposal is related to regulatory science, but does not document any specific FDA priority area for that project. While each center or office may be funding projects that are generally consistent with the priority areas that FDA established, without consistent information from each center or office detailing those connections, the agency is not able to examine obligations across specific priority areas. Standards for internal control in the federal government state that complete and accurate data are needed to make operating decisions and to allocate resources. In addition, to ensure program goals are met, our work encourages agencies to manage efforts that cut across the agency. Without complete information on the allocation of funding across priority areas by centers and offices, FDA cannot ensure that funding is being distributed in line with its strategic plan. In response to our request, FDA identified projects targeted at regulatory science funded from fiscal years 2010 through 2014 totaling more than $507 million. Annual obligations for these projects generally increased during that time. This funding varied across centers and offices, ranging from approximately $450,000 to about $200 million. In addition, while FDA does not systematically track regulatory science obligations by priority area, the agency’s retrospective review showed wide funding variation across priority areas, ranging from about $3 million to approximately $203 million. FDA obligations targeted at regulatory science projects increased from about $73 million in fiscal year 2010 to a peak of about $123 million in fiscal year 2013, then a decline to about $110 million in fiscal year 2014. These funds represent those obligations targeted at specific regulatory science projects and do not include FDA obligations for other activities benefitting regulatory science for which the agency was not able to quantify spending at a project level. (See fig. 2.) Over the 5 years we examined, FDA obligated nearly $507 million for 1,279 regulatory science projects, an average of approximately $400,000 per project. Total obligations for individual projects ranged from $430 to $9.1 million during this time. While FDA obligated funds for some projects for a single year, FDA typically obligated funds for projects in multiple years. FDA obligated 80 percent of these regulatory science funds to intramural projects—those led by FDA researchers. For example, the Office of the Chief Scientist’s grant program includes five intramural grant programs; for these programs, FDA scientists first submit concept papers that are ranked and then submit full proposals that are peer-reviewed. The remaining 20 percent of projects were either extramural or a combination of intramural and extramural. For example: Broad Agency Announcements are extramural, competitive funding announcements supporting regulatory science research programs. Centers of Excellence in Regulatory Science and Innovation are extramural partnerships between FDA and universities to promote cross-disciplinary regulatory science training and research. The Critical Path Initiative started in 2004 to improve medical product development, evaluation, and manufacturing and is used to support intramural research and external collaborations. Most Critical Path Initiative projects were intramural, except for ten projects that were both intramural and extramural. Of the total funding targeted at regulatory science projects, 48 percent was obligated for projects awarded through a non-competitive award process, 39 percent through a competitive award process, and 13 percent through a combination of competitive and non-competitive processes. Officials reported that FDA has traditionally funded regulatory science projects with FDA general appropriations, but projects funded within CDER have also been supplemented by funds collected under user fee acts—specifically the Prescription Drug User Fee Act (PDUFA) and Generic Drug User Fee Amendments of 2012 (GDUFA)—that authorize the collection of funds from industry (including the pharmaceutical and biotechnology industries). PDUFA funds accounted for 2 percent of CDER’s total regulatory science obligations for fiscal years 2010 through 2014, ranging from 1 to 4 percent annually, with approximately $300,000 of PDUFA funds obligated each year for regulatory science. Starting in fiscal year 2013, CDER committed to using a portion of its GDUFA funds for regulatory science. GDUFA funds accounted for 65 percent of CDER’s regulatory science obligations in fiscal years 2013 and 2014 combined, about $17 million (70 percent) and $19 million (62 percent), respectively. (See fig. 3.) In 2013, the addition of funds from GDUFA more than doubled CDER’s annual obligations for regulatory science projects for fiscal years 2010-2013. GDUFA and PDUFA are the only two user fee programs that support targeted regulatory science obligations. In addition, FDA indicated that it funded other efforts benefitting regulatory science, but was unable to quantify spending at the project level. For example, CDER officials told us that funds targeted at regulatory science identified for our review represent only a portion of CDER’s investment in regulatory science. They added that lab programs, like those in the Office of Pharmaceutical Quality and the Division of Applied Regulatory Science within the Office of Translational Sciences, fund the bulk of their projects through the normal CDER budgeting process and therefore are not included among the targeted funds. One such CDER project focused on developing an analytical method that sponsors could use to support that proposed generic and brand name forms of estrogen are chemically equivalent. The work resulted in new recommendations for some estrogen analyses, which were incorporated into a guidance document for the development of generic estrogen products. Of the nine centers and offices that obligated funds targeted at regulatory science from fiscal year 2010 through fiscal year 2014, total obligations ranged from approximately $450,000 by OMH to about $200 million by NCTR. The center and offices within the Office of the Chief Scientist— NCTR, OCET, OMH, ORSI, and OWH—accounted for 65 percent of FDA’s total obligations for projects targeted at regulatory science, with NCTR accounting for 60 percent of the total Office of the Chief Scientist’s obligations. Centers with regulatory responsibilities for medical products (CDER, CBER, and CDRH) accounted for 34 percent of the obligations, and OIP accounted for the remaining 1 percent. The average obligation per project by centers and offices ranged from just over $110,000 for OWH to about $1.1 million for OIP. However, the centers and offices that had the highest total obligations were not necessarily the ones that had the highest average obligation at the project level. For instance, OIP had the highest average obligations per project, more than $1 million, yet it had the second lowest total obligations, at about $5.4 million. (See fig. 4.) Similarly, total obligations associated with each FDA regulatory science priority area varied widely, ranging from about $3 million for projects that focused on global product safety to approximately $203 million for projects that focused on the toxicology priority area. (See fig. 5.) Projects that focused on the clinical evaluations and personalized medicine and medical countermeasures priority areas were among those with the greatest obligations. Average obligations per project ranged from about $250,000 for the projects that focused on the manufacturing and quality priority area to approximately $1.1 million for the projects that focused on the global product safety priority area. However, the FDA priority areas that had the lowest total obligations were not necessarily the ones that had the lowest average obligation at the project level. For example, projects that included a focus on global product safety had the lowest total obligations but they had the highest average obligation per project. Similarly, projects that included a focus on social and behavioral science had the second lowest total obligations but they had the second highest average obligation per project. FDA’s centers and offices make decisions about funding that determine which projects to fund, and they generally obligated project funds across several regulatory science priority areas. Specifically, two-thirds of the centers and offices provided obligations to five or more priority areas. For example, CBER provided obligations to projects that collectively focused on every priority area except global product safety. Nevertheless, there were four centers and offices that directed at least half of their regulatory science obligations to a single priority area. 100 percent of OCET obligations were for medical countermeasures. 62 percent of OIP obligations were for global product safety. 56 percent of NCTR obligations were for toxicology. 50 percent of OWH obligations were for clinical evaluations and personalized medicine. While funding for each of the FDA priority areas generally came from a number of different centers and offices, for all but two priority areas, there was one center or office that accounted for the majority of obligations. Global product safety: 100 percent of the obligations from OIP. Toxicology: 82 percent of the obligations from NCTR. Social and behavioral science: 76 percent of the obligations from CDER. Medical countermeasures: 64 percent of the obligations from OCET. Manufacturing and quality: 56 percent of the obligations from CBER. Emerging technologies: 55 percent of the obligations from CBER. Information sciences: 49 percent of the obligations from NCTR. Clinical evaluations and personalized medicine: 43 percent of the obligations from NCTR. FDA reported that the 17 regulatory science projects that we selected for review helped to advance regulatory science. (See appendix I for additional information about each of these 17 projects.) For each of these 17 projects, FDA identified achievements that we classified as dissemination of project findings, internal changes at FDA, and changes by industry and groups outside of FDA. (See fig. 6.) Dissemination of project findings. For 16 of the 17 selected projects, FDA reported that it disseminated the project results in scientific publications, conferences, FDA workshops, or some combination of all three. Such dissemination provides FDA an opportunity to share new information and understanding internally and with the larger scientific community and may contribute to future regulatory science activities. Internal impacts. FDA described internal impacts resulting from each of the 17 selected projects, many of which related to advancing FDA’s scientific understanding in a particular area. For 12 selected projects, FDA reported that the projects resulted in new information about the topic that FDA was using or considering using for future work. For example, one goal of OCET’s project looking at the feasibility of using electronic health records in public health emergencies was advancing FDA’s understanding of the possibilities and limitations of using electronic health data. The findings from this project indicated that structured data in electronic health records could help the agency assess the risk of adverse events, particularly those that are severe. However, the results also indicated that this near real-time data still had some built-in delays and that the data search process cannot be fully automated. This provided FDA with information it can use as it considers using electronic health records in emergency situations. For 9 selected projects, FDA reported that the results led the agency to plan or conduct additional studies or activities that represent the logical next step in the particular area being studied. For example, initial results from an ORSI study of the use of social media to provide early signals of drug safety concerns showed a relationship between data obtained from social media and from FDA’s adverse event reporting system. Using that information, FDA then conducted a retrospective study for 10 safety concerns to see if there was evidence of those adverse events in social media before FDA became aware of them. FDA has since reported that the analysis identified specific limitations of the tool FDA uses in monitoring safety concerns. Other internal impacts were related to changes in agency practices. For 8 selected projects, FDA reported that the results led to the development of standards, methods, tools, or training for FDA internal use. For example, an OIP project was designed to create a tool to help secure global supply chains against the infiltration of counterfeit or substandard products. The project resulted in the production of a “roadmap” that FDA could use to develop such a system. For 5 selected projects, FDA reported that the results led to either a change in guidance or regulation or the decision to not make a previously proposed change. For example, CDER funded a study of surrogate endpoints that could speed the development of new therapies for breast cancer. FDA used results from this study to inform its development of guidance to industry that described study designs in which a surrogate endpoint may be accepted by FDA as reasonably likely to predict the clinical benefit of a drug. In response to a statutory requirement, CDER also examined whether quantitative information could be added to drug advertising to maximize consumer and health care professional understanding of the benefits and risks of the drug. Based on the results of the study, the Secretary of HHS concluded that quantitative information cannot be readily applied to many drugs and therefore it is not appropriate to issue regulations that would require such information to be added to promotional labeling or advertising. For 5 selected projects, FDA reported that the results led the agency to change aspects of its review process. For example, a CBER study designed to improve influenza vaccine efficacy provided information to FDA that has helped in the review of preclinical animal studies that are included in some drug applications. The study also resulted in FDA offering training for its lab members in an approach to vaccine testing that FDA says is a common part of a product review. In addition, an NCTR project developed a knowledge base about liver toxicity and has used that to advise CDER reviewers about drug- induced liver injury risk for products they were reviewing. External impacts. FDA described external impacts for 8 of the 17 selected projects. These projects resulted in the development of tools or proposed standards for use by industry or the implementation or use of new tools or standards by industry or outside organizations. According to FDA, it can take several years for funded research to result in these types of tools. For 7 selected projects, FDA reported that the results led to the development of tools or proposed standards for use by industry. For example, a CDRH study of radio interference with automated external defibrillators led FDA to recommend to an international commission that standards for these types of defibrillators be modified to account for radio interference. Similarly, FDA officials reported that results from a study funded by OWH looking at degradation of absorbable polymers used in some cardiac stents have provided guidance for industry on the design, manufacturing, and regulation of these absorbable stents. For 6 selected projects, FDA reported that industry or outside organizations have made changes based on the results of these FDA- funded projects. For example, FDA told us NCTR’s studies of Bisphenol A, a chemical found in certain plastics, helped FDA and the European Food Safety Authority, resolve public safety concerns. Similarly, CDRH’s study of total disc replacement devices for the spine led to the development of a test guide published through the American Society of Testing and Materials International that is used by multiple manufacturers. FDA told us that the results from these tests were then part of the manufacturers’ submissions for approval of these devices. For several years, FDA has been aware of the need to improve its scientific base and has established multiple regulatory science initiatives, as well as prioritized areas to address that need. FDA projects targeted at advancing regulatory science have led to internal and external impacts in understanding new science associated with medical products. However, the agency has not identified measurable goals in their strategic plans or reports on strategic priorities, such as specific targets and time frames, for regulatory science. Such goals are a best practice for strategic planning and could enable FDA to assess and report its progress in addressing its identified priority areas and strategically plan and allocate resources for its broader regulatory science initiative. The agency faces another obstacle to its strategic plan without consistent information about centers and offices’ distribution of targeted regulatory science funding among those identified priority areas. The individual centers and offices that decide which projects to fund are either tracking the priority areas in different ways or not all. Our prior work encourages the proactive collection of consistent information and standards for internal control recommend federal agencies have complete and accurate data for making funding decisions. Systematic tracking by each center or office is needed for the agency to examine obligations across, or progress within, specific priority areas and would help the agency to strategically plan for its regulatory science initiative as a whole. In order to improve FDA’s strategic planning for regulatory science efforts, we recommend the Secretary of Health and Human Services direct the Commissioner of FDA to take the following two actions: 1. Develop and document measurable goals, such as targets and time frames, for its regulatory science efforts so it can consistently assess and report on the agency’s progress in regulatory science efforts. 2. Systematically track funding of regulatory science projects across each of its priority areas. We provided a draft of this report to HHS. HHS concurred with our recommendations and provided written comments, which are reprinted in appendix II. In its written comments, HHS agreed with the importance of strategic planning for regulatory science. HHS concurred with our recommendation that FDA should develop and document measurable goals; HHS suggested that agency documents with a targeted focus, such as user fee commitment letters and specific planning documents, are a more appropriate place for such goals than an agency-level strategic plan. In our recommendation to HHS, we do not specify where such goals should be documented. We recognize, as HHS noted in its comments, that advancing regulatory science is an uncertain and non- linear path that can make it challenging to set targets for specific accomplishments. Nevertheless, FDA should develop measureable goals that are related to the impacts that are discussed in HHS’s comments, including the effectiveness and efficiency of FDA’s regulatory review, new pathways for medical product development, enhancements in the agency’s ability to provide useful guidance to sponsors, and new technologies to monitor manufacturing and real world use of approved medical products. As all but one of the agency-wide priority areas are being addressed by projects funded by multiple centers and offices, it is important that FDA develop and document measurable goals that encompass the efforts of multiple centers and offices. HHS also concurred with our recommendation to systematically track funding across FDA’s regulatory science priority areas, and the department identified recent and planned activities of specific centers to improve such tracking We support these efforts and reiterate the importance of FDA systematically track funds agency-wide for each of the priority areas it developed. Systematic tracking of both progress on measurable goals and funding is essential for FDA to strategically plan its regulatory science initiative across the agency. In addition to these general comments, HHS provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Health and Human Services and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or crossem@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. We examined the achievements related to regulatory science for 17 projects funded by the Food and Drug Administration (FDA). Below is a brief description of each project, including examples of achievements resulting from that project, according to information provided by FDA. Title: Created new approaches to identify and understand critical product quality attributes of complex products, such as stem cell- derived products (both animal and human) and complex systems of medical devices. Center or office funding project: Center for Biologics Evaluation and Research (CBER) Priority area(s) covered: Emerging technologies and medical countermeasures Funding obligations and time period of funding: $4.513 million (fiscal years 2010 through 2014) Description of project: Certain stem cells are being used in clinical trials for multiple medical conditions. However, multiple factors, including donor variation and culture conditions may affect the clinical performance of the stem cells either in terms of safety, efficacy, or both. This project was designed to identify product attributes that correlate with specific outcomes and to increase understanding of factors that might affect stem cell-derived safety and efficacy. This project also included the development of methods to quantify various attributes of these cells. Select achievements reported by FDA: The project’s findings affected FDA reviewers’ understanding of stem-cell products. Further, the findings have helped producers of stem cell products with their studies, specifically noting that some sponsors have adopted quantitative methods for assessing the stem cells. The findings were important in the scientific reasoning in two guidance documents. Additional projects have been planned to further advance understanding in this area. Title: Developing new approaches for measuring the quality of next- generation smallpox vaccines. Center or office funding project: CBER Priority area(s) covered: Medical countermeasures Funding obligations and time period of funding: $0.537 million (fiscal years 2010 through 2014) Description of project: The goal of the project was to evaluate factors that affect the safety and efficacy of smallpox vaccines and to develop new methods to evaluate smallpox vaccine quality— specifically, potency. The studies were aimed at helping the development of new smallpox vaccines. Select achievements reported by FDA: Results from the project informed FDA’s understanding of various characteristics of the vaccine, as well as factors that influenced immune responses to the vaccine. FDA also reported that one of the two alternative approaches for evaluating the potency of smallpox vaccines used in clinical trials was considerably faster than traditional methods and could be adapted for future use. In addition, further studies are ongoing in other tests related to smallpox vaccines. Title: Correlates of protective immunity against influenza. Center or office funding project: CBER Priority area(s) covered: Manufacturing and quality, emerging technologies, and medical countermeasures Funding obligations and time period of funding: $1.209 million (fiscal years 2010 and 2011) Description of project: This project was designed to support the development of seasonal and pandemic influenza vaccines by identifying mechanisms that contribute to immunity and developing measures of those responses. Select achievements reported by FDA: This project resulted in the development of a new method that was useful for testing influenza vaccines, the discovery of certain characteristics that are important for producing protective immunity, and the determination of the amount of a vaccine ingredient that is needed to produce immunity. FDA reported that the project provided a foundation for training lab members in development and validation of an approach to vaccine testing, while also providing the basis for an international study led by CBER to assess reproducibility in one of the methods developed in this project. Title: Evaluating a surrogate endpoint that could speed development of new therapies for breast cancer. Center or office funding project: Center for Drug Evaluation and Research (CDER) Priority area(s) covered: Clinical evaluations and personalized medicine Funding obligations and time period of funding: FDA could not determine funding as the research was conducted as part of FDA employees’ regular responsibilities. Description of project: FDA conducted research to assess and evaluate the validity and potential applications of a surrogate endpoint—a measure that can predict, but is not itself a measure of, benefit—in trials of treatments for women with breast cancer. By collaborating with an international working group, FDA researchers were able to use data from more than 12,000 patients. Select achievements reported by FDA: FDA reported that researchers found a potential relationship between the surrogate endpoint and survival. This then provided important information to drug developers for the design of future drug trials. It also informed the development of guidance to industry. Title: Completed three studies and a literature review assessing whether quantitative information could be successfully added to television and print advertisements to maximize audience understanding of benefit information in the piece, including the type of benefit information, different combinations of statistical format, and different graphic representations. Center or office funding project: CDER Priority area(s) covered: Social and behavioral science Funding obligations and time period of funding: $0.270 million (fiscal year 2010) Description of project: This project was composed of four studies designed to investigate whether quantitative information in direct-to- consumer advertisements is helpful for consumers. FDA was asked by Congress to investigate this topic to determine whether such information about the benefits and risks of prescriptions drugs in a standardized format would improve health care decision-making by clinicians, patients, and consumers. Select achievements reported by FDA: FDA reported that, based on these studies and other efforts, the inclusion of certain types of quantitative information can be helpful in some limited circumstances, but a standardized format cannot be readily applied to many drugs. It was therefore not appropriate to issue new regulations that would require such information on promotional labeling or print advertising. The findings have been used internally by an FDA working group that explores direct-to-consumer advertising. FDA reported that the findings have also been used by the external research community. Title: Experimental study of patient information prototypes. Center or office funding project: CDER Priority area(s) covered: Social and behavioral science Funding obligations and time period of funding: $1.613 million (fiscal year 2010) Description of project: This project examined different methods of presenting prescription drug information to patients who obtained a prescription. The study was designed to compare the format in which information is presented, the inclusion of additional context or not, and the order of information about warnings versus information about the efficacy of the drug. Select achievements reported by FDA: The study provided information about patient preferences for and increased comprehension of single page prototypes over the currently available format. These findings have informed FDA’s development of the Patient Medication Information Initiative, which will consider a new regulation to require all prescription drugs to have a single document standardized in content and format that provides prescription information to patients in an accurate, easily understood, and balanced form. Title: Development and validation of a standard test method to assess for impingement of artificial total disc replacement devices in order to provide scientific basis for regulatory guidance and better predict which devices will be clinically successful. Center or office funding project: Center for Devices and Radiological Health (CDRH) Priority area(s) covered: Clinical evaluations and personalized medicine and emerging technologies Funding obligations and time period of funding: $0.753 million (fiscal years 2010 through 2012) Description of project: The primary goal of the project was to develop a new scientific tool to characterize impingement— unintended contact between surfaces of the device—of total disc replacement devices. Impingement can be linked to premature mechanical device failure and was not accounted for in preclinical test methods. As a result, bench testing was not accurately mimicking wear and damage that was being observed clinically. Select achievements reported by FDA: The project resulted in the development of an impingement test guide published through a professional association. As a result, multiple device manufacturers used the guide to perform impingement testing and FDA incorporated those results into their decision making on premarket approval applications and investigational device exemption submissions. Title: Developed standards to reduce the risk of misconnection between different types of small-bore connectors used for intravenous, feeding, neural, blood pressure cuff, and breathing system tubes to prevent serious adverse events. Center or office funding project: CDRH Priority area(s) covered: Manufacturing and quality Funding obligations and time period of funding: FDA could not determine funding as the research was conducted as part of FDA employees’ regular responsibilities. Description of project: Because devices using small bore connectors have been accidentally connected with devices that have different functions and have led to serious adverse events for patients, including deaths, FDA participated in international efforts to standardize connector designs for specific medical applications such that they cannot be interconnected with a device for another medical application. Select achievements reported by FDA: FDA has issued guidance on premarket recommendations for devices that use small-bore connectors intended for use in the gastrointestinal tract. International standards are also being finalized based on this work. Device manufacturers are modifying their devices accordingly. In addition, FDA has developed a website for highlighting relevant information for stakeholders. Title: Created a general testing protocol and test methods for electromagnetic compatibility of automated external defibrillators. Center or office funding project: CDRH Priority area(s) covered: Emerging technologies Funding obligations and time period of funding: FDA could not determine funding as the research was conducted as part of FDA employees’ regular responsibilities. Description of project: A growing number of adverse events and voluntary recalls by manufacturers of automated external defibrillators led FDA to study the effect of electromagnetic interference that had been related to potentially life-threatening failures of this device. Select achievements reported by FDA: As a result of this project, FDA developed test methods to evaluate the susceptibility of automated external defibrillators to radiofrequency interference. FDA recommended an international commission’s standards be modified to account for interference testing at certain frequencies. Title: Physiologically based pharmacokinetic models for Bisphenol A. Center or office funding project: National Center for Toxicological Research (NCTR) Priority area(s) covered: Toxicology Funding obligations and time period of funding: $1.049 million (fiscal years 2010 through 2014) Description of project: Due to concerns about the safety of Bisphenol A, which is used in many consumer plastic products, NCTR developed computational modeling to simulate infant exposure to Bispehnol A to provide FDA with information necessary to complete a safety assessment. Select achievements reported by FDA: The results of the project allowed FDA to predict how much chemical remained after being metabolized and would get into the circulatory system of adults and infants. These models were incorporated into FDA’s updated assessment on Bisphenol A and allowed FDA’s Center for Food Safety and Nutrition, as well as other regulatory bodies, to determine that current uses of Bisphenol A are safe for infants and adults and, further, led the agency to conclude that the traditional safety assessment methods used were overly conservative. Title: Development of liver toxicity knowledge base to empower the FDA review process. Center or office funding project: NCTR Priority area(s) covered: Toxicology and information sciences Funding obligations and time period of funding: $3.426 million (fiscal years 2010 through 2013) Description of project: Drug-induced liver injury is a serious safety concern that is a frequent cause of denied approvals and “black box” warnings on drugs. As a result, FDA was interested in developing a database to improve its understanding and prediction of such liver injury. Select achievements reported by FDA: The project has produced a centralized resource of data and predictive models that are useful for both research and regulation. NCTR has trained CDER reviewers to effectively use the software from this database. NCTR also has received requests from CDER for five consultations to assess the risk of products that it has reviewed and to incorporate the software into training for new reviewers. An extension of the project is also currently under review. Title: Assessing the feasibility of using electronic health record systems to conduct near real-time monitoring of health outcomes, including serious or unexpected adverse events associated with medical countermeasures used during public health emergencies. Center or office funding project: Office of Counterterrorism and Emerging Threats (OCET) Priority area(s) covered: Medical countermeasures Funding obligations and time period of funding: $1.419 million (fiscal year 2013) Description of project: The project was designed as a proof-of- concept feasibility study to learn whether it is possible to extract adverse event data from electronic health records and, if so, whether those data would provide useful information about the safety and effectiveness of medical countermeasures during a public health emergency. Select achievements reported by FDA: FDA learned that electronic health records data could help inform a risk assessment of medical countermeasures based on adverse events; however, there are limits to what can be done. For example, they found it was feasible to detect severe adverse events, but that less severe adverse events were likely to be underreported. Title: Sentinel initiative for surveillance of drugs, vaccines, and blood products used to prevent and treat pandemic influenza. Center or office funding project: OCET Priority area(s) covered: Medical countermeasures Funding obligations and time period of funding: $9.1 million (fiscal year 2011) Description of project: Sentinel is FDA’s system for conducting near real-time active safety surveillance of FDA-regulated medical products through routinely collected electronic healthcare data. This project was designed to expand those capabilities to include preparedness for safety surveillance in response to the use of medical countermeasures, including for pandemic influenza. Select achievements reported by FDA: This project created the capability for FDA to monitor the safety of medical countermeasures— for example, influenza vaccines used during an emergency, such as a pandemic. It also increased the efficiency of linkages between registries of immunization and the Sentinel database, which FDA says is vital during pandemics. Title: Collecting spectral information of foods, pharmaceutical ingredients, and formulated product (as well as its packaging materials) to establish a comprehensive spectral library accessible through the internet. Center or office funding project: Office of International Programs (OIP) Priority area(s) covered: Information sciences Funding obligations and time period of funding: $0.314 million (fiscal year 2013) Description of project: The primary goal of this project was to develop a roadmap for the development of a global library that could potentially be used to protect consumers against fraudulent and adulterated products. Select achievements reported by FDA: A roadmap was produced from this project and FDA intends to use it in discussions with domestic and foreign stakeholders, including other regulatory agencies and manufacturers. Title: Explored the potential for mining social media and other web sources to detect adverse event and safety signals. Center or office funding project: Office of Regulatory Science and Innovation (ORSI) Priority area(s) covered: Information sciences Funding obligations and time period of funding: $0.801 million (fiscal years 2012 through 2014) Description of project: This study evaluated the validity and trustworthiness of using a social media data mining tool to detect drug safety events. It was also designed to evaluate the potential of social media data to provide early signals for drug safety events in postmarketing surveillance and to better understand how these data can be used. Select achievements reported by FDA: FDA initially reported that the data mining tool suggested that user-generated data sources may identify signals not found in the traditional voluntary reporting systems and that there was agreement between data obtained from this tool and that obtained from the FDA Adverse Events Reporting System. FDA conducted an additional study to evaluate if there was evidence in social media for 10 recent MedWatch Safety Alerts prior to FDA becoming aware of them. FDA has since reported that monitoring social media did not provide early safety concerns for the medical products that were monitored in the study. The analysis identified specific limitations of FDA’s tool that was used in monitoring safety concerns. These limitations related to the natural variability of data sources and the difficulties in conducting accurate evaluation of the data. FDA noted that as monitoring social media for safety concerns is a new approach, the agency still needs to establish best practices in order to use it effectively. Title: Consortium for tuberculosis biomarkers. Center or office funding project: ORSI Priority area(s) covered: Clinical evaluations and personalized medicine Funding obligations and time period of funding: $1.425 million (fiscal years 2010 through 2012) Description of project: This project had three main objectives: 1) create protocols, processes, and standards by which a consortium for tuberculosis biomarkers—composed of three organizations central to tuberculosis clinical drug development—would operate; 2) create a repository for receiving, storing, and shipping samples to designated investigators; and 3) establish a peer review panel to review proposals related to the discovery and qualification of tuberculosis biomarkers, especially some surrogate markers. Select achievements reported by FDA: The consortium established protocols for key data elements to be gathered; a consensus set of operating procedures for sample collection, processing, and storage; and quality assurance and monitoring for these activities. The consortium also adopted a peer review process to review applications for access to samples. Title: Sex-based differences in the molecular mechanisms of polymer degradation in drug eluting stents. Center or office funding project: Office of Women’s Health (OWH) Priority area(s) covered: Clinical evaluations and personalized medicine and manufacturing and quality Funding obligations and time period of funding: $0.2 million (fiscal years 2011 through2012) Description of project: This study explored the different breakdown of materials used in biodegradable stents and examined the potential effect of sex on the degradation of these materials. Select achievements reported by FDA: FDA reported that the study found that stent material breakdown varied in different tissues. The findings provide information that can guide FDA and the larger community in the design, manufacture, and evaluation of absorbable stents. FDA reported that this research will support the development of guidance for implants containing certain absorbable components. Marcia Crosse, (202) 512-7114 or crossem@gao.gov. In addition to the contact name above, William Hadley, Assistant Director; Carolyn Garvey; Sandra George; Cathleen Hamann; Carolyn Feis Korman; and Deborah Linares made key contributions to this report.

Summary: FDA has faced challenges regulating medical products, owing in part to rapid changes in science and technology. In 2010, FDA established a regulatory science initiative that identified eight priority areas for medical products where new research was needed to advance its mission. Legislation enacted in 2012 required FDA to establish a plan for measuring its progress on its regulatory science efforts. GAO was asked to examine FDA's progress on its regulatory science efforts related to medical products. In this report, GAO (1) evaluates FDA's strategic planning efforts to address its regulatory science priorities, (2) describes FDA's funding targeted at regulatory science projects, and (3) describes the achievements of selected FDA regulatory science projects. GAO compared related FDA strategic planning documents to federal internal control standards and leading practices for strategic planning. GAO reviewed FDA data on obligations targeted at regulatory science projects for fiscal years 2010 through 2014 and reviewed the achievements FDA reported from a sample of 17 projects, chosen to ensure nine FDA centers and offices and priority areas are represented. The Food and Drug Administration (FDA) lacks measurable goals to assess its progress in advancing regulatory science—the science supporting its effort to assess the products it regulates. The agency issued strategic planning documents in 2011 and 2013 to guide its regulatory science efforts and identify priority areas for conducting work, but these documents do not specify the targets and time frames necessary for the agency to measure progress overall or within each of the eight priority areas related to medical products. According to leading practices for strategic planning, identifying and using consistent measurable goals in planning and progress documents is important to assessing effectiveness. While FDA cited examples of its achievements in regulatory science in a 2015 report, FDA cannot assess how those achievements constitute progress towards its goals. In addition, FDA lacks information about how funding targeted at regulatory science is distributed across the priority areas. Decisions to award these funds are made by individual FDA centers and offices, which generally did not collect information on the associated priority areas of funded projects. Rather, FDA retrospectively identified these areas for the purpose of GAO's review. The lack of consistent information limits FDA's ability to examine obligations across, or progress within, specific priority areas. Standards for internal control in the federal government state that complete and accurate data are needed to make operating decisions and allocate resources. Furthermore, multiple centers or offices fund projects toward a given priority area and leading practices for strategic planning encourage agencies to manage efforts that cut across the agency. For the 17 regulatory science projects GAO reviewed, FDA identified achievements ranging from the dissemination of project findings to changes in both agency and external stakeholder practices. For example, FDA reported that all projects resulted in some type of change within FDA. About half of the projects resulted in the agency developing standards, methods, tools, or training that it could use internally, and about one-third of the projects affected guidance or regulations. FDA also reported that about half of the projects resulted in the development of new tools or standards for use by industry or other stakeholders, in areas such as setting new standards for defibrillators to account for radio interference. To improve strategic planning for regulatory science efforts, FDA should (1) develop and document measurable goals, including targets and time frames, and (2) systematically track funding across its regulatory science priority areas. The Department of Health and Human Services agreed with GAO's recommendations.