This section discusses the offices within FDA and other federal agencies that help oversee imported food products. It also discusses FDA’s approach for overseeing the safety of the imported food products for which it is responsible, including milk, seafood, fruits, and vegetables. FDA’s responsibilities for overseeing the safety of imported food products are divided among its product centers and program offices. For example, the Center for Food Safety and Applied Nutrition (CFSAN) is responsible for regulating food and cosmetics products. FDA’s Office of Regulatory Affairs (ORA) is the lead office for ensuring the safety of imported food through various field activities, such as inspections of firms, review of imported products, examination of products, and sample collection and analysis. ORA works closely with the centers and with the Office of International Programs (OIP), which coordinates some of the agency’s international activities. FDA has staff in China, Chile, Costa Rica, India, Mexico, and the European Union. These foreign offices help ORA obtain foreign scientific and regulatory information, conduct investigations and facility inspections in foreign countries, and facilitate collaboration with foreign regulators in areas of common interest. Primary responsibility for imported food safety is divided between FDA and FSIS, but other federal agencies play a role. For example, CBP supports FDA in its enforcement of its food safety regulations at the border, among other things. CBP’s automated systems process all imported shipments, including food. In addition, NMFS provides fee-for- service inspection services, upon request, to the seafood industry— including domestic and foreign processors, distributors, and other firms— for example, to certify that these seafood firms comply with FDA’s Hazard Analysis and Critical Control Point regulations and other federal food safety standards. Some retailers request this certification as a condition for purchasing seafood products. FDA oversees the safety of imported food by promoting corporate responsibility, examining food prior to it entering into U.S. commerce to help ensure that unsafe food does not enter the country, and responding when unsafe food does enter the country. In 2011, FSMA provided FDA with new provisions that hold imported foods to the same standards as domestic foods and prevent unsafe food from entering the country. These new provisions included new requirements and authorities for FDA, including a requirement to develop a comprehensive plan to expand the food safety capacity of foreign governments; a requirement to increase inspections of foreign food facilities and the authority to enter into cooperative arrangements with foreign governments to facilitate the inspection of foreign food facilities; the authority, under certain circumstances, to require as a condition of admission that imported food be accompanied by certifications or assurances from accredited third-party certification bodies, agencies, or representatives of foreign governments where the food originated; and the authority to require importers to verify that their foreign suppliers meet applicable U.S. food safety standards. Owing in part to the volume of imported food, FDA cannot physically examine every shipment; the agency examines about 1 percent of entry lines annually. FDA electronically screens all imported food shipments to determine which imports to physically examine at the border and which imports to allow into U.S. commerce. The electronic screening process consists of two phases: (1) Prior Notice screening, which is intended to protect against potential terrorist acts and other public health emergencies, and (2) admissibility screening, which is intended to ensure that the food is admissible under the Food, Drug, and Cosmetic Act (FDCA). Imported food products are generally considered admissible if they are in compliance with applicable FDCA regulations that ensure food is not adulterated, misbranded, manufactured or packed under insanitary conditions, or restricted in sale in the country in which it was produced or from which it was exported, among other things. The first phase, Prior Notice screening, requires that an importer, broker, or other entity submit notification to FDA of food being imported or offered for import into the United States before that food arrives at the port of entry. Prior Notice information includes data such as the names and addresses of the manufacturer or grower, importer, owner, and ultimate consignee (the “deliver to” location); FDA product code for the food item; and country of production. Information that FDA uses for Prior Notice review can either be submitted electronically through CBP’s system, which passes the information to FDA, or submitted electronically directly to FDA. FDA targets, screens, and reviews the information to ensure that the information meets the Prior Notice requirements and to determine whether the food potentially poses a terrorism threat or other significant health risk. If such risks are identified, FDA is to work with CBP to examine the shipment upon arrival at the port. If adequate Prior Notice information is not provided, the food is subject to refusal of admission and may not be delivered to the importer, owner, or consignee. If FDA subsequently verifies that adequate Prior Notice information is provided and the food does not appear to pose a terrorism threat, the shipment is allowed to proceed to FDA’s admissibility screening. During the second phase, admissibility screening, FDA electronically screens entry lines using PREDICT to determine their level of risk. “Risk”—reflected by the PREDICT risk score—includes factors such as the inherent health risk of the product, compliance risk associated with firms, facility inspection results, and broker history, among others, and is then compared to all other entry lines within a specified commodity over the past 30 days. If PREDICT determines that an entry line poses a low risk, PREDICT recommends the entry be allowed to enter into U.S. commerce, and another FDA system, called the Operational and Administrative System for Import Support (OASIS), issues a system- generated “May Proceed” message, allowing the product into U.S. commerce without further review. If PREDICT determines that the entry line poses a high risk, an entry reviewer decides whether to examine the entry line for admissibility. The entry reviewer determines if the entry line is under import alert. If the entry line is not under import alert and no further examination is needed, the entry reviewer allows the entry line into U.S. commerce by issuing a manual “May Proceed” message through OASIS. The entry reviewer may determine that a field or laboratory examination is needed for an entry line that is not under an import alert. In this case, an FDA investigator examines the entry either at the port of entry or at another location, such as the importer’s or consignee’s warehouse or a cold storage facility. The investigator may examine a product’s label to determine whether it meets labeling requirements. Additionally, the investigator may examine the shipment for rodent or insect activity or inadequate storage while in transit, among other things. Figure 2 shows FDA investigators examining imported food. If the product does not appear to be violative after the examination, the owner or consignee receives a notice stating that the line is released, and the product is allowed into U.S. commerce. If the product appears to be violative, the investigator may decide to take samples from the product for a laboratory examination to test for rodent or insect activity and other such elements that could confirm that the entry is violative. If the samples indicate that the food is not violative, the owner or consignee receives a notice stating that the line is released, and the product is allowed into U.S. commerce. If the samples are violative, the owner or consignee receives a notice stating that the line is being detained and is subject to refusal. Once the owner or consignee receives a notice stating that the line is being detained and is subject to refusal, the owner or consignee may request that FDA immediately refuse the product, and the product must either be exported or destroyed. If the owner or consignee does not request refusal, then the owner or consignee decides whether to submit testimony or request to “recondition” the product—for example, relabeling the product or converting the product into a type of product not regulated by FDA. If the owner or consignee submits testimony regarding the admissibility of the food or FDA approves a request to recondition the product, then an FDA hearing determines whether the product should be released. If FDA determines that the owner or consignee has provided sufficient information to overcome the appearance of a violation, the owner or consignee receives a notice stating that the product is released. If FDA determines that the owner or consignee’s actions failed to bring the product into compliance, the food must be exported or destroyed. If the product is subject to an import alert, FDA determines whether the product should be detained without physical examination. If FDA determines that the product should not be detained, the owner or consignee receives a notice stating that the product is released. If FDA determines that the product should be detained, FDA issues a notice stating that the line is being detained and is subject to refusal, and the owner or consignee decides whether to export or destroy the product. Figure 3 illustrates the key elements of FDA’s screening process. From 2012 to 2014, about 0.1 percent of all food entry lines were refused and exported or destroyed each year. During this period the countries with the highest number of entry lines that were refused were India, Mexico, and China and the most commonly refused items included rice, herbals and botanicals (not teas), tuna, shrimp and aquaculture- harvested seafood products, and vitamins and minerals. When unsafe food enters the country, FDA may respond by issuing advisories about the affected food. Depending on the circumstances, FDA may also use other enforcement tools, such as seizure, injunction, and administrative detention. FDA can also seek voluntary recalls of unsafe food. If a responsible party does not voluntarily recall the food, FDA may issue a mandatory recall order, provided that certain criteria are met. In early 2010, FDA began an effort to assess foreign food safety systems to determine whether certain other countries have a regulatory system— including food safety statutes, regulations, and an implementation strategy—that is comparable to the U.S. food safety system. Under this effort, FDA in 2010 developed a systems recognition tool to assess the overall food safety systems of foreign countries for foods under FDA’s jurisdiction. FDA completed its on-site review for a systems recognition pilot with New Zealand food safety authorities in late 2010 and established a Systems Recognition Arrangement in 2012. In May 2016, FDA reported that it had established a similar arrangement with Canada. PREDICT uses a variety of data and FDA-created rules—conditional statements that tell PREDICT how to react when encountering particular information—to analyze these data and to identify high-risk imported food shipments. These data include information from FDA sources; non-FDA domestic sources, such as other federal and state agencies; and foreign sources, such as foreign governments. Some of the domestic and foreign sources are open sources—that is, publicly available. FDA does not currently have a documented process for obtaining open source data. PREDICT analyzes all of the data by applying rules that contribute to risk- based scores. PREDICT then recommends potential FDA actions, such as holding an entry line for examination or allowing it to proceed into commerce. PREDICT uses a variety of data to estimate the risk of imported food. These data come from internal FDA sources, other domestic sources, and foreign sources. Figure 4 provides an overview of the types of data sources PREDICT uses. Many of the data PREDICT uses come from sources within FDA. These include electronic sources, such as databases, and human sources, such as FDA officials. Electronic sources. Most of the electronic FDA sources that PREDICT uses are databases that contain historical data about products, firms, and other elements of imported shipments, such as the geographic location of facilities. PREDICT uses numerous kinds of historical data stored in various FDA databases, such as data about (1) the previous field examination and facility inspection history of foreign firms, (2) the track record of importers and brokers, and (3) the types of food products historically imported from other countries. One of the other databases that PREDICT uses contains official FDA safety violation data, such as FDA import alerts and import bulletins. Another database that PREDICT uses contains registration data for facilities that manufacture, process, or pack a specific type of food product, acidified and low-acid canned foods, which are specifically tracked because growth of the botulinum bacterium in canned food may cause a deadly form of food poisoning. Human sources. The human FDA sources that PREDICT draws upon include FDA officials who compile and report data based on their knowledge of products, firms, and other elements of imported shipments, such as country of origin. For example, certain PREDICT rules use FDA’s knowledge and experience to determine the inherent risk of certain food products, and certain other PREDICT rules draw from data reported by FDA entry reviewers about the past record of products or firms. According to FDA officials, PREDICT also uses data from sources outside of FDA, including domestic sources, such as databases from states and other federal agencies. For example, PREDICT uses data from CBP. Specifically, CBP collects entry data from importers for products being imported into the United States and electronically transfers the data to FDA. PREDICT also uses data from other federal agencies, such as from NMFS. NMFS maintains a list of approved seafood establishments, based on its fee-for-service inspections, on its website and may provide this list electronically to FDA upon request. However, FDA only requests information from NMFS on an occasional, as-needed basis. For example, FDA requested information from NMFS in 2010 during the Gulf of Mexico oil spill. More recently, FDA requested NMFS audit reports and facility approvals for certain overseas locations. FDA officials told us that PREDICT also uses data from foreign sources, such as foreign governments. Specifically, FDA’s domestic and overseas offices obtain information from foreign regulatory counterparts on an occasional, as-needed basis about food safety in their respective countries and provide summary information to ORA that may be used in PREDICT. FDA officials told us that the agency may receive nonpublic information from foreign governments only in cases where formal arrangements called Confidentiality Commitments are in place. Of the more than 230 entities from which the United States imports food, FDA has 39 foods-related Cooperative Arrangements with 24 foreign governments—including 1 arrangement with the European Union, which has 28 member countries. FDA officials told us that they are in the process of developing additional cooperative arrangements. An FDA office that is involved in obtaining information from foreign regulatory counterparts is OIP, which has offices in the United States and abroad that help obtain foreign scientific and regulatory information by coordinating with foreign regulators or accessing foreign media sources. One country covered by OIP’s Office of Regional and Country Affairs is Japan. In March 2011, after a 9.0 magnitude earthquake and subsequent tsunami in Japan, the Fukushima Daiichi nuclear power plant suffered extensive damage. The Office of Regional and Country Affairs coordinated with the Japanese government to identify Japanese exports that have a high risk of contamination. As a result, FDA issued Import Alert 99-33: “Detention Without Physical Examination of Products from Japan Due to Radionuclide Contamination.” This alert informs FDA field personnel that they may detain without physical examination certain food shipments from Japan. FDA officials told us that years after the disaster, the agency continues to obtain information from the Japanese government, and the import alert is still in place and was last revised in April 2016. As of March 10, 2014, FDA had tested 1,345 import and domestic samples specifically to monitor for Fukushima contamination. Of the 1,345 samples, 2 were found to contain detectable levels of a contaminant, but the levels posed no public health concern. Both the non-FDA domestic sources and the foreign sources may include open sources—that is, sources that are publicly available, such as newspapers and Internet blogs. Data from open sources may provide information about events—such as food recalls and natural disasters— that could affect the risk of imported foods. Although ORA collects some open source data on its own, it also relies on other FDA offices and federal agencies that gather open source data for their own purposes and that may communicate to ORA information that seems relevant to the import screening mission. For example, OIP’s Europe office monitors news related to FDA-regulated products in Europe. If OIP obtains recall information that it deems significant to FDA-regulated commodities, it communicates the information to ORA and other relevant FDA offices. ORA may also obtain open source data from other federal agencies, such as from the agencies represented at CBP’s Commercial Targeting and Analysis Center, which targets and screens commercial shipments that may pose a threat to the health and safety of U.S. consumers. Open source data have proven useful in the past. For example, in August 2015, an explosion occurred at a warehouse in the port of Tianjin, China, involving hazardous chemicals, including cyanide, posing a potential threat to public health. In the aftermath, OIP, through its China Office, collected information about the explosion from local media sources. As a result, FDA increased surveillance of FDA-regulated imports from manufacturing, processing, and/or packing facilities in the Tianjin area. In this particular instance, FDA also notified importers of this increased level of surveillance. Prior to 2015, FDA had a formal contract dedicated to the collection of open source data for PREDICT. FDA’s 2013 PREDICT operating manual, PREDICT Guide: Rules and Scoring, documented how the agency was to obtain open source data, the type of data to be provided by the contracted company—such as data about natural disasters—and how PREDICT used the data. According to the manual, the PREDICT Open- Source Intelligence Team used open source data to locate news stories pertaining to food safety around the world, looking specifically for geophysical events (e.g., tsunamis, typhoons, and earthquakes) or ecological events (e.g., algal blooms, overuse of antibiotics, and contamination from various spills that could affect FDA-regulated commodities). Analysts then researched each event or action to understand the content and context of the issue, develop an analytic conclusion, and recommend a risk score and expiration date. FDA terminated that contract because the agency determined that it was not cost-effective, given the availability of information from other public sources, such as the Internet and other offices within the agency. The 2015 version of FDA’s PREDICT operating manual does not document the process for identifying the type of open source data to collect, obtaining such data, and determining how PREDICT is to use the data. Instead, FDA relies on ORA officials to informally communicate and obtain information from officials in other FDA offices and from other federal agencies on an ad hoc basis. FDA told us that because the agency did not renew the contract, the 2015 version omitted references to obtaining and using open source information. ORA officials told us they now use a variety of informal and situation-dependent methods to obtain open source intelligence, but these methods are not formally documented. We have previously found that by using informal coordination mechanisms, agencies may rely on relationships with individual officials to ensure effective collaboration and that these informal relationships could end once personnel move to their next assignments. Under federal standards for internal control, agencies are to clearly document internal control, and the documentation is to appear in management directives, administrative policies, or operating manuals. Without a documented process, FDA does not have reasonable assurance that it is consistently identifying the type of open source data to collect, obtaining such data, and determining how PREDICT is to use the data in a regular and systematic manner. A senior ORA official in its Division of Import Operations told us that formalizing and documenting the process by which FDA receives information from other entities would improve the current process for using open source information. FDA creates rules, conditional statements that tell PREDICT how to react when encountering particular situations. For example, if a certain PREDICT rule encounters the term “tamarind” in the description of a product, it should recommend detaining the item without physical examination. This rule was created as the result of an incident in which filth was found in tamarind products. PREDICT’s application of rules generates a cumulative risk score for each entry line; the cumulative risk score includes factors such as the product’s inherent risk and the manufacturing firm’s facility inspection history. The risk score can indicate a negative risk value (signifying low risk) or a positive risk value (signifying higher risk). The individual risk values assigned to the various elements of the entry line help produce the cumulative risk score for the entire entry line and possibly also one or more recommended actions. The application of a rule may prompt a flag (for example, an import alert flag), which may provide information—such as that a manufacturer’s product was recently examined or that the product was refused admission by one or more foreign countries—and recommended an action. For example, it may point out that some of the entry line data are missing or invalid—based on a comparison with other FDA databases—and recommend performing a manual review of databases. After PREDICT has applied the rules to the entry data, entry reviewers analyze the results. The first thing an entry reviewer sees is the main entry review screen, which lists all the current entry lines to be reviewed and their PREDICT risk scores and flags. The entry reviewer can click on a PREDICT risk score to see a table, referred to as the PREDICT Mashup. The PREDICT Mashup displays the individual risk values that contributed to the overall PREDICT risk score for that entry line. For example, the PREDICT Mashup may show that PREDICT assigned the product’s manufacturer a risk-based score of 10 for its facility inspection history, which would signify a high risk. The same Mashup may show that PREDICT assigned the product’s country of origin a risk-based score of 3 for its history, which would signify a lower risk. The entry reviewer can also “roll over,” or move the cursor over, flags to obtain more information. For example, an entry reviewer may roll over a “recent targeted activity” flag—which indicates that this product from this manufacturer was recently examined—to obtain information (sometimes through another database) about when the product was examined and the results of the examination. The entry reviewer uses all of this information to decide how to proceed. For example, the entry reviewer may decide to allow the entry line to proceed into commerce or to hold it for examination or laboratory analysis. Any action taken by the entry reviewer, including the results of any examination, will become part of the history of the facility, which will then contribute to future PREDICT scores. FSMA, which was enacted in 2011, contains provisions that will provide PREDICT with additional data to analyze (i.e., from additional sources) when estimating the risk of imported food. FDA officials told us that even with the changes brought about by FSMA, PREDICT will continue to be used to target imports for examination but will now draw data from even more sources. FDA officials identified five FSMA provisions and authorities as likely to generate the most data for use in PREDICT. Of these five, the first three are as follows: Foreign Supplier Verification Program (FSVP): The FSVP rule requires importers to verify that their foreign suppliers use processes and procedures that provide the same level of public health protection as the hazard analysis and risk-based preventive controls and other applicable requirements of FDCA. For example, importers must develop, maintain, and follow an FSVP that provides adequate assurances that the foreign supplier is producing food that is, among other things, not adulterated (e.g., that it does not contain Salmonella) and is not misbranded with respect to labeling for the presence of major food allergens. Verification activities could include monitoring records, annual on-site inspections or review of such inspections performed by a qualified auditor, and testing and sampling of shipments. The date by which importers must comply with the FSVP regulations depends on a number of factors. Inspection of foreign food facilities: FSMA includes a provision that authorizes FDA to direct resources according to the known safety risks of facilities, especially those that present a high risk. The law directs FDA to inspect at least 600 foreign facilities within 1 year of enactment of FSMA and, in each of the 5 years following that period, to inspect at least twice the number it inspected during the previous year. Laboratory accreditation: FSMA also includes a provision that requires FDA to establish a program for the testing of food by accredited laboratories and to recognize accreditation bodies for accrediting the laboratories, including independent private laboratories. The provision requires owners or consignees to have food products tested by an accredited laboratory in certain circumstances, including when a food product is under an import alert that requires successful consecutive tests. The results of these tests may be submitted to FDA electronically and used to determine compliance and admissibility of the food product. FDA has not yet issued a proposed rule for establishing the program, but an agency official stated that the agency is working on a rule now. The other two FSMA provisions likely to generate data for use in PREDICT are related to third-party certification, for which FDA issued the final rule in November 2015. FSMA directs the establishment of a system for the recognition of accreditation bodies to accredit third-party certification bodies (also known as auditors) to certify that a foreign facility, or any product produced by the facility, meets the applicable FDA food safety requirements. If an accredited third-party certification body or its audit agent discovers a condition that could cause or contribute to a serious risk to public health, the certification body must immediately notify FDA. The following two FSMA provisions rely on third-party certifications and will likely generate data for use in PREDICT: Voluntary Qualified Importer Program (VQIP): FSMA requires FDA to establish a program that offers expedited review and entry to certain participating importers that import food from foreign facilities certified by accredited third-party certification bodies. FDA published draft guidance for VQIP in the Federal Register in June 2015. According to an FDA Fact Sheet on VQIP, the agency expects to begin receiving applications for the program in January 2018. Import certifications: Under FSMA, FDA has the authority, under certain circumstances, to require certifications or other assurances from agencies, or representatives of foreign governments where the food originated, or an accredited third-party certification body. FDA can determine that an article of food requires certification based on, among other factors, the known safety risks associated with the food; the country, territory, or region of origin of the food; or evidence that the food safety systems for that product are inadequate. FDA published the final rule on accreditation of third-party certification bodies in the November 27, 2015, Federal Register. Because some of these FSMA-related programs are still new and not yet fully implemented, the details of how certain data generated by these programs will be integrated into PREDICT are not finalized, but according to FDA officials, PREDICT will use data produced by these programs and authorities in a variety of ways. Implementation of FSVP, for example, will identify importers that are not in compliance, and PREDICT would use that information in assigning risk scores to products from those importers. The food products from suppliers and importers that have a history of noncompliance with food safety regulations will receive higher PREDICT risk scores than those that are consistently in compliance. In addition, if the results from an inspection of a foreign food facility show that the facility is noncompliant, that information would be provided to PREDICT, and PREDICT would use that information in assigning risk scores to imports from that facility. Figure 5 shows how the implementation of FSMA will add to the data sources already used by PREDICT. FDA has assessed the effectiveness of PREDICT by ongoing monitoring of key program data and by conducting an evaluation of the tool in 2013, and it has implemented many, but not all, of the 2013 evaluation’s recommendations to improve PREDICT. Data maintained by agency officials show that PREDICT is working properly to focus reviewers’ attention on food items that have a high probability of being violative. Moreover, when FDA officials conducted an evaluation of the system in 2013, results showed that PREDICT was working well to provide staff with the data they need to make informed entry review decisions. However, the agency has not yet implemented all of the 24 recommendations resulting from that evaluation. FDA officials told us that they conduct monitoring of PREDICT on an ongoing basis and that the data they collect as part of this monitoring show that PREDICT is working as intended—that is, PREDICT is focusing entry reviewers’ attention on items determined to be of higher risk. Data provided by FDA from fiscal years 2012 to 2014 confirmed that in general, PREDICT is fulfilling this role. Our analysis of these data showed that in general, the higher the PREDICT risk score, the more often entry lines were examined and the more often they were found violative. As tables 1, 2, and 3 show, for fiscal years 2012 through 2014, entry lines that received higher PREDICT scores generally were more often selected for a field examination, for a label examination, or for sampling, and entry lines with higher PREDICT scores were more often found violative. In all 3 years we reviewed, items with the highest PREDICT scores— those from 91 to 100—were selected for examination more often than items with lower PREDICT scores. For example, in fiscal year 2012, FDA examined or sampled more than 39,000 entry lines with a PREDICT score from 91 to 100, or about 6.5 percent of all entry lines in this range, nearly double the percentage of entry lines examined in the next-highest 10 point risk score range and at least 6 times more than entry lines with a score of 50 or less. The tables also show that in each of the 3 years we reviewed, the higher the PREDICT score, the higher the percentage of entry lines that were found to be violative. For example, in fiscal year 2014, FDA selected for examination about 45,000 of the nearly 1.3 million entry lines with PREDICT scores of 91 to 100. Of these, nearly 12 percent, or 5,363 line items, were found to be violative, almost double the percentage of violation rates found in the next-highest range of PREDICT scores. In May 2013, FDA completed an internal evaluation of PREDICT’s effectiveness that examined five processes: work planning, examinations and sampling, entry review, rules management, and communication. To conduct this evaluation, FDA officials gathered information through a variety of methods, including analysis of entry line data and the entry review process, interviews with stakeholders from the various FDA centers, and surveys of PREDICT users in the field. Overall, the evaluation showed that PREDICT is helping to expedite the release of low-risk items and to identify violative imports. As a result of the internal evaluation, FDA developed a total of 24 recommendations: 23 for improving the five processes noted above and another recommendation concerning the development of performance metrics. The list below shows the categories of recommendations and the number of recommendations in each category: work planning (4), examination and sampling (4), entry review (6), performance metrics (1). rules management (5), communication (4), and FDA prioritized each of its 24 recommendations according to the feasibility of near-term implementation and the impact on operations or public health, among other things. A recommendation was determined to have high feasibility if resources were available to implement it and the implementation plan was clear. A recommendation was determined to have high impact if, for example, many stakeholders cited the problem that the recommendation was intended to solve. As figure 6 shows, of the 24 recommendations, 6 were determined to be highly feasible, and 16 were determined to be of high impact or medium/high impact; 2 recommendations were determined to be both highly feasible and of high impact. Examples of FDA’s recommendations and how they were prioritized include analyze and set specific “May Proceed” thresholds by commodity (designated as both highly feasible and of high impact), develop a system to test PREDICT rules (designated as highly feasible and of medium impact), and identify sources of delays associated with the sampling process (designated as of medium feasibility and medium impact). FDA has evaluated the 24 recommendations and has implemented many, but not all, of them. According to agency officials, the agency has implemented 15, partially implemented another 6, and not implemented 3 (see fig. 7). Of the 15 recommendations that were implemented, 13 were designated either as highly feasible or of high or medium/high impact. Both of the recommendations determined to be both highly feasible and of high impact were implemented, including the recommendation to analyze and set specific “May Proceed” thresholds by commodity. Of the 6 that were partially implemented—which includes the recommendation to develop a system to test PREDICT rules—5 were designated as either highly feasible or of high or medium/high impact. Finally, of the 3 that were not implemented—which includes the recommendation to identify sources of delays associated with the sampling process—1 was of low feasibility and high impact, 1 was of medium feasibility and high impact, and 1 was of medium feasibility and medium impact. Federal standards for internal control state that agencies are to ensure that the findings of audits and other reviews are promptly resolved. To that end, agencies are to complete, within established time frames, all actions that correct or otherwise resolve the matters brought to management’s attention. According to FDA officials, the reason for partially implementing or not implementing recommendations was often a lack of resources. For example, FDA officials said that a lack of resources was why they only partially implemented the recommendation to develop a system to test PREDICT rules and did not implement the recommendation to identify sources of delays associated with the sampling process. Agency officials agree that implementing the remaining recommendations would improve the effectiveness of PREDICT, but the agency has not yet established time frames for completing the implementation of the remaining recommendations. Establishing a practical timeline for implementing the remaining recommendations as resources become available would provide FDA with reasonable assurance that the improvements are made and that PREDICT remains an effective tool for screening imports. The volume of FDA-regulated imported food continues to grow, as does the need to ensure that such imports are safe. FDA physically examines about 1 percent of food shipment entry lines annually. The agency developed PREDICT to help target food shipments deemed higher risk and subject them to additional scrutiny. FDA’s assessment of PREDICT shows that the tool is generally working to focus FDA’s resources on the examination of food items determined to be of highest risk and expediting the release of lower-risk food items. PREDICT analyzes imported food entry lines using data from various sources, including FDA sources, other domestic sources, and foreign sources. Some of the domestic and foreign sources are open sources, and FDA uses such sources to obtain information about imported foods on an ad hoc basis. However, FDA does not have a documented process for identifying the type of open source data to collect, obtaining such data, and determining how PREDICT is to use the data. Without such a documented process, FDA does not have reasonable assurance that it will consistently obtain open source data for PREDICT in a regular and systematic manner. In addition, FDA’s May 2013 evaluation of PREDICT identified 24 recommendations to improve the tool, and FDA has fully implemented 15 of these recommendations. FDA officials explained that resource constraints have often limited their ability to fully implement all the remaining recommendations. Agency officials agree that implementing these recommendations would improve the effectiveness of PREDICT, but the agency has not yet established time frames for doing so. Establishing a practical timeline for implementing the remaining recommendations would help FDA ensure that the improvements are made and that PREDICT remains an effective tool for helping to prevent high-risk foods from entering the United States. To further enhance FDA’s PREDICT tool and its ability to ensure the safety of imported food, we recommend that the Secretary of Health and Human Services direct the Commissioner of FDA to take the following two actions:  document the process for identifying the type of open source data to collect, obtaining such data, and determining how PREDICT is to use the data and  establish a timeline for implementing, as resources become available, the remaining recommendations from FDA’s 2013 evaluation of PREDICT. We provided the departments of Commerce, Health and Human Services, and Homeland Security a draft of this report for their review and comment. In an e-mail, the Department of Commerce stated that it had no comments. The Department of Health and Human Services generally concurred with the recommendations and provided written comments on the draft, which are summarized below and presented in their entirety in appendix II of this report. The Department of Homeland Security provided technical comments, which we incorporated as appropriate. In its written comments, the Department of Health and Human Services concurred with our first recommendation and concurred in part with the second. For our first recommendation, HHS stated that ORA plans to work with appropriate units across the agency to develop and document a formal process to identify the type of information to collect, how to obtain the information, and how PREDICT may use it. For our second recommendation, HHS stated that it has recently fully implemented one of the internal recommendations from the 2013 internal evaluation that was previously designated as “partially implemented.” However, because HHS did not provide evidence of fully implementing the recommendation, we were unable to verify such implementation and made no change to the report. The Department also stated that it is supportive of the remaining recommendations from its 2013 internal evaluation but noted that establishing an implementation timeline is based on consideration of the feasibility and impact relative to available FDA resources and other FDA priorities, such as FSMA. The department indicated that the goals of the unimplemented recommendations may ultimately be achieved via these other FDA initiatives. We are sending copies of this report to the appropriate congressional committees, the Secretary of Commerce, the Secretary of Health and Human Services, the Secretary of Homeland Security, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or morriss@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff members who made key contributions to this report are listed in appendix III. You asked us to examine how the Food and Drug Administration (FDA) is using Predictive Risk-based Evaluation for Dynamic Import Compliance Targeting (PREDICT) to protect the safety of the U.S. food supply. Our objectives in this report were to examine (1) the data used by PREDICT and how PREDICT analyzes these data to identify high-risk food shipments for examination; (2) how implementation of the FDA Food Safety Modernization Act (FSMA) will affect PREDICT; and (3) the extent to which FDA has assessed the effectiveness of PREDICT and used the results of those assessments to improve the tool. To determine the data used by PREDICT and how PREDICT analyzes these data to identify high-risk food shipments for examination, we reviewed documents that described the data used by PREDICT and interviewed FDA officials. We assessed the data sources, data quality controls, and the methodology used by PREDICT and determined that they were sufficiently reliable to support accurate descriptions of the system inputs, processes, and resulting risk scores. We also assessed the data used by PREDICT to ensure that they were sufficiently reliable to describe factors considered by PREDICT and system rules used to generate risk scores. We interviewed FDA officials in the Office of International Programs to understand how FDA coordinates with foreign governments to obtain data for PREDICT. We also reviewed how the agency identifies, obtains, and uses open source information and compared this process to criteria for documentation found in the federal standards for internal control. To determine the extent to which the implementation of FSMA will affect PREDICT, we reviewed FSMA, FDA’s rules implementing FSMA, and FDA planning documents and interviewed FDA officials. To determine the extent to which FDA has assessed the effectiveness of PREDICT and used the results of those assessments to improve the tool, we interviewed FDA officials about ongoing monitoring of PREDICT, reviewed the 2013 evaluation that FDA has conducted on PREDICT, and assessed the extent to which FDA has implemented the recommendations from the evaluation. We then compared FDA’s actions to the criteria found in the federal standards for internal control regarding the findings of audits and other reviews. In addition, we examined data provided by FDA on PREDICT risk scores and violation rates from fiscal years 2012 through 2014, the most recent years for which data were available, and determined that the data were sufficiently reliable for our purposes, as discussed above. To inform all three objectives, we conducted site visits at four FDA and U.S. Customs and Border Protection (CBP) facilities located at ports of entry located in Baltimore, Maryland; Otay Mesa in San Diego, California; Long Beach California; and Los Angeles, California. We selected these sites to include air, ship, and truck ports of entry; variable geographic locations; the variability of food products that enter through the ports; and proximity to an FDA office. The information we obtained at these sites is not generalizable to all facilities and ports of entry. In addition, we interviewed officials from CBP and the National Marine Fisheries Service to understand how FDA coordinates with other federal agencies to protect the safety of the U.S. food supply. We also interviewed officials from a number of stakeholder organizations that are affected by PREDICT or contributed to PREDICT’s development. We conducted this performance audit from January 2015 to May 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Anne K. Johnson (Assistant Director), Kevin Bray, Ellen Fried, Steven Putansu, Gloria Ross, Stuart Ryba, Sara Sullivan, Vasiliki Theodoropoulos, and Karen Villafana made key contributions to this report.

Summary: Imported food makes up a substantial and growing portion of the U.S. food supply. FDA is responsible for oversight of more than 80 percent of the U.S. food supply. However, because the volume of imported food is so high, FDA physically examines only about 1 percent of imported food annually. In 2011, FDA implemented PREDICT, a computerized tool intended to improve FDA's targeting of imports for examination by estimating the risk of imported products. GAO was asked to review how FDA is using PREDICT to protect the U.S. food supply. This report examines (1) the data used by PREDICT and how PREDICT analyzes these data to identify high-risk food shipments for examination, (2) how implementation of FSMA will affect PREDICT, and (3) the extent to which FDA has assessed the effectiveness of PREDICT and used the assessments to improve the tool. To address these issues, GAO analyzed FDA documents, interviewed FDA officials, and analyzed data from fiscal years 2012 through 2014. The Food and Drug Administration's (FDA) Predictive Risk-based Evaluation for Dynamic Import Compliance Targeting (PREDICT) tool uses a variety of data and analyzes data by applying rules—conditional statements that tell PREDICT how to react when encountering particular information—to generate risk scores for imported food. Many of the data used by PREDICT come from internal FDA sources, such as FDA databases. PREDICT also uses data from sources outside of FDA, such as other federal agencies, states, and foreign governments. Some of the data are open source data—information that is publicly available, such as information from newspapers and websites. FDA's Office of Regulatory Affairs (ORA) relies on other FDA offices and federal agencies to provide open source data for PREDICT, but ORA does not have a documented process for identifying, obtaining, and using the data, relying instead on an ad hoc process. Federal standards for internal control call for agencies to document internal controls. Without a documented process for identifying, obtaining, and using open source data, FDA does not have reasonable assurance that ORA will consistently identify, obtain, and use such data for PREDICT. The implementation of the FDA Food Safety Modernization Act (FSMA), enacted in 2011, will provide PREDICT with additional data for estimating the risk of imported food. FDA identified FSMA provisions likely to generate data, including the Foreign Supplier Verification Program (FSVP), which requires importers to verify that their foreign suppliers use processes and procedures that provide the same public health protection as applicable U.S. requirements. Because FSVP and other FSMA-related programs are still new and not yet fully implemented, the details of how PREDICT will use the data have not been worked out. However, according to FDA officials, the data will be useful. For example, FSVP data will identify suppliers that are not in compliance with standards, and PREDICT will use those data in assigning risk scores to imports from those suppliers. FDA has assessed the effectiveness of PREDICT by monitoring of key data and by conducting an internal evaluation of the system, and it has implemented many, but not all, of its recommendations from the evaluation. GAO's analysis of FDA data from fiscal year 2012 through 2014 shows that in general, PREDICT is working as intended: imported food with higher-risk scores is more likely to be physically examined and to be found in violation of food safety standards or labeling requirements. In May 2013, FDA completed an evaluation of PREDICT that examined five key processes. As a result of the evaluation, FDA developed 24 recommendations for improving PREDICT and prioritized these recommendations based on feasibility and impact. According to FDA, the agency has fully implemented 15, partially implemented 6, and not implemented 3 of the recommendations. FDA officials said that the agency has not fully implemented all recommendations because of a lack of resources. However, federal standards for internal control specify that agencies are to ensure that the findings of reviews are promptly resolved. To that end, agencies are to complete, within established time frames, all actions that correct or otherwise resolve the matters brought to management's attention. Establishing a timeline for implementing the remaining recommendations as resources become available would help ensure that PREDICT continues to remain an effective tool for screening imported food. GAO recommends that FDA take two actions to improve the effectiveness of PREDICT: (1) document the process by which FDA is to identify, obtain, and use open source data and (2) establish a timeline for implementing the remaining recommendations from FDA's 2013 evaluation of PREDICT. FDA generally agreed with GAO's recommendations.