Our May 2009 report on the then-current round of ASP testing found that DHS increased the rigor of ASP testing over that of previous tests, and that a particular area of improvement was in the performance testing at the Nevada Test Site, where DNDO compared the capability of ASP and current-generation equipment to detect and identify nuclear and radiological materials. For example, unlike in prior tests, the plan for the 2008 performance test stipulated that the contractors who developed the equipment would not be involved in test execution. This improvement addressed concerns we previously raised about the potential for bias and provided increased credibility to the results. Nevertheless, based on the following factors, in our report we questioned whether the benefits of the new portal monitors justify the high cost: The DHS criteria for a significant increase in operational effectiveness. Our chief concern with the criteria is that they require only a marginal improvement over current-generation portal monitors in the detection of certain weapons-usable nuclear materials during primary screening. DNDO considers detection of such materials to be a key limitation of current-generation portal monitors. The marginal improvement required of ASPs to meet the DHS criteria is problematic because the detection threshold for the current-generation portal monitors does not specify a level of radiation shielding that smugglers could realistically use. Officials from the Department of Energy (DOE), which designed the threat guidance DHS used to set the detection threshold, and national laboratory officials told us that the current threshold is based not on an analysis of the capabilities of potential smugglers to take effective shielding measures but rather on the limited sensitivity of PVTs to detect anything more than certain lightly shielded nuclear materials. DNDO officials acknowledge that both the new and current-generation portal monitors are capable of detecting certain nuclear materials only when unshielded or lightly shielded. The marginal improvement in detection of such materials required of ASPs is particularly notable given that DNDO has not completed efforts to fine-tune PVTs' software using a technique called "energy windowing" that could improve the PVTs' sensitivity to nuclear materials. DNDO officials expect they can achieve small improvements in sensitivity through energy windowing, but DNDO has not yet completed efforts to fine-tune the PVTs' software. In contrast to the marginal improvement required in detection of certain nuclear materials, the primary screening requirement to reduce the rate of innocent alarms by 80 percent could result in hundreds of fewer secondary screenings per day, thereby reducing CBP's workload. In addition, the secondary screening criteria, which require ASPs to reduce the probability of misidentifying special nuclear material by one-half, address the limitations of relatively small handheld devices in consistently locating and identifying potential threats in large cargo containers. Results of performance testing and field validation. The results of performance tests that DNDO presented to us were mixed, particularly in the ASPs' capability to detect certain shielded nuclear materials during primary screening. The results of performance testing at the Nevada Test Site showed that the new portal monitors detected certain nuclear materials better than PVTs when shielding approximated DOE threat guidance, which is based on light shielding. In contrast, differences in system performance were less notable when shielding was slightly increased or decreased: both the PVTs and ASPs were frequently able to detect certain nuclear materials when shielding was below threat guidance, and both systems had difficulty detecting such materials when shielding was somewhat greater than threat guidance. With regard to secondary screening, ASPs performed better than handheld devices in identification of threats when masked by naturally occurring radioactive material. However, the differences in the ability to identify certain shielded nuclear materials depended on the level of shielding, with increasing levels appearing to reduce any ASP advantages over the handheld identification devices. Other phases of testing uncovered multiple problems in meeting requirements for successfully integrating the new technology into operations at ports of entry. Of the two ASP contractors participating in the current round of testing, one has fallen behind due to severe problems encountered during testing of ASPs' readiness to be integrated into operations at ports of entry ("integration testing"); the problems may require that the vendor redo previous test phases to be considered for certification. The other vendor's system completed integration testing, but CBP suspended field validation testing in January 2009 after 2 weeks because of serious performance problems resulting in an overall increase in the number of referrals for secondary screening compared with existing equipment. DNDO's plans for computer simulations. As of May 2009, DNDO did not plan to complete injection studies--computer simulations for testing the response of ASPs and PVTs to simulated threat objects concealed in cargo containers--prior to the Secretary of Homeland Security's decision on certification even though delays to the ASP test schedule have allowed more time to conduct the studies. According to DNDO officials, injection studies address the inability of performance testing to replicate the wide variety of cargo coming into the United States and the inability to place special nuclear material and other threat objects in cargo during field validation. DNDO had earlier indicated that injection studies could provide information comparing the performance of the two systems as part of the certification process for both primary and secondary screening. However, DNDO subsequently decided that performance testing would provide sufficient information to support a decision on ASP certification. DNDO officials said they would instead use injection studies to support effective deployment of the new portal monitors. Lack of an updated cost-benefit analysis. DNDO had not updated its cost- benefit analysis to take into account the results of ASP testing. An updated analysis that takes into account the testing results, including injection studies, might show that DNDO's plan to replace existing equipment with ASPs is not justified, particularly given the marginal improvement in detection of certain nuclear materials required of ASPs and the potential to improve the current-generation portal monitors' sensitivity to nuclear materials, most likely at a lower cost. DNDO officials said they were updating the ASP cost-benefit analysis and planned to complete it prior to a decision on certification by the Secretary of Homeland Security. Our May report recommended that the Secretary of Homeland Security direct DNDO to (1) assess whether ASPs meet the criteria for a significant increase in operational effectiveness based on a valid comparison with PVTs' full performance potential and (2) revise the schedule for ASP testing and certification to allow sufficient time for review and analysis of results from the final phases of testing and completion of all tests, including injection studies. We further recommended that, if ASPs are certified, the Secretary direct DNDO to develop an initial deployment plan that allows CBP to uncover and resolve any additional problems not identified through testing before proceeding to full-scale deployment. DHS agreed to a phased deployment that should allow time to uncover ASP problems but disagreed with GAO's other recommendations, which we continue to believe remain valid. The results of DNDO's most recent round of field validation testing, which it undertook in July 2009, after our May report was released, raise new issues. In July 2009, DNDO resumed the field testing of ASPs at four CBP ports of entry that it initiated in January 2009 but suspended because of serious performance problems. However, the July tests also revealed ASP performance problems, including two critical performance deficiencies. First, the ASP monitors had an unacceptably high number of false positive alarms for the detection of certain high-risk nuclear materials. According to CBP officials, these false alarms are very disruptive in a port environment in that any alarm for this type of nuclear material would cause CBP to take enhanced security precautions because such materials (1) could be used in producing an improvised nuclear device and (2) are rarely part of legitimate or routine cargo. Furthermore, once receiving an alarm for this type of nuclear material, CBP officers are required to conduct a thorough secondary inspection to assure themselves that no nuclear materials are present before permitting the cargo to enter the country. Repeated false alarms for nuclear materials are also causes for concern because such alarms could eventually have the effect of causing CBP officers to doubt the reliability of the ASP and be skeptical about the credibility of future alarms. Secondly, during the July testing the ASP experienced a "critical failure," which stemmed from a problem with a key component of the ASP and caused the ASP to shut down. Importantly, during this critical failure, the ASP did not alert the CBP officer that it had shut down and was no longer scanning cargo. As a result, were this not in a controlled testing environment, the CBP officer would have permitted the cargo to enter the country thinking the cargo had been scanned, when it had not. According to CBP officials, resolving this issue is important in order to assure the stability and security of the ASP. In addition to these key performance problems, the ASP was not able to reduce referrals to secondary inspection by 80 percent as required by the DHS criteria for a significant increase in operational effectiveness. According to the report from the ASP Field Validation Advisory Panel, a panel made up of officials from CBP, DNDO, and a national laboratory, the ASP was able to reduce referrals to secondary inspection by about 69 percent rather than the 80 percent as required by the DHS criteria. While the performance of the ASP during the July field validation testing raises issues about its potential readiness for deployment, DNDO's proposed solutions to address these performance problems raise additional questions about whether this equipment will provide any meaningful increase in the ability to detect certain nuclear materials. Specifically, to address the problem of false positive alarms indicating the presence of certain nuclear materials, according to DNDO officials, DNDO has modified the ASP to make this equipment less sensitive to these nuclear materials. While this may address the issue of false positive alarms, it will also diminish the ASP capability of detecting a key high-risk nuclear material. Since the ASP modification, DNDO conducted computer simulations using a vendor-provided system called a "replay tool" to examine the effect of the modification on the ASP's performance. According to DNDO officials, the replay tool demonstrated that the modified ASP will still be able to detect certain nuclear materials better than the PVT. However, at this point, DNDO does not plan to retest the ASP at the Nevada Test Site where it can examine the effects of these modifications using actual nuclear materials. As we reported earlier this year, the results of the testing at the Nevada Test Site demonstrated that the ASPs represented a marginal improvement in detecting certain nuclear materials. By reducing the sensitivity to these materials and not retesting the modified ASPs against actual nuclear materials, it is uncertain exactly what improvement in detecting certain nuclear materials these costly portal monitors are providing. While DNDO is reducing the sensitivity of ASPs to certain nuclear materials, it has yet to complete efforts to improve the PVT's ability to detect these same materials through energy windowing. For several years, CBP officials have repeatedly urged DNDO officials to complete this research. However, it was not apparent from our discussions with DNDO officials if this effort is making meaningful progress with the development of energy windowing or when it will be completed. Furthermore, CBP officials stated that, depending on the outcome of this research, energy windowing could be the more cost effective way to improve detection of certain nuclear materials. In our view, ASPs being modified to diminish their capabilities to detect certain nuclear materials raises questions about whether energy windowing might be able to achieve a similar level of performance against these same materials from the PVTs that are already in place. Beyond reducing the sensitivity of ASPs to certain nuclear materials, DNDO also plans to address the issue of critical failures by, among other things, installing an indicator light on the ASP that will alert CBP officers that the ASP has experienced a mission-critical failure and is no longer scanning cargo. While this should address the issue of CBP officers not knowing that the ASP has suffered a critical failure, CBP officials stressed to us the need for the ASP to be stable and secure enough to avoid these shutdowns. In closing, the issues raised by the results of the July 2009 field validation tests provide even greater reason for DNDO to address recommendations from our May 2009 report. In particular, we reiterate the importance of our prior recommendation for DNDO to assess whether ASPs meet the criteria for a significant increase in operational effectiveness based on a valid comparison with PVTs' full performance potential, given that the ASPs will no longer be as effective in detecting certain nuclear materials. Mr. Chairman, this completes my prepared statement. I would be happy to respond to any questions that you or other Members of the Subcommittee may have at this time. For further information about this testimony, please contact me at (202) 512-3841 or aloisee@gao.gov. Dr. Timothy Persons (Chief Scientist), Ned Woodward (Assistant Director), Joseph Cook, and Kevin Tarmann made key contributions to this testimony. Kendall Childers, Karen Keegan, Carol Kolarik, Jonathan Kucskar, Omari Norman, Alison O'Neill, and Rebecca Shea also made important contributions. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.

Summary:
The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is responsible for addressing the threat of nuclear smuggling. Radiation detection portal monitors are key elements in the nation's defenses against such threats. DHS has sponsored testing to develop new monitors, known as advanced spectroscopic portal (ASP) monitors, to replace radiation detection equipment being used at ports of entry. DNDO expects that ASPs may offer improvements over current-generation portal monitors, particularly the potential to identify as well as detect radioactive material and thereby to reduce both the risk of missed threats and the rate of innocent alarms, which DNDO considers to be key limitations of radiation detection equipment currently used by Customs and Border Protection (CBP) at U.S. ports of entry. However, ASPs cost significantly more than current generation portal monitors. Due to concerns about ASPs' cost and performance, Congress has required that the Secretary of Homeland Security certify that ASPs provide a significant increase in operational effectiveness before obligating funds for full-scale ASP procurement. In May 2009, GAO issued a report (GAO-09-655) on the status of the ongoing ASP testing round. This testimony (1) discusses the principal findings and recommendations from GAO's May report on ASP testing and (2) updates those findings based on information from DNDO and CBP officials on the results of testing conducted since the report's issuance. DHS, DNDO, and CBP's oral comments on GAO's new findings were included as appropriate. GAO's May 2009 report on ASP testing found that DHS increased the rigor in comparison with previous tests and thereby added credibility to the test results. However, GAO's report also questioned whether the benefits of the ASPs justify its high cost. In particular, the DHS criteria for a significant increase in operational effectiveness require only a marginal improvement in the detection of certain weapons-usable nuclear materials, which DNDO considers a key limitation of current-generation portal monitors. The marginal improvement required of ASPs is particularly notable given that DNDO has not completed efforts to fine-tune current-generation equipment to provide greater sensitivity. Moreover, the test results showed that ASPs performed better than current-generation portal monitors in detection of such materials concealed by light shielding approximating the threat guidance for setting detection thresholds, but that differences in sensitivity were less notable when shielding was slightly below or above that level. Finally, DNDO had not yet updated its cost-benefit analysis to take into account the results of ASP testing and did not plan to complete computer simulations that could provide additional insight into ASP capabilities and limitations prior to certification even though test delays have allowed more time to conduct the simulations. DNDO officials believed the other tests were sufficient for ASPs to demonstrate a significant increase in operational effectiveness. GAO recommended that DHS assess ASPs against the full potential of current-generation equipment and revise the program schedule to allow time to conduct computer simulations and to uncover and resolve problems with ASPs before full-scale deployment. DHS agreed to a phased deployment that should allow time to uncover ASP problems but disagreed with the other recommendations, which GAO believes remain valid. The results of DNDO's most recent round of field testing raise continuing issues. In July 2009, DNDO resumed the field testing of ASPs that it initiated in January 2009 but suspended because of serious performance problems. However, the July tests also revealed critical performance deficiencies. For example, the ASP had a high number of false positive alarms for the detection of certain nuclear materials. According to CBP, these false alarms are very disruptive in a port environment because any alarm for this type of nuclear material causes CBP to take enhanced security precautions. To address these false alarms, DNDO plans to modify the ASP to make these monitors less sensitive to these nuclear materials and thereby diminishing the ASPs' capability. As GAO reported earlier this year, previous testing results demonstrated that the ASPs represented a marginal improvement in detecting these materials. By reducing the sensitivity to nuclear materials even further, it is uncertain exactly what improvement in detecting these materials the ASPs are providing or whether DNDO might be able to achieve a similar level of performance as the modified ASPs by improving the current-generation portal monitors that are already in place. In addition, the July 2009 testing also identified a critical equipment failure, including an alert malfunction, which DNDO is taking steps to resolve for future testing.