Although the DHS criteria for primary screening require an improved ability to detect certain nuclear materials at operational thresholds, ASPs could meet the criteria for improvement while still failing to detect anything more than lightly shielded material. DNDO officials acknowledge that passive radiation detection equipment, which includes both the new and current-generation portal monitors, is capable of detecting certain nuclear materials only when this material is unshielded or lightly shielded. For this reason, the DOE threat guidance used to set PVTs’ detection threshold is based on the equipment’s limited sensitivity to anything more than lightly shielded nuclear material rather than on the assumption that smugglers would take effective shielding measures. DOE developed the guidance in 2002 and 2003 when CBP began deploying PVTs for primary screening. DOE and national laboratory officials responsible for the guidance told us the assumption of light shielding was based not on an analysis of the capabilities of potential smugglers to take effective shielding measures but rather on the limited sensitivity of PVTs to detect anything more than certain lightly shielded nuclear materials. In contrast, PVTs are more sensitive to the relatively strong radiation signature of other nuclear materials, and the threat guidance assumes a higher level of shielding for setting the operational threshold for detection of such materials. However, even for such materials, the DOE threat guidance assumes that shielding would not exceed a level provided by the contents of an average cargo container. Moreover, DNDO has not completed efforts to fine-tune PVTs’ software and thereby improve sensitivity to nuclear materials. As a result, the criteria compare ASPs to the current performance of PVTs and do not take potential improvements into account, which affects any assessment of “significant” improvement over current technology. DNDO officials expect they can achieve small improvements to PVTs’ performance through additional development of “energy windowing,” a technique currently being used in PVTs to provide greater sensitivity than otherwise possible. Pacific Northwest National Laboratory officials responsible for developing the technique also told us small improvements may be possible, and CBP officials have repeatedly urged DNDO to investigate the potential of the technique. DNDO collected the data needed to further develop energy windowing during the 2008 performance testing at the Nevada Test Site but has not yet funded Pacific Northwest National Laboratory efforts to analyze the data and further develop the technique. Other aspects of the criteria for a significant increase in operational effectiveness require that ASPs either provide more than a marginal improvement in addressing other limitations of current-generation equipment or at least maintain the same level of performance in areas in which the current-generation equipment is considered adequate: The primary screening requirement for an 80 percent reduction in the rate of innocent alarms could result in hundreds of fewer secondary screenings per day, thereby reducing CBP’s workload and delays to commerce. The actual reduction in the volume of innocent alarms would vary and would be greatest at the nation’s busiest ports of entry, such as Los Angeles/Long Beach, where CBP officials report that PVTs generate up to about 600 innocent alarms per day. A DNDO official said the requirement for an 80 percent reduction in innocent alarms was developed in conjunction with CBP and was based on a level that would provide meaningful workload relief. The primary screening criteria requiring that ASPs provide at least the same level of sensitivity to plutonium and medical and industrial isotopes, but not specifying an improvement, were based on DNDO’s assessment that PVTs adequately detect such materials, which have a stronger radiation signature than HEU. In addition, CBP officials said that including medical and industrial isotopes in the criteria addressed a CBP requirement for verifying that those transporting certain quantities of these materials into the United States are properly licensed. The secondary screening requirement that ASPs reduce the probability of misidentifying special nuclear material by one-half addresses the inability of relatively small handheld devices to consistently locate and identify potential threats in large cargo containers. For example, a handheld device may fail to correctly identify special nuclear material if the material is well-shielded or the device is not placed close enough to a radiation source to obtain a recognizable measurement. According to CBP and DNDO, the requirement for a reduction in the average time to conduct secondary screenings is not more specific because the time varies significantly among ports of entry and types of cargo being screened. Improvements to the 2008 round of testing addressed concerns we raised about earlier rounds of ASP testing. However, the testing still had limitations, and the preliminary results are mixed. As we testified in September 2008, DHS’s improvements to the 2008 round of ASP testing addressed concerns we raised about previous tests. A particular area of improvement was in the performance testing at the Nevada Test Site, where DNDO compared the capability of ASP and current-generation equipment to detect and identify nuclear and radiological materials, including those that could be used in a nuclear weapon. The improvements addressed concerns we previously raised about the potential for bias and provided credibility to the results within the limited range of scenarios tested by DNDO. For example, we reported in 2007 that DNDO had allowed ASP contractors to adjust their systems after preliminary runs using the same radiological materials that would be used in the formal tests. In contrast, the plan for the 2008 performance test stipulated that there would be no system contractor involvement in test execution, and no ASP contractors were at the test location on the day we observed performance testing. Furthermore, DNDO officials told us, and we observed, that they did not conduct preliminary runs with threat objects used in the formal tests. In 2007, we reported that DNDO did not objectively test the handheld identification devices because it did not adhere to CBP’s standard operating procedure for using the devices to conduct a secondary inspection, which is fundamental to the equipment’s performance in the field. DNDO addressed this limitation in the 2008 round of performance testing: CBP officers operated the devices and adhered as closely to the standard operating procedure as test conditions allowed. While the test conditions did not allow CBP officers to obtain real-time technical support in interpreting the device’s measurements, as they would in the field to increase the probability of correctly identifying a radiation source, DNDO officials said they addressed this limitation. For example, they treated a decision by a CBP officer to indicate the need for technical support as a correct outcome if the test scenario involved the use of a potential threat, such as HEU. Other aspects of testing, while not specifically addressing concerns we previously raised, also added credibility to the test results. Based on our analysis of the performance test plan, we concluded that the test design was sufficient to identify statistically significant differences between the new technology and current-generation systems when there were relatively large differences in performance. Specifically, DNDO conducted a sufficient number of runs of each scenario used in the 2008 performance testing to identify such differences. With regard to the general conduct of the 2008 round of testing, two aspects, in particular, enhanced the overall rigor of the tests: (1) criteria for ensuring that ASPs met the requirements for each phase before advancing to the next, and (2) the participation of CBP and the DHS Science and Technology Directorate. The test and evaluation master plan established criteria requiring that the ASPs have no critical or severe issues rendering them completely unusable or impairing their function before starting or completing any test phase. In addition, the criteria established a cumulative limit of 10 issues requiring a work-around (e.g., a straightforward corrective step, such as a minor change in standard operating procedures) and 15 cosmetic issues not affecting proper functioning. DNDO and CBP adhered to the criteria even though doing so resulted in integration testing conducted at the Pacific Northwest National Laboratory taking longer than anticipated and delaying the start of field validation. For example, DNDO and CBP did not allow a vendor’s ASP system to complete integration testing until all critical or severe issues had been resolved. The involvement of CBP and the DHS Science and Technology Directorate provided an independent check, within DHS, of DNDO’s efforts to develop and test the new portal monitors. For example, the lead CBP official involved in ASP testing told us that DNDO provided an initial assessment of the severity of issues uncovered during testing, but CBP made the final decision on categorizing them as critical, severe, work-around, or cosmetic issues. CBP also added a final requirement to integration testing before proceeding to field validation to demonstrate ASPs’ ability to operate for 40 hours without additional problems. According to CBP officials, their efforts to resolve issues prior to field validation reflect the importance CBP places on ensuring that ASPs are sufficiently stable and technically mature to operate effectively in a working port of entry and thereby provide for a productive field validation. The DHS Science and Technology Directorate, which is responsible for developing and implementing the department’s test and evaluation policies and standards, will have the lead role in the final phase of ASP testing; the final phase, consisting of 21 days of continuous operation, is scheduled to begin at one seaport after the completion of field validation. The Science and Technology Directorate identified two critical questions to be addressed through operational testing: (1) Will the ASP system improve operational effectiveness (i.e., detection and identification of threats) relative to the current-generation system, and (2) is the ASP system suitable for use in the operational environment at land and sea ports of entry? The suitability of ASPs includes factors such as reliability, maintainability, and supportability. Because the operational testing conducted at one seaport is not sufficient to fully answer these questions—for example, because the testing will not allow threat objects to be inserted into cargo containers—the directorate plans to also conduct an independent analysis of the results from previous test phases, including performance testing. The 2008 testing still had limitations, which do not detract from the test results’ credibility but do require that results be appropriately qualified.