Our report on the latest round of ASP testing found that DHS increased the rigor of ASP testing in comparison with previous tests and that a particular area of improvement was in the performance testing at the Nevada Test Site, where DNDO compared the capability of ASP and current-generation equipment to detect and identify nuclear and radiological materials. For example, unlike in prior tests, the plan for the 2008 performance test stipulated that there would be no system contractor involvement in test execution. Such improvements addressed concerns we previously raised about the potential for bias and provided credibility to the results. Nevertheless, based on the following factors, we continue to question whether the benefits of the new portal monitors justify the high cost: The DHS criteria for a significant increase in operational effectiveness. Our chief concern with the criteria is that they require a marginal improvement over current-generation portal monitors in the detection of certain weapons-usable nuclear materials when ASPs are deployed for primary screening. DNDO considers detection of such materials to be a key limitation of current-generation portal monitors. We are particularly concerned about the marginal improvement required of ASPs because the detection threshold for the current-generation portal monitors does not specify a level of radiation shielding that smugglers could realistically use. DOE and national laboratory officials told us that DOE’s threat guidance used to set the current detection threshold is based not on an analysis of the capabilities of potential smugglers to take effective shielding measures but rather on the limited sensitivity of PVTs to detect anything more than certain lightly shielded nuclear materials. DNDO officials acknowledge that both the new and current-generation portal monitors are capable of detecting certain nuclear materials only when unshielded or lightly shielded. The marginal improvement in detection of such materials required of ASPs is particularly notable given that DNDO has not completed efforts to fine-tune PVTs’ software and thereby improve sensitivity to nuclear materials. DNDO officials expect they can achieve small improvements in sensitivity, but DNDO has not yet funded efforts to fine-tune PVTs’ software. In contrast to the marginal improvement required in detection of certain nuclear materials, the primary screening requirement to reduce the rate of innocent alarms could result in hundreds of fewer secondary screenings per day, thereby reducing CBP’s workload and delays to commerce. In addition, the secondary screening criteria, which require ASPs to reduce the probability of misidentifying special nuclear material by one-half, address the inability of relatively small handheld devices to consistently locate and identify potential threats in large cargo containers. Preliminary results of performance testing and field validation. The preliminary results presented to us by DNDO are mixed, particularly in the capability of ASPs used for primary screening to detect certain shielded nuclear materials. Preliminary results show that the new portal monitors detected certain nuclear materials better than PVTs when shielding approximated DOE threat guidance, which is based on light shielding. In contrast, differences in system performance were less notable when shielding was slightly increased or decreased: Both the PVTs and ASPs were frequently able to detect certain nuclear materials when shielding was below threat guidance, and both systems had difficulty detecting such materials when shielding was somewhat greater than threat guidance. With regard to secondary screening, ASPs performed better than handheld devices in identification of threats when masked by naturally occurring radioactive material. However, differences in the ability to identify certain shielded nuclear materials depended on the level of shielding, with increasing levels appearing to reduce any ASP advantages over the handheld identification devices. Other phases of testing uncovered multiple problems in meeting requirements for successfully integrating the new technology into operations at ports of entry. Of the two ASP vendors participating in the 2008 round of testing, one has fallen behind due to severe problems encountered during testing of ASPs’ readiness to be integrated into operations at ports of entry (“integration testing”); the problems may require that the vendor redo previous test phases to be considered for certification. The other vendor’s system completed integration testing, but CBP suspended field validation after 2 weeks because of serious performance problems resulting in an overall increase in the number of referrals for secondary screening compared with existing equipment. DNDO’s plans for computer simulations. DNDO does not plan to complete injection studies—computer simulations for testing the response of ASPs and PVTs to simulated threat objects concealed in cargo containers—prior to the Secretary of Homeland Security’s decision on certification even though delays to the ASP test schedule have allowed more time to conduct the studies. According to DNDO officials, injection studies address the inability of performance testing to replicate the wide variety of cargo coming into the United States and the inability to place special nuclear material and other threat objects in cargo during field validation. DNDO had earlier indicated that injection studies could provide information comparing the performance of the two systems as part of the certification process for both primary and secondary screening. However, DNDO subsequently decided that performance testing would provide sufficient information to support a decision on ASP certification. DNDO officials said they would instead use injection studies to support effective deployment of the new portal monitors. Lack of an updated cost-benefit analysis. DNDO has not yet updated its cost-benefit analysis to take into account the results of the latest round of ASP testing. An updated analysis that takes into account the results from the latest round of testing, including injection studies, might show that DNDO’s plan to replace existing equipment with ASPs is not justified, particularly given the marginal improvement in detection of certain nuclear materials required of ASPs and the potential to improve the current-generation portal monitors’ sensitivity to nuclear materials, most likely at a lower cost. DNDO officials said they are currently updating the ASP cost-benefit analysis and plan to complete it prior to a decision on certification by the Secretary of Homeland Security. Our report recommended that the Secretary of Homeland Security direct DNDO to (1) assess whether ASPs meet the criteria for a significant increase in operational effectiveness based on a valid comparison with PVTs’ full performance potential and (2) revise the schedule for ASP testing and certification to allow sufficient time for review and analysis of results from the final phases of testing and completion of all tests, including injection studies. We further recommended that, if ASPs are certified, the Secretary direct DNDO to develop an initial deployment plan that allows CBP to uncover and resolve any additional problems not identified through testing before proceeding to full-scale deployment. DHS agreed to a phased deployment that should allow time to uncover ASP problems but disagreed with GAO’s other recommendations, which we continue to believe remain valid. The challenges DNDO has faced in developing and testing ASPs illustrate the importance of following existing DHS policies as well as best practices for investments in complex homeland security acquisitions and for testing of new technologies. The DHS investment review process calls for executive decision making at key points in an investment’s life cycle and includes many acquisition best practices that, if applied consistently, could help increase the chances for successful outcomes. However, we reported in November 2008 that, for the period from fiscal year 2004 through the second quarter of fiscal year 2008, DHS had not effectively implemented or adhered to its investment review process due to a lack of senior management officials’ involvement as well as limited monitoring and resources. In particular, of DHS’s 48 major investments requiring milestone and annual reviews under the department’s investment review policy, 45 were not assessed in accordance with this policy. In addition, many major investments, including DNDO’s ASP program, had not met the department’s requirements for basic acquisition documents necessary to inform the investment review process. As a result, DHS had not consistently provided the oversight needed to identify and address cost, schedule, and performance problems in its major investments. Among other things, our November 2008 report recommended that the Secretary of Homeland Security direct component heads, such as the Director of DNDO, to ensure that the components have established processes to manage major investments consistent with departmental policies. DHS generally concurred with our recommendations, and we noted that DHS had begun several efforts to address shortcomings in the investment review process identified in our report, including issuing an interim directive requiring DHS components to align their internal policies and procedures by the end of the third quarter of fiscal year 2009. In January 2009, DHS issued a memorandum instructing component heads to create acquisition executives in their organizations to be responsible for management and oversight of component acquisition processes. If fully implemented, these steps should help ensure that DHS components have established processes to manage major investments. Based on our body of work on ASP testing, one of the primary lessons to be learned is to avoid the pitfalls in testing that stem from a rush to procure new technologies. GAO has previously reported on the negative consequences of pressures imposed by closely linking testing and development programs with decisions to procure and deploy new technologies, including the creation of incentives to postpone difficult tests and limit open communication about test results. We found that testing programs designed to validate a product’s performance against increasing standards for different stages in product development are a best practice for acquisition strategies for new technologies. In the case of ASPs, the push to replace existing equipment with the new portal monitors led to a testing program that until recently lacked the necessary rigor. Even for the most recent round of testing, DNDO’s schedule consistently underestimated the time required to conduct tests, resolve problems uncovered during testing, and complete key documents, including final test reports. In addition, DNDO’s original working schedule did not anticipate the time required to update its cost-benefit analysis to take into account the latest test results. The schedule anticipated completion of testing in mid-September 2008 and the DHS Secretary’s decision on ASP certification between September and November 2008. However, testing is still not completed, and DNDO took months longer than anticipated to complete the final report on performance testing. As previously mentioned, a number of aspects of the latest round of ASP testing increased the rigor in comparison with earlier rounds and, if properly implemented, could improve the rigor in DHS’s testing of other advanced technologies.