DHS began testing next-generation RPMs with a preliminary small-scale study performed in July 2004 by DOE’s Pacific Northwest National Laboratory (PNNL). (See fig. 3.) Specifically, PNNL performed small- scale, side-by-side comparisons of next-generation RPMs and the RPMs used by CBP. This study found that in some situations, the performance of the next-generation RPMs was better than that of CBP’s RPMs, and that in other situations, the performance of the two was equal. In a follow- up, small-scale study conducted for DHS in July 2005, PNNL found that the next-generation RPMs performed no better than CBP’s currently deployed RPMs but did produce fewer alarms for benign materials. In October 2005, DNDO conducted a number of tests on next-generation RPMs that DHS had commissioned from different manufacturers and used the test results to help evaluate proposals for the ASP program. In July 2006, DNDO awarded contracts for development and production of ASPs. Among other things, according to DNDO officials, the design specifications for ASP stated that it should operate on trucks traveling at 5 miles per hour during primary screening and 2 miles per hour during secondary screening. DNDO began testing these ASPs at DOE’s Nevada National Security Site in February 2007. As we have previously reported, we identified a number of weaknesses in the methods used for these tests that impaired the significance and validity of the test results. Concerned about the performance and cost of ASP, Congress required the Secretary of Homeland Security to certify that ASP would provide a “significant increase in operational effectiveness” before DNDO obligated funds for full-scale procurement of ASPs. In response, DNDO, CBP, and DHS jointly issued criteria in July 2008 for determining whether ASP provided a significant increase in operational effectiveness compared with existing equipment. The criteria generally compare ASP with the RPMs used by CBP under CBP’s standard operating procedure. Specifically, as shown in table 1, there were four criteria for primary screening and two criteria for secondary screening. In addition to these overarching criteria, each phase of testing also had criteria that needed to be met to complete that phase. The final ASP testing campaign began in July 2008 and was concluded in November 2010. DNDO started the campaign with performance testing at DOE’s Nevada National Security Site. The performance tests determined the probability of detection and identification of radiation sources, including special nuclear material that could be used to make a nuclear weapon. CBP then began field validation testing in 2009. Field validation tests are the first step in determining the suitability of fully operational equipment in the actual environment where the equipment is intended to be used. The ASP field validation tests were performed at land border crossings in Detroit, Michigan, and Laredo, Texas, and at seaports in Long Beach, California, and in New York City, New York. CBP made three attempts to complete this testing—in January and February 2009, July and August 2009, and October and November 2010. This testing, which we reviewed for this report, evaluated the compatibility of ASP with CBP’s interdiction system on the basis of 10 additional criteria CBP had specified for completion of this phase of testing. Among other things, these criteria included whether, under normal field operating conditions, ASP improved detection to the required degree compared with the currently deployed CBP equipment. To pass the field validation testing phase, ASP had to meet all 10 criteria. In addition, DNDO’s ASP program manager specified that ASP could not have any major unresolved issues that might render it ineffective or unsuitable for use by CBP. Because of unsatisfactory test results, ASP did not pass field validation testing, which led DHS to cancel the program. Specifically, in CBP’s first and second attempts to pass field validation testing in 2009, ASP did not meet 1 of the 4 criteria for demonstrating a significant increase in operational effectiveness during primary screening–that is, it did not refer at least 80 percent fewer trucks for secondary screening than CBP’s currently deployed RPMs. When CBP began the first field validation test in late January 2009, CBP found ASP produced a higher number of referrals to secondary screening than CBP’s currently deployed RPMs. CBP suspended the test about 2 weeks later in mid-February; CBP officials reported that the problem with referrals required significant corrective actions before testing could resume. DNDO, working with Johns Hopkins University’s Applied Physics Laboratory, determined that the problem could be addressed by revising ASP’s software to raise the thresholds for triggering alarms. CBP conducted a second field validation test of ASP in July and August 2009 using the revised ASP software. CBP and DNDO documents indicate that, with the revised software, ASP was able to reduce referrals to secondary screening by about 70 percent but not by the 80 percent required to demonstrate a significant increase in operational effectiveness. Significantly, according to these documents, the majority of the ASP referrals to secondary screening were for alarms that falsely indicated the presence of special nuclear material that could be used to make a nuclear weapon. CBP officials told us these types of false alarms were very disruptive in a port environment because they caused CBP officers to take enhanced security precautions—specifically, CBP officers had to conduct a thorough inspection to ensure no special nuclear material was present before permitting the cargo to enter the country. Moreover, repeated false alarms for such nuclear materials could cause CBP officers to be skeptical about future alarms. In the course of the second field validation test, ASP also experienced a “critical failure,” which caused it to shut down. Importantly, during this critical failure, ASP did not emit a signal to alert the CBP officer that it was no longer screening cargo. Had this failure not occurred in a controlled testing environment, in which CBP was using its RPMs in tandem with ASP, the CBP officer would have permitted cargo to enter the country unscreened for radiological material. Testing is performed so that issues like this can be discovered and fixed prior to full-scale production and deployment. Following the second field validation test, DNDO’s Acting Director stated that the ASP program would be scaled back to secondary screening only. According to the Acting Director, DNDO’s decision was based on ASP’s performance and cost. Specifically, the Acting Director stated that DHS decided to not pursue ASP for primary screening because ASP did not meet DHS’s criteria for a significant increase in operational effectiveness. However, the Acting Director stated that results to date had shown that ASP met secondary screening criteria by wide margins. The Acting Director also stated that the cost to replace current RPMs with ASP for secondary screening was relatively low—approximately $350 million—compared with the original total program cost of $2 billion to $3 billion to use ASP for primary and secondary screening. This is because the original program called for many more ASPs to be deployed in CBP primary screening than secondary screening. In November 2010, ASP could not pass the third and final field validation test for use in secondary screening because ASP did not meet 6 of the 10 criteria needed to complete the field validation testing phase, according to our review of CBP test documents. According to these documents, 2 of the 6 criteria that ASP did not meet involved issues that would significantly affect operations unless they could be mitigated or resolved. ASP did not meet these criteria in part because of its inability to turn on at the beginning of each day and to continue operating long enough to complete a full day of testing. In addition, when ASP was operating, it produced an excessive number of alarms that incorrectly indicated naturally occurring radioactive material in the cargo might be masking special nuclear material. In some cases, testing of the ASP had to be suspended because the high number of these alarms was backing up traffic for secondary screening, delaying the flow of commerce. According to the CBP documents, the remaining 4 of the 6 criteria that ASP did not meet involved issues that could affect or delay port operations, although not as significantly. For example, ASP did not meet one of these criteria because of difficulty in integrating ASP into a CBP information network. In addition to the 6 criteria that the ASP did not meet, DNDO test documents identified several major unresolved issues that would have needed to be resolved for successful completion of field validation testing. In explaining these issues, DNDO and CBP officials told us that, for example, at certain times of the day, direct sunlight impaired the ability of the ASP to operate properly. See table 2 for a summary of the results for the three field validation tests. In addition to the issues identified during testing, our review identified analytical weaknesses before and after the third field validation test. For example, DNDO was inconsistent in its analysis of how to properly set up the ASP for secondary screening. Specifically, ASPs have separate settings (modes) for primary and secondary screening, and before the third field validation test, DNDO performed an analysis that indicated ASP would perform best during secondary screening if the machines were set in primary screening mode. However, after the third field validation test, DNDO officials analyzed the test results and concluded that ASP would have performed better for secondary screening had the machines been set in secondary screening mode. Furthermore, the analytical basis for a key decision regarding the ASP program is unclear. Specifically, DNDO could not provide us with any supporting analysis on a principal factor that led to the decision to cancel the program. According to DNDO officials, for ASP to work in secondary screening, truck speeds would need to average 2 miles per hour, as indicated in the design specifications.