In the summer of 2005, DNDO tested ASPs from 10 vendors to evaluate their performance capabilities and to select the ASPs that warranted further development and possible procurement. In July 2006, DNDO awarded contracts totaling $1.2 billion over fiveyears to three vendors— Raytheon, Canberra, and Thermo. The Department of Homeland Security Appropriations Act for Fiscal Year 2007 states that “none of the funds appropriated … shall be obligated for full scale procurement of monitors until the Secretary of Homeland Security has certified … that a significant increase in operational effectiveness will be achieved.” Congress enacted a similar requirement in DHS’s fiscal year 2008 appropriation. In hopes of obtaining secretarial certification by June 2007, DNDO tested ASPs at several sites, including the Nevada Test Site, the New York Container Terminal, the Pacific Northwest National Laboratory, and five ports of entry. DNDO conducted the tests at NTS in two phases. DNDO stated that the Phase 1 tests, performed in February-March 2007, attempted to estimate the performance capabilities of the ASPs with a high degree of statistical confidence. DNDO intended these tests to support the Secretary’s decision on whether to certify the ASPs for the purposes of a full-scale production decision, while the Phase 3 tests were intended to help improve the computer algorithms that the ASPs use to identify the specific radiological or nuclear source inside a container. On September 18, 2007, we testified that DNDO’s Phase 1 tests did not constitute an objective and rigorous assessment of the ASPs’ capabilities because, among other things, DNDO conducted preliminary test runs on source materials to be used in the tests, and then allowed the vendors to adjust their ASPs to specifically identify the source materials to be tested. We testified that in our view, DNDO’s approach biased the tests in ways that enhanced the apparent performance of the ASPs. We also noted that the tests did not attempt to estimate the limits of ASPs’ detection abilities—an important concern to those who will use them such as CBP officers. During that hearing, DNDO’s Director stated that, contrary to statements DNDO made in its final Phase 3 test plan, DNDO would use the Phase 3 test results to help support the Secretary’s decision on whether to certify the ASPs for full-scale production. Subsequently, DNDO delayed its anticipated date for secretarial certification to the fall of 2008 in order to conduct additional performance tests and field tests during fiscal year 2008. Because the limitations of the Phase 3 test results are not properly discussed in the Phase 3 test report, the report does not accurately portray the results from the Phase 3 tests and could be misleading. The purpose of the Phase 3 tests was to identify areas in which the ASPs needed improvement. While some of the Phase 3 report addresses this purpose, much of the report compares the performance of the ASPs with each other or with PVTs and RIIDs during the tests. However, because DNDO performed each test a limited number of times, the data it used to make some of these comparisons provide little information about the actual capabilities of the ASPs. The narrative of the report often presents each test result as a single value, although, because of the limited number of test runs, the results would be more thoroughly and appropriately stated as a range of potential values. In addition, the report’s narrative sometimes omits key facts that conflict with DNDO’s analysis of the results. “The primary goals of the testing were to provide information by giving the ASP systems an opportunity to perform against a wider range of radionuclides, shielding, and cargo configurations. To allow for more substantial variation among test cases, the test plan called for a smaller number of trials over a larger range of objects and configurations rather than running sufficient number of repetitions for each individual test case to provide higher statistically significan results.” (p. 2) In these comparisons , results are small sample sizes induce large uncertainties in the estimates of the probabilities being compared [for example: n ≤ 5].” (p.9) “For at 2 mph, the ASP system performances are statistically indistinguishable.” (p.13) “For shielded , performance for all three systems is statistically indistinguishable with probabilities of correct alarm varying approximately between 0.84 and .0.92.” (p.11) The statements imply that the performances of the ASPs were similar because the results were “statistically indistinguishable.” However, given the small number of test runs, it is impossible to determine with a high degree of confidence whether or not the performances were actually similar. Yet the report’s text describing specific results rarely qualifies the results by stating that the test was run only a few times or that the results should not be considered conclusive of the ASPs’ capabilities. “For the source configurations tested, the ASP systems have equal performance down to the lowest source activity tested.” (p. iii) “The PVT systems display lower performance than the ASP systems for and sources.” (p. iv) “When comparing the ASP systems _ mph identification metric with the _ RIID measurements…, it is observed that the RIID performance is poor compared to the ASP systems.” (p. iv) “For bare only, … he probability of correct identification varied between 0.34 and 0.5.” (p.14) However, because each test involved a small number of test runs, these percentages provide little information about the performance capabilities of the ASPs. In fact, because of the small number of test runs, DNDO can only estimate that each ASP can correctly identify the type of source material within a range of values. The fewer the number of test runs, the larger the range. For example, for the ASP that correctly identified the source material 34 percent of the time during the tests, the report text omits the fact that, as shown on an accompanying graph, DNDO can only estimate that the ASP would be able to correctly identify the source between about 10 percent and 65 percent of the time. By stating that the ASP identified the source 34 percent of the time without clarifying that the results came from only a few test runs, the report’s text makes the test results seem more precise than they really are. Similarly, for the ASP that correctly identified the source material 50 percent of the time during the tests, the small number of test runs means that DNDO can only estimate that the ASP would be able to correctly identify the source material between about 15 percent and 85 percent of the time. This range is too wide to have much value in determining how well the ASP may perform in the real world. Although these ranges are clearly shown on the report’s graphs, they are omitted in the report’s descriptions and interpretations of the test results. Similarly, DNDO’s analysis comparing the performances of ASPs and RIIDs fails to consider the uncertainties created by the tests’ small sample sizes. The report states that the RIIDs “performance is poor compared to the ASP systems.” For example, during the tests, one vendor’s ASP correctly identified one type of source material about 50 percent of the time, while the RIIDs correctly identified the same type of source material about 10 percent of the time. However, given the small number of test runs, DNDO cannot be confident that these percentages precisely indicate the performance capabilities of the ASPs and RIIDs. On the basis of the tests, DNDO can only infer that the ASPs’ and RIIDs’ performance capabilities lie somewhere within a relatively large range of values. As these ranges are illustrated in the report’s graphs, it appears that the difference in the performance of the ASPs and RIIDs may not be statistically different for three of the five types of source materials DNDO tested. This does not necessarily mean that the ASPs and RIIDs performed equally well; rather, DNDO did not conduct each test enough times to determine that the superior performance of the ASPs over the RIIDs reflects the capabilities of the ASPs rather than mere chance. “The ASP systems demonstrated detection limits equal to or better than those of any of the PVT systems as configured during testing.” (p.iii) However, the report’s executive summary fails to note that because DNDO used only one type of source material, the results are largely specific to that particular source material and would not necessarily apply to other specific source materials. In fact, for other types of source material, the report shows several instances in which the PVTs were apparently able to detect other types of source materials better than the ASPs. Moreover, other Phase 3 tests showed that simply moving the source material referred to in the above quote to another place in the container affected the relative performances of the ASPs and PVTs. “the ASP systems have the ability to identify sources when placed inside almost all but the thickest shielding configuration tested.” (p.iv) Again, however, DNDO fails to note in its report that, as it explained in its Phase 3 test plan, all the shielding used in the Phase 3 tests represented “light shielding.” The report also fails to state how many specific sources the ASPs could correctly identify or how frequently the ASPs could identify them. In our view, it is not appropriate to use the Phase 3 test report in determining whether the ASPs represent a significant improvement over currently deployed radiation equipment because the limited number of test runs does not support many of the comparisons of ASP performance made in the Phase 3 report. As noted, DNDO’s use of a small number of runs for each test means that DNDO can only be certain that the ASP can correctly identify or detect a source material over a broad range of possible values rather than at a specific rate. This is true even if the ASP was successful every time a test was conducted.