The Department of Homeland Security Appropriations Act for Fiscal Year 2007 states that “none of the funds appropriated…shall be obligated for full scale procurement of monitors until the Secretary of Homeland Security has certified…that a significant increase in operational effectiveness will be achieved.” DNDO noted that certification would meet DHS guidelines for the review and approval of complex acquisitions. Specifically, DNDO stated that the Secretary’s decision would be made in the context of DHS “Key Decision Point 3,” which details the review and approval necessary for DHS acquisition programs to move from the “Capability Development and Demonstration” phase to the “Production and Deployment Phase.” To meet the statutory requirement to certify the ASPs will provide a “significant increase in operational effectiveness,” and requirements outlined in DHS Management Directive 1400, DNDO, with input from subject matter experts, developed a series of tests intended to demonstrate, among other things, ASP performance and deployment readiness. The tests were conducted at several venues, including the Nevada Test Site, the New York Container Terminal, the Pacific Northwest National Laboratory, and five ports of entry. DNDO stated that its request for full-scale production approval would be based upon completed and documented results of these tests. To meet the Secretary’s goal of deploying 225 ASPs by the end of calendar year 2008, Secretarial Certification was scheduled for June 26, 2007. To guide the test operations, DNDO defined a set of Critical Operational Issues that outlined the tests’ technical objectives and provided the baseline to measure demonstrated effectiveness. The purpose of the Critical Operational Issue 1 is to “verify operational effectiveness” of ASPs and determine whether “ASP systems significantly increase operational effectiveness relative to the current generation detection and identification system.” DNDO conducted a series of tests at the Nevada Test Site, the single focus of which, according to DNDO, was to resolve Critical Operational Issue 1. According to DNDO, these tests began in February 2007 and concluded in March 2007. DNDO’s Nevada Test Site test plan, dated January 12, 2007, identified three primary test objectives comparing the operational effectiveness of the ASP systems with existing detection and identification systems at current high-volume operational thresholds. Specifically, DNDO sought to determine the ASPs’ probability to (1) detect and identify nuclear and radiological threats (2) discriminate threat and non-threat radionuclides in primary , and (3) detect and identify threat radionuclides in the presence of non-threat radionuclides. The Nevada Test Site test plan had two key components. First, DNDO developed guidelines for basic test operations and procedures, including test goals and expectations, test tasks and requirements, and roles and responsibilities of personnel involved in the testing, including the ASP contractors. The second component involved the National Institute of Standards and Technology developing test protocols that defined, among other things, how many times a container carrying test materials would need to be driven through portal monitors in order to obtain statistically relevant results. DNDO’s tests at the Nevada Test Site were designed to compare the current system—using PVTs in primary inspections and a PVT and RIID combination in secondary inspections—to other configurations including PVTs in primary and ASPs in secondary, and ASPs in both primary and secondary inspection positions. DNDO tested three ASPs and four PVTs. The ASP vendors included Thermo, Raytheon, and Canberra. The PVT vendors included SAIC, TSA, and Ludlum. According to the test plan, to the greatest extent possible, PVT, ASP, and RIID handheld devices would be operated consistent with approved CBP standard operating procedures. Prior to “formal” collection of the data that would be used to support the resolution of Critical Operational Issue 1, DNDO conducted a series of tests it referred to as “dry runs” and “dress rehearsals.” The purpose of the dry runs was to, among other things, verify ASP systems’ software performance against representative test materials and allow test teams and system contractors to identify and implement software and hardware improvements to ASP systems. The purpose of the dress rehearsals was to observe the ASPs in operation against representative test scenarios and allow the test team to, among other things: develop confidence in the reliability of the ASP system so that operators and data analysts would know what to expect and what data to collect during the formal test, collect sample test data, and determine what errors were likely to occur in the data collection process and eliminate opportunities for error. In addition to improving ASP performance through dry runs and dress rehearsals conducted prior to formal data collection, ASP contractors were also significantly involved in the Nevada Test Site test processes. Specifically, the test plan stated that “ contractor involvement was an integral part of the NTS test events to ensure the systems performed as designed for the duration of the test.” Furthermore, ASP contractors were available on site to repair their system at the request of the test director and to provide quality control support of the test data through real time monitoring of available data. DNDO stated that Pacific Northwest National Laboratory representatives were also on site to provide the same services for the PVT systems. DNDO conducted its formal tests in two phases. The first, called Phase 1, was designed to support resolution of Critical Operational Issue 1 with high statistical confidence. DNDO told us on multiple occasions and in a written response that only data collected during Phase 1 would be included in the final report presented to the Secretary to request ASP certification. According to DNDO, the second, called Phase 3, provided data for algorithm development which targeted specific and known areas in need of work and data to aid in the development of secondary screening operations and procedures. According to DNDO documentation, Phase 3 testing was not in support of the full-scale production decision. Further, DNDO stated that Phase 3 testing consisted of relatively small sample sizes since the data would not support estimating the probability of detection with a high confidence level. On May 30, 2007, following the formal tests and the scoring of their results, DNDO told GAO that it had conducted additional tests that DNDO termed “Special Testing.” The details of these tests were not outlined in the Nevada Test Site test plan. On June 20, 2007, DNDO provided GAO with a test plan document entitled “ASP Special Testing” which described the test sources used to conduct the tests but did not say when the tests took place. According to DNDO, special testing was conducted throughout the formal Phase 1 testing process and included 12 combinations of threat, masking, and shielding materials that differed from “dry run,” “dress rehearsal,” and formal tests. DNDO also stated that the tests were “blind,” meaning that neither DNDO testing officials nor the ASP vendors knew what sources would be included in the tests. According to DNDO, these special tests were recommended by subject matter experts outside the ASP program to address the limitations of the original NTS test plan, including available time and funding resources, special nuclear material sources, and the number of test configurations that could be incorporated in the test plan, including source isotope and activity, shielding materials and thicknesses, masking materials, vehicle types, and measurement conditions. Unlike the formal tests, National Institute of Standards and Technology officials were not involved in determining the number of test runs necessary to obtain statistically relevant results for the special tests. Based on our analysis of DNDO’s test plan, the test results, and discussions with experts from four national laboratories, we are concerned that DNDO used biased test methods that enhanced the performance of the ASPs. In the dry runs and dress rehearsals, DNDO conducted many preliminary runs of radiological, nuclear, masking, and shielding materials so that ASP contractors could collect data on the radiation being emitted, and modify their software accordingly. Specifically, we are concerned because almost all of the materials, and most combinations of materials, DNDO used in the formal tests were identical to those that the ASP contractors had specifically set their ASPs to identify during the dry runs and dress rehearsals. It is highly unlikely that such favorable circumstances would present themselves under real world conditions. A key component of the NTS tests was to test the ASPs’ ability to detect and identify dangerous materials, specifically when that material was masked or “hidden” by benign radioactive materials. Based on our analysis, the masking materials DNDO used at NTS did not sufficiently test the performance limits of the ASPs. DOE national laboratory officials raised similar concerns to DNDO after reviewing a draft of the test plan in November 2006. These officials stated that the masking materials DNDO planned to use in its tests did not emit enough radiation to mask the presence of nuclear materials in a shipping container and noted that many of the materials that DOE program officials regularly observe passing through international ports emit significantly higher levels of radiation than the masking materials DNDO used for its tests. DNDO officials told us that the masking materials used at the Nevada Test Site represented the average emissions seen in the stream of commerce at the New York Container Terminal. However, according to data accumulated as part of DOE’s program to secure international ports (the Megaports program), a significant percentage of cargo passing through one European port potentially on its way to the United States has emission levels greater than the average radiation level for cargo that typically sets off radiation detection alarms. Importantly, DNDO officials told us that the masking materials used at the Nevada Test Site were not intended to provide insight into the limits of ASP detection capabilities. Yet, DNDO’s own test plan for “ASP Special Testing” states, “The DNDO ASP NTS Test Plan was designed to… measure capabilities and limitations in current ASP systems.” In addition, the NTS tests did not objectively test the ASPs against the currently deployed radiation detection system.