DHS began testing next-generation RPMs with a preliminary small-scale study performed in July 2004 by DOE’s Pacific Northwest National Laboratory (PNNL). (See fig. 3.) Specifically, PNNL performed small- scale, side-by-side comparisons of next-generation RPMs and the RPMs used by CBP. This study found that in some situations, the performance of the next-generation RPMs was better than that of CBP’s RPMs, and that in other situations, the performance of the two was equal. In a follow- up, small-scale study conducted for DHS in July 2005, PNNL found that the next-generation RPMs performed no better than CBP’s currently deployed RPMs but did produce fewer alarms for benign materials. In October 2005, DNDO conducted a number of tests on next-generation RPMs that DHS had commissioned from different manufacturers and used the test results to help evaluate proposals for the ASP program. In July 2006, DNDO awarded contracts for development and production of ASPs. Among other things, according to DNDO officials, the design specifications for ASP stated that it should operate on trucks traveling at 5 miles per hour during primary screening and 2 miles per hour during secondary screening. DNDO began testing these ASPs at DOE’s Nevada National Security Site in February 2007. As we have previously reported, we identified a number of weaknesses in the methods used for these tests that impaired the significance and validity of the test results. Concerned about the performance and cost of ASP, Congress required the Secretary of Homeland Security to certify that ASP would provide a “significant increase in operational effectiveness” before DNDO obligated funds for full-scale procurement of ASPs. In response, DNDO, CBP, and DHS jointly issued criteria in July 2008 for determining whether ASP provided a significant increase in operational effectiveness compared with existing equipment. The criteria generally compare ASP with the RPMs used by CBP under CBP’s standard operating procedure. Specifically, as shown in table 1, there were four criteria for primary screening and two criteria for secondary screening. In addition to these overarching criteria, each phase of testing also had criteria that needed to be met to complete that phase. The final ASP testing campaign began in July 2008 and was concluded in November 2010. DNDO started the campaign with performance testing at DOE’s Nevada National Security Site. The performance tests determined the probability of detection and identification of radiation sources, including special nuclear material that could be used to make a nuclear weapon. CBP then began field validation testing in 2009. Field validation tests are the first step in determining the suitability of fully operational equipment in the actual environment where the equipment is intended to be used. The ASP field validation tests were performed at land border crossings in Detroit, Michigan, and Laredo, Texas, and at seaports in Long Beach, California, and in New York City, New York. CBP made three attempts to complete this testing—in January and February 2009, July and August 2009, and October and November 2010. This testing, which we reviewed for this report, evaluated the compatibility of ASP with CBP’s interdiction system on the basis of 10 additional criteria CBP had specified for completion of this phase of testing. Among other things, these criteria included whether, under normal field operating conditions, ASP improved detection to the required degree compared with the currently deployed CBP equipment. To pass the field validation testing phase, ASP had to meet all 10 criteria. In addition, DNDO’s ASP program manager specified that ASP could not have any major unresolved issues that might render it ineffective or unsuitable for use by CBP. Because of unsatisfactory test results, ASP did not pass field validation testing, which led DHS to cancel the program. Specifically, in CBP’s first and second attempts to pass field validation testing in 2009, ASP did not meet 1 of the 4 criteria for demonstrating a significant increase in operational effectiveness during primary screening–that is, it did not refer at least 80 percent fewer trucks for secondary screening than CBP’s currently deployed RPMs. When CBP began the first field validation test in late January 2009, CBP found ASP produced a higher number of referrals to secondary screening than CBP’s currently deployed RPMs. CBP suspended the test about 2 weeks later in mid-February; CBP officials reported that the problem with referrals required significant corrective actions before testing could resume. DNDO, working with Johns Hopkins University’s Applied Physics Laboratory, determined that the problem could be addressed by revising ASP’s software to raise the thresholds for triggering alarms. CBP conducted a second field validation test of ASP in July and August 2009 using the revised ASP software. CBP and DNDO documents indicate that, with the revised software, ASP was able to reduce referrals to secondary screening by about 70 percent but not by the 80 percent required to demonstrate a significant increase in operational effectiveness. Significantly, according to these documents, the majority of the ASP referrals to secondary screening were for alarms that falsely indicated the presence of special nuclear material that could be used to make a nuclear weapon. CBP officials told us these types of false alarms were very disruptive in a port environment because they caused CBP officers to take enhanced security precautions—specifically, CBP officers had to conduct a thorough inspection to ensure no special nuclear material was present before permitting the cargo to enter the country. Moreover, repeated false alarms for such nuclear materials could cause CBP officers to be skeptical about future alarms. In the course of the second field validation test, ASP also experienced a “critical failure,” which caused it to shut down. Importantly, during this critical failure, ASP did not emit a signal to alert the CBP officer that it was no longer screening cargo. Had this failure not occurred in a controlled testing environment, in which CBP was using its RPMs in tandem with ASP, the CBP officer would have permitted cargo to enter the country unscreened for radiological material. Testing is performed so that issues like this can be discovered and fixed prior to full-scale production and deployment. Following the second field validation test, DNDO’s Acting Director stated that the ASP program would be scaled back to secondary screening only. According to the Acting Director, DNDO’s decision was based on ASP’s performance and cost. Specifically, the Acting Director stated that DHS decided to not pursue ASP for primary screening because ASP did not meet DHS’s criteria for a significant increase in operational effectiveness. However, the Acting Director stated that results to date had shown that ASP met secondary screening criteria by wide margins. The Acting Director also stated that the cost to replace current RPMs with ASP for secondary screening was relatively low—approximately $350 million—compared with the original total program cost of $2 billion to $3 billion to use ASP for primary and secondary screening. This is because the original program called for many more ASPs to be deployed in CBP primary screening than secondary screening. In November 2010, ASP could not pass the third and final field validation test for use in secondary screening because ASP did not meet 6 of the 10 criteria needed to complete the field validation testing phase, according to our review of CBP test documents. According to these documents, 2 of the 6 criteria that ASP did not meet involved issues that would significantly affect operations unless they could be mitigated or resolved. ASP did not meet these criteria in part because of its inability to turn on at the beginning of each day and to continue operating long enough to complete a full day of testing. In addition, when ASP was operating, it produced an excessive number of alarms that incorrectly indicated naturally occurring radioactive material in the cargo might be masking special nuclear material. In some cases, testing of the ASP had to be suspended because the high number of these alarms was backing up traffic for secondary screening, delaying the flow of commerce. According to the CBP documents, the remaining 4 of the 6 criteria that ASP did not meet involved issues that could affect or delay port operations, although not as significantly. For example, ASP did not meet one of these criteria because of difficulty in integrating ASP into a CBP information network. In addition to the 6 criteria that the ASP did not meet, DNDO test documents identified several major unresolved issues that would have needed to be resolved for successful completion of field validation testing. In explaining these issues, DNDO and CBP officials told us that, for example, at certain times of the day, direct sunlight impaired the ability of the ASP to operate properly. See table 2 for a summary of the results for the three field validation tests. In addition to the issues identified during testing, our review identified analytical weaknesses before and after the third field validation test. For example, DNDO was inconsistent in its analysis of how to properly set up the ASP for secondary screening. Specifically, ASPs have separate settings (modes) for primary and secondary screening, and before the third field validation test, DNDO performed an analysis that indicated ASP would perform best during secondary screening if the machines were set in primary screening mode. However, after the third field validation test, DNDO officials analyzed the test results and concluded that ASP would have performed better for secondary screening had the machines been set in secondary screening mode. Furthermore, the analytical basis for a key decision regarding the ASP program is unclear. Specifically, DNDO could not provide us with any supporting analysis on a principal factor that led to the decision to cancel the program. According to DNDO officials, for ASP to work in secondary screening, truck speeds would need to average 2 miles per hour, as indicated in the design specifications. On the basis of these specifications, the ASP Governance Board concluded in February 2011 that ASP was not operationally suitable for secondary screening and should not be deployed because, according to CBP officials, it was not possible to control truck speed to an average of 2 miles per hour. However, when we asked DNDO officials for the data or analysis supporting this conclusion, these officials could not provide such support. Moreover, during congressional testimony in July 2012, the Acting Director of DNDO said that it is possible to control truck speed to 2 miles per hour at state-operated truck weigh stations. Accordingly, it is unclear why such speeds cannot be achieved at CBP land border crossings and seaports but can be achieved at truck weigh stations. Conducting lessons learned reviews when programs are cancelled benefits organizations by identifying things that worked well and did not work well in order to improve future acquisitions programs, according to experts we consulted; however, DHS does not have processes in place to ensure such reviews are conducted and reports documenting the results of the review are disseminated. To determine the key benefits of conducting lessons learned reviews, we consulted seven experts in large- scale engineering and acquisitions programs who were identified at our request by the National Academies. (See app. I for a list of the experts we interviewed.) When we asked these experts for their views on lessons learned reviews and reports for cancelled acquisition programs, they generally agreed on the following observations: Lessons learned reviews help to determine the reasons why programs were cancelled including problems with the technology, management, or setting requirements. By identifying problematic practices and causes for program failure, lessons learned reports allow an organization to improve future efforts. Lessons learned reports should be prepared promptly because otherwise, knowledgeable personnel may not be available to contribute to the reports, important details may not be recalled accurately, and dissemination of lessons learned will be delayed. Lessons learned reports should not be optional for an organization. A requirement for lessons learned reports should be institutionalized— lessons learned reports should be required for programs administered by the organization rather than being an ad hoc requirement. However, an appropriate acquisitions executive should have the authority to tailor or waive the lessons learned report based on factors such as the size or cost of a program, its perceived importance and effects, or the perceived causes for failure. Lessons learned reports must be submitted to an appropriate acquisitions executive to be useful. The acquisitions executive should disseminate the lessons learned reports to other appropriate acquisition officials. Lessons learned reports should be disseminated widely; however, they should be edited to preserve the lessons learned while minimizing potential damage to the reputations of people and organizations that were involved with the program. DHS interim acquisition guidance, issued in 2010, also recognizes the benefits of timely lessons learned reviews for cancelled programs, although as guidance it does not constitute a requirement. Similar to the views of the experts identified by the National Academies, the DHS acquisition guidance states that (1) the objective of the reviews is to share lessons learned throughout the department to increase the probability of success for future acquisition programs and (2) the reviews are to take place immediately after programs are cancelled. In addition, an official from the DHS’s Office of Program Accountability and Risk Management (PARM)—DHS’s policy office for acquisitions—told us that it is important to conduct these reviews immediately so that key people remain available and do not forget the factors that contributed to programs being cancelled. Under DHS acquisition guidance, upon completion of the lessons learned review, a report documenting the lessons learned is to be submitted to PARM. DHS acquisition guidance on lessons learned provides general direction on what is to happen when programs are cancelled, but it does not specify processes to follow. PARM officials stated that there were no documented processes in place for (1) conducting timely lessons learned reviews, (2) preparing lessons learned reports, or (3) disseminating lessons learned reports throughout the department. Under the federal standards of internal control, agencies are to clearly document internal controls (i.e., in management directives, administrative policies, or operating manuals), and this documentation should be readily available for examination. PARM officials told us they generally rely on DHS component agencies, such as DNDO, to self-monitor many aspects of their programs including conducting lessons learned reviews, preparing lessons learned reports, and submitting lessons learned reports. In the case of the ASP program, a lessons learned review was conducted, and a lessons learned report was submitted to PARM and disseminated within CBP and DNDO in November 2012 at the explicit direction of DHS’s Under Secretary for Management. The lessons learned report states that the program was cancelled by the Secretary in October 2011. However, before the Under Secretary’s directive, there was confusion about whether such a review should be conducted for cancelled programs, as well as uncertainty about whether the program had been cancelled. DNDO officials told us in March 2012 that, even though they considered the program to have been cancelled by the Secretary in October 2011, they did not intend to conduct a lessons learned review because such reviews were not needed for cancelled programs. Furthermore, there was uncertainty between DNDO and PARM about whether and when the program had been cancelled. In this regard, while DNDO relied on the Secretary’s October 2011 letter, PARM officials told us that the program was not cancelled and would not be cancelled until DNDO and CBP completed certain additional actions such as gathering data to support a future acquisition program. These actions were started but not completed, however, the uncertainty over the program’s status was eventually resolved in July 2012 with a memorandum from the Under Secretary for Management which stated that, in accordance with the Secretary’s October 2011 letter, the program was considered cancelled. In discussing this with DNDO, and DHS officials, they acknowledged that the DHS acquisition guidance is confusing on when programs are considered cancelled and when to begin conducting a lessons learned review. DHS officials stated that they are in the process of revising the guidance to provide greater clarity on these issues. To determine whether the confusion and uncertainty that affected the ASP program were unique to that program, we asked PARM officials to provide us with examples of previous lessons learned reports from any cancelled programs across DHS, including several programs cancelled in 2011 that we had previously reported upon. The PARM officials told us that they did not have any lessons learned reports for these programs or any others in the department. Furthermore, there was confusion within the department about the need to disseminate lessons learned reports. Before the Under Secretary’s memorandum of July 2012 directing DNDO and CBP to prepare and disseminate an ASP lessons learned report, PARM officials had told us there was no requirement to disseminate lessons learned outside individual component agencies. Subsequent to the memorandum, PARM officials told us that they are developing plans to disseminate lessons learned reports through department-wide Centers of Excellence. The lessons that can be learned from such reviews and reports can be extensive and informative. In the case of the ASP program, DNDO and CBP’s lessons learned report identified 32 lessons learned in the following six categories: documentation of planning; structuring and staffing of the program office; coordination between stakeholders and end users regarding technical and operational requirements, as well as costs. (See app. II for a copy of the lessons learned report.) For example, concerning acquisition strategy, the lessons learned report states the government should consider a “commercial first” approach— meaning consider whether commercially available equipment can meet programmatic needs before developing new technologies. Such an approach can reduce costs and risks to the program. Another lesson learned from the ASP program is that the acquisition officials and the end users must work closely together to ensure the needed capability meets operational requirements. An additional lesson learned was to manage the schedule of the acquisition program by events—not the calendar. In other words, the program should not proceed to the next major phase until all the criteria are met for completing the current phase. Furthermore, the report stated that acquisition programs should be flexible enough to respond to major issues that may be discovered after program initiation. Specifically, management should take care not to believe in the program so strongly that early signs of trouble are ignored, or cannot be detected and acted upon in a timely manner. At this point, however, PARM has no assurance that that lessons learned reviews have been or will be conducted for other cancelled programs, as called for in DHS acquisition guidance. Until DHS puts processes in place to ensure component agencies conduct lessons learned reviews and prepare lessons learned reports, and PARM implements its plans for disseminating the reports, DHS risks limiting the probability of success for the department’s future acquisition programs. Success in such acquisition programs is important given the scale of DHS’s planned investments, which DHS has estimated to be many billions of dollars during the next few years. DHS is participating in international tests of next-generation RPMs, including commercially available RPMs that have characteristics similar to ASP, and is working with states to gather data using ASPs at five truck weigh stations. For the international tests, DNDO is working with its European Union (EU) counterparts in the Illicit Trafficking Radiation Assessment Program (ITRAP); this program is currently known as ITRAP+10 because it is being revisited about 10 years after testing was originally conducted. According to the test planning documents, the final report is scheduled for issuance in August 2013. The documents further indicate that DNDO has the following objectives for the ITRAP+10 tests: Provide scientific and technical data on radiological and nuclear detection equipment to policy makers. Identify the best technologies based on results from repeatable testing. Promote harmonization of national and international standards and guidelines. Improve international exchange of information. Provide manufacturers with feedback on how well their equipment performed against standards. Promote new research and development efforts. The commercially available next-generation RPMs that DNDO and the EU are testing come from a variety of vendors and are being tested along with several other categories of radiation detection equipment. This testing is being performed to national and international standards for radiation detection set by the American National Standards Institute (ANSI) and the International Electrotechnical Commission (IEC). DNDO officials told us these tests are not part of any planned acquisition. According to the test plan, Oak Ridge National Laboratory (ORNL) will perform ITRAP+10 tests for DNDO on these next-generation RPMs. The primary planning documents for the testing of the next-generation RPMs were submitted by the National Institute of Standards and Technology and were approved by DNDO and the European Commission’s Joint Research Centre. These documents indicate that, in addition to testing to the standards, ORNL will test to determine the outer limits of what the next-generation commercial RPMs can detect. In addition to participating in the ITRAP+10 tests, DNDO is working with state agencies to gather data using five existing ASPs and installed at state-operated truck weigh stations in five states. According to a DNDO official, data collected will include information on the trucks and cargo passing through the ASPs, as well as any radiation detected by ASP. According to a DNDO official, data collected by the five ASPs will be sent to DNDO’s Joint Analysis Center Collaborative Information System, which informs DNDO about nuclear detection and coordinates responses to nuclear detection alarms. This official told us the data will be analyzed to determine the types of cargo transported and any trends that may be apparent. To its credit, DHS has cancelled the ASP program, and DNDO conducted a lessons learned review about the cancelled program and prepared a lessons learned report. However, DNDO prepared the lessons learned report and disseminated it 2 years after the final ASP testing, and about a year after the Secretary of Homeland Security notified Congress of her decision to cancel the program. Moreover, the report was not prepared until there was a specific directive from the DHS Under Secretary for Management. For cancelled programs, DHS’s acquisition guidance calls for a lessons learned review immediately after cancellation of a program, preparation of a lessons learned report, and dissemination of such a report. However, the acquisition guidance does not constitute an institutional requirement, which the experts we consulted said it should be. Moreover, DHS does not have documented processes for component agencies to follow in (1) conducting timely lessons learned reviews, (2) preparing lessons learned reports, or (3) disseminating lessons learned reports throughout the department, instead relying on such agencies to self-monitor. The amount of time between the final field validation testing of the ASP, the program’s cancellation, and the conducting of a lessons learned review and dissemination of a lessons learned report suggests that timely identification of lessons learned was not a DHS priority. While our review focused on the ASP program, the inability of PARM to identify any lessons learned reports from cancelled DHS programs suggests that the problem is broader than this one program. PARM is developing plans for disseminating future lessons learned reports department-wide. However, until DHS makes lessons learned reviews an institutional requirement, puts processes in place to ensure component agencies conduct such reviews and prepare lessons learned reports, and implements plans for disseminating the reports, DHS may be missing opportunities to improve its chances of success for billions of dollars in future acquisitions. To increase the probability of success for future acquisition programs, we recommend that the Secretary of Homeland Security take the following four actions for cancelled acquisition programs: Make lessons learned reviews an institutional requirement, such as through an agency directive or order or other appropriate means. Put documented processes in place to ensure that component agencies conduct timely lessons learned reviews, and prepare and submit lessons learned reports. Complete and implement plans to disseminate lessons learned reports throughout the department. We provided a draft of this report to DHS for review and comment. In its written response, reproduced in appendix III, DHS agreed with all four of our recommendations. DHS asked us to consider two of the recommendations resolved and closed on the basis of existing guidance and processes that were discussed in the body of this report. DHS outlined steps that it intends to take to address the other two recommendations. We are encouraged by some of the actions planned. However, we do not consider two of the recommendations resolved and closed because, as discussed below, we believe that DHS should take steps to address these recommendations. In DHS’s response to our recommendation to make lessons learned reviews an institutional requirement, DHS commented that it has institutionalized this requirement through existing departmental acquisitions guidance that was discussed in the body of this report. However, as discussed in this report and confirmed by PARM officials during the course of our review, this guidance does not constitute a requirement. DHS’s comments also refer to additional guidance called the Department of Homeland Security (DHS) Capital Planning and Investment Control (CPIC) Guide. However, officials from both PARM and the DHS Office of the Chief Information Officer told us that CPIC applies only to information technology programs, not to other programs such as ASP. In our view, the existing guidance referenced in DHS’s comments contributed to the confusion and uncertainty we discussed in the body of this report, including confusion about whether lessons learned were required for cancelled programs. Accordingly, we continue to believe that DHS needs to take action to address the problem. Similarly, in commenting on our recommendation to put processes in place to ensure that component agencies conduct timely lessons learned reviews, DHS stated that our recommendation is addressed by its existing acquisition oversight process, as evidenced by the completion of a lessons learned report for ASP. However, according to DNDO and CBP’s lessons learned report, the Secretary of DHS cancelled the program in October 2011, but the lessons learned report for this program was not issued until more than a year later. Furthermore, as noted in this report, the ASP lessons learned report is the only example PARM officials could identify of such a report having been completed. In our view, relying on this existing process seems unlikely to ensure that component agencies conduct timely lessons learned reviews. With regard to our final two recommendations—to put documented processes in place to ensure that component agencies prepare and submit lessons learned reports, and to complete and implement plans to disseminate lessons learned reports throughout the department—DHS stated that it is taking several actions. Specifically, DHS is revising and clarifying its acquisitions guidance with respect to lessons learned; it expects to complete these revisions by June 2014. In its comments, DHS stated that it has developed an online forum and library for DHS acquisition professionals that includes an area dedicated to lessons learned and best practices. Furthermore, DHS has designated a lead Center of Excellence and established a time frame of September 2013 for collecting and posting information on lessons learned for cancelled programs. In our view, as DHS completes more lessons learned reports, an online forum and library could become a useful means of disseminating these reports throughout the department. DHS also provided technical comments that we incorporated in the report, as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Homeland Security, the Director of DNDO, the Commissioner of CBP, the Executive Director of PARM, the appropriate congressional committees, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact David C. Trimble at (202) 512-3841 or trimbled@gao.gov or Dr. Timothy M. Persons at (202) 512-6412 or personst@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. At our request, the National Academies identified experts who could highlight leading practices for reviewing cancelled acquisition programs. These experts were identified on the basis on their knowledge of leading practices in large-scale engineering and acquisition programs. The National Academies provided us a list of 10 experts, 7 of whom were available to participate in this effort. Six of the 7 experts who were available were members of the National Academy of Engineering. We conducted a group telephone conference with 6 of the experts. We drafted conclusions from the conference and circulated them to all 7 of the experts for their review and comment. With one exception, the experts reached consensus on all of the observations. One expert felt that lessons learned reports were so important that acquisitions executives should have less authority to tailor or waive the requirements for such reports. In addition to the individuals named above, Ned Woodward, Assistant Director; Gene Aloise; Cheryl R. Arvidson; Antoinette C. Capaccio; Frederick K. Childers; R. Scott Fletcher; Cindy K. Gilbert; David C. Maurer; Cynthia Norris; Katrina E. Pekar-Carpenter; Kiki Theodoropoulos; and Nathan A. Tranquilli made key contributions to this report. Combating Nuclear Smuggling: DHS has Developed Plans for Its Global Nuclear Detection Architecture, but Challenges Remain in Deploying Equipment. GAO-12-941T. Washington, D.C.: July 26, 2012. Combating Nuclear Smuggling: DHS Has Made Some Progress but Not Yet Completed a Strategic Plan for Its Global Nuclear Detection Efforts or Closed Identified Gaps. GAO-10-883T. Washington, D.C.: June 30, 2010. Combating Nuclear Smuggling: Recent Testing Raises Issues About the Potential Effectiveness of Advanced Radiation Detection Portal Monitors. GAO-10-252T. Washington, D.C.: November 17, 2009. Combating Nuclear Smuggling: Lessons Learned from DHS Testing of Advanced Radiation Detection Portal Monitors. GAO-09-804T. Washington, D.C.: June 25, 2009. Combating Nuclear Smuggling: DHS Improved Testing of Advanced Radiation Detection Portal Monitors, but Preliminary Results Show Limits of the New Technology. GAO-09-655. Washington, D.C.: May 21, 2009. Nuclear Detection: Domestic Nuclear Detection Office Should Improve Planning to Better Address Gaps and Vulnerabilities. GAO-09-257. Washington, D.C.: January 29, 2009. Department of Homeland Security: Billions Invested in Major Programs Lack Appropriate Oversight. GAO-09-29. Washington, D.C.: November 18, 2008. Combating Nuclear Smuggling: DHS’s Phase 3 Test Report on Advanced Portal Monitors Does Not Fully Disclose the Limitations of the Test Results. GAO-08-979. Washington, D.C.: September 30, 2008. Combating Nuclear Smuggling: DHS Needs to Consider the Full Costs and Complete All Tests Prior to Making a Decision on Whether to Purchase Advanced Portal Monitors. GAO-08-1178T. Washington, D.C.: September 25, 2008. Combating Nuclear Smuggling: DHS’s Program to Procure and Deploy Advanced Radiation Detection Portal Monitors Is Likely to Exceed the Department’s Previous Cost Estimates. GAO-08-1108R. Washington, D.C.: September 22, 2008. Combating Nuclear Smuggling: DHS’s Decision to Procure and Deploy the Next Generation of Radiation Detection Equipment Is Not Supported by Its Cost-Benefit Analysis. GAO-07-581T. Washington, D.C.: March 14, 2007. Combating Nuclear Smuggling: DNDO Has Not Yet Collected Most of the National Laboratories’ Test Results on Radiation Portal Monitors in Support of DNDO’s Testing and Development Program. GAO-07-347R. Washington, D. C.: March 9, 2007. Combating Nuclear Smuggling: DHS’s Cost-Benefit Analysis to Support the Purchase of New Radiation Detection Portal Monitors Was Not Based on Available Performance Data and Did Not Fully Evaluate All the Monitors’ Costs and Benefits. GAO-07-133R. Washington, D.C.: October 17, 2006. Combating Nuclear Smuggling: Additional Actions Needed to Ensure Adequate Testing of Next Generation Radiation Detection Equipment. GAO-07-1247T. Washington, D.C.: September 18, 2007. Combating Nuclear Smuggling: DHS Has Made Progress Deploying Radiation Detection Equipment at U.S. Ports-of-Entry, but Concerns Remain. GAO-06-389. Washington, D.C.: March 22, 2006.

Summary: Preventing terrorists from smuggling radiological or nuclear material into the United States to carry out an attack is a national priority. DHS's DNDO develops and deploys radiation detection equipment to assist other federal agencies, such as CBP, in intercepting illicit radiological or nuclear materials that could be used to make a radiological dispersive device (dirty bomb) or a crude nuclear bomb. CBP uses RPMs at nearly all land border crossings and seaports to detect radiation in trucks and cargo. DHS recently cancelled acquisition of ASPs, which was originally envisioned as costing from $2 billion to $3 billion. GAO was asked to provide updated information on the ASP program. This report examines, among other things, (1) the results of ASP testing conducted in 2009 and 2010 that led to DHS's decision to cancel the ASP program and (2) the benefits of lessons learned reviews and how DHS captures any lessons learned when programs are cancelled. GAO reviewed testing and acquisition documents and interviewed key agency officials, as well as seven experts the National Academies identified for their knowledge of leading practices in large-scale engineering and acquisition programs. The advanced spectroscopic portal monitor (ASP)--a next-generation radiation portal monitor (RPM) for screening trucks and cargo containers--did not pass field validation tests conducted in 2009 and 2010. The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) intended to replace many currently deployed RPMs and handheld radiation detectors used by U.S. Customs and Border Protection (CBP) with ASPs. However, in the tests, ASP did not meet key requirements to detect radiation and identify its source. For example, ASP triggered too many false alarms from benign, naturally occurring radioactive material in common items such as kitty litter and granite, and it sometimes would not turn on or continue operating long enough to complete a day of testing. In addition, GAO's review identified analytical weaknesses related to the testing and program cancellation, including inconsistencies in DNDO's analysis of the settings used for testing the ASP. The final field validation test was conducted in November 2010, and the Secretary of Homeland Security notified Congress of her decision to cancel the program in October 2011. Conducting lessons learned reviews when programs are cancelled benefits organizations by identifying things that worked well and did not work well in order to improve future acquisitions programs, according to experts GAO consulted. However, DHS does not have processes in place to ensure such reviews are conducted or that the results are disseminated. Experts identified by the National Academies told GAO that lessons learned reviews help identify reasons why programs were cancelled. The experts also said lessons learned reviews should be required and conducted promptly, and the results should be disseminated. At the direction of DHS management, DNDO reviewed the ASP program and submitted and disseminated a lessons learned report in November 2012. This report cited 32 lessons learned including having program officials work closely with end users to ensure equipment meets operational requirements. DHS guidance calls for lessons learned reviews immediately after programs are cancelled and states that the lessons learned are to be shared throughout the department, but this guidance is not a requirement. Before DHS's directive, there was confusion about whether a lessons learned review was needed for the ASP program, and DNDO officials did not intend to conduct such a review. Moreover, DHS officials were unable to provide examples of previous lessons learned reports from other cancelled programs. DHS officials also said they have no process for disseminating such reports but are planning one. DHS should require lessons learned reviews and develop processes to ensure such reviews are done in a timely manner and the results disseminated throughout the department. DHS agreed with all of GAO's recommendations and has planned and taken some actions to address them.