NNSA’s Deputy Administrator for Defense Nuclear Nonproliferation directs the DNN office. The DNN Deputy Administrator reports to the NNSA Administrator and, through the NNSA Administrator, to the Secretary of Energy. Other NNSA staff in the DNN Deputy Administrator’s office include a Principal Assistant Deputy Administrator, a Chief Operating Officer, and a Senior Advisor. Each DNN program office is headed by an Assistant Deputy Administrator (ADA). In August 2012, DNN established a DNN Management Council to serve as its corporate management mechanism. This council, consisting of the NNSA Deputy Administrator for DNN, the DNN Principal Assistant Deputy Administrator, the DNN Chief Operating Officer, the DNN Senior Advisor, and the ADAs, has formulated various DNN program decisions—including on budget, staffing, and priorities—and has addressed other issues relating to the DNN programs on an ad hoc basis. In January 2015, NNSA reorganized the DNN programs into a new program office structure. Prior to the reorganization, DNN activities were arranged into five program offices: Office of Global Threat Reduction; Office of Defense Nuclear Nonproliferation R&D; Office of Nonproliferation and International Security; Office of International Material Protection and Cooperation; and Office of Fissile Materials Disposition. Under the January 2015 DNN reorganization, NNSA officials consolidated these five DNN program offices into four program offices: Office of Defense Nuclear Nonproliferation R&D; Office of Material Management and Minimization; Office of Global Material Security; and Office of Nonproliferation and Arms Control. Appendix III provides more information on the DNN reorganization. In March 2015, NNSA issued a new unclassified strategic plan for NNSA efforts to address future nuclear proliferation and terrorism threats for fiscal years 2016 to 2020 in response to an August 2014 interim report by the Secretary of Energy Advisory Board Task Force on Nuclear Nonproliferation. Among other things, that task force recommended that DOE produce a biannual report on nonweapons national security activities—especially nonproliferation programs—comparable to an annual report NNSA currently produces on the management of the nuclear weapons programs over the next 25 years, known as the Stockpile Stewardship and Management Plan. The DOE Secretary agreed with this recommendation and asked NNSA to begin drafting such a report. The March 2015 plan was presented as the first strategic plan to define and describe DOE and NNSA missions to prevent, counter, and respond to the threats of nuclear proliferation and terrorism, including the DNN programs. For each DNN program, the plan identifies program objectives, priorities, and performance measures; program activities, accomplishments, and challenges; and program plans over NNSA’s fiscal year 2016 to fiscal year 2020 budget horizon. In addition, the plan states that NNSA will issue a classified appendix containing more details on nuclear and radiological risks, threats, and vulnerabilities, and will update portions of the plan annually following the release of the President’s budget request. The OTH initiative has gone through several phases since it began in 2010 and, as the initiative has evolved, NNSA has used a variety of established methods—consistent with those in GAO-developed guidance for designing evaluations and criteria adopted by DOE that were used or recommended by the Office of Management and Budget and the National Academy of Sciences—to assess research and development programs’ relevance, quality, and performance. NNSA used these methods throughout the OTH initiative process to identify and assess future threats, and they included a literature review, structured interviews and workshops, and a peer review of OTH findings by external subject matter experts. NNSA also formed a working group that held meetings and interviews to update and refine threats identified in earlier phases. However, although NNSA used established methods, the way NNSA implemented them had several limitations—including the absence of documented analysis from the literature review, limited documentation of the structured interviews, and limited application of peer review standards. Taken together, the limitations we found in the OTH process raise concerns about the quality of the analyses produced and about the usefulness of the OTH initiative, as it has been implemented so far, as a DNN planning tool. NNSA began the OTH initiative process in 2010 to identify and assess possible future nuclear and radiological proliferation threats and related trends over the next decade to inform DNN program planning and other DNN management decisions. Since the initiative’s inception, the process has evolved over three phases and encompassed a variety of established methods. It has produced three different reports, with a fourth due later in 2015. (See app. II for a list of the unclassified proliferation-related threats and trends identified in each of the three reports to date.) Table 1 summarizes the three OTH phases, which are described in greater detail below. According to NNSA documents and officials, NNSA officials initiated phase 1 of OTH work in October 2010 within the NIS program office to identify key threats and trends that could impact the emerging nuclear security and nonproliferation environment, discuss their implications for the NIS program, and provide recommendations regarding both general direction and specific areas of emphasis for the program for calendar years 2016 to 2020. This NIS specific phase of OTH work included the use of several established methods consistent with those in GAO- developed guidance, including a literature review encompassing several dozen U.S. government documents and other publications, two workshops, and structured interviews with participants from the U.S. government, nongovernmental organizations, and industry. According to GAO-developed guidance of established methods for designing evaluations, a key first step in designing an evaluation is to collect background information—such as by conducting a literature review—to understand a program’s history, related policies, and knowledge base. Such a review can help focus evaluation questions on knowledge gaps, identify design and data collection options used in the past, and provide important context. In addition, to aggregate and generalize results, it is important to collect uniform data from everyone interviewed, such as through structured interviews or self-administered questionnaires, which help ensure that the same questions are asked of all. According to the guidance, designing a consistent set of responses into the data collection process, such as by using a structured interview, helps establish the uniformity of data across units in the sample. The NIS program office produced a report from this phase in July 2011. The report identified 10 overarching trends that could impact the future of the NIS program, grouped under three broad categories: (1) the evolving strategic landscape, (2) proliferation and nuclear terrorism challenges, and (3) existing and potential stresses on current nonproliferation and nuclear security regimes. The report also examined over 50 potential proliferation-related “wild cards”—events considered to be generally unanticipated, even if sometimes speculated, extremely difficult to prepare for, and outside of the U.S. government’s control—that could impact the trends and planning through 2020, such as terrorist or state use of a nuclear weapon. Issues related to the organization of the NIS program and other resource-related issues were not within the scope of analysis in this phase of OTH study work. The report provided recommendations for the U.S. government, DNN, and the NIS program, along with ideas to support their implementation. The report outlined an overall “logic flow” to the approach that had led to the recommendations, described the methodology and the activities that were conducted, identified documents reviewed in the literature review, and identified the individuals who were interviewed or participated in workshops. The second phase of OTH work was initiated following the July 2011 report completed in the first OTH phase. According to NNSA officials, the NNSA Deputy Administrator for DNN asked, after being briefed on the findings from the 2011 NIS program-specific OTH report, that a second phase of OTH work be undertaken to identify potential security and proliferation threats and trends and the implications for all DNN programs for calendar years 2017 to 2021. In this second phase of OTH work, which began in March 2012, NNSA developed formal terms of reference for OTH work specifying the objective, approach, and methodology for evaluating such threats and trends. (See app. IV for an explanation of these terms of reference.) For example, the stated OTH initiative objective was to produce a cross-cutting analysis to help DNN position itself to best meet nuclear security and nonproliferation challenges over the next 5 to 10 years. In this phase, NNSA also formed an OTH working group, consisting of representatives from each DNN program office, tasked with developing an updated analysis. To do its work, this working group met weekly, received briefings from each DNN program office and the DOE Office of Intelligence and Counterintelligence, and convened a peer review group to provide external validation of study results consistent with established methods. Federal agencies and other organizations, such as the National Academies of Sciences, have established standards and principles for peer review of technical and scientific work. For instance, a National Academies report states that peer review is characterized, in part, as being a documented, critical review of assumptions, calculations and methodology, performed by a person having technical expertise in the subject matter to be reviewed who is independent and external of the work being reviewed. As was the case in the first phase of OTH work, analysis of DNN organizational issues and other resources-related issues, such as budget and personnel, were not within the scope of this phase. At the end of the second phase, DNN issued a report in December 2012 describing 12 threats and trends affecting nuclear and radiological security and associated implications for DNN during calendar years 2017 to 2021. According to this report, the findings were based on a synthesis of views expressed by the working group participants, information derived from the NIS program-level work in the first OTH phase, and by the external validators in the peer review group. Specific threats and trends are presented in the report under three broad categories: (1) global security dynamics, (2) state-level proliferation, and (3) the terrorist/nonstate actor threat. This 2012 report also identified the implications that these threat trends may have for U.S. national security and DNN, and discussed potential opportunities for DNN leadership to consider in addressing these threats. For example, under the global security dynamics category, the report discusses the threats and trends stemming from the persistence and escalation of regional conflicts, and suggests the need for proactive— rather than reactive—country-specific efforts to promote nuclear and radiological material security in conflict-prone regions. The report further states that to achieve the goal of mitigating the risk of unsecured nuclear and radiological materials in conflict-prone regions, DNN should consider various approaches for implementing nuclear and radiological material security in countries of concern and coordinate and maintain “whole-of- government” crisis mitigation plans in coordination with DOE, the Department of Defense, and the Department of State. Phase 3 of OTH work began in mid-2013, and it is ongoing. According to NNSA documentation, the OTH working group concluded during this phase that the threats identified in phase 2 and in the December 2012 report will remain fundamental concerns for DNN through calendar year 2023. Those threats include potential state-based proliferation and attempts by nonstate actors to acquire nuclear materials. Nevertheless, the OTH working group has reevaluated and further refined the potential proliferation-related threats identified in the second OTH phase study through use of internal working group meetings and interviews with NNSA program officials. According to NNSA officials, these program officials include staff members from DNN program offices, NNSA’s Office of Emergency Operations, and NNSA’s Office of Counterterrorism and Counterproliferation, as well as officials from the DOE Office of Intelligence and Counterintelligence. An OTH report produced by NNSA in October 2013 presented an updated list and descriptions of 10 threats and trends affecting nuclear and radiological security and nonproliferation, focused on calendar years 2018 to 2023. A key objective, according to NNSA officials, was to provide more specificity to these 10 proliferation threats and trends in order to better support long-range DNN program planning. To that end, NNSA officials involved in the OTH process told us that, following the publication of the October 2013 report, members of the OTH working group worked with officials from the DOE Office of Intelligence and Counterintelligence to prepare more detailed analyses and “map” the threats and trends identified in the October 2013 report to specific regions and countries. The OTH working group is planning to produce a report on the results of the threat mapping process in the summer of 2015. See figure 1 for a timeline illustrating the three phases of the OTH process and the main activities conducted under each phase. Through reviews of NNSA documentation and interviews with NNSA officials, we identified several limitations in the way NNSA implemented the established methods it used (i.e., the literature review, structured interviews, and peer review by external subject matter experts) through the OTH initiative to identify and assess future proliferation-related threats and trends, which raise concerns about the quality and usefulness of the analyses produced through the process. The 2011 OTH report, other NNSA documents related to the OTH initiative, and NNSA officials noted that the process used by the OTH initiative to assess the nonproliferation environment and identify emerging threats and trends over the next decade included a review of relevant literature, conducted in the first OTH phase, consistent with established methods as discussed earlier. The literature collected by NNSA for this review included several dozen U.S. strategy documents, policy speeches, and scholarly articles on topics ranging from overarching regional trends, to technological advances, to predictions related to countries of concern. We found limitations in the literature review conducted in the first OTH phase based in part on GAO-developed guidance for designing evaluations. According to this guidance, which is relevant to conducting evaluations, such as those conducted by the OTH initiative, a systematic approach to designing evaluations enhances quality, credibility, and usefulness. However, NNSA did not follow consistent common methodological practices in its literature review. For example, NNSA did not document the methodology or scope for developing this literature review list, the criteria they used for selecting the sources cited in their review, or to whom the literature review was disseminated, according to an NNSA official. NNSA officials also did not prepare a summary analysis of the findings from the literature review or have other documentation showing how the materials and information gleaned in the literature review were used in the development of the 2011 report on threats and trends through the OTH initiative process consistent with common methodological practice. In response, an NNSA official told us that the collected documents—consisting of four binders of documents altogether—were intended to serve as OTH background and be available for others in the office to read if they chose to do so. Because NNSA officials did not document the methodology and scope or a written summary or analysis of the findings from the OTH literature review used in the 2011 report, it is difficult to assess how the background information was used to provide context in developing the potential proliferation threats and trends through the OTH initiative. These conditions, in turn, raise potential quality concerns related to the validity and credibility of the literature review results. According to NNSA documents we reviewed, an important component of the OTH process was conducting structured interviews in the first OTH phase, consistent with established methods as discussed earlier. NNSA conducted these interviews with more than 40 staff members, officials and experts from DOE and NNSA, the national laboratories, National Security Council, Department of State, Department of Defense, the Intelligence Community, and nongovernmental organizations. According to the NNSA documents, the interviews were intended to help gather a range of perspectives on the key threats and trends that could characterize the future strategic environment, as well as to gather ideas on potential programmatic needs and priorities. We found, based on our interviews with NNSA officials, that the OTH structured interview process conducted in the first OTH phase had limitations including the following: According to NNSA officials, NNSA did not document the rationale for selecting individuals to be interviewed to show that the people selected as data sources provided a balanced range of views, were sufficiently knowledgeable, and that their opinions may be considered candid and accurate, consistent with common methodological practice. Instead, NNSA officials involved in the OTH initiative said that they relied on their own subject matter knowledge to consult with experts with whom they were familiar. According to NNSA officials, NNSA did not have detailed records or analyses of these interviews, although this is called for by established methods and common methodological practices. NNSA officials provided us with a list of the questions that were asked to participants in the structured interviews and a document containing quotes excerpted from these interviews. However, this document did not identify or attribute remarks to any of the interview participants, and NNSA officials said that they did not have individual records or analyses or attempt to systematically synthesize information from these interviews. To ensure consistency and to prevent biased interpretation of results, it is common methodological practice to document individual interviews in records and to synthesize information obtained in interviews. We have found that such practices also help ensure the reliability and validity of the information being collected. It is unclear how DNN used information from the interviews to develop the potential proliferation threats and trends of concern identified in OTH reports. For instance, it is not possible to establish how various views were reconciled or reflected in the OTH analysis and reports. Because of the limitations, NNSA does not have assurance that the structured interviews—in terms of selection of individuals to be interviewed, record keeping, and analysis—provided OTH with objective, balanced, and independent views and quality information. NNSA officials conducted a peer review process to validate the findings developed during the second OTH phase consistent with established practice, as discussed earlier. Specifically, according to NNSA officials, the peer review by external validators consisted of a June 2012 presentation of OTH findings to a group of seven individuals who were briefed on the OTH results and were asked to comment at the meeting. However, this process was not carried out in a way consistent with established peer review standards as we discussed earlier. For example, a 2004 DOE peer review guide defines peer review as a rigorous, formal, and documented evaluation process using objective criteria and qualified and independent reviewers to make a judgment of the technical/scientific/business merit, the actual or anticipated results, and the productivity and management effectiveness of programs and/or projects. Based on our interviews with NNSA officials and individuals whom NNSA identified as external subject matter experts, we identified the following limitations in the peer review process that were not consistent with established peer review standards: According to NNSA officials, NNSA did not have a methodology explaining how individuals were chosen as validators to ensure a balanced range of feedback. However, established peer review standards state the importance of having a process for selecting peer reviewers that ensures a broad and balanced range of expertise and perspectives. Some of these individuals told us that they were not provided written documents to review in advance of or during the meeting, such as a draft of the OTH phase 2 report. However, according to established peer review standards, peer reviewers should be provided with relevant documentation in advance of the peer review session, in order to give reviewers time to familiarize themselves with the subject of the review. According to NNSA officials, NNSA did not document the results of the peer review process, such as through a summary document or record of analysis, consistent with established peer review standards that state the importance of receiving written comments from all peer reviewers and producing a record of these comments. As a result of the departures from these established standards during the OTH peer review process, it is not clear whether the process provided independent, objective opinions reflecting all individual inputs and the full range of reviewer comments, or how these opinions were reconciled and reflected in OTH reports to ensure quality information. Taken together, the limitations we found in the OTH process raise concerns about the quality of the analyses produced and about the usefulness of the OTH initiative—as it has been implemented so far—as a DNN planning tool. The OTH initiative was intended to help inform DNN management in its decisions about the future DNN organization and resources needed to address future threats, but it is not clear how information generated by the OTH initiative informed the January 2015 DNN reorganization. In addition, the OTH initiative was established to institutionalize long-term DNN planning and support development of DNN strategic plans, but the extent to which the OTH initiative informed the March 2015 DOE and NNSA strategic plan to reduce global nuclear threats is unclear. According to NNSA officials, the OTH initiative is in the process of being integrated into a new DNN strategic planning and integration function that will be led out of the office of the DNN Deputy Administrator, and which will manage integration of OTH studies into strategic planning documents, such as future versions of the NNSA strategic plan to reduce global nuclear threats. The OTH initiative was intended to develop and provide information on potential future proliferation threats to help inform DNN management in its decisions about the future DNN organization and resources needed to address those threats. In January 2015, NNSA reorganized the DNN programs into a new program office structure by consolidating the previous five DNN program offices into four offices. However, it is not clear how information generated by the OTH initiative informed the decision to reorganize DNN programs into the new structure. In October 2014, before the January 2015 reorganization, NNSA officials in the DNN Deputy Administrator’s office stated that the then-planned DNN reorganization was informed by the potential future proliferation- related threats and trends identified through the OTH process— specifically those identified in the July 2011 and December 2012 reports prepared during the first and second phases of the OTH initiative. However, these officials could not provide us with documentation or point to specific examples that could demonstrate direct links between the findings in the 2011 and 2012 OTH reports and elements of the DNN reorganization, such as how changes in certain threats or concerns about new or emerging threats prompted changes in program structure. In a subsequent meeting, NNSA officials in the DNN Deputy Administrator’s office stated that the information produced through the OTH process about the changing proliferation threat environment provided general context for the decision to reorganize the DNN programs. They also stated that the initiative was one of several sources of information that DNN management considered in developing the new DNN structure; according to these officials, other sources included information about evolving proliferation threats provided by the intelligence community. NNSA officials in the DNN Deputy Administrator’s office also stated that there were other drivers behind the reorganization, and that it was not solely a threat-based reorganization. Specifically, these officials stated that a key reorganization goal was to achieve greater efficiency in DNN program management by consolidating similar nonproliferation functions that were fragmented among several DNN program offices. For instance, NNSA officials stated that some programs were spread among several DNN program offices working on issues related to the security of nuclear material overseas and that the reorganization would consolidate those functions under a single program office. In addition, these officials told us the reorganization was motivated by uncertain prospects for future joint nuclear security work in Russia and by the completion or review of certain DNN programs, such as an assessment of alternative options for U.S. plutonium disposition. NNSA officials stated that the new structure will provide DNN the capability and flexibility to respond to changes in the future proliferation threat environment. The OTH initiative was established to institutionalize long-term DNN planning and inform development of DNN strategic plans. In March 2015, NNSA issued a strategic plan to reduce global nuclear threats that was presented as the first strategic plan to define and describe NNSA program missions—including DNN programs—to prevent, counter, and respond to future threats of nuclear proliferation and terrorism. The unclassified plan states that a classified appendix will be issued, which will provide more details about future nuclear and radiological threats and NNSA activities to address the evolving global nuclear security environment. On the basis of our review of the unclassified plan and a draft of the classified appendix, as well as our interviews with NNSA officials, we found that the extent to which the OTH initiative informed the new plan is unclear for two reasons. First, the time frames covered by the March 2015 plan and the OTH initiative are not aligned. The March 2015 plan focuses on NNSA program plans over the agency’s 5-year budget planning horizon—i.e., fiscal years 2016 to 2020. However, the objective of the OTH initiative is to assist DNN by analyzing nuclear and radiological security and nonproliferation challenges 5 to 10 years into the future, or beyond NNSA’s 5-year budget planning horizon. For example, in October 2013, the OTH initiative produced an updated analysis of nuclear and radiological security threats and trends for the 2018 to 2023 time frame. Because the March 2015 plan focuses on NNSA program objectives, activities, and plans for the next 5 fiscal years, it is not clear how information produced through the OTH initiative would align with that planning effort. Second, the extent to which the OTH initiative informed development of the new NNSA plan is unclear because of conflicting information in the unclassified plan and the draft of the classified appendix regarding the contributions of the initiative in supporting both documents. The unclassified plan includes several references to the OTH initiative, including the future nuclear security threats and trends identified through the initiative, and the role it will play in developing future versions of the plan. For instance, the document’s executive summary and conclusion state that NNSA will continue to use OTH studies to validate that its efforts are focused on addressing current threats and to anticipate emerging and evolving threats. The plan also states that information from OTH studies, along with information from external sources, will be used to conduct cross-program and program-specific risk assessments and priority setting. In addition, the document describes 5 proliferation-related threat trends that the plan states were identified by the OTH initiative. These 5 threat trends, described in general terms in the plan’s introduction, consolidate most of the 10 threats and trends for the 2018 to 2023 time frame identified in the October 2013 OTH report. (See app. II for a list of the unclassified proliferation-related threats and trends identified in the October 2013 OTH report and the March 2015 plan.) In April 2015, we reviewed a copy of the draft classified appendix to the March 2015 plan, and we discussed the appendix and the unclassified plan with the NNSA official who coordinated production of both documents. Based on our review of the documents and our discussion with the NNSA official, we did not find a clear connection between the plan and the OTH initiative. The draft classified appendix did not contain any references to the OTH initiative or findings of future proliferation related threats generated through its process. Instead, the NNSA official stated that the details on the future proliferation-related threats and trends described in the classified appendix were derived from a classified analysis prepared by an entity within the intelligence community, not from the completed OTH studies. Furthermore, the NNSA official told us that the OTH initiative was on a “separate but parallel track” with preparation of the strategic plan, and that findings from the OTH process were used only to provide general setup and context for the plan—specifically the 5 threat trends discussed in the unclassified plan’s introduction. The official said that information from the OTH studies did not provide the threat basis for the new plan because the threats and trends identified by the initiative were stated in terms that were too general or academic. However, he noted that the OTH initiative was still completing regional studies to map proliferation-related threats and trends identified in the October 2013 OTH report to specific countries and regions, and he told us that findings from the regional studies could inform the next version of the strategic plan, to be issued in 2016. NNSA officials told us that the OTH initiative is in the process of being integrated into a new DNN strategic planning and integration function that was implemented as part of the reorganization, which will be led out of the office of the DNN Deputy Administrator. Among other things, the NNSA official responsible for managing this effort will lead the analysis and coordination of the OTH process and other long-range studies, and manage the integration of these studies into strategic planning documents, such as future versions of the NNSA strategic plan to reduce global nuclear threats. DNN programs remain critical to addressing the serious threats associated with the proliferation of nuclear and radiological weapons. NNSA has taken an important step through the OTH initiative to identify and assess proliferation-related threats and trends over the next decade and evaluate what they mean for the future of the DNN programs. NNSA used established methods for designing the OTH studies, but we found several limitations in the way the NNSA implemented these methods. These limitations raise concerns about the quality and usefulness of the analyses produced through the OTH process, including absence of documented analysis from its literature review; limited documentation of structured interviews; and limited application of established peer review standards. In turn, these limitations raise concerns about the usefulness of the OTH initiative—as it has been implemented so far—as a DNN planning tool. NNSA could enhance the usefulness of the OTH initiative as it moves forward by better implementing established methods— including literature reviews, structured interviews, and peer reviews associated with threat identification and assessment. To ensure that any future NNSA effort—through the OTH initiative or another process—to assess proliferation threats and the implications for DNN produces high-quality information, we recommend that the NNSA Administrator better implement established methods, including literature reviews, structured interviews, and peer reviews. We provided a draft of this report to NNSA for comment. In written comments provided by NNSA (reproduced in app. V), NNSA generally agreed with our findings and recommendation to improve its OTH threat assessment process. Specifically, NNSA stated that as the OTH initiative continues to evolve and mature, NNSA will incorporate the lessons learned from our report into its standard processes. NNSA also noted that it has created an Office of Strategic Planning and Implementation within DNN to manage the OTH initiative and other information and planning tools. We are sending copies of this report to the appropriate congressional committees, the NNSA Administrator, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI. Our objectives were to (1) describe how the National Nuclear Security Administration (NNSA) assessed potential future proliferation threats through its “Over the Horizon” (OTH) initiative and assess the limitations, if any, in the process used by the initiative and (2) examine the extent to which NNSA used information about the potential future proliferation threats assessed through the OTH initiative in DNN organization and planning decisions. To address our first objective, we interviewed members of management in the Office of Defense Nuclear Nonproliferation (DNN)—including the Deputy Administrator for Defense Nuclear Nonproliferation, the DNN Principal Assistant Deputy Administrator, the DNN Senior Advisor, and DNN program office Assistant Deputy Administrators—and other NNSA officials involved in implementing the OTH initiative to obtain views and information on the goals of and the process used by the OTH initiative, such as key dates, activities, reports, and other milestones associated with the process. We also analyzed documentation NNSA officials provided regarding the OTH process and findings from it. In particular, we analyzed the OTH published reports: Office of Nonproliferation and International Security: Over the Horizon Study, 2016-2020, July 2011; Office of Defense Nuclear Nonproliferation Over the Horizon Opportunity Analysis, 2017-2021 December 2012; and Threats and Trends Impacting Nuclear and Radiological Security and Nonproliferation in 2018 to 2023, October 2013. We also interviewed individuals who participated in OTH process activities. NNSA officials provided us documents that identified participants—including NNSA officials, other agency officials, national laboratory representatives, and independent nonproliferation experts—in four activities of the OTH process: (1) October 2010 workshop; (2) November 2010-February 2011 structured interviews; (3) April 2011 workshop; (4) June 2012 meeting of external validators in a peer reviewer group. We established a criterion to select a nonprobability sample of participants in these sessions; we selected individuals that participated in multiple activities of the OTH. This allowed us to gain more comprehensive responses across activities. In selecting participants using this criterion, the list of 146 participants across the four OTH initiative activities was reduced to the 32 participants who were involved in more than one OTH activity. We excluded participants from other U.S. agencies, as well as participants who had only served as activity facilitators. We also excluded NNSA personnel to better ensure candid responses and to aim to avoid a conflict of interest. This narrowed our list down to 18 participants. We contacted all 18 of these participants and scheduled interviews based on their availability. We were able to schedule and conduct interviews with 15 of these participants. We determined that the responses from these participants were consistent enough to constitute saturation, and we did not schedule any additional interviews. We compared the methods NNSA used to established methods for designing evaluations, as discussed in GAO-developed guidance, and in criteria adopted by the Department of Energy that were used or recommended by the Office of Management and Budget and the National Academy of Sciences. To address our second objective, we interviewed NNSA officials regarding the DNN reorganization and the rationale for changing the DNN program structure, and the extent to which information provided through the OTH initiative regarding future proliferation-related threats and trends informed the restructuring of the DNN programs. In addition, we reviewed the March 2015 NNSA strategic plan and draft classified appendix describing the planned activities of DNN and other NNSA programs to prevent, counter, and respond to nuclear terrorism and proliferation threats over the next 5 years, and we interviewed NNSA officials about the plan and the extent to which it was informed by information generated through the OTH initiative. Because it was outside the scope of our review, we did not assess the plan or the reorganization themselves, for example, by comparing them against established criteria for effective strategic planning or effective organizational transformation. We conducted this performance audit from May 2014 to October 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Prevent, Counter, and Respond—A Strategic Plan to Reduce Global Nuclear Threats (FY 2016-2020), March 2015 Securing and managing nuclear and radiological materials will be challenged by the significant amounts of these materials (and possibly increased amounts of weapon-usable nuclear materials), including in regions of concern, as well as by the erosion of control within weak or failing states. Possessing nuclear weapons capabilities still could be seen as salient and desirable for some state and nonstate actors hostile to U.S. and allied interests, putting strains on monitoring, verifying, and maintaining arms control and nonproliferation regimes. The global expansion of civil nuclear power and the wide use of radiological sources may accelerate the spread of dual-use technology and knowledge and increase demands on safety, security, safeguards, and emergency response systems. Expanding global trade volumes and sophistication of illicit procurement networks will increase the opportunities for state and nonstate actors to acquire dual-use nuclear equipment and technology. Prevent, Counter, and Respond—A Strategic Plan to Reduce Global Nuclear Threats (FY 2016-2020), March 2015 Rapidly changing technologies and greater diffusion of dual-use knowledge are expected to provide more ways for terrorists to threaten nuclear security systems and easier acquisition pathways to nuclear weapons capabilities. Rise in “lone wolf” operations Persistent “insider threats” In January 2015, the National Nuclear Security Administration (NNSA) reorganized the Defense Nuclear Nonproliferation (DNN) programs into a new program office structure. Prior to the reorganization, DNN activities were arranged into the following five program offices: Office of Global Threat Reduction. This office implemented the Global Threat Reduction Initiative (GTRI) that consisted of three subprograms: a Highly Enriched Uranium (HEU) Reactor Conversion subprogram that supported conversion of domestic and international civilian research reactors and isotope production facilities from HEU to low-enriched uranium (LEU); a Nuclear and Radiological Material Protection subprogram, which worked to install security upgrades on high-priority nuclear and radiological materials at civilian sites in the United States and other countries; and a Nuclear and Radiological Material Removal subprogram that included activities to (1) support removal and disposal of U.S.-origin HEU, Russian–origin HEU, and other high-risk nuclear materials; (2) develop capabilities to address nuclear materials associated with “emerging threats” and rapidly denuclearize a country through nuclear material removal when such opportunities emerge, such as in Libya in 2004; (3) support removal and disposal of excess or abandoned radiological materials in other countries; and (4) support rapid removal and disposal of radiological materials in the United States. Office of Defense Nuclear Nonproliferation R&D. This office implemented the DNN Research and Development (R&D) Program, which included two subprograms supporting long-term basic and applied research, development, and testing of new technologies: a Proliferation Detection subprogram to (1) improve U.S. capabilities to detect and monitor nuclear weapons production and proliferation of nuclear weapon-usable materials and (2) a Nuclear Detonation Detection subprogram to detect nuclear explosions worldwide. Office of Nonproliferation and International Security (NIS). This office implemented the NIS program that included four subprograms to: (1) develop NNSA nonproliferation and arms control policy and support implementation of bilateral and multilateral nonproliferation requirements stemming from national nonproliferation initiatives, agreements, and treaties; (2) support the International Atomic Energy Agency’s (IAEA) safeguards system and international nuclear security, including a safeguards policy component that maintains qualified and knowledgeable safeguards staff at U.S. national laboratories and IAEA and implements U.S. safeguards obligations at DOE facilities, a safeguards engagement component that provides safeguards implementation training to other countries, a safeguards technology development component that manages a support program to IAEA’s safeguards department for safeguards R&D and resolution of technical safeguards issues, and an international nuclear security component that conducts assessments of the physical protection of U.S.-obligated nuclear material overseas and works with foreign partners and IAEA to develop and implement nuclear security best practices; (3) promote nuclear controls, by strengthening national nuclear export control systems, developing international nuclear forensics capabilities, conducting technical reviews of domestic export licenses, and providing technical support to U.S. government agencies involved in weapons of mass destruction (WMD) interdiction; and (4) support nuclear verification, including activities in support of nuclear warhead dismantlement and fissile materials transparency, nuclear noncompliance verification, and implementation of HEU transparency arrangements. Office of International Material Protection and Cooperation (IMPC). This office implemented the IMPC program that consisted of two major subprograms, a nuclear Material Protection, Control, and Accounting (MPC&A) program and a Second Line of Defense program. The MPC&A program included several subprograms: a Weapons Material Protection subprogram to provide MPC&A upgrades at nuclear weapon material sites in Russia and other countries; a Material Consolidation and Civilian Sites subprogram to consolidate and convert weapon-usable nuclear material stocks in Russia, and to improve security at civilian sites in Russia and in other locations outside of Russia; a National Infrastructure and Sustainability subprogram to assist Russia and other countries in developing national MPC&A infrastructure and supporting the sustainability of U.S.-funded security upgrades. The Second Line of Defense subprogram strengthens the capacity and commitment of foreign countries to deter, detect, and interdict illicit trafficking of nuclear and other radioactive materials by working with foreign governments to deploy radiation detection systems and by providing training, maintenance, and sustainability assistance. Office of Fissile Materials Disposition. This office implemented two programs: (1) an HEU Disposition program to downblend surplus U.S. HEU to LEU and (2) a Plutonium Disposition program to dispose of surplus U.S. plutonium and support disposition of Russian weapon- grade plutonium. The Plutonium Disposition program included budgeted funding for construction of facilities in the United States to dispose of surplus U.S. plutonium. Under the January 2015 DNN reorganization, NNSA officials consolidated these five DNN program offices into the following four program offices: Office of Defense Nuclear Nonproliferation R&D. This office and the DNN R&D program remain unchanged under the new DNN structure. Office of Material Management and Minimization. This office will implement some program activities previously conducted under the Office of Global Threat Reduction and the Office of Fissile Materials Disposition. Specifically, this office will implement (1) HEU reactor conversion activities conducted previously through the GTRI program; (2) activities to remove and dispose of U.S.-origin HEU, Russian– origin HEU, and other high-risk nuclear materials, and to develop capabilities to address “emerging threats” and rapidly denuclearize a country through nuclear material removal that were conducted previously through the GTRI program; and (3) the U.S. uranium disposition and U.S. and Russian plutonium disposition program activities, including management of related plutonium disposition facility construction in the United States, that were conducted previously through the Fissile Materials Disposition program. In the fiscal year 2016 budget request, NNSA has placed funding for construction of the plutonium disposition facilities under a separate program entry in the budget: “Nonproliferation Construction.” However, organizationally, this office will oversee the nonproliferation construction program budget. Office of Global Material Security. This office will incorporate the program activities previously conducted under the IMPC program. In addition, the Global Material Security Program will implement the domestic and international radiological removal and nuclear and radiological material protection functions that were previously under the GTRI program. This new program will also implement two activities conducted previously under the NIS program: international nuclear forensics cooperation efforts and the portion of the international nuclear security program activities involving engagement with foreign partners and IAEA on nuclear security best practices. Office of Nonproliferation and Arms Control. This office will continue to implement most of the activities conducted previously under the NIS program. However, as noted above, international forensics cooperation activities and the portion of international nuclear security activities focused on promoting nuclear security best practices with foreign partner governments and IAEA have been realigned under the Global Material Security Program. This new office will continue to implement international nuclear security program activities to assess physical protection of U.S.-obligated nuclear materials in overseas locations. The following are the terms of reference for the National Nuclear Security Administration’s “Over the Horizon” (OTH) initiative working group created in March 2012, at the beginning of the second OTH study phase. Produce a cross-cutting analysis to help Defense Nuclear Nonproliferation (DNN) position itself to best meet nuclear and radiological security and nonproliferation challenges over the next 5 to 10 years. DNN will establish a working group with representation from all DNN offices with the Office of Nonproliferation and International Security (NIS) as the Chair. The working group will provide regular updates to the DNN Assistant Deputy Administrators/Associate Assistant Deputy Administrators. Work will begin immediately. Define the threat environment and its trends as insight to how the threats may evolve over the next 5 to 10 years, focusing on aspects of the threat most relevant to DNN. This would be informed by the already completed NIS effort. 1. Identify core strengths of the DNN functional bureaus. What do we do best? What are our strongest assets? What do we bring to the table that is unique, particularly with respect to the interagency process? 2. On the basis of the above, identify prospective and emerging nuclear proliferation and nuclear/radiological security challenges, gaps, needs, and opportunities that will require greater attention by the USG and its allies, and where it makes sense for DNN to take a proactive role in meeting these challenges. 3. Based on these challenges, and the status of programmatic efforts, identify gaps, needs, and opportunities. Taking into account that unanticipated events may occur between now and 2021 that could involve a DNN response. 4. Develop a set of recommendations for DNN offices to factor into their future programs and activities consistent with the analysis above. Recommendations will consider scope of programs, resources, and organizational structure. In addition to the individual named above, Jonathan Gill (Assistant Director), Antoinette Capaccio, William Hoehn, Cynthia Norris, Sophia Payind, Steven Putansu, Dan Royer, and Kiki Theodoropoulos made key contributions to this report.

Summary: The threat posed by the proliferation of nuclear and radiological weapons remains one of the most pressing U.S. national security challenges, and these threats are evolving. Information produced by the OTH initiative about future proliferation threats is intended to support long-term DNN planning and other DNN management decisions. The Joint Explanatory Statement Accompanying the National Defense Authorization Act for Fiscal Year 2014 includes a provision for GAO to assess NNSA's nuclear nonproliferation programs. This report (1) describes how NNSA assessed potential proliferation threats through its OTH initiative and assesses the limitations, if any, in the process used by the initiative and (2) examines the extent to which NNSA used information about the potential threats assessed through the OTH initiative in DNN organization and planning decisions. GAO analyzed NNSA documentation on the OTH initiative and NNSA planning documents. GAO also interviewed DOE and NNSA officials, DOE national laboratory representatives, and external subject matter experts and external validators in peer review groups involved in OTH activities. The National Nuclear Security Administration (NNSA)—a semiautonomous agency within the Department of Energy (DOE)—has used a variety of established methods in its “Over the Horizon” (OTH) initiative to assess potential proliferation threats, but the implementation of these methods had several limitations. NNSA used established methods in its OTH initiative, including: (1) a literature review of studies on proliferation threats; (2) structured interviews with participants from the U.S. government, nongovernmental organizations, and industry; and (3) a peer review of OTH findings by external subject matter experts. Although NNSA used established methods, the way NNSA implemented them had several limitations. For example, NNSA officials did not document the rationale for selecting individuals for structured interviews to show that those selected provided a balanced range of views and were sufficiently knowledgeable. NNSA officials also did not have detailed records or analyses of these interviews. Established methods and common methodological practices call for structured interviews to be documented and analyzed. GAO has found that such practices help ensure the reliability and validity of the information collected. Another limitation was that the peer review was not conducted in a way consistent with established standards. For example, NNSA officials did not document the results of the peer review, as called for by established peer review standards, and some reviewers told GAO that it was unclear how their comments had been incorporated into a 2012 OTH report. These limitations raise concerns about the quality of the analyses produced, and about the usefulness of the OTH initiative, as it has been implemented so far, as a planning tool in NNSA's Office of Defense Nuclear Nonproliferation (DNN). It is unclear how information generated by the OTH initiative informed recent organization and planning decisions in the DNN office, even though NNSA established the initiative partly to support decision making on such matters. For instance, in January 2015, NNSA reorganized DNN programs by consolidating five DNN program offices into four offices. The extent to which the OTH initiative informed the reorganization is unclear because NNSA officials could not provide documentation or examples of links between OTH findings and elements of the reorganization. Instead, NNSA officials told GAO that the OTH initiative was one of several sources that informed the reorganization and that there were other reasons for it, including consolidating similar functions to achieve efficiencies. In addition, in March 2015, NNSA issued an unclassified strategic plan that was presented as defining and describing NNSA program missions—including DNN programs—to prevent, counter, and respond to future nuclear proliferation and terrorism threats. However, the extent to which the OTH initiative informed the plan was unclear because the plan and the OTH initiative covered different time frames, and because the plan and a draft classified appendix contained conflicting information about the role of the initiative in the plan's development. GAO is not making a recommendation on these matters because NNSA is creating a new strategic planning function that will oversee the OTH process and manage integration of OTH and other long-range studies into future versions of the NNSA strategic plan on nuclear threats. In conducting any future proliferation threat assessments, through the OTH initiative or another process, NNSA should better implement established methods, including literature reviews, structured interviews, and peer reviews. NNSA agreed with this recommendation and is taking action to address it.