NNSA’s Deputy Administrator for Defense Nuclear Nonproliferation directs the DNN office. The DNN Deputy Administrator reports to the NNSA Administrator and, through the NNSA Administrator, to the Secretary of Energy. Other NNSA staff in the DNN Deputy Administrator’s office include a Principal Assistant Deputy Administrator, a Chief Operating Officer, and a Senior Advisor. Each DNN program office is headed by an Assistant Deputy Administrator (ADA). In August 2012, DNN established a DNN Management Council to serve as its corporate management mechanism. This council, consisting of the NNSA Deputy Administrator for DNN, the DNN Principal Assistant Deputy Administrator, the DNN Chief Operating Officer, the DNN Senior Advisor, and the ADAs, has formulated various DNN program decisions—including on budget, staffing, and priorities—and has addressed other issues relating to the DNN programs on an ad hoc basis. In January 2015, NNSA reorganized the DNN programs into a new program office structure. Prior to the reorganization, DNN activities were arranged into five program offices: Office of Global Threat Reduction; Office of Defense Nuclear Nonproliferation R&D; Office of Nonproliferation and International Security; Office of International Material Protection and Cooperation; and Office of Fissile Materials Disposition. Under the January 2015 DNN reorganization, NNSA officials consolidated these five DNN program offices into four program offices: Office of Defense Nuclear Nonproliferation R&D; Office of Material Management and Minimization; Office of Global Material Security; and Office of Nonproliferation and Arms Control. Appendix III provides more information on the DNN reorganization. In March 2015, NNSA issued a new unclassified strategic plan for NNSA efforts to address future nuclear proliferation and terrorism threats for fiscal years 2016 to 2020 in response to an August 2014 interim report by the Secretary of Energy Advisory Board Task Force on Nuclear Nonproliferation. Among other things, that task force recommended that DOE produce a biannual report on nonweapons national security activities—especially nonproliferation programs—comparable to an annual report NNSA currently produces on the management of the nuclear weapons programs over the next 25 years, known as the Stockpile Stewardship and Management Plan. The DOE Secretary agreed with this recommendation and asked NNSA to begin drafting such a report. The March 2015 plan was presented as the first strategic plan to define and describe DOE and NNSA missions to prevent, counter, and respond to the threats of nuclear proliferation and terrorism, including the DNN programs. For each DNN program, the plan identifies program objectives, priorities, and performance measures; program activities, accomplishments, and challenges; and program plans over NNSA’s fiscal year 2016 to fiscal year 2020 budget horizon. In addition, the plan states that NNSA will issue a classified appendix containing more details on nuclear and radiological risks, threats, and vulnerabilities, and will update portions of the plan annually following the release of the President’s budget request. The OTH initiative has gone through several phases since it began in 2010 and, as the initiative has evolved, NNSA has used a variety of established methods—consistent with those in GAO-developed guidance for designing evaluations and criteria adopted by DOE that were used or recommended by the Office of Management and Budget and the National Academy of Sciences—to assess research and development programs’ relevance, quality, and performance. NNSA used these methods throughout the OTH initiative process to identify and assess future threats, and they included a literature review, structured interviews and workshops, and a peer review of OTH findings by external subject matter experts. NNSA also formed a working group that held meetings and interviews to update and refine threats identified in earlier phases. However, although NNSA used established methods, the way NNSA implemented them had several limitations—including the absence of documented analysis from the literature review, limited documentation of the structured interviews, and limited application of peer review standards. Taken together, the limitations we found in the OTH process raise concerns about the quality of the analyses produced and about the usefulness of the OTH initiative, as it has been implemented so far, as a DNN planning tool. NNSA began the OTH initiative process in 2010 to identify and assess possible future nuclear and radiological proliferation threats and related trends over the next decade to inform DNN program planning and other DNN management decisions. Since the initiative’s inception, the process has evolved over three phases and encompassed a variety of established methods. It has produced three different reports, with a fourth due later in 2015. (See app. II for a list of the unclassified proliferation-related threats and trends identified in each of the three reports to date.) Table 1 summarizes the three OTH phases, which are described in greater detail below. According to NNSA documents and officials, NNSA officials initiated phase 1 of OTH work in October 2010 within the NIS program office to identify key threats and trends that could impact the emerging nuclear security and nonproliferation environment, discuss their implications for the NIS program, and provide recommendations regarding both general direction and specific areas of emphasis for the program for calendar years 2016 to 2020. This NIS specific phase of OTH work included the use of several established methods consistent with those in GAO- developed guidance, including a literature review encompassing several dozen U.S. government documents and other publications, two workshops, and structured interviews with participants from the U.S. government, nongovernmental organizations, and industry. According to GAO-developed guidance of established methods for designing evaluations, a key first step in designing an evaluation is to collect background information—such as by conducting a literature review—to understand a program’s history, related policies, and knowledge base. Such a review can help focus evaluation questions on knowledge gaps, identify design and data collection options used in the past, and provide important context. In addition, to aggregate and generalize results, it is important to collect uniform data from everyone interviewed, such as through structured interviews or self-administered questionnaires, which help ensure that the same questions are asked of all. According to the guidance, designing a consistent set of responses into the data collection process, such as by using a structured interview, helps establish the uniformity of data across units in the sample. The NIS program office produced a report from this phase in July 2011. The report identified 10 overarching trends that could impact the future of the NIS program, grouped under three broad categories: (1) the evolving strategic landscape, (2) proliferation and nuclear terrorism challenges, and (3) existing and potential stresses on current nonproliferation and nuclear security regimes. The report also examined over 50 potential proliferation-related “wild cards”—events considered to be generally unanticipated, even if sometimes speculated, extremely difficult to prepare for, and outside of the U.S. government’s control—that could impact the trends and planning through 2020, such as terrorist or state use of a nuclear weapon. Issues related to the organization of the NIS program and other resource-related issues were not within the scope of analysis in this phase of OTH study work. The report provided recommendations for the U.S. government, DNN, and the NIS program, along with ideas to support their implementation. The report outlined an overall “logic flow” to the approach that had led to the recommendations, described the methodology and the activities that were conducted, identified documents reviewed in the literature review, and identified the individuals who were interviewed or participated in workshops. The second phase of OTH work was initiated following the July 2011 report completed in the first OTH phase. According to NNSA officials, the NNSA Deputy Administrator for DNN asked, after being briefed on the findings from the 2011 NIS program-specific OTH report, that a second phase of OTH work be undertaken to identify potential security and proliferation threats and trends and the implications for all DNN programs for calendar years 2017 to 2021. In this second phase of OTH work, which began in March 2012, NNSA developed formal terms of reference for OTH work specifying the objective, approach, and methodology for evaluating such threats and trends. (See app. IV for an explanation of these terms of reference.) For example, the stated OTH initiative objective was to produce a cross-cutting analysis to help DNN position itself to best meet nuclear security and nonproliferation challenges over the next 5 to 10 years. In this phase, NNSA also formed an OTH working group, consisting of representatives from each DNN program office, tasked with developing an updated analysis. To do its work, this working group met weekly, received briefings from each DNN program office and the DOE Office of Intelligence and Counterintelligence, and convened a peer review group to provide external validation of study results consistent with established methods. Federal agencies and other organizations, such as the National Academies of Sciences, have established standards and principles for peer review of technical and scientific work. For instance, a National Academies report states that peer review is characterized, in part, as being a documented, critical review of assumptions, calculations and methodology, performed by a person having technical expertise in the subject matter to be reviewed who is independent and external of the work being reviewed. As was the case in the first phase of OTH work, analysis of DNN organizational issues and other resources-related issues, such as budget and personnel, were not within the scope of this phase. At the end of the second phase, DNN issued a report in December 2012 describing 12 threats and trends affecting nuclear and radiological security and associated implications for DNN during calendar years 2017 to 2021. According to this report, the findings were based on a synthesis of views expressed by the working group participants, information derived from the NIS program-level work in the first OTH phase, and by the external validators in the peer review group.