Congress enacted the Nuclear Waste Policy Act of 1982 to establish a comprehensive policy and program for the safe, permanent disposal of commercial spent fuel and other highly radioactive wastes in one or more mined geologic repositories. The act charged DOE with (1) establishing criteria for recommending sites for repositories; (2) “characterizing” (investigating) three sites to determine each site’s suitability for a repository (1987 amendments to the act directed DOE to investigate only the Yucca Mountain site); (3) recommending one suitable site to the President, who, if he considered the site qualified for a license application, would submit a recommendation to Congress; and (4) seeking a license from NRC to construct and operate a repository at the approved site. The act created the Office of Civilian Radioactive Waste Management within DOE to manage its nuclear waste program. Since the 1980s, DOE has spent years conducting site characterization studies at the Yucca Mountain site to determine whether it is suitable for a high-level radioactive waste and spent nuclear fuel repository. DOE, for example, has completed numerous scientific studies of the mountain and its surrounding region for water flow and the potential for rock movement, including volcanoes and earthquakes that might adversely affect the performance of the repository. To allow scientists and engineers greater access to the rock being studied, DOE excavated two tunnels for studying the deep underground environment: (1) a five-mile main tunnel that loops through the mountain, with several research areas or alcoves connected to it; and (2) a 1.7-mile tunnel that crosses the mountain (see fig. 2). This second tunnel allows scientists to study properties of the rock and the behavior of water near the potential repository area. In July 2002, Congress approved the President’s recommendation of the Yucca Mountain site for the development of a repository. The Yucca Mountain project is currently focused on preparing an application to obtain a license from NRC to construct a repository. The required application information includes both repository design work and scientific analyses. DOE is engaged in necessary tasks such as compiling information and writing sections of the license application, and is conducting technical exchanges with NRC staff and addressing key technical issues identified by NRC to ensure that sufficient supporting information is provided. It also plans to further develop the design of the repository, including revised designs for the repository’s surface facilities and canisters to hold the waste. DOE is also identifying and preparing potentially relevant documentary material that it is required to make available on NRC’s Web-based information system, known as the Licensing Support Network. This is a critical step because DOE is required to certify that the documentary material has been identified and made electronically available no later than 6 months in advance of submitting the license application. In February 2005, DOE announced that it does not expect the repository to open until 2012 at the earliest, which is more than 14 years later than the 1998 goal specified by the Nuclear Waste Policy Act of 1982. More recently, the conference report for DOE’s fiscal year 2006 appropriations observed that further significant schedule slippages for submitting a license application are likely. Further delays could arise from factors such as the time needed for EPA to establish revised radiation standards for Yucca Mountain and for DOE to revise its technical documents in response. Such delays could be costly because nuclear utilities, which pay for most of the disposal program through a fee on nuclear power, have sued DOE, seeking damages for not starting the removal of spent nuclear fuel from storage at commercial reactors by the 1998 deadline. Estimates of the potential damages vary widely, from DOE’s estimate of about $5 billion to a nuclear industry’s estimate of about $50 billion, but the cost for the damages will likely rise if there are further delays to opening the repository. Given these schedule slippages, Congress has considered other options for managing existing and future nuclear wastes, such as centralized interim storage at one or more DOE sites. The conference report for DOE’s fiscal year 2006 appropriations directed DOE to develop a spent nuclear fuel recycling plan to reuse the fuel. However, according to the policy organization of the nuclear energy industry, no technological option contemplated will eliminate the need to ultimately dispose of nuclear waste in a geologic repository. In October 2005, the project’s Acting Director issued a memorandum calling for the development of wide-ranging plans for the “new path forward,” DOE’s effort to address quality assurance and other challenges prior to applying for a license. To restore confidence in scientific documents that will support the license application, some of the plans will address the need to review and replace USGS work products, a requirement for USGS to certify its scientific work products, and establishing a lead national laboratory to assist the project. Other plans are focused on a new simplified design for the waste canisters and repository facilities, a design that is expected to improve the safety and operation of the repository by eliminating the need to directly handle and process the spent fuel at the repository. Further, this aggressive effort called for management changes, including a transition plan; more rigorous project management, including a new baseline schedule; rescoping existing contracts and developing new contracts; tracking project hiring actions; a financial plan; and new reporting indicators. After DOE submits the license application, NRC plans to take 90 days to examine the application for completeness to determine whether DOE has addressed all NRC requirements. One of the reviews for completeness will include an examination of DOE’s documentation of the quality assurance program to assess whether it addresses all NRC criteria. These criteria include, among other things, organization, design control, document control, corrective actions, quality assurance records, and quality audits. If it deems any part of the application is incomplete, NRC may either reject the application or require that DOE furnish the necessary documentation before proceeding with the detailed technical review of the application. If it deems the application is complete, NRC will docket the application, indicating its readiness for a detailed technical review. Once the application is accepted and placed on the docket, NRC will conduct its 18-month technical review of the application to determine if the application meets all NRC requirements, including the soundness of scientific analyses and preliminary facility design, and NRC quality assurance criteria. If NRC discovers problems with the technical information used to support the application, it may conduct specific reviews, including inspections, to determine the extent and effect of the problem. Because the data, models, and software used in modeling repository performance are integral parts of this technical review, quality assurance plays a key role since it is the mechanism used to verify the accuracy of the information DOE presents in the application. NRC may conduct reviews, including inspections, of the quality assurance program if technical problems are identified that are attributable to quality problems. NRC will hold public hearings chaired by its Atomic Safety and Licensing Board to examine specific topics. After completing the proceedings, the board will forward its initial decision to the NRC commissioners for their review. Finally, within 3 to 4 years from the date that NRC dockets the application, NRC will make a decision to grant the construction authorization, reject the application, or grant the construction authorization with conditions. NRC will grant a construction authorization only if it concludes from its reviews that the repository would meet its reasonable expectation that the safety and health of workers and the public would be protected. DOE has repeatedly experienced quality assurance problems with its work on the Yucca Mountain project. In the late 1980s, DOE had been challenged to fix and develop adequate plans and procedures related to quality assurance. By the late 1990s, audits by GAO, DOE, and others identified recurring quality assurance problems with several aspects of key scientific data, models, and software. Currently, in preparing to submit the license application to NRC, DOE is relying on costly and time-consuming rework to resolve lingering quality assurance problems with the transparency and traceability of data and in project design and engineering documents uncovered during audits and after-the-fact evaluations. DOE has a long-standing history of attempting to address NRC concerns about its quality assurance program. Although NRC will have responsibility for regulating the construction, operation, and decommissioning (closure) phases of the project, its regulatory and oversight role does not begin until DOE submits a license application. As a result, NRC’s role in the project has been limited to providing guidance to DOE to ensure an understanding of NRC regulations and that the years of scientific and technical work will not later be found inadequate for licensing purposes. Specifically, since 1984, NRC has agreed to point out problems it identifies with the quality assurance program so that DOE can take timely corrective action. Initially, this NRC guidance was mainly focused on ensuring that DOE had the necessary quality assurance organization, plans, and procedures. As we reported in 1988, NRC had reviewed DOE’s quality assurance plans and procedures comprising the principal framework of its quality assurance program, and concluded that they were inadequate and did not meet NRC requirements. NRC also concluded that DOE’s efforts to independently identify and resolve weaknesses in the plans and procedures were ineffective. After observing DOE quality assurance audits, NRC determined that the audits were ineffective for measuring whether quality assurance procedures were being effectively implemented. Further, NRC identified additional concerns, during the 1980s, related to DOE management and organizational deficiencies relating to the quality assurance program. Specifically, among other things, NRC found the following: DOE had a small staff and relied heavily on contractors to provide quality assurance oversight. Based on its experience in regulating nuclear power plants, NRC found that these types of organizations frequently developed major quality-related problems. DOE had indirect project control, with administrative and functional control over the project split between different offices. NRC found that such project control arrangements tend to have serious quality assurance-related problems because conflicts can arise between quality and other organizational goals, such as cost and schedule. During a 1984 NRC visit to Nevada, DOE project participants had expressed the opinion that quality assurance is “unnecessary, burdensome, and an imposition.” Further, in 1986, DOE issued a stop- work order to the USGS based on a determination that USGS staff did not appreciate the importance of quality assurance and that USGS work would not meet NRC expectations. NRC believed that organizational attitudes can indicate whether a project is likely to experience problems relating to quality assurance and found such examples troublesome. Finally, based in part on the information obtained from its oversight activities, NRC concluded, in 1989, that DOE and its key contractors had yet to develop and implement an acceptable quality assurance program. However, by March 1992, NRC came to the conclusion that DOE had made significant progress in improving its quality assurance program. NRC noted that DOE had addressed many of its concerns, specifically that, among other things, (1) all of the contractor organizations had developed and were in the process of implementing quality assurance programs that met NRC requirements, (2) quality assurance management positions had been filled with full-time DOE personnel with appropriate knowledge and experience, and (3) DOE had demonstrated that it is capable of evaluating and correcting deficiencies in the overall quality assurance program. Nevertheless, in October 1994, NRC found problems with quality assurance, particularly with the site contractor’s ability to effectively implement corrective actions and DOE’s ability to oversee the site contractor’s quality assurance program. As DOE’s quality assurance program matured, it resolved NRC concerns about its organization, plans, and procedures, and in the late 1990s began successfully detecting new quality assurance problems in three areas critical to the repository’s successful performance: the adequacy of the data sources, the validity of scientific models, and the reliability of computer software developed at the site. These problems surfaced in 1998 when DOE began to run the initial version of its performance assessment model. Specifically, DOE was unable to ensure that critical project data had been collected and tracked back to the original sources. In addition, DOE did not have a standardized process for developing scientific models used to simulate a variety of geologic events or an effective process for ensuring that computer software used to support the scientific models would work properly. As required by DOE’s quality assurance procedures, the department conducted a root cause analysis and issued a corrective action plan in 1999. After corrective actions were taken, DOE considered the issues resolved. However, in 2001, similar deficiencies associated with models and software resurfaced. DOE attributed the recurrence to ineffective procedures and corrective actions, improper implementation of quality procedures by line managers, and personnel who feared reprisal for expressing quality concerns. Recognizing the need to correct these recurring problems, DOE conducted a comprehensive root cause analysis that included reviews of numerous past self-assessments and independent program assessments, and identified weaknesses in management systems, quality processes, and organization roles and responsibilities. Following the analysis, in July 2002, DOE issued its Management Improvement Initiatives (Initiatives) that addressed quality problems with software and models. In addition, DOE added other corrective actions to address management weaknesses that it found in areas such as roles and responsibilities, quality assurance processes, written procedures, corrective action plans, and work environment. However, DOE continued to face difficulties in resolving quality assurance problems concerning the data, software, and modeling to be used in support of the licensing application: Data management. As part of NRC’s quality assurance requirements, data used to support conclusions about the safety and design of the repository must be either collected under a quality assurance program or subjected to prescribed testing procedures to ensure the data are accurate for their intended use. In addition, the data supporting these conclusions must also be traceable back to its original source. In 1998, DOE identified quality assurance problems with the quality and traceability of data—specifically that some data had not been properly collected or tested to ensure their accuracy and that data used to support scientific analysis could not be properly traced back to their source. DOE again found similar problems in April and September 2003, when a DOE audit revealed that some data sets did not have the documentation necessary to trace them back to their sources; the processes for data control and management were unsatisfactory; and faulty definitions were developed, which allowed unqualified data to be used. Software management. DOE quality assurance procedures require that software used to support analysis and conclusions about the performance and safety of the repository be tested or created in such a way to ensure that it is reliable. From 1998 to 2003, multiple DOE audits found recurring quality assurance problems that could affect confidence in the adequacy of software codes. For example, in 2003, DOE auditors found problems related to software similar to those found previously in areas such as technical reviews, software classification, planning, design, and testing. Further, a team of industry professionals hired by DOE to assess quality assurance problems with software reported in February 2004 that these problems kept recurring because DOE did not assess the effectiveness of its corrective actions and did not adequately identify the root causes of the problems. Model validation. Models are used to simulate natural and environmental conditions at Yucca Mountain, and to demonstrate the performance of the future repository over time. However, before models can be used to support the license application, DOE must demonstrate through a process called validation that the models are able to accurately predict geologic events. In 1998, a team of project personnel evaluated the models and determined that 87 percent did not comply with the validation requirements. In 2001, and again in 2003, DOE audits found that project personnel were not properly following procedures— specifically in the areas of model documentation, model validation, and checking and review. Further, the 2003 audit concluded that previous corrective actions designed to improve validation and reduce errors in model reports were not fully implemented. After many years of working to address these quality assurance problems with data, software, and models, DOE had mostly resolved these problems and closed the last of the associated condition reports by February 2005. As DOE prepares to submit the Yucca Mountain project license application to NRC, it has relied on costly and time consuming rework to ensure that the documents supporting the application are accurate and complete. Specifically, DOE has relied on inspections and rework by DOE personnel to resolve quality assurance problems with the traceability and transparency of technical work products. These efforts to deal with quality problems at the end, rather than effectively ensuring that work organizations are producing quality products from the beginning, add to the project’s cost and could potentially delay DOE’s submission of the license application to NRC. In addition, DOE’s efforts indicate that some corrective actions have been ineffective in resolving problems with the quality assurance process. Further, DOE is now detecting quality assurance problems in design and engineering work that are similar to the quality assurance problems it experienced with its scientific work in the late 1990s. Although DOE did not initiate its major effort to address these problems until 2004, the department and NRC for years had known of quality assurance problems with the traceability and transparency of technical work products called Analysis and Model Reports (AMR). AMRs are a key component of the license application, and contain the scientific analysis and modeling data demonstrating the safety and performance of the planned repository. Among other quality requirements, AMRs must be traceable back to their original source material and data, and must also be transparent in justifying and explaining their underlying assumptions, calculations, and conclusions. In 2003, based in part on these problems as well as DOE’s long-standing problems with data, software, and modeling, NRC conducted an independent evaluation of three AMRs. The scope of the review was to determine if the AMRs met NRC requirements for being traceable, transparent, and technically appropriate for their use in the license application. NRC found significant problems. First, in some cases DOE was not transparent in explaining the basis on which it was reaching conclusions. For example, in some circumstances, DOE selected a single value from a range of data without sufficient justification. Other times, DOE did not explain how a range of experimental conditions were representative of repository conditions. Second, where DOE did sufficiently explain the basis for a conclusion, it did not always provide the necessary technical information, such as experimental data, analysis, or expert judgment, to trace the support for that explanation back to source materials. For example, DOE did not explain how information on one type of material provided an appropriate comparison for another material. Moreover, while DOE had identified similar problems in the past, the actions taken to correct them did not identify and resolve other deficiencies. NRC concluded that these findings suggested that other AMRs possibly had similar problems, and that if not resolved, such problems could delay NRC’s review of the license application as it would need to conduct special inspections to resolve any issues it found with the quality of technical information. To address problems of traceability and transparency, DOE in the spring of 2004 initiated an effort called the Regulatory Integration Team (RIT) to perform a comprehensive inspection and rework of the AMRs to ensure they met NRC requirements and expectations. According to DOE officials, the RIT involved roughly 150 full-time personnel from DOE, USGS, and multiple national laboratories such as Sandia, Los Alamos, and Lawrence Livermore. First, the RIT screened all of the approximately 110 AMRs and prioritized its efforts on 89 that needed additional rework. Ten AMRs were determined to be acceptable, and 11 were canceled because they were no longer needed to support the license application. According to DOE officials, approximately 8 months later, the RIT project was completed at a cost of about $20 million, with a total of over 3,700 problems and issues addressed or corrected. In February 2005, in a letter to DOE, the site contractor stated that the RIT effort was successful and that the AMRs had been revised to improve traceability and transparency. Subsequently, however, additional problems with traceability and transparency have been identified, requiring further inspections and rework. For example, after the March 2005 discovery of e-mails from USGS employees written between May 1998 and March 2000 implying that employees had falsified documentation of their work to avoid quality assurance standards, DOE initiated a review of additional AMRs that were not included in the scope of the 2004 RIT review. The additional AMRs contained scientific work performed by the USGS employees and had been assumed by the RIT to meet NRC requirements for traceability and transparency. However, according to DOE officials, DOE’s review determined that these AMRs did not meet NRC’s standards, and additional rework was required. Further, similar problems were identified as the focus of the project shifted to the design and engineering work required for the license application. In February 2005, the site contractor determined that in addition to problems with AMRs, similar traceability and transparency problems existed in the design and engineering documents that comprise the Safety Analysis Report—the report necessary for demonstrating to NRC how the facilities and other components of the repository site will meet the project’s health, safety, and environmental goals and objectives. In a root cause analysis of this problem, the site contractor noted that additional resources were needed to inspect and rework the documents to correct the problems. DOE cannot be certain that it has met continuous improvement goals for implementing its quality assurance requirements, a commitment DOE made at the closure of its Management Improvement Initiatives (Initiatives) in April 2004. At that time, DOE told us it expected that the progress achieved with the initiatives would continue and that its performance indicators would enable it to assess further progress and direct management attention as needed. However, DOE’s performance indicators, as well as a second management tool—trend evaluation reports—have not been effective for this purpose. More specifically, the indicators panel did not highlight the areas of concern covered by the initiatives and had weaknesses in assessing progress because the indicators kept changing. The trend evaluation reports also did not focus on tracking the concerns covered by the Initiatives, had technical weaknesses for identifying significant and recurring problems, had inconsistently tracked progress in addressing problems, and could not fully analyze projectwide problems. In addition, the trend reports’ tracking of problems for which corrective actions were already being taken was at times overly influenced by judgments about whether additional management action was warranted rather than the problems’ significance. By the time that the actions called for by the Initiatives had been completed in April 2004, project management had already developed the indicators panel, which DOE refers to as the annunciator panel, to use at monthly management meetings to monitor project performance. The panel was a single page composed of colored blocks representing selected performance indicators and their rating or level of performance. A manager viewing the panel would be able to quickly see the color rating of each block or indicator. For example, red indicated degraded or adverse performance warranting significant management attention; yellow indicated performance warranting increased management attention or acceptable performance that could change for the worse; and green indicated good performance. The panel represented a hierarchy of indicators in which the highest level, or primary, indicators were shown; secondary indicators that determined the primary indicators’ ratings were shown for some primary indicators; but lower third- or fourth-level indicators were not shown. Our review analyzed a subset of these indicators that DOE designated as the indicators that best predict performance in areas affecting quality. While we were conducting our review, DOE suspended preparation of the panel after August 2005 while it reconsiders its use of indicators to monitor project performance. DOE had also suspended preparation of the panel from late 2004 to early 2005 in order to make substantial revisions. These revisions were made, in part, to emphasize fewer, more important indicators for management attention. The Initiatives raised concerns about five key areas of management weakness as adversely affecting the implementation of quality assurance requirements: 1. Roles and responsibilities were becoming confused as the project transitioned from scientific studies to activities supporting licensing. The confusion over roles and responsibilities was undermining managers’ accountability for results. The Initiatives’ objective was to realign DOE’s project organization to give a single point of responsibility for project functions, such as quality assurance and the Corrective Action Program, and hold the project contractor more accountable for performing the necessary work in accordance with quality, schedule, and cost requirements. 2. Product quality was sometimes being achieved through inspections by the project’s Office of Quality Assurance rather than being routinely implemented by the project’s work organizations. As a result, the Initiatives sought to increase work organizations’ responsibility for being the principle means for achieving quality. 3. Work procedures were typically too burdensome and inefficient, which impeded work. The Initiatives sought to provide new user-friendly and effective procedures, when necessary, to allow routine compliance with safety and quality requirements. 4. Multiple corrective action programs existed, processes were burdensome and did not yield useful management reports, and corrective actions were not completed in a timely manner. The Initiatives sought to implement a single program to ensure that problems were identified, prioritized, and documented and that timely and effective corrective actions were taken to preclude recurrence of problems. 5. The importance of a safety-conscious work environment that fosters open communication about concerns was not understood by all managers and staff, and they had not been held accountable when inappropriately overemphasizing the work schedule, inadequately attending to work quality, and acting inconsistently in practicing the desired openness about concerns. Through issuing a work environment policy, providing training on the policy, and improving the Employee Concerns Program, the Initiatives sought to create an environment in which employees felt free to raise concerns without fear of reprisal and with confidence that issues would be addressed promptly and appropriately. As shown in table 1, the Initiatives’ effectiveness indicators for tracking progress in addressing these management weaknesses did not have equivalent performance indicators visible in the annunciator panel when it was prepared for the last time, using August 2005 data. Two of the Initiatives’ key areas of concern—(1) roles, responsibilities, authority, and accountability; and (2) work procedures—and their associated effectiveness indicators were not represented in the panel’s visible or underlying indicators. The Initiatives’ effectiveness indicator for tracking trends in recurring problems also was not represented. In other cases, the Initiatives’ effectiveness indicators were represented in underlying lower-level indicators that had very little impact on the rating of the visible indicator. An example is the Initiatives’ indicator for timely completion of employee concerns. The panel’s related visible indicator was work environment, whose rating was based on 4 secondary and 23 tertiary indicators. Of the third-level indicators, two were for timeliness of completion of employee concerns, and combined they contributed 3 percent toward the rating of the work environment indicator. As a result of the weighting of these many underlying indicators, ratings for individual lower-level indicators could be different from the visible indicator. For example, in August 2005, the work environment indicator showed good performance. However, the ratings of four underlying indicators from the project’s employee survey on the work environment—collectively accounting for 25 percent of the work environment indicator’s score— indicated the need for increased management attention. Moreover, some of the Initiatives’ indicators, such as the work organizations’ self- identification of significant problems, had their impact on visible indicators diluted by the inclusion of other indicators that were not focused solely on the detection of significant problems. Another shortcoming of the annunciator panel was that frequent changes to the indicators hindered the ability to identify problems for management attention and track progress in resolving them. The indicators could change in many ways, such as changes in their definition, calculation, or data sources used in calculations, or from the deletion or addition of a subindicator. When such changes were made to the indicators, progress became less clear because changes in reported performance levels may have been the result of the indicator changes rather than actual performance changes. Some of the indicators for key project processes with quality elements changed from one to five times during the 8-month period from April 2004 through November 2004. Even after the major revision of the panel in early 2005, most of the performance indicators tracking quality issues continued to change over the next 6 months—that is, from March 2005 through August 2005. As shown in table 2, only one of the five relevant indicators did not change during this period. One indicator was changed four times during the 6-month period, resulting in it being different in more months than it remained the same. Moreover, the panel was not always available to identify problems and track progress. The panel was not created for December 2004, January 2005, and February 2005 because it was undergoing a major revision. At that time, DOE told NRC that the performance indicators for the panel were revised to reflect the change in the work as the project moved into the engineering, procurement, and construction phase. DOE also reduced the total number of visible indicators from 60 to 30 to focus on fewer, more critical aspects of project management. Panels with the new indicators were then produced for 6 months, starting with March 2005 and ending after August 2005. This second interruption of the panels resulted from another major revision to the indicators; this time, indicators are being made congruent with project work as designated by DOE’s “new path forward,” again to focus on fewer, more important activities. In December 2005, a senior DOE official told us that the project would begin to measure key activities, but without use of the panel. According to DOE, some of the Initiatives’ areas of concern and their associated effectiveness indicators—for example, trends in quality problems related to roles and responsibilities—were being captured, at least partially, in the project’s quarterly trend evaluation reports rather than in the performance indicators. However, the trend reports are a management tool designed more to identify emerging and unanticipated problems than to monitor progress with already identified problems, such as those addressed by the Initiatives. In developing these reports, trend analysts seek to identify patterns and trends in condition reports (CR), which document problematic conditions through the project’s Corrective Action Program. The trend reports analyze CRs for more significant problems (Levels A and B) and minor problems (Level C), but not at Level D (opportunities for improvement). The trend analysis typically separates the reported problems into categories such as organizational unit, type of problem, and cause. These categories are intended to provide insights into the problems. For example, analysis might reveal that most occurrences of a particular type of problem are associated with a certain organization. In practice, DOE missed opportunities to use trend reports to call attention to progress in the Initiatives’ areas of concern. For example, the Initiatives sought to clarify roles and responsibilities within and between DOE and BSC to ensure clear accountability for project results during the project’s transition from scientific studies to the design and engineering activities necessary to license a repository. Similar organizational transition problems were identified in the November 2004 trend report. While that report attributed increases in the number of causal factors associated with change management, supervisory methods, and work organization to recent BSC reorganizations and changes in the project from science-based to design and engineering activities, it did not specifically mention issues of roles and responsibilities or that roles and responsibilities was an Initiatives’ area of concern. However, an analysis of the cause of the problems noted in various significant condition reports, which is performed for certain condition reports and outside of the process of developing trend reports, found evidence of weaknesses in the organizational interfaces among BSC organizations, as well as between BSC and DOE. According to this cause analysis, these organizational interface weaknesses were associated with some manner of change and represented weaknesses in the definition of roles and responsibilities. Trend reports are generally based on condition reports, and problems with roles and responsibilities seem to be identified in cause analyses rather than in the condition reports themselves. Similarly, DOE missed an opportunity to use trend reports to discuss the Initiatives’ goal that the project’s line or work organizations become more accountable for self-identifying significant problems. The August 2005 trend report briefly cited an evaluation of a CR highlighting the low rate of self-identification of significant problems during the previous quarter and reported the evaluation’s conclusion that it was not a problem warranting management attention. However, the trend report did not mention that about 35 percent of significant problems were self-identified during the previous quarter, while the Initiatives’ goal was that 80 percent of significant problems would be self-identified. Thus, the trend report missed an opportunity to either raise a performance problem or pose the question of whether the Initiatives’ goal needed to be reassessed. Beyond whether they effectively tracked the Initiatives’ areas of concern, trend reports face important obstacles, in general, to adequately identify recurrent and significant problems: Recurring or similar conditions can be difficult to clearly identify for management’s attention and resolution. A trend report noted that there will be few cases where recurrent conditions are obvious because each condition slightly differs. Trend analysis tends to focus on the number of CRs issued, but the number of CRs does not necessarily reflect the significance of a problem. For example, the number of CRs involving requirements management decreased by over half from the first quarter to the second quarter of fiscal year 2005. However, this decrease was not a clear sign of progress. Not only did the number rise again in the third quarter, but the May 2005 trend report also noted that the number of all condition reports had dropped during the second quarter. According to the report, the volume of CRs in the first quarter had been high because of reviews of various areas, including requirements management. Another example is the records management problem. The November 2005 trend report stated that a records management problem identified in various CRs, despite accounting for about 50 percent of all business administration problems, reflected an underlying error rate of less than 1 percent and thus was not a significant problem. The lack of an increasing trend in the number of reported problems does not necessarily mean the lack of a significant problem for management attention. Knowing the appropriate level of performance, regardless of the trend, is difficult without having clearly appropriate benchmarks from organizations engaged in activities similar to the Yucca Mountain project. Such benchmarks would clarify, for example, whether a project’s percentages of human performance errors compare favorably, regardless of whether the numbers are increasing. Similarly, the trend in the number and types of CRs during any period is not necessarily a sign of improvement or worsening conditions. Trends can be attributed to various factors, including increases in the number of audits or self- assessments, which can lead to more CRs being issued. At the time of analysis, some trend data may not be sufficiently reliable or complete to ensure sound findings for management’s attention. For example, although some actions were taken in December 2004 to ensure that cause and other codes were properly assigned, a BSC audit in June 2005 again raised questions about the consistency of the coding. With respect to completeness, the fourth quarter report for 2005 noted that 28 percent of the Level B CRs did not have a cause code at the time of the trend analysis, and one finding was presented even though two-thirds of the data was missing. Due, in part, to these obstacles and changes to how the analysis is done, trend reports have not consistently determined the significance of problems or performed well in tracking progress in resolving problems. For example, trend reports have questionably identified significant human performance problems and ineffectively tracked progress in resolving the problem because of no clearly appropriate or precise benchmark for performance, inconsistent focus on the problem, and unreliable data on cause codes. The February 2004 trend report identified a human performance problem based on Yucca Mountain project data showing the project’s proportion of skill-based errors to all human performance errors was two times higher than benchmark data from the Institute of Nuclear Power Operations (INPO). The report used this comparison to suggest that the project needed to adopt successful commercial nuclear practices for addressing skill-based errors. However, the report cautioned that other comparisons with these INPO data may not be appropriate because of differences in the nature, complexity, and scope of work performed, but did not explain why the report’s comparison of INPO data for skill-based errors to the Yucca Mountain project should be an exception to this caution. The May 2004 trend report repeated this comparison to INPO, finding skill-based errors three times higher than the benchmark data. However, this INPO benchmark has not been used in subsequent reports. The November 2004 trend report redefined the problem as the predominance of human performance errors in general, rather than the skill-based component of these errors—but later reports reinterpreted this predominance as not a problem. The problem with skill-based errors was unclear in the November 2004 report because these errors were showing a decreasing trend, a finding that was attributed as likely the result of unreliable assignment of cause codes. Instead, the report cited an adverse trend based on the fact that the human performance cause category accounted for over half of the total number of causes for condition reports prepared during the quarter. Under the project’s trend analysis guidelines, this large predominance of human performance causes—in contrast to management, communication or procedure, and other cause categories— was designated an adverse trend. Nevertheless, by February 2005, trend reports began interpreting this predominance as generally appropriate, given the type of work done by the project. That is, the project’s work involves mainly human efforts and little equipment, while work at nuclear power plants involves more opportunities for errors caused by equipment. In our view, this interpretation that a predominance of human performance errors would be expected implies an imprecise benchmark for appropriate performance. Although trend reports continued to draw conclusions about human performance problems, the February 2005 report indicated that any conclusions were hard to justify because of data reliability problems with cause coding. For example, the majority of problems attributed to human performance causes are minor, or Level C, problems that receive less rigorous cause analysis, such as not completing a form. This less rigorous analysis tends to reveal only individual human errors—that is, human performance problems—whereas more rigorous analysis tends to reveal less immediately obvious problems with management and procedures. Trend reports have also inconsistently tracked progress in resolving the problem associated with the “flow-down” of requirements into the project’s procedures—that is, with ensuring that program, regulatory, and statutory requirements are identified, allocated, and assigned to the project organizations that are responsible for applicable activities. Such requirements management problems can result in inadequate control over design inputs and, possibly, inputs to scientific models. Progress with this problem was less clear because of inconsistent methods of categorizing requirements management problems over time. Initially, based on reviews of annual trends in condition reports, the September 2004 and November 2004 trend reports observed a systemic and continuing problem in the flow- down of requirements from BSC’s Project Requirements Document and identified this as an adverse trend. In subsequent reports, the requirements flow-down problem was variously treated as an aspect of requirements management or records management, or as a latent management weakness or weak change management. When treated as an aspect of these broader problems, the significance of the original flow-down problem and any progress in resolving it became diluted and less clear. The primary focus eventually became requirements management, which the February 2005 trend report designated as a potential trend, whereas the flow-down problem had earlier been designated an adverse trend. Consequently, as a result of this change, the flow-down of requirements got less direct attention and analysis—for example, receiving only a footnote in the August 2005 trend report stating that the April 2004 condition report issued to address the adverse trend was still overseeing implementation of corrective actions. In addition, because trend reports examine only condition reports issued to BSC, they do not always assess the projectwide significance of problems such as requirements management. When analyzing one category of issues associated with requirements management, the November 2005 report stated that BSC and DOE shared the process problems, which cannot be adequately addressed by just one of the organizations. However, for a second category of these issues, the report did not analyze most of the condition reports because 6 of the 10 relevant reports were assigned to DOE. For a third category of issues, no analysis or recommendation was provided because all of the reports were assigned to DOE and therefore did not fall within the scope of the trend report. The tracking of problems for which corrective actions are already being taken appeared at times to be overly influenced by judgments, rather than the problems’ significance, about whether additional management action is warranted. As a result, problems might be rated as less significant, or not tracked further. The situation of assigning a lower rating to a problem’s significance was apparently caused by the fact that ratings were simultaneously an assessment of a problem’s significance and of the need for management action. In its current formulation, DOE’s rating categories cannot accurately represent both the assessment of a problem’s significance and a judgment that additional actions are not needed because the designated rating category will distort one or the other. For instance, the November 2005 trend report analyzed the four categories of requirements management issues and designated one category that included problems with requirements flow-down as a “monitoring trend”—defined as a small perturbation in numbers that does not warrant action but needs to be monitored closely. Describing this trend as a small perturbation, or a disturbance in numbers, did not accurately describe the report’s simultaneous recognition that significant process problems spanned both BSC and DOE and the fact that the numbers and types of problems were consistently identified over the previous three quarters. A more understandable explanation for the low rating is that designating the problem at any higher level of significance would have triggered guidelines involving the issuance of a condition report, which, according to the judgment expressed in the report, was not needed. Specifically, the report indicated that existing condition reports have already identified and were evaluating and resolving the problem, thereby eliminating the need to issue a new condition report. By rating the problem at the lowest level of significance and not calling for additional actions, the trend report did not sufficiently draw management’s attention to the problem. The trend report’s assessment did not convey that other serious problems might have been raised by the additional condition reports. At about the same time that the trend report judged that no new condition reports were necessary, an Employee Concerns Program’s investigation of requirements management resulted in 14 new condition reports—3 at the highest level of significance and 8 at the second-highest level of significance. For example, the Employee Concerns Program’s investigation resulted in condition reports calling for an analysis of the collective significance of the numerous existing condition reports and an assessment of whether the quality assurance requirement for complete and prompt remedial action had been met. As a result of the investigation and a concurrent DOE root cause analysis, during the December 2005 Quarterly Management Meeting with NRC, DOE stated that strong actions were required to address the problems with its requirements management system and any resulting uncertainty about the adequacy of its design products. Trend reports identified significant problems in the February 2005 report but did not continue to track the problems after a separate analysis identified ongoing improvement actions. According to the trend report, Level B condition reports collectively indicated organizational weaknesses associated with change management involving cross-departmental interfaces. The trend report recommended that management focus on these problems, and cited a condition report that would further investigate them. The cause analysis for that condition report and a related condition report found that the problems were well-known, in part through a BSC review, and related to a variety of ongoing BSC improvement actions. Since this was a broad category of problems with many initiatives under way, the cause analysis recommended no new actions other than for management to remain aware of the problems. However, the trend reports that followed provided no further analyses to focus management’s awareness on these problems or to assess progress in resolving them. In October 2005, DOE announced an aggressive series of proposed changes to the design, organization, and management of the Yucca Mountain project, but this effort—known as the “new path forward”—will face substantial challenges. Some key challenges facing DOE are (1) determining the extent of problems and restoring confidence in the documents supporting the license application after the discovery of e-mails raising the potential of falsified records, (2) settling design issues and associated problems with requirements management, and (3) replacing key personnel and managing the transition of new managers and other organizational challenges. The current Acting Director of the Office of Civilian Radioactive Waste Management (OCRWM) stated that DOE will not announce a schedule for submitting a license application until DOE addresses these important quality assurance and other challenges. Since DOE is still formulating its plans, it is too early to determine whether the new path will resolve these challenges. In March 2005, after announcing the discovery of USGS e-mails suggesting the possible violation of quality assurance requirements, including the falsification of records, DOE has taken steps to address lingering concerns about the adequacy of the scientific work related to the flow of water into the repository and whether similar quality assurance problems are evident in other e-mails relevant to the licensing application. Specifically, DOE is (1) conducting an extensive review of approximately 14 million e-mails to determine whether these e-mails raise additional quality assurance concerns and whether they might be relevant to the licensing process, and (2) reworking the technical documents created by USGS personnel to ensure that the science underlying the conclusions on water infiltration are correct and supportable in the license application. The Acting Director of OCRWM has stated that DOE will not submit a license application until these efforts are complete. Consequently, given the early planning stage of these efforts, it is unknown how long this will delay the submission of a license application. As part of the licensing process, DOE is required to publicly disclose all documents relevant to the licensing application, including e-mails, by posting them on DOE’s public Web site, which is accessible through the NRC-sponsored, Internet-based Licensing Support Network (LSN). To satisfy schedule requirements, DOE must certify that relevant documents have been posted to the network and made available for public review 6 months before the submission of the license application. In preparation for submitting the license application by December 2004, in June of that year, DOE submitted almost 700,000 e-mails to the LSN that had been reviewed by their original authors and determined to be relevant to the licensing process. They were part of a group of approximately 6 million archived e- mails authored by individuals still associated with the project. However, in August 2004, NRC’s Atomic Safety and Licensing Board ruled that DOE had not met its regulatory obligation to make all relevant documentary material available. Specifically, DOE had not reviewed a group of approximately 4 million archived e-mails authored by individuals no longer affiliated with the project to determine whether the e-mails were relevant to the licensing process. As part of its effort to address the board’s ruling, BSC began a review of e-mails authored by employees who were not currently working at the project. During this review, the contractor discovered and brought forward e-mails between USGS scientists working on water infiltration models that raised questions of the potential falsification of technical information in order to sidestep quality assurance requirements. Following the discovery of the e-mails, DOE conducted a search to determine if there were similar e-mails in the approximately 1 million e- mails previously determined relevant for licensing. However, the DOE Inspector General reported in November 2005 that there was no evidence that the project requirements for identifying and addressing conditions adverse to quality, such as those contained in the USGS e-mails, were considered during the initial review of e-mails. Further, among the approximately 10 million e-mails that had already been reviewed for the licensing process, they found additional e-mails that identified possible conditions adverse to quality that had not been identified by project personnel as requiring further review. The DOE Inspector General recommended, among other things, that DOE (1) expand the review of archived e-mails to include both those deemed relevant and those deemed not relevant to the licensing process, and ensure that conditions adverse to quality are appropriately identified, investigated, reported, and resolved; and (2) ensure that current and future e-mails are reviewed for possible conditions adverse to quality and that such conditions are appropriately addressed under the Corrective Action Program (CAP) system. DOE accepted the Inspector General’s recommendations. Specifically, DOE agreed to develop a corrective action plan to expand the review of archived e-mails to ensure that conditions adverse to quality are appropriately identified and processed under the CAP system. In addition to this review, the DOE Inspector General opened a criminal investigation into the USGS e-mails in March 2005. As of December 2005, the investigation was still in progress. According to NRC on-site representatives, completing these e-mail reviews will be challenging because DOE now has to screen millions of e-mails to ensure that records were not falsified. Further, many of these e-mails were written by employees who no longer work at the project or may be deceased, making it difficult to learn their true meaning and context. Moreover, if additional e-mails are found that raise quality assurance concerns, DOE may have to initiate further review, inspections, or rework to address the newfound problems. NRC officials stated that it takes the issue of potentially falsified documents by USGS employees very seriously, wants a full understanding of the situation regarding the USGS e-mails, and will conduct follow-up in this area. Because NRC wants DOE to submit a high-quality license application, it has encouraged DOE to take the time and actions necessary to fully and adequately resolve these and other quality assurance issues. Immediately following the discovery of the USGS e-mails, DOE undertook a scientific investigation into the technical documents created by USGS personnel. In October 2005, DOE began developing an action plan for reviewing, validating, augmenting, and replacing USGS work products that had come under scrutiny. Although the plan is not yet complete, the Acting Director told us that the license application would not be submitted until the USGS work is replaced and there is confidence that all requirements have been met. In an effort to ensure that the scientific work underlying water infiltration modeling is accurate, DOE is working to corroborate the original work by engaging multiple agencies and organizations to rework the models. For example, DOE has (1) had its lead project contractor work with the Idaho National Laboratories to extensively review the software and data used in the original science work, (2) engaged Sandia National Laboratories to rework the model and calculations using different software than was used originally, and (3) also asked USGS to rework the models. Consequently, when this additional rework is completed, DOE will have four sets of analysis (including the original scientific work) with which they can evaluate, compare, and corroborate results. DOE will then pick one set of scientific analysis for inclusion in the license application, and work to explain and defend its choice. In October 2005, DOE announced significant changes to the design of the Yucca Mountain repository to simplify the project and improve its safety and operation. However, these changes will also require additional design and engineering work that will add uncertainty about the timing of the submission of a license application. DOE had been considering a design where radioactive waste would be shipped to the Yucca Mountain site, removed from its shipping container, placed and sealed in a special disposal container, and finally moved into the underground repository. As a result, DOE contemplated handling the waste up to four separate times. In late 2003, DOE engineers began identifying potential safety problems with this approach. First, possible fissures or holes in the cladding surrounding the spent nuclear fuel accidentally caused during the handling of the waste could cause air to mix with the fuel and oxidize. Consequently, this radioactive oxidized material could then leak and be dispersed into the air. Second, DOE engineers determined that the original facility design would not be able to adequately control the levels of radioactivity in the buildings where the waste would be repackaged before being moved in the repository. To address these problems, DOE researched a series of options, including only accepting radioactive waste that had already decayed to the point where oxidization would not be problematic, and testing the waste shipments for oxidization and treating them at another site before they arrived at the repository. In addition, DOE also considered changing the design by filling the processing buildings with inert gas to prevent oxidization and revising the electrical and ventilation systems. According to a DOE official, these options were impractical or added complexity to the design. However, in October 2005, DOE proposed a new design that relies on uniform canisters that would be filled and sealed before being shipped, eliminating the need for direct handling of the waste prior to being placed in the repository. As a result, DOE will not have to construct several extremely large buildings costing millions of dollars for handling radioactive waste. DOE believes this change will improve the safety, operation, and long-term performance of the repository. However, this change will also pose a challenge to the project because of the widespread implications and the unknown time and effort required to implement it. For example, to implement the new design, DOE will need to, among other things, get approval from the Energy Systems Acquisition Advisory Board for a new project plan, which, among other things, includes details on the conceptual design, cost estimates, risk management efforts, and acquisition strategies; plan, design, and produce standardized canisters for the transportation coordinate this new approach with commercial nuclear power plants, NRC, and government organizations that plan on shipping waste to the project; and revise procurement and contracting plans to support the new design. Finally, DOE will need to perform the detailed design and engineering work required to implement the new design, and create new technical documents to support the license application. However, before it can present its new plans and perform this design and engineering work, DOE officials have stated that it will need to resolve long-standing quality assurance problems involving requirements management. Requirements management is the process that ensures the broad plans and regulatory requirements affecting the project are tracked and incorporated into specific engineering details. According to DOE’s root cause analyses, low- level documents were appropriately updated and revised to reflect high- level design changes through fiscal year 1995. However, from 1995 through 2002, many of these design documents were not adequately maintained and updated to reflect current designs and requirements. Further, a document that is a major component of the project’s requirements management process was revised in July 2002, but has never been finalized or approved. Instead, the project envisioned a transition to a new requirements management system after the planned submission of the license application in December 2004. However, for various reasons, the license application was not submitted at that time, and the transition to a new requirements management system was never implemented. As a result, the document refers to the out-of-date NRC regulations contained in 10 CFR part 60, and not the regulations in 10 CFR part 63 that were finalized in October 2002. The scope and cause of requirements management problems have been identified in multiple DOE and NRC reviews. Multiple condition reports issued in 2004 and 2005 have identified problems with requirements management. Due to these condition reports and NRC concerns that repetitive deficiencies and the failure to implement timely corrective actions could have direct implications on the quality of the planned license application, NRC performed a review of Corrective Action Program documents related to the requirements management program in the late summer of 2005. NRC determined that these reports identified approximately 35 deficiencies related to requirements management. Because the requirements management documents are not current and the new requirements management system has not been implemented, NRC concluded that there does not appear to be a requirements management mechanism in place. Further, based on the number of reports and other issues identified by DOE audits, NRC concluded that the project’s Corrective Action Program was not effective in, among other things, eliminating the repeated identification of deficiencies relating to requirements management or initiating the actions to identify and appropriately address the root cause of these problems. In September 2005, DOE began reviewing the root causes associated with CR-6278, a condition report identifying problems with requirements management. As part of the review, DOE personnel analyzed 135 condition reports and other events and allegations. Among other things, this review found that DOE expectations for requirements management were diluted and eventually neglected, that DOE reduced funding for requirements management due to reductions in its annual budget, and that these and other events caused the requirements management process to become “completely dysfunctional” from July 2002 to the time of the review in the fall of 2005. The analysis identified the root causes of these conditions as DOE’s failure to fund, maintain, and rigidly apply a requirements management system. In November 2005, a team of DOE personnel concluded an investigation into an employee’s concerns regarding requirements management. The team substantiated all of the concerns they investigated and found instances of failures and breakdowns in the requirements management process. For example, among other things, the team found that no procedure was developed to describe how requirements management was to occur; some existing requirements management procedures were not implemented; and project management was aware of these conditions but corrective actions were deferred because the planned requirements management system was expected to address the problem. As a contributing factor, the team also observed that the project’s lead contractor had not implemented a “traditional systems engineering approach” as it did not have, among other things, typical engineering management plans or a separate systems engineering organization responsible for requirements management. As a result of the investigation, the team initiated 14 condition reports, 13 of which identified quality-related problems. To address these problems, on December 19, 2005, DOE issued a stop-work order on design and engineering for the surface facility and certain other technical work. DOE stated that the root cause analysis for CR 6278 and the investigation into employee concerns revealed that the project has not maintained or properly implemented its requirements management system, resulting in inadequacies in the design control process. This stop-work order will be in effect until, among other things, the project’s lead contractor improves the requirements management system; validates that processes exist and are being followed; and requirements are appropriately traced to implementing mechanisms and products. Further, DOE will establish a team to take other actions necessary to prevent inadequacies in requirements management and other management systems from recurring. An example of the potential risks of a breakdown with requirements management was noted during a BSC audit on the design process in March 2005. NRC on-site representatives observing this audit reported that the audit team noted problems with inconsistencies between the design documents of the planned fuel-handing facility that would be receiving, preparing, and packaging the waste before it is placed in the repository. The original set of requirements specified that no water from a fire protection system was to be used in the fuel-handling areas of the facility because under certain scenarios, water used for fire suppression could facilitate an accidental nuclear reaction, a condition known as criticality. Later, as the project began to review the design of the fuel-handling facility, the design was changed to allow the use of water sprinklers in the fuel- handling areas of the facility to suppress possible fires. NRC noted that personnel working on the design knew of the inconsistencies between older and newer design documents, but no formal tracking mechanism had been provided to ensure that those issues were rectified. According to an NRC on-site representative in December 2005, this was an example of a concern with requirements management, and that repetitive and uncorrected issues associated with the requirements management process could have direct implications on the quality of the license application. While the project may be able to resolve these inconsistencies through an informal process, the lack of a formal design control and requirements management process increases the risk that not all such problems will be addressed. These requirements management problems are potentially significant because if the high-level engineering needs of the project are not accurately or completely reflected in the detailed design, then the quality of the license application may be compromised and cause delays in the license application review process. For example, according to a 1989 speech prepared by NRC’s Office of General Counsel stressing the importance of quality assurance, a West Coast nuclear power plant experienced similar quality assurance problems with requirements management. After a license was issued by NRC, power plant personnel discovered that the wrong diagrams were used to develop design requirements. As a result of this and other quality assurance weaknesses identified by NRC, the license was suspended and the power plant was required to initiate an independent program to verify the correctness of the design. Further, NRC reopened hearings on the issue of the adequacy of the power plant’s quality assurance program related to the plant’s design. In October 2005, DOE announced a “new path forward” that would create a new project schedule and financial plan to address the completion of scientific and engineering work in support of a license application. However, DOE faces challenges to successfully implementing the new path, in terms of managing the transition, program and organizational complexities, and the continuity of management. According to DOE managers involved with planning the new path forward, the organizational transition could take several months to complete. It is too early to determine whether DOE’s new effort will resolve quality assurance issues and move the project forward to the submission of a license application. Accountability for quality and results, which was identified as a significant transition issue in the Initiatives, will likely pose a challenge for managing the transition to the new path forward. The Initiatives sought to clarify roles and responsibilities within and between DOE and contractor organizations to ensure clear accountability for results and quality during the transition from OCRWM’s organization, processes, procedures, and skills supporting scientific studies to those supporting the activities necessary to license a repository. As the project realigns organizations, processes, procedures, and skills to support the new path forward, it will also be faced with the challenge of ensuring that accountability is not undermined during the transition. For instance, according one DOE manager, transitioning project work to a lead laboratory under a direct contract with DOE could pose a significant challenge for quality assurance because the laboratories are currently working under BSC quality assurance procedures and will now have to develop their own procedures. Implicitly recognizing the importance of accountability issues, elements of the new path forward seek to address issues that can negatively affect quality assurance and project management in general. For instance, the new path includes plans for developing and transmitting requirements to USGS for the certification of scientific work. In addition, a senior project official told us that the lead laboratory would provide a single point of accountability that will enhance the quality of the science work. The Acting Director indicated that OCRWM’s management structure may have to be reorganized to have a single manager clearly accountable for each of the new path’s major tasks in science, engineering, and licensing. Moreover, the project is developing new performance indicators to allow the project to assess important activities under the new path forward. Outside of the new path, as the result of a September 2005 DOE Inspector General report on accountability problems with managing contract incentives, OCRWM agreed to develop a comprehensive corrective action plan to provide clearer and more objective performance standards in the BSC contract. Program complexity and other project characteristics are also likely to pose challenges to managing quality assurance. Based on its experience with licensing and regulating nuclear power plants, NRC observed in the mid-1980s that the Yucca Mountain project’s characteristics, such as a large and complicated program, increased the likelihood of major quality-related problems. Although the new path is intended to simplify design, licensing, and construction, the project remains a complicated program that seeks to both restore confidence in its scientific studies and pursue new design and engineering activities. As a result, the project has to manage quality assurance issues simultaneously in both areas. Moreover, the project involves a complicated organizational structure. The project will continue contracting work with BSC, USGS, and the Sandia National Laboratory, which involves working with organizations in various locations. In our 1988 report, we noted that the geographic distance between the various organizations may hamper OCRWM’s quality assurance communication and oversight objectives. The project also faces management challenges related to ensuring management continuity at the project, since DOE has experienced turnover in 9 of 17 key management positions since 2001. To ensure the right managers move the project forward to licensing, the project has a recruitment effort for replacing key departing managers. In the past year, the project has lost key managers through the departures of the director of Project Management and Engineering, the director of the License Application and Strategy, the director of Quality Assurance, and the contractor’s general manager. According to NRC on-site representatives in August and October 2005, management turnover is a concern for NRC because it would like to see continuity of qualified managers rather than a series of acting managers. Recruiting replacement managers can impact project continuity, and newly acting managers may not take full rein of project tasks. However, the Acting Director told us that the recruitment process is an opportunity to improve project managers and staff, but recruiting the right people is challenging for various reasons—for example, government salaries are less than those in industry, and employment clauses restrict subsequent employment in related industries. Finally, since new directors sometimes give new direction to the project, a critical issue for sustaining the current new path forward is continuity with OCRWM’s director. This position was occupied by three individuals between late 1999 and early 2005. The last OCRWM director assumed the position in April 2002, started the Management Improvement Initiatives in 2002, and left the position in February 2005. The current Acting Director began functioning in his position in the summer of 2005, and initiated the new path forward in October 2005. DOE is currently awaiting congressional confirmation of a nominee to take the director position. However, the Acting Director told us he expects that the new path forward will be sustained because it has been endorsed by the Secretary of Energy. DOE’s Yucca Mountain project has been wrestling with quality assurance problems for a long time. Now, after more than 20 years of project work, DOE is again faced with substantial quality assurance and other challenges to submit a fully defensible license application to NRC. Unless these challenges are effectively addressed, further delays on the project are likely. Furthermore, even as DOE faces new quality assurance challenges, it cannot be certain it has resolved past problems, largely because the department has not been well served by management tools—specifically, its performance indicators and trend evaluation reports—that have not effectively identified and tracked progress on significant and recurring problems. First, the management tools have provided limited coverage of the areas of concern identified in the Management Improvement Initiatives and thus have not enabled DOE managers to effectively monitor progress in these important areas. Second, the tools have often not reflected the full extent or significance of problems because their scope has been limited and not based on projectwide analysis. Third, the trend evaluation reports have, at times, not accurately characterized problems because reliable and complete data and appropriate performance benchmarks were not available at the time of analysis. Fourth, frequent changes in performance indicators and the way analysis is done have made it difficult to accurately identify trends over time. Fifth, the tools’ rating categories have sometimes been misleading as to the significance of problems because the ratings tend to be skewed by the fact that corrective actions were already being taken, without considering their effectiveness or considering the significance of the problem on its own terms. These shortcomings with the tools limit project managers’ ability to direct and oversee such a large and complex undertaking as constructing an underground repository for nuclear wastes. Further complicating DOE’s ability to manage the project are the vacancies in key managerial positions for the quality assurance program and elsewhere on the project. The tools become even more important for new managers who need to quickly understand project management issues. To improve the effectiveness of DOE’s efforts to monitor performance in key areas at the Yucca Mountain project, including quality assurance, we recommend that the Secretary of Energy direct the Director, Office of Civilian Radioactive Waste Management, to take the following five actions to strengthen the project’s management tools: Reassess the coverage that the management tools provide for the areas of concern identified in the Management Improvement Initiatives and ensure that performance in these important areas is effectively monitored, especially in light of the more recent condition reports and associated cause analyses, trend reports, and other reviews indicating continuing problems. Base future management tools, such as the trend evaluation reports, on projectwide analysis of problems, unless there are compelling reasons for a lesser scope. Establish quality guidelines for trend evaluation reports to ensure sound analysis when reporting problems for management’s attention. Such guidelines should address, among other things, having reliable and complete data and appropriate benchmarks. To the extent practicable, make analyses and indicators of performance consistent over time so that trends or progress can be accurately identified and, where changes to analyses or indicators are made for compelling reasons, provide a clear history of the changes and their impact on measuring progress. Focus the management tools’ rating categories on the significance of the monitored condition, not on a judgment of the need for management action. We provided DOE and NRC with draft copies of this report for their review and comment. In oral comments, DOE agreed with our recommendations and provided technical and editorial comments that we have incorporated in the report, as appropriate. We also incorporated, as appropriate, NRC’s oral editorial comments, which primarily served to clarify its role. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to interested congressional committees and Members of Congress, the Secretary of Energy, and the Chairman of the Nuclear Regulatory Commission. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or wellsj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. The objectives of this review were to determine (1) the history of the Yucca Mountain project’s quality assurance problems since the project’s start in the 1980s, (2) the Department of Energy’s (DOE) tracking of quality problems and progress implementing quality assurance requirements since our April 2004 report, and (3) challenges that DOE faces as it continues to address quality assurance issues within the project. In addition, we were asked to provide information about implementation of the project’s Employee Concerns Program and the types of concerns raised in recent years through the program. To determine the history of the project’s quality assurance problems, we reviewed our prior reports and those of DOE’s Office of the Inspector General concerning the Yucca Mountain project. We also reviewed internal DOE evaluations and audit reports written about the quality assurance program and Nuclear Regulatory Commission (NRC) reports and NRC- prepared summaries of NRC and DOE quarterly management meetings, technical exchange meetings, and quality assurance meetings dating to early 2004. In addition, we reviewed letters and communications between DOE and NRC regarding quality assurance from the NRC Web archives from the late 1980s. Furthermore, we reviewed plans for the Regulatory Integration Team (RIT) and subsequent correspondence between Bechtel/SAIC Company, LLC (BSC), DOE’s management contractor for the Yucca Mountain project, and DOE. Moreover, we discussed quality assurance issues with officials of DOE’s Office of Civilian Radioactive Waste Management (OCRWM), including the Acting Director and Deputy Director, at DOE headquarters in Washington, D.C., and at its field office in Las Vegas. In addition, we interviewed representatives of Navarro Quality Services, a DOE subcontractor, as well as BSC, and NRC officials in the agency’s field office in Las Vegas, Nevada, and at its headquarters in Rockville, Maryland. To determine DOE’s tracking of quality problems and progress implementing quality assurance requirements since our April 2004 report, we interviewed OCRWM, BSC, and NRC officials about the status of these efforts since the issuance of our prior report. We also reviewed DOE’s Management Improvement Initiatives (2002), DOE’s Management Improvement Initiatives Transition Approach (2003), and our 2004 report to understand the history of the improvement efforts. To understand DOE’s management tools to monitor problems and progress, we reviewed the available performance indicators panels from April 2004 through August 2005, when it was last produced; the documentation on the individual indicators applied to August 2005 data; and the quarterly trend reports from the fourth quarter of fiscal year 2003 through the fourth quarter of fiscal year 2005. We also reviewed information from condition reports and examined documentation on DOE’s Quality Assurance Requirements and Description (issued in August 2004), BSC’s Trend Evaluation and Reporting, and DOE’s Procedure: Condition Reporting and Resolution (issued in November 2005). To determine challenges that DOE faces as it continues to address quality assurance issues within the project, we reviewed information from condition reports, NRC on-site representative reports, DOE Inspector General reports, and an OCRWM Office of Concerns Program’s investigative report on past quality assurance problems and DOE’s efforts to address them. We obtained information on turnover in key management positions at DOE and BSC since 2000. In addition, we discussed with DOE and NRC officials DOE’s difficulties in addressing recurring quality assurance problems and the quality assurance implications of the Yucca Mountain project moving from the site characterization phase to design and licensing. Also, to better understand issues and challenges, we attended quarterly meetings held between DOE and NRC in Rockville in September 2005 and Las Vegas in December 2005. To identify recent employee concerns related to quality assurance, such as falsification of records and a safety-conscious work environment, as well as to identify the actions taken to address those concerns, we reviewed all concerns received by the OCRWM and BSC Employee Concerns Programs from January through November 2005. For the OCRWM program, we reviewed all employee concerns files to identify concerns related to quality assurance. For the BSC program, we first read summary descriptions of each concerns file, and reviewed the concerns files for only those we identified as related to quality assurance. We then conducted a content analysis of all concerns files that we reviewed. Next, our three team members reached consensus about the correct classification of a concern as a quality assurance problem, such as potential falsification of records. Finally, through a second review of concerns files, we verified our recorded information for those concerns that seemed to be important illustrations of problems. In addition, we also spot-checked a sample of OCRWM and BSC concerns received in 2005 to verify the accuracy of their placement in various concerns categories. We found that the concerns were generally categorized accurately. We performed our work from July 2005 through January 2006 in accordance with generally accepted government auditing standards. NRC expects licensees to establish a safety conscious work environment— that is, one in which (1) employees are encouraged to raise concerns either to their own management or to NRC without fear of retaliation and (2) employees’ concerns are resolved in a timely and appropriate manner according to their importance. NRC encourages but does not require licensees to establish employee concerns programs to help achieve such a work environment, and both DOE and BSC have established such programs. DOE’s Employee Concerns Program is currently operated under the requirements of DOE Order 442.1A, but the department, in anticipation of becoming a licensee, is in the process of establishing the program to meet NRC expectations. DOE and contractor employees at the Yucca Mountain project may raise concerns about quality, safety, or other work environment issues—such as harassment, intimidation, retaliation, and discrimination—through various means. Employees are encouraged to resolve concerns at the lowest possible level in the organization, in the following order: Use normal supervisory channels, such as by raising an issue to a manager for resolution. Initiate a condition report through the Corrective Action Program—a process in which any employee can formally identify a problem on the project, such as with policies, procedures, or the work environment, and have the issue investigated and, if necessary, fixed through corrective actions. Submit a concern via e-mail, telephone, or in person to one of the project’s two Employee Concerns Programs—a BSC program for BSC employees and other subcontractors and another run by DOE for either DOE or BSC employees. Contact NRC directly. The DOE and BSC concerns programs are intended to supplement rather than replace the resolution of problems through managers or the Corrective Action Program. DOE and BSC Employee Concerns Programs have each established a communication network to allow employees to register concerns. These networks include brochures and regular newsletters on the programs and numerous links to the program on the project’s intranet, where employees can obtain concerns forms. Both the DOE and BSC concerns programs of the Yucca Mountain project have four main steps: 1. Employees notify the concerns program staff about issues that they feel should be corrected, such as safety or health issues; harassment, intimidation, retaliation, or discrimination; concerns raised through the Corrective Action Program; and quality assurance problems. 2. The concerns program staff document and handle the concern in accordance with the requirements of DOE Order 442.1A. 3. The concerns program notifies the employees of the results of the investigation and notifies management of any deficiencies. 4. Project management develops corrective actions for deficiencies, and the program validates that the concerns have been effectively addressed by the actions. Under DOE Order 442.1A, concerns may be addressed through an investigation by the concerns program staff, an independent investigation, a referral, a transfer, or a dismissal of the concern. Employees can request or waive confidentiality. If a concern is submitted anonymously, interpreting the main issues and problems is left up to the concerns program staff, and action on the concern may be limited if the submitted information does not clearly or sufficiently define the concern. The concerns program may conduct its own investigation of the concern. Alternatively, it may refer the concern to another project organization for investigation or resolution. After the results of the investigation or resolution are reported to the concerns program within a specified period, the concerns program accepts the results or requires additional actions. In other cases, concerns may be transferred to another organization with the appropriate subject matter responsibility or expertise, such as the Office of Human Relations, Office of General Counsel, or Office of the Inspector General. After investigating a concern, the concerns programs determines whether the concern is substantiated, partially substantiated, unsubstantiated, or indeterminate. If a concern is substantiated or partially substantiated, the investigation results are presented to the responsible senior managers. A concern is considered indeterminate when evidence is insufficient to substantiate a concern or allow for a conclusion to be drawn. Some concerns can be resolved through a noninvestigative resolution, a method to address concerns promptly when minimal effort is required for resolution. Some resolutions involve the development of management corrective action plans that are tracked until they are closed. In addition, for deficiencies that identify systemic problems, the concerns programs may file a condition report through the Corrective Action Program. Moreover, DOE and contractor employees are required to report certain conditions or alleged conditions to DOE’s Office of the Inspector General under DOE Order 221.1, which covers waste, fraud, and abuse. The concerns program handles some employee concerns in this way. From January through November 2005, DOE’s concerns program opened 139 employee concerns for investigation, and the BSC concerns program opened 112 concerns for investigation. DOE’s concerns program places concerns into 14 categories, while the BSC program uses 20 categories. For both DOE and BSC, the category receiving by far the most concerns for calendar year 2005 was management: “management/mismanagement” for DOE and “management practices” for BSC. According to DOE, management concerns generally involved conditions related to management behavior, policy practice, budget allocation, or use of resources. According to the manager of BSC’s program, about half of the concerns in the management practices category involve hiring and human relations issues and the other half involve organizational policies and other issues. The “quality” category accounts for a relatively small portion of total concerns—18 percent of concerns for the DOE program and 4 percent for the BSC program. Tables 3 and 4 show the concerns received by the DOE and BSC programs for January through November 2005. The Employee Concerns Programs, which are designed to provide an alternative to raising issues through the Corrective Action Program and issuing condition reports, have been playing an active and sometimes key role in identifying and addressing quality assurance problems, as can be seen in the following examples: As part of an effort to identify e-mails relevant to the licensing process and that therefore should be included in the Licensing Support Network, BSC employees in late 2004 discovered e-mails suggesting potential falsification of technical records. The e-mails were submitted to the Employee Concerns Program in March 2005 and were eventually reported to the DOE Inspector General for investigation. The quality assurance issues raised by the e-mails have resulted in a substantial effort by DOE to restore confidence in the quality of technical documents that will support its license application to construct the repository. In mid-2005, the DOE concerns program referred to the project’s senior management an employee’s allegation that the project’s schedule was taking priority over quality in the review of technical documents. In this instance, the Office of Concerns Program Manager negotiated with senior management to address the time and resource needs for ensuring quality assurance, rather than simply communicating to the organization that quality should take priority over the schedule. As the result of an employee’s concerns referred to DOE by NRC in mid- 2005, the Employee Concerns Program initiated an extensive investigation of issues related to requirements management. That investigation substantiated the employee’s concerns and led to the issuance of 14 condition reports for problem resolution. Signifying the importance of this issue, DOE discussed problems with requirements management with NRC at their quarterly meeting in December 2005. The Employee Concerns Programs’ role in identifying and addressing quality assurance and other issues is dependent upon employees’ willingness to submit concerns, but the employees’ willingness has sometimes been in doubt. A late 2004 DOE survey of project employees indicated, for example, that less than two-thirds of employees were confident that submitted concerns would be thoroughly investigated and appropriately resolved. DOE recognizes the need to improve employee trust and willingness to use the concerns program, and both the DOE and BSC program are engaged in outreach efforts. However, employees’ willingness to submit concerns may be affected by factors outside the programs’ control. According to a DOE manager, the project’s recent and pending workforce reductions may account for a decreasing number of concerns submitted to the DOE program in late 2005. Based on OCRWM Employee Concerns Program data, the program averaged about 13 concerns a month from January through November 2005. However, the number of monthly concerns dropped to 5 in October and 3 in November 2005. During our review of concerns opened for investigation from January 2004 through November 2005, we did not identify any concerns alleging problems similar to the falsification of technical records suggested by the USGS e-mails. Although we found records of an early 2004 concern about an instance of inappropriate management of a technical document, this instance was resolved and did not appear to be an intentional or systematic effort to falsify records. The manager of the BSC program told us of a concern raised about another set of e-mails, but this concern was not about record falsification. The manager of the DOE program told us that she had not seen any reportable allegations of falsification of technical records since she took her position in July 2004. In addition to the contact named above, Raymond Smith (Assistant Director), Casey Brown, John Delicath, James Espinoza, and Terry Hanford made key contributions to this report.

Summary: The Department of Energy (DOE) is working to obtain a license from the Nuclear Regulatory Commission (NRC) to construct a nuclear waste repository at Yucca Mountain in Nevada. The project, which began in the 1980s, has been beset by delays. In a 2004 report, GAO raised concerns that persistent quality assurance problems could further delay the project. Then, in 2005, DOE announced the discovery of employee e-mails suggesting quality assurance problems, including possible falsification of records. Quality assurance, which establishes requirements for work to be performed under controlled conditions that ensure quality, is critical to making sure the project meets standards for protecting public health and the environment. GAO was asked to examine (1) the history of the project's quality assurance problems, (2) DOE's tracking of these problems and efforts to address them since GAO's 2004 report, and (3) challenges facing DOE as it continues to address quality assurance issues within the project. DOE has had a long history of quality assurance problems at the Yucca Mountain project. In the 1980s and 1990s, DOE had problems assuring NRC that it had developed adequate plans and procedures related to quality assurance. More recently, as it prepares to submit a license application for the repository to NRC, DOE has been relying on costly and time-consuming rework to resolve lingering quality assurance problems uncovered during audits and after-the-fact evaluations. DOE announced, in 2004, that it was making a commitment to continuous quality assurance improvement and that its efforts would be tracked by performance indicators that would enable it to assess progress and direct management attention as needed. However, GAO found that the project's performance indicators and other key management tools were not effective for this purpose. For example, the management tools did not target existing areas of concern and did not track progress in addressing them. The tools also had weaknesses in detecting and highlighting significant problems for management attention. DOE continues to face quality assurance and other challenges. First, DOE is engaged in extensive efforts to restore confidence in scientific documents because of the quality assurance problems suggested in the discovered e-mails between project employees, and it has about 14 million more project e-mails to review. Second, DOE faces quality assurance challenges in resolving design control problems associated with its requirements management process--the process for ensuring that high-level plans and regulatory requirements are incorporated into specific engineering details. Problems with the process led to the December 2005 suspension of certain project work. Third, DOE continues to be challenged to manage a complex program and organization. Significant personnel and project changes initiated in October 2005 create the potential for confusion over roles and responsibilities--a situation DOE found to contribute to quality assurance problems during an earlier transition.