NNSA carries out the department’s nuclear weapons research missions at three weapons laboratories—Lawrence Livermore, Los Alamos, and Sandia national laboratories. These three laboratories have primarily a science and technology mission, which focuses on maintaining the nation’s nuclear weapons stockpile; preventing nuclear proliferation; and furthering basic scientific knowledge in chemistry, structural biology, and mathematics. In addition to their primary mission, the three laboratories perform work for other federal agencies, such as supporting homeland security efforts, and they coordinate research efforts with DOE’s Office of Science national laboratories in areas such as climate change and nanotechnology. In support of these various missions, contractors at the laboratories may carry out major construction projects, as well as projects to clean up radioactive and hazardous wastes from decades of producing materials or components for nuclear weapons. NNSA relies heavily on contractors to carry out its work, making effective federal oversight crucial to accomplishing its missions. At each of the laboratories, about 100 NNSA staff at the site office have responsibility for overseeing the work performed under contract by thousands of contractor employees. The contractors, in turn, may subcontract out major portions of their work, especially in mission-support areas such as constructing and maintaining facilities. Although NNSA has no direct relationship with these subcontractors, it is ultimately responsible for ensuring that all work, whether done by the prime contractor or its subcontractors, is performed in a manner consistent with the contract, including with all requirements for nuclear and worker safety. NNSA’s contracts for the three laboratories generally provide for reimbursing contractors for allowable costs plus an additional fee. The total fee available to the contractors may include a base, or fixed, amount that is guaranteed and an “at-risk” amount that is tied to performance measures in the contract. To help strengthen accountability, the department established a new contract provision in 1999 that allows it to reduce the fee otherwise earned if a contractor does not meet certain environmental, safety, and health performance standards. DOE regulations and directives set forth requirements for ensuring that nuclear facilities are operated safely to protect workers and the public. NNSA’s primary approach to ensuring nuclear and worker safety is to incorporate these regulations and directives into contracts. These rules require contractors to develop and maintain documentation that (1) describes the work to be performed; (2) evaluates all potential hazards and accident conditions; (3) contains appropriate controls, including technical requirements, that will eliminate or minimize the risk of hazards; and (4) delineates procedures and practices for operating the facilities safely. This documentation is commonly referred to as the facility’s documented safety basis. In addition, DOE regulations require that radiation doses to workers at DOE facilities be maintained within prescribed limits. NNSA’s laboratories and facilities, with few exceptions, are not regulated by the Nuclear Regulatory Commission or by the Occupational Safety and Health Administration. Instead, DOE and NNSA provide internal oversight of the three weapons laboratories at several different levels. NNSA provides direct oversight of the laboratories and the contracts through its site offices. In addition, NNSA headquarters staff offices, such as the offices of Defense Programs and Nuclear Nonproliferation, provide funding and program direction to the site offices. DOE’s Office of Enforcement and Office of Independent Oversight and Performance Assurance (now called the Office of Environment, Safety and Health Evaluations) and NNSA’s Chief of Defense Nuclear Safety and Senior Advisor for Environmental Safety and Health also provide oversight of laboratory activities to ensure nuclear and worker safety. Finally, the Safety Board, an independent oversight organization created by Congress in 1988, provides advice and recommendations to the Secretary of Energy to help ensure adequate protection of public health and safety at all of the department’s defense nuclear facilities, including those at the three weapons laboratories. As part of its independent oversight, the Safety Board has full-time representatives at the Los Alamos and Lawrence Livermore laboratories to work with the NNSA site offices and to observe contractor work activities at the site’s nuclear facilities. From 2000 through 2007, the three NNSA weapons laboratories have been troubled by persistent safety problems, including accidents and violations of nuclear safety rules designed to protect laboratory employees and the public. Our review of nearly 100 internal and external safety reviews since 2000 found that factors contributing to safety problems stemmed largely from weaknesses in NNSA’s management of safety issues at the weapons laboratories—weaknesses that leave the laboratories vulnerable to continued, and potentially serious, safety problems. From 2000 through 2007, nearly 60 accidents or near misses—each serious enough to be investigated—have occurred at the three NNSA weapons laboratories. The accidents have included radiation exposures, inhalation of toxic vapors, electrical shocks, and injuries during construction projects or maintenance activities. Fortunately, no one has been killed, but many of these accidents have resulted in serious worker injuries or facility damage. (Appendix II lists the major accident investigations at the three weapons laboratories since 2000.) For example: In 2000, seven workers at a Los Alamos plutonium-processing and - handling facility received significant doses of radiation from plutonium released into the air from a faulty unit, known as a glovebox, that shields people working with radioactive materials. When plutonium is inhaled, it can damage cells or raise a person’s risk of getting cancer. In this incident, a technician was trying to determine why the glovebox system was not operating properly; seven other workers were in the room at the time. As the technician was working, a fitting in the system leaked plutonium into the air, setting off alarms. Although the eight workers left the room at once, at least four of them were exposed to radiological releases much higher than the allowable annual exposure limits set in regulation, raising their cancer risk. The workers were provided immediate treatment. An internal DOE accident investigation found a number of factors behind this accident—which, because of the number of workers involved and the potential radiological doses, ranked among the top 10 worst radiological intake accidents in 41 years of data gathering by DOE and its predecessor agencies. These contributing factors included inadequate design and configuration of the glovebox and its auxiliary systems, lack of communication between workgroups tasked with maintaining different parts of interconnected systems, weaknesses in the technician’s training, and informal operations in the plutonium-handling facility. Moreover, according to the investigation report, the Los Alamos Laboratory had apparently failed to apply lessons learned from previous contamination releases in the same facility—including a similar event 2 years before, involving the same glovebox and some of the same people. In 2002, at another Los Alamos unit, liquid chlorine dioxide formed unexpectedly during an experiment and then exploded, sending debris into the air with enough force to destroy the fume hood where the experiment was taking place and to knock out pieces of wall, ceiling, and concrete. One of the two researchers present during the experiment noticed a rapid rise in temperature in the experimental apparatus, and both researchers fled the room seconds before the explosion, thus averting serious injury or death. According to an independent investigation of this accident, the experiment was changed to use 100 percent chlorine gas instead of 4 percent chlorine gas, a change that warranted a formally changed hazard control plan; yet only informal evaluations, without adequate analyses, review, or authorization, were done. As a result, the researchers failed to recognize the potential for formation of liquid chlorine dioxide and carried out the altered experiment inside a vessel that could not withstand the high pressure of the unanticipated liquid chlorine dioxide. According to the investigation report, this accident represents a case in which division management, line management, and workers had not adequately evaluated or ensured implementation of existing safety requirements. In 2003, an accident at a construction site on the New Mexico campus of Sandia National Laboratories seriously injured two ironworkers who were part of a crew of three installing a steel stairway in one building’s open stairwell. As the crew was hoisting and positioning a stair section near the top of the stairwell, a temporary hoisting beam slipped and fell; it struck the first worker’s hardhat on its way down and crushed his foot before hitting the ground. Other parts of the hoisting apparatus also collapsed, cutting another worker’s shin and knocking over a third worker. The first worker was hospitalized for a week; the second worker required six stitches to close the wound on his leg; the third worker escaped injury. The accident investigation report stated that neither the installation of the temporary hoisting beam nor the lifting of the stair section conformed to safety requirements. The report further noted that lack of clarity in safety requirements and poor communication between NNSA’s Sandia site office, project management, and subcontractors contributed to this preventable accident. Since 2000, Los Alamos and Lawrence Livermore have also been cited a total of eight times for violating nuclear safety rules. These rules are intended to protect workers and the public from nuclear hazards, including unintended nuclear explosions and radiation exposure, and under federal law, DOE has the authority to impose fines, or civil penalties, on contractors that violate them. In general, the rules (1) require analyses of work to be performed in a nuclear facility so as to identify potential hazards and operate the facility at an acceptably low level of risk and (2) spell out controls needed to ensure the safety of workers and the public. The eight citations levied since 2000 against Los Alamos and Lawrence Livermore laboratories have carried total penalties of nearly $4 million for violations of a number of nuclear safety requirements by, for example: failing to test safety equipment, such as fire-alarm systems, before beginning work to ensure proper operation; failing to follow protective procedures for handling radioactive materials; failing to label areas that contained high levels of radiation; illegally storing radioactive waste in a facility that lacked proper operating documentation; and failing to maintain proper documentation for the safe operation of nuclear facilities. In addition to accidents serious enough to warrant formal investigation and violations of nuclear safety rules, the three laboratories have experienced a number of less serious accidents and near misses. For example, from 2004 to mid-2007, the three laboratories have reported 97 worker injuries serious enough to require off-site medical attention and more than 150 electrical and mechanical near-miss incidents where serious injury could have occurred. Other reviews have also raised concerns about safety at the laboratories. In 2004, for instance, DOE’s Office of Independent Oversight and Performance Assurance and the Safety Board both raised concerns about safety management at Lawrence Livermore Laboratory’s plutonium-handling facility, including concerns over the adequacy of fire-suppression and ventilation systems in case of an accident. At both Los Alamos and Lawrence Livermore laboratories, such persistent safety problems (combined with concerns about security at Los Alamos) ultimately resulted in the temporary closure, or stand-down, of certain of the laboratories’ facilities. On July 16, 2004, the director of Los Alamos Laboratory suspended all laboratory operations, except those specifically designated as critical, to address safety and security concerns. The ensuing 10-month shutdown cost taxpayers an estimated $121 million to $370 million in lost productivity. Similarly, on January 15, 2005, the director of Lawrence Livermore Laboratory suspended all programmatic work at the site’s plutonium-handling facility, largely because of numerous unresolved safety issues and failure to address these issues adequately. The facility did not return to full operation for 16 months. During the stand-downs, both laboratories conducted comprehensive investigations into the causes of the numerous safety and security problems and found hundreds of deficiencies in both areas, which ranged from muddled lines of authority to overly complex and unclear safety policies and procedures to inadequate documentation and training. Despite the stand-downs, however, all three laboratories have continued to experience accidents warranting formal investigation, as well as violations of nuclear safety rules. For example, of the nearly 60 accidents investigated at the three weapons laboratories since 2000, 15 of them have occurred since the stand-downs. In addition, Los Alamos and Lawrence Livermore have both been cited for nuclear safety violations since the stand-downs were declared. Accidents included the following: In 2005, a worker at Los Alamos received and opened a package containing radioactive material delivered from another Los Alamos site and unknowingly contaminated himself, his clothing, and things he later touched; the contamination was not detected for 11 days. The shippers assumed the receiver would know that radiological contamination was possible and would act accordingly, and they did not test the package for contamination before shipping. The receiver, in contrast, assumed the package was uncontaminated because he had not been alerted otherwise. When the worker left that day, he was not screened for potential contamination because the room he was working in was not designated as a radiological control area. Over the next days and weeks, the worker unwittingly spread contamination to his home, to relatives’ homes in Kansas and Colorado, and to other locales at Los Alamos. In addition, he handled some otherwise nonradioactive parts, which also became contaminated and were shipped to Pennsylvania. The officials investigating this accident found a number of failures to follow safety procedures, unverified assumptions, and undocumented requirements; according to their report, “all of the accident’s causal factors were well established” before the accident. In 2006, an electrician working alone on a project to replace rooftop air conditioners at Lawrence Livermore missed a step while climbing a ladder mounted on the building. The worker fell and sustained multiple fractures of his wrist, shoulder, and pelvis, along with other injuries; he was hospitalized for nearly a month. The officials investigating this accident explicitly stressed the “significance of this seemingly simple accident—a worker slipped and fell from a ladder” because workers frequently climb similar ladders, the potential consequences of a fall are serious, and remedies—from ladder design to worker training—are straightforward and easy to put in place. Other safety problems have also occurred since the stand-downs. For example, the three laboratories have reported 33 electrical shock incidents since 2005. In one case at Los Alamos, two employees operating a generator-powered winch received electrical shocks on multiple occasions over a 4-day period without stopping work to report the shocks; the winch then malfunctioned, and the employees reported the shocks. At Sandia, a subcontractor employee received an electrical shock requiring the attention of paramedics after touching a “hot” screw on a 120-volt receptacle he was testing. In our review of nearly 100 safety studies—including accident investigations and independent assessments by the Safety Board and others since 2000—we found that factors contributing to safety problems stemmed largely from weaknesses in how NNSA manages safety at the weapons laboratories. These contributing factors generally fall into three key areas: A relatively lax attitude toward safety procedures. Accident investigations and other reviews of the weapons laboratories have repeatedly found an informal or lax attitude toward safety. Specifically, reviews have cited weaknesses such as (1) laboratory management that does not consistently and effectively emphasize the importance of working safely and following prescribed safety procedures, (2) employees who rely on their own expertise and knowledge of work hazards rather than following safety procedures, and (3) subcontractors who understand and implement safety procedures inadequately. The Safety Board and others have cautioned that such lax safety attitudes—including employees’ reluctance to question potential safety problems or inadequate leadership insistence on safety—create an environment where workers become complacent, and accidents occur. At the Los Alamos plutonium-handling facility, multiple accidental releases of airborne plutonium since 1996— including the 2000 incident involving seven workers and another one in 2003 involving the same group of employees in the same facility—led the investigators of the 2003 accident to conclude that “the organizational safety culture has evolved to one of complacency towards safety such that workers and managers fail to respect the hazards present in the workplace, and risks to workers are accepted without understanding the magnitude of those risks.” Study after study has highlighted the informality of laboratory operations and the lack of emphasis on safety throughout, from division management levels to individual worker levels. As a result of lax attitudes over the years, the laboratories have repeatedly failed to prevent what many reports and reviews regard as preventable accidents and near misses. Weaknesses in identifying safety problems and taking appropriate corrective actions. Fundamental weaknesses in the laboratories’ ability to accurately identify and fully understand safety problems and take appropriate corrective actions have hindered safety performance. Many reviews have cited (1) an inability to learn from past incidents, (2) a lack of rigorous self-assessments by the laboratories to identify problems, and (3) a failure to develop appropriate or timely corrective actions to mitigate these problems as factors contributing to recurring accidents. Several investigations stressed that accidents could have been prevented had lessons from previous accidents been learned and properly applied. Inadequate oversight by NNSA site offices. Many reviews have pointed out continuing deficiencies in federal oversight of the laboratories, including that oversight was insufficiently formal or documented (for example, that roles and responsibilities for safety were not clearly and consistently delineated). Such weaknesses have been exacerbated by staff shortages at the site offices, specifically, (1) unfilled positions resulting in too few staff available to serve as NNSA’s eyes and ears at the laboratories and (2) shortages in staff with adequate technical expertise. For example, positions for critical senior nuclear safety officials at both the Lawrence Livermore and Los Alamos site offices went unfilled for more than a year. These safety evaluations have repeatedly indicated that key management weaknesses have contributed to the laboratories’ continuing safety problems and that accidents could have been prevented had weaknesses been properly addressed. Together, these safety evaluations indicate that unless corrected, the weaknesses create conditions that leave the laboratories vulnerable to continued—and potentially more serious— safety problems. Steps taken by NNSA and its contractors include on-site efforts to address weaknesses in three key areas, as well as mechanisms to hold contractors more accountable for safety. NNSA and its contractors have been taking steps intended to address weaknesses in three key areas: safety culture, systems for identifying and correcting safety problems, and federal oversight: Safety culture. Since at least 2006, NNSA and its contractors have been taking steps to reinvigorate NNSA’s key safety improvement effort, called integrated safety management. Launched in 1996, integrated safety management was designed to respond to concerns raised by the Safety Board about the lack of formal, standardized procedures throughout DOE for ensuring that hazardous activities were carried out safely. The effort was intended to raise safety awareness and provide a formal process for employees to integrate safety into work activities by requiring employees to (1) define the scope of work, (2) analyze the hazards associated with that scope of work, (3) develop and implement hazard controls to address possible safety issues, (4) perform work within those controls, and (5) provide a feedback system for continuing to improve safety. This program aims to instill in every individual at the laboratories a sense of responsibility for working safely. Despite the program’s longevity and the soundness of the concepts behind integrated safety management, many safety reviews have stated that the program has not been fully or successfully implemented. In the decade since it began, NNSA and laboratory contractors have developed policies and procedures under program guidelines, but the laboratories have been unable to ensure that managers and employees consistently follow these policies and procedures in their work. Many of the accident and other reports we examined specifically cited ineffective implementation of integrated safety management at NNSA’s laboratories as a key factor contributing to the accidents. To remedy these recognized shortcomings, NNSA is revising its guidance to clarify integrated safety management requirements, and the laboratories have been taking various steps to reemphasize the principles of integrated safety management. First, according to laboratory officials, the laboratory directors have publicly stressed safety by, for example, at Sandia making unannounced monthly visits to different laboratory units to observe operations firsthand. At Lawrence Livermore, the laboratory director holds monthly performance reviews requiring his associate directors to report on specific safety metrics for their division. Second, several hundred managers and employees at all three laboratories have undergone training on why accidents happen and how to prevent them, in part through better communication and teamwork. In addition, Los Alamos and Sandia site office officials told us, the laboratories have been hiring staff from contractors at other DOE sites or from other programs where adherence to safety procedures has been more consistent, a move they believe will help shift the safety culture at the laboratories. Identifying safety problems and taking corrective actions. NNSA and its contractors at the laboratories have been taking steps to better identify safety problems and appropriate corrective actions. For example, Lawrence Livermore has created a new process in which teams of workers and managers annually review and assess implementation of work practices to identify deficiencies in safety procedures or other opportunities to improve safety. Sandia has also begun to standardize its annual self-assessment process for identifying safety and other problems, although officials told us that the new approach cannot yet provide consistent and useful information across laboratory divisions. Two laboratories (Los Alamos and Sandia) have also created new processes and computer systems for managing safety issues. Previously, reviews found deficiencies in processes for assigning and tracing accountability for safety problems. Both laboratories now have management boards that review identified safety issues and assign responsibility for those issues to individual managers, who must analyze and address the identified problems. To complete the process, individual managers must sign off on the fixes they have directed and either have the issue re-reviewed by the assigning board or have independent verification that the corrective action was completed. Further, the laboratories and site offices have begun using new software systems to help them track safety issues. Specifically, Los Alamos and Sandia have been improving their electronic management systems for tracking safety deficiencies and associated corrective actions, and two of the site offices, at Sandia and Lawrence Livermore, are using new integrated software systems intended to help the site offices track safety issues at the labs and document oversight efforts. Previously, this information was stored in multiple systems across the labs and site offices, which made it more difficult to track overall safety efforts. NNSA site office oversight. The site offices have initiated efforts to address concerns about inadequate federal oversight by instituting more- formal oversight procedures, seeking to fill vacant positions, and providing additional training. The site offices at Lawrence Livermore and Sandia have revised their operating procedures and documentation on staff responsibilities, qualifications of technical staff, and schedules for evaluating laboratory operations. Los Alamos and Lawrence Livermore site officials told us they have begun to fill vacant positions, including hiring a senior nuclear safety expert at Lawrence Livermore who directly advises the site office manager. The site offices told us that they have also been formalizing their process to provide training related to general scientific and technical expertise, applicable regulations, contract administration, and safety management. Many of these efforts are still under way, however, and their effect on safety performance is not clear. To hold its contractors more accountable for safety performance, NNSA has incorporated into its contracts at Los Alamos and Sandia new contract mechanisms that provide for additional fee or contract extensions for meeting annual safety and other goals. Under the new contract incentives, contractors can earn substantially larger fees—or, in the case of Sandia, a one-year contract extension as well—if they improve safety performance. At Los Alamos, incentive fees are offered for improving safety documentation and decreasing rates of illness and injury, for example. In 2005, Sandia had the opportunity to earn the 1-year extension but could not do so, primarily because of safety problems. NNSA officials we spoke with were hopeful that these contract incentives would foster greater accountability but said that improvements could take years to achieve. Moreover, officials expressed concern that incentives to reduce accidents could actually lead to underreporting, rather than actual reductions in the number of accidents. NNSA also is expecting to hold contractors more accountable through a newly implemented regulation—referred to as the “851 rule”—that requires contractors to follow worker safety requirements and imposes penalties for violations. Promulgated in response to a 2002 congressional requirement, and similar to nuclear safety regulations, the worker safety regulation (effective as of February 2007) encourages contractors to report violations of worker safety requirements and provides for DOE’s Office of Enforcement to levy civil penalties carrying monetary fines up to $70,000 per day. As of September 2007, DOE had not yet levied any fines against its contractors. NNSA faces two principal challenges in its continuing efforts to improve safety at the nation’s weapons laboratories. First, the agency has no way to determine the effectiveness of its safety improvement efforts, in part because those efforts rarely incorporate outcome-based performance measures. Second, concerns have arisen over the agency’s shift in its oversight approach to rely more heavily on contractors’ own safety management controls. NNSA does not have effective outcome-based performance measures that would enable it to evaluate the impact of individual improvement initiatives on safety performance. When asked what impact integrated safety management has had on safety performance, for example, NNSA and contractor officials at the laboratories described positive trends in measures such as illness and injury rates. However, in a December 2005 report reviewing NNSA’s implementation of integrated safety management at seven of its sites, including the weapons laboratories, the Safety Board noted that, although the illness and injury rates had been declining, the number of serious accidents, nuclear safety enforcement actions, and other safety occurrences had not declined. According to the Safety Board, this evidence indicated that the integrated safety management program had not reduced the number of serious safety problems, and the Safety Board suggested that NNSA develop a way to evaluate the effectiveness of integrated safety management. Yet effective performance measures were not included in the action plan to revitalize integrated safety management; rather, the planned actions were primarily process-oriented, such as developing new policies or manuals or providing additional training. The few measures that were included in the plan focused on, for example, defining annual performance measures in contracts or increasing the use of measures related to repeated incidents, with no indication of how these measures might help gauge effectiveness. We have found that NNSA and its contractors have not consistently managed safety improvement efforts using a disciplined approach incorporating substantive outcome measures and a system to evaluate its efforts’ effectiveness. Such an approach, often taken by high-performing organizations, generally includes four key elements: (1) defining clear goals, (2) developing an implementation strategy that sets milestones and establishes responsibility, (3) establishing results-oriented outcome measures to gauge progress toward the goals, and (4) using results- oriented data to evaluate the effectiveness of the effort and making additional changes where warranted. We have previously recommended that DOE develop and use this systematic approach in future improvement efforts. In response to this recommendation, the department issued a directive (DOE Notice 125.1) in October 2003 that adopted these principles. In February 2004, we reported on challenges at Los Alamos and Lawrence Livermore national laboratories, including problems with ensuring the safe operations of nuclear facilities and recommended that NNSA include in its contracts for the two laboratories a requirement that the contractors manage future improvement efforts in accordance with the October 2003 directive, to better ensure that its efforts are effective. Nevertheless, we found little indication that either NNSA or the contractors have been using the systematic approach specified by the October 2003 directive, and the approach in the directive has not been incorporated into the laboratory contracts. Rather than following the recommended systematic approach, the safety improvement efforts described by NNSA and its contractors echo previous attempts while continuing to lack useful measures of effectiveness. Specifically, the laboratory contractors have been providing managers and workers with additional safety training, but the contractors have not instituted any systems to evaluate whether the trainees have put into practice what they have been taught. In another example, the laboratories have changed their systems for tracking identified safety problems and the corrective actions taken to address those problems. But these systems still measure whether corrective actions were completed and completed on time, rather than how effective the actions were in addressing underlying weaknesses. What outcome-based measures do exist to evaluate safety performance—specifically, accident, illness, and injury rates—consider just part of the safety situation and do not address underlying management weaknesses that allow these incidents to recur. In short, NNSA has no objective way of determining whether improvement efforts are effective, whether these efforts will correct long-standing safety problems, or whether reduced accident rates are merely coincidental. Without stronger performance measures, NNSA and its contractors have no way of knowing whether the time and money invested in their improvement initiatives have actually resulted in safer laboratories. Given the persistent nature of safety problems at the laboratories, it appears that either the identification of the underlying causes or the corrective actions taken have been inadequate. A crucial step in the October 2003 directive is to fully understand problems and their underlying causes so that corrective actions will be effective. Yet over the past decade, NNSA and laboratory contractors have developed corrective action plans that were essentially reactive—responding to findings and recommendations from one or another internal or external report— without consistently taking the next step of identifying deeper, systemic weaknesses and taking steps to mitigate these weaknesses. Moreover, for at least two of the weapons laboratories, neither the safety problems nor the efforts to correct them are new. As we reported in February 2004, NNSA had put into place contract mechanisms and requirements to address known problems in areas including nuclear safety. Although this effort was intended to strengthen management and federal oversight of nuclear safety at Los Alamos and Lawrence Livermore, most of the measures included in the contracts were aimed at establishing processes or developing plans. In reports issued as recently as 2007, the Safety Board and others have again raised similar concerns about safety management weaknesses and suggested that NNSA and its contractors have not fully understood the safety problems or their underlying causes and have not identified and implemented the appropriate corrective actions. It thus appears likely that agency efforts will continue to be disjointed, and incidents and vulnerabilities could continue. NNSA has revised its laboratory contractor oversight policy to rely more on the contractors’ own systems of management controls to identify and correct safety problems. We and others have expressed concerns in the past, however, about these changes to its oversight policy and the increased emphasis and reliance on the contractors’ systems of management controls. In its draft policy of August 2003, NNSA proposed relying more on contractor assurance systems and self-assessments to identify and correct problems in all areas of operations, including safety. NNSA would then use a risk-based approach to its oversight, tailoring the extent of federal oversight to the quality and completeness of the contractor’s assurance system and the extent to which NNSA could rely on the contractor’s system to identify and correct problems effectively. In our February 2004 report, we acknowledged the potential benefits of a risk- based approach to federal oversight, but we also raised concerns about NNSA’s ability to effectively carry out this approach while successfully meeting its responsibility for safety. Furthermore, we recommended that NNSA retain sufficient independent federal oversight of contractors’ activities to fulfill its responsibilities for protecting workers, the public, and the environment. In addition, the Safety Board, in a series of public meetings in late 2003 and early 2004, expressed concerns about NNSA’s proposed oversight policy and stressed that NNSA should not delegate responsibility for the inherently high-risk area of operations at its nuclear facilities. The Safety Board was concerned about both the adequacy and the quality of federal oversight; it was also concerned that the contractors’ systems of management controls had yet to be proven effective. In response to these concerns, NNSA revised its oversight policy to outline how contractors’ systems of management controls, federal line management oversight, and independent reviews would work together to ensure effective operations, including safety. To specifically address the Safety Board’s concerns about high-hazard operations, the revised policy requires additional, and more rigorous, federal oversight of nuclear facilities and other high-hazard operations. The departmentwide oversight policy’s stated objective is to ensure that contractor assurance systems and federal oversight programs are comprehensive and integrated for all aspects of operations essential to mission success. According to the policy, contractor assurance systems should identify and address program and performance deficiencies and opportunities for improvement, provide the means and requirements to report deficiencies, establish and effectively implement corrective and preventive actions, and share lessons learned across all aspects of operations. Regardless, NNSA lacks a cohesive implementation strategy for how it will maintain appropriate levels of oversight of its laboratory contractors’ safety performance under this revised policy. At the site offices, oversight consists of a collection of activities, such as observations of work activities and reviews of contractor data, but it is not clear how these activities will fit into NNSA’s overall oversight structure. Furthermore, the NNSA site offices lack their own clear goals for improving oversight but instead equate improved oversight to ensuring that the contractors meet contract goals and annual performance measures. For example, at the Lawrence Livermore site office, the goals and outcomes for fiscal year 2007 included (1) ensuring that the contractor completes all required nuclear facility safety documentation, (2) ensuring that lessons learned from the 2005 plutonium facility stand-down are implemented at the laboratory’s other nuclear facilities, and (3) ensuring that the contractor implements the new worker safety rule. Yet these three goals and outcomes are still geared more toward process rather than safety improvements and are generally activities that the contractor should already have completed. Specifically, one of these goals—ensuring that the contractor completes all required nuclear facility safety documentation to identify the potential nuclear hazards and mitigation plans to protect workers, the public, and the environment—has been part of Lawrence Livermore’s contract requirements since 2001. Yet the laboratory had nuclear facilities that lacked completed safety documentation until September 2007. Furthermore, no clear criteria or results-oriented outcome measures exist for determining when a contractor assurance system is mature and reliable enough for NNSA to depend on the contractor for identifying and correcting safety problems. Without such measures, NNSA has no assurance that contractors can and will effectively identify and correct safety problems. In line with what we reported 3 years ago, we continue to believe it is premature for NNSA to rely so heavily on the contractors to maintain laboratory safety. Given the perennial safety problems at the laboratories, coupled with NNSA’s and contractors’ continued inability to clearly demonstrate progress in remedying weaknesses, it is unclear how this revised system will enable NNSA to maintain an appropriate level of oversight of safety performance at the weapons laboratories. The NNSA weapons laboratories, which conduct important but potentially dangerous research, have experienced persistent safety problems despite years of effort to make the laboratories safer. Although dozens of reviews since 2000 have repeatedly highlighted significant safety management problems at the laboratories, and NNSA and contractors have been taking steps aimed at improving safety, many of the steps appear to be revision or repackaging of earlier efforts, with few new approaches to correcting underlying problems. A key shortcoming may be that—despite a DOE- wide directive requiring that improvement initiatives include results- oriented outcome measures—neither NNSA nor its contractors have developed performance measures suitable for assessing the effect, if any, of safety improvement efforts on identified safety weaknesses. As a result, NNSA has no assurance that the resources expended on safety improvement efforts will successfully remedy fundamental weaknesses or strengthen laboratory safety. Furthermore, we remain concerned about NNSA’s recent shift to relying more on contractors to police themselves at a time when the laboratories remain vulnerable to safety problems, including accidents. We and others have raised concerns that although effective oversight of laboratory safety requires a strong, qualified federal presence, federal oversight remains problematic. Until contractors at the weapons laboratories can demonstrate improved safety performance, and until their efforts to address underlying management weaknesses are effective, our misgivings remain about NNSA’s ability to maintain effective independent oversight. As the responsible owner of these weapons laboratories, NNSA must be able to demonstrate that it is carrying out sufficient independent federal oversight of contractors’ activities to fulfill its responsibilities for protecting the health and safety of workers, the public, and the environment. Unless NNSA addresses these fundamental challenges and adopts a more structured and disciplined approach to improvement efforts and federal oversight, the weapons laboratories will continue to be vulnerable to safety problems and potentially serious consequences. To strengthen safety management and oversight at the nation’s weapons laboratories, we recommend that the Secretary of Energy direct the Administrator of NNSA to take the following four actions: Ensure that safety improvement initiatives comply with DOE Notice 125.1, which requires, in particular, that improvement initiatives be carried out in a systematic manner, with effective performance measures based on outcomes, not process. Negotiate with the weapons laboratories to include in their contracts a requirement that safety improvement initiatives be managed in a manner consistent with the best practices of high-performing organizations, as defined in accordance with the framework established in DOE Notice 125.1. Ensure that as NNSA implements its proposed oversight and contractor assurance policy, it develops a clear plan and specific measures that enable it to (1) determine when a contractor’s assurance system is sufficiently mature and reliable to identify and address safety problems at the weapons laboratories effectively and (2) retain sufficient independent federal oversight of safety to fulfill its responsibilities associated with protecting workers, the public, and the environment. Report annually to Congress on progress toward making the weapons laboratories safer, including the status and effectiveness of safety improvement initiatives, using outcome-based performance measures. We provided a draft of this report to NNSA for its review and comment. In written comments, NNSA’s Associate Administrator for Management and Administration stated that NNSA generally agreed with the report and its recommendations. NNSA’s written comments on our draft report are included in appendix IV. While generally agreeing with the facts in our report and its corresponding recommendations, NNSA sought to provide additional context in which our findings could be viewed. Specifically, NNSA stated that it believes that, given the numbers of employees, the period of years covered, and the high-hazard work that is performed at these laboratories, safety at the laboratories has been impressive. NNSA suggests that the ladder incident we describe does not exemplify lax safety. Yet this view overlooks the fact that even an accident as simple as a ladder fall can result in serious personal injuries. NNSA’s own accident investigation report stressed the “significance of this seemingly simple accident” because the consequences can be serious, and the remedies are relatively easy to put into place. Furthermore, we cited a number of examples illustrating the range and severity of accidents at the laboratories, including major radiation exposures. We remain concerned that such safety incidents are symptoms of more pervasive problems. NNSA also stated that, contrary to our criticism of its oversight of the weapons laboratories, it believes that oversight of safety at the laboratories is excellent. NNSA offers as evidence the existence of safety evaluations performed by its offices and other DOE offices. While we agree that reports by NNSA’s offices and other DOE offices are useful, our report focused on oversight at the NNSA site office level. In fact, NNSA’s safety evaluations themselves point out the same long-standing concerns about the adequacy of NNSA site office oversight. In addition, NNSA pointed to a decrease in its illness and injury rates at the weapons laboratories as evidence of the effectiveness of federal oversight of safety at the weapons laboratories. We acknowledge in the report that NNSA and its contractors described recent positive trends in safety measures such as illness and injury rates. We remain concerned about relying solely on this measure as evidence of improved safety performance because a number of factors could affect these rates, including instances of underreporting. In addition, illness and injury rates are not useful in indicating performance in nuclear safety, where a single accident can carry serious consequences not only for workers but for the public at large. Finally, as we stated in our report, a December 2005 Safety Board report noted that, although illness and injury rates had declined, the number of serious accidents and nuclear safety enforcement actions had not declined. We continue to believe that until NNSA adopts a more disciplined approach to improvement efforts and federal oversight—an approach that incorporates meaningful performance indicators—the laboratories cannot assure that safety improvement efforts have been effective or will be sustained. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the date of this letter. At that time, we will send copies of this report to the Secretary of Energy; the Administrator, NNSA; and appropriate congressional committees. We will also make copies available to others on request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. If you or your staff have any questions on this report, please contact me at (202) 512-3841 or Aloisee@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V. To determine the safety problems that have occurred at the weapons laboratories and contributing factors, we reviewed documents, including federal laws and regulations describing safety requirements for nuclear safety and for worker safety and health, Department of Energy (DOE) policies and procedures regarding safety management, and about 100 relevant reports issued since 2000 evaluating safety issues at the three weapons laboratories: Lawrence Livermore, Los Alamos, and Sandia national laboratories. Reports included inspections or reviews of the weapons laboratories by DOE’s Office of Independent Oversight and Performance Assurance, the National Nuclear Safety Administration’s (NNSA) Chief of Defense Nuclear Safety, the Defense Nuclear Facilities Safety Board (Safety Board), and our past reports. Reports also included all type A and type B accident investigation reports for the three weapons laboratories and, when possible, any contractor-led accident investigations. We also reviewed all enforcement actions for violations of nuclear safety rules taken against the laboratories by DOE’s Office of Enforcement. We reviewed the factors these reports identified as contributing to safety problems and categorized them into three key areas, using an analytical tool developed in consultation with our methodologist. We also analyzed safety performance data provided by DOE, specifically, safety incident data contained in DOE’s Occurrence Reporting and Processing System and Computerized Accident/Incident Reporting System. We determined that these data were sufficiently reliable for the purposes of this report. We discussed the safety problems and contributing factors, as well as our categorization of them, with representatives from DOE’s Office of Independent Oversight and Performance Assurance, NNSA’s Chief of Defense Nuclear Safety and Senior Advisor for Environmental Safety and Health, and the Safety Board, as well as with DOE and NNSA headquarters officials. In addition, we visited the three weapons laboratories and met with NNSA officials and contractors to discuss the factors we identified as contributing to safety problems at the laboratories. To identify the steps NNSA and its contractors have taken to improve safety management and address underlying management weaknesses, we reviewed agency documents, including implementation plans; laboratory contracts; and, to the extent it was available, documentation on safety improvement initiatives. We also interviewed officials at the three laboratories and NNSA site offices to discuss efforts taken by NNSA and the laboratories to improve safety and to more specifically address the areas of concern we identified. We also discussed efforts to improve safety performance with independent experts, including officials from DOE’s Office of Enforcement and representatives from the Safety Board. To determine the challenges that remain to effective management and oversight of safety performance at the weapons laboratories, we reviewed and analyzed relevant GAO reports on safety issues at the laboratories; recommendations made, if any; steps taken in response, if any; and issues remaining. We reviewed relevant DOE, NNSA, and contractor documents, including DOE policies and orders, site office strategies and plans, laboratory contracts, and annual evaluations of contractor performance. We also discussed challenges with DOE and NNSA officials; contractor officials; and independent experts, including officials from the Safety Board. We performed our work in accordance with generally accepted government auditing standards, which included an assessment of data reliability, from September 2006 through September 2007. Department of Energy, National Nuclear Safety Administration. Type B Accident Investigation of the Americium Contamination Accident at the Sigma Facility, Los Alamos National Laboratory, New Mexico, July 14, 2005. Washington, D.C.: January 2006. Department of Energy, Office of Security and Safety Performance Assurance. Inspection of Environment, Safety, and Health Programs at the Los Alamos National Laboratory. Washington, D.C.: November 2005. Department of Energy, National Nuclear Safety Administration. Type B Accident Investigation of the Acid Vapor Inhalation on June 7, 2005, in Technical Area 48, Building RC-1, Room 402, Los Alamos National Laboratory. Washington, D.C.: June 2005. Tarantino, Frederick A., et al. LANL Investigation of a Laser Eye Injury. LA-UR-04-6229. Los Alamos, New Mexico: Los Alamos National Laboratory, 2004. Department of Energy, National Nuclear Safety Administration. Type B Accident Investigation of the August 5, 2003, Plutonium-238 Multiple Uptake Event at the Plutonium Facility, Los Alamos National Laboratory, New Mexico. Washington, D.C.: December 2003. Hargis, Barbara, et al. Unanticipated Formation and Explosion of Liquid Chlorine Dioxide in a Parr Reaction. LA-CP-02-206. Los Alamos, New Mexico: Los Alamos National Laboratory, 2002. Department of Energy, Office of Security and Safety Performance Assurance. Inspection of Environment, Safety, and Health Programs at the Los Alamos National Laboratory. Washington, D.C.: 2002. Department of Energy, National Nuclear Safety Administration. Type B Accident Investigation of the Mineral Oil Leak Resulting in Property Damage at the Atlas Facility, Los Alamos National Laboratory, New Mexico. Washington, D.C.: March 2001. Department of Energy, National Nuclear Safety Administration. Type A Accident Investigation of the March 16, 2000, Plutonium-238 Multiple Intake Event at the Plutonium Facility, Los Alamos National Laboratory, New Mexico. Washington, D.C.: July 2000. Department of Energy, Office of Health, Safety and Security. Inspection of Environment, Safety, and Health Programs at the Lawrence Livermore National Laboratory. Washington, D.C.: May 2007. Department of Energy, National Nuclear Safety Administration. Type B Accident Investigation of the July 31, 2006, Fall from Ladder Accident at the Lawrence Livermore National Laboratory, Livermore, California. Washington, D.C.: October 2006. Department of Energy, Office of Security and Safety Performance Assurance. Inspection of Environment, Safety, and Health Programs at the Lawrence Livermore National Laboratory. Washington, D.C.: December 2004. Department of Energy, National Nuclear Safety Administration. Type B Accident Investigation Board Report of the June 2002 High Radiation Dose to Extremities in Building 151, Lawrence Livermore National Laboratory, Livermore, California. Washington, D.C.: Department of Energy, October 2002. Department of Energy, Office of Security and Safety Performance Assurance. Inspection of Environment, Safety, and Health Programs at the Lawrence Livermore National Laboratory. Washington, D.C.: July 2002. Department of Energy, Office of Security and Safety Performance Assurance. Inspection of Environment, Safety, and Health Programs at the Sandia National Laboratories. Washington, D.C.: 2005. Department of Energy, National Nuclear Safety Administration. Type B Accident Investigation of the March 20, 2003, Building 752 Stair Installation Accident at the Sandia National Laboratories, New Mexico. Washington, D.C.: Department of Energy, April 2003. Department of Energy, Office of Security and Safety Performance Assurance. Inspection of Environment, Safety, and Health Programs at the Sandia National Laboratories, New Mexico. Washington, D.C.: 2003. Defense Nuclear Facilities Safety Board. Safety Management of Complex, High-Hazard Organizations. DNFSB/TECH-35. Washington, D.C.: December 2004. Defense Nuclear Facilities Safety Board. Integrated Safety Management: The Foundation for a Successful Safety Culture. DNFSB/TECH-36. Washington, D.C.: December 2005. Significant multiple deficiencies in work control, quality improvement, and radiation protection resulting in exposure of eight employees to airborne radioactive material (estimated to be 1 of 10 worst radiological exposures over past 41 years). Failure to address long-standing deficiencies at an additional nuclear facility, which should have been identified and corrected during routine assessments and reviews. Management failures leading to establishing and operating an unauthorized nuclear facility for 5 years by storing radioactive waste in a facility without a safety evaluation and necessary controls. Failure of management processes, including oversight and assessments to identify inventory of nuclear materials that required analysis and controls. Numerous work process and radiological control violations resulting in exposure of workers to radioactive material and contamination of facility. Numerous failures to follow nuclear safety requirements and repeated work and radiological control deficiencies. Long- standing weaknesses in recognizing and addressing nuclear safety deficiencies. Significant work control deficiencies resulting in two workers receiving greater than annual allowed doses of radioactive material and exposure of five workers to toxic vapors. Severity levels increased because of long-standing nature of underlying problems and failure of management assessments and controls to identify or correct such problems. Fifteen separate violations of nuclear safety rules, which reflect continuing safety performance deficiencies over past several years. Deficiencies in work controls and quality improvement. Lack of fundamental improvements noted since stand-down of facilities. Failure to adequately address or take steps to correct programmatic weaknesses previously identified in maintaining and adhering to documents that form the safety basis for nuclear facilities. Failure to perform work using established controls over work processes. Radiation protection deficiencies resulting in significant radiological overexposure to one worker. Inadequate radiological controls and failure to implement a required hazard assessment. Long-standing and repeated failures to effectively track and correct radiological program deficiencies. Significant failure of management to properly oversee the correction of repeated problems. Weaknesses in determining underlying causes and corrective action plans. In addition to the individual named above, Janet E. Frisch, Assistant Director; Carole Blackwell; Timothy Bober; Candace Carpenter; Ellen W. Chu; Doreen Eng; Daniel Feehan; Nancy Kintner-Meyer; Thomas Laetz; Mehrzad Nadji; Omari Norman; Keith Rhodes; Rebecca Shea; and William R. Swick made contributions to this report. Department of Energy: Consistent Application of Requirements Needed to Improve Project Management. GAO-07-518. Washington, D.C.: May 11, 2007. National Nuclear Security Administration: Security and Management Improvements Can Enhance Implementation of the NNSA Act. GAO-07- 428T. Washington, D.C.: January 31, 2007. National Nuclear Security Administration: Additional Actions Needed to Improve Management of the Nation’s Nuclear Programs. GAO-07-36. Washington, D.C.: January 19, 2007. DOE Contracting: Better Performance Measures and Management Needed to Address Delays in Awarding Contracts. GAO-06-722. Washington, D.C.: June 30, 2006. Stand-down of Los Alamos National Laboratory: Total Costs Uncertain; Almost All Mission-Critical Programs Were Affected but Have Recovered. GAO-06-83. Washington, D.C.: November 18, 2005. Department of Energy: Further Actions Are Needed to Strengthen Contract Management for Major Projects. GAO-05-123. Washington, D.C.: March 18, 2005. National Nuclear Security Administration: Key Management Structure and Workforce Planning Issues Remain As NNSA Conducts Downsizing. GAO-04-545. Washington, D.C.: June 25, 2004. Department of Energy: Mission Support Challenges Remain at Los Alamos and Lawrence Livermore National Laboratories. GAO-04-370. Washington, D.C.: February 27, 2004. Contract Reform: DOE’s Policies and Practices in Competing Research Laboratory Contracts. GAO-03-932T. Washington, D.C.: July 10, 2003. Nuclear Security: NNSA Needs to Better Manage Its Safeguards and Security Program. GAO-03-471. Washington, D.C.: May 30, 2003. Department of Energy: Status of Contract and Project Management Reforms.GAO-03-570T. Washington, D.C.: March 20, 2003. Contract Reform: DOE Has Made Progress, but Actions Needed to Ensure Initiatives Have Improved Results. GAO-02-798. Washington, D.C.: September 13, 2002. Nonproliferation R&D: NNSA’s Program Develops Successful Technologies, but Project Management Can Be Strengthened. GAO-02-904. Washington, D.C.: August 23, 2002. Nuclear Security: Lessons to Be Learned from Implementing NNSA’s Security Enhancements. GAO-02-358. Washington, D.C.: March 29, 2002. Department of Energy: NNSA Restructuring and Progress in Implementing Title 32. GAO-02-451T. Washington, D.C.: February 26, 2002. Department of Energy: Fundamental Reassessment Needed to Address Major Mission, Structure, and Accountability Problems. GAO-02-51. Washington, D.C.: December 21, 2001. National Laboratories: Better Performance Reporting Could Aid Oversight of Laboratory-Directed R&D Program. GAO-01-927. Washington, D.C.: September 28, 2001. Nuclear Security: DOE Needs to Improve Control Over Classified Information. GAO-01-806. Washington, D.C.: August 24, 2001. Department of Energy: Follow-Up Review of the National Ignition Facility. GAO-01-677R. Washington, D.C.: June 1, 2001. Department of Energy: Views on Proposed Legislation on Civil Penalties for Nuclear Safety Violations by Nonprofit Contractors. GAO-01-548T. Washington, D.C.: March 22, 2001. Nuclear Weapons: Improved Management Needed to Implement Stockpile Stewardship Program Effectively. GAO-01-48. Washington, D.C.: December 14, 2000. National Ignition Facility: Management and Oversight Failures Caused Major Cost Overruns and Schedule Delays. GAO/RCED-00-141. Washington, D.C.: August 8, 2000. Department of Energy: Views on Proposed Civil Penalties, Security Oversight, and External Safety Regulation Legislation. GAO/T-RCED-00- 135. Washington, D.C.: March 22, 2000. Nuclear Security: Security Issues at DOE and Its Newly Created National Nuclear Security Administration. GAO/T-RCED-00-123. Washington, D.C.: March 14, 2000. Department of Energy: Views on DOE’s Plan to Establish the National Nuclear Security Administration. GAO/T-RCED-00-113. Washington, D.C.: March 2, 2000. Nuclear Security: Improvements Needed in DOE’s Safeguards and Security Oversight. GAO/RCED-00-62. Washington, D.C.: February 24, 2000. Department of Energy: Need to Address Longstanding Management Weaknesses. GAO/T-RCED-99-255. Washington, D.C.: July 13, 1999. Nuclear Weapons: DOE Needs to Improve Oversight of the $5 Billion Strategic Computing Initiative. GAO/RCED-99-195. Washington, D.C.: June 28, 1999. Department of Energy: DOE’s Nuclear Safety Enforcement Program Should Be Strengthened. GAO/RCED-99-146. Washington, D.C.: June 10, 1999. National Laboratories: DOE Needs to Assess the Impact of Using Performance-Based Contracts. GAO/RCED-99-141. Washington, D.C.: May 7, 1999. Department of Energy: Key Factors Underlying Security Problems at DOE Facilities. GAO/T-RCED-99-159. Washington, D.C.: April 20, 1999.

Summary: Federal officials, Congress, and the public have long voiced concerns about safety at the nation's nuclear weapons laboratories: Lawrence Livermore, Los Alamos, and Sandia. The laboratories are overseen by the National Nuclear Security Administration (NNSA), while contractors carry out the majority of the work. A recent change to oversight policy would result in NNSA's relying more on contractors' own management controls, including those for assuring safety. This report discusses (1) the recent history of safety problems at the laboratories and contributing factors, (2) steps taken to improve safety, and (3) challenges that remain to effective management and oversight of safety. To address these objectives, GAO reviewed almost 100 reports and investigations and interviewed key federal and laboratory officials. The nuclear weapons laboratories have experienced persistent safety problems, stemming largely from long-standing management weaknesses. Since 2000, nearly 60 serious accidents or near misses have occurred, including worker exposure to radiation, inhalation of toxic vapors, and electrical shocks. Although no one was killed, many of the accidents caused serious harm to workers or damage to facilities. Accidents and nuclear safety violations also contributed to the temporary shutdown of facilities at both Los Alamos and Lawrence Livermore in 2004 and 2005. Yet safety problems persist. GAO's review of nearly 100 reports issued since 2000 found that the contributing factors to these safety problems generally fall into three key areas: relatively lax laboratory attitudes toward safety procedures, laboratory inadequacies in identifying and addressing safety problems with appropriate corrective actions, and inadequate oversight by NNSA site offices. NNSA and its contractors have been taking some steps to address safety weaknesses at the laboratories. Partly in response to continuing safety concerns, NNSA has begun taking steps to reinvigorate a key safety effort--integrated safety management--originally started in 1996. This initiative was intended to raise safety awareness and provide a formal process for employees to integrate safety into every work activity by identifying potential safety hazards and taking appropriate steps to mitigate these hazards. NNSA and its contractors have also begun taking steps to develop or improve systems for identifying and tracking safety problems and the corrective actions taken in response. Finally, NNSA has initiated efforts to strengthen federal oversight at the laboratories by improving hiring and training of federal site office personnel. NNSA has also taken steps to strengthen contractor accountability through new contract mechanisms. Many of these efforts are still under way, however, and their effect on safety performance is not clear. NNSA faces two principal challenges in its continuing efforts to improve safety at the weapons laboratories. First, the agency has no way to determine the effectiveness of its safety improvement efforts, in part because those efforts rarely incorporate outcome-based performance measures. The department issued a directive in 2003 requiring use of a disciplined approach for managing improvement initiatives, often used by high-performing organizations, including results-oriented outcome measures and a system to evaluate the effectiveness of the initiative. Yet GAO found little indication that NNSA or its contractors have been managing safety improvement efforts using this approach. Second, in light of the long-standing safety problems at the laboratories, GAO and others have expressed concerns about the recent shift in NNSA's oversight approach to rely more heavily on contractors' own safety management controls. Continuing safety problems, coupled with the inability to clearly demonstrate progress in remedying weaknesses, make it unclear how this revised system will enable NNSA to maintain an appropriate level of oversight of safety performance at the weapons laboratories.