In accordance with scientific custom and/or statutory mandates, several offices within EPA have used peer review for many years to enhance the quality of science within the agency. In May 1991, the EPA Administrator established a panel of outside academicians to, among other things, enhance the stature of science at EPA and determine how the agency can best ensure that sound science is the foundation for the agency’s regulatory and decision-making processes. In March 1992, the expert panelrecommended that, among other things, EPA establish a uniform peer review process for all scientific and technical products used to support EPA’s guidance and regulations. In response, EPA issued a policy statement in January 1993 calling for peer review of the major scientific and technical work products used to support the agency’s rulemaking and other decisions. However, the Congress, GAO, and others subsequently raised concerns that the policy was not being consistently implemented throughout EPA. The congressional concern resulted in several proposed pieces of legislation that included prescriptive requirements for peer reviews. Subsequently, in June 1994 the EPA Administrator reaffirmed the central role of peer review in the agency’s efforts to ensure that its decisions rest on sound science and credible data by directing that the agency’s 1993 peer review policy be revised. The new policy retained the essence of the prior policy and was intended to expand and improve the use of peer review throughout EPA. Although the policy continued to emphasize that major scientific and technical products should normally be peer reviewed, it also recognized that statutory and court-ordered deadlines, resource constraints, and other constraints may limit or preclude the use of peer review. According to the Executive Director of the Science Policy Council, one of the most significant new features of the 1994 action was the Administrator’s directive to the agency’s Science Policy Council to organize and guide an agencywide program for implementing the policy. The policy and procedures emphasize that peer review is not the same thing as other mechanisms that EPA often uses to obtain the views of interested and affected parties and/or to build consensus among the regulated community. More specifically, EPA’s policy and procedures state that peer review is not peer input, which is advice or assistance from experts during the development of a product; stakeholders’ involvement, which is comments from those people or organizations (stakeholders) that have significant financial, political, or other interests in the outcome of a rulemaking or other decision by EPA; or public comment, which is comments obtained from the general public on a proposed rulemaking and may or may not include the comments of independent experts. While each of these activities serves a useful purpose, the policy and procedures point out that they are not a substitute for peer review. For example, as noted in EPA’s Standard Operating Procedures, public comments on a rulemaking do not necessarily solicit the same unbiased, expert views as are obtained through peer review. In order to accommodate the differences in EPA’s program and regional offices, the policy assigned responsibility to each program and regional office to develop standard operating procedures and to ensure their use. To help facilitate agencywide implementation, EPA’s Science Policy Council was assigned the responsibility of assisting the offices and regions in developing their procedures and identifying products that should be considered for peer review. The Council was also given the responsibility for overseeing the agencywide implementation of the policy by promoting consistent interpretation, assessing agencywide progress, and developing revisions to the policy, if warranted. However, EPA’s policy specifies that the Assistant and Regional Administrators for each office are ultimately responsible for implementing the policy, including developing operating procedures, identifying work products subject to peer review, determining the type and timing of such reviews, and documenting the process and outcome of each peer review conducted. Our objectives, scope, and methodology are fully described in appendix I. Two years after EPA established its peer review policy, implementation is still uneven. EPA acknowledges this problem and provided us with a number of examples to illustrate the uneven implementation. At our request, the Science Policy Council obtained information from EPA program and regional offices and provided us with examples in which, in their opinion, peer review was properly conducted; cases in which it was conducted but not fully in accordance with the policy; and cases in which peer review was not conducted at all. The following table briefly summarizes the cases they selected; additional information on these nine cases is provided in appendix II. According to the Executive Director of the Science Policy Council, this unevenness can be attributed to several factors. First, some offices within EPA have historically used peer review, while others’ experience is limited to the 2 years since the policy was issued. For example, in accordance with scientific custom, the Office of Research and Development (ORD) has used peer review for obtaining critical evaluations of certain work products for more than 20 years. Additionally, statutes require that certain work products developed by EPA be peer reviewed by legislatively established bodies. For example, criteria documents developed by ORD for the National Ambient Air Quality Standards must receive peer review from EPA’s Science Advisory Board (SAB), and pesticide documents must receive peer review from the Scientific Advisory Panel. In contrast, some EPA regional offices and areas within some EPA program offices have had little prior experience with peer review. In addition to these offices’ varying levels of experience with peer review, the Science Policy Council’s Executive Director and other EPA officials said that statutory and court-ordered deadlines, budget constraints, and difficulties associated with finding and obtaining the services of qualified, independent peer reviewers have also contributed to peer review not being consistently practiced agencywide. A report by the National Academy of Public Administration confirmed that EPA frequently faces court-ordered deadlines. According to the Academy, since 1993 the courts have issued an additional 131 deadlines that EPA must comply with or face judicial sanctions. Also, as explained to us by officials from EPA’s Office of Air and Radiation (OAR), just about everything EPA does in some program areas, such as Clean Air Act implementation, is to address either legislative or court-ordered mandates. Others have attributed EPA’s problems with implementing peer review in the decision-making process to other factors. For example, in its March 1995 interim report on EPA’s research and peer review program within the Office of Research and Development, the National Academy of Sciences’ National Research Council noted that, even in EPA’s research community, knowledge about peer review could be improved. The Council’s interim report pointed out that “although peer review is widely used and highly regarded, it is poorly understood by many, and it has come under serious study only in recent years.” Although we agree that the issues EPA and others have raised may warrant further consideration, we believe that EPA’s uneven implementation is primarily due to (1) confusion among agency staff and management about what peer review is, what its significance and benefits are, and when and how it should be conducted and (2) ineffective accountability and oversight mechanisms to ensure that all products are properly peer reviewed by program and regional offices. Although the policy and procedures provide substantial information about what peer review entails, we found that some EPA staff and managers had misperceptions about what peer review is, what its significance and benefits are, and when and how it should be conducted. For example, officials from EPA’s Office of Mobile Sources (OMS) told the House Commerce Committee in August 1995 that they had not had any version of the mobile model peer reviewed. Subsequently, in April 1996, OMS officials told us they recognize that external peer review is needed and that EPA plans to have the next iteration of the model peer reviewed. However, when asked how the peer review would be conducted, OMS officials said they plan to use the public comments on the revised model they receive as the peer review. As EPA’s policy makes clear, public comments are not the same as nor are they a substitute for peer review. We found a similar misunderstanding about what peer review entails in a regional office we visited. The region prepared a product that assesses the impacts of tributyl tin—a compound used since the 1960s in antifouling paints for boats and large ships. Although regional staff told us that this contractor-prepared product had been peer reviewed, we found that the reviews were not in accordance with EPA’s peer review policy. The draft product received some internal review by EPA staff and external review by contributing authors, stakeholders, and the public; however, it was not reviewed by experts previously uninvolved with the product’s development nor by those unaffected by its potential regulatory ramifications. When we pointed out that—according to EPA’s policy and the region’s own peer review procedures—these reviews are not a substitute for peer review, the project director said that she was not aware of these requirements. In two other cases we reviewed, there was misunderstanding about the components of a product that should be peer reviewed. For example, in the Great Waters study—an assessment of the impact of atmospheric pollutants in significant water bodies—the scientific data were subjected to external peer review, but the study’s conclusions that were based on these data were not. Similarly, in the reassessment of dioxin—a reexamination of the health risks posed by dioxin—the final chapter summarizing and characterizing dioxin’s risks was not as thoroughly peer reviewed. More than any other, this chapter indicated EPA’s conclusions based on its reassessment of the dioxin issue. In both cases, the project officers did not have these chapters peer reviewed because they believed that the development of conclusions is an inherently governmental function that should be performed exclusively by EPA staff. However, some EPA officials with expertise in conducting peer reviews disagreed, maintaining that it is important to have peer reviewers comment on whether or not EPA has properly interpreted the results of the underlying scientific and technical data. In addition to the uncertainty surrounding the peer review policy, we also noted problems with EPA’s accountability and oversight mechanisms.