Peer review is well established as a mechanism for assuring the quality, credibility, and acceptability of individual and institutional work products. This assurance is accomplished by having the products undergo an objective, critical review by independent reviewers. Peer review has long been used by academia, professional organizations, industry, and government. Within EPA, peer review has taken many different forms, depending upon the nature of the work product, the relevant statutory requirements, and office-specific practices and needs. In keeping with scientific custom and/or congressional mandates, several offices within EPA have used peer review for many years to enhance the quality of science within the agency. In response to a panel of outside academicians’ recommendations in 1992, EPA issued a policy statement in 1993 calling for peer review of the major scientific and technical work products used to support the agency’s rulemaking and other decisions. However, the Congress, GAO, and others subsequently raised concerns that the policy was not being implemented consistently across the agency. In response to these concerns, in 1994 EPA reaffirmed the central role that peer review plays in ensuring that the agency’s decisions are based on sound science and credible data and revised its 1993 policy. The new policy, while retaining the essence of the prior one, was intended to expand and improve the use of peer review throughout EPA. The 1994 policy continued to stress that major products should normally be peer reviewed, but it also recognized that statutory and court-ordered deadlines, resource limitations, and other constraints might limit or even preclude the use of peer review. The policy applied to major work products that are primarily scientific or technical in nature and that may contribute to the basis for policy or regulatory decisions. In contrast, other products used in decision-making are not covered by the policy, nor are the ultimate decisions themselves. While peer review can take place at several different points along a product’s development, such as during the planning stage, it should be applied to a relatively well-developed product. The 1994 policy also clarified that peer review is not the same thing as the peer input, stakeholders’ involvement, or public comment—mechanisms used by EPA to develop products, to obtain the views of interested and affected parties, and/or to build consensus among the regulated community. While each of these mechanisms serves a useful purpose, the policy points out that they are not a substitute for peer review because they do not necessarily solicit the same unbiased, expert views that are obtained through peer review. EPA’s policy assigned responsibility to each Assistant and Regional Administrator to develop standard operating procedures and to ensure their use. To help facilitate consistent EPA-wide implementation, EPA’s Science Policy Council—chaired by EPA’s Deputy Administrator—was directed to help the offices and regions develop their procedures and identify products that should be peer reviewed. The Council was also given the responsibility for assessing agencywide progress and developing any needed changes to the policy. However, the ultimate responsibility for implementing the policy was placed with the Assistant and Regional Administrators. We found that—2 years after EPA established its peer review policy— implementation was still uneven. We concluded that EPA’s uneven implementation was primarily due to (1) inadequate accountability and oversight to ensure that all products are properly peer reviewed by program and regional offices and (2) confusion among agency staff and management about what peer review is, what its significance and benefits are, and when and how it should be conducted. According to the Executive Director of the Science Policy Council, the unevenness could be attributed to a number of factors. First, while some offices within EPA—such as the Office of Research and Development (ORD)—have historically used peer review for many years, other program offices and regions have had little prior experience. In addition, the Director and other EPA officials told us that statutory and court-ordered deadlines, budget constraints, and problems in finding and obtaining qualified, independent peer reviewers also contributed to the problem. EPA’s oversight primarily consisted of a two-part reporting scheme that called for each office and region to annually list (1) the candidate products nominated for peer review during the upcoming year and (2) the status of the products previously nominated. If a candidate product was no longer scheduled for peer review, the list had to note this and explain why peer review was no longer planned. Although we found this to be an adequate oversight tool for tracking the status of previously nominated products, we pointed out that it does not provide upper-level managers with sufficient information to ensure that all products warranting peer review have been identified. This fact, together with the misperceptions about what peer review is and the deadlines and budget constraints that project officers often operate under, has meant that the peer review program to date has largely been one of self-identification, allowing some important work products to go unlisted. According to the Science Policy Council, reviewing officials would be much better positioned to determine if the peer review policy and procedures are being properly and consistently implemented if, instead, EPA’s list contained all major products along with what peer review is planned and, if none, the reasons why not. We noted that the need for more comprehensive oversight is especially important given the policy’s wide latitude in allowing peer review to be forgone in cases facing time and/or resource constraints. As explained by the Executive Director of EPA’s Science Policy Council, because so much of the work that EPA performs is in response to either statutory or court-ordered mandates and the agency frequently faces budget uncertainties or limitations, an office under pressure might argue for nearly any given product that peer review is a luxury the office cannot afford in the circumstances. However, as the Executive Director of the Science Advisory Board (SAB)told us, not conducting peer review can sometimes be more costly to the agency in terms of time and resources. He told us of a recent Office of Solid Waste rulemaking concerning a new methodology for delisting hazardous wastes in which the Office’s failure to have the methodology appropriately peer reviewed resulted in important omissions, errors, and flawed approaches in the methodology; these problems will now take from 1 to 2 years to correct. The SAB also noted that further peer review of the individual elements of the proposed methodology is essential before the scientific basis for this rulemaking can be established. Although EPA’s policy and procedures provide substantial information about what peer review entails, we found that some EPA staff and managers had misperceptions about what peer review is, what its significance and benefits are, and when and how it should be conducted. Several cases we reviewed illustrate this lack of understanding about what peer review entails. Officials from EPA’s Office of Mobile Sources (OMS) told the House Commerce Committee in August 1995 that they had not had any version of the mobile model peer reviewed. Subsequently, in April 1996, OMS officials told us they recognize that external peer review is needed and that EPA planned to have the next iteration of the model so reviewed. We found similar misunderstandings in several other cases we reviewed. EPA regional officials who produced a technical product that assessed the environmental impacts of tributyl tin told us that the contractor-prepared product had been peer reviewed. While we found that the draft product did receive some internal review by EPA staff and external review by contributing authors, stakeholders, and the public, it was not reviewed by experts independent of the product itself or of its potential regulatory ramifications. When we pointed out that—according to EPA’s policy and the region’s own peer review procedures—these reviews are not a substitute for peer review, the project director said that she was not aware of these requirements. In two other cases we reviewed, there were misunderstandings about the components of a product that should be peer reviewed. For example, in the Great Waters study—an assessment of the impact of atmospheric pollutants in significant water bodies—the scientific data were subjected to external peer review, but the study’s conclusions that were based on these data were not. Similarly, in the reassessment of dioxin—an examination of the health risks posed by dioxin—the final chapter summarizing and characterizing dioxin’s risks was not as thoroughly peer reviewed. In both cases, the project officers did not have the conclusions peer reviewed because they believed that the development of conclusions is an inherently governmental function that should be performed exclusively by EPA staff. However, some EPA officials with expertise in conducting peer reviews disagreed, maintaining that it is important to have peer reviewers comment on whether or not EPA has properly interpreted the results of the underlying scientific and technical data. EPA’s quality assurance requirements also state that conclusions should be peer reviewed. During our review, we found that EPA had recently taken a number of steps to improve the peer review process. Although we believed that these steps should prove helpful, we concluded that they did not fully address the previously-discussed underlying problems and made some recommendations for improvement. EPA agreed with our findings and recommendations and has recently undertaken steps to implement them. While it is too early to gauge the effectiveness of these efforts, we are encouraged by the attention peer review is receiving by the agency’s upper-level management. Near the completion of our review, in June 1996, EPA’s Deputy Administrator directed the Science Policy Council’s Peer Review Advisory Group and ORD’s National Center for Environmental Research and Quality Assurance to develop an annual peer review self-assessment and verification process to be conducted by each office and region. The self-assessment was to include information on each peer review completed during the prior year as well as feedback on the effectiveness of the overall process. The verification would consist of the signature of headquarters, laboratory, or regional directors to certify that the peer reviews were conducted in accordance with the agency’s policy and procedures. If the peer review did not fully conform to the policy, the division director or the line manager must explain significant variances and actions needed to limit future significant departures from the policy. The self-assessments and verifications were to be submitted and reviewed by the Peer Review Advisory Group to aid in its oversight responsibilities.