Scientific research has traditionally been considered to be valuable to society. The knowledge gained from federally supported research leads to the development of new products and processes. Research results also provide information needed to make policy decisions. The Congress is concerned that quality research is used to provide a rational basis for federal rule- making, as with regulations established to protect human health and safety and the environment. Peer review has been used to judge the quality of science for over 3 centuries. Historically, peer review has been used extensively in the selection of proposed research projects and to a lesser extent to evaluate R&D programs and their likely effects. The Congress and the President have taken steps to help improve the management of and accountability for federal R&D spending. In 1976, the Congress established the Office of Science and Technology Policy, which serves as a source of scientific, engineering, and technological analysis and judgment for the President and assists him in providing leadership and coordination for federal R&D programs. In November 1993, by executive order, the President also established the Cabinet-level National Science and Technology Council to coordinate federal R&D and to establish clear national goals for science and technology investments. Through this Council, the Office of Science and Technology Policy helps the President coordinate science, space, and technology policy and programs across the federal government and leads an interagency effort to develop and implement science and technology policies and budgets across federal agencies. In 1993, the Congress passed the Government Performance and Results Act to encourage efficiency, effectiveness, and accountability in federal programs. The act requires federal agencies to produce strategic performance plans with annual targets and annual performance reports that explain whether those targets have been met. A recent report by the Committee on Science, Engineering, and Public Policy concluded that the most effective means of evaluating federally funded research programs is expert review. The best known form of expert review is peer review; two other forms are relevance review that examines whether the research program focuses on an agency’s mission and international benchmarking that determines whether the research is at the forefront of scientific and technological knowledge. There is no written definition of peer review that applies across the federal government. Officials at the Office of Science and Technology Policy described peer review as a merit-based process or independent merit assessment, generally used in decisions about which research projects to fund. Individual agencies define peer review somewhat differently; however, all of the agencies’ definitions or descriptions of peer review contained the fundamental concept of a review of technical or scientific merit by individuals with sufficient technical competence and no unresolved conflict of interest. Peers generally are considered to be scientists or engineers who have qualifications and expertise equivalent to those of the researcher whose work they review. In addition, peers must be capable of making an independent judgment of the merits and relevance of the research. Officials at the Office of Science and Technology Policy said that peers do not have to be external to the funding agency, as long as there is no unresolved conflict of interest. Officials at the Office of Science and Technology Policy said that there is no governmentwide legislation or policy that requires agencies to conduct peer reviews or dictates how the reviews should be conducted. However, the office advocates the use of peer review and provides guidance to agencies on the use of peer review to assess the quality of research. For example, the office works with the Office of Management and Budget to promote peer review. Beginning with the fiscal year 1996 budget cycle, the two offices have jointly provided annual direction to agencies encouraging them to emphasize the funding of peer-reviewed research over nonpeer-reviewed research. As a result, Office of Science and Technology Policy officials said that agencies have shifted funding toward more peer-reviewed research. The guidance also encourages agencies to solicit proposals from many researchers on how to solve research problems. Officials at the Office of Science and Technology Policy said that agencies’ peer review practices should be flexible and tailored to agency missions and type of research, and that specific uniform practices should not be dictated for every agency or all federally funded research. A variety of peer review methods is viewed by the agencies as both appropriate and essential, reflecting the varying nature of the research and its purposes, the differences in research timelines, the broad spectrum of R&D performers, and the varying funding mechanisms, such as grants, contracts, and cooperative agreements. For example, agency officials noted that specific peer reviews can have quite different purposes such as the review of research results prior to publication versus the review of agency or program performance. While peer review has come to be viewed by some observers as the best assurance that quality criteria will prevail over social, economic, and political considerations, others view peer review as an element of elitism in science that tends to discount such concerns as economic considerations. Officials at the National Science Foundation said that under certain conditions, over emphasis on peer review for funding decisions may discourage funding of innovative research because peers generally tend to view research somewhat conservatively. Officials at the Office of Science and Technology said that this phenomenon has been noted by other agencies, and the Office is initiating an interagency effort to examine how peer review practices (and other federal R&D policies) might better facilitate innovation. “Science agencies must devise assessment strategies that are appropriate to the nature of scientific processes and to the enabling role of fundamental science in support of over-arching national goals... should be designed to...respond to surprises, pursue detours, and revise program agendas in response to new scientific information and technical opportunities essential to the future well-being of all our people.” All the agencies that we contacted identified policies, orders, or other internal guidance regarding the conduct of peer review. Some of these policies are legislatively mandated. For example, the Agricultural Research, Extension, and Education Reform Act of 1998 requires the U.S. Department of Agriculture to peer review Department-funded research.The law also requires grantees to arrange for a peer review of special grants mandated by the Congress. Overall, we found that eight agencies—the Agricultural Research Service, the Cooperative State Research, Education, and Extension Service, the National Institute of Standards and Technology, the National Institutes of Health, the Department of Energy, the Environmental Protection Agency, the National Aeronautics and Space Administration, and the Federal Aviation Administration—have laws or regulations that require peer review of competitively selected grant proposals. The peer review practices differ among and within federal research agencies in two ways. First, agencies use peer review in varying degrees to assess the merit of research at different stages in the research process, including selecting research projects for funding; monitoring in-progress research; and evaluating research products prior to publication. Second, the implementation of the peer review process varies. The following highlights the extent to which agencies use peer review and some of the various ways they implement peer review. Appendixes I through XII provide descriptions of the agencies’ peer review practices, to the extent that peer review is used, for each of the 12 agencies included in our review. All the agencies conduct peer reviews to help determine which competitive research proposals to fund. All 12 agencies also use peer review to help determine funding for at least a portion of their other research, including peer reviews of the agencies’ intramural or internal research proposals or plans. The agencies use a combination of external and internal reviewers with subject matter expertise. However, Federal Aviation Administration officials said that their peer reviewers are primarily agency employees who are not involved in the project but have the required subject matter expertise. The agencies conduct the peer reviews by mail, panels or committees, or a combination of methods. They also differ in the number of reviewers used in the process. For example, the Cooperative State Research, Education, and Extension Service uses panels of outside experts to review competitive research proposals. One Cooperative State Research, Education, and Extension Service program, the National Research Initiative, uses panels that meet throughout the year. Each panel member reviews about 30 research proposals and provides written comments on about 20 of the proposals. Mail reviews and ad hoc reviewers are also used when additional expertise is needed. In contrast, the National Aeronautics and Space Administration generally conducts reviews by mail to obtain specialized expertise on technical issues and approaches, followed by reviews by panels of 7 to 10 experts with broader perspectives to reconcile differences among the mail reviewers’ comments. The agencies use various criteria to assess proposed research, including technical or scientific merit, relevance to agency mission and priorities, and the qualifications of the researcher. Agency officials responsible for selecting research for funding generally consider the peer reviewers’ advice or recommendations along with other financial and management factors to make their funding decisions. In some circumstances, agencies use these same types of peer review processes to assess the merit of research that is not funded through competitive selection, generally this research is internal to the agencies. For example, the Forest Service conducts peer reviews of its scientists’ research study plans prior to approval by project leaders. The U.S. Geological Survey conducts peer reviews of all intramural project proposals to ensure technical quality prior to final approval and implementation. At the Agricultural Research Service, internal peers, who are not involved in the funding decision, review detailed project plans. While all 12 agencies provided examples of peer reviews of research that are in-progress, there is much variation in the frequency and purpose of those reviews. The purposes of these reviews include assessments of research projects to determine if funding should be renewed or to assess the progress of on-going research at the program level, or at research stations or laboratories. For example, in-progress Cooperative State Research, Education, and Extension Survey projects that are assessed for renewal within 2 to 3 years compete with new proposals in the same merit review process. If the review panel considers the research progress to be unsatisfactory, the research project will not receive additional funding. The National Aeronautics and Space Administration requires that for any project that continues for more than 3 years, the researcher must submit a new proposal, which is subject to external peer review. The National Institutes of Health uses boards of outside experts to peer review its on-going intramural research, and the Agricultural Research Service convenes panels in a workshop format to review intramural research projects. The U.S. Geological Survey’s peer review guidance requires that on-going programs undergo external peer review about every 5 years. The National Oceanic and Atmospheric Administration also has peer reviews of the work funded and completed at its laboratories. Every 3 to 5 years, qualified peers evaluate program accomplishments and impacts in the context of the resources invested in them. At the National Institute of Standards and Technology, the National Research Council’s Board on Assessment of the Institute’s programs annually conducts reviews of the technical quality and relevance of planned, ongoing, and completed laboratory programs. On a cyclical basis, the Forest Service Deputy Chief’s Program Reviews use external peers in the evaluations of a research station’s overall program to improve program results. Generally, the agency officials said that their agencies encourage their scientists to publish research results in professional journals that conduct peer reviews of manuscripts prior to accepting them for publication. Some agencies also peer review draft work products, manuscripts, or other research results prior to publication. For example, the Agricultural Research Service, the Forest Service, the Environmental Protection Agency, the National Oceanic and Atmospheric Administration’s National Marine Fisheries Service, the National Institutes of Health, and the U.S. Geological Survey review their scientists’ manuscripts prior to publication in in-house technical reports or professional journals. Agencies also use peer-review techniques to aid priority setting, program development, and personnel evaluation. For example, the Federal Aviation Administration’s annual programming and budgeting process includes program-level peer reviews of proposed and in-progress research by a 30-member, legislatively authorized Research, Engineering, and Development Advisory Committee, established under the Aviation Safety Research Act of 1988, as amended. This process results in recommendations about the merit of the research and funding priorities in the Federal Aviation Administration’s six program areas. The Environmental Protection Agency’s Science Advisory Board, consisting of scientists, engineers, and economists from academia, industry, and environmental communities, reviews the technical basis for the agency’s science policy positions, including scientific documents used to support environmental regulations. Independent scientists provide advice to the Department of Energy on the quality, relevance, and productivity of its laboratory research, in conjunction with program reviews and advisory committee oversight. Last, the Agricultural Research Service and the Forest Service also consider the required peer reviews of their individual scientist’s research accomplishments to constitute additional checks of the quality of their research. In addition to peer reviews, most agencies also conduct various types of internal reviews as checks on the quality of their research. These reviews are generally conducted by supervisors or managers and are, therefore, not independent reviews of the research. Agencies conduct these quality assurance reviews to assess the merit of proposed research, to assess the progress of on-going research, and to evaluate research results. These reviews occur at both the project and program level. TThe following are examples of agencies’ internal reviews. Scientists at the Cooperative State Research, Education, and Extension Service review noncompetitive proposals for formula-funded awards and for congressionally mandated research. Cooperative State Research, Education, and Extension Service scientists also review annual progress reports prepared by the researchers. Likewise, Agricultural Research Service line managers review annual project progress reports submitted by the Service’s lead scientists. These reviews may lead to revisions of program plans. National Institute of Standards and Technology officials track the results of research funded by the Advanced Technology Program, using quarterly progress reports and annual meetings with recipients. The National Aeronautics and Space Administration project managers and staff frequently comment on technical, management, and financial aspects of a proposal, since science reviewers may not be qualified experts in these fields. The Federal Aviation Administration’s Civil Aeromedical Institute, which conducts mostly intramural research, reviews the technical details of proposed projects. According to Federal Aviation Administration officials, the Institute employs some of the world’s best scientists, so the number of outside experts is limited. Research performed under Federal Aviation Administration procurement contracts is reviewed during the annual programming and budgeting process. Subsequently, when a decision is made to fund research through a contract, Federal Aviation Administration technical staff monitor the work. In addition, the Agricultural Research Service, the Cooperative State Research, Education, and Extension Service, the Forest Service, the National Institute of Standards and Technology, the National Institutes of Health, the National Oceanic and Atmospheric Administration, the Department of Energy, and the National Science Foundation use advisory committees to help establish research programs or priorities. For example, section 103 of the Agricultural Research, Extension, and Education Reform Act of 1998 requires that the National Agricultural Research, Extension, Education, and Economics Advisory Board annually review the Department of Agriculture’s research priority setting. A Forest Research Advisory Committee to the Secretary of Agriculture, made up of university, industry, and interest group representatives, also reviews and comments on the Forest Service’s current and future research priorities. The National Institute of Standards and Technology statutory Visiting Committee on Advanced Technology meets quarterly to review agency research policies, budget, organization, and programs. A few agency officials also discussed their use of peer review in their efforts to comply with the Results Act requirements for reporting performance measures, goals, and outcomes of their research. For example, the Federal Aviation Administration’s annual programming and budget process includes reviews of research outcomes and outputs and is performed in the context of the Results Act. A Department of Agriculture report stated that the Cooperative State Research, Education, and Extension Service’s Results Act report might potentially provide a mechanism to more systematically monitor and report research performance and accomplishments. The National Science Foundation is trying to track the results of its research efforts through annual and final reports on each project and plans to follow-up for years after completion to identify retrospective impacts from the research. While the agencies said that they conduct either peer reviews or other quality reviews for almost all of their research, there are small amounts of research that may not be reviewed. For example, officials from the Agricultural Research Service, the National Institute of Standards and Technology, and the Department of Energy said that they did not always review specific research proposals when the agency is directed to perform the research. In addition, research performed jointly with outside entities, such as with cooperative research and development agreements, may not always be fully subjected to review, depending on factors such as the nature of the partnership and the presence of proprietary information. The Congress directs the Agricultural Research Service to allocate about $10 million annually to external researchers for specific cooperative agreements. While the Agricultural Research Service does not usually peer review this research prior to funding, it does review progress reports that the researcher is required to submit annually. Officials from the National Institutes of Health said that supplemental funding requested to carry on previously reviewed and funded research might not be subject to peer review. The Environmental Protection Agency’s peer review guidance recognizes circumstances that might preclude peer review, including research methodology that is widely accepted and research for which the regulatory activity has been terminated. We provided a draft of this report to the Agricultural Research Service, the Cooperative State Research, Education, and Extension Service, and the Forest Service within the U.S. Department of Agriculture; the Department of Commerce; the Department of Energy; the Environmental Protection Agency; the National Institutes of Health; the Department of the Interior; the National Aeronautics and Space Administration; the National Science Foundation; the Department of Transportation; and the Office of Science and Technology Policy for review and comment. We obtained comments from each of the above agencies. Generally, the agencies concurred that the report provided an accurate portrayal of their peer review practices. Some of the agencies suggested technical changes to the report to help ensure an accurate description of their peer review practices, and we incorporated the agencies’ comments. An official at the Department of Energy pointed out that our statement that peer review practices vary is not without policy connotations and is subject to misinterpretation. He said that peer review practices should vary among and within the agencies. To address this concern, we added additional agency views on the need for flexibility in the peer review process. To define what is meant by peer review and to describe the federal government’s peer review policy, we reviewed studies of government peer review, previous GAO reports, and documentation provided by the 12 agencies included in our review. We also interviewed officials from the Office of Science and Technology Policy and reviewed that office’s guidance and its and the Office of Management and Budget’s annual budget direction to federal agencies. To describe the peer review practices of 12 agencies, we obtained and compared descriptive information on peer review at each agency to identify what the various practices were and to determine whether the practices were uniform among and within the agencies. The agencies provided legislation, policies, manuals, and other documentation, which we reviewed, related to the agencies’ implementation of peer review. The agencies also provided fiscal year 1999 research and development budget data. To obtain the agencies’ rationales for their practices and the reasons for the variations among the agencies and programs and to obtain information about practices that were not formally documented, we interviewed officials knowledgeable about and responsible for conducting peer reviews of scientific research at each agency’s Headquarters headquarters office. To identify other quality assurance reviews the agencies conducted in addition to or in lieu of peer reviews, we reviewed agency documentation and interviewed agency officials. Because the 12 agencies’ practices were carried out at numerous research sites at headquarters, field offices, laboratories, research stations, and grantee locations across the United States, we did not attempt to verify the extent to which the many different practices reported by the agencies were being implemented and carried out. From our interviews with agency officials, we also identified research that did not receive any peer review or other quality assurance review and the agencies’ rationale for not conducting reviews of this research. Our work was performed from August 1998 through March 1999 in accordance with generally accepted government auditing standards. As arranged with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 10 days after its date. At that time, we will send copies of this report to Dan Glickman, Secretary, Department of Agriculture; William M. Daley, Secretary, Department of Commerce; Bill Richardson, Secretary, Department of Energy; Donna E. Shalala, Secretary, Department of Health and Human Services, ; Bruce Babbitt, Secretary, Department of the Interior; Rodney E. Slater, Secretary, Department of Transportation; D. James Baker, Under Secretary, National Oceanic and Atmospheric Administration; Floyd P. Horn, Administrator, Agricultural Research Service; Colien Hefferan, Acting Administrator, Cooperative State Research, Education, and Extension Service; Mike Dombeck, Chief, Forest Service; Daniel S. Goldin, Administrator, National Aeronautics and Space Administration; Carol M. Browner, Administrator, Environmental Protection Agency; Jane F. Garvey, Administrator, Federal Aviation Administration; Raymond G. Kammer, Director, National Institute of Standards and Technology; Harold E. Varmus, Director, National Institutes of Health; Rita R. Colwell, Director, National Science Foundation; Jacob J. Lew, Director, Office of Management and Budget; Neal Lane, Director, Office of Science and Technology Policy; and Thomas Casadevall, Director, U.S. Geological Survey. We will also make copies available to others on request. If you or your staff have any questions concerning this report, please call me at 202-512-3841. Major contributors to this report are listed in appendix XIII. The following presents a description of the Agricultural Research Services’ (ARS’) peer review and other quality assurance review practices. The Agricultural Research Service, an agency of the U.S. Department of Agriculture, conducts research on foods, fibers, soil, water, and other natural resources. ARS’ mission is to solve technical agricultural problems of broad scope and high national priority. ARS carries out this mission through 23 national programs. ARS does not have a regulatory mission but develops methods and technologies used by other regulatory agencies within the Department, such as the Animal and Plant Health Inspection Service and the Food Safety Inspection Service, and by other federal agencies such as the Food and Drug Administration and the Environmental Protection Agency. ARS’ funding for fiscal year 1999 is about $813 million. Most research projects are conducted in-house by ARS scientists and between ARS scientists and states, local governments, private firms, and institutions through cooperative research and other types of agreements. Only about $4 million of ARS’ annual appropriation is competitively awarded to external scientists. According to agency officials, ARS spends about $800,000 annually to conduct peer reviews. ARS defines peer review as the evaluation of the conceptual, relevance, and technical soundness of research by highly qualified scientists active in the same or closely related research fields. ARS has a multilayered system of complimentary peer reviews that includes reviews of the technical merit of planned research projects prior to funding, reviews of research results prior to the publication of the results, and reviews of ongoing research programs. Various ARS directives and manuals provide the guidance for conducting these peer reviews, which are performed by both internal and external experts and external customers and stakeholders. ARS is revising its project merit review system during fiscal year 1999 to satisfy the additional requirements of the Agricultural Research, Extension, and Education Reform Act of 1998, which requires merit review panels composed of a majority of external peers for all ARS research projects. ARS’ Research Project Documentation Manual provides the guidance for conducting the project plan merit reviews prior to deciding which projects to fund. The agency manages 1,100 on-going research projects, which operate at 102 locations and are assigned to specific laboratories based on program plans. The projects span 3 to 5 years, and each year 200 to 300 projects require funding or reduction decisions. ARS’ internal peers, who are not involved in the funding decision, review detailed project plans. In addition, the list of peers must include three individuals from universities and three individuals from customer or stakeholder groups. Each reviewer provides comments by mail. ARS managers request the reviews and act on the results. The ARS scientist who prepared the project plan must respond in writing to reviewers’ comments. ARS officials said that the new project merit review system will consist of peer review panels that will review the merit of research project proposals related to each of the 23 national programs. To ensure that the new system meets the letter and spirit of the law, ARS officials said that they will draw on the expertise of other science agencies, such as the National Science Foundation, the National Academy of Sciences, and the Cooperative State Research, Education, and Extension Service. ARS’ national program staff manages the peer review process. Currently, peers are selected by ARS area directors in consultation with both in-house and external experts in relevant fields. Under the revised system, the national program staff will select peer reviewers from candidate pools. The ARS scientist whose plan is reviewed suggests names of scientific peers for consideration. Criteria for all reviewer selections include a high level of subject matter knowledge and accomplishment in scientific research. According to ARS officials, the majority of reviewers come from outside ARS, and the continuous involvement of customers, stakeholders, and ARS managers helps ensure high-quality research. ARS reviews of proposals and all scientific research are based on six assessment criteria: overall scientific value; probability of success; adequacy of approach and excellence of research procedure; adequacy of literature review and the researcher’s knowledge; extent of duplication of other research; and reviewers’ suggestions for improvement. As congressionally mandated, about $4 million of ARS’ annual appropriation is competitively awarded through grants to external scientists to conduct research in specific program areas. These proposals are peer- reviewed by a mix of internal and external peer reviewers. Periodically, ARS area offices organize in-depth location reviews. These reviews are conducted by panels, usually in a workshop format with panel members from ARS, other agencies, academia, and the private sector and generally require several days to complete. Location reviews may evaluate the quality and impact of research performed on several projects. These reviews have multiple objectives and provide input to a variety of management issues such as personnel, productivity, quality, and financial management. In addition, ARS forms ad hoc panels to address more immediate research issues as needed. These panels gather information on specific problems and identify possible corrective actions. ARS requires its scientists to report research results, generally through peer-reviewed scientific journals. ARS research leaders may require that research manuscripts be reviewed by two or more internal or external peers prior to submission to a scientific journal. These reviews are obtained by mail, and the authors must address reviewers’ comments before the manuscript is approved for publication. According to ARS officials, publication in a peer-reviewed journal is a means to ensure the quality and productivity of the research. ARS also uses panels of in-house scientific peers to evaluate scientists’ research accomplishments every 3 to 5 years. The panels anonymously evaluate the scientist’s research contributions by documenting the impact of the research using peer and stakeholder input. The results of these reviews determine a scientist’ grade promotion. An ARS directive documents the panel procedures, including the assurance of diversity in panel selection. Supervisors and line managers also conduct reviews of research. National program staff research managers conduct internal project level reviews to help determine the impact of the research, correct problems, and plan future direction. ARS lead scientists prepare annual project progress reports, which document accomplishments, published manuscripts, and patents. The reports are reviewed by line managers and submitted to the national program staff, and are used to adjust or revise program plans. ARS is now introducing a revised system, which will include an annual summary progress report highlighting major research accomplishments for each of the 23 national programs. ARS plans to place these reports on the Internet and to solicit comments from customers, stakeholders, and scientific peers. The national program staff reportedly spends about one-third to one-half of their time conducting reviews of projects and programs in the 23 major program areas. ARS officials said that the individual scientist’s annual performance appraisal is also a basic tool for assessing scientists’ accountability for the accomplishment of research project objectives. ARS’ program planning process includes setting priorities and ensuring that research is relevant to the agency’s mission. Internal sources and external customers and stakeholders, including the Congress, policymakers, consumer groups, private industry, and academia, provide input to the planning process through national program planning workshops. These reviews help with the development of the agency’s strategic plan and the national programs. ARS also plans to utilize the National Agricultural Research, Extension, Education, and Economics Advisory Board to oversee the implementation of the peer review system and ARS’ priority-setting process. The Congress directs ARS to allocate about $10 million annually to external researchers for specific cooperative agreements. ARS usually does not conduct peer review of this research prior to awarding the funds. However, outside researchers must provide annual progress reports that are reviewed by ARS’ managers. The following presents a description of the Cooperative State Research, Education, and Extension Service’s (CSREES’) peer review and other quality assurance review practices. The Cooperative State Research, Education, and Extension Service, within the U.S. Department of Agriculture, was created in 1995 through a merger of the Cooperative State Research Service and the Extension Service. CSREES funds research to address problems of national and regional importance to agriculture, forestry, and related sciences. The agency’s fiscal year 1999 research budget is about $465 million. About $237 million is designated for noncompetitive formula funding land grant universities. The remainder of the budget funds competitive grants ($150 million) and congressionally mandated special research grants ($78 million). CSREES has no regulatory role, but its research can be used by regulatory agencies. For example, the Environmental Protection Agency may use results from CSREES’ Pesticide Management Alternatives Program to determine if changes should be made to regulations on pesticide use. CSREES does not have a formal definition of peer review. However, the agency is developing definitions of both peer review and merit review as part of an agency initiative studying how the agency will comply with the Agriculture’s new legislative mandates related to peer review. Section 103 of the Agricultural Research, Extension, and Education Reform Act of 1998 (the act) requires “peer review” of all CSREES research grants issued on a competitive basis and “merit review” of competitive extension and education grants but does not define these terms. A CSREES official said that merit review would probably be defined as an evaluation of a project or program to determine its technical quality and relevance to program goals. Peer review will probably be defined as a method for conducting merit review that uses people with qualifications and expertise to conduct research similar to that being reviewed. CSREES funds both competitive and noncompetitive research, but generally external peer review applies only when entities compete for funding. The act requires that all research funded by Agriculturethe Department be peer-reviewed and that a peer review panel assess the merit and relevance of the research at least every 5 years. A CSREES official said that flexibility will be critical in developing methods used to select reviewers and conduct reviews, depending on the types of decisions required and the field of science involved. For example, a goal of the Pesticide Alternatives Program is to determine the most important pesticides and crops to investigate. Therefore, two peer review panels were convened to review proposals: one with experts in the related scientific field and another with experts in policy and in research and funding priorities. All programs that award competitive grants use formal documented peer review methods and select external peers. The $100 million National Research Initiative Competitive Grants Program, which funds high-priority, fundamental, and mission-linked research revised its peer review rules in 1994. A program description that includes peer review-related instructions is now published annually. The competitive grant application process is open to anyone in the United States, including other Agriculture scientists. The requirement to conduct peer reviews of research grant applications is found in 7 C.F.R. part 3411, which specifies the requirements for composition of peer review groups and ad hoc reviewers, including the need to include experts from universities, industry, and private consultants and from a variety of locations. This regulation also lists detailed evaluation criteria in three categories: scientific merit; qualifications of proposed project personnel and adequacy of facilities; and relevance to long-range improvements in, and sustainability of, U. S. agriculture. In fiscal year 1998, the National Research Initiative Competitive Grants Program, CSREES’ largest competitive grant program, with an appropriation of $98 million, spent $4.3 million for its peer reviews of competitive research proposals. An agency official said that peer review is expensive because of the cost of the panel process. The agency pays panel managers on an hourly basis for up to 40 days of work plus travel expenses; panel members receive $150 a day honorarium plus travel expenses. CSREES has legislative authority to use program rather than administrative funds to pay for peer review panel costs. In the case of the National Research Initiative Competitive Grants Program, specific peer review procedures differ among the 27 research areas. Generally, panels of outside experts conduct reviews, but mail and ad hoc reviewers are also used when additional scientists are needed or the subject is specialized. A rotating position of Panel Manager chairs the review panel for each program. These positions are generally filled by outside experts who work for approximately 3 weeks a year. Each panel member reviews about 30 proposals and provides written comments for about 20. Three written reviews are required for any project recommended for award. According to a departmental report on the quality of research,CSREES scientists select panel members and provide leadership by overseeing the review process. The scientists may share administrative responsibilities with panel managers, serve as panel Chairs if there is no outside panel manager, and are to ensure adherence to documented conflict of interest and confidentiality rules. Most peer review panels are exempt from the Federal Advisory Committee Act. For this reason, the panels can reach consensus on their recommendations for funding proposals while maintaining confidentiality. According to a CSREES official, the selecting officials accept the panels’ recommendations with few exceptions. CSREES funds about 20 percent of the applications it receives. A summary and copies of written reviews are sent to the researcher, and only names of awardees are made public. There is no appeals process for applicants that are turned down, but feedback is provided, and applicants can reapply the following year. Section 103 of the aAct requires colleges to peer review all research as criteria for submitting grant proposals. However, a CSREES official said that although many colleges peer review their research, CSREES will continue to conduct peer reviews. When competitively funded research projects are scheduled for renewal either annually or every 2 to 3 years, progress reports are peer-reviewed, and the projects compete for funding with new proposals under the competitive grants merit review process. If the review panel considers the progress unsatisfactory, the project will not be funded. A CSREES official said that 40 to 50 percent of renewals are funded through this process. CSREES internally reviews the noncompetitive research proposals it funds, as well as the progress of research projects and programs. Authority to approve noncompetitive research projects is delegated to CSREES scientists. Academic institutions submit research project proposals that are funded noncompetitively through legislatively established formulas. To comply with the requirement in the Agricultural Research, Extension, and Education Reform Act, grantees are now required to certify that the proposal has received a peer review at the institution level. CSREES scientists will continue to conduct a merit review of the technical quality. In addition, section 212 of the act requires that grantees arrange for peer review of research funded through congressionally mandated special grants. CSREES scientists review the technical merit of the special grant proposals and may obtain views from nationally recognized experts in the research area. If a proposal is found lacking, CSREES staff work with the author to improve the proposal. Department heads and experiment station directors, within the originating region of the proposal, review proposals for regional research projects, which involve collaborative efforts of researchers at several universities and laboratories. Criteria for the selection of regional projects are: significance and scope of problem, level of interdependency and cooperation among participants, quality of science, and the avoidance of unnecessary duplication of research. According to Agriculture’s report, department or program reviews are conducted internally at the request of land-grant institutions, with some outside scientists conducting evaluations of programs. Site visits are made when possible to laboratories and regional project locations. Annual progress reports, which are required by CSREES’ “Special Terms and Conditions” for research grants, are reviewed internally. All grantees are required to provide a final technical report for review when the grant expires. While CSREES has not had a formal systematic process for evaluating research performance, other than through annual progress reports, a CSREES official said that the agency is reporting the performance and outcomes of its research, as required by the Results Act. One reporting concern is that research may not always have a positive outcome, as negative outcomes are part of good science and the learning process. The departmental report states that potentially, the performance report required by the Results Act will provide a mechanism to more systematically monitor and report research performance and accomplishments. University partners provide information on research performance and outcomes that CSREES uses in developing its performance report. Consensus regarding national CSREES funding priorities is developed through input from a wide array of private and public sector stakeholders, including various commodity groups, and associations. In addition, CSREES receives advice from the National Agricultural Research Extension, Education, and Economics Advisory Board. The national research initiatives program receives advice and oversight from its Board of Directors, which meets twice a year. Peer review panels also evaluate programs and make suggestions for revising the annual call for grant proposals. All research, either at the project or program level, is peer reviewed by CSREES or grantee institutions. The following presents a description of the Forest Service’s (FS’) peer review and other quality assurance review practices. The Forest Service, within the U.S. Department of Agriculture, has two primary authorities for conducting research. The first, the Forest and Rangeland Renewable Resources Planning Act of 1974, which authorizes FS to conduct natural resource assessments, provide periodic reviews of its research activities, and to project a research program every 5 years.The second, the Forest and Rangeland Renewable Resources Research Act of 1978, which authorizes the forestry research program. In response, FS conducts primarily intramural research through seven Forestry Research Stations. The stations manage a total of 550 scientists in 163 Research Work Units located at various sites across the nation, including land grant colleges. The agency’s fiscal year 1999 R&D budget is $197 million. Approximately 10 percent of the budget will fund extramural research, which generally is performed through standard research agreements with outside research partners at academic institutions or companies, or cooperative agreements with individual consultants. Under these agreements, which usually span 1or 2 years, FS scientists are substantially involved in conducting the research. Peer reviews are considered to be essential research activities, and FS does not track the costs of conducting the reviews. FS scientists spend about 10 percent of their time serving as peer reviewers for FS research, external journals, and other agencies, including the National Science Foundation. FS defines peer review as technical or scientific review and defines peers as scientists who have scientific expertise in and knowledge of the discipline that is being reviewed. Peers include researchers and administrators from other research stations, FS headquarters staff scientists, university researchers and research administrators, and interest group scientists. Reviewers must be independent of the research under review. They are selected or invited to participate in reviews for which they are deemed best qualified, based on their scientific reputation, standing in their discipline, and specific knowledge of the research area. Peers are sometimes selected to provide training or personal growth opportunities in different programs and issues. Particular attention is paid to obtaining a balanced set of viewpoints and ensuring gender and ethnic diversity, and research administrators are responsible for ensuring that a wide range of peers are used. Although peer review is not explicitly required by statute, agency officials said that peer review requirements documented in FS’ manual satisfy the law’s requirement that FS encourage cooperators and grantees to use the best available scientific skills from a variety of disciplines in and outside the fields of agriculture and forestry. FS officials said that peer review is the foundation of scientific credibility and reputation and the cornerstone of its quality assurance and control efforts. Therefore, all research is peer reviewed. The peer reviews occur at six different points in the research process: preparation of when research work unit descriptions are prepared; when preparation of problem analyses are prepared; when preparation of study plans are prepared; when the Deputy Chief’s makes his or her program reviews and the Chief’s overviews of research stations; when preparation of research publications, articles, and other research products are prepared; and when scientists’ position descriptions are evaluated Research Grade Evaluation Guide evaluations of scientist position descriptions. To help determine research priorities and direction, and which research to perform, FS research work unit descriptions, problem analyses, and study plans are subject to peer review. A research work unit description is a concise summary of the work unit’s mission, problems to be solved through research, reason the problems were selected, proposed research approach, planned accomplishments, and staffing needs. The criteria for setting research priorities or selecting specific problems for study are subjective and depend on the nature of the decision being made. Peer reviews lead to decisions about prioritizing problems and general approaches to solving them, level of resources needed, accomplishments, and likely benefits. The descriptions are revised about every 5 years and are finalized after several iterations of review and comment. Technical assistance visits to research stations provide peer review of the research details. Prior to final approval, the descriptions are submitted to FS headquarters for another peer review. The Station Director then approves the description and the Deputy Chief for Research and Development concurs with decisions contained in the description. The problem analysis contains precise definitions of the research problem, benefits to customers, and the likelihood that the research will provide a solution. The analysis is a detailed plan prepared by a project leader or designated scientist after approval of the research work unit description. The author selects peer reviewers based on his or her knowledge of their interests and capabilities. Following the peer review, an Assistant Station Director for Research or a Program Manager must approve the problem analysis. Last, a study plan is prepared, usually by the scientist assigned to perform the research. The plan defines the research objectives and detailed methods to be applied. The plan is submitted usually by mail, to peers who are selected by the author or project leader. Following peer review, the researcher modifies the study plan as needed needed, and the Project Leader or another research manager approves the plan. The peer review comments and researcher responses are documented with the study plan files. FS does not have any documented instructions regarding conflict of interest in peer reviews and considers this to be an ethics issue. A FS task group is currently developing a policy on professional ethics to ensure against misconduct in scientific research. FS officials said that the peer review process has built in checks and balances to ensure independence. For example, obtaining a minimum of three reviews, or inviting 10 to 20 people to participate in a research work unit description review, ensures that a diversity of views will be obtained. The officials said that often researchers seek out their harshest critics to review their work to get early insight into concerns. Individual researchers are responsible for soliciting peer reviews and deciding how to respond to review comments. These decisions ultimately affect their scientific reputation and standing in their field. Less than 5 percent (or about $1 million) of the extramural budget funds competitive and noncompetitive research grants. For example, grants are awarded to university researchers in response to solicitations for mission-related research. The grant agreements include study plans that are subject to the same review process as is the intramural research discussed above. A study plan can span many years, such as a 45-year study of the impacts of tree- cutting methods. Such long-range plans are reviewed every 3 to 5 years. Program reviews and technical assistance visits help station and headquarters managers reach agreement on research priorities and direction and ensure the research is still needed to address priorities. Program reviews normally cover several research work units, and technical assistance visits usually cover a single unit’s research. All program reviews have peers, including senior external customers, such as state foresters and university deans. In addition, the Deputy Chief’s program reviews and Chief’s overview of research stations evaluate a research station’s overall program and are conducted on a cyclical basis to improve the station’s overall research program results. Seven review topics are addressed during these reviews, with the objective of uniformly evaluating the research program to ensure appropriate quality and quantity of results. According to FS officials, publications play a key role in the quality assurance and quality control process for research and development. Peer reviews of manuscripts prior to publication are part of the process and help ensure that the quality of the research is high and results are significant enough to warrant dissemination. Other indicators of quality exist, such as science citation abstracts, book reviews, publication request rates, book sales by vendors, invitations to speak, and requests for consultations, but these are not substitutes for the peer review of publications. Three peers generally review research products prepared by FS researchers or external research partners. Products include manuscripts for publication in external journals, research papers, research notes, and general technical reports. For extramural research, the necessity to obtain peer review is part of the cooperative agreement. The research partner, the FS, or both may conduct the reviews. For internal FS publications, and publications in external journals that do not require peer review, the names of reviewers are documented on a manuscript approval form, and the author must provide a justification for any reviewer’s comments to which he or she did not respond. A FS official said that users of FS’ research results or members of stakeholder groups also assist in performing peer reviews of various programs. Their opinions on the utility and quality of the results and program priorities help the agency fulfill its responsibilities to improve customer service. Another type of peer review that helps ensure the quality of FS research is the evaluation of individual scientist position descriptions performed by panels of four peers, two from the same general discipline as the position being evaluated and two from a different discipline. Reviews include the nature of the scientist’s research assignment, level of supervision received, originality and creativity required by the assignment, and qualifications and scientific contributions. Personnel specialists select the peers with advice from research administrators. FS’ Manual describes other reviews of stations’ research in addition to peer review. Station Directors and Assistant Directors for Research conduct reviews of a unit’s overall research as necessary, but at least biennially. Informal reviews are conducted with research work units, as needed, to evaluate individual scientist’s programs and improve research results. Under the cooperative agreements, FS monitors the research conducted by its partners by comparing required progress reports with the tasks outlined in the study plan. In addition, the Forest Research Advisory Committee to the Secretary of Agriculture, a formal group of university, industry, and interest group representatives, annually reviews the President’s budget and offers advice on current and future priorities. Research stations often waive the requirement that three individuals review a manuscript prior to publication if the journal has a policy of securing “blind” peer reviews of drafts. Peer reviews are also not required if the research has previously been published in a peer-reviewed journal or FS publication. The following presents a description of the National Institute of Standards and Technology’s (NIST’s) peer review and other quality assurance review practices. The National Institute of Standards and Technology (NIST), founded as the National Bureau of Standards in 1901, is a nonregulatory agency within the Department of Commerce. NIST’s primary mission is to promote U.S. economic growth by working with industry to develop and apply technology, measurements, and standards. NIST carries out this mission through four major programs: the Measurement and Standards Laboratories, the Advanced Technology Program, the Manufacturing Extension Partnership, and the Malcolm Baldrige National Quality Program. The following information focuses on the Measurement and Standards Laboratories and the Advanced Technology Program, NIST’s primary R&D programs. The Measurement and Standards Laboratories in Gaithersburg, Maryland, and Boulder, Colorado, work with industry and government agencies to advance measurement science and develop standards. NIST’s standards for weight, size, volume, and other physical quantities are used to ensure accuracy, fairness, and efficiency for more than $3 trillion worth of goods and services annually. For example, the standards for x-rays, drugs, and DNA testing ensure the safety and efficacy of millions of medical procedures each year. NIST also develops new standards to support advances in semiconductor electronics manufacture, communications, and information technology. In 1988, NIST’s mission was expanded to include new programs designed to further support U.S. industry. One of these programs, the Advanced Technology Program, provides co-funding to companies to initiate high-risk research to develop promising technologies with potential for broad economic and technological benefits across society. For fiscal year 1999, NIST’s budget authority of $641 million includes about $233 million for R&D in the Measurement and Standards Laboratories and about $178 million for external R&D co-funded by the Advanced Technology Program. NIST defines peer review as an evaluation by at least two internal or external technical experts familiar with the technical aspects of the proposals or programs and able to offer informed judgments about the technical quality of proposed external research and of internal research programs. The Department of Commerce’s administrative procedure provides guidance on competitive reviews for the award of external funding in discretionary grant programs, including research grants co-funded by the Advanced Technology Program and external grants awarded by the Measurement and Standards Laboratories. NIST uses a mix of internal and external reviewers to guide decision-making in the award of external R&D grants and to evaluate the technical quality of its internal research programs. The reviews are tailored to the particular purpose and structure of each program. The Advanced Technology Program uses a combination of internal and external reviewers at various stages to determine which research project proposals to fund. In addition, NIST is developing an external program-level peer review for the evaluation of the technical quality of the Advanced Technology Program. In the Measurement and Standards Laboratories, NIST uses external peer review to evaluate the technical quality of the laboratories’ programs. NIST also uses separate internal peer review processes to competitively award the small number of external research grants the laboratories fund each year to supplement their internal R&D efforts. Using a combination of internal and external reviewers, the Advanced Technology Program uses peer review to determine which proposed research projects to fund. NIST’s extramural Advanced Technology Program provides multiyear funding to single companies and industry-led joint ventures. Research proposals are peer reviewed, as specified in 15 C.F.R. 295.4. At the beginning of each competition, a Source Evaluation Board is formed. Board members are government employees, including technical experts and specialists in business and economics from NIST and other government agencies. After the proposals are screened by the Board and determined to be complete and responsive, the Board arranges to have the proposals reviewed by external technical and business experts. Generally, the Board obtains three technical and three business reviews. All board members and reviewers must sign nondisclosure statements, agree to protect proprietary information, and certify that they have no conflict of interest. The reviewers score each proposal numerically and also provide a written critique for Board consideration. The proposals deemed by the Board to be of highest quality are designated semifinalist proposals, and the proposers are invited to NIST for an oral question and answer session with the Board. Following the oral reviews, the Board ranks the proposals based on a number of factors, including the scientific and technical merit of the research and the potential for broad-based economic benefits. The Source Selecting Official, a senior official from the Advanced Technology Program named by NIST for each competition, makes the final funding decisions. The selection officials can deviate from the Source Evaluation Board’s ranking to attain portfolio balance. NIST uses both internal and external peer review to evaluate the effectiveness of its programs. Internal peer review, including program reviews by upper management, is used extensively in all programs. External peer review is used principally to evaluate the technical quality of the Measurement and Standards Laboratories’ programs. Each year since 1959, the National Research Council, an advisory group within the National Academy of Sciences, has coordinated an external peer review of NIST laboratory programs by members of industry and academia. Currently, about 150 scientists and engineers assess the technical quality of NIST’s laboratory programs through the Council’s Board on Assessment of NIST Programs. Panel members visit NIST both individually and in groups, meeting with laboratory management and staff to discuss planned, on-going, and completed programs to determine the technical quality and relevance of the laboratory programs. The evaluation process typically entails a review of specific projects and consultation with individual researchers. Board members use the following criteria: technical merit, appropriateness to NIST missions, effectiveness, adequacy of planning, adequacy of human and physical resources, appropriateness of other agency funding, industrial impact of research, and the integration of the laboratories and NIST missions. The Board annually issues a report on its assessment. The National Academy of Sciences reviews the report according to procedures approved by its Report Review Committee. NIST and the Measurement and Standards Laboratories use the findings and recommendations to improve program quality and to guide decision-making. Subsequent Board assessments and reports consider how NIST has addressed previous findings. NIST’s statutory advisory committee, the Visiting Committee on Advanced Technology, meets quarterly to review NIST’s policies, budget, organization, and programs. The committee, composed of members of industry, academia, and government, is appointed by the Secretary of Commerce. The committee examines the programmatic direction and management of all of NIST’s programs, providing broad external review and guidance. Committee findings and recommendations are summarized each year in an annual report that is submitted to the Secretary of Commerce and transmitted by the Secretary to the Congress. Agency officials also track the results of research efforts funded by the Advanced Technology Program. Using quarterly progress reports and annual meetings with the recipient, the assessments look at the economic impact of the research projects. Program officials monitor completed projects for 6 years. NIST occasionally conducts congressionally earmarked projects that are not separately reviewed. Such projects account for a tiny fraction of NIST’s program funds and are reviewed according to the procedures of the sponsoring program. For example, earmarked projects in the Measurement and Standards Laboratories are reviewed as part of the National Research Council’s Board of Assessment process. Currently, NIST is funding only one earmarked grant. The Board on Assessment does not review the small amount of intramural research conducted under Cooperative Research and Development Agreements. These agreements cover joint research efforts in which both NIST and a cooperating company provide staff, equipment, facilities, and/or funds for a project of mutual interest. These projects can involve intellectual property owned by the private companies and are protected by law from disclosure by NIST. The nonproprietary NIST contribution is subject to the same internal review as all projects and is not excluded from the scope of the annual assessment. The following presents a description of the National Oceanic and Atmospheric Administration’s (NOAA’s) peer review and other quality assurance review practices. Established in 1970, the National Oceanic and Atmospheric Administration (NOAA) is a bureau within the Department of Commerce. NOAA’s mission is to describe and predict changes in the Earth’s environment and to conserve and manage wisely the nation’s coastal and marine resources. NOAA’s research is used to support policy decisions about fishery management, disseminate data about the Earth’s climate, and issue weather reports/warnings. NOAA consists of five major offices: the National Marine Fisheries Service, the National Ocean Service, the Office of Oceanic and Atmospheric Research, the National Weather Service, and the National Environmental Satellite Data and Information Service. The Office of Oceanic and Atmospheric Research, the National Ocean Service, and the National Marine Fisheries Service do much of the research within NOAA. There are 12 environmental research laboratories, 11 joint academic institutes, 29 sea grant colleges, and 6 undersea research centers that conduct research within the Office of Oceanic and Atmospheric Research. The National Ocean Service sponsors research through its coastal ocean research centers and focussed programs with other offices. The National Marine Fisheries Service conducts its’ research through 5 regional science centers in about 30 laboratories. These laboratories are overseen by the National Marine Fisheries Service Science’s Advisory Board, composed of five5 Science Center Directors and the Director of Science and Technology. The National Marine Fisheries Service also has extensive collaborations with academia, many through cooperative agreements and grants. NOAA’s fiscal year 1999 R&D budget is $577 million. NOAA defines peer review as an organized and objective method for evaluating proposed, ongoing, and completed scientific work, by individuals and/or committees who have equal or pre-eminent standing in the pertinent field of research and knowledge of the type of work being reviewed. Virtually all of NOAA’s research portfolio is evaluated by some type of peer review process. Peer review is used at both the project and program levels to certify the correctness of procedures, establish the plausibility of results, and allocate scarce resources. Currently, NOAA has no comprehensive peer review policy. Thus, each office has its own policies, which have accrued over time. For the most part, all external research and internal research is peer reviewed; however, the peer review methods differ among and within the offices. For example, the Office of Oceanic and Atmospheric Research, the National Ocean Service, and the National Marine Fisheries Service conduct their peer reviews by either peer panel, mail, or a combination of the two. Reviewers are selected from a variety of sources, including academia, industry, government, the international science community, and from within NOAA. The Office of Oceanic and Atmospheric Research has two ways that it peer reviews competitive research proposals. If the Office puts out a request for proposals itself, the proposals that are submitted are peer reviewed by the Office and then assigned to individual research universities or institutes funded by the Office. If the research universities or institutes put out a request for proposals, then the individual university or institute peer reviews the proposals submitted. However, each laboratory, university, or institute has its own way of conducting peer reviews within the Office’s guidelines. Research is judged on rationale, scientific merit, innovativeness, the qualifications of the principal investigator, user relationships, relationships to NOAA’s priorities, programmatic justification, linkages, and costs. In the National Ocean Service, competitive research proposals are peer reviewed in a two two-step process. First, the proposals are distributed to knowledgeable individuals for anonymous review. Second, a review panel of knowledgeable individuals are provided of knowledgeable individuals are provided the proposals and reviews for discussion and separate ranking by each panel member. Proposed research is judged on scientific rationale, technical merit, qualifications of the researchers, and the cost of the proposed work. Within the National Marine Fisheries Service, each proposal’s peer review method depends on the products or programs reviewed and is developed by its sponsoring organization—a particular fisheries laboratory, a Regional Fishery Management Council, and/or the peer review panel itself. The schedule or format may also depend on whether or not the proposal is legislatively mandated. Within the Service, proposals are reviewed by mail, peer panels, committees, and agency staff. Every 3 to 5 years, qualified peers review the research programs of research universities, laboratories, or institutes in the Office of Oceanic and Atmospheric Research. This review evaluates the programmatic accomplishments and impacts in the context of resources invested. Each set of reviewers visiting the site consists of representatives from the program, stakeholders, external technical experts, or members of established Office of Oceanic and Atmospheric Research review panels. The university, laboratory, or institute reviewed develops the review presentation based on Office of Oceanic and Atmospheric Research criteria. Most of NOAA’s research is published in peer-reviewed journals. Principal scientists at each laboratory are evaluated in large part on their publications in peer-reviewed journals. The National Marine Fisheries Service’s Scientific Publications Office publishes two peer-reviewed journals of fishery science work performed by NOAA and non-NOAA scientists. Scientific publications undergo an internal technical review by two management levels within the Service. Additional peer review may be requested during the internal technical review. NOAA has recently established a Science Advisory Board that is officially chartered in accordance with the Federal Advisory Committee Act. The 15-member Board is composed of eminent scientists, engineers, resource managers, and educators reflecting the full breadth of NOAA’s responsibilities, as well as the ethnic and gender diversity of the United States. The Board advises NOAA’s Administrator on long- and short-range strategies for research, education, and the application of science to resource management. The Board is currently considering its potential role in the oversight of NOAA’s various science review panels. The Board’s operations and support functions are within the Office of the Chief Scientist. NOAA’s Office of Research and Technology Applications conducts the technical review and selection process for applicants to the Small Business Innovative Research Program. The program was established, among other things, to foster participation by minority and disadvantaged researchers. A certain portion (2.5 percent in fiscal year 1999) of NOAA’s external research and development budget is set aside for program funding. The review is performed by a variety of experts whose views are collected by mail. Scientific findings and conclusions of the National Marine Fisheries Service programs are sometimes reviewed by opposing parties in court when fisheries management decisions are litigated. Since all NOAA research is evaluated by some type of peer review process, agency officials provided no examples of other quality assurance reviews. All NOAA research is subjected to peer review or other technical reviews by NOAA or its customers. The following presents a description of the U.S. Department of Energy’s (DOE’s) peer review and other quality assurance review practices. Created in 1977, DOE’s mission is to foster a secure and reliable energy system that is environmentally and economically sustainable, to be a responsible steward of the nation’s nuclear weapons, to clean up its facilities, and to support continued U.S. leadership in science and technology. The agency conducts research and development on a variety of topics, including fossil, fusion, and nuclear energy production; energy conservation; renewable energy; biological and environmental research; materials science; engineering and geoscience; advanced computing; high-energy and nuclear physics; nuclear waste management; environmental remediation; radiation; nuclear stockpile management; nuclear nonproliferation; and the Human Genome Project. DOE’s research can affect a broad spectrum of federal policies and regulations. For example, DOE generates federal energy-efficiency rules for the manufacture, testing, and labeling of major home appliances and certain commercial products. The Environmental Protection Agency’s Office of Radiation Protection and the Nuclear Regulatory Commission have used the results of DOE’s research as part of the background used to set radiation standards. In addition, agency research was used to set standards for mobile pollution sources and fuel regulations under the Motor Vehicle Information and Cost Savings Act. DOE’s research and development budget for fiscal year 1999 is $7.8 billion. Approximately 80 percent of the budget will support research, research facilities, and related activities within the Department and its national laboratory system. The remaining 20 percent will support external research conducted by industry, universities, public and private research institutions, not-for-profit organizations, and research and development consortia through Department-awarded grants, cooperative agreements and contracts, and laboratory-awarded research subcontracts. Because of its diversity, DOE’s peer review practices are guided by a variety of laws and regulations. The Federal Acquisition Regulation, the DOE Acquisition Regulation, and the Competition in Contracting Act guide the agency’s peer review practices for research and development contracts. Research grants and cooperative agreements, which are awarded through a merit-based selection process, follow the Department’s Financial Assistance Rules, as promulgated in the Code of Federal Regulations 10 C.F.R. Part 600. DOE has no formal definition of peer review, but practices peer review as merit review with peer evaluation—a formal, competent, and objective evaluation process using specified criteria and the review and advice of qualified peers. Peers must be technically competent in the scientific or technical field under review and must be free from conflict of interest. Peers may come from any source, including industry, academia, private and nongovernmental institutions, government agencies, and their associated laboratories. DOE uses merit review with peer evaluation to guide research direction and to assess research progress. External research is peer-reviewed in conjunction with the preaward competitive selection process. This research is also reviewed as part of the award renewal process. Reviews of laboratory research occur at both the laboratory and departmental oversight levels. In addition, laboratories, user facilities, and major research divisions have committees of outside experts that provide periodic peer reviews of research relevance and quality. Research results are also extensively published in peer-reviewed journals. The methods for conducting reviews are tailored to each situation. The following provides examples of the different peer review practices among DOE’s programs. With few exceptions, merit review with peer evaluation guides DOE research, including that by its research laboratories. For example, regulations governing the Financial Assistance Program require peer review and competitive selection. The regulations specify that each grant proposal normally receive a minimum of three reviews per proposal by technically qualified experts in the proposed field, followed by a peer review panel. Proposals are peer-reviewed for scientific excellence. The Office of Science & Technology, in the Environmental Management Program, Project Selection Reviews, for new research and development activities, combine the judgments of technical peers and potential users of the results. In addition, research subcontracted by DOE’s national laboratories to outside researchers is governed by contract provisions, unless otherwise justified through formal documentation. These provisions require competitive selection processes, including merit review with peer evaluation. Peer review is applied to the selection and approval of most laboratory field work proposals. Field work proposals are the means by which the laboratories formally propose future work and seek authorization for expending research and development funds. In the Office of Science, all field work proposals are required to be peer-reviewed for quality by external, independent experts. Each laboratory research program is reviewed annually. For example, the Technology Development Program of the Office of Environmental Management uses teams of subject matter specialists from technical, regulatory, business, and stakeholder perspectives. In addition, peer review is used to allocate available time and to select the experiments conducted at specialized research facilities located at DOE’s laboratories. Such facilities include accelerators for the study of high-energy physics and the world’s most powerful computers and lasers. At the laboratories, each director’s discretionary research and development program and the laboratory field work proposals are reviewed. The Laboratory Directed Research and Development Program provides certain laboratory directors discretionary funds (up to 6 percent of their laboratory’s budget) to develop new scientific ideas and opportunities and to initiate new directions. The laboratories rely on individual scientific investigators and the scientific leadership of the laboratory to identify opportunities that will contribute to scientific and institutional goals. Peer review is also used in conjunction with the evaluation of on-going research. While the substance of the reviews is similar, such as considering the quality and relevance of the research and the investigator’s or research group’s record of accomplishment, the nature of the reviews can differ. For example, the Office of International Health Programs uses independent, external review panels to conduct in-progress reviews. The Office of Science & Technology within the Environmental Management Program conducts technical reviews of continuing projects in their third year of support or when reaching engineering demonstration, or when considered a new start, through a formal process externally managed by the American Society of Mechanical Engineers. The Society selects reviewers who assess technical excellence, relevance, progress, and productivity. In addition, for new environmental-management technologies, mid-year progress reviews are held annually for each program element, with potential users assessing the applicability and performance requirements. Publication in open literature constitutes another form of peer review. Publication of original work is considered essential at DOE, and the scientists it supports (both external and internal) are continually evaluated by the quality of their original research, as indicated, in part, by publications in archival, peer-reviewed journals. Retrospectively, scientists who are independent of the laboratory conduct reviews of laboratory research in conjunction with program reviews and advisory committee oversight. These reviews provide advice on the quality, relevance, and productivity of laboratory-conducted research. The following are three examples of such reviews. The Office of Science regularly conducts retrospective peer reviews of research and development programs throughout the Department, which includes an evaluation of a sampling of research projects. Individual programs also conduct reviews. . The Office of Defense Programs uses an Inertial Confinement Fusion Advisory Committee, constituted under the Federal Advisory Committee Act, which reports directly to the Assistant Secretary for Defense Programs, to assess program results. For highly classified research, the Department interacts with the Department of Defense for customer feedback on program performance. The Office of Civilian Radioactive Waste Management uses peer review to help assess the quality and validity of completed technical work and to ensure the quality of data for use in adjudicatory hearings. Because of the U.S. Nuclear Regulatory Commission’s role under the Nuclear Waste Policy Act, the Commission has provided guidance on the conduct of peer review. A primary selection criterion for peer reviewers is independence. When there is a potential or an apparent conflict of interest that may bring the independence of a participant into question, a documented rationale is included in the peer review report. Many of DOE’s energy technology development and related research and development programs are deliberately designed to accommodate industrial partners. In various ways, these industrial partners provide opportunities for external merit review by engaging themselves as full participants in planning, executing, and commercializing the research and development. Such reviews extend beyond the peer review procedures that characterize science programs. For most major technological development programs, the formulation and enforcement of a comprehensive Quality Assurance Program is required. For the Energy Efficiency Program, quality control involves three stages: peer review for basic research, merit review for applied research, and market review for judging commercial application. Under reforms begun in 1994, all of the Department’s new contracts for the management and operation of its national laboratories require regular, performance-based merit reviews of the contractor’s performance. Colleagues, laboratory superiors, and administrators at DOE headquarters evaluate the research and development projects. The 9 nine multiprogram national laboratories also have various industrial advisory panels to review research. In addition, all research subcontracted by the laboratories to outside researchers is governed by contract provisions that generally require periodic evaluations of the subcontractor’s performance. Panels constituted under the Federal Advisory Committee Act frequently advise DOE program administrators on program content, quality, future directions, and priorities. For example, the Office of Science uses advisory committees for recommendations on basic energy sciences, biological and environmental research, high-energy physics, nuclear sciences, and fusion energy. Similarly, the Office of civilian Radioactive Waste Management has standing advisory committees and just completed a 2-year participatory peer review. For classified nuclear weapons design-related research, where no broad industrial, university, or other independent source of expertise exists, a process of merit review exists within DOE’s Defense Programs laboratories. For example, every 5 years, with annual updates, the three Defense Programs laboratories review the nuclear weapons in the active stockpile through a formal internal peer review “Weapons Appraisal Process.” The University of California, the contractor that operates the Lawrence Livermore and Los Alamos laboratories, also uses a President’s Council Panel on National Security to assess the nuclear weapons program. Each of the laboratories’ directors also appoints review committees for each of the laboratories’ divisions, with members coming almost exclusively from industry and academia but sometimes from DOE and its contractors. The committees report to the laboratory directors with an assessment of the division’s technical and scientific quality. The directors, in turn, file a self-assessment with a review Council convened by the President of the University of California. From this process, the president reports to DOE on the laboratories’ technical and scientific quality. Finally, additional reviewing bodies such as JASON (a civilian science advisory group), the National Academy of Sciences, the Nuclear Weapons Council, and other senior advisory groups review DOE’s Defense Program’s research and development program. According to DOE officials, most congressional mandates and earmarks, which designate projects and the institutions to conduct them, are not subject to the peer review process in deference to the congressional directives. However, once a grant is funded, it is likely to receive merit review before being competitively renewed, unless waived with a written determination by the project administrator. When merit review is not conducted before an award’s renewal, the award must be considered to be noncompetitive and must meet different selection requirements.Whenever the merit review system is not used for applications and proposals, the Director of Grants and Contracts must obtain written prior approval for a different review procedure. Very rarely are contracts peer-reviewed when sole-source selection is used, but the administrator making this decision must justify this process. In addition, nonreviewed grants cannot be extended for more than 6 years; periodic reviews of the research results are another check. The following presents a description of the Environmental Protection Agency’s (EPA’s) peer review and other quality assurance review practices. The mission of the Environmental Protection Agency, established in 1970, is to protect human health and to safeguard the natural environment. EPA is organized into 10 geographic regions and nine major offices: Administration and Resources Management; Air and Radiation; the Chief Financial Officer; International Activities; Policy; Prevention, Pesticides, and Toxic Substances; Research and Development; Solid Waste and Emergency Response; and Water. EPA’s fiscal year 1999 operating budget is about $547 million, excluding state revolving funds for the design and construction of waste water and drinking water systems. Peer review is defined as a documented critical review of a specific, major scientific or technical work product, conducted by qualified, independent individuals. EPA distinguished this peer review definition from what it classifies as “peer input” which is interaction during the development of an evolving agency work product, to provide an open exchange of data, insights, and ideas. The difference between peer review and peer input is the independence of the peer reviewers and their level of involvement. Peer reviewers should have no material stake in the proposal reviewed and should have had no substantial involvement in the development of the proposal. Generally, all research funded by the Office of Research and Development is peer peer-reviewed, as legislatively mandated. In 1993, the administrator issued a policy that statement that major and scientifically and technically based work products related to major agency decisions should normally be peer-reviewed. In response to one of our 1994 reports, each assistant and regional administrator developed a set of standard operating procedures for peer review. In June 1994, the current administrator reissued to a policy statement reaffirming the central role of peer review to ensure that EPA policy decisions rest on sound, credible science and data. After one of our 1996 reports found that EPA’s implementation of these peer review procedures remained uneven, EPA’s Science Policy Council issued a peer-review practices handbook for agencywide use in February 1998. This handbook concentrates mainly on guidelines for the retrospective review of work products used to support EPA decisions. The following provides examples of the different peer review practices among EPA’s offices. The Office of Research and Development’s Science to Achieve Results Program issues requests for research proposals. These proposals are selected through a rigorous peer review process in which panels of independent researchers from relevant fields review all proposals. The reviewers use evaluation criteria that emphasize the quality of science as well as the responsiveness to the program request. Proposals that are rated very good or excellent by the panels are subjected to a programmatic review within EPA to ensure a balanced research portfolio. Office for Research and Development scientists and Program or Regional Office staff conduct the programmatic reviews, based on their knowledge of program priorities and how the research proposals complement the intramural research program. The reviewers recommend proposals for funding to the National Center for Environmental Research and Quality Assurance. Research proposals not under the Science to Achieve Results Program, such as congressional earmarks and unsolicited research proposals, are also peer reviewed for technical merit. For EPA contracts and other assistance agreements, the approach used to peer review a major scientific or technical work is left up to the peer review leader and the EPA’s decision maker who base their decision on the nature of the topic and the intended final product. The Office of Research and Development has established the Board of Scientific Counselors, composed of nationally-recognized scientific and engineering experts. The primary functions of the board are to evaluate office science and engineering research programs, laboratories, and research management practices; recommend actions to improve their quality and strengthen their relevance to the mission of EPA; and to evaluate and provide advice concerning the utilization of peer review within the office to enhance the quality of science in EPA. In addition, the Science Advisory Board—a Federal Advisory Committee Act committee with deliberations open to the public—functions as a technical peer review panel, providing consultation and advisory reviews of work products at various points prior to their completion. Consisting of 10 committees covering such topics as health, radiation, air quality and drinking water, the board draws scientists, engineers, and economists from academia, industry, and the environmental community. EPA has always encouraged the publication of its scientific and technical material in peer-reviewed literature as a means of obtaining independent, external review of its work products. The Science Advisory Board often evaluates and reviews the technical basis for a science policy position adopted by EPA. For example, the board sometimes serves as a council of peers in cases where action is necessary in order to address emerging environmental risk before all the rigors of scientific proof are met. The board also reviews selected scientific documents that are used as the basis for environmental regulations. Important, major EPA rulemakings, including those determined as “significant” ($100 million impact or more) by the Office of Management and Budget, lend themselves to extensive external peer review. Generally, more extensive and involved peer review with external peers is indicated for work with more complex science, greater cost implications, or a more controversial issue. Other projects that are under strict time constraints, are of lesser impact or less controversial, may be reviewed internally, or by a combination of internal and external peer review. Group discussions among the reviewers can be very helpful, but individual reviews are easier, faster, and less expensive and may be more appropriate at the early stages of a product’s development or for products with less impact and complexity. The Office of Research and Development tracks how many peer reviews are conducted each year across EPA and does a qualitative review of whether or not the reviews were conducted according to EPA policies. Managers and peer review leaders are also expected to document the results of completed peer reviews. These results, in conjunction with discussions during the peer review process, are intended to help EPA to ensure that the scientific and technical support for its decisions withstand independent scrutiny. EPA’s Science Policy Council Peer Review Handbook recognizes some circumstances that might preclude the peer review of a major work product. These include products prepared using widely accepted methodology; those for which the regulatory activity has been terminated; those with court-ordered deadlines that may limit or eliminate time for an adequate peer review; those for which resources to conduct reviews are limited; and products that were previously reviewed, if a new application does not differ significantly from the original application. The following presents a description of the National Institutes of Health’s (NIH’s) peer review and other quality assurance review practices. Begun in 1887, the National Institutes of Health (NIH) has grown to become one of the world’s foremost biomedical research centers and the federal government’s focal point for biomedical research in the United States. NIH is one of eight agencies in the U.S. Department of Health and Human Services’ Public Health Service. Comprised of 18 Institutes, 6 Centers, and a Library, NIH has 75 buildings on more than 300 acres in Bethesda, Maryland. NIH’s mission is to uncover new knowledge that will lead to better health for everyone. To this end, NIH conducts research in its own laboratories; supports the research of nonfederal scientists in universities, medical schools, hospitals, and research institutions in the United States and abroad; helps to train research investigators; and fosters the communication of biomedical information. Research ranges from basic understanding of biological processes and the human genome to clinical trials for ways to control infectious diseases and tests of dental sealants for children’s teeth. NIH research is used by regulatory agencies such as the Food and Drug Administration, whose mission is to ensure that food, drugs, and medical devices are safe. In fiscal year 1999, NIH’s budget for R&D is $14.9 billion. About 82 percent of the budget will be spent on grants, contracts, or similar awards to organizations outside the agency. These awards comprise the extramural, or external, research program. The remainder of the budget supports NIH’s intramural research and administrative support for both the extramural and intramural programs. Peer review is conducted under authority of the Public Health Service Act, as amended. More specific policies and procedures are outlined in the U.S. Department of Health and Human Services and Public Health Service regulations, the Public Health Service Grants Administration Manual, the Public Health Service Grants Policy Statement, and NIH’s manuals and handbooks. The Public Health Service Act states that members of peer review group shall be individuals who by virtue of their training or experience are eminently qualified to give expert advice on scientific and technical merit. The act also includes provisions to ensure that there is no conflict of interest among the reviewers. Agency officials said that peer review includes the expert evaluation of scientific merit by independent reviewers. Almost all research funded by NIH is peer- reviewed. Panels of outside experts review extramural research projects for scientific merit prior to funding the research project. Institute directors have the legal funding responsibility for both extramural and intramural research. Outside reviewers generally review the intramural research program retrospectively. Advisory council and institute/center staffs also conduct numerous program-level reviews. The Center for Scientific Review, formerly the Division of Research Grants, is the focal point for the conduct of peer review for the external research program. Referral officers review the contents of some 10,000 applications each grant cycle and, using written guidelines, assign an application to a Scientific Review Group. These review groups judge a proposal’s or project’s scientific and technical merit, assign priority scores, and make budget recommendations. The specific criteria used to assess the merit of research project applications vary with the types of applications reviewed. Criteria for grants include significance, approach, innovation, investigator, and environment. Criteria for contract projects and proposals include significance, availability of technology and resources, anticipated practical uses of the results, and adequacy of the methodology. In addition, NIH policy requires that all applications are reviewed for the adequacy of their plans to include, recruit, and retain both genders, minorities, and children as research subjects and the adequacy of proposed protection for humans, animals, or the environment. To ensure independence, almost all peer review is performed by outsiders. The act and NIH regulations require that no more than 25 percent of reviewers are from within the agency; the average is only about 1 percent. The Scientific Review Groups are generally composed of 18 to 20 individuals, primarily from academia, to review as many as 60 to 100 proposals. The Scientific Review Administrator recommends and the Director of NIH appoints review group members from among the active and productive researchers in the biomedical community to serve for multiyear terms. Criteria for selecting the reviewers include demonstrated scientific expertise, a doctoral degree or its equivalent, mature judgment, balanced perspective and objectivity, an ability to work effectively in a group, an interest in serving, and an adequate representation of women and minority scientists. Membership is frequently supplemented by temporary members and written outside opinions. When a proposed research topic does not match a review group’s specialties, or when an application sent to the appropriate review group might create a conflict of interest, NIH may convene a Special Emphasis Panel to conduct the review. The Scientific Review Groups usually meet together in person three times a year for 2 to 3 days but sometimes use teleconferencing. Mail reviews are used only as an adjunct to a full panel review. NIH officials said that by definition, peer review recommendations are considered as advice only. However, while other factors such as maintaining a variety of research topics and the need to support newly emerging areas of science are considered, most extramural awards follow peer review recommendations. Of about 40,000 grant applications submitted to NIH each year, up to 30 percent are funded. For each institute, a National Advisory Council mandated by the Congress meets three to four times a year to conduct second-level reviews of all eligible grant applications. As mandated by the Congress, these advisory groups typically include about two-thirds outside scientists and one-third lay members, such as lawyers, economists, and members of patient and disease advocacy groups. These councils may also have ex officio members representing other federal agencies. Councils make recommendations to the Institute director about funding particular meritorious grants that are seen as very important but which may not have received the best scores from the scientific reviewers and, in general, ensure that the scientific peer review process has been conducted appropriately. The need to continue funding projects over multiple years is an important criterion when deciding to fund new projects. In any given year, only about 25 percent of the total funds allocated for research projects is available for new projects that may change the course of a line of research or move research into an entirely new area. Peer review of ongoing external projects occur at the time a request is made for renewed support. The Scientific Review Groups evaluate these efforts with the new proposals. Boards of scientific counselors reviews the technical and scientific quality of each institute’s ongoing intramural research. Like the other NIH advisory committees discussed previously above, the boards are established under the Federal Advisory Committee Act to review, discuss, and evaluate institute, center, and division research programs, projects, and investigators. The boards meet two or three times a year. The boards also review and evaluate tenured NIH investigators at least once every 4four years and tenure-track scientists must be reviewed mid-point in their tenure-track and prior to conversion to tenure. In addition, the boards may choose to review the work of permanent staff scientists or other intramural scientists. Chosen mainly from outside the government, board membership includes internationally recognized authorities in one of the fields of research under review. For continuity, NIH policy states that members generally serve for overlapping 5-year terms, if possible. NIH administrators encourage scientists they fund externally or employ internally to publish the results of their work. Abstracts, manuscripts, or written material by employees must be reviewed and approved prior to publication. Typically, this process also entails further peer review by the journal. NIH administrators can follow citation indexes, in which peer-reviewed articles that are cited are compiled, in order to gauge the relevance and success of funded researchers. To review agency policies, each institute convenes national advisory councils with members from the public and from the medical and scientific communities with expertise relevant to the NIH’s missions. These councils may also review and comment on special initiatives proposed by the institute or, for example, on research training policies. In addition, the boards of scientific counselors provide evaluation and advice on scientific directions of the laboratory, tenure actions under consideration, resource allocation, specific projects projects, including new areas of development, and other administrative matters. In 1996, the NIH Director created a Peer Review Oversight Group to advise on the development and implementation of trans-NIH policies to ensure that the review processes keep pace with current advances in research. NIH also conducts reviews of its directors, the senior executives who manage the agency’s institutes and centers. In each institute and center, Scientific Directors supervise research. An ad hoc committee with at least four members reviews each Scientific Director’s performance every 4 to 6 years. This committee is established by and reports to the Institute or Center Director’s Advisory Council or Board, which, in turn, makes recommendations to the Institute or Center Director. Membership on the ad hoc committees varies and can include an Advisory Council or Board member, a former Institute intramural scientist, a senior scientist, and a scientific administrator. Occasionally, administrative supplements–additional funds necessary to carry on approved work–are not peer- reviewed. However, the project and its administrative supplement are reviewed if the project is continued beyond its original schedule, at which time it competes with new projects. The following presents a description of the United States Geological Survey’s (USGS’s) peer review and other quality assurance review practices. The United States Geological Survey (USGS), as the principal research agency of the Department of the Interior, provides biological, geologic, topographic, and hydrologic information that contributes to the wise management of the Nation’s natural resources and promotes the health, safety, and wellbeing of the people. USGS has four research divisions: Biological Resources, Water Resources, Geologic, and National Mapping Program. The divisions support research conducted by USGS scientists, and through competitive grants to external scientists. USGS provides information resulting from its research that consists of maps, databases, and descriptions and analyses of the water, energy, biological, and mineral resources, land surface, underlying geologic structure, and dynamic processes of the Earth. USGS’ fiscal year 1999 R&D budget is $567 million. The four divisions’ research spending for fiscal year 1998 was about $545 million with about $507 million spent on intramural research. The USGS defines peer review as scientific and technical review by qualified scientific or technical experts in the relevant discipline(s). Individuals chosen as reviewers are recognized experts in the appropriate field. However, reviewers are not to be involved in the preparation, development, or execution of the program, project, or product being reviewed. USGS policy provides for peer review of all research. In early 1996, a draft Department of the Interior policy called for each Bureau to develop a peer review policy. In response, USGS implemented a Bureau-wide peer review policy. At that time, the USGS divisions had review policies that varied in content and scope. The divisions were directed to submit proposed review policies or guidelines for evaluation by USGS’ Program Council and approval by the Policy Council. The “USGS Review Policy” covers (1) subjects for review; (2) review definitions; (3) the management of reviews; (4) timing, frequency, and documentation of reviews;, and (5) the departmental peer review guidelines. USGS directed that the peer review policy be applied to all aspects of the acquisition, interpretation, application, and dissemination of scientific and technical information. USGS’ peer reviews are conducted to select among competitive grant proposals and intramural projects, to evaluate a proposal’s application to questions of public policy, and to ensure the technical quality and relevance of research at all stages of development. USGS’ guidance to the divisions allowed them to manage reviews in different ways and at different stages, depending on the subjects under review, and required them to retain independence and rigor in the review process. In general, managers who supervise research programs are responsible for ensuring that reviews are conducted and the findings are documented. The four divisions have each developed different peer review practices that reflect USGS’ guidance. USGS’ policy states that scientific proposals for new projects and programs or for significant changes in existing projects and programs will be subject to peer review or other review as appropriate prior to final approval and implementation. Reviews must comply with the Department-level guidelines. The guidelines present factors that indicate the need for peer review, including large commitments of funds to new or extended research projects, communications regarding scientific subjects of a controversial nature, and major research projects performed for USGS under contract. Determining the need for peer review is the responsibility of the project or program supervisor. When peer reviews are conducted, in most cases, reviewers should be external to the bureau and should include two or more peer scientists. Officials of the Biological Resources Division, Water Resources Division, and the Geologic Division said the divisions review all research proposals prior to funding. The National Mapping Program is planning to implement a process to review all proposals for its research program beginning in fiscal year 1999. Generally, the criteria applied by divisions to select reviewers for project and program level reviews are knowledge and expertise in an area. Divisions’ evaluations of the research criteria include determining whether objectives are met, results are relevant and timely, and evidence is adequate. USGS guidance stipulates that subsequent periodic review of projects and programs are to be conducted. Ongoing programs are to undergo an independent external peer review about every 5 years, with the number and mix of internal and external reviewers determined by the individual responsible for the review. Each division determines the specific frequency of reviews for ongoing projects. The divisions’ implementations of the review guidelines have resulted in some deviations among reviews of projects. All four divisions review research projects and programs while in-progress, but the timing of the reviews differs. For example, all Geologic Division projects are peer-reviewed annually after the work has been undertaken, and each program is subjected to formal external review every 5 years by groups such as the National Academy of Sciences. The National Mapping Program conducts quarterly project reviews. The Water Resources Division has a standing contract with the National Research Council for broad reviews of the division, which address different subjects each year. These reviews result in recommendations for changes to program direction and improvement. Scientific products require the Director’s approval prior to general release and a peer review, if appropriate, is required. Qualified, technical experts in the relevant discipline(s) review research products prior to dissemination. For example, the Water Resources and the National Mapping divisions require review of manuscripts by two peer colleagues. Procedures, implemented by the Divisions for peer review of publications, call for review by scientists who do not have any involvement in the project. The Geologic and Water Resources Divisions also hold authors of reports and other products responsible, through the peer review process associated with the release of publications, to interact with fellow scientists in addressing comments, suggestions, and criticisms. The “USGS Review Policy” identifies three types of reviews that are to be conducted in addition to peer review: management review, policy review, and editorial review of publications. USGS supervisors and managers conduct the management reviews of programs, proposals, projects, products, and customer information. For example, these reviews are used to select and approve peer reviewers and types of reviews; ensure technical soundness, relevance, and priority; confirm that objectives are met; ensure proper animal care and use; and verify that the media, style, technical level, and content of publications fit the intended audience. Policy reviews include review of programs, proposals, projects, and products to ensure compliance with pertinent policies and mandates prior to receiving “Director’s Approval” for general release of research products. Editorial reviews conducted by qualified personnel check for expression, clarity, mechanical condition, organization and layout, effective and appropriate presentation and use of illustrations and tables for the intended audience, completeness of annotation for references, and, if appropriate, conformity with USGS’ style, usage, and format. The divisions also implement additional reviews. For example, the Biological Research Division has administrative, national program, and science center reviews in addition to the cited USGS reviews. Most Biological Resources Division programs and projects receive annual and or “mid-point” management and technical reviews. The Water Resources Division and National Mapping Program conduct numerous reviews of programs and technical disciplines that include biannual and 3-year reviews. The National Mapping Program does not require peer review for directed research projects being performed for an external customer. However, such research is reviewed for its relevance to the goals and objectives of the National Mapping Program. The customer reviews the projects, and the contract or other written agreement establishes the review criteria. All USGS research is subjected to peer review or other technical reviews. The following presents a description of the National Aeronautics and Space Administration’s (NASA’s) peer review and other quality assurance review practices. The National Aeronautics and Space Administration (NASA) conducts research in earth science; space science; life and microgravity sciences, and applications; and aeronautics and space transportation technology. Within these program areas, 10 NASA centers fund research. NASA’s fiscal year 1999 R&D budget is $10.1 billion. In fiscal year 1998, NASA spent $3.5 billion of its $10.5 billion budget on research funded through grants, cooperative agreements, and contracts with other agencies, industry, and academia. The remainder of the R&D budget funded procurement contracts for nonscientific investigations such as the development of hardware for the international space station, communication satellites, shuttle modifications, and facilities construction. The funding mechanism is determined by the research goal. For example, NASA officials said that a contract agreement is appropriate for a product or service that will be used by NASA. Grants and cooperative agreements are appropriate when research is conducted to accomplish a public purpose. NASA’s research may be used to support other agencies’ policy decisions, such as those related to regulations of the Environmental Protection Agency or Federal Aviation Administration. NASA defines peer review as scientific evaluation by an independent in-house specialist, a specialist outside NASA, or both, of proposals submitted in response to NASA research announcements, announcements of opportunity, and cooperative agreement notices. Peer Review is also used to evaluate unsolicited proposals. Peer reviews evaluate relevance to NASA’s objectives, intrinsic merit that includes scientific or technical merit of research methods, the researcher’s capabilities and qualifications, and cost. All NASA research, including research resulting from unsolicited proposals, is subject to peer review, and peer review is conducted primarily to award funds on the basis of scientific merit. NASA officials said that external peer review is essential for high-quality, relevant research. NASA’s Federal Acquisition Regulations Supplements dictate that peer review will be the method used to evaluate and select research for funding. NASA is developing a series of instructions on the implementation of peer review. Currently, each NASA office works with its own contractors that arrange and manage the logistics of peer reviews. Policies for peer review are documented through the instructions in the research announcements, which are written by NASA scientists. NASA’s Sponsored Research Business Activity, which is responsible for implementing research business policies and procedures, has a competitive procurement effort underway to obtain a single contract to manage the logistics of peer reviews to gain consistency among programs in the reviews of proposals. NASA’s offices now have five different contracts providing logistical support for peer reviews. NASA also has efforts underway to develop uniform instructions for submitting research proposals and to increase uniformity in development of research announcements. Intramural research conducted by NASA scientists is normally funded through awards resulting from the same open competitive solicitations that are used to select extramural research. Research proposals from NASA centers and other federal laboratories are considered together and treated identically to those from industry and academia. Specific peer review methods differ somewhat among NASA offices or disciplines depending on, for example, cost, resources brought to bear, or the experience of the reviewers. Some offices have standing committees, some ad hoc committees, and some a combination of these for individual announcements. However, the use of NASA experts to evaluate and document findings of proposal reviews is universal across NASA. Peers include scientists from public and private academic institutions, industry, government laboratories, and foreign countries. Criteria for peer selection include the research they have conducted, publications, knowledge and experience, and ability to conduct an impartial review. NASA and peer review support contractors maintain databases of discipline experts to identify peers. Acknowledged experts in a discipline and proposal authors may also suggest other qualified reviewers. The officials said that using external peers ensures fresh view points, alternative perspectives, and state-of-the-art understanding. The authors of a proposal are not involved in its review, and peers are screened to ensure that they have no conflict of interest. Reviews are conducted by mail or by panel meetings, depending on the logistics specific to the review. Mail reviews are conducted to allow for the selection of reviewers with very specialized expertise on technical and scientific issues and technical approach. Often a panel review is conducted to reconcile differences among mail reviews and put the proposed research in a larger scientific and programmatic context. In fiscal year 1998, five NASA research offices released a total of 57 research announcements and received 5,048 proposals. Of these, 3,778 proposals were peer-reviewed by mail, and 136 panels met with a total membership of 1,287 peers. An official of the Office of Space Science, which received 2,599 proposals, said that his office held 91 peer review panel meetings throughout the year. Each panel consisted of 7 to 10 members. NASA receives a few unsolicited proposals, and these are usually peer-reviewed only by mail. Panel members are solicited by telephone or letter and are reimbursed for their travel costs to attend panel meetings. NASA officials said that peers working on a voluntary basis contribute to the integrity of the process. Specific criteria for reviews are unique to each research solicitation, but the criteria should be modeled after basic evaluation factors, including relevance to NASA objectives, intrinsic merit, and cost. Panels do not apply overall scores, but rate each proposal on scientific and technical merit using a 5-point scale. In addition, NASA program goals and objectives, and cost-effectiveness of proposed budgets are factors in the review process for individual proposals. However, a NASA official said that there are wide variations among NASA offices in the extent that they rely on peer review panels to assess these factors. Panels are sometimes asked to recommend assemblages of proposals that best meet focused program objectives. Program managers weigh results of the panels’ reviews against program requirements, costs, and scientific risk to ensure a focused and well- balanced program. If the NASA selecting official determines a proposal is relevant to the agency’s mission, it generally will be funded if the panel rated it high in technical merit. Peer reviews are not normally used to evaluate in-progress or completed research, although a yearly progress report is required before a yearly funding allotment is provided to the researcher. However, if a research project continues beyond 3 years, the researcher must submit a new proposal that is subject to full external peer review and competes with new proposals for funding. According to NASA, the amount and quality of the research results are judged by the research community and through publication in scientific journals and NASA’s Scientific and Technical Information Program. This program manages and disseminates results of basic and applied R&D to reduce unnecessary duplication and improve the productivity and cost-effectiveness of the research. The program requires conformance with review requirements prior to acceptance of research for publication. Project managers and staff are frequently asked to comment on management and financial aspects of research proposals, since as a rule science reviewers are not qualified experts in these fields. These nontechnical reviews are presented with scientific reviews to the selecting official. There are a few instances where internal or program reviews alone may be conducted for proposals that involve minor funding for nonresearch activities, such as presentations at workshops or symposia, or a situation which requires a quick response to an unexpected research opportunity. Normally, the program manager conducts mid-point or annual reviews of research projects without input from multiple experts. NASA managers also annually review required research progress reports to determine whether funding should be continued. Also, reports of findings and new knowledge presented at the conclusion of a research effort are evaluated internally to determine if additional support should be provided to the grantee. A final report that is required at the conclusion of a grant or research effort must include citations of all published papers resulting from the work. Internal reviews of research results are based primarily on the publications that have appeared in the peer-reviewed literature, the importance of the results or, in some cases, the rigorous evaluation of results as a part of international research assessments. All research is reviewed, through either external peer review or internal NASA review. The following presents a description of the National Science Foundation’s (NSF’s) peer review and other quality assurance review practices. The National Science FoundationNSF is an independent federal agency, with the goal of promoting and advancing scientific and engineering progress in the United States as well as ensuring the nation’s supply of scientists, engineers, and science educators. Of its fiscal year 1999 R&D budget of $2.7 billion, $2.5 billion is allocated for basic and applied research. NSF supports extramural research and education in most fields of science and engineering, through about 200 programs. The research is funded through grants and cooperative agreements with almost 2,000 colleges, universities, and other research and education organizations from all parts of the United States. NSF annually receives about 30,000 proposals requesting new or renewed support for research, graduate and postdoctoral fellowships, and math, science, and engineering education projects. About 10,000 new awards are made annually. NSF uses the terms peer review and merit review interchangeably, and reviews of research proposals are “merit review with peer evaluation.” NSF reviews involve knowledgeable peers from the scientific and engineering communities as the keystone of their system. conducted appropriately, and the Director of NSF reports annually to the Board on the merit review system. Proposals received for research under the competitive grants program are reviewed by a scientist, engineer, or educator serving as an NSF program officer, and usually by 3 to 10 other persons outside NSF who are experts in the particular field represented by the proposal. Authors of proposals are invited to suggest names of persons they believe are especially well qualified to review the proposal or persons they would prefer not review the proposal. These suggestions serve as one source in the reviewer selection process at the program officer’s discretion. Program officers may obtain comments on proposals from mail reviews, assembled review panels, or site visits before recommending final action on proposals. Senior NSF staff further review the program officer’s recommendations for awards. The division director receives a recommendation and decides whether the proposal should be declined or recommended for award. Normally, final programmatic approval occurs at the division level. Then, the Division of Grants and Agreements reviews the business, financial and policy implications of the proposal, before issuing a grant or other agreement. The judgments of the peers as to the extent that proposals address the review criteria are vital for informing NSF staff and influencing funding recommendations. NSF relies on the judgment of program officers to make funding recommendations that address NSF strategic goals. NSF research grants are awarded for 1 to 5 years, and large ongoing projects (in terms of the number of investigators involved, the time frames of the project, or the dollar amount of the grant) are evaluated by outside experts who visit the research sites. The reviewers are familiar with the scientists and engineers who are conducting the research. The results of these reviews are used in decisions on whether to continue funding of these projects. track results of the research efforts it supports, through annual and final reports on each project and plans to follow-up for years after the research is completed to identify impacts resulting from the research. Investigators receiving grants submit annual progress reports that are reviewed by the program director. At a program director’s prerogative, up to 5 percent of the research budget can be used for newly emerging research areas that are reviewed by NSF staff but are not necessarily subjected to external peer review. This is done because the nature of peer review to select research for funding that is somewhat conservative and may not normally result in the funding of high risk or exploratory projects. NSF believes it is important to conduct some exploratory research to further expand knowledge in certain areas. In some instances, these grants are awarded for ideas that may need quick decisions, such as a case where earthquake information requires immediate study. All research funded by NSF is reviewed through internal NSF review and, in most cases, also through external peer review. The following presents a description of the Federal Aviation Administration’s (FAA’s) peer review and other quality assurance review practices. Within the Department of Transportation, the Federal Aviation Administration’s (FAA) mission is to provide for the safe, secure, and efficient movement of air traffic consistent with national security concerns. The FAA’s Research, Engineering, and Development Program develops and validates the technologies, systems, designs, and procedures for the full range of the agency’s operational and regulatory activities. The agency’s 1999 budgets for research, engineering, and development and Advanced Technology Development and Prototyping is $202.6 million. During the agency’sFAA rulemaking process, the Aviation Rulemaking Advisory Committee, an FAA and industry group, discusses the confidence level in the research that supports a proposed rule. failure, and to counteract terrorist acts. FAA grants are generally awarded to applicants from colleges, universities, and other nonprofit research institutions. Other appropriate research institutions, such as for-profit organizations and governmental entities, may also qualify to perform research in aviation security. Peer reviews are technical reviews performed by in-house and external experts with qualifications equal with those of the researcher whose work is being reviewed. Peers must be knowledgeable about the area to be reviewed. FAA conducts peer reviews of proposed research projects funded through FAA grants and cooperative agreements. Through an advisory council, the agency also conducts program-level peer reviews of planned research, ongoing research, and the outputs of completed research for their usefulness to FAA and industry. The 1990 FAA Research, Engineering, and Development Authorization Act provided grant authority to the agency for the first time and required a review and evaluation process to ensure that research proposals have adequate merit and relevancy to FAA mission. FAA officials also said that they are cognizant of the annual budget guidance from OSTP and OMB regarding the need to conduct peer review of competitively awarded research proposals. FAA has developed a formal grant application process. Grants and cooperative agreements are administered by FAA’s Office of Aviation Research, and the Research Grants Program Office also reports to the Research, Engineering, and Development Advisory Committee. FAA issues one broad solicitation each year outlining major categories for needed research and receives proposals to undertake specific R&D projects on an on-going basis. Each proposal is evaluated on its own merit, rather than being compared with other proposals. The Research Grants Program documentation states that following an administrative review; each proposal will be reviewed carefully for merit by a technical evaluation team consisting of three or more qualified people. A FAA representative is to be designated as the team leader and is responsible for developing an overall rating based on the ratings of the team members. FAA officials said that peer reviewers are primarily FAA employees, because they have the required subject matter expertise. Ninety percent of the proposals are reviewed by three individual peers not involved in the project, who are selected by the program experts. Reviewers attest they have no conflict of interest. Reviewers receive guidance on the use of four review criteria: intrinsic value, relevance to FAA mission, technical soundness of the proposal, and research performance competence. However, a FAA official said that these reviews are highly subjective. FAA also conducts peer reviews of on-going research projects and programs. For example, FAA provides annual funding for the Joint FAA/NASA University Program, a consortium comprised of the Massachusetts Institute of Technology, Ohio University, and Princeton University. FAA and NASA’s technical experts conduct triannual reviews of the program’s ongoing research in areas such as intelligent flight control systems, weather hazard avoidance, satellite navigation, cockpit displays, and intelligent air traffic management. The agency also conducts reviews of the work in progress in its Centers of Excellence. For example, for the Operations Research Center of Excellence, the agency and the center’s partners from industry and academia conduct an annual research review as well as hold biannual steering committee meetings to ensure that the research is on track and pertinent to the needs of FAA. The agency encourages intramural and extramural researchers to publish results in journals that conduct peer review prior to the acceptance of the results for publication. The Civil Aeromedical Institute reported that it conducts peer reviews of manuscripts and other work products prior to publication. objectives, plans, approaches, content, and accomplishments. The Federal Aviation Administration Research, Engineering, and Development Authorization Act of 1990 expanded the committee’s responsibilities and set the membership to not more than 30 representatives from research Centers of Air Transportation Excellence, universities, corporations, user groups, associations, consumers, and other government agencies. During the annual process, subcommittees of the advisory committee review FAA’s proposed research and research that is already in-progress and make recommendations about priorities and merit of the research. Evaluation criteria for assessing research project descriptions include, among others, Results Act considerations, congressional mandates, mission relevance, research outcomes and outputs, and whether the project plan describes a credible, well understood, work effort. Documented procedures are used to develop merit scores assigned to proposed research project descriptions. During the annual programming and budgeting process FAA target area teams, which include internal research sponsors from the six program areas, are heavily involved in prioritizing and planning all research efforts and in decisions about what research to fund. In addition, FAA program staff conduct reviews of some research programs once or twice a year, or as needed. For example, a FAA official said that program review teams include a FAA Associate Administrator, Department of Transportation representatives, and FAA stakeholders. On an ad hoc basis, FAA solicits assistance from external organizations such as the National Academy of Science to review its research efforts. between FAA and industry or opportunities to share data from related research. The Civil Aeromedical Institute conducts intramural research, and reviews are primarily internal. The Institute employs some of the world’s best scientists in its narrow fields of research, so the number of outside experts is limited. Internal reviews are conducted of Aeromedical Research Resumes, which describe technical details of proposed research projects. In addition, an Internal Medical Research Group of FAA laboratory and headquarters staff, and one outside expert, meets four times a year to review research proposals and quarterly progress reports. Research performed under procurement contracts is reviewed during the annual programming and budgeting process. After this process establishes research needs and budgets, FAA technical staff prepare statements of work for research they want to perform under contracts. These same staff review subsequent proposals, and after contracts are awarded, they are expected to monitor the researcher’s performance. In addition, contract deliverables are reviewed by internal experts prior to acceptance by the agency. All FAA research is reviewed by FAA, its industry stakeholders, or others. Arleen L. Alleman James S. Crigler The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.

Summary: Pursuant to a congressional request, GAO studied the peer review and other quality assurance processes that federal agencies use in conducting scientific research and development, focusing on: (1) defining what is meant by peer review; (2) describing the federal government's peer review policy; (3) describing the peer review practices of 12 federal agencies that conduct scientific research; (4) describing other agencies' quality assurance reviews; and (5) identifying which research is not subject to review. GAO noted that: (1) there is no written governmentwide definition of peer review; (2) officials at the Office of Science and Technology Policy (OSTP) and at the agencies GAO contacted generally concur that peer review is defined as a process that includes an independent assessment of the technical, or scientific merit of research by peers who are scientists with knowledge and expertise equal to that of the researchers whose work they review; (3) there is no uniform federal policy for conducting peer reviews; (4) through annual budget guidance to federal agencies, OSTP and the Office of Management and Budget encourage funding of research projects that are peer reviewed over those that are not reviewed; (5) officials at OSTP said that peer review practices should not be dictated uniformly for every agency or for all types of federally funded research; (6) rather, the practices should be tailored to agency missions and type of research; (7) each of the 12 agencies that GAO contacted had a variety of policies, orders, or other internal guidance regarding the conduct of peer review; (8) to varying degrees, the 12 agencies use peer review to: (a) assess the merit of competitive and noncompetitive research proposals; (b) determine whether to continue or renew research projects; (c) evaluate the results of the research prior to the publication of those results; (d) establish annual budget priorities for research programs; and (e) evaluate program and scientist performance; (9) all of the agencies use peer review to assess competitive research proposals; (10) the methods for conducting peer reviews vary among and within the agencies; (11) most of the agencies that GAO reviewed also use reviews by agency supervisors or program managers to assess the quality of research proposals, to check the quality of in-progress research, and to evaluate program performance; (12) generally, these quality assurance reviews are not considered independent assessments--a key criterion in the peer review process; (13) these quality assurance reviews occur at both the project and program levels; (14) while agencies reported that almost all research is reviewed either through peer reviews or other quality assurance reviews, a small amount of research may not be reviewed by the agencies in certain circumstances; and (15) examples of research that may be funded without being reviewed include projects that are congressionally mandated or projects that use widely accepted methodologies.