[Federal Register Volume 69, Number 82 (Wednesday, April 28, 2004)]
[Notices]
[Pages 23230-23242]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 04-9572]


=======================================================================
-----------------------------------------------------------------------

OFFICE OF MANAGEMENT AND BUDGET


Revised Information Quality Bulletin on Peer Review

AGENCY: Office of Management and Budget, Executive Office of the 
President.

ACTION: Notice and request for comments.

-----------------------------------------------------------------------

SUMMARY: The Office of Management and Budget (OMB), in consultation 
with the Office of Science and Technology Policy (OSTP), is re-
proposing its new guidance designed to realize the benefits of 
meaningful peer review of the most important science disseminated by 
the Federal Government. This Notice requests comment on the revised 
Bulletin, now entitled ``Revised Information Quality Bulletin on Peer 
Review.'' OMB originally requested comment on its ``Proposed Bulletin 
on Peer Review and Information Quality,'' published in the Federal 
Register on September 15, 2003. We received 187 comments during the 
public comment period, listened to discussion at a public workshop at 
the National Academy of Sciences (NAS), and carried out an interagency 
review. This process led to a substantially revised Bulletin, which 
incorporates many of the diverse perspectives and suggestions voiced 
during the comment period. The public comments are posted at: http://www.whitehouse.gov/omb/inforeg/2003iq/iq_list.html. A summary of the 
public and agency comments, including responses by OMB and OSTP, is

[[Page 23231]]

available at http://www.whitehouse.gov/omb/inforeg/infopoltech.html#iq.
    The revised Bulletin incorporates a number of changes:
     Providing in the preamble, a more extensive 
discussion of why government-wide peer review guidance is needed;
     Providing more discretion to federal agencies in 
determining what type of peer review mechanism is appropriate for 
specific information products;
     Providing exemptions for time-sensitive medical, 
public health and safety information and other compelling 
circumstances;
     Indicating in a savings clause that the guidance 
does not create any new rights for litigation against federal agencies;
     Defining a more transparent process for public 
participation in peer review planning; and
     Requiring the most rigorous form of peer review 
only for highly influential scientific assessments.
    This Bulletin is part of an ongoing effort to improve the quality, 
objectivity, utility, and integrity of information disseminated by the 
Federal Government to the public. This Bulletin would be issued under 
the authority of section 515 of the Treasury and General Government 
Appropriations Act for Fiscal Year 2001 (Pub. L. 106-554; H.R. 5658); 
44 U.S.C. 3504(d)(1), 3506(a)(1)(B); Executive Order No. 12866, as 
amended. Part I of the SUPPLEMENTARY INFORMATION below provides the 
Preamble to the Bulletin. Part II provides the text of the proposed 
bulletin.

DATES: Interested parties should submit comments to the Office of 
Information and Regulatory Affairs (OIRA), Office of Management and 
Budget, at the address shown below on or before May 28, 2004.

John D. Graham,
Administrator, Office of Information and Regulatory Affairs.

SUPPLEMENTARY INFORMATION:

Introduction

    On September 15, 2003, OIRA published a draft Peer Review Bulletin 
for public comment. We received 187 comments during the public comment 
period, participated in a public workshop at the National Academy of 
Sciences (NAS), and undertook an interagency review process. This 
process led to a substantially revised Bulletin, which incorporates 
many of the diverse perspectives and suggestions voiced during the 
comment period.
    As almost all commenters recognized, peer review is an important 
way to enhance the quality of information. When done in an open, 
rigorous manner, independent peer review improves both the quality of 
scientific information and the public's confidence in the integrity of 
science.
    Under this Bulletin, agencies must undertake a peer review of 
influential scientific information before they disseminate the 
information to the public. Different types of peer review are 
appropriate for different types of information products, and agencies 
are granted under this Bulletin appropriate discretion to weigh the 
benefits and costs of using a particular peer review mechanism for a 
particular information product. This Bulletin leaves the selection of a 
peer review mechanism for influential scientific information to the 
agency's discretion. Based on public and agency comments, we also 
exempted various types of information products from the requirements of 
this Bulletin, including time-sensitive medical, health, and safety 
determinations, in order to ensure that peer review does not unduly 
delay the release of time-sensitive findings.
    This Bulletin also imposes minimum requirements for the peer review 
of highly influential scientific assessments, which are a subset of 
influential scientific information. A scientific assessment is an 
evaluation of a body of scientific or technical knowledge which 
typically synthesizes multiple factual inputs, data, models, 
assumptions, and/or applies best professional judgment to bridge 
uncertainties in the available information. Although the proposed 
Bulletin imposed heightened peer review requirements on a broader array 
of information products, we agree with some commenters that, in order 
to ensure that the Bulletin is not too costly or rigid, more intensive 
peer review should be restricted to the more important information 
disseminated by the Federal Government.
    Even for this category of highly influential scientific 
assessments, the revised Bulletin leaves broad discretion to the agency 
formulating the peer review plan. In general, an agency conducting a 
peer review of a highly influential scientific assessment must ensure 
that the peer review process is transparent by making available to the 
public a written charge to the peer reviewers, the peer reviewers' 
report, and the agency's response to the peer reviewers' report. The 
agency selecting peer reviewers must ensure that the reviewers possess 
the necessary expertise. In addition, the agency must address 
reviewers' potential conflicts of interest (including those stemming 
from ties to regulated businesses) and independence from the agency. In 
response to comments, this revised Bulletin encourages agencies to 
consider using the panel selection criteria employed by the NAS. The 
use of a transparent process, coupled with the selection of objective 
and independent peer reviewers, should improve the quality of 
government science while promoting public confidence in the integrity 
of the government's scientific products.

Peer Review

    Peer review is one of the important procedures used in science to 
ensure that the quality of published information meets the standards of 
the scientific community. It is a form of deliberation involving an 
exchange of judgments about the appropriateness of methods and the 
strength of the author's inferences.\1\ Peer review occurs when a draft 
product is reviewed for quality by specialists who were not involved in 
producing the draft.
---------------------------------------------------------------------------

    \1\ Carnegie Commission on Science, Technology, and Government, 
Risk and the Environment: Improving Regulatory Decision Making, 
Carnegie Commission, New York, 1993: 75.
---------------------------------------------------------------------------

    The peer reviewer's report is an evaluation or critique that is 
used by the authors of the draft to improve the product. Peer review 
typically evaluates the clarity of hypotheses, the validity of the 
research design, the quality of the data collection procedures, the 
robustness of the methods employed, the appropriateness of the methods 
for the hypotheses being tested, the extent to which the conclusions 
follow from the analysis, and the strengths and limitations of the 
overall product.
    Peer review has diverse purposes. Editors of scientific journals 
use reviewer comments to help determine whether a draft scientific 
article is of sufficient quality, importance, and interest to a field 
of study to justify publication. Research funding organizations often 
use peer review to evaluate research proposals. In addition, some 
federal agencies make use of peer review to obtain evaluations of draft 
information products that contain important scientific determinations.
    Peer review should not be confused with public comment and other 
stakeholder processes. The selection of participants in a peer review 
is based on expertise, independence, and the absence of conflict of 
interest. Furthermore, notice-and-comment procedures for agency 
rulemaking do not provide an adequate substitute for peer review, as 
disinterested experts--especially those most knowledgeable in a field--
often do not file public comments with federal agencies.

[[Page 23232]]

    The critique provided by a peer review often suggests ways to 
clarify assumptions, findings, and conclusions. For instance, peer 
reviews can filter out biases and identify oversights, omissions, and 
inconsistencies.\2\ Peer review also may encourage authors to more 
fully acknowledge limitations and uncertainties. In some cases, 
reviewers might recommend major changes to the draft, such as 
refinement of hypotheses, reconsideration of research design, 
modifications of data collection or analysis methods, or alternative 
conclusions. However, peer review does not always lead to specific 
modifications in the draft product. In some cases, a draft is in 
excellent shape prior to being submitted for review. In others, the 
authors do not concur with changes suggested by one or more reviewers.
---------------------------------------------------------------------------

    \2\ William W. Lowrance, Modern Science and Human Values, Oxford 
University Press, New York, NY 1985: 85.
---------------------------------------------------------------------------

    Peer review may take a variety of forms, depending upon the nature 
and importance of the product. For example, the reviewers may represent 
one scientific discipline or a variety of disciplines; the number of 
reviewers may range from a few to more than a dozen; the names of each 
reviewer may be disclosed publicly or may remain anonymous (e.g., to 
encourage candor); the reviewers may be blinded to the authors of the 
report or the names of the authors may be disclosed to the reviewers; 
the reviewers may prepare individual reports or a panel of reviewers 
may be constituted to produce a collaborative report; panels may do 
their work electronically or they may meet together in person to 
discuss and prepare their evaluations; and reviewers may be compensated 
for their work or they may donate their time as a contribution to 
science or public service.
    For large, complex reports, different reviewers may be assigned to 
different chapters or topics. Such reports may be reviewed in stages, 
sometimes with blinded, confidential reviews that precede a public 
process of panel review. As part of peer review, there may be 
opportunity for written and/or oral public comments on the draft 
product.
    The results of peer review are often only one of the criteria used 
to make decisions about journal publication, grant funding, and 
information dissemination. For instance, the editors of scientific 
journals (rather than the peer reviewers) make final decisions about a 
manuscript's appropriateness for publication based on a variety of 
considerations. In research-funding decisions, the reports of peer 
reviewers often play an important role, but the final decisions about 
funding are often made by accountable officials based on a variety of 
considerations. Similarly, when a government agency sponsors peer 
review of its own draft documents, the peer review reports are an 
important factor in information dissemination decisions, but are rarely 
the sole consideration. Agencies are not expected to cede their 
discretion with regard to dissemination or use of information to peer 
reviewers; accountable agency officials must make the final decisions.

The Need for Stronger Peer Review Policies

    There are a multiplicity of science advisory procedures used at 
federal agencies and across the wide variety of scientific products 
prepared by agencies.\3\ In response to congressional inquiry, the U.S. 
General Accounting Office documented the variability in both the 
definition and implementation of peer review across agencies.\4\ The 
Carnegie Commission on Science, Technology and Government \5\ has 
highlighted the importance of ``internal'' scientific advice (within 
the agency) and ``external'' advice (through scientific advisory boards 
and other mechanisms).
---------------------------------------------------------------------------

    \3\ Sheila Jasanoff, The Fifth Branch: Science Advisors as 
Policy Makers, Harvard University Press, Boston, 1990.
    \4\ U.S. General Accounting Office, Federal Research: Peer 
Review Practices at Federal Agencies Vary, GAO/RCED-99-99, 
Washington, DC, 1999.
    \5\ Carnegie Commission on Science, Technology, and Government, 
Risk and the Environment: Improving Regulatory Decision Making, 
Carnegie Commission, New York, 1993: 90.
---------------------------------------------------------------------------

    A wide variety of authorities have argued that peer review 
practices at federal agencies need to be strengthened.\6\ Other 
arguments focus on specific types of scientific products (e.g., 
assessments of health, safety and environmental hazards).\7\ Indeed, 
the Congressional/Presidential Commission on Risk Assessment and Risk 
Management suggests that ``peer review of economic and social science 
information should have as high a priority as peer review of health, 
ecological, and engineering information.''\8\
---------------------------------------------------------------------------

    \6\ National Academy of Sciences, Peer Review in the Department 
of Energy--Office of Science and Technology, Interim Report, 
National Academy Press, Washington, DC, 1997; National Academy of 
Sciences, Peer Review in Environmental Technology Development: The 
Department of Energy--Office of Science and Technology, National 
Academy Press, Washington, DC, 1998; National Academy of Sciences, 
Strengthening Science at the U.S. Environmental Protection Agency: 
Research-Management and Peer-Review Practices, National Academy 
Press, Washington, DC, 2000; U.S. General Accounting Office, EPA's 
Science Advisory Board Panels: Improved Policies and Procedures 
Needed to Ensure Independence and Balance, GAO-01-536, Washington, 
DC, 2001; U.S. Environmental Protection Agency, Office of Inspector 
General, Pilot Study: Science in Support of Rulemaking 2003-P-00003, 
Washington, DC, 2002; Carnegie Commission on Science, Technology, 
and Government, In the National Interest: The Federal Government in 
the Reform of K-12 Math and Science Education, Carnegie Commission, 
New York, 1991; U.S. General Accounting Office, Endangered Species 
Program: Information on How Funds Are Allocated and What Activities 
are Emphasized, GAO-02-581, Washington, DC 2002.
    \7\ National Research Council, Science and Judgment in Risk 
Assessment, National Academy Press, Washington, DC, 1994.
    \8\ Presidential/Congressional Commission on Risk Assessment and 
Risk Management, Risk Commission Report, Volume 2, Risk Assessment 
and Risk Management in Regulatory Decision-Making, 1997:103.
---------------------------------------------------------------------------

    Some agencies have formal peer review policies, while others do 
not. Even agencies that have such policies do not always follow them 
prior to the release of important scientific products.
    Prior to the development of this Bulletin, there were no 
government-wide standards concerning when peer review is required and, 
if required, what type of peer review processes are appropriate. No 
formal interagency mechanism existed to foster cross-agency sharing of 
experiences with peer review practices and policies. Despite the 
importance of peer review for the credibility of agency scientific 
products, the public lacks a consistent way to determine when an 
important scientific information product is being developed by an 
agency, the type of peer review planned for that product, or whether 
there will be an opportunity to provide comments and data to the 
reviewers.
    This Bulletin establishes minimum standards for when peer review is 
required for scientific information and the types of peer review that 
should be considered by agencies in different circumstances. It also 
establishes a transparent process for public disclosure of peer review 
planning, including the establishment of an agenda that describes the 
peer review process that the agency has chosen for each of its 
forthcoming influential scientific information products.

Legal Authority for the Bulletin

    This Bulletin is issued under the Information Quality Act and OMB's 
general authorities to oversee the quality of agency information, 
analyses, and regulatory actions. In the Information Quality Act, 
Congress directed OMB to issue guidelines to ``provide policy and 
procedural guidance to Federal agencies for ensuring and maximizing the 
quality, objectivity, utility and integrity

[[Page 23233]]

of information'' disseminated by Federal agencies. Public Law 106-554, 
515(a). The Information Quality Act was crafted as an amendment to the 
Paperwork Reduction Act, 44 U.S.C. 3501 et seq., which requires OMB, 
among other things, to ``develop and oversee the implementation of 
policies, principles, standards, and guidelines to * * * apply to 
Federal agency dissemination of public information.'' In addition, 
Executive Order 12866, 58 FR 51735 (Oct. 4, 1993), establishes that 
OIRA is ``the repository of expertise concerning regulatory issues,'' 
and it directs OMB to provide guidance to the agencies on regulatory 
planning. E.O. 12866, section 2(b). The Order also requires that 
``[e]ach agency shall base its decisions on the best reasonably 
obtainable scientific, technical, economic, or other information.'' 
E.O. 12866, section 1(b)(7). Finally, OMB has general authority to 
manage the agencies under the purview of the President's Constitutional 
authority to oversee the unitary Executive Branch. See, e.g., the 
Budget and Accounting Procedures Act of 1950, as amended, 31 U.S.C. 
1111; Reorganization Plan No. 2 of 1970, 84 Stat. 2085; Executive Order 
11541, 35 FR 10737 (July 1, 1970); Executive Order 12866. All of these 
authorities support this Bulletin.

The Requirements of This Bulletin

    This Bulletin addresses peer review of scientific information 
disseminations that contain findings or conclusions that represent the 
official position of one or more Departments or agencies of the Federal 
Government.

Section I: Definitions

    Section I provides definitions that are central to this Bulletin. 
Several terms are identical to or based on those used in OMB's 
government-wide information quality guidelines 67 FR 8452 (Feb. 22, 
2002), and the Paperwork Reduction Act, 44 U.S.C. 3501 et seq. The term 
``Administrator'' means the Administrator of the Office of Information 
and Regulatory Affairs in the Office of Management and Budget. The term 
``agency'' includes all agencies subject to the Paperwork Reduction 
Act, see 44 U.S.C. 3502(1). The term ``Information Quality Act'' means 
Section 515 of the Treasury and General Government Appropriations Act 
for Fiscal Year 2001 (Pub. L. 106-554; H.R. 5658).
    The term ``dissemination'' means agency initiated or sponsored 
distribution of information to the public. Dissemination does not 
include distribution limited to government employees or agency 
contractors or grantees; intra- or inter-agency use or sharing of 
government information; or responses to requests for agency records 
under the Freedom of Information Act, the Privacy Act, the Federal 
Advisory Committee Act or similar law. This definition also excludes 
distribution limited to correspondence with individuals or persons, 
press releases, archival records, public filings, subpoenas and 
adjudicative processes. Finally, ``dissemination'' also excludes 
information distributed for peer review in compliance with this 
Bulletin, provided that the distributing agency includes an appropriate 
and clear disclaimer on the information, as explained more fully below.
    In the context of this Bulletin, the definition of 
``dissemination'' also goes beyond the definition in OMB's government-
wide information quality guidelines to address the need for peer review 
prior to official dissemination of the information product. In cases 
where a draft report or other information is released by an agency for 
purposes of peer review, a question may arise as to whether the draft 
report constitutes an official ``dissemination'' under information-
quality guidelines. Normally, draft reports undergoing peer review are 
not intended as disseminations--because they are not yet final--and 
thus Section I instructs agencies to make this clear by presenting the 
following disclaimer in the report: ``THIS INFORMATION IS DISTRIBUTED 
SOLELY FOR THE PURPOSE OF PRE-DISSEMINATION PEER REVIEW UNDER 
APPLICABLE INFORMATION QUALITY GUIDELINES. IT HAS NOT BEEN FORMALLY 
DISSEMINATED BY [THE AGENCY] AND SHOULD NOT BE CONSTRUED TO REPRESENT 
ANY AGENCY DETERMINATION OR POLICY.''
    This disclaimer should appear on each page of a draft report in 
cases where the information is highly relevant to specific policy or 
regulatory deliberations. Agencies also should discourage state, local, 
international and private organizations from using information in draft 
reports that are undergoing peer review. Draft influential scientific 
information being presented at scientific meetings prior to peer review 
must include the disclaimer: ``THE VIEWS IN THIS REPORT (PRESENTATION) 
ARE THOSE OF THE AUTHOR(S) AND DO NOT NECESSARILY REPRESENT THE VIEWS 
OF THE FUNDING AGENCY.''
    For the purposes of the peer review Bulletin, the term ``scientific 
information'' means factual inputs, data, models, analyses, or 
scientific assessments related to such disciplines as the behavioral 
and social sciences, public health and medical sciences, life and earth 
sciences, engineering, or physical sciences. This includes any 
communication or representation of knowledge such as facts or data, in 
any medium or form, including textual, numerical, graphic, 
cartographic, narrative, or audiovisual forms. This definition includes 
information that an agency disseminates from a web page, but does not 
include the provision of hyperlinks on a Web page to information that 
others disseminate. This definition excludes opinions, where the 
agency's presentation makes clear that an individual's opinion, rather 
than a statement of fact or of the agency's views, is being offered.
    The term ``influential scientific information'' means the 
scientific information the dissemination of which the agency reasonably 
can determine will have or does have a clear and substantial impact on 
important public policies or private sector decisions. In OMB's 
government-wide information quality guidelines, the term ``influential 
information'' is used in the context of ``influential scientific, 
financial, or statistical information.'' However, this Bulletin only 
covers ``influential scientific information.''
    For the purposes of this Bulletin, the term ``scientific 
assessment'' means an evaluation of a body of scientific or technical 
knowledge which typically synthesizes multiple factual inputs, data, 
models, assumptions, and/or applies best professional judgment to 
bridge uncertainties in the available information. These assessments 
include, but are not limited to, state-of-science reports, technology 
assessments, weight-of-evidence analyses, meta-analyses, risk 
assessments, toxicological profiles of substances, integrated 
assessment models, hazard determinations, exposure assessments, or 
health, ecological, or safety assessments. The assessment will often 
draw upon knowledge from multiple disciplines.

Section II: Peer Review of Influential Scientific Information

    Section II requires each agency to subject ``influential'' 
scientific information to peer review prior to dissemination. For 
dissemination of influential scientific information, Section II 
provides agencies broad discretion in determining what type of peer 
review is appropriate and what procedures should be employed to select 
appropriate reviewers.
    The National Academy of Public Administration suggests that the 
intensity of peer review should be

[[Page 23234]]

commensurate with the significance of the information being 
disseminated and the likely implications for policy decisions.\9\ 
Furthermore, agencies need to consider tradeoffs between depth of peer 
review and timeliness.\10\ More rigorous peer review is necessary for 
information that is based on novel methods or presents complex 
challenges for interpretation. Furthermore, the need for rigorous peer 
review is greater when the information contains precedent-setting 
methods or models, presents conclusions that are likely to change 
prevailing practices, or is likely to affect policy decisions that have 
a significant impact.
---------------------------------------------------------------------------

    \9\ National Academy of Public Administration, Setting 
Priorities, Getting Results: A New Direction for EPA, National 
Academy Press, Washington, DC, 1995:23.
    \10\ Presidential/Congressional Commission on Risk Assessment 
and Risk Management, Risk Commission Report, 1997.
---------------------------------------------------------------------------

    This tradeoff can be considered in a benefit-cost framework. The 
costs of peer review are the direct costs of the peer review activity, 
and the potential delay in government and private actions that can 
result from peer review. The benefits of peer review are equally clear: 
the insights offered by peer reviewers may lead to policy with more 
benefits and/or fewer costs. In addition to contributing to strong 
science, peer review, if performed fairly and rigorously, can build 
consensus among stakeholders and reduce the temptation for courts and 
legislators to second-guess agency actions.\11\ While it will not 
always be easy for agencies to quantify the benefits and costs of peer 
review, we encourage agencies to approach peer review from a benefit-
cost perspective.
---------------------------------------------------------------------------

    \11\ Mark R. Powell, Science at EPA: Information in the 
Regulatory Process, Resources for the Future, Washington, D.C., 
1999: 148, 176; Sheila Jasanoff, The Fifth Branch: Science Advisors 
as Policy Makers, Harvard University Press, Boston, 1990: 242.
---------------------------------------------------------------------------

    Regardless of the peer review mechanism chosen, agencies should 
strive to ensure that their peer review practices are characterized by 
both scientific integrity and process integrity. ``Scientific 
integrity,'' in the context of peer review, refers to such issues as 
``expertise and balance of the panel members, the identification of the 
scientific issues and clarity of the charge to the panel, and the 
quality, focus and depth of the discussion of the issues by the panel, 
the rationale and supportability of the panel's findings, and the 
accuracy and clarity of the panel report.'' ``Process integrity'' 
includes such issues as ``transparency and openness, avoidance of real 
or perceived conflicts of interest, a workable process for public 
comment and involvement,'' as well as adhering to defined 
procedures.\12\
---------------------------------------------------------------------------

    \12\ ILSI Risk Sciences Institute, ``Policies and Procedures: 
Model Peer Review Center of Excellence,'' 2002: 4. Available at 
http://rsi.ilsi.org/file/Policies&Procedures.pdf.
---------------------------------------------------------------------------

    When deciding what type of peer review mechanism is appropriate for 
a specific information product, agencies will need to consider at least 
the following issues: individual versus panel review; timing; the scope 
of the review; the selection of reviewers; disclosure; public 
participation; and disposition of reviewer comments. These issues are 
relevant to any peer review under this Bulletin.
Individual Versus Panel Review
    Letter reviews by several experts generally will be more 
expeditious than convening a panel of a dozen or more experts. 
Individual letters are more appropriate when a draft document covers 
only one discipline or when premature disclosure of a sensitive report 
to a public panel could cause harm to government or private interests. 
When time and resources warrant, panels are preferable, as they tend to 
be more deliberative than individual letter reviews and the reviewers 
can learn from each other. There are also multi-stage processes in 
which confidential letter reviews are conducted prior to release of a 
draft document for public notice and comment, followed by a formal 
panel review.
    These more rigorous and expensive processes are appropriate for 
highly complex, multidisciplinary, and more important documents, 
especially those that are novel or precedent-setting.
Timing of Peer Review
    As a general rule, it is most useful to consult with peers early in 
the process of producing an information product. For example, in the 
context of risk assessments, it is valuable to have the choice of input 
data and the specification of the model reviewed by peers before the 
agency invests time and resources in implementing the model and 
interpreting the results. ``Early'' peer review occurs in time to 
``focus attention on data inadequacies in time for corrections.'' \13\
---------------------------------------------------------------------------

    \13\ Testimony of Bruce Alberts, PhD., President, National 
Academy of Sciences, February 24, 1998, Hearing on S. 981, before 
Senate Committee on Governmental Affairs.
---------------------------------------------------------------------------

    When an information product is a critical component of rule-making, 
it is important to obtain peer review before the agency announces its 
regulatory options so that any technical corrections can be made before 
the agency becomes invested in a specific approach or the positions of 
interest groups have hardened. If review occurs too late, it is 
unlikely to contribute to the course of a rulemaking. For instance, use 
of peer review is more often regarded as ``generally successful'' when 
it occurs ``early'' in the agency's deliberative process. Furthermore, 
investing in a more rigorous peer review early in the process ``may 
provide net benefit by reducing the prospect of challenges to a 
regulation that later may trigger time consuming and resource-draining 
litigation.'' \14\
---------------------------------------------------------------------------

    \14\ Fred Anderson, Mary Ann Chirba Martin, E. Donald Elliott, 
Cynthia Farina, Ernest Gellhorn, John D. Graham, C. Boyden Gray, 
Jeffrey Holmstead, Ronald M. Levin, Lars Noah, Katherine Rhyne, 
Jonathan Baert Wiener, ``Regulatory Improvement Legislation: Risk 
Assessment, Cost-Benefit Analysis, and Judicial Review,'' Duke 
Environmental Law and Policy Forum, Fall 2000, vol. XI (1): 132.
---------------------------------------------------------------------------

Scope of the Review
    The ``charge'' contains the instructions to the peer reviewers 
regarding the objective of the peer review and the specific advice 
sought. The importance of the information, which shapes the goal of the 
peer review, influences the charge. For instance, the goal of the 
review might be to determine the utility of a body of literature for 
drawing certain conclusions about the feasibility of a technology or 
the safety of a product. In this context, an agency might ask reviewers 
to determine the relevance of conclusions drawn in one context for 
other contexts (e.g., different exposure conditions or patient 
populations).
    The charge to the reviewers should be determined in advance of the 
selection of the reviewers. In drafting the charge, it is important to 
remember the strengths and limitations of peer review. Peer review is 
most powerful when the charge is specific and steers the reviewers to 
specific technical questions while also directing reviewers to offer a 
broad evaluation of the overall product.
    Uncertainty is inherent in science, and in many cases individual 
studies do not produce conclusive evidence. Rather, what is being 
reviewed in the case of scientific assessments is a scientific judgment 
rather than ``scientific fact.'' \15\ Specialists attempt to reach a 
consensus by weighing the accumulated evidence. As such, it is 
important that peer reviewers be asked to ensure that scientific 
uncertainties are clearly identified and characterized. Furthermore, 
since not all uncertainties will have an equal effect on the 
conclusions drawn, reviewers can be asked to ensure that the potential

[[Page 23235]]

implications of the uncertainties for the technical conclusions drawn 
are clear. Within this context, peer reviewers can make an important 
contribution by distinguishing scientific facts from professional 
judgments. Reviewers might be asked to provide advice on reasonable 
judgments that can be made from the scientific evidence, but the charge 
should make clear that the reviewers are not to provide advice on the 
policy (e.g., the amount of uncertainty that is acceptable or the 
amount of precaution that should be embedded in an analysis). Such 
considerations are the purview of the government.\16\ In addition, peer 
reviewers might be asked to consider value-of-information analyses that 
identify whether more research is likely to decrease key 
uncertainties.\17\ Value-of-information analysis was suggested for this 
purpose in the reports of the Presidential/Congressional Commission on 
Risk Assessment and Risk Management.\18\ A description of additional 
research that would appreciably influence the conclusions of the 
assessment might help an agency target any additional research 
resources available for this problem.
---------------------------------------------------------------------------

    \15\ Mark R. Powell, Science at EPA: Information in the 
Regulatory Process, Resources for the Future, Washington, DC, 1999: 
139.
    \16\ Ibid.
    \17\ Granger Morgan and Max Henrion, ``The Value of Knowing How 
Little You Know,'' Uncertainty: A Guide to Dealing with Uncertainty 
in Quantitative Risk and Policy Analysis, Cambridge University 
Press, 1990: 307.
    \18\ Presidential/Congressional Commission on Risk Assessment 
and Risk Management, Risk Commission Report, 1997, Volume 1: 39, 
Volume 2: 91.
---------------------------------------------------------------------------

Selection of Reviewers
    Expertise. The most important factor in selecting reviewers is 
expertise: ensuring that the selected reviewer has the knowledge, 
experience, and skills necessary to perform the review. In cases where 
the document being reviewed spans a variety of scientific disciplines 
or areas of technical expertise, reviewers who represent the necessary 
spectrum of knowledge should be chosen. For instance, expertise in 
applied mathematics and statistics is essential in the review of 
models, thereby allowing an audit of calculations and claims of 
significance and robustness based on the numeric data.\19\ For some 
reviews, evaluation of biological plausibility is as important as 
statistical modeling.
---------------------------------------------------------------------------

    \19\ William W. Lowrance, Modern Science and Human Values, 
Oxford University Press, New York, NY 1985: 86.
---------------------------------------------------------------------------

    Balance. Reviewers should also be selected to represent a diversity 
of scientific perspectives relevant to the subject. On most 
controversial issues, there exists a range of respected scientific 
viewpoints regarding interpretation of the available literature. 
Inviting reviewers with competing views on the science may lead to a 
sharper, more focused peer review. Indeed, as a final layer of review, 
some organizations (e.g., the National Academy of Sciences) 
specifically recruit reviewers with strong opinions to test the 
scientific strength and balance of their reports.
    Independence. In its narrowest sense, independence in a reviewer 
means that the reviewer was not involved in producing the draft 
document to be reviewed. However, for peer review of some documents, a 
broader view of independence is often necessary to assure credibility 
of the process. Reviewers are generally not employed by the agency or 
office producing the document. As the National Academy of Sciences has 
stated, ``external experts often can be more open, frank, and 
challenging to the status quo than internal reviewers, who may feel 
constrained by organizational concerns.'' \20\ The Carnegie Commission 
on Science, Technology, and Government notes that ``external science 
advisory boards serve a critically important function in providing 
regulatory agencies with expert advice on a range of issues.'' \21\ 
However, the choice of reviewers requires a case-by-case analysis. In 
some instances, reviewers employed by other federal and state agencies 
may be sufficiently independent.
---------------------------------------------------------------------------

    \20\ National Research Council, Peer Review in Environmental 
Technology Development Programs: The Department of Energy's Office 
of Science and Technology, National Academy Press, Washington, D.C., 
1998: 3.
    \21\ Carnegie Commission on Science, Technology, and Government, 
Risk and the Environment: Improving Regulatory Decision Making, 
Carnegie Commission, New York, 1993: 90.
---------------------------------------------------------------------------

    A related issue raised by some commentators is whether government-
funded scientists in universities and consulting firms have sufficient 
independence from the federal agencies that support their work to be 
appropriate peer reviewers for those agencies.\22\ This concern can be 
mitigated in situations where the scientist determines the hypothesis 
to be tested or the method to be developed, which effectively creates a 
buffer between the scientist and the agency. Similarly, when an agency 
awards grants through a competitive process that includes peer review, 
the agency's potential to influence the scientist's research is 
limited. As such, when a scientist is awarded a government research 
grant through an investigator-initiated, peer-reviewed competition, 
there generally should be no question as to that scientist's ability to 
offer independent scientific advice to the agency on other projects. 
This contrasts, for example, to a situation in which a scientist has a 
consulting or contractual arrangement with the agency or office 
sponsoring a peer review. Likewise, when the agency and a researcher 
work together to design or implement a study, there is less 
independence from the agency. Furthermore, if a scientist has 
repeatedly served as a reviewer for the same agency, some may question 
whether that scientist is sufficiently independent from the agency to 
be employed as a peer reviewer on agency-sponsored projects.
---------------------------------------------------------------------------

    \22\ Lars Noah, ``Scientific `Republicanism': Expert Peer Review 
and the Quest for Regulatory Deliberation'', Emory Law Journal, 
Atlanta, Fall 2000:1066.
---------------------------------------------------------------------------

    As the foregoing suggests, independence issues pose a complex set 
of questions which much be considered by agencies when peer reviewers 
are selected. In general, agencies should make an effort to rotate peer 
review responsibilities across the available pool of qualified 
reviewers, recognizing that in some cases repeated service by the same 
reviewer is needed because of essential expertise.
    Some agencies have built entire organizations to provide 
independent scientific advice while other agencies tend to employ ad 
hoc scientific panels on specific issues. Respect for the independence 
of reviewers may be enhanced if an agency collects names of potential 
reviewers based on considerations of expertise and reputation for 
objectivity from the public, including scientific or professional 
societies. The Department of Energy's use of the American Society of 
Mechanical Engineers to identify potential peer reviewers from a 
variety of different scientific societies provides an example of how 
professional societies can assist in the development of an independent 
peer review panel.\23\
---------------------------------------------------------------------------

    \23\ American Society for Mechanical Engineers, Assessment of 
Technologies Supported by the Office of Science and Technology, 
Department of Engineering: Results of the Peer Review for Fiscal 
Year 2002, ASME Technical Publishing, Danvers, MA, 2002.
---------------------------------------------------------------------------

    Conflict of Interest. The National Academy of Sciences defines 
``conflict of interest'' as any financial or other interest that 
conflicts with the service of an individual on the review panel because 
it could impair the individual's objectivity or could create an unfair 
competitive advantage for a person or organization.\24\ This standard 
provides a

[[Page 23236]]

useful benchmark for agencies to consider in selecting peer reviewers. 
Agencies should make a special effort to examine prospective reviewers' 
potential financial conflicts, including significant investments, 
consulting arrangements, employer affiliations and grants/contracts. 
Financial ties of potential reviewers to regulated entities and 
regulatory agencies should be scrutinized when the information being 
reviewed is likely to be relevant to regulatory policy. The inquiry 
into potential conflicts goes beyond financial investments and business 
relationships and includes work as an expert witness, consulting 
arrangements, honoraria and sources of grants and contracts. To prevent 
any real or perceived conflicts of interest with potential reviewers 
and questions regarding the independence of reviewers, we refer 
agencies to federal ethics requirements, applicable standards issued by 
the Office of Government Ethics, and the prevailing practices of the 
National Academy of Sciences. Specifically, peer reviewers who are 
federal employees (including special government employees) are subject 
to federal requirements governing conflicts of interest. See, e.g., 18 
U.S.C. 208; 5 CFR part 2635. With respect to reviewers who are not 
federal employees, agencies should adopt or adapt the prevailing 
practices of the NAS regarding committee composition, conflicts, and 
balance \25\ and/or the applicable ethics requirements that have been 
developed by the U.S. government, including the standards of the Office 
of Government Ethics.\26\ Both NAS and the Federal Government recognize 
that under certain circumstances some conflict may be unavoidable in 
order to obtain the necessary expertise. See, for example, 18 U.S.C. 
208(b)(3).
---------------------------------------------------------------------------

    \24\ National Academy of Science, ``Policy and Procedures on 
Committee Composition and Balance and Conflicts of Interest for 
Committees Used in the Development of Reports,'' May 2003: Available 
at: http://www.nationalacademies.org/coi/index.html.
    \25\ Ibid.
    \26\ United States Office of Government Ethics, ``Standards of 
Ethical Conduct for Employees of the Executive Branch,'' Washington, 
D.C., 2002. Available at: http://www.usoge.gov/pages/forms_pubs_otherdocs/fpo_files/reference/rfsoc_02.pdf.
---------------------------------------------------------------------------

Disclosure Policies: Anonymous Versus Identified Reviewers
    In choosing the appropriate peer review mechanism, agencies must 
balance the need for confidentiality of reviews with the need for 
transparency. In a journal review, the most common practice is to keep 
the names and affiliations of the reviewers confidential. This 
confidentiality is designed to encourage reviewers to be candid in 
their evaluations of the draft product under review. Such 
confidentiality may also encourage participation by qualified 
scientists. However, in the context of peer review of government 
products, such confidentiality may not always add to the credibility of 
the review process. Where the issue under review is likely to have 
large public or private sector impacts, the agency may decide that more 
transparency is in the public interest. In such cases, disclosure of 
the slate of reviewer names and their qualifications can strengthen 
public confidence in the peer review process. It may be feasible to 
disclose information about reviewers without disclosing their specific 
opinions. The degree of public disclosure of information about 
reviewers should balance the need for transparency with the need to 
protect the privacy of scientists.
Public Participation
    Public comments can be important in shaping expert deliberations. 
Agencies may decide that peer review should precede an opportunity for 
public comment to ensure that the public receives the most 
scientifically strong product (rather than one that may change 
substantially as a result of peer reviewer suggestions). However, there 
are situations in which public participation in peer review is an 
important aspect of obtaining a high-quality product through a credible 
process. Agencies, however, should avoid open-ended comment periods, 
which may delay completion of peer reviews and complicate the 
completion of the final work product.
    Public participation can take a variety of forms, including 
opportunities to provide oral comments before a peer review or requests 
to provide written comment to the peer reviewers. Another option is for 
agencies to publish a ``request for comment'' or other notice in which 
they solicit public comment before a panel of peer reviewers performs 
its work.
Disposition of Reviewer Comments
    A peer review is considered completed once the Agency considers and 
addresses the reviewers' comments. All reviewer comments should be 
given reasonable consideration and be incorporated where relevant and 
valid. As part of the peer review planning process, agencies should 
determine whether they will consider reviewer comments confidential or 
make them available to the public once the reviewed document is 
disseminated. For instance, in the context of risk assessments, the 
National Academy of Sciences recommends that peer review include a 
written evaluation made available for public inspection.\27\ Reviewers 
should be informed about how their comments will be disseminated, 
whether they will be disclosed with attribution, or whether they will 
be summarized without attribution. In cases where there is a public 
panel, the agency should plan publication of both the peer review 
report(s) and the Agency's response to peer reviewer comments.
---------------------------------------------------------------------------

    \27\ National Research Council, Risk Assessment in the Federal 
Government: Managing the Process, National Academy Press, 
Washington, DC, 1983.
---------------------------------------------------------------------------

Section III: Peer Review of Highly Influential Scientific Assessments

    Whereas Section II leaves most of the considerations regarding the 
form of the peer review to the agency's discretion, Section III 
requires a more rigorous form of peer review for highly influential 
scientific assessments. The requirements of Section II of this Bulletin 
apply to Section III. In addition, Section III has some specific 
requirements, which are discussed below. In planning a peer review 
under Section III, agencies typically will have to devote greater 
resources and attention to the issues discussed in Section II, i.e., 
individual versus panel review; timing; the scope of the review; the 
selection of reviewers; disclosure; public participation; and 
disposition of reviewer comments.
    The term ``scientific assessment'' means an evaluation of a body of 
scientific or technical knowledge which typically synthesizes multiple 
factual inputs, data, models, assumptions, and/or applies best 
professional judgment to bridge uncertainties in the available 
information. These assessments include, but are not limited to, state-
of-science reports, technology assessments, weight-of-evidence 
analyses, meta-analyses, risk assessments, toxicological profiles of 
substances, integrated assessment models, hazard determinations, 
exposure assessments, or health, ecological, or safety assessments. 
Typically, the data and models used in scientific assessments have 
already been subject to some form of peer review (e.g., refereed 
journal peer review or peer review under Section II of this Bulletin).
    A scientific assessment is considered ``highly influential'' if the 
agency or the OIRA Administrator determines that the dissemination 
could have a clear and substantial impact on important public policies 
(including regulatory actions) or private sector decisions with a 
potential effect of more than $500 million in any one year or that the

[[Page 23237]]

dissemination involves precedent setting, novel and complex approaches, 
or significant interagency interest. One of the ways information can 
exert economic impact is through the costs or benefits of a regulation 
based on the disseminated information. The qualitative aspect of this 
definition may be most useful in cases where it is difficult for an 
agency to predict the potential economic effect of dissemination. If 
information is covered by Section III, an agency is required to adhere 
to the peer-review procedures specified in Section III.
    With regard to the selection of reviewers, Section III(2)(a) 
emphasizes consideration of expertise and balance. Expertise refers to 
the required knowledge, experience and skills required to perform the 
review whereas balance refers to the need for diversity in scientific 
perspective and disciplines. We emphasize that the term ``balance'' 
here refers not to balancing of stakeholder or political interests but 
rather to a broad and diverse representation of respected perspectives 
and intellectual traditions within the scientific community.
    Section III(2)(b) instructs agencies to consider barring 
participation by scientists with a conflict of interest. The conflict 
of interest standards for Sections II and III of the Bulletin are 
identical. As discussed under Section II, those peer reviewers who are 
federal employees, including Special Government Employees, are subject 
to applicable statutory and regulatory standards for federal employees. 
For non-government employees, agencies should adopt or adapt the 
applicable ethical standards used by the Federal Government and/or the 
NAS.
    Section III(2)(c) instructs agencies to ensure that reviewers are 
independent of the agency sponsoring the review. Scientists employed by 
the sponsoring agency are not permitted to serve as reviewers for 
highly influential scientific information. This does not preclude 
Special Government Employees, such as academics appointed to advisory 
committees, from serving as peer reviewers. Agencies (or their 
contractors) should seek and consider potential reviewers who have been 
nominated based on their expertise and objectivity by the public, 
including scientific and professional societies. We considered whether 
a reviewer is independent of the agency if that reviewer receives a 
substantial amount of research funding from the agency sponsoring the 
review. Research grants that were awarded to the scientist based on 
investigator-initiated, competitive, peer-reviewed proposals do not 
generally raise issues of independence. However, significant consulting 
and contractual relationships with the agency may raise issues of 
independence or conflict, depending upon the situation. Repeated use of 
the same reviewer in multiple assessments may raise issues of 
independence unless the particular reviewer's expertise is essential. 
Agencies can generally avoid the effect of use of the same reviewer by 
rotating membership across the available pool of qualified reviewers. 
Similarly, when using standing panels of scientific advisors, we 
suggest rotating membership among qualified scientists in order to 
obtain fresh perspectives and reinforce the reality and perception of 
independence from the agency. Section III(3)(c) also requires agencies 
to consider the prevailing selection practices used by the National 
Academy of Sciences, since they were designed to ensure independence 
from sponsors in the Federal Government.
    Section III(3) requires agencies to provide reviewers with 
sufficient background information, including access to key studies, 
data and models, to perform their role as peer reviewers. In this 
respect, the peer review envisioned in Section III is more rigorous 
than some forms of journal peer review, where the reviewer is often not 
provided access to underlying data or models. Reviewers should be 
informed of applicable access, objectivity, reproducibility and other 
quality standards under federal information quality laws.
    Section III(4) addresses opportunity for public participation in 
peer review, and provides that the agency should, wherever possible, 
provide for public participation. In some cases, an assessment may be 
so sensitive that it is critical that the agency's assessment achieve a 
high level of quality before it is publicized. In those situations, a 
rigorous yet confidential peer-review process may be appropriate, prior 
to public release of the assessment. If an agency decides to make a 
draft assessment publicly available at the onset of a peer review 
process, the agency shall, whenever possible, provide a vehicle for the 
public to provide written comments, make an oral presentation before 
the peer reviewers, or both. When written public comments are received, 
the agency should ensure that peer reviewers receive copies of comments 
that address significant scientific issues with ample time to consider 
them in their review.
    Section III(5) requires that agencies instruct reviewers to prepare 
a peer review report that describes the nature and scope of their 
review and their findings and conclusions. The report should disclose 
the name of each peer reviewer and a brief description of their 
organizational affiliation, credentials and relevant experiences. When 
the agency uses a panel, the peer review report should either summarize 
the views of the group as a whole (including any dissenting views) or 
summarize the views of individual reviewers (with or without 
attribution of specific views to specific names). The agency must also 
prepare a written response to the peer review report, indicating 
whether the agency agrees with the reviewers and what actions the 
agency has taken or plans to take to address the points made by 
reviewers. The agency is required to disseminate the peer review report 
and the agency's response to the report on the agency's web site, 
including all the materials related to the peer review such as charge 
statement, peer review report, and agency response to the review.
    Section III(6) authorizes but does not require an agency to 
commission an entity independent of the agency to select peer reviewers 
and/or manage the peer review process in accordance with this section. 
The entity may be a scientific or professional society, a firm 
specializing in peer review, or a non-profit organization with 
experience in peer review.

Section IV: Alternative Procedures

    Peer review as described in this Bulletin is only one of many 
procedures that agencies can employ to ensure an appropriate degree of 
pre-dissemination quality of influential scientific information. As an 
alternative to complying with Sections II and III of this Bulletin, an 
agency may instead (1) rely on scientific information produced by the 
National Academy of Sciences, (2) commission the National Academy of 
Sciences to peer review an agency draft scientific information product, 
or (3) employ an alternative procedure or set of procedures, 
specifically approved by the OIRA Administrator in consultation with 
OSTP, that ensures that the scientific information product meets 
applicable information-quality standards. For example, an agency might 
choose to commission a respected third party other than the NAS (e.g., 
the Health Effects Institute or the National Commission on Radiation 
Protection and Measurement) to conduct an assessment or series of 
related assessments. The purpose of Section IV is to encourage 
innovation in the methods used to ensure pre-dissemination quality 
control of influential scientific information.

[[Page 23238]]

Section V: Peer Review Planning

    Section V requires agencies to begin a systematic process of peer 
review planning for influential scientific information and highly 
influential scientific assessments that the agency plans to disseminate 
in the foreseeable future. A key feature of planning is a web site 
listing of forthcoming influential scientific disseminations that is 
regularly updated by the agency, at least every six months. Each entry 
on the list of forthcoming disseminations should include a preliminary 
title of the planned report, a short paragraph describing the subject 
and purpose of the planned report, and an agency contact person. In 
addition, the agency should briefly describe its peer review plan, 
including the anticipated number of reviewers (3 or less; 4-10; more 
than 10), whether they shall work as individuals or a panel, and a 
succinct description of the primary disciplines or types of skills, 
expertise and experience needed in the review.
    In addition, each peer review plan shall include the following: (1) 
Whether reviewers will be selected by the agency or by a designated 
outside organization; (2) whether the public, including scientific or 
professional societies, will be asked to nominate potential peer 
reviewers; (3) whether there will be opportunities for the public to 
comment on the work product to be peer reviewed, and if so, how and 
when these opportunities will be provided; and (4) whether or not the 
agency will provide peer reviewers copies of significant and relevant 
public comments prior to doing their work.
    The peer review agenda will allow agencies to gauge the extent of 
public interest in the peer review process for influential scientific 
information. The agenda can also be used by the public to monitor 
agency compliance with this Bulletin. The Bulletin requires agencies to 
update their peer review agenda at least every six months. However, in 
some cases--particularly for highly influential scientific assessments 
and other particularly important information products--more frequent 
updates of existing entries on the agenda, or the addition of new 
entries to the agenda, may be warranted. When new entries are added to 
the agenda of forthcoming reports and other information products, the 
public should be provided with sufficient time to comment on the 
agency's peer review plan for that report or product. Agencies shall 
consider public comments on the peer review plan. Agencies are 
encouraged to offer some form of listserve for members of the public 
who would like to be notified by email each time an agency's peer 
review agenda has been updated.
    The peer review planning requirements of this Bulletin are designed 
to be implemented in phases. Specifically, the planning requirements of 
the Bulletin will go into effect for documents subject to Section III 
of the Bulletin (highly influential scientific assessments) four months 
after publication. However, the planning requirements do not go into 
effect for documents subject to Section II of the Bulletin until one 
year after publication. It is expected that agency experience with the 
planning requirements of the Bulletin for the smaller scope of 
documents encompassed in Section III will be used to inform 
implementation of these planning requirements for the larger scope of 
documents covered under Section II.

Section VI: Certification in the Administrative Record

    If an agency relies on influential scientific information subject 
to the requirements of this Bulletin in support of a regulatory action, 
the agency shall include in the administrative record for that action a 
certification that explains how the agency has complied with this 
Bulletin and the Information Quality Act. Relevant materials are to be 
placed in the administrative record.

Section VII: Safeguards and Waivers

    Section VII establishes basic procedures to protect privacy and 
confidentiality concerns, and to allow for waiver of the requirements 
of the Bulletin where necessary. First, peer review must be conducted 
in a manner that respects privacy interests, confidential business 
information, and intellectual property. Second, the agency head may 
waive or defer some or all of the peer review requirements of Sections 
II or III of this Bulletin if there is a compelling rationale for 
waiver or deferral. If the agency head waives the peer review 
requirements prior to dissemination, peer review should be conducted as 
soon as practicable thereafter.

Section VIII: Exemptions

    There are a variety of situations where agencies need not conduct 
peer review under this Bulletin. These include, for example, 
disseminations of sensitive information related to national security, 
foreign affairs, or negotiations involving international treaties and 
trade where compliance with this Bulletin would interfere with the need 
for secrecy or promptness.
    An information product is not covered by the Bulletin unless it 
represents an official view of one or more Departments or agencies of 
the Federal Government. Since the Bulletin covers only official 
``disseminations'' of the U.S. government, it does not cover 
information products released by government-funded scientists (e.g., 
those supported extramurally or intramurally by federal agencies, or 
those working in state or local governments with federal support) if 
those information products are not represented as the views of the 
agency or Department supporting the research. In cases where the 
imprimatur of the Federal Government is not intended, government-funded 
scientists are advised to include a statement with their disseminated 
work indicating that ``the views in this report are those of the 
author(s) and do not necessarily represent the views of the funding 
agency''.
    This Bulletin does not cover official disseminations that arise in 
adjudications and permit proceedings, unless the agency determines that 
the influential dissemination is scientifically or technically novel 
(i.e., a major change in accepted practice) and likely to have 
precedent-setting influence on future adjudications or permit 
proceedings. This exclusion is intended to cover, among other things, 
licensing, approval and registration processes for specific products 
and development activities, as well as site-specific disseminations 
such as those made under Superfund or the National Environmental Policy 
Act (NEPA). The Bulletin also does not directly cover information 
supplied to the government by third parties (e.g., studies by private 
consultants, companies and private, non-profit organizations, or 
research institutions such as universities). However, if a Department 
or agency plans to disseminate information supplied by a third party 
(i.e., using this information to support decisions, thereby adopting 
this information as an official dissemination), the requirements of the 
Bulletin apply, assuming the dissemination is ``influential''.
    The Bulletin does not cover time-sensitive medical, health, and 
safety disseminations (for this purpose, ``health'' includes public 
health, or plant or animal infectious diseases), or disseminations 
based primarily on data from a recent clinical trial that was 
adequately peer reviewed before the trial began.
    This Bulletin covers original data and formal analytic models used 
by agencies in Regulatory Impact Analyses (RIAs). However, the RIA 
documents themselves are already reviewed through an interagency review 
process

[[Page 23239]]

under EO 12866 that involves application of the principles and methods 
defined in OMB Circular A-4. In that respect, RIAs are excluded from 
coverage by this Bulletin, although agencies are encouraged to have 
RIAs reviewed by peers within the government for adequacy and 
completeness. One model for such a review prior to submission to OIRA 
is offered by the Interagency Economic Peer Review (IEPR). The IEPR 
comprises agency economists engaged in benefit-cost analysis from 
across the Federal Government.
    The Bulletin does not cover accounting, budget, and financial 
information including that which is generated or used by agencies that 
focus on interest rates, banking, currency, securities, commodities, 
futures, or taxes.
    Routine statistical information released by federal statistical 
agencies (e.g., periodic demographic and economic statistics) and the 
analysis of these data to compute standard indicators and trends (e.g., 
unemployment and poverty rates) is excluded from this Bulletin.
    The Bulletin does not cover information disseminated in connection 
with rules that materially alter entitlements, grants, user fees, or 
loan programs or the rights and obligations of recipients thereof, 
other than influential scientific information disseminated in 
connection with non-routine rules in this category.
    In general, the Bulletin does not impose new peer-review 
requirements on information that has already been adequately peer 
reviewed. Under the terms of the Bulletin, agencies should exercise 
discretion in determining when a draft information product has already 
been adequately peer reviewed. The mere existence of a public comment 
process (e.g., notice-and-comment procedures under the Administrative 
Procedures Act) does not constitute adequate peer review, because it 
does not assure that qualified, impartial specialists in relevant 
fields have performed a critical evaluation of the agency's draft 
product.\28\ For both Sections II and III of this Bulletin, principal 
findings, conclusions and recommendations in official reports of the 
National Academy of Sciences are generally presumed to have been 
adequately peer reviewed. Publication in a refereed scientific journal 
may mean that adequate peer review has been performed. However, because 
the intensity of journal review is highly variable, there may be cases 
in which an agency determines that a more rigorous or transparent 
review process is necessary. For instance, an agency may determine a 
particular journal review process did not address all of the questions 
that the agency should address before publishing a report. In addition, 
because science primarily advances through further research in which 
new data challenges prior theories, prior peer review and publication 
is not by itself sufficient grounds for determining that no further 
review is necessary.
---------------------------------------------------------------------------

    \28\ William W. Lowrance, Modern Science and Human Values, 
Oxford University Press, New York, NY 1985: 86.
---------------------------------------------------------------------------

    Congress has assigned the NAS a special role in advising the 
Federal Government on scientific and technical issues. The peer-review 
procedures of the NAS are generally quite rigorous, and thus agencies 
should presume that major findings from NAS reports have been 
adequately peer reviewed.
    If information is disseminated pursuant to an exemption to this 
Bulletin, subsequent disseminations are not automatically exempted. For 
example, if influential scientific information is first disseminated in 
the course of an exempt agency adjudication, but is later disseminated 
in the context of a non-exempt rulemaking, the subsequent dissemination 
will be subject to the requirements of this Bulletin even though the 
first dissemination was not.

Section IX: OIRA and OSTP Responsibilities

    OIRA, in consultation with OSTP, is responsible for overseeing 
agency implementation of the requirements of this Bulletin. In order to 
foster learning about peer review practices across agencies, OIRA and 
OSTP shall form an interagency workgroup on peer review that meets 
regularly, discusses progress and challenges, and recommends 
improvements to peer review practices under the Bulletin.

Section X: Effective Date and Existing Law

    The requirements of this Bulletin, with the exception of Section V, 
apply to information disseminated on or after four months after 
publication of this Bulletin. However, the Bulletin does not apply to 
information products that are already being addressed by an agency-
initiated peer review process (e.g., a draft is already being reviewed 
by a formal scientific advisory committee established by the agency). 
An existing peer review mechanism mandated by law should be implemented 
by the agency in a manner as consistent as possible with the practices 
and procedures outlined in this Bulletin. As noted above, the 
requirements in Section V apply to ``highly influential scientific 
assessments,'' as designated in Section III of the Bulletin, within 
four months of publication of the final Bulletin. The requirements in 
Section V apply to documents subject to Section II of the Bulletin one 
year after publication of the final Bulletin.

Section XI: Judicial Review

    This Bulletin is intended to improve the internal management of the 
executive branch and is not intended to create any new right or 
benefit, substantive or procedural, enforceable at law or in equity, 
against the United States, its departments, agencies, or other 
entities, its officers or employees, or any other person. Nor does this 
Bulletin abridge any existing rights of action. Consistent with current 
law, materials generated during the peer review process may be 
considered by courts adjudicating existing rights of action.

Bulletin for Peer Review

I. Definitions

    For purposes of this Bulletin--
    1. The term ``Administrator'' means the Administrator of the Office 
of Information and Regulatory Affairs in the Office of Management and 
Budget;
    2. The term ``agency'' has the same meaning as in the Paperwork 
Reduction Act, 44 U.S.C. 3502(1);
    3. The term ``dissemination'' means agency initiated or sponsored 
distribution of information to the public (see 5 CFR 1320(d) 
(definition of ``Conduct or Sponsor'')). Dissemination does not include 
distribution limited to government employees or agency contracts or 
grantees; intra- or inter-agency use or sharing of government 
information; or responses to requests for agency records under the 
Freedom of Information Act, the Privacy Act, the Federal Advisory 
Committee Act or similar law. This definition also excludes 
distribution limited to correspondence with individuals or persons, 
press releases, archival records, public filings, subpoenas and 
adjudicative processes. The term ``dissemination'' also excludes 
information distributed for peer review in compliance with this 
Bulletin, provided that the distributing agency includes a clear 
disclaimer on the information as follows: ``THIS INFORMATION IS 
DISTRIBUTED SOLELY FOR THE PURPOSE OF PRE-DISSEMINATION PEER REVIEW 
UNDER APPLICABLE INFORMATION QUALITY GUIDELINES. IT HAS NOT

[[Page 23240]]

BEEN FORMALLY DISSEMINATED BY [THE AGENCY] AND SHOULD NOT BE CONSTRUED 
TO REPRESENT ANY AGENCY DETERMINATION OR POLICY'';
    4. The term ``influential scientific information'' means scientific 
information the dissemination of which the agency reasonably can 
determine that dissemination of which will have or does have a clear 
and substantial impact on important public policies or private sector 
decisions;
    5. The term ``Information Quality Act'' means section 515 of the 
Treasury and General Government Appropriations Act for Fiscal Year 2001 
(Pub. L. 106-554; H.R. 5658);
    6. The term ``scientific assessment'' means an evaluation of a body 
of scientific or technical knowledge which typically synthesizes 
multiple factual inputs, data, models, assumptions, and/or applies best 
professional judgment to bridge uncertainties in the available 
information. These assessments include, but are not limited to, state-
of-science reports, technology assessments, weight-of-evidence 
analyses, meta-analyses, risk assessments, toxicological profiles of 
substances, integrated assessment models, hazard determinations, 
exposure assessments, or health, ecological, or safety assessments, and
    7. The term ``scientific information'' means factual inputs, data, 
models, analyses, or scientific assessments related to the behavioral 
and social sciences, public health and medical sciences, life and earth 
sciences, engineering, or physical sciences. This includes any 
communication or representation of knowledge such as facts or data, in 
any medium or form, including textual, numerical, graphic, 
cartographic, narrative, or audiovisual forms. This definition includes 
information that an agency disseminates from a web page, but does not 
include the provision of hyperlinks to information that others 
disseminate. This definition does not include opinions, where the 
agency's presentation makes clear that what is being offered is 
someone's opinion rather than fact or the agency's views.

II. Peer Review of Influential Scientific Information

    1. In General: To the extent permitted by law, each agency shall 
have a peer review conducted on all influential scientific information 
that the agency intends to disseminate. Agencies need not, however, 
have peer review conducted on information that has already been 
subjected to adequate peer review.
    2. Adequacy of Peer Review: To be considered ``adequate'' for 
purposes of the preceding paragraph, a peer review need not comply with 
all of the requirements of this Bulletin. An agency may deem a prior 
peer review adequate if it determines that the peer review was 
sufficiently rigorous in light of the novelty and complexity of the 
science to be reviewed and the benefit and cost implications. For both 
Sections II and III of this Bulletin, principal findings, conclusions 
and recommendations in official reports of the National Academy of 
Sciences are generally presumed to have been adequately peer reviewed.
    3. Choice of Peer Review Mechanism: When planning a peer review for 
influential scientific information, the agency shall select an 
appropriate peer review mechanism based on the novelty and complexity 
of the science to be reviewed and the benefit and cost implications. 
Depending on these factors, appropriate peer review mechanisms can 
range from review by qualified specialists within the Federal 
Government to formal review by an independent body of experts outside 
the government. Peer reviewers shall be selected on the basis of 
necessary technical or scientific expertise, and should not have 
participated in development of the work product.
    4. Conflicts: In order to properly handle participation by 
scientists with a conflict of interest, the agency--or the entity 
selecting the peer reviewers--shall (i) ensure that those reviewers 
serving as federal employees (including special government employees as 
defined in 18 U.S.C. 202(a)) comply with applicable federal ethics 
requirements (ii) apply or adapt the federal ethics requirements for 
reviewers who are not federal employees; and (iii) consider the 
conflict of interest policy used by the National Academy of Sciences, 
including principles regarding potential financial conflicts arising 
from factors such as a reviewers' investments, employer and business 
affiliations, grants, contracts and consulting income. For scientific 
assessments relevant to specific regulations, a reviewer's financial 
ties to both regulated entities (e.g., businesses) and the agency 
should be examined.
    5. Transparency: A detailed summary or copy of the reviewers' 
comments, as a group or individually, shall be made available to the 
public and, where appropriate, be made part of the administrative 
record for related agency actions. Agencies shall consider the comments 
of the reviewers.

III. Additional Peer Review Requirements for Highly Influential 
Scientific Assessments

    1. Applicability: This section applies to influential scientific 
information which the agency or the Administrator determines is a 
scientific assessment that:
    (i) Could have a clear and substantial impact on important public 
policies (including regulatory actions) or private sector decisions 
with a potential effect of more than $500 million in any year, or
    (ii) Involves precedent setting, novel, and complex approaches, or 
significant interagency interest.
    2. Selection of Reviewers:
    a. Expertise and Balance: Peer reviewers shall be selected to 
provide the necessary expertise, experience and skills, including 
specialists from multiple disciplines, as necessary. The group of 
reviewers shall be sufficiently broad and diverse to fairly represent 
the relevant scientific perspectives and fields of knowledge.
    b. Conflicts: In order to properly handle participation by 
scientists with a conflict of interest, the agency--or the entity 
selecting the peer reviewers--shall (i) ensure that those reviewers 
serving as federal employees (including special government employees) 
comply with applicable federal ethics requirements; (ii) apply or adapt 
the federal ethics requirements for reviewers who are not federal 
employees; and (iii) consider the conflict of interest policy used by 
the National Academy of Sciences, including principles regarding 
potential financial conflicts arising from factors such as a reviewers' 
investments, employer and business affiliations, grants, contracts and 
consulting income. For scientific assessments relevant to specific 
regulations, a reviewer's financial ties to both regulated entities 
(e.g., businesses) and the agency should be examined.
    c. Independence: In order to ensure participation by scientists who 
are independent of the agency sponsoring the review, the agency--or 
entity selecting the reviewers--shall (i) bar participation by 
scientists employed by the agency sponsoring the review unless the 
reviewer's service as a peer reviewer defines the government employment 
(i.e., special government employees); (ii) consider requesting the 
nomination of potential reviewers based on expertise and objectivity 
from the public, including scientific and professional societies; and 
(iii) consider the prevailing selection practices of the National 
Academy of Sciences

[[Page 23241]]

concerning ties of a potential committee member to the sponsoring 
agency. Agencies should avoid repeated use of the same reviewer on 
multiple assessments unless his or her participation is essential. 
Agencies are encouraged to rotate membership on panels across the pool 
of qualified reviewers. Research grants that were awarded to scientists 
based on investigator-initiated, competitive, peer-reviewed proposals 
generally do not raise issues as to independence or conflicts.
    3. Information Access: The agency--or entity managing the peer 
review--shall provide the reviewers with sufficient information--
including background information about key studies or models--to enable 
them to understand the data, analytic procedures, and assumptions used 
to support the key findings or conclusions of the draft assessment. 
Reviewers shall be informed of applicable access, objectivity, 
reproducibility and other quality standards under the federal laws 
governing information access and quality.
    4. Opportunity for Public Participation: If the agency decides to 
make a draft assessment publicly available at the same time it is 
submitted for peer review (or during the peer review process), the 
agency shall, whenever practical, provide to peer reviewers a 
compilation or summary of relevant public comments on the draft 
assessment that address significant scientific or technical issues. 
When there is sufficient public interest, the agency--or entity 
managing the peer review--shall consider establishing a public comment 
period for a draft report and sponsoring a public meeting where oral 
presentations on scientific issues can be made to the peer reviewers by 
interested members of the public. Time limits for public participation 
shall be specified.
    5. Peer Review Reports: The agency--or entity managing the peer 
review--shall instruct peer reviewers to prepare a report that 
describes the nature of their review and their findings and 
conclusions. The peer review report should either summarize the views 
of individual reviewers (either with or without specific attributions, 
as long as the reviewers are informed in advance of the agency's plans 
for disclosure) or represent the views of the group as a whole 
(including any dissenting views). The peer review report shall also 
disclose the names, organizational affiliations, and a short paragraph 
on the credentials and relevant experiences of each peer reviewer. The 
agency is required to prepare a written response to the peer review 
report explaining: The agency's agreement or disagreement; any actions 
the agency has undertaken or will undertake in response to the report; 
and (if applicable) the reasons the agency believes those actions 
satisfy any key concerns or recommendations in the report. The agency 
shall disseminate the final peer review report and the agency's written 
statement of response on the agency's Web site, and all the materials 
related to the peer review (charge statement, peer review report, and 
agency response) shall be included in the administrative record for any 
related agency action.
    6. Selection and Management of Peer Review Panel: The agency may 
commission entities independent of the agency to select peer reviewers 
and/or manage the peer review process in accordance with this section.

IV. Alternative Procedures

    As an alternative to complying with Sections II and III of this 
Bulletin, an agency may instead: (i) Rely on a scientific information 
produced by the National Academy of Sciences; (ii) commission the 
National Academy of Sciences to peer review an agency draft scientific 
information product; or (iii) employ an alternative scientific 
procedure or process, specifically approved by the Administrator in 
consultation with OSTP, that ensures that the scientific information 
product satisfies applicable information quality standards. The 
alternative procedure(s) may be applied to a single report or group of 
reports.

V. Peer Review Planning

    1. Peer Review Agenda: Each agency shall post on its Internet Web 
site, and update at least every six months, an agenda designating all 
planned and ongoing influential scientific information subject to 
Section II and highly influential scientific assessments subject to 
Section III of this Bulletin.
    2. Peer Review Plans:
    a. General Requirements: For each entry on the agenda that is 
subject to this Bulletin, the agency shall describe the peer review 
plan. Each peer review plan shall include: (i) A paragraph including 
the title, subject and purpose of the planned report, as well as an 
agency contact to whom inquiries may be directed to learn the specifics 
of the plan; (ii) whether the review will be conducted by a panel or 
individual letters; (iii) the anticipated number of reviewers (3 or 
less; 4-10; or more than 10); and (iv) a succinct description of the 
primary disciplines or types of expertise needed in the review.
    b. Designations: Each peer review plan shall designate the 
following: (i) Whether reviewers will be selected by the agency or by a 
designated outside organization; (ii) whether the public, including 
scientific or professional societies, will be asked to nominate 
potential peer reviewers; (iii) whether there will be opportunities for 
the public to comment on the work product to be peer reviewed, and if 
so, how and when these opportunities will be provided; and (iv) whether 
the agency will provide peer reviewers copies of significant and 
relevant public comments prior to doing their work.
    c. Agenda Updates: Agencies are encouraged to offer a listserve to 
alert interested members of the public when new entries are added or 
updated.
    d. Public Comment: Agencies shall establish a mechanism for 
allowing the public to comment on the adequacy of the peer review plans 
and designations. Agencies must consider public comments on peer review 
plans.

VI. Certification in the Administrative Record

    If an agency relies on influential scientific information or a 
highly influential scientific assessment subject to the requirements of 
this Bulletin in support of a regulatory action, it shall include in 
the administrative record for that action a certification explaining 
how the agency has complied with the requirements of this Bulletin and 
the Information Quality Act.

VII. Safeguards and Waivers

    1. Privacy and Confidentiality: Peer review shall be conducted in a 
manner that respects (i) privacy interests; (ii) confidential business 
information; and (iii) intellectual property.
    2. Waiver: The agency head may waive or defer some or all of the 
peer review requirements of Section II and III of this Bulletin where 
warranted by a compelling rationale. If the agency head waives the peer 
review requirements prior to dissemination, peer review should be 
conducted as soon as practicable thereafter.

VIII. Exemptions

    Agencies need not have peer review conducted on information that 
is:
    1. Related to national security, foreign affairs, or negotiations 
involving international trade or treaties where compliance with this 
Bulletin would interfere with the need for secrecy or promptness;
    2. Produced by government-funded scientists (e.g., those supported 
extramurally or intramurally by federal agencies or those working in 
state or

[[Page 23242]]

local governments with federal support) if those information products 
are not represented as the views of a Department or agency. To qualify 
for this exemption, scientists are advised to include in their 
information product a clear disclaimer that ``the views in this report 
are those of the author(s) and do not necessarily represent the views 
of the funding agency'';
    3. Disseminated in the course of an individual agency adjudication 
or permit proceeding (including a registration, approval, licensing, 
site-specific determination), unless the agency determines that the 
influential dissemination is scientifically or technically novel and 
likely to have precedent-setting influence on future adjudications and/
or permit proceedings;
    4. A medical, health, or safety dissemination where the agency 
determines that the dissemination is time-sensitive or is based 
primarily on data from a recent clinical trial that was adequately peer 
reviewed before the trial began.
    5. An agency regulatory impact analysis or regulatory flexibility 
analysis subject to interagency review under Executive Order 12866;
    6. Routine statistical information released by federal statistical 
agencies (e.g., periodic information about unemployment and poverty 
rates);
    7. Accounting, budget, and financial information, including that 
which is generated or used by agencies that focus on interest rates, 
banking, currency, securities, commodities, futures, or taxes; or
    8. Information disseminated in connection with rules that 
materially alter entitlements, grants, user fees, or loan programs, or 
the rights and obligations of recipients thereof, except that 
influential scientific information disseminated in connection with non-
routine rules is not exempt.

IX. Responsibilities of OIRA and OSTP

    OIRA, in consultation with OSTP, shall be responsible for 
overseeing implementation of the requirements of this Bulletin. An 
interagency group, chaired by OSTP and OIRA, shall meet periodically to 
foster better understanding about peer review practices and to assess 
progress in the implementation of this Bulletin.

X. Effective Date and Existing Law

    The requirements of this Bulletin, with the exception of those in 
Section V (Peer Review Planning), apply to information disseminated on 
or after four months after publication, except that they do not apply 
to information for which an agency has already commenced a peer-review 
process. Any existing peer review mechanisms mandated by law should be 
employed in a manner as consistent as possible with the practices and 
procedures laid out herein. The requirements in Section V apply to 
``highly influential scientific assessments,'' as designated in Section 
III of this Bulletin, within four months of publication. The 
requirements in Section V apply to documents subject to Section II of 
this Bulletin one year after publication.

XI. Judicial Review

    This Bulletin is intended to improve the internal management of the 
executive branch, and is not intended to create any new right or 
benefit, substantive or procedural, enforceable at law or in equity, 
against the United States, its departments, agencies, or other 
entities, its officers or employees, or any other person. Nor does this 
Bulletin abridge any existing rights of action. Consistent with current 
law, materials generated during the peer review process may be 
considered by courts adjudicating existing rights of action.

[FR Doc. 04-9572 Filed 4-27-04; 8:45 am]
BILLING CODE 3110-01-P