Peer Review: EPA Needs Implementation Procedures and Additional Controls
(Letter Report, 02/22/94, GAO/RCED-94-89).
In May 1991, the head of the Environmental Protection Agency (EPA)
convened a panel of outside academicians to review the role of science
at EPA and evaluate how the agency could meet its goal of using sound
science as the basis for decision-making. The panel reported the next
year that science was of uneven quality at EPA, causing the agencies'
policies and regulations to be perceived as lacking strong scientific
support. The panel recommended that EPA establish a uniform peer review
process for all scientific and technical products used to support EPA
guidance and regulations. A February 1993 GAO report (GAO/RCED-93-77R)
commented on weaknesses in internal controls over EPA's peer review
process that contributed to the premature release of a draft EPA report
on environmental tobacco smoke. This report examines (1) EPA's efforts
to implement a uniform policy on peer reviews and (2) its controls over
documents sent to outside reviewers.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: RCED-94-89
TITLE: Peer Review: EPA Needs Implementation Procedures and
Additional Controls
DATE: 02/22/94
SUBJECT: Internal controls
Accountability
Environmental policies
Publications
Policy evaluation
Evaluation methods
Scientific research
Information dissemination operations
Research reports
Agency reports
**************************************************************************
* This file contains an ASCII representation of the text of a GAO *
* report. Delineations within the text indicating chapter titles, *
* headings, and bullets are preserved. Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are *
* identified by double and single lines. The numbers on the right end *
* of these lines indicate the position of each of the subsections in the *
* document outline. These numbers do NOT correspond with the page *
* numbers of the printed product. *
* *
* No attempt has been made to display graphic images, although figure *
* captions are reproduced. Tables are included, but may not resemble *
* those in the printed version. *
* *
* A printed copy of this report may be obtained from the GAO Document *
* Distribution Facility by calling (202) 512-6000, by faxing your *
* request to (301) 258-4066, or by writing to P.O. Box 6015, *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time. *
**************************************************************************
Cover
================================================================ COVER
Report to the Chairman, Subcommittee on Oversight and Investigations,
Committee on Energy and Commerce, House of Representatives
February 1994
PEER REVIEW - EPA NEEDS
IMPLEMENTATION PROCEDURES AND
ADDITIONAL CONTROLS
GAO/RCED-94-89
Peer Review
Abbreviations
=============================================================== ABBREV
EPA - Environmental Protection Agency
AOQPS - Office of Air Quality Planning and Standards
ORD - Office of Research and Development
Letter
=============================================================== LETTER
B-253525
February 22, 1994
The Honorable John D. Dingell
Chairman, Subcommittee on
Oversight and Investigations
Committee on Energy and Commerce
House of Representatives
Dear Mr. Chairman:
In May 1991, the Administrator, Environmental Protection Agency
(EPA), established a panel of outside academicians to review the role
of science at EPA and evaluate how the agency can meet the goal of
using sound science as the foundation for the agency's
decision-making. In March 1992, the expert panel reported that
science at EPA is of uneven quality, and as a result, the agency's
policies and regulations are frequently perceived as lacking strong
scientific support. The panel recommended that EPA establish a
uniform peer review process for all scientific and technical products
used to support EPA guidance and regulations. For this review, we
interpreted peer review as the final critical evaluation of
scientific and technical program products by independent experts.
In 1993, we reported on the premature release, by an external
reviewer, of a draft compendium to an EPA report on environmental
tobacco smoke.\1 In that report, we cited weaknesses in internal
controls over EPA's peer review process as contributing to the
premature release. You then asked us to review EPA's policies and
practices regarding the peer review of scientific and technical
materials published or issued by the agency. Specifically, we
examined (1) EPA's efforts to implement a uniform policy on peer
reviews and (2) its controls over documents sent to outside
reviewers.
--------------------
\1 Environmental Tobacco Smoke (GAO/RCED-93-77R, Feb. 8, 1993).
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
In response to the panel's recommendations, in January 1993, the EPA
Administrator issued a peer review policy statement requiring that
technically based products undergo peer reviews. However, the policy
statement does not define technically based products and does not
specify the implementation procedures or steps needed to perform peer
reviews. Ongoing efforts to define technically based products and
develop implementation procedures have been delayed due to concerns
raised by the various EPA offices that will have to implement the
policy. These concerns are about, among other things, identifying
which products to review, as well as the scope, timing, and cost of
the reviews. To resolve these issues, EPA established a work
group--consisting of representatives from its program and regional
offices and the Council of Science Advisors\2 --to develop specific
implementation procedures. However, EPA has not established
milestones or deadlines for this group. Plans are to address, in
phases over an unspecified period of time, concerns that are
hampering agencywide implementation of the policy statement.
EPA does not have consistent agencywide controls over products being
sent to external peer reviewers to help reduce the possibility that
the products will be prematurely released and perceived as agency
policy. EPA has no procedures to ensure that documents sent to
external peer reviewers are stamped as drafts and/or include a
disclaimer stating that the documents are drafts and do not
necessarily represent the views of the agency. Also, EPA does not
always have a written agreement with outside peer reviewers that they
will not release draft documents without the agency's approval. EPA
work groups developing peer review implementation procedures have no
plans to address these problems. Therefore, the potential continues
to exist that documents may be prematurely released by reviewers and
perceived as EPA's position.
--------------------
\2 The Council consists of science advisors from each of EPA's
program and regional offices.
BACKGROUND
------------------------------------------------------------ Letter :2
Nearly a decade ago, EPA learned that the author of seven of the
eight key studies it used to support proposed revisions to the carbon
monoxide standard was being investigated by the Food and Drug
Administration and the Veterans Administration for the falsification
of research. Although peer reviews were conducted for these studies,
they did not adequately address all relevant concerns. For example,
the peer reviews had not looked closely at the author's data or
methodology because of his high esteem within the scientific
community. When the investigators questioned the scientific validity
of the studies, EPA delayed the standards, audited the research data,
and concluded that the studies were unreliable for its purposes.
In response to concerns that some of EPA's decisions could be based
on unreliable data, the Administrator wanted to institute a uniform
peer review policy that would ensure that all of the agency's
decisions would be based on objective, credible, unbiased scientific
and technological data. EPA's 5 program offices\3 and 10 regional
offices generate scientific and technical data and analyses to
support the agency's regulatory, enforcement, and standard-setting
decisions. Such data and analyses are also developed by external
organizations under contract to various EPA offices.
--------------------
\3 The Offices of Air and Radiation, Pesticides and Toxic Substances,
Water, Solid Waste and Emergency Response, and Research and
Development.
PEER REVIEW POLICY
IMPLEMENTATION DELAYED
------------------------------------------------------------ Letter :3
The EPA Administrator wanted to issue a uniform policy that included
implementation procedures. However, continuing concerns by various
EPA office managers over the diversity of products (ranging from
basic health research to permits to operate a facility at a specific
site) developed by the offices and the availability of time and
resources to do the increased number of reviews delayed the
implementation of a uniform peer review policy statement.
An internal work group met in April 1993 to begin developing specific
procedures for managers to use in their peer review decisions. The
group was to consider the concerns raised by the various offices and
determine what effect the extra time needed to do peer reviews would
have on EPA's ability to meet statutory and court-ordered deadlines.
EPA officials said that the issue of meeting deadlines was raised
because EPA has had difficulty meeting deadlines even without the
requirement to conduct peer reviews supporting scientific and
technical products.
The availability of resources and independent reviewers to perform
the increased number of reviews is another issue that EPA needs to
consider in developing a peer review implementation plan. EPA
officials said that doing more reviews will increase costs as the
agency usually pays external peer reviewers between $150 and $300 a
day. Some work group members were concerned that EPA would not have
the funds for the increased costs, particularly in austere budget
periods. Also, some EPA officials were concerned that selecting peer
reviewers and handling their comments might take additional staff.
Furthermore, the availability of independent reviewers was a concern
because some products address highly technical subjects for which
there are very few experts available. Some of these experts may not
be independent since they may have done work on products for
companies that could be affected by decisions EPA makes on the basis
of these products.
The broad range of EPA products is another issue that needs to be
considered. Some work group members believe that it is difficult to
develop one set of procedures that will satisfy the many different
types of products developed by EPA's various offices. For example,
the Office of Research and Development (ORD) usually develops more
traditional research products that may be used by many different
researchers outside of EPA as well as within the agency. However,
the Office of Air Quality Planning and Standards (OAQPS), within the
Office of Air and Radiation, develops technical products that may be
designed to meet specific information needs and thus limited in scope
and use by others. In addition, regional offices usually develop
products that are even more limited in scope. For example, regional
offices may develop data and analyses to allow a specific facility an
operating permit. These data and analyses are site-specific and
would have limited use and be of concern only to those living near
the facility.
The work group has made some preliminary conclusions regarding peer
review procedures. For example, the group has endorsed a flexible
peer review policy. Under this policy, if a product has national
ramifications (e.g., standards for radon levels in homes) or is
controversial (e.g., a report concerning health effects from
environmental tobacco smoke), it should receive a peer review; if it
is more site-specific (e.g., data to support noncontroversial permit
applications), it may need only a limited peer review or none at all.
The work group has also established four committees to address some
of the offices' concerns. For example, committees have been
established to develop (1) guidelines for peer reviews of regional
offices' work, (2) helpful hints of what to do and what not to do on
program-office- level peer reviews, (3) a mechanism to include peer
review as part of the regulatory negotiation\4 process instead of
after it, and (4) a model procurement instrument to help individual
offices obtain external peer review services.
The work group has also identified other concerns. For example,
flexibility was identified as a very important component of the
implementation plan. However, as of December 1993, the work group
had not addressed how it plans to deal with flexibility issues such
as determining which products should receive peer reviews,
determining the extent of the reviews, and developing a conceptual
framework to prevent flexibility from generating unjustifiable
inconsistency. As of December 1993, EPA officials did not know when
uniform peer review procedures would be available--no deadline has
been set for completing them.
--------------------
\4 Regulatory negotiation brings interested parties, such as
industry, environmental groups, and government agencies, together to
agree on how a regulation should be worded to meet all of their
concerns, thereby expediting the issuance of regulations and avoiding
costly litigation.
EPA NEEDS TO STRENGTHEN ITS
PEER REVIEW POLICY
---------------------------------------------------------- Letter :3.1
Because EPA lacks uniform implementing procedures for its new peer
review policy, some of its offices still do not consistently employ
such reviews on products developed by their own staffs or by
contractors. For example, OAQPS does not require that all of the
products it uses to support decisions receive peer reviews. In
addition, ORD often uses EPA personnel to review products it develops
rather than using independent reviewers, the generally accepted way
to do peer reviews. Specifically, we found that:
OAQPS uses many scientific and technical documents, which may be
developed by various sources, including its own staff, ORD, and
contractors, to support air quality standards and regulations.
However, OAQPS officials said that the office does not require
these documents to undergo peer review. In examining products
developed by one of its contractors, we found that neither the
contractor nor OAQPS had subjected them to peer review. Yet at
least one of these products, the "Economic Impact of Air
Pollutant Emission Standards for New Municipal Waste Combustors"
report, was used to support standards and guidance for municipal
waste combustors. According to the EPA officials involved in
overseeing the contractor, the agency does not require a peer
review of such products.
ORD officials said that before the January 1993 policy statement,
they had a policy that all research and technical documents
undergo peer reviews. However, the objectivity of some reviews
could be challenged. For instance, one product, "Mutagenicity
of Emissions from the Simulated Burning of Scrap Rubber Tires,"
which was developed by ORD's Air and Energy Engineering Research
Laboratory, had been reviewed by two other EPA employees--one
from the branch that developed the product and the other from
OAQPS. According to ORD officials, this report received a peer
review at the same level as most other ORD products. The
officials said that EPA employees may be used when not enough
independent reviewers are available. However, according to
officials from the Robert Wood Johnson Foundation and the
National Institute of Environmental Health Sciences, such a
review by EPA's staff is not credible since the review was not
performed by independent parties.
Another weakness in EPA's peer review process is that authors of some
scientific and technical documents help select reviewers. For
example, in some cases, the Atmospheric Research and Exposure
Assessment Laboratory uses a contractor to recommend individuals who
are qualified to review products. An EPA project officer said that a
list of potential reviewers is given to the product's author, who can
eliminate any reviewer he or she considers unsuitable. The revised
list is then returned to the contractor, who makes the final
selection.
CONCERNS OF EPA SCIENCE
ADVISOR OVER THE PEER REVIEW
PROCESS
---------------------------------------------------------- Letter :3.2
The expert panel also recommended that the EPA Administrator appoint
a science advisor to ensure that credible scientific information for
the agency's guidance and decisions is available from both EPA
scientists and the scientific community. In November 1992, a
position was created and filled by an experienced scientific
administrator from outside EPA. The EPA Science Advisor is the focal
point for science in the agency as well as for the work groups
developing peer review implementation procedures. In discussing the
delays and difficulties the agency is having in implementing the
January peer review policy, the Science Advisor had the following
concerns:
The need to find a way to ensure that the peer review policy is
consistently applied throughout EPA's various program and
regional offices and laboratories. The Science Advisor
identified, for example, 14 different ways unit managers could
obtain a peer review, ranging from one by an independent expert
within EPA to an extensive review by the National Academy of
Sciences. The Advisor said that without criteria, unit managers
would inevitably differ to too great an extent when considering
the complexity and importance of products and the type and
degree of peer review a product would receive.
The need for a mechanism to ensure accountability. To help devise
such a mechanism, the Science Advisor has asked a work group in
the Office of Program Planning and Evaluation to consider this
issue as part of its efforts to evaluate EPA's entire regulatory
development process. The Advisor said that members of this
group are aware that EPA's structure is inefficient and
sometimes incapable of dealing with issues, such as peer
reviews, that cut across division lines. This group's work,
which started in June 1993, is expected to impact the peer
review implementation procedures--the group is trying to
identify a way to hold managers accountable for ensuring that
their products meet all the applicable requirements, including
the peer review policy. However, the group has not yet
identified an acceptable accountability mechanism. According to
the Science Advisor, possibly the only means to ensure
accountability under EPA's current structure would be for the
EPA Administrator to require evidence of peer review before
products go to printing. Inserting accountability into the
implementation procedures may further delay the release of
products. As with the work group addressing overall peer
review, the group addressing peer review accountability has just
begun its efforts and has not set a date for recommending a
solution.
Effectively communicating the policy throughout EPA so that
managers know that all products should be considered for peer
review. The Science Advisor said that while the January policy
was not specific as to which products to review,
contractor-provided products are also subject to the policy.
The Advisor was concerned when he learned that some of the
project managers we interviewed were either not aware of the
policy or did not believe it applied to contractor-provided
products. He further explained that without some method to hold
managers accountable for doing peer reviews that some may ignore
the policy.
Confusion between peer involvement and peer review. The Science
Advisor said that some offices within EPA routinely use
employees from the same office as the author to conduct peer
reviews. He explained, however, that this is not peer review
but is more closely associated with peer involvement. He
believes that for EPA to avoid later criticism, project managers
should use independent reviewers from outside their own office
to conduct peer reviews. The Advisor said that while peer
involvement is a valuable tool, it is no substitute for peer
review.
CONTROLS NEEDED OVER EPA
PRODUCTS THAT ARE INDEPENDENTLY
REVIEWED
------------------------------------------------------------ Letter :4
EPA does not have procedures to ensure that products sent to external
individuals for peer review are clearly identified as draft products
and are not prematurely released. As we previously reported, a draft
report on environmental tobacco smoke was prematurely released by a
reviewer. In that instance, the reviewer released a copy of the
technical compendium to the draft report. This document had not
received a final review by EPA managers to determine if its contents
met agency policy. Also, the document had not been marked as a draft
to ensure that anyone receiving it would know that it might not
represent agency policy. The compendium, which later proved to be
controversial, also contained a chapter that was written by the
reviewer. As of this date, EPA has not developed procedures to
preclude the premature release of documents.
EPA products are sometimes sent to external individuals who perform
the reviews for free. Some of the reviewers are not under
contractual or other written agreement to refrain from releasing EPA
products without prior approval. In our February 1993 report, we
discussed the premature release of documents by a peer reviewer that
was doing the review without financial compensation and thus had no
contract with EPA. Also, EPA did not have any other written
agreement with the reviewer that he would not release the document
without EPA's approval. The technical compendium was not stamped
"Draft," nor did the cover letter accompanying it contain a
disclaimer that would have alerted the reader that the document was a
draft. The Director of EPA's Indoor Air Division said that it would
have been good practice for the cover letter to have included a
disclaimer stating that the document was not a final EPA product.
While no assurance exists that a disclaimer would discourage the
premature release of draft documents, EPA would, at least, have a
sound basis for clarifying that such documents are subject to
revision and do not represent the agency's position. However, none
of the work groups have been tasked to address this issue.
CONCLUSIONS
------------------------------------------------------------ Letter :5
By developing an agencywide policy requiring that its scientific and
technical products undergo peer reviews, EPA moved toward ensuring
that its decisions and regulations are based on objective, credible
information. However, because this policy is very general, relying
on each internal unit to tailor its peer reviews, and because EPA has
not yet adopted implementation procedures, peer reviews remain
inconsistent and not fully effective. In addition, controls over
materials undergoing peer reviews by outside individuals remain weak
as EPA does not obtain written agreement from all reviewers that they
will not release documents without EPA's approval. Moreover,
documents are not always marked as drafts and do not always have an
accompanying disclaimer stating that they do not necessarily
represent agency policy.
EPA has established a work group to address the implementation
problems, but progress has been slow, and the agency has not
established a deadline for completing this work. EPA's difficulties
in developing implementation procedures to accompany the peer review
policy demonstrate the importance of establishing deadlines for the
work group. Also, it is not clear when EPA will address such issues
as how to hold managers at all levels accountable for their peer
review decisions, how to ensure that reviewers' independence is not
unduly influenced by the fact that products' authors have the
authority to eliminate unwanted reviewers, and how to ensure that
products developed by contractors receive appropriate peer reviews.
Until such implementation procedures are established, the universal
peer review policy is not likely to have much effect.
RECOMMENDATIONS
------------------------------------------------------------ Letter :6
In order for the agency's peer review policy to be successful, we
recommend that the EPA Administrator set a schedule for developing,
completing, and implementing agencywide peer review procedures.
Also, the Administrator should develop and implement controls that
protect against the premature release of documents by external peer
reviewers.
AGENCY COMMENTS
------------------------------------------------------------ Letter :7
We discussed the information contained in this report with EPA
officials, including the Science Advisor and the Director of the
Office of Technology Transfer and Regulatory Support, who generally
agreed with its facts. Their comments are included where
appropriate. However, at your request, we did not obtain written
agency comments on a draft of this report.
SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :8
We conducted our review from January through December 1993 in
accordance with generally accepted government auditing standards.
Our work focused primarily on examining the peer review of research
products for EPA's air and radiation programs and obtaining peer
review information from EPA Regions I and IV to get an understanding
of concerns EPA regional offices may be having with implementing the
policy. We interviewed officials and obtained documents from EPA's
Offices of Air and Radiation, and Research and Development and the
agency's laboratories at the Research Triangle Park in North
Carolina. We also interviewed two of the four members of the panel
that had reviewed EPA's previous peer review policy and officials at
the National Institute of Environmental Health Sciences in Durham,
North Carolina; the Department of Energy's Argonne National
Laboratory in Argonne, Illinois; the Research Triangle Institute at
the Research Triangle Park; and the Robert Wood Johnson Foundation in
Princeton, New Jersey.
---------------------------------------------------------- Letter :8.1
Unless you publicly announce its contents earlier, we plan no further
distribution of this report until 30 days after the date of this
letter. At that time, we will send copies to the EPA Administrator
and make copies
available to others upon request. This report was prepared under the
direction of Peter F. Guerrero, Director, Environmental Protection
Issues, who can be reached at (202) 275-6111 if you have any
questions. Major contributors to this report are listed in appendix
I.
Sincerely yours,
Keith O. Fultz
Assistant Comptroller General
MAJOR CONTRIBUTORS TO THIS REPORT
=========================================================== Appendix I
RESOURCES, COMMUNITY, AND
ECONOMIC DEVELOPMENT DIVISION,
WASHINGTON, D.C.
--------------------------------------------------------- Appendix I:1
William F. McGee, Assistant Director
NORFOLK REGIONAL OFFICE
--------------------------------------------------------- Appendix I:2
Philip Bartholomew, Evaluator
James Hayward, Evaluator