Peer Review: EPA's Implementation Remains Uneven (Testimony, 03/11/97,
GAO/T-RCED-97-95).
GAO discussed its recent report on the Environmental Protection Agency's
(EPA) implementation of its peer review policy, focusing on EPA's: (1)
progress in implementing its peer review policy; and (2) efforts to
improve the peer review process.
GAO noted that: (1) despite some recent progress, peer review continues
to be implemented unevenly; (2) although GAO found some cases in which
EPA's peer review policy was properly followed, it also found cases in
which key aspects of the policy were not followed or in which peer
review was not conducted at all; (3) GAO believes that two of the
primary reasons for this uneven implementation are: (a) inadequate
accountability and oversight to ensure that all relevant products are
properly peer reviewed; and (b) confusion among EPA's staff and
management about what peer review is, its importance and benefits, and
how and when it should be conducted; (4) EPA officials readily
acknowledge this uneven implementation and, during the course of GAO's
work, had a number of efforts under way to improve the peer review
process; (5) although GAO found these efforts to be steps in the right
direction, it concluded that EPA was not addressing the underlying
problems that GAO had identified; (6) accordingly, GAO recommended that
EPA ensure that: (a) upper-level managers have the information they need
to know whether or not all relevant products have been considered for
peer review; and (b) staff and managers are educated about the need for
and benefits of peer review and their specific responsibilities in
implementing policy; (7) EPA agreed with GAO's recommendations and has
several efforts under way to implement them; (8) for example, EPA plans
to initiate a peer review training program for its managers and staff in
June 1997; and (9) while it is still too early to be certain if these
efforts will be fully successful, GAO is encouraged by the high-level
attention being paid to this very important process.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: T-RCED-97-95
TITLE: Peer Review: EPA's Implementation Remains Uneven
DATE: 03/11/97
SUBJECT: Environmental research
Evaluation methods
Internal controls
Research reports
Policy evaluation
Environmental policies
Human resources training
Quality assurance
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Before the House Subcommittee on Energy and Environment, Committee on
Science, House of Representatives
For Release
on Delivery
Expected at
1 p.m. EST
Tuesday
March 11, 1997
PEER REVIEW - EPA'S IMPLEMENTATION
REMAINS UNEVEN
Statement by Stanley J. Czerwinski, Associate Director,
Environmental Protection Issues,
Resources, Community, and Economic
Development Division
GAO/T-RCED-97-95
GAO/RCED-97-95T
(160384)
Abbreviations
=============================================================== ABBREV
EPA -
GAO -
ORD -
SAB -
OMS -
============================================================ Chapter 0
Mr. Chairman and Members of the Subcommittee:
We are pleased to be here today to discuss our recent report on the
Environmental Protection Agency's (EPA) implementation of its peer
review policy.\1 As you know, peer review is the critical evaluation
of scientific and technical work products by independent experts to
enhance the products' quality, credibility, and acceptability.
Furthermore, peer review can actually help avoid costly and
time-consuming delays by helping to steer product development along
the most efficient, effective course. The EPA's current peer review
policy--updated in 1994 and currently under evaluation by the agency
in the light of our report's findings--stresses the importance of
such reviews and calls for the peer review of all major scientific
and technical work products that may eventually play an important
role in key agency decisions. In the light of the critical role that
peer review plays in supporting the agency's important decisions, we
assessed EPA's (1) progress in implementing its peer review policy
and (2) efforts to improve the peer review process.
In summary, we found that:
-- Despite some recent progress, peer review continues to be
implemented unevenly. Although we found some cases in which
EPA's peer review policy was properly followed, we also found
cases in which key aspects of the policy were not followed or in
which peer review was not conducted at all. We believe that two
of the primary reasons for this uneven implementation are (1)
inadequate accountability and oversight to ensure that all
relevant products are properly peer reviewed and (2) confusion
among EPA's staff and management about what peer review is, its
importance and benefits, and how and when it should be
conducted.
-- EPA officials readily acknowledge this uneven implementation
and, during the course of our work, had a number of efforts
under way to improve the peer review process. Although we found
these efforts to be steps in the right direction, we concluded
that they were not addressing the underlying problems that we
had identified. Accordingly, we recommended that EPA ensure
that (1) upper-level managers have the information they need to
know whether or not all relevant products have been considered
for peer review and (2) staff and managers are educated about
the need for and benefits of peer review and their specific
responsibilities in implementing the policy. EPA agreed with
our recommendations and has several efforts under way to
implement them. For example, EPA plans to initiate a peer
review training program for its managers and staff in June 1997.
While it is still too early to be certain if these efforts will
be fully successful, we are encouraged by the high-level
attention being paid to this very important process.
--------------------
\1 Peer Review: EPA's Implementation Remains Uneven
(GAO/RCED-96-236, Sept. 24, 1996).
BACKGROUND
---------------------------------------------------------- Chapter 0:1
Peer review is well established as a mechanism for assuring the
quality, credibility, and acceptability of individual and
institutional work products. This assurance is accomplished by
having the products undergo an objective, critical review by
independent reviewers. Peer review has long been used by academia,
professional organizations, industry, and government. Within EPA,
peer review has taken many different forms, depending upon the nature
of the work product, the relevant statutory requirements, and
office-specific practices and needs.
In keeping with scientific custom and/or congressional mandates,
several offices within EPA have used peer review for many years to
enhance the quality of science within the agency. In response to a
panel of outside academicians' recommendations in 1992,\2 EPA issued
a policy statement in 1993 calling for peer review of the major
scientific and technical work products used to support the agency's
rulemaking and other decisions. However, the Congress, GAO,\3 and
others subsequently raised concerns that the policy was not being
implemented consistently across the agency. In response to these
concerns, in 1994 EPA reaffirmed the central role that peer review
plays in ensuring that the agency's decisions are based on sound
science and credible data and revised its 1993 policy.
The new policy, while retaining the essence of the prior one, was
intended to expand and improve the use of peer review throughout EPA.
The 1994 policy continued to stress that major products should
normally be peer reviewed, but it also recognized that statutory and
court-ordered deadlines, resource limitations, and other constraints
might limit or even preclude the use of peer review. The policy
applied to major work products that are primarily scientific or
technical in nature and that may contribute to the basis for policy
or regulatory decisions. In contrast, other products used in
decision-making are not covered by the policy, nor are the ultimate
decisions themselves. While peer review can take place at several
different points along a product's development, such as during the
planning stage, it should be applied to a relatively well-developed
product.
The 1994 policy also clarified that peer review is not the same thing
as the peer input, stakeholders' involvement, or public
comment--mechanisms used by EPA to develop products, to obtain the
views of interested and affected parties, and/or to build consensus
among the regulated community. While each of these mechanisms serves
a useful purpose, the policy points out that they are not a
substitute for peer review because they do not necessarily solicit
the same unbiased, expert views that are obtained through peer
review.
EPA's policy assigned responsibility to each Assistant and Regional
Administrator to develop standard operating procedures and to ensure
their use. To help facilitate consistent EPA-wide implementation,
EPA's Science Policy Council--chaired by EPA's Deputy
Administrator--was directed to help the offices and regions develop
their procedures and identify products that should be peer reviewed.
The Council was also given the responsibility for assessing
agencywide progress and developing any needed changes to the policy.
However, the ultimate responsibility for implementing the policy was
placed with the Assistant and Regional Administrators.
--------------------
\2 Safeguarding the Future: Credible Science, Credible Decisions
(EPA/600/9-91/050, Mar. 1992).
\3 Peer Review: EPA Needs Implementation Procedures and Additional
Controls (GAO/RCED-94-89, Feb. 22, 1994).
IMPLEMENTATION REMAINS UNEVEN
---------------------------------------------------------- Chapter 0:2
We found that--2 years after EPA established its peer review policy--
implementation was still uneven. We concluded that EPA's uneven
implementation was primarily due to (1) inadequate accountability and
oversight to ensure that all products are properly peer reviewed by
program and regional offices and (2) confusion among agency staff and
management about what peer review is, what its significance and
benefits are, and when and how it should be conducted.
According to the Executive Director of the Science Policy Council,
the unevenness could be attributed to a number of factors. First,
while some offices within EPA--such as the Office of Research and
Development (ORD)--have historically used peer review for many years,
other program offices and regions have had little prior experience.
In addition, the Director and other EPA officials told us that
statutory and court-ordered deadlines, budget constraints, and
problems in finding and obtaining qualified, independent peer
reviewers also contributed to the problem.
INADEQUATE OVERSIGHT TO
ENSURE CONSISTENT
IMPLEMENTATION
-------------------------------------------------------- Chapter 0:2.1
EPA's oversight primarily consisted of a two-part reporting scheme
that called for each office and region to annually list (1) the
candidate products nominated for peer review during the upcoming year
and (2) the status of the products previously nominated. If a
candidate product was no longer scheduled for peer review, the list
had to note this and explain why peer review was no longer planned.
Although we found this to be an adequate oversight tool for tracking
the status of previously nominated products, we pointed out that it
does not provide upper-level managers with sufficient information to
ensure that all products warranting peer review have been identified.
This fact, together with the misperceptions about what peer review is
and the deadlines and budget constraints that project officers often
operate under, has meant that the peer review program to date has
largely been one of self-identification, allowing some important work
products to go unlisted. According to the Science Policy Council,
reviewing officials would be much better positioned to determine if
the peer review policy and procedures are being properly and
consistently implemented if, instead, EPA's list contained all major
products along with what peer review is planned and, if none, the
reasons why not.
We noted that the need for more comprehensive oversight is especially
important given the policy's wide latitude in allowing peer review to
be forgone in cases facing time and/or resource constraints. As
explained by the Executive Director of EPA's Science Policy Council,
because so much of the work that EPA performs is in response to
either statutory or court-ordered mandates and the agency frequently
faces budget uncertainties or limitations, an office under pressure
might argue for nearly any given product that peer review is a luxury
the office cannot afford in the circumstances.
However, as the Executive Director of the Science Advisory Board
(SAB)\4 told us, not conducting peer review can sometimes be more
costly to the agency in terms of time and resources. He told us of a
recent Office of Solid Waste rulemaking concerning a new methodology
for delisting hazardous wastes in which the Office's failure to have
the methodology appropriately peer reviewed resulted in important
omissions, errors, and flawed approaches in the methodology; these
problems will now take from 1 to 2 years to correct. The SAB also
noted that further peer review of the individual elements of the
proposed methodology is essential before the scientific basis for
this rulemaking can be established.
--------------------
\4 The SAB is a legislatively established body of independent experts
that provides advice to the EPA Administrator on scientific and
engineering issues.
PEER REVIEW POLICY AND
PROCEDURES NOT WELL
UNDERSTOOD
-------------------------------------------------------- Chapter 0:2.2
Although EPA's policy and procedures provide substantial information
about what peer review entails, we found that some EPA staff and
managers had misperceptions about what peer review is, what its
significance and benefits are, and when and how it should be
conducted. Several cases we reviewed illustrate this lack of
understanding about what peer review entails. Officials from EPA's
Office of Mobile Sources (OMS) told the House Commerce Committee in
August 1995 that they had not had any version of the mobile model\5
peer reviewed. Subsequently, in April 1996, OMS officials told us
they recognize that external peer review is needed and that EPA
planned to have the next iteration of the model so reviewed.
We found similar misunderstandings in several other cases we
reviewed. EPA regional officials who produced a technical product
that assessed the environmental impacts of tributyl tin\6 told us
that the contractor-prepared product had been peer reviewed. While
we found that the draft product did receive some internal review by
EPA staff and external review by contributing authors, stakeholders,
and the public, it was not reviewed by experts independent of the
product itself or of its potential regulatory ramifications. When we
pointed out that--according to EPA's policy and the region's own peer
review procedures--these reviews are not a substitute for peer
review, the project director said that she was not aware of these
requirements.
In two other cases we reviewed, there were misunderstandings about
the components of a product that should be peer reviewed. For
example, in the Great Waters study--an assessment of the impact of
atmospheric pollutants in significant water bodies--the scientific
data were subjected to external peer review, but the study's
conclusions that were based on these data were not. Similarly, in
the reassessment of dioxin--an examination of the health risks posed
by dioxin--the final chapter summarizing and characterizing dioxin's
risks was not as thoroughly peer reviewed.\7 In both cases, the
project officers did not have the conclusions peer reviewed because
they believed that the development of conclusions is an inherently
governmental function that should be performed exclusively by EPA
staff. However, some EPA officials with expertise in conducting peer
reviews disagreed, maintaining that it is important to have peer
reviewers comment on whether or not EPA has properly interpreted the
results of the underlying scientific and technical data. EPA's
quality assurance requirements also state that conclusions should be
peer reviewed.\8
--------------------
\5 The mobile model is one of the primary tools used by EPA, states,
and localities to calculate the estimated emissions reduction
benefits of pollution control activities called for in state
implementation plans.
\6 Tributyl tin is a compound that has been used since the 1960s in
antifouling paints for boats and large ships.
\7 Although the entire product was reviewed by EPA's Science Advisory
Board, the Board expressed dissatisfaction that the risk
characterization chapter did not receive prior peer review.
\8 EPA Requirements for Quality Management Plans (EPA QA/R-2, Aug.
1994). This document establishes the criteria and mandatory
specifications for quality assurance and quality control activities.
EPA'S ACTIONS TO IMPROVE THE
PEER REVIEW PROCESS
---------------------------------------------------------- Chapter 0:3
During our review, we found that EPA had recently taken a number of
steps to improve the peer review process. Although we believed that
these steps should prove helpful, we concluded that they did not
fully address the previously-discussed underlying problems and made
some recommendations for improvement. EPA agreed with our findings
and recommendations and has recently undertaken steps to implement
them. While it is too early to gauge the effectiveness of these
efforts, we are encouraged by the attention peer review is receiving
by the agency's upper-level management.
EPA'S PAST EFFORTS
-------------------------------------------------------- Chapter 0:3.1
Near the completion of our review, in June 1996, EPA's Deputy
Administrator directed the Science Policy Council's Peer Review
Advisory Group and ORD's National Center for Environmental Research
and Quality Assurance to develop an annual peer review
self-assessment and verification process to be conducted by each
office and region. The self-assessment was to include information on
each peer review completed during the prior year as well as feedback
on the effectiveness of the overall process. The verification would
consist of the signature of headquarters, laboratory, or regional
directors to certify that the peer reviews were conducted in
accordance with the agency's policy and procedures. If the peer
review did not fully conform to the policy, the division director or
the line manager must explain significant variances and actions
needed to limit future significant departures from the policy. The
self-assessments and verifications were to be submitted and reviewed
by the Peer Review Advisory Group to aid in its oversight
responsibilities. According to the Deputy Administrator, this
expanded assessment and verification process would help build
accountability and demonstrate EPA's commitment to the independent
review of the scientific analyses underlying the agency's decisions
to protect public health and the environment.
During our review, we also found a number of efforts under way within
individual offices and regions to improve their implementation of
peer review. For example, the Office of Water drafted additional
guidance to further clarify the need for, use of, and ways to conduct
peer review. The Office of Solid Waste and Emergency Response formed
a team to help strengthen the office's implementation of peer review
by identifying ways to facilitate good peer review and addressing
barriers to its successful use. Additionally, EPA's Region 10 formed
a Peer Review Group with the responsibility for overseeing the
region's reviews.
We concluded that the above efforts should help address the problems
we found. However, we also concluded that the efforts aimed at
improving the oversight of peer review fell short by not ensuring
that all relevant products had been considered for peer review and
did not require documenting the reasons why products were not
selected. Similarly, we noted that the efforts aimed at better
informing staff about the benefits and use of peer review would be
more effective if they were done consistently throughout the agency.
EPA'S CURRENT PLANS
-------------------------------------------------------- Chapter 0:3.2
EPA agreed with our findings and conclusions and has recently
undertaken a number of steps to implement our recommendations. On
November 5, 1996, the Deputy Administrator asked ORD's Assistant
Administrator, in consultation with the other Assistant
Administrators, to develop proposals to strengthen the peer review
process. In response, ORD's Assistant Administrator proposed a
three-pronged approach consisting of (1) audits of a select number of
work products to determine how well the peer review policy was
followed; (2) a series of interviews with office and regional staff
involved with peer review to determine the processes used to
implement the policy; and (3) training to educate and provide help to
individuals to improve the implementation of the peer review policy.
Significantly, the Deputy Administrator has echoed our message that
EPA needs to improve its oversight to ensure that all appropriate
products are peer reviewed. In a January 14, 1997, memorandum to the
Assistant and Regional Administrators, the Deputy stated, "I want you
to ensure that your lists of candidates for peer review are
complete." To help accomplish this goal, each organization is
directed to use, among other things, EPA's regulatory agenda and
budget planning documents to help identify potential candidates for
peer review.
While we agree that this should prove to be a useful tool, we
continue to encourage EPA to expand its existing candidate list to
include all major work products, along with explanations of why
individual products are not nominated for peer review. An
all-inclusive list such as this will be extremely useful to those
overseeing the peer review process to determine whether or not all
products have been appropriately considered for peer review.
-------------------------------------------------------- Chapter 0:3.3
In summary, peer review is critical for improving the quality of
scientific and technical products and for enhancing the credibility
and acceptability of EPA's decisions that are based on these
products. We are encouraged by the renewed attention EPA is giving
to improving the peer review process. Although it is too early for
us to gauge the success of these efforts, the involvement of the
agency's upper-level management should go a long way to ensure that
the problems we identified are resolved. Mr. Chairman, this
concludes my prepared statement. I will be happy to respond to your
questions or the questions of Subcommittee members.
*** End of document. ***