Peer Review: EPA's Implementation Remains Uneven (Letter Report,
09/24/96, GAO/RCED-96-236).

Peer review is the critical evaluation of scientific and technical work
products by independent experts. Senior officials at the Environmental
Protection Agency (EPA) have said that peer review is an important
mechanism for enhancing the quality, credibility, and acceptability of
products that may ultimately form the basis of regulations and other key
agency decisions. Properly implemented, peer review can also save money
by steering product development along the most efficient, effective
course, thereby avoiding costly and time-consuming delays. EPA's current
policy, issued in June 1995, expands the agency's prior policies and
practices and continues to stress that major scientific and technical
work products should be peer reviewed. This report assesses EPA's (1)
progress in implementing its peer review policy and (2) efforts to
improve the peer review process.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  RCED-96-236
     TITLE:  Peer Review: EPA's Implementation Remains Uneven
      DATE:  09/24/96
   SUBJECT:  Accountability
             Policy evaluation
             Internal controls
             Research and development
             Environmental policies
             Scientific research
             Research reports

             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Report to Congressional Requesters

September 1996

PEER REVIEW - EPA'S IMPLEMENTATION
REMAINS UNEVEN

GAO/RCED-96-236

Peer Review at EPA

(160339)


Abbreviations
=============================================================== ABBREV

  AJ - Alaska Juneau (Gold Mine)
  EPA - Environmental Protection Agency
  GAO - General Accounting Office
  OAR - Office of Air and Radiation
  OMS - Office of Mobile Sources
  OPPE - Office of Policy, Planning and Evaluation
  OPPTS - Office of Prevention, Pesticides, and Toxic Substances
  ORD - Office of Research and Development
  OSW - Office of Solid Waste
  OSWER - Office of Solid Waste and Emergency Response
  RCRA - Resource Conservation and Recovery Act
  SAB - Science Advisory Board
  SPC - Science Policy Council
  TBT - tributyl tin
  WTI - Waste Technologies Industries

Letter
=============================================================== LETTER


B-272925

September 24, 1996

The Honorable Christopher S.  Bond
Chairman, Committee on Small Business
United States Senate

The Honorable Lauch Faircloth
Chairman, Subcommittee on Clean Air, Wetlands,
 Private Property, and Nuclear Safety
Committee on Environment and Public Works
United States Senate

The Honorable Don Nickles
Chairman, Subcommittee on Energy Production
 and Regulation
Committee on Energy and Natural Resources
United States Senate

Peer review is the critical evaluation of scientific and technical
work products by independent experts.  Within the Environmental
Protection Agency (EPA), the agency's senior leadership has stated
that peer review is an important mechanism for enhancing the quality,
credibility, and acceptability of products that may ultimately form
the basis of regulations and other key decisions by the agency. 
Properly implemented, peer review can also conserve resources by
steering product development along the most efficient, effective
course, thereby avoiding costly and time-consuming delays.  EPA's
current policy, issued in June 1994, expands on the agency's prior
policy and practices.  The new policy continues to emphasize that
major scientific and technical work products should normally be peer
reviewed.  In light of the key role that peer review plays in
developing regulations, you asked us to assess EPA's (1) progress in
implementing its peer review policy and (2) efforts to improve the
peer review process. 


   RESULTS IN BRIEF
------------------------------------------------------------ Letter :1

Although EPA has made progress in implementing its peer review
policy, after nearly 2 years, implementation remains uneven.  While
we found cases in which the peer review policy was followed, we also
found cases in which important aspects of the policy were not
followed or peer review was not conducted at all.  Two primary
reasons for this unevenness are (1) confusion among agency staff and
management about what peer review is, what its significance and
benefits are, and how and when it should be conducted and (2)
inadequate accountability and oversight mechanisms to ensure that all
relevant products are properly peer reviewed.  For example, some
agency officials told us that the public comments obtained through
the rulemaking process would suffice for peer review of their work
products, although EPA's peer review procedures state that these are
not substitutes for peer review.  In another instance, a model of
EPA's that had significant ramifications for many states' air
pollution control programs did not receive peer review. 

EPA officials readily acknowledged this uneven implementation and
identified several of the agency's recent efforts to improve the peer
review process.  For example, since April 1996, two offices have
provided additional guidance for their offices to help reduce
confusion about what peer review entails.  This guidance clarifies
the need for, use of, and ways to conduct peer review.  Also, because
of concern about the effectiveness of the existing accountability and
oversight mechanisms for ensuring proper peer review, EPA's Deputy
Administrator recently established procedures intended to help build
accountability and demonstrate EPA's commitment to the independent
review of the scientific analyses underlying the agency's decisions. 
These efforts are steps in the right direction.  However, educating
all staff about the merits of and procedures for conducting peer
review would increase the likelihood that peer review is properly
implemented agencywide.  Furthermore, by ensuring that all relevant
products have been considered for peer review and that the reasons
for those not selected have been documented, EPA's upper-level
managers will have the necessary information to ensure that the
policy is properly implemented. 


   BACKGROUND
------------------------------------------------------------ Letter :2

In accordance with scientific custom and/or statutory mandates,
several offices within EPA have used peer review for many years to
enhance the quality of science within the agency.  In May 1991, the
EPA Administrator established a panel of outside academicians to,
among other things, enhance the stature of science at EPA and
determine how the agency can best ensure that sound science is the
foundation for the agency's regulatory and decision-making processes. 
In March 1992, the expert panel\1 recommended that, among other
things, EPA establish a uniform peer review process for all
scientific and technical products used to support EPA's guidance and
regulations.  In response, EPA issued a policy statement in January
1993 calling for peer review of the major scientific and technical
work products used to support the agency's rulemaking and other
decisions.  However, the Congress, GAO,\2 and others subsequently
raised concerns that the policy was not being consistently
implemented throughout EPA.  The congressional concern resulted in
several proposed pieces of legislation that included prescriptive
requirements for peer reviews. 

Subsequently, in June 1994 the EPA Administrator reaffirmed the
central role of peer review in the agency's efforts to ensure that
its decisions rest on sound science and credible data by directing
that the agency's 1993 peer review policy be revised.  The new policy
retained the essence of the prior policy and was intended to expand
and improve the use of peer review throughout EPA.  Although the
policy continued to emphasize that major scientific and technical
products should normally be peer reviewed, it also recognized that
statutory and court-ordered deadlines, resource constraints, and
other constraints may limit or preclude the use of peer review. 
According to the Executive Director of the Science Policy Council,
one of the most significant new features of the 1994 action was the
Administrator's directive to the agency's Science Policy Council to
organize and guide an agencywide program for implementing the policy. 

The policy and procedures emphasize that peer review is not the same
thing as other mechanisms that EPA often uses to obtain the views of
interested and affected parties and/or to build consensus among the
regulated community.  More specifically, EPA's policy and procedures
state that peer review is not

  -- peer input, which is advice or assistance from experts during
     the development of a product;

  -- stakeholders' involvement, which is comments from those people
     or organizations (stakeholders) that have significant financial,
     political, or other interests in the outcome of a rulemaking or
     other decision by EPA; or

  -- public comment, which is comments obtained from the general
     public on a proposed rulemaking and may or may not include the
     comments of independent experts. 

While each of these activities serves a useful purpose, the policy
and procedures point out that they are not a substitute for peer
review.  For example, as noted in EPA's Standard Operating
Procedures, public comments on a rulemaking do not necessarily
solicit the same unbiased, expert views as are obtained through peer
review. 

In order to accommodate the differences in EPA's program and regional
offices, the policy assigned responsibility to each program and
regional office to develop standard operating procedures and to
ensure their use.  To help facilitate agencywide implementation,
EPA's Science Policy Council was assigned the responsibility of
assisting the offices and regions in developing their procedures and
identifying products that should be considered for peer review.  The
Council was also given the responsibility for overseeing the
agencywide implementation of the policy by promoting consistent
interpretation, assessing agencywide progress, and developing
revisions to the policy, if warranted.  However, EPA's policy
specifies that the Assistant and Regional Administrators for each
office are ultimately responsible for implementing the policy,
including developing operating procedures, identifying work products
subject to peer review, determining the type and timing of such
reviews, and documenting the process and outcome of each peer review
conducted. 

Our objectives, scope, and methodology are fully described in
appendix I. 


--------------------
\1 Safeguarding the Future:  Credible Science, Credible Decisions
(EPA/600/9-91/050, Mar.  1992). 

\2 Peer Review:  EPA Needs Implementation Procedures and Additional
Controls (GAO/RCED-94-89, Feb.  22, 1994); and Environmental
Protection:  EPA's Problems With Collection and Management of
Scientific Data and Its Efforts to Address Them (GAO/T-RCED-95-174,
May 12, 1995), testimony for the Senate Committee on Appropriations. 


   IMPLEMENTATION REMAINS UNEVEN
------------------------------------------------------------ Letter :3

Two years after EPA established its peer review policy,
implementation is still uneven.  EPA acknowledges this problem and
provided us with a number of examples to illustrate the uneven
implementation.  At our request, the Science Policy Council obtained
information from EPA program and regional offices and provided us
with examples in which, in their opinion, peer review was properly
conducted; cases in which it was conducted but not fully in
accordance with the policy; and cases in which peer review was not
conducted at all.  The following table briefly summarizes the cases
they selected; additional information on these nine cases is provided
in appendix II. 



                                         Table 1
                         
                              Examples of Uneven Peer Review

                                                                                 Peer
                                                            Peer       Policy    review
                                                            review     not       not
Scientific and technical                                    policy     fully     conducte
work products\a               Brief description of product  followed   followed  d
----------------------------  ----------------------------  ---------  --------  --------
Eastern Columbia Plateau      Support document prepared by  X
Aquifer System                EPA to designate the plateau
                              area as a sole-source
                              aquifer

WTI (Waste Technologies       Document assessing human      X
Industry) Incinerator         health and ecological risks
                              associated with operations
                              of a commercial incineration
                              facility

Dioxin Resssessment           Reexamination of the health              X
                              risks associated with this
                              chemical by-product

Great Waters Study\b          Study of atmospheric                     X
                              deposition of pollutants to
                              the Great Lakes, Lake
                              Champlain, Chesapeake Bay,
                              and coastal waters

Unit Pricing\c                Literature assessment of                 X
                              unit pricing programs as a
                              pollution prevention
                              mechanism for residential
                              municipal solid waste
                              generation

AJ Mine Technical Assistance  Technical report assessing               X
Report                        the impact of a proposed
                              disposal method for a gold
                              mine's tailings

Methodology for Establishing  Proposed rulemaking to                   X
Hazardous Waste Exit          establish a new methodology
Criteria\b                    for delisting hazardous
                              wastes

Mobile 5A Emissions Model     Key tool used by EPA,                              X
                              states, and local areas to
                              calculate estimated emission
                              reduction benefits of
                              various pollution control
                              activities

Tributyl Tin Impacts          Assessment of the impacts of                       X
                              an ingredient in antifouling
                              paints used on boats and
                              ships
-----------------------------------------------------------------------------------------
\a These are working titles generally used by EPA; complete titles
can be found in appendix II. 

\b Although this work product was originally identified as one that
fully met EPA's peer review policy, after our review and discussion
the Science Policy Council's Executive Director agreed that the peer
review of this product did not fully meet the policy. 

\c This work product was referred to us by the EPA Science Advisory
Board subsequent to the Science Policy Council's identification of
work products. 

According to the Executive Director of the Science Policy Council,
this unevenness can be attributed to several factors.  First, some
offices within EPA have historically used peer review, while others'
experience is limited to the 2 years since the policy was issued. 
For example, in accordance with scientific custom, the Office of
Research and Development (ORD) has used peer review for obtaining
critical evaluations of certain work products for more than 20 years. 
Additionally, statutes require that certain work products developed
by EPA be peer reviewed by legislatively established bodies.  For
example, criteria documents developed by ORD for the National Ambient
Air Quality Standards must receive peer review from EPA's Science
Advisory Board (SAB), and pesticide documents must receive peer
review from the Scientific Advisory Panel.\3 In contrast, some EPA
regional offices and areas within some EPA program offices have had
little prior experience with peer review.  In addition to these
offices' varying levels of experience with peer review, the Science
Policy Council's Executive Director and other EPA officials said that
statutory and court-ordered deadlines, budget constraints, and
difficulties associated with finding and obtaining the services of
qualified, independent peer reviewers have also contributed to peer
review not being consistently practiced agencywide.  A report by the
National Academy of Public Administration confirmed that EPA
frequently faces court-ordered deadlines.  According to the Academy,
since 1993 the courts have issued an additional 131 deadlines that
EPA must comply with or face judicial sanctions.  Also, as explained
to us by officials from EPA's Office of Air and Radiation (OAR), just
about everything EPA does in some program areas, such as Clean Air
Act implementation, is to address either legislative or court-ordered
mandates. 

Others have attributed EPA's problems with implementing peer review
in the decision-making process to other factors.  For example, in its
March 1995 interim report\4 on EPA's research and peer review program
within the Office of Research and Development, the National Academy
of Sciences' National Research Council noted that, even in EPA's
research community, knowledge about peer review could be improved. 
The Council's interim report pointed out that "although peer review
is widely used and highly regarded, it is poorly understood by many,
and it has come under serious study only in recent years." Although
we agree that the issues EPA and others have raised may warrant
further consideration, we believe that EPA's uneven implementation is
primarily due to (1) confusion among agency staff and management
about what peer review is, what its significance and benefits are,
and when and how it should be conducted and (2) ineffective
accountability and oversight mechanisms to ensure that all products
are properly peer reviewed by program and regional offices. 


--------------------
\3 The SAB is a legislatively established body of independent experts
that provides advice to the EPA Administrator on scientific and
engineering issues.  The Scientific Advisory Panel was legislatively
established under the Federal Insecticide, Fungicide and Rodenticide
Act to perform peer reviews of studies related to pesticides and to
ensure that an adequate scientific basis exists for regulatory
actions related to pesticides. 

\4 Interim Report of the Committee on Research and Peer Review in
EPA, Board on Environmental Studies and Toxicology, Commission on
Geosciences, Environment, and Resources, and Commission on Life
Sciences, National Academy of Sciences, National Research Council
(Mar.  1995).  This study was requested by the Congress in its fiscal
year 1995 appropriations for EPA. 


      PEER REVIEW POLICY AND
      PROCEDURES NOT WELL
      UNDERSTOOD
---------------------------------------------------------- Letter :3.1

Although the policy and procedures provide substantial information
about what peer review entails, we found that some EPA staff and
managers had misperceptions about what peer review is, what its
significance and benefits are, and when and how it should be
conducted.  For example, officials from EPA's Office of Mobile
Sources (OMS) told the House Commerce Committee in August 1995 that
they had not had any version of the mobile model\5 peer reviewed. 
Subsequently, in April 1996, OMS officials told us they recognize
that external peer review is needed and that EPA plans to have the
next iteration of the model peer reviewed.  However, when asked how
the peer review would be conducted, OMS officials said they plan to
use the public comments on the revised model they receive as the peer
review.  As EPA's policy makes clear, public comments are not the
same as nor are they a substitute for peer review. 

We found a similar misunderstanding about what peer review entails in
a regional office we visited.  The region prepared a product that
assesses the impacts of tributyl tin--a compound used since the 1960s
in antifouling paints for boats and large ships.  Although regional
staff told us that this contractor-prepared product had been peer
reviewed, we found that the reviews were not in accordance with EPA's
peer review policy.  The draft product received some internal review
by EPA staff and external review by contributing authors,
stakeholders, and the public; however, it was not reviewed by experts
previously uninvolved with the product's development nor by those
unaffected by its potential regulatory ramifications.  When we
pointed out that--according to EPA's policy and the region's own peer
review procedures--these reviews are not a substitute for peer
review, the project director said that she was not aware of these
requirements. 

In two other cases we reviewed, there was misunderstanding about the
components of a product that should be peer reviewed.  For example,
in the Great Waters study--an assessment of the impact of atmospheric
pollutants in significant water bodies--the scientific data were
subjected to external peer review, but the study's conclusions that
were based on these data were not.  Similarly, in the reassessment of
dioxin--a reexamination of the health risks posed by dioxin--the
final chapter summarizing and characterizing dioxin's risks was not
as thoroughly peer reviewed.\6

More than any other, this chapter indicated EPA's conclusions based
on its reassessment of the dioxin issue.  In both cases, the project
officers did not have these chapters peer reviewed because they
believed that the development of conclusions is an inherently
governmental function that should be performed exclusively by EPA
staff.  However, some EPA officials with expertise in conducting peer
reviews disagreed, maintaining that it is important to have peer
reviewers comment on whether or not EPA has properly interpreted the
results of the underlying scientific and technical data. 


--------------------
\5 The mobile model is one of the primary tools used by EPA, states,
and local areas to calculate the estimated emissions reduction
benefits of the pollution control activities called for in state
implementation plans. 

\6 Although the entire product was reviewed by the SAB, the Board
expressed dissatisfaction that the chapter on risk characterization
did not receive prior peer review.  Appendix II contains additional
information on this case and the other eight we reviewed. 


      INADEQUATE MECHANISMS TO
      ENSURE CONSISTENT
      IMPLEMENTATION
---------------------------------------------------------- Letter :3.2

In addition to the uncertainty surrounding the peer review policy, we
also noted problems with EPA's accountability and oversight
mechanisms.  EPA's current oversight mechanism primarily consists of
a two-part reporting scheme:  Each office and region annually lists
(1) the candidate products nominated for peer review during the
upcoming year and (2) the status of products previously nominated. 
If a candidate product is no longer scheduled for peer review, the
list must note this and explain why peer review is no longer planned. 
Agency officials said this was the most extensive level of oversight
to which all program and regional offices could agree when the peer
review procedures were developed. 

Although this is an adequate oversight mechanism for tracking the
status of previously nominated products, it does not provide
upper-level managers with sufficient information to ensure that all
products warranting peer review have been identified.  This, when
taken together with the misperceptions about what peer review is and
with the deadlines and budget constraints that project officers often
operate under, has meant that the peer review program to date has
largely been one of self-identification, allowing some important work
products to go unlisted.  According to the Science Policy Council's
Executive Director, reviewing officials would be much better
positioned to determine if the peer review policy and procedures are
being properly and consistently implemented if, instead, EPA's list
contained all major products along with what peer review is planned
and, if none, the reasons why not. 

The need for more comprehensive accountability and oversight
mechanisms is especially important given the policy's wide latitude
in allowing peer review to be forgone in cases facing time and/or
resource constraints.  As explained by EPA's Science Policy Council's
Executive Director, because so much of the work that EPA performs is
in response to either statutory or court-ordered mandates and the
agency frequently faces budget uncertainties or limitations, an
office under pressure might argue for nearly any given product that
peer review is a luxury the office cannot afford in the
circumstances. 

However, as the Executive Director of EPA's Science Advisory Board
told us, not conducting peer review can sometimes be more costly to
the agency in terms of time and resources.  He told us of a recent
rulemaking by the Office of Solid Waste concerning a new methodology
for delisting hazardous wastes in which the program office's failure
to have the methodology appropriately peer reviewed resulted in
important omissions, errors, and flawed approaches in the
methodology, which will now take from 1 to 2 years to correct.  The
SAB also noted that further peer review of individual elements of the
proposed methodology is essential before the scientific basis for
this rulemaking can be established. 


   EPA'S ACTIONS TO IMPROVE THE
   PEER REVIEW PROCESS
------------------------------------------------------------ Letter :4

EPA has recently taken a number of steps to improve the peer review
process.  Although these steps should prove helpful, they do not
fully address the underlying problems discussed above. 


      AGENCYWIDE EFFORTS
---------------------------------------------------------- Letter :4.1

In June 1996, EPA's Deputy Administrator directed the Science Policy
Council's Peer Review Advisory Group and ORD's National Center for
Environmental Research and Quality Assurance to develop an annual
peer review self-assessment and verification process to be conducted
by each office and region.  The self-assessment will include
information on each peer review completed during the prior year as
well as feedback on the effectiveness of the overall process.  The
verification will consist of the signature of headquarters,
laboratory, or regional directors to certify that the peer reviews
were conducted in accordance with the agency's policy and procedures. 
If the peer review did not fully conform to the policy, the division
director or the line manager will explain significant variances and
actions needed to limit future significant departures from the
policy.  The self-assessments and verifications will be submitted and
reviewed by the Peer Review Advisory Group to aid in its oversight
responsibilities.  According to the Deputy Administrator, this
expanded assessment and verification process will help build
accountability and demonstrate EPA's commitment to the independent
review of the scientific analyses underlying the agency's decisions
to protect public health and the environment.  These new
accountability and oversight processes should take full effect in
October 1996. 

ORD's National Center for Environmental Research and Quality
Assurance has also agreed to play an expanded assistance and
oversight role in the peer review process.  Although the details had
not been completed, the Center's Director told us that his staff will
be available to assist others in conducting peer reviews and will try
to anticipate and flag the problems that they observe.  In addition,
the Center recently developed an automated Peer Review Panelist
Information System--a registry with information on identifying and
contacting potential reviewers according to their areas of expertise. 
Although the system was designed to identify potential reviewers of
applications for EPA grants, cooperative agreements, and fellowships,
the Center's Director stated that the registry (or similarly designed
ones) could also be used to identify potential peer reviewers for
EPA's technical and scientific work products. 


      OFFICE-SPECIFIC EFFORTS
---------------------------------------------------------- Letter :4.2

Recognizing that confusion remains about what peer review entails,
the Office of Water recently drafted additional guidance that further
clarifies the need for, use of, and ways to conduct peer review.  The
Office has also asked the Water Environment Federation to examine its
current peer review process and to provide recommendations on how to
improve it.  The Federation has identified the following areas of
concern, among others, where the program should be improved:  (1) the
types of, levels of, and methodologies for peer review; (2) the
sources and selection of reviewers; (3) the funding/resources for
peer review; and (4) the follow-up to, and accountability for, peer
review.  Similarly, OAR's Office of Mobile Sources proposed a Peer
Review/Scientific Presence Team in March 1996 to help OMS personnel
better understand the principles and definitions involved in the peer
review process.  In addition to promoting greater understanding, this
team would also help identify products and plan for peer review, as
well as facilitate and oversee the conduct of peer reviews for OMS'
scientific and technical work products. 

The Office of Solid Waste and Emergency Response recently formed a
team to support the Administrator's goal of sound science through
peer review.  The team was charged with strengthening the program
office's implementation of peer review by identifying ways to
facilitate good peer review and addressing barriers to its successful
use.  In May 1996, the team developed an implementation plan with a
series of recommendations that fall into the following broad
categories:  (1) strengthening early peer review planning; (2)
improving the ability of the Assistant Administrator to manage peer
review activities; (3) providing guidance and examples to support the
staff's implementation of peer review; and (4) developing mechanisms
to facilitate the conduct of peer reviews. 

EPA's Region 10 formed a Peer Review Group with the responsibility
for overseeing the region's reviews.  In March 1996, the group had a
meeting with the region's senior management, where it was decided to
later brief mid-level managers on the importance of peer review and
their peer review responsibilities.  Agreement was also reached to
have each of the region's offices appoint a peer review contact who
will receive training from the Peer Review Group and be responsible
for managing some peer reviews and for coordinating other major peer
review projects. 

The above agencywide and office-specific efforts should help address
the confusion about peer review and the accountability and oversight
problems we identified.  However, the efforts aimed at better
informing staff about the benefits and use of peer review are not
being done fully in all offices and would be more effective if done
consistently throughout the agency.  Similarly, the efforts aimed at
improving the accountability and oversight of peer review fall short
in that they do not ensure that each office and region has considered
all relevant products for peer review and that the reasons are
documented when products are not selected. 


   CONCLUSIONS
------------------------------------------------------------ Letter :5

Despite some progress, EPA's implementation of its peer review policy
remains uneven 2 years after it became effective.  Confusion remains
about what peer review entails and how it differs from the mechanisms
that EPA uses to obtain the views of interested and affected parties. 
Furthermore, the agency's accountability and oversight mechanism
provides too much leeway for managers to opt out of conducting peer
reviews without having to justify or document such decisions.  The
annual listing of only those products that have been selected for
peer review has not enabled upper-level managers to see what products
have not been nominated for peer review nor the reasons for their
exclusion. 

A more useful tool would be to have the list contain all planned
major products with detailed information about the managers'
decisions about peer review.  For example, if peer review is planned,
the list would contain--as the current procedures already
require--information on the type and timing of it.  More
significantly, if the managers elect to not conduct peer review on
individual products, the list would provide an explanation of why the
products are not being nominated.  This process would provide
upper-level managers with the necessary information to determine
whether or not all products have been appropriately considered for
peer review. 

We acknowledge that there are other difficulties in properly
conducting peer reviews.  However, we believe that as EPA strengthens
the implementation of its peer review policy and gains more
widespread experience with the process, the agency will be better
positioned to address these other issues. 


   RECOMMENDATIONS
------------------------------------------------------------ Letter :6

To enhance the quality and credibility of its decision-making through
the more widespread and consistent implementation of its peer review
policy, we recommend that the Administrator, EPA, do the following: 

  -- Ensure that staff and managers are educated about the need for
     and benefits of peer review; the difference between peer review
     and other forms of comments, such as peer input, stakeholders'
     involvement, and public comment; and their specific
     responsibilities in implementing the policy. 

  -- Expand the current list of products nominated for peer review to
     include all major products, along with explanations of why
     individual products are not nominated for peer review. 


   AGENCY COMMENTS
------------------------------------------------------------ Letter :7

We provided copies of a draft of this report to the Administrator of
EPA for review and comment.  In responding to the draft, EPA
officials stated that the report was clear, instructive, and fair. 
The officials also provided us with some technical and presentational
comments that we have incorporated as appropriate. 


---------------------------------------------------------- Letter :7.1

We conducted our review from February 1996 through August 1996 in
accordance with generally accepted government auditing standards.  A
detailed discussion of our scope and methodology appears in appendix
I. 

As arranged with your offices, unless you publicly announce its
contents earlier, we plan no further distribution of this report
until 15 days after the date of this letter.  At that time, we will
send copies to the Administrator of EPA and other interested parties. 
We will also make copies available to others upon request. 

Please call me at (202) 512-6111 if you or your staff have any
questions.  Major contributors to this report are listed in appendix
III. 

Peter F.  Guerrero
Director, Environmental
 Protection Issues


OBJECTIVES, SCOPE, AND METHODOLOGY
=========================================================== Appendix I

The Chairmen of the Senate Small Business Committee; the Subcommittee
on Clean Air, Wetlands, Private Property, and Nuclear Safety, Senate
Committee on Environment and Public Works; and the Subcommittee on
Energy Production and Regulation, Senate Committee on Energy and
Natural Resources, asked us to assess the Environmental Protection
Agency's (EPA) (1) progress in implementing its peer review policy
and (2) efforts to improve the peer review process. 

To assess the status of EPA's implementation of its peer review
policy, we reviewed relevant documents and discussed the agency's use
of peer review with officials from EPA's Science Policy Council;
Office of Air and Radiation (Washington, DC, Durham, NC, and Ann
Arbor, MI); Office of Water; Office of Program Planning and
Evaluation; Office of Solid Waste and Emergency Response; and Office
of Prevention, Pesticides, and Toxic Substances (Washington, DC);
Office of Research and Development (Washington, DC and Research
Triangle Park, NC); and EPA Region 10 (Seattle, WA). 

We also interviewed and obtained documents from officials with the
National Academy of Sciences; the Water Environment Federation; the
National Environmental Policy Institute; and the Natural Resources
Defense Council. 

We reviewed a selection of scientific and technical products to
obtain examples of how EPA's program and regional offices were
implementing the peer review policy.  We asked officials from EPA's
Science Policy Council and Science Advisory Board to identify
products that, in their opinion, fell into the following categories: 
(1) those that fully complied with the policy; (2) those that
received some level of peer review but did not fully comply with the
policy; and (3) those that should have received but did not receive
peer review.  We then interviewed the officials responsible for the
products to determine how decisions were made about the products'
peer review. 

To assess EPA's efforts to improve the peer review process, we
reviewed relevant documents and discussed the agency's recent,
ongoing, and planned improvements with officials from EPA's Science
Policy Council; Science Advisory Board; and the program and regional
offices identified above. 

We conducted our review from February though August 1996 in
accordance with generally accepted government auditing standards. 


PEER REVIEW ACTIVITIES FOR NINE
SCIENTIFIC AND TECHNICAL WORK
PRODUCTS
========================================================== Appendix II

At our request, the Science Policy Council obtained information from
EPA program and regional offices and provided us with examples
illustrating the current uneven implementation of EPA's peer review
policy.  This list was further augmented by the Executive Director of
the Science Advisory Board.  Although these products are not
necessarily a representative sample, the Executive Director of EPA's
Science Policy Council stated that these cases provide good
illustrations of how the level of peer review within EPA remains
uneven.  We have grouped the cases below according to whether (1)
EPA's peer review policy was followed, (2) the policy was not fully
followed, or (3) a peer review was not conducted but should have
been. 


   PEER REVIEW POLICY FOLLOWED
-------------------------------------------------------- Appendix II:1


      EASTERN COLUMBIA PLATEAU
      AQUIFER SYSTEM
------------------------------------------------------ Appendix II:1.1

In January 1993, EPA Region 10 received a petition from a local
environmental group to designate the Eastern Columbia Plateau Aquifer
System as a "Sole-Source Aquifer" under the Safe Drinking Water Act. 
The technical work product was entitled Support Document for Sole
Source Aquifer Designation of the Eastern Columbia Plateau Aquifer
System.  Under the act, EPA may make this designation if it
determines that the aquifer is the principal or sole source for the
area's drinking water.  Once so designated, EPA would then review
federally assisted projects in the area to determine if these
activities could contaminate the aquifer. 

In August 1994, EPA prepared a draft document that presented the
technical basis for the designation.  Technical questions were raised
by commentors that prompted EPA to convene a panel of experts to
review the document.  The panel was given a list of specific
technical issues to address, the draft document, and the supporting
materials.  The peer review panel convened July 26-27, 1995, to
discuss their views. 

The peer reviewers were chosen by asking several "stakeholder"
organizations, including local governments, an environmental
organization, and the United States Geological Survey, to nominate
respected scientists with expertise in areas such as hydrogeology. 
From more than 15 nominees, a selection committee of EPA staff from
outside Region 10 chose 6 peer review panel members.  Although one
stakeholder group expressed dissatisfaction that their candidate was
not chosen for the panel, they eventually agreed that the panel
fairly and objectively reviewed the support document. 

In July 1995, EPA received the peer review panel's report and is
still in the process of responding to the panel's comments and those
received from the public. 


      WASTE TECHNOLOGIES
      INDUSTRIES INCINERATOR
------------------------------------------------------ Appendix II:1.2

Waste Technologies Industries (WTI) began limited operation of a
hazardous waste incinerator in East Liverpool, Ohio, in April 1993. 
Although permitted for operation under the Clean Air Act, the Clean
Water Act, and the Resource Conservation and Recovery Act, the
facility became the focus of national attention and controversy due
to several concerns.  For example, it was being built near populated
areas and an elementary school, and the public was skeptical about
industries' management of commercial incinerators, the ability of
government agencies to regulate them, and whether the existing laws
and regulations are sufficient to protect public health and the
environment.  The WTI site was chosen, in part, because of its
proximity to steel mills, chemical plants, and other industries
generating hazardous waste suitable for incineration.  When fully
operational, this site will incinerate over 100,000 tons of hazardous
wastes annually. 

The original permit for WTI had been based solely on the modeled
effects of direct inhalation exposures and had not included other
exposure scenarios, such as indirect exposure through the food chain. 
Because of such risk assessment omissions and the controversy
associated with the facility, EPA decided to conduct an on-site risk
assessment of the cumulative human health and ecological risks
associated with the operations of this facility, as well as such
risks from accidents at the facility, and to publish its findings
prior to the full operation of the WTI site. 

According to the Senior Science Advisor for the Office of Solid Waste
and Emergency Response, peer review was envisioned early in the
process and occurred at several stages, including peer review of the
agency's approach to addressing these issues and peer review of the
entire report, including the conclusions and recommendations.  She
also said that about $120,000, or nearly 20 percent of all extramural
funds that EPA spent on this over 3-year effort, went to cover peer
review costs. 


   PEER REVIEW POLICY NOT FULLY
   FOLLOWED
-------------------------------------------------------- Appendix II:2


      DIOXIN REASSESSMENT
------------------------------------------------------ Appendix II:2.1

EPA began to assess the risks of dioxin in the early 1980s, resulting
in a 1985 risk assessment that classified the chemical as a probable
human carcinogen, primarily on the basis of animal studies available
at that time.  The implications of additional advances in the early
1990s were uncertain:  some maintained that dioxin's risks were not
as great as earlier believed, while others made the opposite
argument.  Given the growing controversy, in April 1991 EPA decided
to work closely with the broader scientific community to reassess the
full range of dioxin risks.  The draft product, which was released
for public comment in September 1994, contained an exposure document
and a health effects document.  The last chapter of the health
effects document characterized the risks posed from dioxin by
integrating the findings of the other chapters. 

Nearly the entire document received formal peer reviews by experts
from outside of EPA prior to the draft's release for public comment. 
However, the integrating chapter on risk characterization was not as
thoroughly peer reviewed as the rest of the document.  Following the
public comment period, both documents were submitted to the Science
Advisory Board (SAB) for its review.  While the SAB commended EPA on
most of the product, it had considerable problems with the risk
characterization chapter and faulted EPA for not having the chapter
externally peer reviewed prior to the SAB's review: 

     "The importance of this .  .  .  demands that the highest
     standards of peer review extend to the risk characterization
     itself.  Although it can be argued that this is in fact being
     carried out by this SAB [Science Advisory Board] Committee,
     submitting the risk characterization chapter for external peer
     review prior to final review by the SAB would serve to
     strengthen the document, and assure a greater likelihood of its
     acceptance by the scientific community-at-large.  It is
     recommended strongly that:  a) the risk characterization chapter
     undergo major revision; and b) the revised document be peer
     reviewed by a group of preeminent scientists, including some
     researchers from outside the dioxin "community" before returning
     to the SAB."\7

Members of Congress also criticized EPA's risk characterization
document and its lack of peer review.  In the House and Senate
reports on the fiscal year 1996 appropriations bill for EPA, concerns
were raised that the draft document

"does not accurately reflect the science on exposures to dioxins and
their potential health effects['] .  .  .  EPA selected and presented
scientific data and interpretations .  .  .  dependent upon
assumptions and hypotheses that deserve careful scrutiny[,] .  .  . 
and inaccuracies and omissions .  .  .  were the result of the
Agency's failure to consult with and utilize the assistance of the
outside scientific community .  .  ."\8

The committees directed EPA to respond to the SAB's concerns and
consult with scientists in other agencies in rewriting the risk
characterization chapter.  The House committee also restricted EPA
from developing any new rules that raise or lower dioxin limits on
the basis of the risk reassessment. 

As of July 1996, EPA was in the process of responding to the
committees', SAB's, and the public's comments.  The risk
characterization chapter is being subjected to a major revision and
will be peer reviewed by external scientific experts prior to
referral back to the SAB.  The SAB will then be asked to evaluate
EPA's response to their suggestions and the adequacy of the
additional peer review conducted on the draft report. 


--------------------
\7 An SAB Report:  A Second Look at Dioxin, Review of the Office of
Research and Development's Reassessment of Dioxin and Dioxin-like
Compounds by the Dioxin Reassessment Review Committee
(EPA-SAB-EC-95-021, Sept.  1995). 

\8 H.  Rept.  104-201, pp.  53-54; Senate Rept.  104-140, p.  89. 


      GREAT WATERS PROGRAM
------------------------------------------------------ Appendix II:2.2

Section 112(m) of the Clean Air Act Amendments of 1990 required EPA
to determine if atmospheric inputs of pollutants into the Great
Waters warrants further reductions of atmospheric releases and to
report the agency's findings to the Congress 3 years after the act's
enactment.  The Great Waters program includes the Great Lakes, Lake
Champlain, Chesapeake Bay, and the coastal waters.  EPA made its
first report to the Congress in May 1994. 

The scientific and technical data in this report, Deposition of Air
Pollutants to the Great Waters:  First Report to Congress, were peer
reviewed by 63 reviewers.  The reviewers represented a number of
different perspectives, including academia, industry, environmental
groups, EPA offices, other federal and state agencies, and Canadian
entities.  According to the Great Waters Program Coordinator, the
reviewers were given copies of all the report chapters, except the
conclusions and recommendation chapter, so that they could prepare
for a peer review workshop.  The reviewers then met to discuss the
report and provide EPA with their views. 

EPA expended a great deal of effort to ensure that the science in the
report was peer reviewed; however, the program coordinator said the
agency did not have the conclusions and recommendations chapter peer
reviewed.  The decision not to peer review this chapter was based on
the belief by those directing the program that these were the
agency's opinions based on the information presented and thus an
inherently governmental function not subject to peer review. 
However, others within EPA believe that nothing should be withheld
from peer review and said that the conclusions should have been peer
reviewed to ensure that they were indeed consistent with the
scientific content. 


      UNIT PRICING REPORT
------------------------------------------------------ Appendix II:2.3

Residential unit pricing programs involve charging households
according to the amount, or number of units, of garbage that they
produce.  In accordance with the principle that the polluter pays,
unit pricing provides a financial incentive for reducing municipal
waste generation and enhancing recycling.  EPA's Office of Policy,
Planning and Evaluation (OPPE) used a cooperative agreement to have
an assessment prepared of the most significant literature on unit
pricing programs to determine the degree to which unit pricing
programs meet their stated goals.  The paper, which was completed in
March 1996, highlights those areas where analysts generally agree on
the outcomes associated with unit pricing, as well as those areas
where substantial controversy remains.  Unit pricing is still
voluntary in the United States, according to the project officer;
however, he said EPA believes that the more information that
municipalities have readily available as they make long-term solid
waste landfill decisions, the more likely these local governments are
to employ some form of unit pricing as a disincentive to the
continued unrestrained filling of landfills. 

The OPPE project director had the report internally peer reviewed by
three EPA staff knowledgeable about unit pricing.  The report was not
externally peer reviewed, he said, because it is designed to be used
only as a reference guide by communities that are considering
implementing some type of unit pricing program to reduce waste, and
because EPA does not intend to use the report to support any
regulatory actions. 


      TECHNICAL ANALYSIS OF AJ
      MINE'S TAILING IMPOUNDMENT
------------------------------------------------------ Appendix II:2.4

The Alaska Juneau (AJ) Gold Mine project was a proposal by the Echo
Bay, Alaska, company to reopen the former mine near Juneau.  The
proposal entailed mining approximately 22,500 tons of ore per day
and, after crushing and grinding the ore, recovering gold through the
froth flotation and carbon-in-leach (also called cyanide leach)
processes.  After the destruction of residual cyanide, the mine
tailings would be discharged in a slurry form to an impoundment that
would be created in Sheep Creek Valley, four miles south of downtown
Juneau. 

An environmental impact statement was prepared on the proposal in
1992.  Because the project would require permits for fill materials
and discharging wastewater into surface waters, EPA's regional staff
developed a model to predict the environmental ramifications of the
proposal.  According to regional staff, a careful analysis of the
proposal was important because the issues in this proposal could
potentially set a precedent for similar future proposals. 

EPA went through three iterations of the model.  The first model was
presented in a report entitled A Simple Model for Metals in the
Proposed AJ Mine Tailings Pond.  The report was reviewed by an
engineer in EPA's Environmental Research Laboratory and a firm that
worked for the City and Borough of Juneau.  The second model was a
customized version of one developed by EPA's Research Laboratory. 
After receiving comments from the firm representing Echo Bay, ORD
laboratories, the Corps of Engineers, and others, EPA decided to also
use another model to evaluate the proposal's potential environmental
effects.  In 1994, EPA prepared a technical analysis report on the
proposal.  The report received peer review by several of the same
individuals who commented on the models, as well as others.  Although
the reviewers had expertise in the subject matter, several were not
independent of the product's development or its regulatory and/or
financial ramifications. 

Based partially on the model's predictions, it became evident that
EPA would withhold permit approval for the project.  Accordingly,
Echo Bay developed an alternative design for its project.  In May
1995, EPA hired a contractor to prepare a supplemental environmental
impact statement that will assess the revised project's ecological
effects.  The agency plans to have the impact statement peer
reviewed. 


      METHODOLOGY FOR ESTABLISHING
      HAZARDOUS WASTE EXIT
      CRITERIA
------------------------------------------------------ Appendix II:2.5

Under the Resource Conservation and Recovery Act (RCRA), EPA is not
only responsible for controlling hazardous wastes but also for
establishing procedures for determining when hazardous wastes are no
longer a health and/or ecological concern.  As such, EPA's Office of
Solid Waste (OSW) developed a new methodology for establishing the
conditions under which wastes listed as hazardous may be delisted. 
This methodology was presented in an OSW report, Development of Human
Health Based and Ecologically Based Exit Criteria for the Hazardous
Waste Identification Project (March 3, 1995), which was intended to
support the Hazardous Waste Identification Rule.  The intent of this
rule is to establish human health-based and ecologically based waste
constituent concentrations--known as exit criteria--for constituents
in wastes below which listed hazardous wastes would be reclassified
and become delisted as a hazardous waste.  Such wastes could then be
handled as a nonhazardous solid waste under other provisions of RCRA. 
OSW's support document describes a proposed methodology for
calculating the exit concentrations of 192 chemicals for humans and
about 50 chemicals of ecological concern for five types of hazardous
waste sources; numerous release, transport, and exposure pathways;
and for biological effects information. 

Although years of effort went into developing the proposed
methodology, OSW did not have its proposed rule and supporting risk
analyses peer reviewed until very late in the methodology development
process.  Peer review by the Science Advisory Board was later chosen
as the most appropriate mechanism for ensuring credible science. 
According to the SAB's May 1996 report, the program office's failure
to have the proposed methodology appropriately peer reviewed resulted
in important omissions, errors, and flawed approaches in the
methodology which will now take from 1 to 2 years to correct.  The
report further noted that

     "The Subcommittee is seriously concerned about the level of
     scientific input and the degree of professional judgment that,
     to date, have been incorporated into the methodology
     development.  It was clear to the Subcommittee that there has
     been inadequate attention given to the state-of-the-science for
     human and ecological risk assessment that exists within EPA, let
     alone in the broader scientific community, in the development of
     the overall methodology, the identification of individual
     equations and associated parameters, the selection of models and
     their applicability, and the continual need for sound scientific
     judgment."

The SAB also noted that further peer review of individual elements of
the proposed methodology is essential before the scientific basis can
be established.  The SAB concluded that the methodology at present
lacks the scientific defensibility for its intended regulatory use. 

According to SAB's Executive Director, this is a case where the
program office's decision to not conduct a peer review of the key
supporting elements of a larger project resulted in extra cost and
time to the agency, as well as missed deadlines.  He pointed out that
the experience on this one effort had now, he believed, caused a
cultural change in the Office of Solid Waste, to the extent that they
now plan to have peer consultation with the SAB on several upcoming
lines of effort. 


   PEER REVIEW NOT CONDUCTED
-------------------------------------------------------- Appendix II:3


      MOBILE 5A MODEL
------------------------------------------------------ Appendix II:3.1

Mobile 5A, also known as the mobile source emissions factor model, is
a computer program that estimates the emissions of hydrocarbons,
carbon monoxide, and nitrogen oxide for eight different types of
gasoline-fueled and diesel highway motor vehicles.  The first mobile
model, made available for use in 1978, provided emissions estimates
only for tailpipe exhaust emissions from passenger cars.  Since that
time, major updates and improvements to the mobile model have
resulted in the addition of emissions estimates for evaporative
(nontailpipe exhaust) emissions and for uncorrected in-use
deterioration due to tampering or poor maintenance, according to the
OMS Emission Inventory Group Manager.  Also, other categories of
vehicles, such as light-duty trucks and motorcycles, have been added
over the years, she said. 

The development of the next generation model, Mobile 6, is currently
under way.  As with other models, the mobile model exists because
precise information about the emissions behavior of the approximately
200 million vehicles in use in the United States is not known,
according to the Group Manager.  The primary use of the mobile model
is in calculating the estimated emissions reductions benefits of
various actions when applied to the mobile sources in an area.  For
example, the mobile model can estimate the impact of participating in
a reformulated gasoline program, or of using oxygenated fuels in an
area, or of requiring periodic inspection and maintenance of selected
vehicle categories.  In essence, the mobile model is one of the
primary tools that EPA, states, and localities use to measure the
estimated emissions reduction effectiveness of the pollution control
activities called for in State Implementation Plans. 

None of the previous mobile models has been peer reviewed.  However,
EPA has obtained external views on the model through stakeholders'
workshops and experts' meetings; one of the largest of these meetings
involved over 200 stakeholders, according to OMS officials.  The
agency recognizes that these workshops and meetings are not a
substitute for peer review and, in a reversal of the agency's views
of 10 months ago, EPA now plans to have Mobile 6 peer reviewed, they
said.  Several constraints, such as the limited number of unbiased
experts available to do peer review in some fields and the resources
for compensating reviewers, still have to be overcome, they added. 


      TRIBUTYL TIN IMPACTS
------------------------------------------------------ Appendix II:3.2

Tributyl tin (TBT) is a compound used since the 1960s as an
antifouling ingredient for marine paints.  In the 1970s, antifouling
paints were found to adversely affect the environment.  Although
restrictions were placed on TBT by the United States and a number of
other countries in the 1980s, elevated levels of TBT continue to be
found in marine ecosystems.  In light of the uncertain human health
and environmental effects of TBT, an interagency group consisting of
EPA Region 10 officials, the Washington State Departments of Ecology
and Natural Resources, the National Oceanographic and Atmospheric
Administration, the U.S.  Army Corps of Engineers, and others was
formed to derive a marine/estuarine sediment effects-based cleanup
level (or screening level) for TBT.  In April 1996, a
contractor-prepared report was issued with recommended screening
levels; EPA regional staff served as the project managers and made
significant contributions to the revisions to and final production of
the report. 

Although an EPA project manager maintains that the report was peer
reviewed, the reviews did not meet the requirements of EPA's peer
review policy nor the region's standard operating procedures for
conducting peer reviews.  While the report was reviewed by members of
the interagency group, other experts who provided input to the
report, the affected regulated community, and the general public,
there was not an independent review by experts not associated with
preparing the report or by those without a stake in its conclusions
and recommendations.  When we explained to the project manager why
EPA's Science Policy Council characterized the report as not having
received peer review, the project manager acknowledged that she was
not familiar with either EPA's peer review policy or the region's
standard operating procedures.  EPA is currently in the process of
responding to the comments it has received. 


MAJOR CONTRIBUTORS TO THIS REPORT
========================================================= Appendix III

RESOURCES, COMMUNITY, AND ECONOMIC
DEVELOPMENT DIVISION, WASHINGTON,
D.C. 

Stanley J.  Czerwinski, Associate Director
Susan D.  Kladiva, Assistant Director
Gregory A.  Kosarin, Evaluator-in-Charge
Richard A.  Frankel, Senior Scientist

NORFOLK REGIONAL OFFICE

James R.  Beusse, Senior Evaluator
Philip L.  Bartholomew, Staff Evaluator


*** End of document. ***