Nuclear Waste: Further Actions Needed to Increase the Use of Innovative
Cleanup Technologies (Chapter Report, 09/25/98, GAO/RCED-98-249).

Pursuant to a congressional request, GAO reviewed the Department of
Energy's (DOE) Office of Environmental Management's (EM) efforts to
deploy innovative cleanup technologies, focusing on: (1) the extent to
which innovative technologies developed by the Office of Science and
Technology (OST) have been deployed at DOE sites and how this rate of
deployment compares with the rates of other government organizations
that develop environmental technologies; (2) what progress EM has made
in overcoming obstacles to deploying innovative technologies at DOE
cleanup sites; and (3) what EM is doing to increase the deployment of
innovative technologies.

GAO noted that: (1) OST has initiated 713 technology development
projects and has reported that 152 projects have been deployed one or
more times, for an overall deployment rate of 21 percent; (2) GAO found
many errors in the office's deployment data and estimates that EM has
deployed between 88 and 130 of these projects, for an overall deployment
rate of 12 to 18 percent; (3) OST overstated its deployment information
because it had not previously maintained comprehensive deployment data;
compiled the data rapidly in response to congressional requests; and
lacked procedures for compiling the data; (4) in comparison with the
deployment rates of other programs that demonstrate environmental
technologies--the Environmental Protection Agency's Superfund Innovative
Technology Evaluation Program and the Department of Defense's
Environmental Security Technology Certification Program--OST's
deployment rate for projects at comparable stages of development falls
between the rates of these two programs; (5) however, comparisons of
OST's deployment rate with the rates of other organizations' programs
must be viewed with caution because no organization is fully comparable
with OST, and the deployment rate is not the only possible measure of
success for research and development programs; (6) as DOE's
Environmental Management program has matured, its waste cleanup sites
have made progress in overcoming some obstacles to implementing
innovative technologies; (7) other obstacles that are internal to the
operations of EM and its OST continue to slow the use of innovative
technologies, including the lack of: (a) involvement by technology users
in the development of cleanup technologies by OST; and (b) technical
assistance by OST to help sites select and implement technologies; (8)
after congressional hearings in May 1997, EM initiated changes in its
organization and processes to increase the deployment of innovative
technologies; (9) some of these initiatives address the internal
obstacles limiting deployment; (10) however, the office has not yet
improved developer-user cooperation in individual technology development
projects; (11) OST does not consistently and rigorously use its existing
decisionmaking process for managing the development of innovative
technologies; and (12) EM has yet to determine how it will: (a) provide
technical assistance to sites in selecting and implementing innovative
technologies; and (b) make modifications to completed technologies to
meet sites' specific needs and conditions.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  RCED-98-249
     TITLE:  Nuclear Waste: Further Actions Needed to Increase the Use 
             of Innovative Cleanup Technologies
      DATE:  09/25/98
   SUBJECT:  Environmental policies
             Technology transfer
             Pollution control
             Environmental research
             Comparative analysis
             Environmental monitoring
             Nuclear waste disposal
             Cost control
             Data integrity
             Nuclear waste management
IDENTIFIER:  DOE Environmental Management Program
             Superfund Innovative Technology Evaluation Program
             DOD Environmental Security Technology Certification Program
             Hanford (WA)
             Savannah River (SC)
             Oak Ridge (TN)
             Ohio
             California
             OST Accelerated Site Technology Deployment Program
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


NUCLEAR WASTE - FURTHER ACTIONS
NEEDED TO INCREASE THE USE OF
INNOVATIVE CLEANUP TECHNOLOGIES

GAO/RCED-98-249

DOE's Innovative Cleanup Technologies

(141094)


Abbreviations
=============================================================== ABBREV

  ASEM - Assistant Secretary for Environmental Management
  ASTD - Accelerated Site Technology Deployment
  DOE - Department of Energy
  EM - Environmental Management
  EMAB - Environmental Management Advisory Board
  EPA - Environmental Protection Agency
  ESTCP - Environmental Security Technology Certification Program
  GAO - General Accounting Office
  OST - Office of Science and Technology
  R&D - research and development
  SITE - Superfund Innovative Technology Evaluation

Letter
=============================================================== LETTER


B-280628

September 25, 1998

Congressional Committees

As requested, this report addresses the deployment rate for
technologies developed by the Office of Science and Technology in the
Department of Energy's (DOE) Environmental Management program and
compares this rate with those of other similar organizations.  The
report also addresses obstacles to the use of innovative technologies
at DOE's cleanup sites and the Environmental Management program's
efforts to overcome these obstacles. 

As arranged with your offices, unless you publicly announce its
contents earlier, we plan no further distribution of this report
until 15 days after the date of this letter.  At that time, we will
provide copies of the report to interested congressional committees,
the Secretaries of Energy and Defense, the Administrator of the
Environmental Protection Agency, the Director of the Office of
Management and Budget, and other interested parties.  We will also
make copies available to others on request. 

Please call me at (202) 512-8021 if you or your staff have any
questions.  Major contributors to this report are listed in appendix
V. 

(Ms.) Gary L.  Jones
Associate Director, Energy,
 Resources, and Science Issues


List of Committees

The Honorable Tom Bliley
Chairman
The Honorable John D.  Dingell
Ranking Minority Member
Committee on Commerce
House of Representatives

The Honorable Joe Barton
Chairman
The Honorable Ron Klink
Ranking Minority Member
Subcommittee on Oversight
 and Investigations
Committee on Commerce
House of Representatives



EXECUTIVE SUMMARY
============================================================ Chapter 0

The Department of Energy (DOE) estimates that it could save $20
billion or more by using innovative technologies to clean up the
hazardous and radioactive contaminants resulting from the production
of nuclear weapons at its facilities.  To achieve these savings, the
Office of Science and Technology (OST),\1 within DOE's Office of
Environmental Management (EM), is developing technologies that could
reduce cleanup costs, accelerate cleanups, provide methods for
cleanup activities for which there are no existing cost-effective
technologies, and/or reduce risks to cleanup workers and the public. 
However, previous reports by GAO and others have identified obstacles
that make it difficult to select and use innovative technologies at
DOE sites. 

Because of concern about the benefits returned from the approximately
$2.5 billion invested in OST since 1989, the House Committee on
Commerce and its Subcommittee on Oversight and Investigations
requested that GAO review EM's efforts to deploy innovative
technologies.  Specifically, GAO was asked to determine (1) to what
extent innovative technologies developed by OST have been deployed
(used) at DOE sites and how this rate of deployment compares with the
rates of other government organizations that develop environmental
technologies; (2) what progress EM has made in overcoming obstacles
to deploying innovative technologies at DOE cleanup sites; and (3)
what EM is doing to increase the deployment of innovative
technologies. 


--------------------
\1 This office was originally named the Office of Technology
Development, but it was later renamed because basic science research
for waste cleanup was added to its responsibilities. 


   BACKGROUND
---------------------------------------------------------- Chapter 0:1

In 1989, the Congress directed DOE to establish a research program to
develop technologies to reduce environmental hazards and restore the
environment.  OST develops environmental technologies from the early
stages, which involve laboratory experimentation, through the later
stages, which involve the demonstration and testing of technologies'
performance.  OST is organized into "focus areas" that concentrate on
the major cleanup problems facing DOE sites.  Among DOE's cleanup
challenges being addressed by OST's technology research and
development are the following:  massive underground tanks that
contain high-level radioactive wastes, migrating areas of hazardous
and radioactive substances in groundwater, acres of contaminated
soil, and thousands of buildings no longer in use that require
decontamination and dismantlement.  DOE sites are responsible for
selecting the technologies to be used in cleanup projects.  These
selections are subject to the review and approval of the
Environmental Protection Agency (EPA) and the state agencies that
regulate DOE's cleanups. 


   RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:2

The Office of Science and Technology has initiated 713 technology
development projects and has reported that 152 projects have been
deployed one or more times, for an overall deployment rate of 21
percent.  GAO found many errors in the office's deployment data and
estimates that Environmental Management has deployed between 88 and
130 of these projects, for an overall deployment rate of 12 to 18
percent.  The Office of Science and Technology overstated its
deployment information because it had not previously maintained
comprehensive deployment data; compiled the data rapidly in response
to congressional requests; and lacked procedures for compiling the
data, such as a formal definition of what constitutes a deployment. 
In comparison with the deployment rates of other programs that
demonstrate environmental technologies--the Environmental Protection
Agency's Superfund Innovative Technology Evaluation Program and the
Department of Defense's Environmental Security Technology
Certification Program--the Office of Science and Technology's
deployment rate for projects at comparable stages of development
falls between the rates of these two programs.  However, comparisons
of the Office of Science and Technology's deployment rate with the
rates of other organizations' programs must be viewed with caution
because no organization is fully comparable with the Office of
Science and Technology, and the deployment rate is not the only
possible measure of success for research and development programs. 

As DOE's Environmental Management program has matured, DOE waste
cleanup sites have made progress in overcoming some obstacles to
implementing innovative technologies, such as addressing the concerns
of regulators and public stakeholders.  However, other obstacles that
are internal to the operations of the Office of Environmental
Management and its Office of Science and Technology continue to slow
the use of innovative technologies.  These obstacles include the lack
of (1) involvement by technology users in the development of cleanup
technologies by the Office of Science and Technology and (2)
technical assistance by the Office of Science and Technology to help
sites select and implement technologies. 

After congressional hearings in May 1997, the Office of Environmental
Management initiated changes in its organization and processes to
increase the deployment of innovative technologies.  Some of these
initiatives address the internal obstacles limiting deployment.  For
example, one initiative provided for involving technology users in
setting overall plans and priorities for the Office of Science and
Technology, such as the general areas of research in which the office
should invest.  However, the office has not yet improved
developer-user cooperation in individual technology development
projects.  Furthermore, the Office of Science and Technology does not
consistently and rigorously use its existing decision-making process
for managing the development of innovative technologies.  This
process would require technology users to be involved at various
stages in a technology's development.  In addition, the Office of
Environmental Management has yet to determine how it will (1) provide
technical assistance to sites in selecting and implementing
innovative technologies and (2) make modifications to completed
technologies to meet sites' specific needs and conditions.  These
steps could increase the use of completed technology development
projects when they present benefits over the use of conventional
approaches. 


   PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:3


      DEPLOYMENT RATE OF
      OST-DEVELOPED TECHNOLOGIES
-------------------------------------------------------- Chapter 0:3.1

OST developed deployment data in response to a November 1996 request
from the Subcommittee on Oversight and Investigations, House
Committee on Commerce.  As of January 1998, OST's database showed
that EM had initiated 713 technology projects since OST's inception. 
On the basis of its independent verification and analysis of OST's
deployment data, GAO estimates that EM actually has deployed from 88
to 130 projects, to achieve an overall deployment rate of 12 to 18
percent for the 713 projects.  In contrast, OST has reported that 152
of the 713 technology projects initiated since its inception have
been deployed.  Thus, according to OST's data, about 1 in 5 OST
technologies have been deployed one or more times, for an overall
deployment rate of 21 percent.  (See app.  I for more detail on GAO's
methodology.)

OST's inaccurate deployment data resulted from several factors. 
Specifically, OST had not maintained comprehensive deployment data
and compiled the data quickly in response to the congressional
request.  In addition, the lack of a formal definition of deployment
led to differing understandings among the personnel responsible for
compiling the data.  OST has begun to establish procedures for
entering and updating project data and, in August 1998, formalized a
definition of deployment.  In the interim, however, OST has continued
to report inaccurate data to the Congress.  For example, OST's latest
semiannual report to the Congress, dated April 1998, stated that,
through September 1997, over 140 OST-sponsored technologies had been
deployed. 

GAO compared OST's deployment rate with those of the Environmental
Protection Agency's Superfund Innovative Technology Evaluation
Program and the Department of Defense's Environmental Security
Technology Certification Program.  (See app.  II for more detail on
these comparisons.) For projects at comparable later stages of
development, OST's deployment rate ranges from 28 to 45 percent,\2
while the Superfund program's rate is 59 percent and the
Environmental Security program's rate is 38 percent.  However,
comparisons of OST's deployment rate with the rates of other
organizations must be viewed with caution for several reasons.  For
example, GAO found few organizations that engage in the range of
environmental research OST performs, and none of the 10 organizations
that GAO contacted routinely tracked deployment data on their
projects.  It should also be noted that, while GAO was asked to
develop deployment information, the deployment rate is not the only
performance measure relevant to research and development programs and
is more applicable to technologies in the later stages of
development.  Furthermore, developers of later-stage technologies
believe that the deployment rate is an incomplete performance measure
and that cost savings or some measure of dollar impact should also be
used to evaluate a program's success.  EM has used the completion of
demonstrations as a performance measure for OST for several years and
is considering developing additional measures more relevant to
earlier stages of technology development, as well as a performance
measure addressing cost savings resulting from using innovative,
rather than conventional, technologies. 


--------------------
\2 Since GAO used only a sample of OST's projects to make its
estimates, these estimates have a margin of error.  The range stated
reflects this margin of error at the 95-percent confidence level. 
See app.  I for details about GAO's methodology. 


      OBSTACLES TO DEPLOYING
      INNOVATIVE TECHNOLOGIES AT
      DOE SITES
-------------------------------------------------------- Chapter 0:3.2

As EM's cleanup program has matured, it has addressed several of the
obstacles to using innovative technologies reported by GAO, EM, and
others.  For example, one reported obstacle was a lack of awareness
among DOE field staff about available technologies.  In visits to
DOE's waste cleanup sites, GAO found that federal and contractor
staff had become better informed than in the past about innovative
technologies that are relevant to their cleanup needs.  In addition,
DOE sites and their regulators had improved their working
relationships, and, in cases where innovative technologies were
selected, DOE sites had found ways to address regulatory concerns. 
In several cases, sites had phased in technologies to increase the
regulators' confidence in the technologies' performance. 

On the other hand, several obstacles that are internal to EM's and
OST's operations continue to hinder the deployment of OST-developed
technologies.  Specifically, OST has not involved users as
technologies are being developed.  OST did not comprehensively
identify the sites' technology needs until October 1996 and has not
involved technology users in the decision-making process (referred to
as the gates system) during the development of technologies.  As one
result of the lack of user involvement, site staff find that some OST
technologies are generic and do not meet their specific conditions
and performance requirements.  For example, officials at DOE's
Hanford site tried two OST technologies that promised to characterize
the nature and extent of contamination in soil more quickly than
conventional methods and thus enable the site to remediate the
contamination sooner.  The officials, however, rejected the
technologies because they were not designed to work effectively in
Hanford's arid soil. 

Furthermore, when technologies require modification to fit a site's
specific situation, site officials told GAO that it is not clear who
should pay for these modifications.  For example, Hanford officials
wanted to use OST's Electrical Resistance Tomography equipment to
help detect leaks in waste tanks from outside the tanks, but Hanford
did not have funding to fine-tune the equipment for its needs.  While
OST has data identifying potential uses for its technologies at
specific cleanup projects, EM lacks a policy on who is responsible
for paying for any necessary modifications to adapt the OST-developed
technologies for specific field uses. 

Finally, OST has infrequently provided technical assistance to sites
during technology selection or implementation.  GAO recommended in
1994 that EM give OST a formal role in technology selection
decisions.\3 However, the recommendation was not implemented because
site personnel lack confidence in OST's ability to provide expert
technical advice and assistance and are therefore reluctant to allow
OST a formal role in their technology selections.  In visits to five
DOE sites, GAO found that the sites infrequently sought technical
assistance from OST.  According to a report by an advisory board to
EM, OST staff are not always well informed about technologies
developed by organizations other than OST.  OST's managers recognize
that staff in its focus areas do not always have sufficient technical
expertise to assist sites with their specific cleanup problems. 


--------------------
\3 Department of Energy:  Management Changes Needed to Expand Use of
Innovative Cleanup Technologies (GAO/RCED-94-205, Aug.  10, 1994). 


      EM'S EFFORTS TO INCREASE
      DEPLOYMENT
-------------------------------------------------------- Chapter 0:3.3

In response to congressional hearings and concerns, the Assistant
Secretary for EM identified actions intended to increase the
deployment of innovative technologies, such as establishing the
Technology Acceleration Committee, which is composed of upper
managers from EM headquarters and field offices; identifying
responsibilities and performance measures for deployment; and
requiring sites to develop deployment plans.  These actions were
completed over time--some in the fall of 1997 and others in the
spring and summer of 1998.  For instance, for fiscal year 1998, EM
established performance measures on technology deployment for its
sites.  In addition to EM's actions, OST is requiring multiyear plans
for its five focus areas; and in February 1998, EM established user
steering committees for the focus areas to assist with these plans
and priority setting. 

However, continued attention by EM's upper management to deployment
is not ensured because the Technology Acceleration Committee may not
remain in effect following the departure of EM's Assistant Secretary
and because planned deployment measures for managers have not been
implemented.  Although the actions initiated by the Assistant
Secretary for EM were to include deployment measures in the annual
performance expectations of EM's upper managers, EM has not carried
out this aspect of its planned performance measures.  Instead, EM
managers' involvement in the Technology Acceleration Committee was
considered to be a sufficient means of holding the managers
accountable for deployment.  This Committee has not met since January
1998, and EM is considering establishing a new executive committee of
senior managers to address EM issues, including deployment. 

Furthermore, the actions specified by the Assistant Secretary for EM
do not include user involvement in individual OST projects--the type
of involvement needed to ensure that completed technologies address
sites' requirements.  Although the new user steering committees help
OST in its overall planning and priority setting, user involvement in
individual technology development projects was not addressed by the
actions specified by the Assistant Secretary for EM.  Meanwhile,
existing processes that would require user involvement are not used. 
In 1993, OST developed the decision-making system known as the gates
system, which (1) establishes requirements for technology projects at
various stages of development and (2) identifies decision points,
called gates, at which projects are evaluated and "go/no-go"
decisions are made.  The gates system includes requirements for user
involvement at the various development stages.  For example, gate
2--moving from applied research to exploratory development--requires
that research be linked to the specific needs of end-users at sites
and that end-users' performance requirements are specified.  However,
OST's use of the gates system has been spotty, in part because a
rigorous application of its requirements might indicate that some
projects should be terminated for reasons such as the lack of an
identified customer, according to several EM officials. 

In addition, the actions specified by the Assistant Secretary for EM
are not designed to increase the deployment--and return on
investment--of completed OST technologies.  While site officials
consider OST's technologies too generic, the actions do not address
modifying those technologies so that they could have additional
cost-effective uses.  The initiatives also do not specify policies,
resources, or processes for providing sites with technical assistance
on innovative technologies. 


   RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:4

To ensure the deployment of future technology development projects
and increase the use of innovative technologies that OST has
completed, GAO recommends that EM (1) consistently use the gates
system to make decisions about OST's projects and to increase the
involvement of users in technology projects, (2) use existing data to
identify technologies that could be used cost-effectively for site
cleanup projects and identify funds for modifying these technologies
if needed, and (3) establish centers of expertise for innovative
technologies by using existing focus areas or another approach if
needed and require that a representative from one of these centers
participate in the technology selection process on each cleanup
project.  GAO also makes several other recommendations to improve
deployment data and upper management's attention to technology
deployment. 


   AGENCY COMMENTS
---------------------------------------------------------- Chapter 0:5

GAO provided a draft of this report to DOE for its review and
comment.  Overall, DOE stated that it agreed with the findings and
recommendations of the draft report.  DOE did provide specific
comments and corrections, and GAO made changes where appropriate. 
DOE's comments are included as appendix III.  GAO also provided
portions of the draft of this report concerning the Environmental
Security Technology Certification Program to the Department of
Defense and portions concerning the Superfund Innovative Technology
Evaluation Program to the Environmental Protection Agency.  The
Department of Defense stated that it found the report to be correct
with respect to its program and provided specific comments and
corrections that GAO incorporated where appropriate.  The
Department's comments are included as appendix IV.  Similarly, the
Acting Director of the Environmental Protection Agency's Office of
Resources Management and Administration told GAO that the information
on the Superfund Innovative Technology Evaluation Program contained
in the report was summarized and reported accurately. 


INTRODUCTION
============================================================ Chapter 1

Nearly a decade ago, the Department of Energy (DOE) embarked on a
mission to deal with the environmental legacy of the Cold War.  This
DOE mission, which is expected to continue for many years, involves a
number of activities, with the most ambitious and far-ranging being
the cleanup of the hazardous and radioactive contaminants that
resulted from the production of nuclear weapons at DOE facilities. 
The challenges of this task are technical, institutional, and
economic.  For example, thousands of tons of radioactive waste must
be treated and put into permanent storage; contaminated soil must be
stabilized; contaminated water must be treated; and nuclear reactors
and materials-processing facilities must be decontaminated,
decommissioned, and demolished.  In some cases, no safe and effective
technology is currently available to address the more complex
contamination problems. 

In June 1998, DOE estimated that it will be very expensive, about
$150 billion, to clean up the legacy of the Cold War.\1 However, DOE
has also stated that the cost of cleaning up its contaminated
facilities and sites can be significantly reduced through the use of
innovative cleanup technologies.  It supports the development of
these technologies through its Office of Science and Technology
(OST), within the Office of Environmental Management (EM). 


--------------------
\1 Accelerating Cleanup:  Paths to Closure, DOE/EM-0362, June 1998. 


   OST ESTABLISHED TO DEVELOP
   INNOVATIVE TECHNOLOGIES
---------------------------------------------------------- Chapter 1:1

In 1989, DOE established EM to clean up and restore its contaminated
facilities and sites in compliance with federal and state
environmental laws and regulations.  The Congress also directed the
Secretary of Energy to establish "a program of research for the
development of technologies useful for (1) the reduction of
environmental hazards and contamination resulting from defense waste,
and (2) environmental restoration of inactive defense waste disposal
sites."

In response, DOE established the Office of Technology Development
within EM to develop innovative technologies to support the waste
cleanup and restoration efforts of EM's program offices--the Offices
of Waste Management, Environmental Restoration, and Nuclear Material
and Facility Stabilization.  The Office of Technology Development was
renamed the Office of Science and Technology in 1994, when basic
science research for waste cleanup was added to its responsibilities. 
OST's projects are intended to produce technologies that could
accelerate cleanups, reduce costs, enable cleanup activities for
which there are no existing cost-effective technologies, or reduce
risks to cleanup workers.  From fiscal year 1990 through fiscal year
1998, the Congress appropriated approximately $2.5 billion for OST's
development of innovative waste cleanup technologies,\2 and OST has
initiated over 700 projects.  OST's budget for technology development
activities in fiscal year 1998 is about $220 million.  OST requested
a total of $180.5 million for technology development activities for
fiscal year 1999.\3

OST develops technology at DOE's national laboratories, private
companies under contract to OST, and universities.  Although OST is
responsible for technology development, DOE waste sites are
responsible for selecting the technologies they will use, with the
review and approval of the Environmental Protection Agency (EPA) and
state agencies that regulate DOE's cleanups, and with input from the
public involved with the site.\4

To serve sites' needs for cleanup technology, OST is organized into
five major remediation and waste management problem areas (termed
"focus areas").  OST first established focus areas in 1994 in order
to better serve the cleanup sites by concentrating technology
resources on each of the major cleanup problems DOE faces.  OST
currently has the following five focus areas: 

  -- Mixed Waste Characterization, Treatment, and Disposal.  Known as
     "mixed waste," this focus area addresses the large inventory of
     mixed, low-level, and transuranic waste;\5

  -- Radioactive Tank Waste Remediation.  Known as "tanks," this
     focus area addresses the hundreds of large storage tanks
     containing over 100 million gallons of radioactive waste;

  -- Subsurface Contaminants.  This focus area addresses hazardous
     and radioactive contaminants in soil and groundwater and the
     remediation challenges posed by numerous DOE landfills;

  -- Deactivation and Decommissioning.  This focus area addresses the
     deactivation, decommissioning, and disposal of aging and
     contaminated DOE weapons complex facilities; and

  -- Plutonium Stabilization and Disposition.  This focus area
     addresses the over 20 tons of excess plutonium that must be
     stabilized. 

OST has established a lead field office to manage each focus area. 
For example, the Savannah River site manages the Subsurface
Contaminants Focus Area.  EM has also established site technology
coordination groups in each of its field offices to identify sites'
technology needs, provide information to OST and its focus areas, and
communicate information about OST's technology development projects
to the cleanup sites. 

In 1994, the Assistant Secretary for EM established the Environmental
Management Advisory Board (EMAB) to provide the Assistant Secretary
with information, advice, and recommendations on issues confronting
the EM program, including advice on the development and deployment of
innovative technology for waste cleanup.  EMAB has about 25 members
from industry; academia; and private, federal, tribal, state, and
local environmental groups.  EMAB has been very active in studying
OST and recommending improvements in its operations. 


--------------------
\2 Another $353 million was appropriated for basic science research
and for activities not directly related to technology development. 
While OST continues to be responsible for basic research, these other
responsibilities have since been moved to other parts of EM or DOE. 

\3 Figures for fiscal years 1998 and 1999 exclude funds for basic
science research and funds to be transferred to the Small Business
Innovative Research Program. 

\4 Remediation activities at DOE's facilities are governed by the
Comprehensive Environmental Response, Compensation, and Liability Act
of 1980, as amended, and the Resource Conservation and Recovery Act
of 1976, as amended.  These acts lay out the requirements for
identifying waste sites, studying the extent of their contamination
and identifying possible remedies, and involving the public in making
decisions about the sites. 

\5 Mixed waste contains both radioactive and hazardous waste. 
Low-level waste contains radioactive waste not classified as
high-level waste, spent nuclear fuel, transuranic waste, or
radioactive by-product material.  Transuranic waste contains
radioactive elements that are above uranium in the Periodic Table of
elements--that is, elements that have an atomic number greater than
92.  Transuranic waste is produced artificially (during a man-made
nuclear reaction). 


   USE OF INNOVATIVE TECHNOLOGY
   MAY HAVE BENEFITS, INCLUDING
   THE REDUCTION OF CLEANUP COSTS
---------------------------------------------------------- Chapter 1:2

DOE believes it will be very costly and take many years to clean up
its waste sites if only conventional technology is used.  For
example, using the conventional method of removing contaminants from
groundwater can involve pumping and treating the water for 30 years
or more.  In addition, no technology exists to address some cleanup
problems.  For example, no technology exists for some aspects of
removing and treating the radioactive waste now in large tanks at
several major DOE facilities.  Furthermore, some cleanup activities
could be dangerous or impossible for cleanup workers unless
innovative technologies, such as remote robotic devices to clean
inside radioactive waste tanks, are used. 

Those in the Congress and in DOE who led the effort to establish OST
believed that the use of innovative technology would reduce the cost
of waste cleanup.  For example, in 1995, DOE estimated that it would
cost between $200 billion and $350 billion and take another 75 years
to complete the cleanup.  However, DOE also estimated that the use of
new technologies could reduce cleanup costs by a minimum of $9
billion to as much as $80 billion, depending on the cleanup
scenario.\6 More recently, in 1997, the Army Corps of Engineers
reviewed cost savings estimates developed by OST for 37 of its
technology projects and concluded that these 37 projects could
potentially save about $20 billion over the use of conventional
technology.  DOE believes that cleanup costs could significantly
exceed current estimates if innovative technology is not used. 


--------------------
\6 Estimating the Cold War Mortgage:  The 1995 Baseline Environmental
Management Report (DOE/EM, Mar.  1995). 


   GAO HAS REPORTED IN THE PAST ON
   PROBLEMS IN MANAGING TECHNOLOGY
   DEVELOPMENT AND DEPLOYMENT
---------------------------------------------------------- Chapter 1:3

We have issued a number of reports and testified on the operation and
management of EM's technology program.  Among other things, we have
identified obstacles to the deployment of innovative technology at
DOE's cleanup sites.  In 1992, we reported that EM had not
established key management tools, such as cost estimates and
schedules, and decision points for evaluating technology projects.\7
In January 1993, EM implemented a management plan for the technology
program that incorporated our recommendations.  The program
established cost estimates and schedules for projects.  EM also
developed decision points (called gates) and related requirements for
evaluating projects and making "go/no-go" decisions. 

In 1994, we reported that officials at DOE cleanup sites may not be
familiar with innovative technologies and may fear that using new
technologies may lead DOE to miss cleanup deadlines if the technology
fails to perform as expected.\8 In response to our report, OST took
several actions, including establishing the site technology
coordination groups discussed earlier, to improve communication on
sites' technology needs and the capabilities of newly developed
technologies.  In addition, to help ensure that development
activities were concentrated on the most pressing cleanup needs, EM
restructured its technology development program into the focus areas. 

In 1996, we reported that EM had not coordinated technology
development to prevent duplication of effort, particularly between
OST and the Office of Waste Management, which together had 60
projects to develop equipment to melt and immobilize waste.\9 A key
reason for the duplication was EM's lack of a comprehensive list of
technology development projects.  EM subsequently developed a list of
its technology projects.  We also found that more technology projects
were being started at the sites where the focus areas were physically
headquartered.  In following up on this situation in 1997, we found
that this concentration had decreased. 

In 1997, we testified before the Subcommittee on Oversight and
Investigations, House Committee on Commerce, that OST appeared to
have made some improvements in its project management, but we had
continuing concern about the extent of use of OST-developed
technologies at DOE's waste cleanup sites and the validity of OST
data on deployments and expected cost savings.\10 OST had also
proposed a new initiative, Accelerated Site Technology Deployment
(ASTD),\11 to facilitate the use of its technologies.  We expressed
several concerns about the likely effectiveness of this initiative,
which provides funding to DOE sites for the first use of an
innovative technology.  OST provided a total of approximately $26
million to 14 ASTD projects in fiscal year 1998. 


--------------------
\7 Cleanup Technology:  Better Management for DOE's Technology
Development Program (GAO/RCED-92-145, Apr.  10, 1992). 

\8 Department of Energy:  Management Changes Needed to Expand Use of
Innovative Cleanup Technologies (GAO/RCED-94-205, Aug.  10, 1994). 

\9 Energy Management:  Technology Development Program Taking Action
to Address Problems (GAO/RCED-96-184, Jul.  9, 1996). 

\10 Cleanup Technology:  DOE's Program to Develop New Technologies
for Environmental Cleanup (GAO/T-RCED-97-161, May 7, 1997). 

\11 Formerly called the Technology Deployment Initiative. 


   OBJECTIVES, SCOPE, AND
   METHODOLOGY
---------------------------------------------------------- Chapter 1:4

The Chairman and Ranking Minority Member of the House Committee on
Commerce and the Chairman and Ranking Minority Member of its
Subcommittee on Oversight and Investigations asked us to review EM's
Office of Science and Technology.  Specifically, we were asked to
determine (1) to what extent innovative technologies developed by OST
have been deployed (used) at DOE sites and how this rate of
deployment compares with the rates of other government organizations
that develop environmental technologies; (2) what obstacles exist to
deploying innovative technologies at DOE sites; and (3) what EM is
doing to overcome obstacles to deploying innovative technologies. 

To determine the extent to which OST-developed technologies have been
deployed at DOE sites, we obtained deployment information on OST's
projects from an OST management information system.  This information
provided project names and numerical identifiers, research stage,
deployment sites (if any), and other project information, as of
January 1998.  We also obtained information about OST's use and
definition of the term "deployment" and OST's procedures for entering
and updating the information in this system.  In order to assess the
accuracy of OST's deployment data, we used a random sample of the
projects that OST listed as deployed, and we verified the claimed
deployments with site operations officials.  Upon finding a
significant error rate, we used our sample results to estimate a
range for the actual number of OST project deployments.  The
methodology for our verification is described in appendix I. 

To compare the rate of deployment for OST's technologies with the
rates of other government organizations that develop environmental
technologies, we used database searches and contacts with federal
agency officials to identify federal government programs that develop
environmental technologies.  We contacted the eight government
programs whose research and development work was most comparable to
OST's in mission and scope, and two of these programs--EPA's
Superfund Innovative Technology Evaluation Program and the Department
of Defense's Environmental Security Technology Certification
Program--were able to provide deployment data for comparison with
OST's data.  We also contacted two private sector organizations that
develop environmental technologies but found that they did not
maintain deployment data.  Because the two federal programs providing
deployment data conduct technology demonstrations but not earlier
phases of research and development, we identified OST projects that
had reached a similar stage of maturity to provide an equitable group
for comparison.  (See app.  II for a detailed discussion of these two
programs.)

To identify obstacles that exist to deploying innovative technologies
at DOE sites, we first reviewed past reports on this subject by GAO,
DOE, and advisory groups to DOE.  In order to obtain more current and
specific information about obstacles to deployment and EM's progress
in overcoming them, we visited five DOE sites:  Hanford (Washington
State), Savannah River (South Carolina), Oak Ridge (Tennessee),
Fernald (Ohio), and Lawrence Livermore National Laboratory
(California).  We selected these sites to provide varied
perspectives:  The first three sites are among the largest DOE
cleanup sites; Fernald is far along in its cleanup efforts and
represents a medium-sized cleanup effort; and Lawrence Livermore has
a smaller cleanup effort and budget.  For the site visits, we
identified OST-developed technologies that were either selected for
use at the site, considered for use but not selected, or potentially
applicable to the site's cleanup problems.  We identified
technologies to discuss with site officials from our meetings with
managers of OST's focus areas, records maintained by EM's Office of
Environmental Restoration, and discussions with headquarters EM
officials.  The technologies were judgmentally selected to provide
coverage of (1) EM's various cleanup challenges and the related OST
focus areas and (2) innovative technologies selected or not selected
for use. 

We discussed with EM field personnel and contractor staff the
obstacles they faced to using particular technologies, the ways they
addressed and overcame these obstacles (for deployed technologies),
and the reasons they did not select the technologies.  We discussed a
total of 30 OST-developed technologies with one or more of the five
sites and obtained documentation on related selection decisions.  We
analyzed this information to identify (1) commonly cited obstacles to
deployment and (2) the means by which sites overcame these obstacles
in those cases in which OST-developed technologies were selected or
in use. 

In order to identify EM's actions to overcome deployment obstacles,
we reviewed a memo from the Assistant Secretary for EM that directed
a number of actions to increase deployment, and we obtained
information about the status and results of these actions.  We also
interviewed OST managers to identify additional actions under way
within OST and obtained related documentation.  To assess the
adequacy of these actions, we compared the EM and OST actions to the
obstacles to deployment that we had identified. 

We provided a draft of this report to DOE for its review and comment,
and a draft of chapter 2 and appendix II to the Department of Defense
and EPA for their review and comment.  DOE's and the Department of
Defense's written comments and our responses are included in
appendixes III and IV. 

We performed our review from August 1997 through September 1998 in
accordance with generally accepted government auditing standards. 


EM HAS DEPLOYED LESS THAN 20
PERCENT OF OST'S PROJECTS
============================================================ Chapter 2

While OST has initiated 713 technology development projects, we
estimate that EM has deployed between 88 and 130 of these projects,
for an overall deployment rate of 12 to 18 percent.  In contrast, OST
has reported that 152 projects have been deployed one or more times,
for an overall deployment rate of 21 percent.  OST's overstated
deployment information is the result of several factors, including
its rapid compilation of deployment data in response to congressional
requests and the lack of a formal definition of what constitutes a
deployment. 

Most organizations we contacted, including some private technology
developers, did not track deployment data comparable to OST's.  We
contacted eight government programs and two private sector programs
engaged in environmental technology research and found only two that
could provide data on deployment.  In comparing data from these two
organizations and OST, we found that OST's deployment rate was close
to that of the 2-year-old Environmental Security Technology
Certification Program (ESTCP) in the Department of Defense and
somewhat lower than the rate for the 12-year-old Superfund Innovative
Technology Evaluation (SITE) program in EPA.  However, it is
important to recognize that the value of deployment rate comparisons
with other organizations is limited.  To assess the overall
performance of a research and development (R&D) program like OST,
other measures in addition to deployment would be relevant. 


   EM HAS DEPLOYED ABOUT 88 TO 130
   OF OST'S TECHNOLOGY PROJECTS
---------------------------------------------------------- Chapter 2:1

OST developed deployment data in response to a November 1996 request
from the Subcommittee on Oversight and Investigations, House
Committee on Commerce.  As of January 1998, OST's database showed
that EM had initiated 713 technology development projects since OST's
inception.  On the basis of our verification and analysis of these
data, using a 95-percent confidence interval, we estimate that EM
actually deployed from 88 to 130 projects, to achieve an overall
deployment rate of 12 to 18 percent for the 713 projects.\1 In
contrast, OST has reported that 152 of the 713 technology projects
initiated since its inception have been deployed.  Thus, according to
OST's data, about 1 in 5 OST technologies have been deployed one or
more times, for an overall deployment rate of 21 percent. 

We also found that OST had overstated the number of deployment
instances reported for each technology project.  OST's database
listed a total of 283 deployment instances\2 for the 152 projects
claimed as deployed.  We estimate that of the 283 deployment
instances claimed by OST, only 137 to 216 have actually occurred. 
Table 2.1 lists OST's data, the error rates we found, and our
estimates of actual deployments based on the error rate found in our
sample of 30 projects. 



                               Table 2.1
                
                Deployment of OST-Developed Technologies

                                                                 GAO's
                                                             estimates
                                                                    of
                                                            deployment
OST's claimed deployments          Error rate found by GAO         s\a
------------------------------  --------------------------  ----------
152 projects deployed                           27 percent   88 to 130
                                                              projects
                                                              deployed
283 deployment instances                        38 percent  137 to 216
                                                            deployment
                                                             instances
----------------------------------------------------------------------
\a Since we used only a sample of projects to make our estimates,
these estimates have a margin of error.  The range shown reflects
this margin of error at the 95-percent confidence level. 


--------------------
\1 To verify OST's deployment data, we contacted site personnel for a
randomly selected sample of the projects that OST claimed had been
deployed.  See app.  I for a detailed explanation of the confidence
interval and our verification methodology. 

\2 For example, OST reported four deployment instances for the Waste
Inspection Tomography technology project:  one use at Hanford, one at
Idaho National Environmental Engineering Laboratory, and two at
Lawrence Livermore National Laboratory.  This project thus counts as
1 of the 152 projects OST claimed as deployed and 4 of the 283
deployment instances. 


   SEVERAL FACTORS HAVE LED TO
   OST'S INACCURATE DEPLOYMENT
   DATA
---------------------------------------------------------- Chapter 2:2

OST's inaccurate deployment data resulted from several factors. 
Specifically, OST compiled deployment data quickly, in response to a
congressional request that came 7 years after the program's
inception, because it had not previously maintained comprehensive
data.  In addition, the lack of a formal definition for deployment
led to differing understandings among the focus area personnel
responsible for compiling the data.  Finally, OST has begun only
recently to establish procedures for entering and updating project
data.  If such procedures had been in place early on, they would have
uncovered the need to formalize the definition of deployment. 


      INITIAL COMPILATION OF
      DEPLOYMENT DATA
-------------------------------------------------------- Chapter 2:2.1

Some inaccuracy in OST's deployment data may have been due to the
fact that the data were compiled quickly.  OST prepared deployment
data in response to a November 1996 request from the Chairman of the
House Committee on Commerce.  Previously, OST had not maintained
comprehensive deployment data on its projects.  Instead, OST tended
to focus its performance measures on completed demonstrations.  For
the November 1996 request, OST gathered deployment data for its
projects over a period of several months and provided the information
to the Chairman in April 1997.  At that time, OST reported that 150
projects had been deployed and an additional 41 projects had been
selected for use in the future (for a total of 191 past and future
deployments). 


      LACK OF A FORMAL DEFINITION
      FOR DEPLOYMENT
-------------------------------------------------------- Chapter 2:2.2

Another reason for OST's inaccurate data has been the lack of a
formal definition of deployment, leading to different understandings
among the focus area personnel who collected the deployment data
about what should be counted as a deployment.  According to OST
managers, while gathering data to respond to the Committee's request,
OST headquarters officials told focus area personnel to refer to an
earlier definition of implementation for the meaning of deployment
but did not distribute new written guidance.  This definition, which
OST had formalized and distributed in April 1996, defined
implementation to mean that the technology was used or selected for
use to meet specified user performance measures (e.g., completion of
an assessment or treatment of waste for disposal).  However,
officials of the Subsurface Contaminants Focus Area provided us with
a definition of deployment that they received along with the
instructions for responding to the Committee's data request.  This
definition stated that the number of deployments means the number of
"hot" demonstrations (that is, demonstrations in radioactive
environments) and that deployment site means the location of a hot
demonstration. 

We found that OST focus area personnel entering the data frequently
regarded demonstrations as deployments.  For example, OST counted as
a deployment the use of a characterization technology called Laser
Ablation/Mass Spectroscopy at the Pacific Northwest National
Laboratory in Washington State.  In response to our questions, site
contractor officials stated the technology's use at the laboratory
had been a demonstration in which data derived from the laser
technology were compared with data derived from a conventional
technology.  At this time, the site cannot rely upon the laser
technology to accomplish its goals for characterization. 


      STATUS OF OST EFFORTS TO
      IMPROVE DATA QUALITY
-------------------------------------------------------- Chapter 2:2.3

While OST has issued a definition of deployment and is taking other
steps to improve data quality, written procedures for data
verification have not yet been developed.  In August 1998, the Acting
Deputy Assistant Secretary for OST issued a memo that formally
defined deployment.  The definition appropriately emphasized that
deployments must accomplish site objectives, such as the completion
of assessments, cleanups, or the treatment and disposal of wastes. 
The memo stated that this definition is to be used for performance
measurement.  OST has also completed a data verification effort for
those projects considered deployed during fiscal year 1997.  It used
verification by site personnel and other data sources to improve the
accuracy of this portion of the data. 

According to OST officials, the office intends to continue similar
verification efforts in the future.  However, these data verification
plans are not reflected in OST's draft procedures for its database,
which do not specify a method of data verification.  The procedures,
drafted in January 1998, identify OST's focus areas as responsible
for entering data and ensuring their quality and completeness.  These
procedures also require that the data on ongoing projects be updated
at least once per quarter (every 3 months).  However, the draft
procedures do not identify any means by which the data are to be
verified or spot-checked for accuracy.  While site technology
coordination groups can comment on the deployments listed in the
database, the procedures do not state any requirement for data review
and concurrence by these groups or other site officials.  The Acting
Deputy Assistant Secretary for OST told us that OST plans to obtain
further advice about verification methods and then develop written
procedures. 

OST has not yet determined whether, or to what extent, to verify data
from the years prior to 1997 because of the time and resources
involved.  According to the Acting Deputy Assistant Secretary for
OST, the office is seeking clarification from the House Committee on
Commerce on the degree of accuracy or certainty needed. 


   DEPLOYMENT RATE FOR OST
   TECHNOLOGIES FALLS BETWEEN THE
   RATES OF TWO OTHER
   ENVIRONMENTAL TECHNOLOGY
   PROGRAMS
---------------------------------------------------------- Chapter 2:3

We compared the deployment rate for OST's technologies with the
deployment rates for technologies sponsored by EPA's SITE program and
the Department of Defense's ESTCP.  The SITE program is engaged
solely in the environmental technology demonstration and
implementation stages of R&D.  Similarly, ESTCP demonstrates and
validates technologies and funds environmental technologies that have
progressed to the stage at which field demonstrations are warranted. 
Taking into account the limitations of this comparison, OST's
deployment rate for projects at comparable stages of development
falls between the rates of the two organizations that provided data,
as shown in table 2.2.  (App.  II discusses in detail how we
developed each comparison.)



                               Table 2.2
                
                OST's Deployment Rate Compared With the
                Rates of Other Environmental Technology
                         Demonstration Programs

                                            Comparison
Other demonstration programs                      rate      OST rate\a
--------------------------------------  --------------  --------------
Superfund Innovative Technology                    59%       28 to 45%
 Evaluation program
Environmental Security Technology                  38%       28 to 40%
 Certification Program\b
----------------------------------------------------------------------
\a OST deployment rates based on projects at comparable stages of
maturity. 

\b ESTCP data based on the program's first 2 years of operation. 

Source:  GAO's analyses of data provided by the agency programs. 


   DEPLOYMENT RATE COMPARISONS
   BETWEEN OST AND OTHER
   ENVIRONMENTAL TECHNOLOGY
   DEVELOPERS MAY HAVE LIMITED
   USEFULNESS
---------------------------------------------------------- Chapter 2:4

Comparisons of OST's deployment rate with the rates of other
organizations must be viewed with caution when assessing how well EM
is doing in deploying OST-developed technologies.  We found few
organizations that engage in the range of environmental research OST
performs, and no organization we contacted routinely tracked
deployment data on its projects.  Data provided by the two
organizations differed widely in source and composition.  Finally,
many individuals we contacted question whether a deployment rate is a
sufficient benchmark for successful R&D. 


      A SMALL NUMBER OF
      ENVIRONMENTAL TECHNOLOGY
      DEVELOPMENT PROGRAMS
      MAINTAIN DEPLOYMENT DATA
-------------------------------------------------------- Chapter 2:4.1

Most organizations we contacted, including some private technology
developers, did not track deployment data comparable to OST's.  Of
the eight government programs and two private sector programs engaged
in environmental technology research we contacted, only the SITE
program and ESTCP could provide data on deployment.  Even these two
programs needed to compile their information so that it could be
expressed as deployment rates.  Table 2.3 shows the entities that we
contacted. 



                               Table 2.3
                
                  Environmental Technology Development
                           Entities Contacted

Entities                                  Program
----------------------------------------  ----------------------------
Environmental Protection Agency           Superfund Innovative
                                          Technology Evaluation
                                          Program

Department of Defense                     Environmental Security
                                          Technology Certification
                                          Program

                                          Strategic Environmental
                                          Research and Development
                                          Program

                                          Chief of Naval Operations;
                                          Environmental Research,
                                          Development, Test and
                                          Evaluation Program

                                          U.S. Army Corps of Engineers
                                          Waterways Experiment
                                          Station; Environmental
                                          Laboratory

                                          U.S. Air Force Research
                                          Laboratory; Air Base and
                                          Environmental Technology
                                          Division

                                          Air Force Center for
                                          Environmental Excellence

                                          U.S. Air Force; Environment,
                                          Safety and Occupational
                                          Health--Technical Planning
                                          Integrated Product Team

Private sector organizations              Petroleum Environmental
                                          Research Forum

                                          DuPont Corporate Remediation
                                          Group
----------------------------------------------------------------------
Furthermore, we found that only one of the other government programs
listed in table 2.3 engaged in nearly the full range of environmental
R&D that OST performs.  OST's R&D includes basic science research,
applied research and engineering development, field testing and
demonstration, and implementation by the end user
(commercialization).  Most of the governmental organizations we
contacted performed either the early stages of R&D or the later
stages, but not both.  Technology development efforts undertaken at
the early stages have more unknowns and are likely to involve a
greater risk of failure than efforts at the later stages.  Since we
would expect performance results to differ for each stage, meaningful
comparisons can only be made among projects or programs that are at
similar stages of R&D maturity. 

Two organizations provided us with very different types of data. 
EPA's SITE program had accumulated survey data on the number of
contracts their technology vendors had obtained over about 8 years. 
We agreed that a contract for use could be considered deployment of
the technology.  As to be expected, the survey responses were less
than 100 percent, unlike the OST and ESTCP data, which include all of
these agencies' technology projects.  Therefore, the data from EPA's
SITE program are incomplete, and the deployment rate for SITE could
actually be higher.  The Department of Defense's ESTCP provided a
description of the transition (deployment) status for all of its
projects from the program's first 2 years of existence.  Since ESTCP
is a relatively new program, its deployment data are based on a
limited number of projects and may be less representative of the
program's future performance.  We did not verify the accuracy of
these organizations' deployment data, but we reviewed their available
project summaries and believe the organizations' approaches were
reasonable responses to our request.  Nevertheless, differences in
how the programs defined deployment, and whether they counted
incomplete projects, will affect computed rates. 


      DEPLOYMENT RATE MAY BE AN
      INCOMPLETE MEASURE OF OST'S
      PERFORMANCE
-------------------------------------------------------- Chapter 2:4.2

As we have previously reported, measuring the performance of R&D
programs is difficult.\3 Performance measures used in other federal
R&D programs include the scientific peer review of projects, numbers
of patents issued, and studies of publications.  Recent R&D
management literature suggests that certain measures, such as the
number of patents issued, are best suited to earlier stages of
research, while outcome measures, such as deliverables and customer
satisfaction, are more relevant for later-stage research.  In this
context, a deployment rate measure would be most useful when applied
to more mature projects.  At the same time, program managers need to
assess how successful the program has been at selecting early-stage
projects with high potential for future payoff. 

Officials in a number of programs we contacted told us that
deployment has only recently been raised as a possible performance
measure.  Furthermore, programs performing earlier stages of R&D were
less likely to have any deployment data.  Developers of later-stage
technologies believed that the deployment rate is an incomplete
performance measure, and that cost savings or some measure of dollar
impact should also be used to evaluate program success.  EM is
considering developing a performance measure that would assess cost
savings from the use of innovative technologies. 

EM and OST recognize that deployment is not the only relevant measure
of success in technology development.  We reviewed performance
measures established for OST for fiscal years 1994 through 1997 and
found that completing demonstrations of technologies and the number
of technologies made available for use--that is, number that have
completed development--were the main performance measures used.  In
fiscal year 1998, OST's performance measures are (1) demonstrate 35
new technologies, (2) make 40 alternative technologies available for
use with cost and engineering data, and (3) perform 49 deployments of
new technologies.  As described in more detail in chapter 4,
performance measures for deploying innovative technologies are also
being applied to EM's field operations offices in fiscal year 1998,
and OST is considering developing additional performance measures for
its focus areas that address technologies in various stages of
development. 


--------------------
\3 Measuring Performance:  Strengths and Limitations of Research
Indicators (GAO/RCED-97-91, Mar.  21, 1997). 


SOME OBSTACLES CONTINUE TO SLOW
DEPLOYMENT OF INNOVATIVE
TECHNOLOGIES
============================================================ Chapter 3

As EM's cleanup program has matured, several of the obstacles to
using innovative technologies reported previously by us, EM, and
others have been addressed.  For instance, DOE sites and their
regulators have improved their working relationships, and, in cases
where innovative technologies were selected, DOE sites have found
ways to address regulator concerns about whether these technologies
will achieve required objectives.  However, some obstacles, internal
to EM and OST program operations, continue to slow the deployment of
innovative technologies, and, in some cases, have led OST to spend
millions of dollars for technologies that the cleanup sites do not
want.  The most significant and continuing of these internal
obstacles has been EM's and OST's failure to involve users
sufficiently in the design and development of technology targeted for
use at the cleanup sites.  As a result, OST has developed generic
technologies that do not meet site-specific needs or that require
modification to make them usable by the site.  However, EM has not
clearly defined responsibilities and funding sources for modifying
technologies among OST and potential technology users.  Furthermore,
OST still has no clearly defined role in helping sites select the
appropriate technology and infrequently provided technical assistance
in the cases we reviewed. 

Several factors contribute to these problems.  First, prior to 1996,
OST had not comprehensively assessed users' technology needs and
linked these needs with technology development efforts.  Second, OST
has not fully implemented its system for monitoring, and if
necessary, modifying or terminating ongoing technology development
projects--a system that would require interaction with technology
users. 


   SITES HAVE OVERCOME OBSTACLES
   TO USING SOME INNOVATIVE
   TECHNOLOGIES
---------------------------------------------------------- Chapter 3:1

DOE's field and contractor staff face a number of challenges when
attempting to use an innovative cleanup technology.  Past reports by
us, EM, and advisory groups have catalogued the challenges:  the
perceived risks of exceeding projected costs or failing to meet time
schedules; the need to convince regulators and stakeholders of the
advantages of innovative technology; and technical problems,
including the need to modify a technology to make it fit a specific
situation.  However, as the EM technology program has matured and
site personnel, regulators, and stakeholders have become more aware
of the benefits of using some innovative technologies, some obstacles
have diminished in importance.  Furthermore, when the use of a new
technology is clearly and significantly advantageous, cleanup sites
make a strong effort to overcome any obstacles to its use. 

Specifically, when regulators and stakeholders are concerned about a
technology's effectiveness, sites have provided additional data or
testing and, occasionally, modified technology to satisfy some
concern or implemented an innovative technology in phases to obtain
performance data.  For example, according to Hanford officials, using
a new technology to encapsulate certain carbon-based waste would be
much less costly than incinerating it.  State regulations, however,
called for incinerating such waste.  Nonetheless, Hanford persisted
and obtained a waiver from the state to encapsulate the waste.  At
Oak Ridge, DOE and its contractors wanted to use a frozen soil
barrier to contain a relatively small pool of water that had been
contaminated with reactor waste.  However, regulators and
stakeholders were skeptical that this innovative technology would
work and be cost effective.  Oak Ridge demonstrated the technology to
obtain cost and performance data and provided this information to
regulators and stakeholders.  The technology has since gained wide
acceptance by these groups. 

Some technology may have to be modified to satisfy regulatory
concerns.  For example, Hanford officials wanted to test an
innovative technology for cleaning up contaminated soil, which they
believed was better than current methods.  However, regulators were
concerned about the possible expulsion of carbon tetrachloride
contaminants into the air.  Hanford officials convinced the
regulators to allow them to experiment with the new technology by
offering to add a filter to the equipment to catch any contaminants. 
The modification was a low-cost and easy addition to the equipment. 

In some cases, sites implement a technology in phases to obtain
performance data and to assure themselves and convince regulators and
stakeholders of the technology's viability.  For example, OST funded
the development of a robotics device, called Houdini, that could help
clean up waste in tanks.  Oak Ridge, with the help of the
manufacturer, adapted Houdini to help clean up radioactive waste
stored at the bottom of the site's large tanks.  However, because
Houdini had never been used to clean up radioactive waste, no
information was available on the device's performance and
reliability.  Oak Ridge therefore had to implement Houdini in phases
using nonradioactive "cold" testing; followed by treatability tests
in a lower radiation environment; and finally, "hot" testing on the
radioactive waste in its tanks. 

Field officials also told us that the projects in which OST and an EM
operating group get involved as a joint venture seem to work well. 
In these cases, OST provides funding and some technical assistance,
and the operating group also provides funding and implements the
project.  If there are also partners from industry, they further
enhance the chances for success.  For example, at Hanford, OST and
EM's environmental restoration group are participating in a large
project to demonstrate a number of technologies that can be used to
put Hanford's old, shutdown reactors into safe interim storage. 
Hanford officials were convinced that if the demonstrated
technologies were successful, the time needed to prepare the reactors
for storage could be cut by 7 years. 

The demonstration project started in 1996, with contributions
totaling about $8 million from OST and about $16 million from the
environmental restoration group.  However, the project did not have
the extra money to make needed refinements and modifications to
technologies.  Consequently, Hanford officials suggested partnering
with private contractors who would assume the risk and cost of
getting the technologies to perform.  OST's Deactivation and
Decommissioning Focus Area, which routinely works with the private
sector, helped to bring about this partnership with private
contractors.  As of July 1998, the project had successfully
demonstrated 20 technologies and deployed 13 of them at Hanford's C
Reactor, two other Hanford reactors, and a number of other DOE
reactors throughout the complex.  In addition, the technologies have
been transferred to the commercial reactor sector and will be used to
help put the nuclear power plant in Chernobyl, Ukraine, into safe
storage. 


   OST HAS DEVELOPED INNOVATIVE
   TECHNOLOGIES WITHOUT SUFFICIENT
   USER INVOLVEMENT
---------------------------------------------------------- Chapter 3:2

Despite the progress that has been made, some obstacles internal to
EM and OST operations continue to slow the deployment of innovative
technologies.  In particular, OST has developed technologies that
tend to be generic solutions to cleanup problems and, if usable at
all, have to be modified to fit a site's specific problem.  These
problems occur in part because OST had not, until 1996,
comprehensively assessed the technology needs of the cleanup sites
and has not involved potential technology users in the development of
technology that might be used to address specific cleanup problems. 
Without user involvement, there have been no identified customers for
some of the technology that OST has sponsored.  For example, of the
107 technologies that OST has completed, 31 technologies, costing $71
million to develop, have not been used by cleanup sites.\1


--------------------
\1 From OST data as of Jan.  1998, unadjusted for errors. 


      CLEANUP SITES CHARGE THAT
      TECHNOLOGY DOES NOT MEET
      THEIR NEEDS
-------------------------------------------------------- Chapter 3:2.1

According to EM field and contractor personnel responsible for waste
cleanup, in many cases, OST technologies do not meet their needs. 
They said that OST has many times assumed that "one-size-fits-all"
and therefore has developed generic solutions to cleanup problems. 
However, these solutions either do not fit a site's specific needs or
must be modified before they can be used.  For example, Fernald
workers needed portable equipment that would allow them to
characterize contamination within buildings without climbing ladders
to obtain samples from contaminated areas.  OST said that, although
its laser-induced fluorescence imaging equipment had not been
field-tested, the equipment had been designed to meet needs such as
Fernald's.  However, when Fernald workers attempted to use the
equipment, they found that it was not ready for field use.  It was
cumbersome (not really "portable") and light interfered with
measurement readings.  As a result, the equipment was not usable and
was returned to the manufacturer for modifications.  Consequently,
Fernald personnel continued to take samples from the contaminated
building areas by hand.  Although they realized that the OST
equipment had not been thoroughly tested before they tried it,
Fernald officials said, they believed that if OST had involved them
in the design and development of the equipment, the problems would
have been avoided, or at least identified and corrected earlier. 

Similarly, officials at DOE's Hanford site tried two OST technologies
that promised to support faster remediation of contaminated soil but
had to reject them because they were not designed to work in
Hanford's arid soil.  The officials said that the concept for faster
remediation of contaminated soil was attractive and probably would
have been acceptable to Hanford's regulators, but the generic design
of the technologies did not meet Hanford's specific needs. 

Furthermore, some site officials said that they would like to use
some OST technologies, but the technologies require modification to
fit the site's situation.  They pointed out that it is not clear who
should make and pay for these modifications.  For example, a project
manager at DOE's Savannah River Site told us that he would like to
use more innovative technology in his projects, but it is unclear who
is responsible for making site-specific modifications, and his
program does not have funding to make such modifications.  At
Hanford, officials were interested in using OST's Electrical
Resistance Tomography to help detect leaks in their high-level
radioactive waste tanks.  (Hanford has 67 known or suspected leaking
tanks.) However, a Hanford official said that the technology needed
substantial fine-tuning to make it work on the Hanford tanks, and no
funding was available to pay for this.  He said that it was unclear
who is responsible for funding modifications to OST technologies. 

When only minor, inexpensive modifications are required, site
representatives said that they have made and usually paid for them. 
But other technologies that are of interest to sites would require
more extensive and more expensive modifications.  Without a clear
policy on who is responsible for modifying the technology and paying
for the modification, the sites are likely to reject the innovative
technology and select a known alternative. 


      USER NEEDS FOR TECHNOLOGY
      NOT FULLY ASSESSED UNTIL
      RECENTLY
-------------------------------------------------------- Chapter 3:2.2

Until its reorganization in 1994, OST did not involve the cleanup
sites in identifying technologies that need to be developed and did
not conduct comprehensive needs assessments until 1996 and 1997. 
Therefore, most of the technologies developed through OST were not
based on a comprehensive assessment of the technology needs of those
responsible for cleaning up DOE waste sites.  Instead, OST consulted
with its developers at the national laboratories in deciding which
technologies it would sponsor to solve sites' cleanup problems. 
These technical solutions, according to potential technology users,
tend toward the "one-size-fits-all" development philosophy. 

We reported in 1994 that technology needs had not been
comprehensively identified to allow prudent research decisions nor
had various environmental program offices in headquarters and in the
field worked together effectively to identify and evaluate all of the
possible technology solutions available.\2 In 1995, and again in
1996, the Environmental Management Advisory Board (EMAB) told the
Assistant Secretary of Environmental Management that the lack of a
comprehensive assessment linking identified needs with technology
development efforts was a "primary barrier" to technology deployment. 
EMAB said that technology development and deployment must be linked
together as a single system. 

Site technology coordination groups, established in 1994, made early
attempts to assess the needs of potential technology users.  However,
because OST considered data from these early surveys unreliable, it
and the site groups developed guidance and worksheets for a more
comprehensive assessment, which the site technology coordination
groups carried out in October 1996.  In October 1997, an updated
needs assessment and a database that matches technology needs with
appropriate existing technology or the future efforts of technology
developers was completed, according to the director of OST's Office
of Technology Systems. 


--------------------
\2 Department of Energy:  Management Changes Needed to Expand Use of
Innovative Cleanup Technologies (GAO/RCED-94-205, Aug.  10, 1994). 


      OST HAS NOT INVOLVED USERS
      IN TECHNOLOGY DEVELOPMENT
-------------------------------------------------------- Chapter 3:2.3

In addition to not involving the cleanup sites in identifying
technology needs, OST has not sufficiently involved users in
designing technologies and monitoring their development to help
ensure that they meet users' needs.  In 1992, we recommended that EM
institute a technology development management system with explicit
decision points at which the technology would be assessed to
determine whether development should continue or be terminated.  OST
established its "Technology Investment Decision Model" (called the
"gates system") to do this.  The gates system satisfies our 1992
recommendation and was intended to be "a user-oriented
decision-making process for managing technology development and for
linking technology-development activities with cleanup operations."
However, OST has not fully implemented the gates system and thus
cannot be certain that appropriate technology is developed to meet
the needs of DOE's cleanup sites. 

Under OST's gates system, the focus areas are to assess a
technology's development at six stages, from basic research through
implementation.  At each stage, the focus area is to make a go/no-go
decision, with input from potential users.  The critical decision
points include the following: 

  -- Gate 1:  Entrance Into Applied Research Stage.  To pass through
     gate 1 and enter this stage, a proposed technology must be shown
     to address national interests and priority environmental needs. 
     EM guidance states that if a technology does not address a
     specific need, it should not pass through gate 1. 

  -- Gate 2:  Entrance Into Exploratory Development Stage.  To pass
     through gate 2 and enter this stage, a technology has to be
     linked with the specific needs of an identified user. 

  -- Gate 3:  Entrance Into Advanced Development Stage.  To pass
     through gate 3 and enter this stage, the technology must be able
     to meet an identified user's specific performance requirements. 
     In addition, it must be documented that the research to develop
     the technology is expected to produce results consistent with
     the user's time frame for deployment and implementation. 

  -- Gate 4:  Entrance Into Engineering Development Stage.  To pass
     through gate 4 and enter this stage, the technology must be
     shown to meet the user's specific needs in a timely manner.  In
     addition, it must be documented that the proposed innovative
     technology will be more cost-effective than current methods or
     other emerging technology. 

  -- Gate 5:  Entrance Into Demonstration Stage.  To pass through
     gate 5 and enter this stage, the identified user for the
     technology must make a commitment to deploy the technology if it
     meets performance requirements.  In addition, the user must
     agree to share in the cost of and the responsibility for
     demonstrating the technology. 

  -- Gate 6:  Entrance Into Implementation Stage.  To pass through
     gate 6 into implementation, the technology must successfully
     complete a "real world" demonstration, either at a DOE site or
     another location, using actual waste streams and/or anticipated
     operating conditions.  In addition, it must be documented that
     the technology has proven to be viable, cost-effective, and
     applicable to the users' needs. 

As this discussion of the gates system shows, OST's focus areas must
identify a user for the technology in the early stages of
development.  Furthermore, this user must stay involved throughout
the development process to ensure that the technology will meet the
needs and implementation schedule of the user. 

OST, however, has not fully implemented its gates system to involve
potential users in the assessment of technology that it is
developing, and, in some cases, OST has not identified an end user
for the technology.  Furthermore, a review by EM and EMAB
representatives, completed in late 1997, revealed that OST's focus
areas do not consistently use the gates system and do not
consistently involve potential technology users in technology
development decisions. 

EMAB has pointed out in numerous reports that OST has failed to
rigorously apply the gates system.  EMAB has stated that OST should
use the gates system to identify and terminate technologies that have
no identified customer, are not cost-effective, or have limitations
that may increase the risk of failure when used.  According to the
Chairman of EMAB's Committee on Technology Development and Transfer,
OST officials told him that they did not rigorously apply the gates
system because it yielded results that OST and technology developers
at the laboratories did not like--that is, indicating that some
technology projects should be terminated. 

Similarly, representatives of one of OST's focus areas told us that
OST does not rigorously use the gates system because it would force
OST to terminate technologies that have no identified customer, do
not meet users' needs, are technically limited, or have some other
fault.  The manager of the Subsurface Contaminants Focus Area told us
that his focus area had rigorously applied the gates system and
terminated some technologies, which led to a confrontation with the
laboratories developing the technologies. 

The Director of OST's Office of Technology Systems told us that the
gates system was never fully implemented because staff were confused
by other evaluations and OST's reorganization into focus areas, which
were taking place at the time the gates system was instituted.  He
said that the gates system was currently not being used but would be
reinstituted in the future.  According to the Acting Deputy Assistant
Secretary of OST, the criteria of the gates system are still valid,
but when focus areas tried to use the gates system, their approach
was inappropriate and did not work.  Specifically, he stated that
focus areas set up panels to periodically review their projects
according to the gate criteria.  Instead, the gates system was
intended to be used on an ongoing basis, so that the focus areas
could determine whether requirements for the various stages of
technology development, including user involvement, had been met. 
According to this official, OST's intent was that technology
developers and technology users have frequent interaction. 


   OST HAS NOT PROVIDED SOME
   NEEDED TECHNICAL ASSISTANCE
---------------------------------------------------------- Chapter 3:3

OST has not fulfilled its role of providing technology users with the
technical advice and assistance that they need to identify solutions
to cleanup problems and to help implement those solutions.  Focus
areas' ability to provide technical help varies widely, although this
was a principal mission when these groups were established in 1994. 
Some site officials responsible for cleanup told us that they are
reluctant to try new technologies without a reliable source for
advice and assistance, but some are reluctant to seek help from the
focus areas because they do not trust the focus areas' abilities. 


      FOCUS AREAS' CAPABILITY TO
      PROVIDE TECHNICAL ASSISTANCE
      VARIES
-------------------------------------------------------- Chapter 3:3.1

EM established the focus areas in part to provide technology users
with technical advice and assistance.  However, EMAB has consistently
noted the lack of technical knowledge in some focus areas and
suggested that this problem be addressed.  Similarly, we found that
cleanup sites are skeptical of the technical expertise of some focus
areas and rarely call upon them for assistance. 

EMAB believes that the focus areas need to become experts not only in
OST-sponsored technology but also in other domestic and foreign
technology that might help solve waste cleanup problems.  EMAB
reported in January 1998 that some focus areas do not know the
state-of-the-art technology for their area.  The Chairman of EMAB's
Committee on Technology Development and Transfer told us that the
Tanks Focus Area and the Deactivation and Decommissioning Focus Area
seem capable, but he said that EMAB is concerned about the capability
of the Subsurface Contaminants Focus Area, which has the largest
workload by type of waste problem. 

During our visits to five cleanup sites, we found that the sites
infrequently sought technical assistance from OST and its focus
areas.  Site officials said that technical assistance would be
helpful in deploying new technologies, but some are not convinced of
the focus areas' technical expertise.  Furthermore, they preferred to
go directly to a vendor for technical assistance because the vendor
was much more knowledgeable than OST. 


      SITES ARE RELUCTANT TO ALLOW
      OST A ROLE IN MAJOR
      TECHNOLOGY DECISIONS
-------------------------------------------------------- Chapter 3:3.2

In 1994, we recommended that OST be given a formal role in sites'
selections of technologies to solve cleanup problems.  For example,
OST could formally take part in sites' feasibility studies to
identify and analyze technologies that could potentially solve a
specific waste cleanup problem and to help a site decide which
technology to use.  However, some site officials told us that OST and
its focus areas are not familiar enough with their sites' waste
cleanup problems and appropriate solutions.  They said that our
recommendation was not taken because site officials are skeptical of
OST's ability to provide quality technical advice and assistance and
therefore are reluctant to allow OST more of a role in selecting
cleanup technologies for their sites. 

The Acting Deputy Assistant Secretary for OST told us that he is
aware of this problem and has directed the focus areas to become more
technically competent and supportive.  He said that providing
technical assistance should be routine for the focus areas; they
should be out in the field providing this help, not waiting in the
office for the sites to call them.  He emphasized that if the focus
areas are not able to provide expert technical assistance, he will
look to other groups, perhaps the national laboratories, to provide
needed technical assistance. 


EM'S EFFORTS TO INCREASE
DEPLOYMENT DO NOT FULLY ADDRESS
COOPERATION BETWEEN TECHNOLOGY
DEVELOPERS AND USERS
============================================================ Chapter 4

EM management devoted little attention to the deployment of
innovative technologies until a congressional oversight hearing in
May 1997 criticized EM's performance in deploying technology. 
Following the hearing, the Assistant Secretary of EM issued a
memorandum in July 1997, directing OST and other EM offices to
initiate specified actions designed to facilitate technology
deployment.  Some of these actions have already been completed, and
the remainder were to be completed by September 30, 1998.  These
actions establish responsibilities, require the development of
performance measures for technology deployment, establish the
Technology Acceleration Committee of upper-level EM and field
managers, require sites to develop deployment plans, and continue the
Accelerated Site Technology Deployment program that funds individual
projects.  OST has additional initiatives under way, including
establishing technology-user steering committees and developing
multiyear plans for technology development. 

However, EM's efforts only partially address the internal obstacles
limiting deployment.  On the positive side, EM has established
deployment performance measures for field sites and required sites to
develop deployment plans.  Users' involvement in developing overall
plans and priorities for OST's work is also improving.  On the other
hand, although the initiatives provided for upper management
attention through the Technology Acceleration Committee, the future
of this Committee is uncertain because of the departure of EM's
Assistant Secretary, who established it.  According to EM officials,
a broader executive committee addressing EM issues may take its
place.  EM did not carry out its plans to include deployment in the
annual performance expectations of its senior managers, considering
their membership in the then-active Technology Acceleration Committee
to be sufficient to hold managers accountable.  In addition, EM has
not yet improved developer-user cooperation for individual projects. 
Specifically, EM's initiatives do not require OST to use its existing
decision process for technology development (the gates system), which
would require user involvement at various stages in the development
process.  Furthermore, EM has yet to determine how it will provide
deployment assistance to cleanup sites to (1) more routinely provide
technical assistance in selecting and implementing innovative
technologies and (2) make modifications to completed technologies to
better meet sites' needs when it is cost-effective to do so. 


   EM HAS ESTABLISHED INITIAL
   DEPLOYMENT PERFORMANCE MEASURES
   FOR FIELD SITES AND IS
   DEVELOPING ADDITIONAL MEASURES
---------------------------------------------------------- Chapter 4:1

In a July 1997 memo, EM's Assistant Secretary stated that technology
deployment is the responsibility of all senior EM management,
including the managers of EM's operating groups, OST, and field
offices.  EM management had not previously emphasized technology
deployment, and this was the first formal assignment of
responsibility for deployment.  The Assistant Secretary also directed
that performance measures based on technology deployment be
established for those groups involved with deployment efforts and be
included in the performance expectations for senior managers.  In
response, EM has instituted or is planning performance measures
addressing the deployment of innovative technologies at several
levels:  (1) DOE field sites undergoing cleanup, (2) contractors that
manage the DOE field sites, and (3) OST and its focus areas.  Field
sites were also required to submit deployment plans addressing both
their overall approach to utilizing innovative technologies and their
plans to achieve deployments in specific cleanup projects.  EM
continues to refine its performance measures and has asked EMAB for
advice about improving performance measures at the various levels to
help increase deployment. 


      PERFORMANCE MEASURES AND
      DEPLOYMENT PLANS FOR FIELD
      SITES
-------------------------------------------------------- Chapter 4:1.1

In responding to our written inquiry to EM management in March 1998,
the Acting Deputy Assistant Secretary for OST stated that "in
analyzing the most appropriate and optimum way" to accelerate
technology deployment, EM management concluded that deployment goals
can best be achieved by holding those at the point of implementation
of new technology--the field sites--responsible for deployment.  EM
has established two indicators to measure field sites' efforts to use
innovative technology to clean up waste sites:  (1) the number of
technologies deployed annually and (2) life-cycle cost savings
resulting from the use of innovative technology.  For the present,
annual targets for the number of deployments are based on the amount
of annual EM funding a site receives.  EM established a target that
requires field offices to agree to deploy one new technology for
every $100 million in annual funding that they receive.  For example,
DOE's Oak Ridge site will receive about $600 million in EM funding in
fiscal year 1998 and is therefore expected to use six new
technologies a year in its effort to clean up nuclear waste.  For
fiscal year 1998, field sites have agreed to deploy a total of 49 new
technologies, which can be from OST or other sources.  OST believes
that the majority of these new technologies will be ones that it has
sponsored. 

Field sites must also submit site-specific deployment plans for
innovative technologies.  The plans, most of which were submitted in
May and June 1998,\1

describe the sites' overall approaches to deploying innovative
technologies, such as processes for identifying deployment
opportunities and involving regulators.  The plans also specify
opportunities to deploy innovative technologies in the sites' cleanup
projects.  For instance, the plans describe the schedule for
technology deployments, projected benefits from using the
technologies, and funding requirements. 

In the future, EM may establish performance targets for field sites
that are based on the amount of savings that would be produced by
using innovative, rather than conventional, technology over the life
of a project.  These measures were not established in fiscal year
1998 because EM lacked a standard methodology for calculating cost
savings.  However, in March 1998, EM completed a draft of a
standardized process for calculating these savings. 


--------------------
\1 One of the sites required to develop a deployment plan--Savannah
River--submitted a draft plan in July 1998 and had not developed a
final plan at the time of our review. 


      PERFORMANCE MEASURES FOR
      DOE'S CONTRACTORS
-------------------------------------------------------- Chapter 4:1.2

The need for contract incentives for the use of innovative
technologies has been broadly recognized by EM managers in
headquarters and the field.  Each of the five sites that we visited
had used performance measures addressing deployment for the site's
management contractor.  Some sites have experimented with different
approaches to determine which measures work best.  For example, at
Savannah River, DOE tried performance-based incentives for its
contractor in 1995 and 1996 that were based on the number of
innovative technologies used and the associated cost savings; then,
in 1997 and 1998, it switched to incentives based on the cost savings
achieved--regardless of whether conventional or innovative
technologies were used.  According to DOE's Assistant Manager for
Environmental Quality at Savannah River, over half the cost savings
that the contractor achieved in environmental restoration in 1997
came from the use of innovative technologies, and he believes that
the cost savings measure has worked the best in providing incentives
for using innovative technologies.  At Lawrence Livermore National
Laboratory, which participates in a number of OST technology
projects, the contractor's performance measures address both using
innovative technologies in the laboratory's cleanup activities and
supporting their use at other sites. 


      PERFORMANCE MEASURES FOR OST
-------------------------------------------------------- Chapter 4:1.3

OST's performance will also be measured on the basis of technology
development and related deployment.  For example, OST's performance
goals for fiscal year 1998 include demonstrating 35 new technologies;
finishing the development of 40 "alternative" technologies; and,
along with the cleanup sites, taking responsibility for the 49
deployments of technology to be used in waste cleanup projects. 

According to the Acting Deputy Assistant Secretary for OST, several
additional performance measures are under consideration for OST's
focus areas to help ensure that the technologies still in development
are "deployable" when they are completed.  These measures include
whether the focus areas' projects address high-priority technology
needs and whether end users consider the technologies under
development to be viable solutions to their needs. 


      FURTHER IMPROVEMENTS PLANNED
      FOR PERFORMANCE MEASURES
-------------------------------------------------------- Chapter 4:1.4

In a June 1998 meeting, EMAB presented its analysis, prepared at EM's
request, of how EM should improve performance measures for technology
development and deployment.  Among other things, EMAB emphasized that
the use of performance measures must be supported by EM's leadership
and that performance measures for EM's technology research,
development, and deployment must be integrated with similar measures
for site cleanup programs.  EMAB also suggested that EM's Technology
Acceleration Committee review and improve existing research and
development performance measures.  The Acting Deputy Assistant
Secretary for OST told us that EMAB's advice would be considered in
designing additional performance measures for OST's focus areas.  As
of September 1998, EM was still in the process of identifying and
improving performance measures to help ensure that cost-effective
innovative technologies are used for waste cleanup. 


   EM IS INCREASING USER INPUT TO
   OST'S PLANNING AND PRIORITY
   SETTING
---------------------------------------------------------- Chapter 4:2

EM has established a mechanism--a user steering committee for each of
OST's focus areas--to engage technology users in setting overall
plans and priorities for the work of the focus areas.  The committees
include the senior managers of DOE field sites (such as sites with
tank waste for the Tanks Focus Area) and headquarters officials
appropriate to the focus area.  These committees are to work on
budgeting, planning, and setting directions for the R&D investments
of the focus areas.  The committees are modeled after the practice of
the Tanks Focus Area, which set up such a committee in 1996.  The
committees for the other focus areas began organizing in February
1998. 

Among other things, user steering committees will help focus areas
develop their multiyear program plans.  OST is initiating these
5-year plans to manage and measure focus areas' performance under the
requirements of the Government Performance and Results Act of 1993.\2
OST plans to complete the first set of plans by December 31, 1998,
and to develop the plans annually for the upcoming 5 years.  In
addition, at their meetings in the spring of 1998, the user steering
committees provided input to the focus areas' proposed fiscal year
2000 budgets. 


--------------------
\2 This act requires executive departments and agencies to establish
long-term strategic goals and annual performance goals and measures
for their programs. 


   CONTINUED UPPER MANAGEMENT
   ATTENTION TO DEPLOYMENT IS NOT
   ENSURED
---------------------------------------------------------- Chapter 4:3

While the EM and OST initiatives have begun to address internal
barriers to the deployment of innovative technologies, continued
attention by EM's upper management to deployment is not ensured.  The
attention may not continue because (1) the future of the Technology
Acceleration Committee is uncertain and (2) deployment measures have
not been included in the contracts of EM's senior managers. 


      UPPER MANAGEMENT COMMITTEE'S
      FUTURE IS UNCERTAIN
-------------------------------------------------------- Chapter 4:3.1

In response to the July 1997 memo by EM's Assistant Secretary, the
Technology Acceleration Committee, composed of senior-level managers
from EM headquarters and the field, was organized and met in
September 1997.  This Committee's purpose is to "provide corporate
leadership to ensure an aggressive effort to deploy alternative and
more effective technologies through full integration of the
technology development and user organizations." According to the
Committee's draft charter, it would meet at least once per quarter. 
The Committee met again in January 1998, but has not met since. 
According to the Acting Deputy Assistant Secretary of OST, the
Committee has been inactive because it reported directly to EM's
Assistant Secretary, who left the Department in January 1998. 
According to the Acting Assistant Secretary for EM, EM is considering
establishing a broader executive committee of senior managers to
address EM issues, including the deployment of innovative
technologies. 

To date, the Technology Acceleration Committee has increased
communication among OST, EM line offices, and field offices.  It has
discussed issues such as clarifying deployment responsibilities,
involving technology users throughout the technology development
process, and improving incentives for contractors.  The Committee
also directed the establishment of user steering committees for focus
areas.  Because the user steering committees have members from EM's
headquarters and field offices, we believe that the existence of the
Technology Acceleration Committee facilitated this innovation.  The
Acting Deputy Assistant Secretary of OST agreed with the importance
of the Committee but thought that a broader executive committee of
senior officials could address technology deployment and other EM
issues. 

Even with these improvements, unresolved issues affecting technology
deployment still exist and could benefit from the attention of EM's
upper management.  As noted above, EMAB suggested that the Technology
Acceleration Committee review and improve R&D performance measures. 
In addition, the site-specific deployment plans state that a number
of issues need to be resolved, such as learning the possible effects
of EM's increased use of fixed-price contracting and private
financing (referred to as "privatization") on the use of innovative
technologies.\3

For example, the deployment plan of the Ohio field office raises
privatization as a policy issue requiring guidance from headquarters,
stating that most fixed-price bidders will use technologies with
which they are familiar.  As a result, the plan states, technologies
that were developed at considerable expense may not be deployed
because of bidders' reluctance to assume a risk of failure.  In our
visits to field sites, we observed instances in which the use of
OST-developed technologies was uncertain because EM planned to
solicit fixed-price bids for cleanup work and the technology selected
would depend on the choice of the private firm winning the contract. 
For instance, the Houdini robot was designed for retrieving
radioactive wastes from silos at the Fernald site.  However, when EM
decided to solicit fixed-price bids for waste retrieval from
Fernald's silos, the Houdini robot was instead used in the
radioactive waste tanks at Oak Ridge.  Fernald had not yet received
bids at the time of our visit, and environmental remediation
officials told us that the companies bidding for this work will
define which waste retrieval tools they would use--Houdini might or
might not be included. 


--------------------
\3 Our report entitled Department of Energy:  Alternative Financing
and Contracting Strategies for Cleanup Projects (GAO/RCED-98-169, May
29, 1998) provides further details about privatization. 


      EM HAS NOT ESTABLISHED
      DEPLOYMENT PERFORMANCE
      EXPECTATIONS IN ITS SENIOR
      MANAGERS' CONTRACTS
-------------------------------------------------------- Chapter 4:3.2

In his July 1997 memo, the Assistant Secretary for EM stated that,
beginning in October 1997, performance expectations for EM's senior
managers in headquarters and the field would be developed to require
the deployment of alternative and more effective technologies. 
However, the Acting Deputy Assistant Secretary for OST, in response
to our written inquiry to EM management, stated that
technology-related performance measures would not be included in
senior managers' performance contracts and that senior managers are
held responsible for technology deployment through their membership
in the Technology Acceleration Committee.  However, as noted above,
this Committee has not met since January 1998, and its future is
uncertain. 


   EM AND OST INITIATIVES DO NOT
   FULLY ADDRESS USER INPUT TO
   TECHNOLOGY PROJECTS AND USER
   NEED FOR DEPLOYMENT ASSISTANCE
---------------------------------------------------------- Chapter 4:4

EM's and OST's current efforts and initiatives only partially address
the internal obstacles to deployment that were discussed in chapter
3.  Specifically, the new initiatives do not reinforce the need for
OST's focus areas to use the technology development gates system and
do not provide for OST's deployment assistance to help sites select
new waste-cleanup technologies or modify existing technologies for
site use. 


      INITIATIVES DO NOT STIPULATE
      THE USE OF THE GATES SYSTEM
-------------------------------------------------------- Chapter 4:4.1

Although EM's initiatives involve users in setting the overall plans
and priorities of OST's focus areas, they do not fully address the
need for detailed user input on individual technology projects.  The
Acting Deputy Assistant Secretary for OST told us that the focus
areas need to use OST's existing gates system to obtain user input
into the design and development of cleanup technology.  Furthermore,
he said that it is necessary to use this system to help prevent the
development of technologies that do not meet sites' needs, a problem
discussed in chapter 3.  However, in contrast to these statements of
support for the gates system, we found that EM's new initiatives do
not require its use nor identify an alternative means to ensure that
technology developers and users communicate and cooperate about
individual technology development projects. 


      EM HAS NOT FULLY ADDRESSED
      DEPLOYMENT ASSISTANCE TO
      CLEANUP SITES
-------------------------------------------------------- Chapter 4:4.2

EM and OST initiatives have not fully addressed two areas that must
be considered when deploying innovative technologies:  (1) providing
technical assistance to sites on innovative technologies and (2)
modifying completed technologies for use at specific sites.  One
potential vehicle for providing deployment assistance--OST's new
Accelerated Site Technology Deployment program--has not increased
technical assistance in most cases and did not have the benefit of
information that EM now has that EM could use to improve its priority
setting for deployment assistance. 


         TECHNICAL ASSISTANCE FOR
         SITES IS NOT YET
         WELL-DEFINED
------------------------------------------------------ Chapter 4:4.2.1

EM and OST have not yet identified sources of expertise and
procedures or developed a policy for routinely providing technical
assistance on innovative technologies to DOE sites.  OST recognizes
that focus areas should more frequently provide technical assistance
to sites when they are selecting and beginning to implement
technologies and that this assistance should address innovative
technologies developed by other sources as well as by OST.  EMAB has
questioned whether the focus areas currently have the expertise
needed to provide such assistance.  The Acting Deputy Assistant
Secretary for OST acknowledged that the focus areas vary in their
degree of expertise and ability to provide technical assistance.  He
noted that the Tanks Focus Area works closely with one of the
national laboratories, which can provide in-depth expertise, and
stated that the other focus areas need to develop a roster of
technical experts who can be consulted for particular site cleanup
problems that the focus areas cannot solve.  Furthermore, the Acting
Deputy Assistant Secretary stated that performance measures that
encourage focus areas to provide technical assistance will be needed. 

Some initial steps have been taken to involve OST in selecting
technology for environmental restoration sites.  In fiscal year 1998,
the Office of Environmental Restoration began including OST in its
processes for providing sites with information and support for
technology selection decisions.  OST is contributing funding and the
technical support of its focus area staff to this program.  However,
OST does not have a similar involvement with EM's Office of Waste
Management or Office of Nuclear Material and Facility Stabilization. 

EM lacks a policy on whether OST should provide technical assistance
for major cleanup actions routinely or only if requested by a site. 
While the management-level Technology Acceleration Committee reached
an "understanding" that the focus areas' role should include
technical support to end users for deployment, the Committee did not
identify resources, procedures, or policies for such technical
assistance.  According to the Acting Deputy Assistant Secretary for
OST, policies and procedures for providing technical assistance will
be one of the elements addressed in the business system redesign
currently under way in OST, and procedures may be completed by the
end of 1998.\4


--------------------
\4 OST sought advice about its business processes from a management
consulting firm, which completed a draft study in Mar.  1998.  The
firm suggested ways to improve and streamline OST's budgeting and
priority setting, roles and responsibilities for technology services,
information management, and tracking of ongoing projects. 


         EM HAS NOT ADDRESSED NEED
         TO MODIFY SOME COMPLETED
         TECHNOLOGIES
------------------------------------------------------ Chapter 4:4.2.2

The initiatives do not address a barrier to deployment that we
discussed in chapter 3--the lack of a mechanism and resources for
modifying completed technologies for use at specific sites.  In fact,
none of the initiatives, action plans, or meetings of the Technology
Acceleration Committee even raise this issue.  Officials at three of
the five sites we visited told us that OST sometimes considers its
technology development work completed before technologies are ready
for specific applications in the field.  The Acting Deputy Assistant
Secretary for OST agreed that this is a problem and told us that,
while the Tanks Focus Area develops technologies fully to the point
of use, technologies from the other focus areas were not always ready
for field use.  For example, he stated that the Mixed Waste Focus
Area had not tested its thermal treatment technologies on actual
radioactive waste.  The Acting Deputy Assistant Secretary stated that
sites and focus areas should work together to enable and jointly fund
the first use of an OST-developed technology. 

While joint OST and site support for deployment has occurred for some
projects--including the reactor safe-storage project at Hanford and
the use of the Houdini robot in tanks at Oak Ridge that are described
in chapter 3--EM lacks an overall policy, procedure, and designation
of responsibilities for situations in which OST-developed
technologies may require modification for site use.  Nor has EM
identified resources for this purpose, except to the extent that some
projects under the Accelerated Site Technology Deployment program may
address this need.  According to the Director of OST's Office of
Technology Systems, focus areas consider any funding needs for
technology modifications when requested by sites.  He noted that such
requests would compete for limited funding with the focus areas'
technology projects. 

EM has data that could be used to identify OST technologies that
might have additional cost-effective deployments.  Sites'
Accelerating Cleanup plans, issued in draft in June 1997 and most
recently submitted in June 1998, provide a comprehensive compilation
of sites' technology needs, as well as detailed information on each
cleanup project across the DOE complex.\5 OST has developed a
database, called a linkage table, that identifies links between its
completed and ongoing projects and the sites' technology needs.  EM
could identify OST-developed technologies that could provide
cost-effective solutions to sites' needs and set priorities for
deployment assistance to cleanup projects, including technical
assistance and technology modifications, if needed.  OST's
Deactivation and Decommissioning Focus Area has already used this
database to contact potential technology users at the sites and
inquire whether the focus area can provide assistance.  However, OST
has not required its focus areas to do this. 


--------------------
\5 The Assistant Secretary for EM initiated the plans--now known as
Accelerating Cleanup:  Paths to Closure--in 1996 to speed up the
cleanup process and identify the steps needed to complete the cleanup
at each site.  EM also issues a national summary of sites' plans. 


         OST'S ACCELERATED SITE
         TECHNOLOGY DEPLOYMENT
         PROGRAM DOES NOT FULLY
         MEET SITES' NEEDS FOR
         DEPLOYMENT ASSISTANCE
------------------------------------------------------ Chapter 4:4.2.3

One potential vehicle for deployment assistance is OST's Accelerated
Site Technology Deployment (ASTD) program, begun in fiscal year 1998. 
OST funded 14 ASTD projects at 12 sites to deploy innovative
technologies in cleanup projects.  The approximately $26 million that
OST provided to site projects in fiscal year 1998 resulted in an
additional investment of about $708 million from the sites over the
life of the projects.\6 OST identified potential ASTD projects
through site proposals and competitively evaluated the proposals to
select projects to fund.  Selection criteria included the technical
merit of the approach, interest in deploying the technologies at
multiple locations, and commitment of additional funding by the site. 

While ASTD may be helping these selected projects in addressing
obstacles to deployment, the program has not fostered interaction
among technology developers and users in many instances.  For
example, we found that OST's focus areas provided technical
assistance to only 5 of the 14 ASTD projects, and national laboratory
personnel who had helped to develop some of the technologies provided
technical assistance to 2 additional ASTD projects.  It should also
be noted that technical assistance and technology modifications on a
smaller scale than the current ASTD projects may be appropriate in
some cases. 


--------------------
\6 OST plans to provide additional funds to some of the 14 projects
in fiscal year 1999 and beyond.  According to the Director of the
Office of Technology Systems, OST plans to provide about $21 million
in fiscal year 1999. 


CONCLUSIONS AND RECOMMENDATIONS
============================================================ Chapter 5

Having spent more than $2 billion and 9 years on over 700 innovative
cleanup technology projects, EM and OST recognize that the cleanup
program can only benefit from these efforts if the innovative
technologies that have been developed are successfully deployed.  To
promote deployment, EM and OST have initiated a number of actions
aimed at improving the relationship between technology developers in
OST and the users at EM's cleanup sites.  However, we are concerned
that the committees and processes that EM and OST are now creating
will be ineffective if they are not accompanied by more fundamental
changes in how EM conducts technology development and deployment. 

We believe that EM and OST need to take three relatively
straightforward actions to increase the deployment of existing
innovative technologies First, OST must make sure that it has
adequate technical expertise to assist users in evaluating and
implementing innovative technologies that it and others have
developed.  The focus areas are the logical source for this
expertise; however, if they are unable to meet this need, other
centers of expertise, possibly in the national laboratories, need to
be developed. 

Second, we continue to believe that OST staff, equipped with the
appropriate expertise, need to be formally involved in evaluating and
selecting technologies for use at the cleanup sites.  We believe that
the program's experience has shown that without a specific
requirement to bridge the gap between developers and users, each
party will continue to operate in its own environment, with users
deploying only those technologies with which they are familiar, and
OST developing technologies that are generic and not designed for
specific situations. 

Third, existing innovative technologies could be implemented, as we
found repeatedly, if they could be modified or fine-tuned to address
a specific site cleanup problem.  Information now exists from sites'
Accelerating Cleanup plans and OST's linkage tables to identify
technologies that can be modified to fit specific situations. 
However, such modification takes money, and without specific action
by EM management, neither users nor developers are likely to provide
these funds on their own.  For example, if OST uses its funds to
fine-tune an existing technology, it is reducing the funds available
for its other missions.  Similarly, users can logically view the use
of their funds to modify a technology as taking away resources that
they need for other cleanups.  However, EM's experience, for example,
from the project for safe storage of the C reactor at the Hanford
site or from the ASTD program, has shown that successful deployment
can occur if both parties make a financial commitment. 

Additional technology development will be needed to address
technology problems for which no cost-effective solution exists, such
as high-level waste tanks at Hanford.  To ensure the deployment of
technologies that are currently under development or will be
developed, EM does not need additional processes and procedures. 
Rather, it needs to rigorously and consistently apply its current
gates system.  Consistent use of this system by focus areas would
help ensure that technology developers and users communicate and
cooperate throughout the development of individual technologies, and
that, if technologies are not living up to their potential or there
is not adequate commitment from users, the project can be terminated
and the funds redirected to more productive uses. 

Ensuring that these actions are taken consistently will require the
commitment of top management in the EM program.  The Technology
Acceleration Committee is a sound idea; however, it has already
missed a planned meeting, and we are concerned that it could easily
slip into disuse.  We believe that continuing a committee of senior
EM managers is a key element in ensuring that top management is
focused on formulating policy for technology deployment.  An
additional important element is the establishment of performance
measures that hold EM's top managers accountable for technology
deployment.  While EM has made clear to field managers that they are
responsible for deploying innovative technologies, this commitment
needs to be reflected throughout the organization if additional
innovative technologies are to be successfully deployed. 

Finally, with an increased emphasis on deployment, EM will need more
accurate data than it currently has on deployment efforts.  A
verification effort similar to the one we undertook will be needed to
provide valid data on future deployments.  On the other hand, we
recognize that improving data on prior deployments may not be
cost-effective.  Therefore, reporting existing data as estimates
could lend more credibility to the data and the overall program.  In
addition, EM has recognized that deployment is not the only relevant
measure of success in technology development.  EM's recent efforts to
develop additional performance measures for the entire program are a
step in the right direction. 


   RECOMMENDATIONS
---------------------------------------------------------- Chapter 5:1

To increase the deployment of existing technologies and ensure that
technologies developed in the future are used, we recommend that the
Secretary of Energy direct the Assistant Secretary for Environmental
Management to

  -- direct the Deputy Assistant Secretary for the Office of Science
     and Technology to establish centers of expertise for innovative
     technologies by using existing focus areas or another approach
     if needed and require that a representative from one of these
     centers participate in the technology selection process on each
     cleanup project;

  -- direct the cleanup programs and OST to (1) use existing data to
     identify OST-developed technologies that can be cost-effectively
     modified to meet sites' needs and (2) identify funds to modify
     these technologies if needed;

  -- direct that the gates system be used rigorously and consistently
     as a decision-making tool for managing technology development
     projects and as a vehicle for increasing developer-user
     cooperation;

  -- use their annual performance expectations to hold EM
     headquarters managers responsible for increasing the deployment
     of innovative technology; and

  -- implement a system to verify the accuracy of future deployment
     data and label any existing data that have not been verified as
     an estimate. 


   AGENCY COMMENTS AND OUR
   EVALUATION
---------------------------------------------------------- Chapter 5:2

Overall, DOE agreed with the recommendations in our report.  In doing
so, DOE offered information regarding actions it had taken or
intended to take that it believed were responsive to our
recommendations.  However, DOE's responses to two of the
recommendations suggest that the actions described would not be fully
responsive to these recommendations.  DOE's comments are included as
appendix III. 

In response to our recommendation that OST establish centers of
expertise and include a representative from one of these centers in
the technology selection process, DOE indicated a willingness to act
on our recommendation but offered few specifics, especially with
respect to involving OST in the technology selection process.  In
1994, we also recommended that OST be given a formal role in the
technology selection process.  During our current review, we found
that this recommendation had not been implemented primarily because
site officials were skeptical about OST's ability to provide quality
technical advice and were therefore reluctant to allow OST more of a
role in selecting cleanup technologies.  We believe that it will take
more specific actions by OST, beyond the generalized user steering
committees cited in its response, to develop credible expertise and
thus gain a role in the technology selection process. 

In response to our recommendation that DOE rigorously and
consistently use the gates system as a decision-making tool for
managing technology development, DOE also agreed with the
recommendation but noted that it had incorporated the gates system
into its system of peer review.  While we recognize the value of peer
review as a mechanism for obtaining independent technical judgments
about projects OST is pursuing, we note that peer review can occur
infrequently over the life of a project and after significant
decisions are made.  Therefore, we do not believe that peer review is
a substitute for focus area managers using a disciplined
decision-making system that involves users throughout the technology
development and deployment process. 


METHODOLOGY USED TO ASSESS THE
RELIABILITY OF OST'S DATA
=========================================================== Appendix I

This appendix describes the methodology we used to assess the
reliability of the Office of Science and Technology's (OST) data in
order to address the question of the extent to which innovative
technologies developed by OST have been deployed (used) at the
Department of Energy's (DOE) waste cleanup sites and to compare this
rate of deployment with the rates of other government organizations
that develop environmental technologies. 

To answer this question, we obtained an electronic file from OST
containing project data from OST's inception in 1989 through January
1998.  OST had initially developed most of these data in response to
other congressional inquiries.  The data included, among other
things, each project's total cost to OST, research stage, number of
times deployed and associated deployment sites.  We limited our
reliability assessments to data on the number of times each project
was deployed and the deployment sites identified in the database. 


   METHODOLOGY
--------------------------------------------------------- Appendix I:1

We used attribute sampling techniques and verified claimed technology
deployments with site operations officials or with individuals
familiar with the technology's use at that site.  We used our sample
results to estimate a range for the actual number of OST project
deployments. 


      UNIVERSE
------------------------------------------------------- Appendix I:1.1

The OST data listed 713 projects initiated since the program's
inception.  The records for 152 of these projects indicated the
technology was deployed at one or more sites.  We chose to verify the
deployments for this group of projects because (1) no clear-cut way
exists to verify that a technology has not been used at any DOE site
and (2) we believed that the risk was low that OST neglected to count
significant numbers of deployed projects, thereby understating its
success. 


      SAMPLE AND VERIFICATION
------------------------------------------------------- Appendix I:1.2

We randomly selected a sample of 30 projects from the 152 projects
that OST claimed were deployed 1 or more times.  For each project in
our sample, we contacted all of the deployment sites listed in OST's
database under the project.  To the extent possible, we verified
project deployment either with DOE Environmental Management (EM) or
contractor officials having operational duties at the sites.  At the
time of our reliability assessment, OST lacked a formal definition of
deployment.  We developed the following definition on the basis of
(1) how the term was used by the Committee requesting our review, (2)
what the need was for comparability with the two other organizations
providing deployment data, and (3) what technology users thought was
reasonable.  We considered a technology to be deployed if, at any
site listed,

  -- the cleanup or waste management project was completed using the
     technology;

  -- the cleanup or waste management project is currently using the
     technology;

  -- a contract is in place for use of the technology in the future;
     or

  -- a demonstration either accomplished the cleanup or
     characterization goals, was expanded to accomplish these goals,
     or resulted in a contract for the use of the technology at that
     site. 


      ESTIMATIONS
------------------------------------------------------- Appendix I:1.3

Using our sample results, we estimated a range for the actual number
of projects deployed 1 or more times.  Because we used a probability
sample of deployed projects to estimate the number of actual
deployments, this estimate has a measurable precision, or sampling
error, which may be expressed as a plus/minus figure.  The sampling
error indicates how closely we can reproduce from a sample the
results that we would obtain if we were to take a complete count of
the universe, using the same verification methods.  By adding the
sampling error to and subtracting it from the estimate, we can
develop upper and lower bounds for each estimate.  The resulting
range is called a confidence interval.  Sampling errors and
confidence intervals are stated at a certain confidence level--in
this case, 95 percent.  The 95-percent confidence level means that in
95 out of 100 instances, the sampling procedure we used would produce
a confidence interval containing the universe value we are
estimating. 

As a result, we estimate that a total of 88 to 130 OST projects have
been deployed 1 or more times.  We used this estimated range of
actual deployments to compute the estimated range of OST's deployment
rate.  The deployment rate represents the proportion of deployed
projects to the total number of technology development projects
started by OST.  For our estimate of the deployment rate, we
substituted the number of deployed projects reported by OST with the
range we estimated above.  As a result, we computed that OST actually
deployed from 88 to 130 of the 713 projects reported from 1989 to
January 1998, or 12 to 18 percent. 

Owing to multiple deployments for some projects, the OST database
reported 283 deployment instances (locations) for the 152 deployed
projects, naming a site for most instances.  For example, the Waste
Inspection Tomography project was listed as deployed once at Idaho
National Environmental Engineering Laboratory, once at Hanford, and
twice at Lawrence Livermore National Laboratory, for a total of four
deployment instances. 

Using the same verification sample, we performed cluster sampling
analysis to estimate a range for the actual total number of
deployment instances for all OST projects.  From this analysis, we
estimate that the number of deployment instances ranges from 137 to
216.  Figure I.1 illustrates how we used the same universe and sample
to verify both project deployments and the total number of deployment
instances claimed. 

   Figure I.1:  Universe and
   Sample of OST Projects

   (See figure in printed
   edition.)


OST'S DEPLOYMENT RATE COMPARED
WITH THE RATES OF TWO GOVERNMENTAL
ENVIRONMENTAL TECHNOLOGY
DEMONSTRATION PROGRAMS
========================================================== Appendix II

Two government programs that demonstrate innovative environmental
technologies provided deployment data for comparison with OST's data: 
the Environmental Protection Agency's (EPA) Superfund Innovative
Technology Evaluation (SITE) program and the Department of Defense's
Environmental Security Technology Certification Program (ESTCP).  As
discussed in chapter 2, we found that any deployment rate comparisons
have limited usefulness.  Nevertheless, deployment data from these
programs provide some evidence that OST has deployed its technologies
at rates lower than or close to rates experienced by other
governmental programs in this area of research and development.  This
appendix describes the two demonstration programs and the comparisons
we made. 

Unlike OST, which develops technologies beginning with basic research
and progressing through their demonstration and implementation, the
SITE program and ESTCP only demonstrate and implement existing
technologies.  In order to provide equitable comparisons, we
identified those OST projects at stages of maturity similar to SITE
and ESTCP projects, resulting in two subgroups of OST projects to use
for comparison.  Furthermore, we adjusted OST's deployment rates for
the two subgroups to reflect the errors that we found in OST's
deployment data for each group.  The methodology used for these
adjustments is described in appendix I.  We then compared the
adjusted deployment rates for the two subgroups of OST projects with
the deployment rates for SITE and ESTCP. 


   SUPERFUND INNOVATIVE TECHNOLOGY
   EVALUATION
-------------------------------------------------------- Appendix II:1

The SITE program has been demonstrating innovative environmental
technologies for about 12 years.  EPA initiated the program in
response to the 1986 Superfund Amendments and Reauthorization Act. 
The program's mission is to encourage the development and
implementation of (1) innovative treatment technologies for
remediating hazardous waste sites and (2) characterization and
monitoring technologies for evaluating the nature and extent of
contamination at hazardous waste sites.  The SITE program does not
perform earlier stages of technology development but demonstrates
technologies in order to evaluate their effectiveness.  It provides
methodologies for demonstrations and funds associated sampling and
testing.  The technology vendors generally pay for all technology
equipment.  According to a program official, the SITE program's
fiscal year 1997 budget was $6.8 million. 

SITE published its demonstration results for all projects completed
through December 1996 in EPA SITE Technology Profiles, Ninth Edition. 
To gather information for this publication, the SITE program had
surveyed its vendors, asking them, for example, to report how many
contracts or jobs they have obtained to use the SITE-demonstrated
technologies.  Although they had not previously analyzed the survey
data in this manner, SITE officials agreed to compute a deployment
rate for SITE technologies by counting a technology project as
deployed if its vendor reported one or more contracts or jobs to use
the technology.  According to SITE officials, because up to 10 years
had elapsed since the first demonstrations and some vendors had
changed management or gone out of business, about 64 percent of
vendors responded to the surveys.  SITE computed a deployment rate
for all 80 completed demonstrations, whether or not the vendor
responded to the surveys.  As a result, the SITE deployment rate may
be based on incomplete reporting of deployments and could actually be
higher. 

The SITE program reported that 59 percent of its demonstration
projects were deployed at least once.\1 For projects at a similar
stage of maturity, we estimate that OST deployed from 28 to 45
percent.  We computed this OST deployment rate for projects OST
identified as either having ended the demonstration stage of
development or having entered the implementation stage.  OST listed
191 projects in this subgroup and claimed that 94 were deployed 1 or
more times.  From our verification sample of 30 projects, 18 projects
fell into this subgroup, and we found four errors.  Using the same
statistical methods described in appendix I, we estimate that OST
actually deployed from 53 to 86 of these relatively mature projects,
to produce an estimated deployment rate of 28 to 45 percent, which is
lower than the rate for the SITE program. 


--------------------
\1 We did not test the reliability of SITE or ESTCP data, but we
believe the data to be reasonable on the basis of our review of
project descriptions. 


   ENVIRONMENTAL SECURITY
   TECHNOLOGY CERTIFICATION
   PROGRAM
-------------------------------------------------------- Appendix II:2

The Department of Defense's ESTCP demonstrates and validates
technologies and funds environmental technologies that have
progressed to the stage where field demonstrations are warranted. 
ESTCP has existed only since fiscal year 1995.  Since the program is
relatively new, its deployment data are based on a limited number of
projects and may be less representative of the program's future
performance.  Like the SITE program, ESTCP demonstrates technologies
that other research organizations have brought to the demonstration
stage.  ESTCP also encourages investment by the technology user or
vendor.  ESTCP operated on a budget of about $22 million in fiscal
year 1997. 

ESTCP provided a description of the transition status (the
technology's actual use or selection for future use) for 32 ESTCP
projects started in fiscal years 1995 or 1996.\2

Because the number of projects was small, we decided to use a
deployment rate for comparison that was based upon the transition
status of all projects ESTCP began in its first 2 years of
existence--regardless of whether they had been completed. 

ESTCP reported that, for projects started in the program's first 2
years of existence, 38 percent were deployed at least once.  For
projects at a similar stage of maturity, we estimated that OST
deployed from 28 to 40 percent.  Our estimate was based on OST
projects that had entered demonstration or later stages, regardless
of whether OST had ended the project.\3 OST listed 290 projects in
this subgroup and said that 129 were deployed 1 or more times.  From
our verification sample of 27 projects that fell into this subgroup,
we found six errors.  Using the same statistical methods as for the
SITE program comparison, we estimated that OST actually deployed from
80 to 115 of these projects, to produce an estimated deployment rate
of 28 to 40 percent for all of its projects reaching demonstration or
later stages.  This rate is similar to that achieved by ESTCP in its
first 2 years of existence. 



(See figure in printed edition.)Appendix III

--------------------
\2 In fiscal years 1995 and 1996, ESTCP started 33 projects; however,
we removed 1 project from the analysis because its objective was not
technology deployment. 

\3 The subgroups of OST projects that we used for comparison differ
in that the SITE program's comparison subgroup included all OST
projects that had ended the demonstration stage and/or entered into
the implementation stage, while the comparison subgroup for ESTCP
included all OST projects that had at least entered the demonstration
or implementation stage of development. 


COMMENTS OF THE DEPARTMENT OF
ENERGY
========================================================== Appendix II



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)


The following are GAO's comments on DOE's letter dated August 19,
1998. 

GAO'S COMMENTS

1.  We concur with this comment and have made changes to the report
where appropriate. 

2.  See chapter 5 for our reply to DOE's response to our
recommendation that OST establish centers of expertise and include a
representative from one of these centers in the technology selection
process. 

3.  See chapter 5 for our reply to DOE's response to our
recommendation that DOE rigorously and consistently use the gates
system as a decision-making tool for managing technology development. 




(See figure in printed edition.)Appendix IV
COMMENTS OF THE DEPARTMENT OF
DEFENSE
========================================================== Appendix II


The following is GAO's comment on the letter from the Department of
Defense, dated August 28, 1998. 

GAO'S COMMENT

1.  We concur with this comment and have made changes to the report
where appropriate. 


MAJOR CONTRIBUTORS TO THIS REPORT
=========================================================== Appendix V

RESOURCES, COMMUNITY, AND ECONOMIC
DEVELOPMENT DIVISION

James Noï¿½l, Assistant Director
Rachel J.  Hesselink, Evaluator-In-Charge
Jack Paul, Senior Evaluator
Ilene M.  Pollack, Senior Evaluator

SEATTLE FIELD OFFICE

Patricia J.  Rennie, Senior Evaluator
Evan Stoll, Programmer Analyst

OFFICE OF GENERAL COUNSEL

Susan W.  Irwin, Senior Attorney


*** End of document. ***