Department of Energy: Major Construction Projects Need a Consistent Approach for Assessing Technology Readiness to Help Avoid Cost Increases and Delays

Department of Energy: Major Construction Projects Need a
Consistent Approach for Assessing Technology Readiness to Help
Avoid Cost Increases and Delays (27-MAR-07, GAO-07-336).

The Department of Energy (DOE) spends billions of dollars on
major construction projects that help maintain the nuclear
weapons stockpile, conduct research and development, and process
nuclear waste so that it can be disposed of. Because of DOE's
long-standing project management problems, GAO determined the
extent to which (1) DOE's major construction projects are having
cost increases and schedule delays and the major factors
contributing to these problems and (2) DOE ensures that project
designs are sufficiently complete before construction begins to
help avoid cost increases and delays. We examined 12 DOE major
projects with total costs of about $27 billion, spoke with
federal and contractor officials, and reviewed project management
documents.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-336
ACCNO: A67345
TITLE: Department of Energy: Major Construction Projects Need a
Consistent Approach for Assessing Technology Readiness to Help
Avoid Cost Increases and Delays
DATE: 03/27/2007
SUBJECT: Contract oversight
Cost overruns
Critical technologies
Design specifications
Facility construction
Performance measures
Program evaluation
Program management
Research and development facilities
Schedule slippages
Strategic planning
Technology assessment
Work measurement
Contract mismanagement

******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-336

* [1]Results in Brief
* [2]Background
* [3]Most Major Projects Have Exceeded Original Costs and Are Yea
* [4]DOE Does Not Consistently Measure Technology Readiness to En

* [5]DOE Does Not Consistently Assess Technology Readiness
* [6]Other Federal Agencies Use a Standard Method for Measuring a

* [7]Conclusions
* [8]Recommendations for Executive Action
* [9]Agency Comments and Our Evaluation
* [10]National Ignition Facility
* [11]Mixed Oxide Fuel Fabrication Facility
* [12]Pit Disassembly and Conversion Facility
* [13]Waste Treatment and Immobilization Plant
* [14]Spallation Neutron Source
* [15]Salt Waste Processing Facility
* [16]Tritium Extraction Facility
* [17]Highly Enriched Uranium Materials Facility
* [18]Depleted Uranium Hexafluoride 6 Conversion Facility
* [19]Chemistry and Metallurgy Research Facility Replacement
* [20]GAO Contact
* [21]Staff Acknowledgments
* [22]GAO's Mission
* [23]Obtaining Copies of GAO Reports and Testimony

* [24]Order by Mail or Phone

* [25]To Report Fraud, Waste, and Abuse in Federal Programs
* [26]Congressional Relations
* [27]Public Affairs

Report to the Subcommittee on Energy and Water Development, and Related
Agencies, Committee on Appropriations, House of Representatives

United States Government Accountability Office

GAO

March 2007

DEPARTMENT OF ENERGY

Major Construction Projects Need a Consistent Approach for Assessing
Technology Readiness to Help Avoid Cost Increases and Delays

GAO-07-336

Contents

Letter 1

Results in Brief 4
Background 7
Most Major Projects Have Exceeded Original Costs and Are Years Late,
Principally Because of Ineffective DOE Project Oversight and Contractor
Management 9
DOE Does Not Consistently Measure Technology Readiness to Ensure That
Critical Technologies Will Work as Intended before Construction Begins 18
Conclusions 26
Recommendations for Executive Action 27
Agency Comments and Our Evaluation 28
Appendix I Scope and Methodology 31
Appendix II Information on the 12 Department of Energy Major Projects
Reviewed 34
Appendix III Independent Studies Reviewed 36
Appendix IV Survey Results for Primary Factors Affecting Cost and Schedule
on Nine Projects with Cost or Schedule Changes 42
Appendix V Definitions of Technology Readiness Levels 44
Appendix VI Comparison of DOD's Product Development Process with DOE's
Project Management Process 47
Appendix VII Comments from the Department of Energy 48
Appendix VIII GAO Contact and Staff Acknowledgments 50

Tables

Table 1: Changes in Estimated Total Project Cost for DOE Major
Construction Projects 10
Table 2: Changes in Estimated Project Schedules for DOE Major Construction
Projects 11
Table 3: Reasons for Cost Increases and Schedule Delays 12

Abbreviations

DOD Department of Defense
DOE Department of Energy
EM Office of Environmental Management
ITP In-Tank Precipitation
NASA National Aeronautics and Space Administration
NNSA National Nuclear Security Administration
PDRI Product Definition Rating Index
TPC total project cost
TRL technology readiness level

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office
Washington, DC 20548

March 27, 2007

The Honorable Peter J. Visclosky
Chairman
The Honorable David L. Hobson
Ranking Member
Subcommittee on Energy and Water Development, and Related Agencies
Committee on Appropriations
House of Representatives

The Department of Energy (DOE) spends billions of dollars on major
construction projects that, among other things, are used to help maintain
the nuclear weapons stockpile, conduct research and development in the
areas of high-energy physics and nuclear physics, and process nuclear
waste into forms suitable for longer-term storage or permanent disposal.
DOE oversees the construction of facilities primarily at government-owned,
contractor-operated sites throughout the nation. In July 2006, DOE revised
its dollar threshold defining a construction project as "major"; it is now
at $750 million, up from $400 million when we began our review in December
2005. The following 12 projects included in our review had estimated
project costs exceeding the original threshold.^1 The total cost of these
projects is currently estimated at about $27 billion.^2 These 12 projects
and their locations are as follows:

o Chemistry and Metallurgy Research Facility Replacement--Los
Alamos National Laboratory, Los Alamos, New Mexico.
o Depleted Uranium Hexafluoride 6 Conversion Facility--Portsmouth,
Ohio, and Paducah, Kentucky.
o Highly Enriched Uranium Materials Facility--Y-12 National
Security Complex, Oak Ridge, Tennessee.
o Linac Coherent Light Source--Stanford Linear Accelerator Center,
Menlo Park, California.
o Microsystems and Engineering Sciences Applications--Sandia
National Laboratories, Albuquerque, New Mexico.
o Mixed Oxide Fuel Fabrication Facility--Savannah River Site,
Aiken, South Carolina.
o National Ignition Facility--Lawrence Livermore National
Laboratory, Livermore, California.
o Pit Disassembly and Conversion Facility--Savannah River Site,
Aiken, South Carolina.
o Salt Waste Processing Facility--Savannah River Site, Aiken,
South Carolina.
o Spallation Neutron Source--Oak Ridge National Laboratory, Oak
Ridge, Tennessee.
o Tritium Extraction Facility--Savannah River Site, Aiken, South
Carolina.
o Waste Treatment and Immobilization Plant--Hanford Site,
Richland, Washington.

These major projects require the construction of large building
complexes and the development of innovative cleanup and other
technologies. Many of these technologies are developed for the
project or are applied in a new way. DOE project directors are
responsible for managing these major projects and overseeing the
contractors that design and construct the facilities. In doing so,
project directors follow specific departmental directives,
policies, and guidance for project management. Among these are DOE
Order 413.3A and Manual 413.3-1, which establish protocols for
planning and executing a project. The protocols require DOE
projects to go through a series of five critical decisions as they
enter each new phase of work. Two of the decisions made before
construction begins are key: (1) formally approving the project's
definitive cost and schedule estimates as accurate and
complete--an approval that is to be based on a review of the
project's completed preliminary design, and (2) reaching agreement
that the project's final design is sufficiently complete and that
resources can be committed toward procurement and construction. To
oversee projects and approve these critical decisions, DOE
conducts its own reviews, often with the help of independent
technical experts. In addition, projects are regularly subject to
reviews by DOE's Office of Engineering and Construction Management
and it's Office of Inspector General, the Department of Defense's
(DOD) U.S. Army Corps of Engineers, the Defense Nuclear Facilities
Safety Board, and the National Research Council, among others.

We and others have reported over the past decade that project
management weaknesses have impaired these projects. For example,
for the Waste Treatment and Immobilization Plant, we reported that
DOE's use of a "fast-track, design-build" approach to
construction--in which design and construction activities
overlap--has been problematic for highly complex, first-of-a-kind
facilities. We found that the designs for these facilities were
not sufficiently complete for construction to begin, which has
resulted in significant cost increases and schedule delays.^3

In this context, we determined the extent to which (1) DOE's
active major construction projects are experiencing cost increases
and schedule delays and the key factors contributing to these
problems and (2) DOE ensures that project designs are sufficiently
complete before construction begins to help avoid cost increases
and schedule delays.^4

To determine the extent to which DOE projects are experiencing
cost increases or schedule delays and factors contributing to
these problems, we sent a survey to the 12 DOE directors of major
projects and reviewed the project management documents for these
projects--6 projects that were above the $750 million threshold, 2
estimated to cost between the previous $400 million threshold and
$750 million, and 4 estimated to cost between $300 million and
$400 million. (App. II describes these projects.) These 12
projects are managed by DOE's Office of Science, Office of
Environmental Management, or National Nuclear Security
Administration (NNSA). We conducted site visits and analyzed
independent project studies for the 9 projects that experienced
cost increases or schedule delays. During the course of our
review, we identified a method used by the DOE project director
for the Pit Disassembly and Conversion Facility to systematically
assess the extent to which a technology is sufficiently developed
for its intended purpose. The project director based this method
on a system developed by the National Aeronautics and Space
Administration (NASA). We had previously reported on the use of a
similar assessment system--technology readiness levels (TRL),
which DOD has adopted for its major projects.^5 We obtained and
reviewed documents regarding these two assessment systems.

To determine the extent to which DOE ensures that project designs
are sufficiently complete before construction, we reviewed in
detail 5 of the 12 projects that were approaching or had recently
begun construction. Specifically, we obtained information on the
extent to which project designs were, or are expected to be,
complete before beginning construction, and the actions DOE has
taken to ensure that technologies used in these designs are
sufficiently ready to begin construction.

Because we and others have previously expressed concern about the
data reliability of a key DOE project management tracking
database--the Project Assessment and Reporting System--we did not
develop conclusions or findings on the basis of information
generated through that system.^6 Instead, we collected information
directly from project site offices. In addition, we spoke with
officials from DOE program offices and DOE's Office of Engineering
and Construction Management in Washington, D.C. We provided
interim briefings to the Subcommittee on the status of our work in
May and September, 2006. We performed our work between December
2005 and January 2007, in accordance with generally accepted
government auditing standards. Appendix I contains a detailed
description of our scope and methodology.

Results in Brief

Nine of the 12 DOE major projects we reviewed have exceeded their
original cost estimates and/or experienced schedule delays,
principally because of ineffective DOE project oversight and poor
contractor management, according to independent studies we
reviewed and interviews we conducted with DOE and contractor
project officials. Specifically, 8 of the 12 projects experienced
cost increases ranging from $79.0 million to $7.9 billion, and 9
of the 12 projects are behind schedule by 9 months to more than 11
years. Major factors cited for these cost increases and delays
included the following:

o Ineffective DOE project oversight. For all 9 projects
experiencing cost increases or schedule delays, poor DOE oversight
was a key contributing factor. Project oversight problems included
inadequate systems for measuring contractor performance, approval
of construction activities before final designs were sufficiently
complete, ineffective project reviews, and insufficient DOE
staffing and project management experience.
o Poor contractor management. Eight of the 9 major projects
experienced cost increases and/or schedule delays, in part because
contractors did not effectively manage the development and
integration of the technology used in the projects, including not
accurately anticipating the cost and time that would be required
to carry out the highly complex tasks involved. For example, the
National Ignition Facility has had over $1 billion in cost
increases and years of schedule delays owing in part to technology
integration problems, according to the DOE project director. Other
examples of poor contractor performance included inadequate
quality assurance for the Highly Enriched Uranium Materials
Facility, which resulted in concrete work that did not meet design
specifications. The subsequent suspension of construction
activities and rework added to the project's estimated cost and
schedule.

DOE officials also explained that a now-defunct policy may have
contributed to increased costs and delays for several projects we
examined. Until 2000, DOE required contractors to prepare cost and
schedule estimates early in the project, before preliminary
designs were completed. These estimates were used to establish a
baseline for measuring contractor performance and tracking any
cost increases or schedule delays. However, these estimates often
were based on early conceptual designs and, thus, were subject to
significant change as more detailed designs were developed. To
improve the reliability of these estimates, DOE issued a new order
in October 2000 that required the preparation of a cost estimate
range at the start of preliminary design, and delayed the
requirement for a definitive cost and schedule baseline estimate
until after preliminary design was completed. Consequently, DOE
officials explained, the new policy should result in improved
estimates and a more accurate measure of cost and schedule
performance.

To effectively assess technology readiness, NASA pioneered and DOD
has adopted a process for measuring and communicating technology
readiness for first-of-a-kind technology applications. This
process uses a nine-point scale for assessing TRLs. Using this
scale, a technology would receive a higher TRL value (e.g., TRL 7)
if it has been successfully demonstrated in an operational
environment, compared with a technology that has been demonstrated
only in a laboratory test (e.g., TRL 4). Several DOE project
directors we spoke with agreed that a consistent, systematic
method for assessing technology readiness would help standardize
terminology, make technology assessments more transparent, and
help improve communication among project stakeholders before they
make critical project decisions.

To improve oversight and decision making for DOE's major
construction projects, we are recommending that the Secretary of
Energy evaluate and consider adopting a disciplined and consistent
approach to assessing TRLs for projects with critical
technologies.

DOE provided comments to us based on a draft of the report. DOE
agreed with our recommendations but suggested revisions that would
first allow them to conduct a pilot application on selected
projects to better understand the technology readiness assessment
process and evaluate its potential use. We revised our
recommendations as appropriate. DOE suggested that our report is
too narrowly focused on technology assessment, and that we
inappropriately calculated cost increases and schedule delays
using preliminary estimates that were only intended for internal
DOE planning. We believe that our recommendations were justifiably
based on our finding that DOE has not systematically ensured that
project designs, including critical technologies reflected in
these designs, have been demonstrated to work as intended prior to
construction. We also believe it was appropriate, when necessary,
to measure cost and schedule changes using the initial estimates
that were developed at the end of conceptual design, as specified
in DOE's project management policy in effect prior to 2000. We
note that these estimates were, in some instances, the only
initial estimates available and had been used by DOE to inform the
Congress of the estimated cost and schedule of the projects while
it was seeking initial project funding. We also incorporated
technical changes in this report where appropriate on the basis of
detailed comments provided by DOE.

Background

To meet its diverse missions, DOE pays its contractors billions of
dollars each year to implement hundreds of projects, ranging from
hazardous waste cleanups at sites in the weapons complex to the
construction of scientific facilities. Many of these complex,
unique projects are designed to meet defense, energy research,
environmental, and fissile materials disposition goals. They often
rely on technologies that are unproven in operational conditions.
In recent years, DOE's budget has been dominated by the monumental
task of environmental restoration and waste management to repair
damage caused by the past production of nuclear weapons.

DOE has long had a poor track record for developing designs and
cost estimates and managing projects. We reported in 1997 that
from 1980 to 1996, 31 of DOE's 80 major projects were terminated
prior to completion, after expenditures of over $10 billion; 15 of
the projects were completed, but most of them were finished behind
schedule and with cost overruns; and the remaining 34 ongoing
projects also were experiencing schedule slippage or cost
overruns.^7 In addition, for over a decade, DOE's Office of
Inspector General, the National Academy of Sciences, and others
have identified problems with DOE's management of major
construction projects. Projects were late or never finished;
project costs increased by millions and sometimes billions of
dollars; and environmental conditions at the sites did not
significantly improve. According to the National Research
Council,^8 DOE's construction and environmental remediation
projects take much longer and cost about 50 percent more than
comparable projects by other federal agencies or projects in the
private sector.^9 A 2004 assessment of departmental project
management completed by the Civil Engineering Research Foundation
recommended, among other things, that DOE develop a core group of
highly qualified project directors and require peer reviews for
first-of-a-kind and technically complex projects when the
projects' preliminary baselines are approved.^10

To address project management issues, DOE began a series of
reforms in the 1990s that included efforts to strengthen project
management practices. To guide these reforms, the department
formed the Office of Engineering and Construction Management in
1999. The reforms instituted to date have included planning,
organizing, and tracking project activities, costs, and schedules;
training to ensure that federal project managers had the required
expertise to manage projects; increasing emphasis on independent
reviews; and strengthening project reporting and oversight.

Most Major Projects Have Exceeded Original Costs and Are Years
Late, Principally Because of Ineffective DOE Project Oversight
and Contractor Management

The estimated costs of many of the DOE major construction projects
we reviewed have significantly exceeded original estimates and
schedules have slipped. On the basis of our analysis of
independent project studies and interviews with project directors,
cost growth and schedule slippage occurred principally because of
ineffective DOE project oversight and poor contractor project
management. Furthermore, unreliable initial cost and schedule
estimates resulting from a now-defunct policy may have been a
contributing factor, according to DOE project officials. Although
external factors, such as additional security and safety
requirements, contributed to cost growth and delays, the
management of these requirements was complicated by ineffective
and untimely DOE communication and decision making.

Eight of the 12 DOE projects we reviewed had increases in
estimates of total project cost (TPC) ranging from $79.0 million
to $7.9 billion. As table 1 shows, the percentage of cost increase
for these 8 projects ranged from 2 percent to over 200 percent.

Table 1: Changes in Estimated Total Project Cost for DOE Major
Construction Projects

Source: GAO analysis of DOE data.

aIn 2000, DOE changed its requirements for establishing initial
cost and schedule estimates. Prior to 2000, these estimates were
established at the end of conceptual design. After 2000, DOE
required initial estimates to be completed later in the
project--at the end of preliminary design. For projects beginning
prior to 2000, and for projects beginning after 2000 that had not
yet completed preliminary design, we used the TPC estimates
prepared after conceptual design. For additional details on our
methodology, see appendix I.

bWe calculated the percentages of cost increases on the basis of
constant 2007 dollars to make them comparable across projects and
to show real increases in cost while excluding increases due to
inflation.

cNNSA officials, in commenting on our draft report, stated that
initial and current cost estimates for the Mixed Oxide Fuel
Fabrication Facillity and the Pit Disassembly and Conversion
Facility should not be used in this analysis because neither
project has an approved budget quality baseline. Nevertheless, we
included the estimates in this analysis because both projects have
been in an extended period of project design, without an approved
budget-quality baseline, for about 10 years, and the estimates
provided here are the only estimates available.

dEstimate may change when DOE approves contractor's revised TPC in
2007.

In addition, as shown in table 2, 9 of the 12 projects experienced
schedule delays ranging from 9 months to more than 11 years. Of
the 9 projects, 7 had schedule delays of at least 2 years or more.

^1For this review, we lowered the threshold to $300 million out of concern
that some projects not considered major could later be defined as major
because of cost increases.

^2This estimate includes design and construction costs, but does not
reflect the total life-cycle costs of the projects, such as operating and
maintenance costs.

^3GAO, Hanford Waste Treatment Plant: Contractor and DOE Management
Problems Have Led to Higher Costs, Construction Delays, and Safety
Concerns, [28]GAO-06-602T (Washington, D.C.: Apr. 6, 2006).

^4A forthcoming GAO report will address actions taken by DOE to improve
overall project management.

^5GAO, Best Practices: Better Management of Technology Development Can
Improve Weapon System Outcomes, [29]GAO/NSIAD-99-162 (Washington, D.C.:
July 30, 1999); Defense Acquisitions: Assessments of Selected Major
Weapons Programs, [30]GAO-06-391 (Washington, D.C.: Mar. 31, 2006); and
Defense Acquisitions: Space-Based Radar Effort Needs Additional Knowledge
before Starting Development, [31]GAO-04-759 (Washington, D.C.: July 23,
2004).

^6GAO, Department of Energy: Further Actions Are Needed to Strengthen
Contract Management for Major Projects, [32]GAO-05-123 (Washington, D.C.:
Mar. 18, 2005); and Civil Engineering Research Foundation, Independent
Research Assessment of Project Management Factors Affecting Department of
Energy Project Success (Washington, D.C: July 12, 2004).

^7GAO, Oversight of DOE's Major Systems, [33]GAO/RCED-97-146R (Washington,
D.C.: Apr. 30, 1997).

^8The National Research Council was organized by the National Academy of
Sciences to advise the federal government on matters related to science
and technology.

^9National Research Council, Improving Project Management in the
Department of Energy (Washington, D.C.: July 1999).

^10Civil Engineering Research Foundation, Independent Research Assessment
of Project Management Factors Affecting Department of Energy Project
Success (Washington, D.C.: July 12, 2004).

Dollars in millions
Initial total
project cost (TPC) Current TPC Percentage
Project estimate^a estimate increase^b
Mixed Oxide Fuel Fabrication
Facility^c $1,400 $4,699 205%
Waste Treatment and
Immobilization Plant 4,350 12,263 143
Highly Enriched Uranium
Materials Facility 251 549 102
National Ignition Facility 1,199 $2,248 59
Salt Waste Processing Facility 440 680^d 50
Pit Disassembly and Conversion
Facility^c 1,700 2,694 40
Tritium Extraction Facility 384 506 15
Spallation Neutron Source 1,333 1,412 2
Depleted Uranium Hexafluoride 6
Conversion Facility 346 346 0
Chemistry and Metallurgy
Research Facility Replacement 837 837 0
Microsystems and Engineering
Sciences Applications 518 518 0
Linac Coherent Light Source 379 379 0

Table 2: Changes in Estimated Project Schedules for DOE Major Construction
Projects

Year mission Initial Current Schedule
need was completion completion delay as of
Project approved date estimate date estimate February 2007
Pit Disassembly and 1997 06/2005 03/2017 11 years, 9
Conversion Facility months
Mixed Oxide Fuel 1997 09/2004 03/2016 11 years, 6
Fabrication months
Facility
Waste Treatment and 1995 07/2011 11/2019 8 years, 4
Immobilization months
Plant
National Ignition 1993 10/2003 03/2009 5 years, 5
Facility months
Depleted Uranium 2000 03/2006 06/2008 2 years, 3
Hexafluoride 6 months
Conversion
Facility^a
Salt Waste 2001 07/2009 09/2011^b 2 years, 2
Processing Facility months
Tritium Extraction 1995 06/2005 07/2007 2 years, 1
Facility month
Highly Enriched 1999 04/2008 03/2010 1 year, 11
Uranium Materials months
Facility
Spallation Neutron 1996 09/2005 06/2006^c 9 months
Source
Chemistry and 2002 03/2014 03/2014 Not
Metallurgy Research applicable
Facility
Replacement
Microsystems and 1999 01/2009 01/2009 Not
Engineering applicable
Sciences
Applications
Linac Coherent 2001 03/2009 03/2009 Not
Light Source applicable

Source: GAO analysis of DOE data.

aThis project reported a schedule delay but did not report an increase in
the estimated total project cost (TPC). According to the DOE project
director, the original cost estimate was probably too high and was not
well supported.

bAccording to DOE officials, schedule may slip further when the contractor
submits its revised TPC to DOE in July 2007.

cProject was completed on this date. Transition to operations has begun.

As table 3 shows, ineffective DOE project oversight and poor contractor
management were frequently cited reasons for cost increases and schedule
delays for the projects we reviewed, according to our review of
independent studies of the 9 projects experiencing cost growth and
schedule delays and our follow-up interviews with DOE project directors.
Project officials, in commenting on our draft report, were concerned that
table 3 might misrepresent the overall successful execution and completion
of some projects, such as the Spallation Neutron Source, and that some
problems may have already been addressed. Nevertheless, to clarify our
main purpose for table 3, our intent is to show broad categories of major
reasons for cost increases and schedule delays, regardless of when they
occurred or whether they have been adequately addressed.

Table 3: Reasons for Cost Increases and Schedule Delays

Poor External factors
DOE project contractor (e.g.,
Project oversight management safety/security)
Depleted Uranium Hexafluoride 6
Conversion Facility X X X
Highly Enriched Uranium Materials
Facility X X X
Mixed Oxide Fuel Fabrication
Facility X X X
National Ignition Facility X X
Pit Disassembly and Conversion
Facility X X X
Salt Waste Processing Facility X X
Spallation Neutron Source X X
Tritium Extraction Facility X X X
Waste Treatment and Immobilization
Plant X X X
Total 9 8 7

Source: GAO analysis of independent project studies and interviews with
DOE project directors (a list of the project studies we reviewed is
included in app. III).

The DOE project oversight issues mentioned in table 3 include the
following:

o inadequate systems for measuring contractor performance;
o approval of construction activities before final designs were
sufficiently complete;
o ineffective project reviews;
o insufficient DOE staffing and experience;
o inadequate use of project management controls;
o lack of headquarters assistance and oversight support of field
project directors;
o failure to detect contractor performance problems, including
inadequate federal inspection activities; and
o poor government cost estimates, including inadequate funding for
contingencies.

DOE's lack of adequate systems to measure contractor performance
was cited in a December 2005 DOE Inspector General review of the
Mixed Oxide Fuel Fabrication Facility. The Inspector General
criticized DOE's NNSA for failing to approve a baseline against
which to measure contractor performance and relying on outdated
cost plans.^11 According to the report, NNSA relied on confusing
and misleading information detailed in the monthly project reports
to monitor progress and track costs--reports that the contractor
acknowledged as being "useless for evaluating performance or
managing the project." Furthermore, although the contractor
reported unfavorable cost and schedule variances for months, these
variances were inaccurate and meaningless because performance was
being compared against a 2-year-old plan. NNSA, in commenting on
our draft report, stated that project oversight and contractor
management problems identified in previous GAO, Inspector General,
and other independent assessments have led to extensive
improvements to the project, and that major findings identified
during a recent independent review have been successfully
addressed.

Similarly, DOE's approval of construction activities before final
designs were sufficiently complete has contributed significantly
to project cost growth and schedule delays. As we have previously
reported, the accelerated fast-track, design-build approach used
for the Waste Treatment and Immobilization Plant, a highly complex
first-of-a-kind nuclear facility, resulted in significant cost
increases and schedule delays.^12 DOE also allowed the contractor
on another project, the Tritium Extraction Facility, to begin
construction before the final design was completed to meet
schedule commitments. According to a 2002 DOE Inspector General
report on the project,^13 this revised acquisition strategy of
simultaneous design and construction directly resulted in at least
$12 million in project overruns.

The contractor management issues mentioned in table 3 include the
poor management of technological challenges, among other
contractor performance issues, according to DOE project directors.
Cost increases and schedule delays for 6 of the 9 projects were
due in part to contractors' poor management of the development and
integration of technologies used in project designs by, among
other things, not accurately anticipating the cost and time that
would be required to carry out the highly complex tasks
involved.^14 For example:

o The National Ignition Facility had over $1 billion in cost
overruns and years of schedule delays, in large part because of
technology integration problems. The requirements for the National
Ignition Facility--the use of 192 high-power laser beams focused
on a single target in a "clean room" environment--had not been
attempted before on such a large scale. According to the DOE
project director, early incorrect assumptions about the original
facility design and the amount of work necessary to integrate the
technologies and assemble the technical components contributed to
about half of the project's cost increases and schedule delays.
o The design of the Mixed Oxide Fuel Fabrication Facility has
presented technical challenges in adapting the design of a similar
plant in France to the design needs of this project. Although the
technological challenge related to adopting the process designs
from the French designs was not the primary contributor to the
project's cost increases and schedule delays, according to NNSA
officials, it has affected the project's complexity. The basic
technology--combining plutonium oxide with depleted uranium to
form fuel assemblies for use in commercial power reactors--has
been previously demonstrated in France. However, the DOE project
director told us that the DOE facility design must, among other
things, account for processing surplus weapon-grade plutonium, a
different type of material than processed in the French facility,
and must be adapted to satisfy U.S. regulatory and other local
requirements. In addition, the DOE facility faced the
technological challenge of reducing the scale of components used
in the French facility. Although definitive cost estimates are not
yet available, expected costs for completing this project have
grown by about $3.3 billion since 2002, and the schedule has been
extended by more than 11 years, in part because the contractor did
not initially understand the project's complexity and
underestimated the level of effort needed to complete the work.
NNSA explained that the capability of the reference plants
currently in operation in France, and by extension, the Mixed
Oxide Fuel Fabrication Facility process design, is currently being
demonstrated by several prototype fuel assemblies manufactured
with weapon-grade plutonium oxide, which are currently being
successfully used in a reactor in South Carolina.
o For the Waste Treatment and Immobilization Plant, a technology
application used on the project had not been tested before
construction. Filters, widely used in the water treatment
industry, were being designed for the project to concentrate and
remove radioactive particles in liquid waste, a new application
for the filters. Although tests are currently under way to
demonstrate the effectiveness of this application, project
officials conceded that these filters may still not be appropriate
for the project.

Other contractor performance problems are illustrated by two
examples. First, DOE cited the contractor working on the Highly
Enriched Uranium Materials Facility for inadequate quality
assurance that resulted in concrete work that did not meet design
specifications. The subsequent suspension of construction
activities and rework added to the project's estimated cost and
schedule. Second, the DOE project director of the Depleted Uranium
Hexafluoride 6 Conversion Facility told us that the project was
delayed 2 years because the contractor (1) did not have experience
in government contracts, (2) underestimated the design effort
needed, and (3) failed to properly integrate the operations of
three separate organizations it managed.

As table 3 shows, external factors were cited as also contributing
to cost growth and schedule delays, such as additional work to
implement requirements for higher levels of safety and security in
project operations, among other things. For example, design rework
for 4 of the projects occurred in response to external safety
oversight recommendations by the Defense Nuclear Facilities Safety
Board that large DOE construction projects meet a certain level of
personnel safety, and that their designs be robust enough to
withstand certain seismic events. In addition, owing to new
security requirements implemented after September 11, 2001,
project officials on the Highly Enriched Uranium Materials
Facility had to redesign some aspects of the project to ensure
that heightened security measures were addressed.

While DOE faced additional requirements for safety and security,
it did not always reach timely decisions on how to implement these
requirements, which contributed significantly to cost increases
and schedule delays for the Salt Waste Processing Facility. The
DOE project director for this project told us the Defense Nuclear
Facilities Safety Board had expressed concerns in June 2004, 5
months after the preliminary design was started, that the facility
design might not ensure nuclear wastes would be adequately
contained in the event of earthquakes. However, DOE did not decide
how to address this concern until 17 months later, as the project
continued to move forward with the existing project design.
According to the project director, better and more timely
discussions between site officials and headquarters to decide on
the actions needed to adequately address these safety and security
requirements might have hastened resolution of the problem, and up
to 1 year of design rework might have been avoided. The delay, the
director told us, added $180 million to the total project cost and
extended the schedule by 26 months. In commenting on our draft
report, EM officials noted that it is now requiring a more
rigorous safety analysis earlier in the decision-making process.

Other external factors also contributed significantly to cost
increases and delays for 2 interrelated projects we reviewed--the
Mixed Oxide Fuel Fabrication Facility and the Pit Disassembly and
Conversion Facility. Project officials for these projects told us
that 25 to 50 percent of the cost increases and over 70 percent of
the schedule delays they experienced were the direct result of
Office of Management and Budget funding constraints and
restrictions resulting from international agreements with Russia.
That is, work that is delayed to a subsequent year because of
funding constraints and other work restrictions can delay project
completion, which likely increases total project costs. Similarly,
Office of Science officials, commenting on our draft report,
stated that external factors caused the largest percentage cost
increase and schedule delay for the Spallation Neutron Source,
including a reduced level of funding appropriated at a time when
project activities and costs were increasing considerably.
However, congressional funding was reduced in fiscal year 2000
because of concerns about poor project oversight and management in
the early stages of this project.

DOE officials also explained that a now-defunct policy may have
contributed to increased costs and delays for several projects we
examined. Until 2000, DOE required contractors to prepare cost and
schedule estimates early in the project, before preliminary
designs were completed. These estimates were used to establish a
baseline for measuring contractor performance and tracking any
cost increases or schedule delays. However, these estimates often
were based on early conceptual designs and, thus, were subject to
significant change as more detailed designs were developed. To
improve the reliability of these estimates, DOE issued a new order
in October 2000 that required the preparation of a cost estimate
range at the start of preliminary design, and delayed the
requirement for a definitive cost and schedule baseline estimate
until after the preliminary design was completed. Consequently,
DOE officials explained, the new policy should result in improved
estimates and a more accurate measure of cost and schedule
performance.

We also sent a survey to DOE project directors for all 12 projects
asking them to identify key events that led to the greatest cost
increases or schedule delays, and the major factors contributing
to these key events. However, no individual factors were
identified as being major contributors to the cost increases or
schedule delays. In responding to our survey, DOE project
directors cited several factors that affected changes in cost and
schedule. However, when asked to rate the relative significance of
these factors for their impact on cost and schedule changes, the
project directors generally did not judge them to be significant
contributors to the changes. The most frequently cited factors
were

o an absence of open communication, mutual trust, and close
coordination;
o changes in "political will" during project execution (e.g.,
project changes resulting from political decisions, both internal
and external to the project);
o interruptions in project funding; and
o project managers' lack of adequate professional experience.

(For detailed survey results covering these four factors, see app.
IV.)

In contrast to the cost increases and schedule delays incurred on
most of the projects we reviewed, 3 projects had not yet
experienced cost increases or schedule delays--Microsystems and
Engineering Sciences Applications, the Linac Coherent Light
Source, and the Chemistry and Metallurgy Research Facility
Replacement. DOE project officials identified key conditions that
they believed helped avoid those cost increases and delays. These
conditions included

o active oversight--that is, the DOE project directors were never
"blindsided" by contractor issues;
o a lack of technological complexity;
o an effective system to measure contractor performance;
o reliable cost estimates;
o effective communication with and integration of all
stakeholders; and
o sustained leadership.

However, we observed that the Linac Coherent Light Source and the
Chemistry and Metallurgy Research Facility Replacement facilities
are still in a relatively early stage in the project development
process, and thus it may be too early to gauge the overall success
of either project. Additionally, because none of these 3 projects
are highly technologically complex, they may be less susceptible
to the types of problems associated with other projects we
reviewed that experienced cost increases and delays.

DOE Does Not Consistently Measure Technology Readiness to Ensure
That Critical Technologies Will Work as Intended before
Construction Begins

Although DOE requires its final designs to be sufficiently
complete before beginning construction, it has not systematically
ensured that the critical technologies reflected in project
designs are technologically ready. Recognizing that a lack of
technology readiness can result in cost overruns and schedule
delays, other federal agencies, such as NASA and DOD, have issued
guidance for measuring and communicating technology readiness.

DOE Does Not Consistently Assess Technology Readiness

Only 1 of the 5 projects we reviewed to determine how DOE ensures
that project designs are sufficiently complete before
construction--projects that were approaching or had recently begun
construction--had a systematic assessment of technology readiness
to determine whether the project components would work
individually or collectively as expected in the intended
design.^15 Specifically, the DOE project director for the Pit
Disassembly and Conversion Facility systematically measured and
assessed readiness levels for each critical component of the
overall project.^16 The assessment was based on a method developed
by NASA, that is, rating each technology from 0 to 10 in terms of
relative maturity. Because the project has not yet begun
construction, we could not determine whether the technology
readiness assessment has helped project managers to avoid cost
increases or schedule delays during construction. However,
according to DOE and contractor officials responsible for the
project, the assessment helped focus management attention during
project design on critical technologies that may require
additional resources to ensure that they are sufficiently ready
before construction begins. In reviewing the assessment, however,
we noted that project officials had not updated the assessment
tool for this project for over 3 years. DOE's project director
acknowledged the delay in updating the assessment and responded
that he plans to begin updating the assessment annually.

The other 4 projects did not have systematic assessments of
technological readiness. Therefore, the risk associated with the
technology may not be clearly and consistently understood across
all levels of management. Formally approving the project's cost
and schedule estimates as accurate and complete, or proceeding
into construction, without having clearly assessed evidence of
technology readiness can result in cost overruns and schedule
delays.

DOE's experience with the Waste Treatment and Immobilization Plant
is a case in point. Specifically, technology known as "pulse jet
mixers"^17 was used in the design of a subsystem intended to
prepare radioactive material for processing. However, this
technology had not been used previously in this application, and
it did not work in tests as expected, even after construction had
already begun. Consequently, DOE incurred about $225 million in
redesign costs and over 1 year in schedule delays, according to
the DOE project director.

Over the past several years, we and others have stressed the
importance of assessing technology readiness to complete projects
successfully, while avoiding cost increases and schedule delays.
Specifically, by 1999, we reported that organizations using best
practices recognize that delaying the resolution of technology
problems until production or construction can result in at least a
10-fold cost increase.^18 Furthermore, we reported that delaying
the resolution until after the start of production could increase
costs by 100-fold. Reporting on similar concerns, the National
Research Council has identified factors common to large
construction projects--in the areas of cost, schedule, and
scope--that help to ensure projects are completed successfully.^19
Among key technical conditions for defining project scope, the
council stated, is a project plan that is based on employing the
best available, state-of-the-art technology, but not experimental
or unproven technology. As such, employing a consistent,
systematic method for measuring the extent to which technology is
still experimental or unproven is of critical importance.

An assessment of technology readiness is even more crucial at
certain points in the life of a project--particularly as DOE
decides to accept a project's (1) preliminary design and formally
approve the project's cost and schedule estimates as accurate and
complete and (2) final design as sufficiently complete so that
resources can be committed toward procurement and construction.
Proceeding through these critical decision points without a
credible and complete technology readiness assessment can lead to
problems later in the project. Specifically, if DOE proceeds with
the project when technologies are not yet ready, there is less
certainty that the technologies specified in the preliminary or
final designs will work as intended. Project managers may then
need to modify or replace these technologies to make them work
properly, which can result in costly and time-consuming redesign
work.

Moreover, modifying the design of a facility after construction
has already begun can be expensive and time consuming. First,
changes to an already designed work plan are not necessarily
subject to competition because the new work can occur through
"change orders"--that is, modifications to existing contracts.
These change orders can be expensive, according to DOE project
directors. Second, worker productivity can be lost if, for
example, extra downtime results from delays to interrelated
construction work. Finally, tearing down and rebuilding items
already constructed, such as concrete floors, walls, and doors,
might be necessary to accommodate a design change.

DOE's experience in the predecessor project to the Salt Waste
Processing Facility--the In-Tank Precipitation (ITP) project
process--at the Savannah River Site illustrates the potential
consequences of proceeding with technology that is not
sufficiently ready. As we reported in 2000, the ITP project was
selected in 1983 as the preferred method for separating highly
radioactive material from 34 million gallons of liquid stored at
the Savannah River site--a step considered necessary to
effectively handle this large quantity of waste.^20 A 1983 test
using the ITP technology on a tank containing 500,000 gallons of
waste resulted in a significant buildup of benzene--a highly
explosive and hazardous compound. The buildup of benzene was more
than the tank instruments could register. Nevertheless, project
managers decided to proceed with the project. In 1985, DOE
estimated that it would take about 3 years and $32 million to
construct the ITP facility. After a number of delays, the ITP
facility was constructed and began start-up operations in 1995,
which were halted because of safety concerns about the amount of
benzene that the facility generated. In 1998, after about a decade
of delays and costs of almost $500 million, DOE suspended the
project because it did not work as safely and efficiently as
designed. This suspension put an effective remedy for treating
high-level waste at the Savannah River Site years behind schedule.
DOE then directed its contractor to begin a process to identify
and select an alternative technology, which has developed into the
current project intended to treat this waste--the Salt Waste
Processing Facility project.

In response to our concerns about the 4 projects without
systematic assessments of technology readiness, DOE project
directors explained that they have alternative methods for
assessing readiness. They are required to submit a project
execution plan, which includes an assessment of risks, including
technological risks, and a plan for mitigating risks. They also
rely upon independent reviews, including extensive design reviews,
before making critical decisions to accept designs, and cost and
schedule estimates, or to proceed with construction. For example,
DOE's Office of Engineering and Construction Management formally
reviews major projects in an effort to ensure that the designs are
sufficiently complete to begin construction. Specifically, an
external independent readiness review is performed, often using
the services of various independent technical experts, that, at a
minimum, is intended to verify the readiness of the project to
proceed into construction or to identify remedial action. Finally,
several DOE project directors stated that they intentionally have
avoided using fast-track, design-build approaches because of the
many problems it posed for the Waste Treatment and Immobilization
Plant project. The DOE project directors of the 5 DOE projects
that are nearing, or have recently begun construction, told us
they have completed, or expect to complete prior to construction,
85 to 100 percent of their projects' final design.

In addition to following the more standard approaches for managing
projects, such as preparing risk assessment plans, some DOE
offices have developed their own tools for assessing the readiness
of projects. For example, DOE's Office of Environmental Management
(EM) uses a Product Definition Rating Index (PDRI) as a tool to
assess how well a project is planned, and whether it is ready to
proceed to the next project phase. Project elements rated include
cost, schedule, scope/technical, management planning and control,
and external factors. Among the 77 project elements rated, 2
involve technology--the identification of technology development
requirements, and the testing and evaluation of the technology to
be used. While the project technologies are collectively given a
ranking with this tool, the PDRI does not represent a rigorous
examination of the demonstrated readiness of each critical
technology for its application in the project. Furthermore, not
all EM projects we examined were using this tool.

DOE's design reviews, risk assessments, and other actions to
monitor design completion are extensive and certainly have merit.
However, we found that these actions alone do not provide
consistent and transparent assurance that all technologies are
sufficiently ready because they do not use a consistent and
systematic method of measurement. DOE's project design reviews,
for example, do not always clearly distinguish between technology
that has been demonstrated to work as expected in the intended
design versus a judgment that the technology has potential for
reaching a specific level of readiness.

The external review of the technologies for the Mixed Oxide Fuel
Fabrication Facility illustrates the shortfalls in DOE's current
approach to assessing technology readiness and communicating the
results of those assessments.^21 The report concluded, among other
things, that the method chosen by the contractor is the most
rigorous and comprehensive, and should result in the most
successful technology transfer possible. Furthermore, the review
team was very impressed with the rigor with which designs and
design changes were being managed, finding ample evidence
verifying that the exact design process used by the French was
being transferred to the United States facility. Although the
external reviewers seemed to be impressed with many aspects of the
design transfer, and concluded that the technologies should not be
problematic, they had identified some key concerns about
technology readiness in the body of their final report. The
reviewers did not explain how they reconciled their conclusion
with their concerns. To reconcile these differences, we obtained
several clarifying statements from DOE's project director,
technical experts, and one of the study's authors. These
clarifying statements appear to support the reviewers'
conclusions. However, without these statements, the level of
technological readiness was not readily evident because the
independent review lacked consistent, systematic criteria and a
method for measuring the degree of readiness or clearly
communicating assessment results, and the review was not
transparent.

DOE does not consistently assess technology readiness of project
technologies because its project management guidance lacks
comprehensive standards for systematically measuring and
communicating the readiness of project technologies. Specifically,
DOE lacks consistent metrics for determining technology readiness
departmentwide, terminology to facilitate effective communication,
and oversight protocols for reporting and reviewing technology
readiness levels. DOE project management guidance is contained in
two key documents--DOE Order 413.3A and Manual 413.3-1. Although
the manual requires final designs to be sufficiently complete
before beginning construction, it does not specify how
technologies reflected in project designs are to be assessed for
readiness--to determine that they have been sufficiently
demonstrated to work as intended. Consequently, critical decisions
made without standard measures are susceptible to varying
interpretations of the actual technology readiness attained and
the level needed for a project to proceed, which can easily vary
among projects and among officials within a single project.

Other Federal Agencies Use a Standard Method for Measuring and
Communicating Technology Readiness

Other federal agencies have recognized the importance of ensuring
that technologies have been sufficiently demonstrated for their
intended purpose and have issued standard guidance for measuring
and communicating TRLs. In particular, recognizing the need to
measure the readiness level of project technologies, NASA began
using a systematic method of measurement in the mid-1990s. NASA
incorporated a structured TRL approach into guidance on integrated
technology planning.

Similarly, to improve DOD management of risk and technology
development, the Deputy Under Secretary of Defense (Science and
Technology) officially endorsed, in a July 2001 memorandum, the
use of TRLs in new major programs. In 2002, DOD issued mandatory
procedures for major defense acquisition programs and major
automated information system acquisition programs, which
identified technology readiness as a principal element of program
risk. The procedures require the military services' science and
technology officials to conduct a systematic assessment of
critical technologies that are identified in major weapon systems
programs before starting engineering and manufacturing development
and production. Using TRLs is the preferred method, and approval
must be obtained from the Deputy Under Secretary if an equivalent
alternative method is used, according to the Deputy Under
Secretary's memorandum. Importantly, the procedures stated that
TRLs are a measure of demonstrated technical maturity--they do not
discuss the probability of occurrence (i.e., the likelihood of
attaining required maturity) or the impact of not achieving
technology maturity.

Both NASA and DOD use a nine-point scale to measure technology
readiness, from a low of TRL 1 (basic principles observed) to a
high of TRL 9 (total system used successfully in project
operations). (App. V contains the definitions of these nine TRLs.)
For example, a subsystem prototype that has been successfully
demonstrated in an operational environment would receive a higher
TRL value (i.e., TRL 7) than a technological component that has
been demonstrated in a laboratory test (i.e., TRL 4). In our
previous work, we recommended to the Secretary of Defense that key
project technologies used in weapons systems be demonstrated in an
operational environment, reaching a high maturity level--analogous
to TRL 7--before deciding to commit to a cost, schedule, and
performance baseline for development and production of the weapon
system.^22 In response to our recommendation, DOD has agreed that
if a technology does not achieve a score of TRL 6 or 7, project
managers must develop a plan to bring the technology to the
required readiness level before proceeding to the next project
phase.

Use of TRLs is not by itself a cure-all for managing critical
technologies, but TRLs can be used in conjunction with other
measures to improve the way projects are managed. For example,
according to studies by NASA, DOD, and others, TRLs can

o provide a common language among the technology developers,
engineers who will adopt/use the technology, and other
stakeholders;
o improve stakeholder communication regarding technology
development--a by-product of the discussion among stakeholders
that is needed to negotiate a TRL value;
o reveal the gap between a technology's current readiness level
and the readiness level needed for successful inclusion in the
intended product;
o identify at-risk technologies that need increased management
attention or additional resources for technology development to
initiate risk-reduction measures; and
o increase transparency of critical decisions by identifying key
technologies that have been demonstrated to work or by
highlighting still immature or unproven technologies that might
result in high project risk.

Two DOE headquarters offices have attempted to systematically
assess technology readiness. First, under the Office of Nuclear
Energy, a DOE contractor preparing a congressional report used a
TRL method to compare the maturity of advanced fuel cycle
technologies. In addition, in 2000, DOE's Office of Science and
Technology, under EM, issued a report that defined a process for
assessing technology maturity of EM projects.^23 However,
according to an EM official, the office decided to discontinue
using this assessment process because it was considered overly
burdensome. As a result, DOE devolved responsibility for managing
technology readiness to the contractor level.

According to several DOE project directors we spoke with, a
consistent, systematic method for assessing technology readiness
would help achieve a number of objectives: that is, standardize
terminology, make technology assessments more transparent, and
improve communication among project stakeholders before they make
critical project decisions. DOE project managers also acknowledged
that TRLs could improve project management departmentwide, and
some managers are now attempting to use this tool to assess
technology maturity. The DOE project director for the Waste
Treatment and Immobilization Plant told us that a senior DOE
official encouraged him to begin using TRLs. He is consulting with
DOD officials knowledgeable about using the TRL method and expects
to develop a TRL tool and have TRL determinations for major parts
of the project in 2007. (App. VI compares DOD's product
development process with DOE's project management process for
major projects.)

Conclusions

The magnitude of the cost increases and schedule delays for DOE's
major projects is cause for serious rethinking of how DOE manages
them. To its credit, DOE has completed, or expects to complete
prior to construction, 85 to 100 percent of project design work
for the 5 projects we reviewed that have recently begun or are
nearing construction. However, DOE has not systematically
addressed another key factor--the readiness level of the
technologies it expects to use in these projects. DOE lacks
comprehensive standards in DOE Order 413.3A and Manual 413.3-1 for
systematically measuring and communicating the readiness of
project technologies. Specifically, the department lacks
consistent metrics for determining technology readiness
departmentwide, terminology, and oversight protocols for reporting
and reviewing TRLs. Without consistent measurement and
communication of the readiness of technologies, DOE does not have
a basis for defining the acceptable level of technological risk
for each project, making critical decisions on accepting the
validity of a project's total estimated cost and schedule, or
proceeding with construction.

Other federal agencies have recognized the need to consistently
measure and communicate technology readiness to help avoid cost
increases and delays that result from relying on immature
technologies. DOD, for example, requires its managers to use a TRL
process to measure technology readiness and generally requires a
TRL 7 (as we had recommended) before system development and
demonstration. In contrast, as DOE's poor track record for
managing the technological complexity of major projects shows, DOE
has not systematically measured the readiness of critical project
technologies before it approves definitive cost and schedule
estimates or begins construction. Furthermore, without a
systematic method for measuring technological readiness, DOE
cannot effectively communicate within the department and to the
Congress whether projects are at risk of experiencing cost
increases and schedule delays associated with technology problems.

Recommendations for Executive Action

To improve decision making and oversight for major DOE
construction projects, including how project technology readiness
is measured and reported, we recommend that the Secretary of
Energy evaluate and consider adopting a disciplined and consistent
approach to assessing TRLs for projects with critical technologies
that includes the following three actions:

o Develop comprehensive standards for systematically measuring and
communicating the readiness of project technologies. At a minimum,
these standards should (1) specify consistent metrics for
determining technology readiness departmentwide, (2) establish
terminology that can be consistently applied across projects, and
(3) detail the oversight protocols to be used in reporting and
reviewing TRLs. In preparing these standards, DOE should consider
lessons learned from NASA and DOD, and its own experience in
measuring technology readiness. If DOE's evaluation results in the
decision to adopt these standards, it should incorporate them into
DOE Order 413.3A and Manual 413.3-1, and provide the appropriate
training to ensure their proper implementation.
o Direct DOE Acquisition Executives to ensure that projects with
critical technologies reach a level of readiness commensurate with
acceptable risk--analogous to TRL 7--before deciding to approve
the preliminary design and commit to definitive cost and schedule
estimates, and at least TRL 7 or, if possible, TRL 8 before
committing to construction expenses.
o Inform the appropriate committees and Members of Congress of any
DOE decision to approve definitive cost and schedule estimates, or
to begin construction, without first having ensured that project
technologies are sufficiently ready (at TRL 7 or 8). This
information should include specific plans for mitigating
technology risks, such as developing backup technologies to offset
the effects of a potential technology failure, and appropriate
justification for accepting higher technological risk.

Agency Comments and Our Evaluation

We provided a draft of this report to DOE for its review and
comment. DOE's written comments are reproduced in appendix VII.
DOE agreed with our recommendations but suggested revisions that
would allow it to first conduct a pilot application on selected
projects to better understand the technology readiness assessment
process and evaluate its potential use. We revised our
recommendations to give DOE this flexibility. DOE also provided
detailed technical comments, which we have incorporated into our
report as appropriate.

DOE also expressed several specific concerns with our draft
report. First, DOE stated that while our draft broadly asserts
that DOE project management has led to increases in cost and
schedule, our recommendations are narrowly focused on technology
assessment. We agree that our draft states that DOE project
management has led to cost increases and schedule delays, a
conclusion we reached on the basis of our contact with DOE project
directors and our review of numerous studies and reports on DOE
major projects. Our recommendations address technology assessment,
a critical project management activity, because they were
developed primarily on the basis of our specific finding that DOE
lacks a systematic approach to ensure that final project designs,
including critical technologies reflected in these designs, have
been demonstrated to work as intended prior to construction. This
report explains that delaying resolution of technology problems
until construction can potentially lead to significant cost
increases and schedule delays.

Second, DOE stated that our draft report inappropriately
characterizes cost and schedule growth from a small sample of
projects by using preliminary cost and schedule estimates that are
intended for internal DOE planning. To clarify, the scope of our
review included an evaluation of DOE's major construction
projects. In addition, our report explains that DOE changed its
project management policy in 2000 to allow cost and schedule
estimates to be prepared later in the project--at the end of
preliminary design. Prior to this new policy, project directors
submitted cost and schedule estimates earlier in the project
development phase--at the end of conceptual design. For projects
under way prior to the policy in 2000, we used post-2000 validated
baseline estimates, if available. Otherwise, we used earlier
estimates since these were the only estimates available and had
been previously used by DOE to inform Congress of the total
expected project cost and schedule while seeking initial project
funding. We also note that for the five projects that were started
after the new policy in 2000, we used the validated project
baseline estimates recommended by DOE, if available.

Third, DOE suggested we revise table 3 in our report to more
clearly identify the correlation between cost and schedule growth
and technology maturity. As our report states, the information in
table 3 was drawn from the results of our review of independent
studies involving the projects we reviewed and the results of our
interviews with DOE project directors. Our report explains that
cost increases and schedule delays for 6 of the 9 projects shown
in the table were due in part to contractors' poor management of
the development and integration of technologies used in the
project designs.

Finally, DOE stated that it is unclear how the factors cited in
appendix IV, such as communication, and changes in "political
will," among other things, led to our recommendation to assess
technology readiness. Although not all of the factors cited in our
survey have a link to our recommendation on technology readiness,
one factor in particular--absence of communication--is addressed
in our recommendation. Specifically, we recommended that the
Secretary of Energy consider developing comprehensive standards
for systematically measuring and communicating the readiness of
project technologies, including the establishment of terminology
that is to be consistently applied across projects.

^11Department of Energy, Office of Inspector General, Audit Report: Status
of the Mixed Oxide Fuel Fabrication Facility, DOE/IG-0713 (Washington,
D.C.: December 2005).

^12 [60]GAO-06-602T .

^13Department of Energy, Office of Inspector General, Audit Report: The
Department of Energy's Tritium Extraction Facility, DOE/IG-0560
(Washington, D.C.: June 2002).

^14These 6 projects are the Mixed Oxide Fuel Fabrication Facility,
National Ignition Facility, Pit Disassembly and Conversion Facility,
Spallation Neutron Source, Tritium Extraction Facility, and Waste
Treatment and Immobilization Plant.

^15These 5 projects are the Chemistry and Metallurgy Research Facility
Replacement, Depleted Uranium Hexafluoride 6 Conversion Facility, Mixed
Oxide Fuel Fabrication Facility, Pit Disassembly and Conversion Facility,
and Salt Waste Processing Facility.

^16Los Alamos National Laboratory, Options for the Development and Testing
of the Pit Disassembly and Conversion Facility Government-Furnished
Design, LA-UR-03-3926 (Los Alamos, New Mexico: June 11, 2003).

^17Pulse jet mixers, which do not have moving parts, use compressed air to
continuously mix tank waste so that it can be properly prepared for
further processing. While such devices have previously been used
successfully in other applications, they have never been used for mixing
wastes with high-solid content like those at the Waste Treatment and
Immobilization Plant.

^18 [61]GAO/NSIAD-99-162 and GAO, Joint Strike Fighter Acquisition: Mature
Critical Technologies Needed to Reduce Risks, [62]GAO-02-39 (Washington,
D.C.: Oct. 19, 2001).

^19Improving Project Management.

^20GAO, Department of Energy: Uncertainties and Management Problems Have
Hindered Cleanup at Two Nuclear Waste Sites, [63]GAO/T-RCED-00-248
(Washington, D.C.: July 12, 2000).

^21Burns and Roe Enterprises, Inc., External Independent Review of the
Basis of Design for the Aqueous Polishing Process for the Mixed Oxide Fuel
Fabrication Facility at The Savannah River Site for the U.S. Department of
Energy Office of Engineering and Construction Management and National
Energy Technology Laboratory Report, BREI-LSP-R-06-01 (Oradell, New
Jersey: March 2006).

^22 [64]GAO/NSIAD-99-162 .

^23Department of Energy, Tracking Technology Maturity in DOE's
Environmental Management Science and Technology Program; Revision 1
(Washington, D.C.: Jan. 1, 2001).

We are sending copies of the report to interested congressional
committees, the Secretary of Energy, and the Director of the
Office of Management and Budget. We will make copies available to
others on request. In addition, the report will also be available
at no charge on the GAO Web site at http://www.gao.gov .

If you or your staffs have any questions about this report, please
contact me at (202) 512-3841 or [email protected]. Contact points
for our Offices of Congressional Relations and Public Affairs may
be found on the last page of this report. Other staff contributing
to the report are listed in appendix VIII.

Gene Aloise
Director, Natural Resources and Environment

Appendix I: Scope and Methodology

To determine the extent to which the Department of Energy's (DOE)
major construction projects have experienced cost increases and
schedule delays and the factors that have contributed to these
problems, we identified (1) active DOE major line-item
construction projects that have current total project cost
estimates above the $750 million threshold--DOE's criteria for
"major construction projects," and (2) the projects with estimates
above $400 million--the DOE threshold for major projects until
July 2006. We also identified those projects above $300 million to
account for any projects that may pass the $400 million
threshold.^1 In all, we identified the following 12 projects:

o Five of these 12 projects began before DOE moved its requirement
for firm cost and schedule estimates to later in the project: the
National Ignition Facility, the Mixed Oxide Fuel Fabrication
Facility, the Pit Disassembly and Conversion Facility, the
Spallation Neutron Source, and the Tritium Extraction Facility. We
used the estimates at the end of conceptual design, as reported by
project directors, for the initial project cost and schedule
estimates.
o Four of the remaining 7 projects had cost and schedule estimates
completed at the end of preliminary design, according to the new
DOE guidelines: the Highly Enriched Uranium Materials Facility,
Microsystems and Engineering Sciences Applications, the Depleted
Uranium Hexafluoride 6 Conversion Facilities, and the Linac
Coherent Light Source. For these projects, we considered the
estimates as reported by project directors to be the initial cost
and schedule estimates.
o One project, the Waste Treatment and Immobilization Plant, began
after DOE moved the requirement for firm cost and schedule
estimates to later in the project. However, DOE initially exempted
the contractor from submitting firm cost and schedule estimates.
Therefore, we used the estimates reported by the project director
to be the initial cost and schedule estimates.
o The final 2 projects, although falling under the new DOE
requirements, had yet to complete their preliminary design at the
time of our review: the Chemistry and Metallurgy Research Facility
Replacement and the Salt Waste Processing Facility. For these
projects, we considered the cost and schedule estimates at the end
of conceptual design reported by project directors to be the
initial project cost and schedule estimates.

To identify cost increases and schedule delays, and the factors
that may have contributed to these changes, we surveyed DOE
project directors, interviewed DOE and contractor project
personnel, and reviewed project management documents for 12 major
projects. These 12 projects are managed by DOE's Office of
Science, Office of Environmental Management (EM), or National
Nuclear Security Administration (NNSA). (See app. II for
information on these projects.)

Our survey asked DOE project directors of the 12 projects to
identify the degree to which cost and schedule estimates may have
changed and the reasons for these changes, and to describe the
events and conditions that led to any changes. Eight of the 12
project directors responded that their projects had experienced
cost increases and schedule delays, and 1 project director
reported only a schedule delay. For these 9 projects, we asked
project directors to (1) identify the top three events that led to
the cost and schedule delays and (2) indicate to what extent
certain factors may have contributed to the event that led to the
largest percentage cost increase or schedule delay. The factors
included in the survey instrument were based on the results of a
National Research Council study that listed essential or important
conditions needed for the successful completion of major
projects.^3 We asked project directors to identify the extent to
which the lack of these conditions may have contributed to any
cost and schedule delays. (App. IV shows key survey results for
these 9 projects.)

In addition to reviewing project documentation, we conducted site
visits for the 9 projects that had experienced cost and schedule
changes, and we analyzed (1) studies of these projects completed
by DOE's Office of Inspector General and (2) external independent
project reviews conducted under the direction of DOE's Office of
Engineering and Construction Management in Washington, D.C. We
interviewed federal project directors of the 3 projects that had
not experienced cost increases or schedule delays to obtain
information on factors they believe are important in avoiding such
increases.

To determine the extent to which DOE ensures that project designs
are sufficiently complete before construction, we obtained
additional information from project directors on 5 projects that
were approaching, or had recently begun, construction. During our
review, we obtained information on the extent project designs
were, or are expected to be, complete before beginning
construction, and the actions DOE had taken to ensure technologies
used in these designs are sufficiently ready to begin
construction. For 2 of these 5 projects, we applied a tool we
previously had used to assess DOD programs--the tool enables
project directors to characterize the readiness level of each
technology being developed for use in aircraft and other military
applications. In addition, we spoke with officials from DOE
program offices and DOE's Office of Engineering and Construction
Management in Washington, D.C.

We provided interim briefings to the Subcommittee on Energy and
Water Development, House Committee on Appropriations, on the
status of our work in May and September, 2006. We performed our
work between December 2005 and January 2007, in accordance with
generally accepted government auditing standards.

^1We excluded the Yucca Mountain Repository project, with a total
estimated cost of $23 billion, from our review due to its uniqueness and
the fact that we have recently reported on the project and currently have
an ongoing review. Also, to review projects with sufficient maturity, we
included only the projects that were at least 1 year past completion of
conceptual design.

^2GAO, Department of Energy: Further Actions Are Needed to Strengthen
Contract Management for Major Projects, [65]GAO-05-123 (Washington, D.C.:
Mar. 18, 2005); and Civil Engineering Research Foundation, Independent
Research Assessment of Project Management Factors Affecting Department of
Energy Project Success (Washington, D.C.: July 12, 2004).

^3National Research Council, Improving Project Management in the
Department of Energy (Washington, D.C.: July 1999).

Appendix II: Information on the 12 Department of Energy Major
Projects Reviewed

Project DOE program office Project purpose/objectives
Chemistry and National Nuclear Relocate and consolidate
Metallurgy Research Security mission-critical analytical
Facility Replacement Administration chemistry, material
characterization, and research and
development capabilities to ensure
continuous national security
mission support beyond 2010.
Depleted Uranium Office of Design and construct facilities at
Hexafluoride 6 Environmental Portsmouth, Ohio, and Paducah,
Conversion Facility Management Kentucky, to convert the
Department of Energy's existing
inventory of depleted uranium
hexafluoride into a more stable
form for disposal or beneficial
reuse.
Highly Enriched National Nuclear Project will construct a highly
Uranium Materials Security secure, state-of-the-art facility
Facility Administration for consolidating and storing
highly enriched uranium, resulting
in cost savings and an increased
security posture.
Linac Coherent Light Science Provide laser-like radiation in
Source the X-ray region of the spectrum
that is 10 billion times greater
in peak brightness than any
existing X-ray light source. The
project will apply these
high-brightness X-rays to
experiments in the chemical,
material, and biological sciences.
Microsystems and National Nuclear Provide state-of-the-art national
Engineering Sciences Security complex that will provide for the
Applications Administration design, integration, prototyping,
and qualification of microsystems
into components, subsystems, and
systems within the nuclear weapons
stockpile.
Mixed Oxide Fuel National Nuclear Facility will combine surplus
Fabrication Facility Security weapon-grade plutonium oxide with
Administration depleted uranium to form mixed
oxide fuel assemblies that will be
irradiated in United States
commercial nuclear reactors. Once
irradiated and converted into
spent fuel, the resulting
plutonium can no longer be readily
used for nuclear weapons.
National Ignition National Nuclear Provide experimental capability to
Facility Security assess nuclear weapons physics,
Administration providing critical data that will
allow the United States to
maintain its technical
capabilities in nuclear weapons in
the absence of underground
testing, and to advance fusion as
an energy source.
Pit Disassembly and National Nuclear Eliminate surplus Russian and
Conversion Facility Security United States plutonium and highly
Administration enriched uranium by disassembling
surplus nuclear weapons pits and
converting the resulting plutonium
metal to a powder form that can
later be fabricated into mixed
oxide fuel to produce nuclear fuel
assemblies for use in commercial
nuclear reactors.
Salt Waste Office of Meet site cleanup goals and reduce
Processing Facility Environmental significant environmental and
Management health/safety risk by construction
of a facility to treat large
quantities of waste from
reprocessing and nuclear materials
production operations at the
Savannah River Site. Process will
separate waste, solidify it in
glass, and send it to federal
repositories for disposal.
Spallation Neutron Science Provide next generation,
Source short-pulse spallation neutron
source for neutron scattering, to
be used by researchers from
academia, national and federal
labs, and industry for basic and
applied research and technology
development in the fields of
condensed matter physics,
materials sciences, magnetic
materials, polymers and complex
fluids, chemistry, biology, earth
sciences, and engineering.
Tritium Extraction National Nuclear To replenish the tritium needs of
Facility Security the nuclear weapons stockpile, the
Administration facility will extract tritium
produced in a commercial nuclear
reactor for use in nuclear weapons
development.
Waste Treatment and Office of The plant will separate high-level
Immobilization Plant Environmental from low-level radioactive waste
Management currently stored in underground
tanks, processing and solidifying
all high-level waste and a
substantial portion of the
low-level waste, and will treat
the remaining low-level waste.

Source: DOE.

Appendix III: Independent Studies Reviewed

National Ignition Facility

Department of Energy, Office of Inspector General. Audit Report:
Status of the National Ignition Facility Project. DOE/IG-0598.
Washington, D.C.: April 28, 2003.

GAO. Department of Energy: Status of Contract and Project
Management Reforms. [35]GAO-03-570T . Washington, D.C.: March 20,
2003.

GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to
Ensure Initiatives Have Improved Results. [36]GAO-02-798 .
Washington, D.C.: September 13, 2002.

GAO. Department of Energy: Follow-up Review of DOE's National
Ignition Facility. [37]GAO-01-677R . Washington, D.C.: June 1,
2001.

GAO. National Ignition Facility: Management and Oversight Failures
Caused Major Cost Overruns and Schedule Delays.
[38]GAO/RCED-00-141 and [39]GAO/RCED-00-271 . Washington, D.C.:
August 8, 2000.

The Mitre Corporation. NIF Ignition. JSR-05-340. McLean, VA: June
29, 2005.

Mixed Oxide Fuel Fabrication Facility

Burns and Roe Enterprises, Inc. External Independent Review of the
Mixed Oxide Fuel Fabrication Facility (MFFF) Project Critical
Decision (CD) 2/3 Baseline: Performance Baseline (CD-2) and Start
of Construction (CD-3) Review. BREI-L-R-06-03. Oradell, NJ: July
7, 2006.

Burns and Roe Enterprises, Inc. External Independent Review of the
Basis of Design for the Aqueuous Polishing Process.
BREI-SLP-R-06-01. Oradell, NJ: March 27, 2006.

Civil Engineering Research Foundation. Independent Research
Assessment of Project Management Factors Affecting Department of
Energy Project Success. Washington, D.C.: July 12, 2004.

Department of Energy, Office of Inspector General. Audit Report:
Status of the Mixed Oxide Fuel Fabrication Facility. DOE/IG-0713.
Washington, D.C.: December 21, 2005.

Pit Disassembly and Conversion Facility

Department of Energy, Office of Inspector General. Audit Report:
National Nuclear Security Administration's Pit Disassembly and
Conversion Facility. DOE/IG-0688. Washington, D.C.: May 3, 2005.

Los Alamos National Laboratory. Options for the Development and
Testing of the Pit Disassembly and Conversion Facility
Government-Furnished Design. LA-UR-03-3926. Los Alamos, NM: June
11, 2003.

Waste Treatment and Immobilization Plant

Bechtel National, Inc. Hanford Tank Waste Treatment and
Immobilization Plant, May 2006 Estimate at Completion. Hanford
Site, WA: May 31, 2006.

Bechtel National, Inc. Comprehensive Review of the Hanford Tank
Waste Treatment and Immobilization Plant Estimate at Completion.
CCN 132848. Hanford Site, WA: March 31, 2006.

Bechtel National, Inc. Comprehensive Review of the Hanford Waste
Treatment Plant Flowsheet and Throughput. CCN132846. Hanford Site,
WA: March 17, 2006.

Bechtel National, Inc. Hanford Tank Waste Treatment and
Immobilization Plant, December 2005 Estimate at Completion
Executive Summary. Hanford Site, WA: January 30, 2006.

Department of the Army Corp of Engineers. Complete Statement of
Kim Callan, to the Subcommittee on Energy and Water Development,
Committee on Appropriations, United States House of
Representatives. Washington, D.C.: April 6, 2006.

Department of Energy. External Independent Review, Independent
Cost Review, CD-3C Review of the Waste Treatment and
Immobilization Plant Project. Hanford Site, WA: September 2002.

Department of Energy. External Independent Review CD-3B Review of
the Waste Treatment and Immobilization Plant Project. Hanford
Site, WA: April 2002.

GAO. Hanford Waste Treatment Plant, Contractor and DOE Management
Problems Have Led to Higher Costs, Construction Delays, and Safety
Concerns. [40]GAO-06-602T . Washington, D.C.: April 6, 2006.

GAO. Further Actions Are Needed to Strengthen Contract Management
for Major Projects. [41]GAO-05-123 . Washington, D.C.: March 18,
2005.

GAO. Nuclear Waste: Absence of Key Management Reforms on Hanford's
Cleanup Project Adds to Challenges of Achieving Cost and Schedule
Goals. [42]GAO-04-611 . Washington, D.C.: June 9, 2004.

GAO. Status of Contract and Project Management Reforms.
[43]GAO-03-57T . Washington, D.C.: March 20, 2003.

GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to
Ensure Initiatives Have Improved Results. [44]GAO-02-798 .
Washington, D.C.: September 13, 2002.

GAO. Nuclear Waste: Hanford Tank Waste Program Needs Cost,
Schedule, and Management Changes. [45]GAO/RCED-93-99 . Washington,
D.C.: March 8, 1993.

LMI Government Consulting. Hanford Waste Treatment and
Immobilization Plant After-Action Fact-Finding Review. DE535T1.
McLean, VA: January 2006.

LMI Government Consulting. External Independent Review, Follow-up
Review, Waste Treatment and Immobilization Plant (WTP) Out
Briefing. Washington, D.C.: March 14, 2003.

Spallation Neutron Source

Civil Engineering Research Foundation. Independent Research
Assessment of Project Management Factors Affecting Department of
Energy Project Success. Washington, D.C.: July 12, 2004.

Department of Energy, Office of Inspector General. Audit Report:
Progress of the Spallation Neutron Source Project. DOE/IG-0532.
Washington, D.C.: November 19, 2001.

Department of Energy. Review Committee Report on the Baseline
Review of the Spallation Neutron Source (SNS) Project. Washington,
D.C.: July 15, 1999.

Department of Energy. Technical, Cost, Schedule, and Management
Review of the Spallation Neutron Source Project. Washington, D.C.:
January 28, 1999.

GAO. Department of Energy: Status of Contract and Project
Management Reforms. [46]GAO-03-570T . Washington, D.C.: March 20,
2003.

GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to
Ensure Initiatives Have Improved Results. [47]GAO-02-798 .
Washington, D.C.: September 13, 2002.

GAO. Department of Energy: Challenges Exist in Managing the
Spallation Neutron Source Project. [48]GAO/T-RCED-99-103 .
Washington, D.C.: March 3, 1999.

Salt Waste Processing Facility

Department of Energy, Office of Inspector General. Audit Report:
Salt Processing Project at the Savannah River Site. DOE/IG-0565.
Washington, D.C.: August 27, 2002.

Institute for Regulatory Science. Technical Peer Review Report of
the Review Panel on Salt Waste Processing Facility Technology
Readiness. CRTD-Vol. 75. Danvers, MA: October 31, 2003.

Tritium Extraction Facility

Civil Engineering Research Foundation. Independent Research
Assessment of Project Management Factors Affecting Department of
Energy Project Success. Washington, D.C.: July 12, 2004.

Department of Energy, Office of Inspector General. Audit Report:
The Department of Energy's Tritium Extraction Facility.
DOE/IG-0560. Washington, D.C.: June 24, 2002.

GAO. Department of Energy: Further Actions Are Needed to
Strengthen Contract Management for Major Projects. [49]GAO-05-123
. Washington, D.C.: March 18, 2005.

GAO. Department of Energy: Status of Contract and Project
Management Reforms. [50]GAO-03-570T . Washington, D.C.: March 20,
2003.

GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to
Ensure Initiatives Have Improved Results. [51]GAO-02-798 .
Washington, D.C.: September 13, 2002.

GAO. Nuclear Weapons: Design Reviews of DOE's Tritium Extraction
Facility. [52]GAO/RCED-98-75 . Washington, D.C.: March 31, 1998.

National Nuclear Security Administration. Program Review of the
Estimate to Complete Tritium Extraction Facility (TEF) at Savannah
River Site. Washington, D.C.: August 29, 2002.

Highly Enriched Uranium Materials Facility

BWXT Y-12. Highly Enriched Uranium Materials Facility Project
Causal Analysis Report. Oak Ridge, TN: March 6, 2006.

Department of Energy. Limited External Independent Review for
Baseline Change Proposal Review. Oak Ridge, TN: August 31, 2004.

Department of Energy, Office of Inspector General. Audit Report,
Design of the Uranium Storage Facility at the Y-12 National
Security Complex. DOE/IG-0643. Washington, D.C.: March 19, 2004.

Department of Energy, Office of Inspector General. Audit Report,
Reestablishment of Enriched Uranium Operations at the Y-12
National Security Complex. DOE/IG-0640. Washington, D.C.: February
24, 2004.

Department of Energy. External Independent Review - Performance
Baseline Review of the Highly Enriched Uranium Materials Facility
Project. Oak Ridge, TN: June 2003.

Depleted Uranium Hexafluoride 6 Conversion Facility

Department of Energy. Report on the Independent Project Review of
the Depleted Uranium Hexafluoride Conversion Project. Washington,
D.C.: October 8, 2004.

Department of Energy, Office of Inspector General. Audit Report:
Depleted Uranium Hexafluoride Conversion. DOE/IG-0642. Washington,
D.C.: March 18, 2004.

GAO. Department of Energy: Status of Contract and Project
Management Reforms. [53]GAO-03-570T . Washington, D.C.: March 20,
2003.

LMI Government Consulting. DUF6 Conversion Project CD-3 Corrective
Action Plan Review. DE538T1. McLean, VA: October 2005.

LMI Government Consulting. Construction Readiness EIR (for CD-3)
of the Depleted Uranium Hexafluoride Conversion Project. DE534T1.
McLean, VA: June 2005.

LMI Government Consulting. DUF6 Conversion Project CD-3C
Construction Readiness Review Preliminary Draft. Washington, D.C.:
May 20, 2005.

LMI Government Consulting. DUF6 Limited Conversion Plan Project
External Independent Review for the Office of Engineering and
Construction Management. DE428T1. McLean, VA: June 2004.

Chemistry and Metallurgy Research Facility Replacement

Jupiter Corporation. External Independent Review of the Chemistry
and Metallurgy Research Building Replacement Project. Approve
Performance Baseline and Approve Start of Construction.
CD-2A/CD-3A. Wheaton, MD: October 14, 2005.

Appendix IV: Survey Results for Primary Factors Affecting Cost
and Schedule on Nine Projects with Cost or Schedule Changes

Source: GAO.

Survey results for primary factors
To a
To a To a To a very
To no limited moderate great great No
Factor/Project extent extent extent extent extent answer
Absence of open communication, mutual trust, and close coordination
Depleted Uranium
Hexafluoride 6
Conversion Facility X
Highly Enriched
Uranium Materials
Facility X
Mixed Oxide Fuel
Fabrication
Facility X
National Ignition
Facility X
Pit Disassembly and
Conversion Facility X
Salt Waste
Processing Facility X
Spallation Neutron
Source X
Tritium Extraction
Facility X
Waste Treatment and
Immobilization
Plant X
Total 0 4 2 1 0 2
Changes in "political will" during project execution (e.g., project
changes resulting from political decisions--includes politics internal and
external to the project )
Depleted Uranium
Hexafluoride 6
Conversion Facility X
Highly Enriched
Uranium Materials
Facility X
Mixed Oxide Fuel
Fabrication
Facility X
National Ignition
Facility X
Pit Disassembly and
Conversion Facility X
Salt Waste
Processing Facility X
Spallation Neutron
Source X
Tritium Extraction
Facility X
Waste Treatment and
Immobilization
Plant X
Total 2 2 2 0 2 1
Interruptions in planning and committing budget funds
Depleted Uranium
Hexafluoride 6
Conversion X
Highly Enriched
Uranium Materials
Facility X
Mixed Oxide Fuel
Fabrication
Facility X
National Ignition
Facility X
Pit Disassembly and
Conversion Facility X
Salt Waste
Processing Facility X
Spallation Neutron
Source X

Survey results for primary factors
To a
To a To a To a very
To no limited moderate great great No
Factor/Project extent extent extent extent extent answer
Tritium Extraction Facility X
Waste Treatment and
Immobilization Plant X
Total 3 0 1 2 2 1
Project managers did not
have adequate professional
experience
Depleted Uranium
Hexafluoride 6 Conversion X
Highly Enriched Uranium
Materials Facility X
Mixed Oxide Fuel Fabrication
Facility X
National Ignition Facility X
Salt Waste Processing
Facility X
Spallation Neutron Source X
Tritium Extraction Facility X
Pit Disassembly and
Conversion Facility X
Waste Treatment and
Immobilization Plant X
Total 2 1 4 0 0 2

Source: GAO analysis of DOD data.

Appendix V: Definitions of Technology Readiness Levels

Technology
readiness level Basic objective of Tests and
(TRL) Level involved TRLs Components Integration environment
1. Basic Studies. Research to prove None. None. Desktop, "back
principles feasibility. of envelope"
observed and environment.
reported.
2. Technology Studies. Research to prove None. Paper studies Academic
concept and/or feasibility. indicate environment. The
application components emphasis here is
formulated. ought to work still on
together. understanding
the science but
beginning to
think about
possible
applications of
the scientific
principles.
3. Analytical Pieces of components. Research to prove No system No attempt at Uses of the
and feasibility. components, just integration; observed
experimental basic laboratory still trying properties are
critical research equipment to see postulated and
function to verify physical whether experimentation
and/or principles. individual with potential
characteristic parts of the elements of
proof of technology subsystem
concept. work. Lab begins. Lab work
experiments to validate
with pieces of
available technology
components without trying
show they to integrate.
will work. Emphasis is on
validating the
predictions made
during earlier
analytical
studies to
ensure that the
technology has a
firm scientific
underpinning.
4. Component Low fidelity Demonstrate Ad Hoc and Available Tests in
and/or breadboard. technical available components controlled
breadboard feasibility and laboratory assembled laboratory
validation in functionality. components are into environment. Lab
lab surrogates for subsystem work at less
environment. system components breadboard. than full
that may require Interfaces subsystem
special handling, between integration,
calibration, or components although
alignment to get are starting to see
them to function. realistic. if components
Not fully will work
functional but together.
representative of
technically
feasible approach.
5. Component High fidelity Demonstrate Fidelity of Fidelity of Laboratory
and/or breadboard/brass-board technical components and subsystem environment
breadboard (e.g., nonscale or feasibility and interfaces are mock up modified to
validation in form components). functionality. improved from TRL improves approximate
relevant 4. Some special (e.g., from operational
environment. purpose components breadboard to environment.
combined with brassboard). Increases in
available Integration accuracy of the
laboratory issues become controlled
components. defined. environment in
Functionally which it is
equivalent but not tested.
of same material or
size. May include
integration of
several components
with reasonably
realistic support
elements to
demonstrate
functionality.
6. System/ Subsystem closely Demonstrate Subsystem is high Components Relevant
Subsystem configured for applicability to fidelity functional are environment
model or intended project intended project prototype with functionally inside or
prototype application. and subsystem (very near same compatible outside the
demonstration Demonstrated in integration. material and size (and very laboratory, but
in relevant relevant environment. of operational near same not the eventual
environment. (Shows will work in (Specific to system). Probably material and operating
desired intended includes the size of environment. The
configuration). application in integration of many operational testing
project.) new components and system). environment does
realistic Component not reach the
supporting integration level of an
elements/subsystems into system operational
if needed to is environment,
demonstrate full demonstrated. although moving
functionality. out of
Partially controlled
integrated with laboratory
existing systems. environment into
something more
closely
approximating
the realities of
technology's
intended use.
7. Subsystem Subsystem configured Demonstrate Prototype improves Prototype not Operational
prototype for intended project applicability to to preproduction integrated environment, but
demonstration application. intended project quality. Components into intended not the eventual
in an Demonstrated in and subsystem are representative system but environment.
operational operational integration. of project onto Operational
environment. environment. components surrogate testing of
(Specific to (material, size, system. system in
intended and function) and representational
application in integrated with environment.
project.) other key Prototype will
supporting be exposed to
elements/subsystems the true
to demonstrate full operational
functionality. environment on a
Accurate enough surrogate
representation to platform,
expect only minor demonstrator, or
design changes. test bed.
8. Total Full integration of Applied/Integrated Components are Subsystem Demonstration,
system subsystems to show into intended right material, performance test, and
completed, total system will meet project size, and function meets evaluation
tested, and requirements. application. compatible with intended completed.
fully operational system. application Demonstrates
demonstrated. and is fully system meets
integrated procurement
into total specifications.
system. Demonstrated in
eventual
environment.
9. Total System meeting Applied/Integrated Components are Subsystem has Operational
system used intended operational into intended successfully been testing and
successfully requirements. project performing in the installed and evaluation
in project application. actual successfully completed.
operations. environment--proper deployed in Demonstrates
size, material, and project that system is
function. systems. capable of
meeting all
mission
requirements.

Appendix VI: Comparison of DOD�s Product Development Process
with DOE�s Project Management Process

Appendix VII: Comments from the Department of Energy

Appendix VIII: GAO Contact and Staff Acknowledgments

GAO Contact

Gene Aloise, (202) 512-3841

Staff Acknowledgments

In addition to the individual named above, Michaela Brown, Rudy
Chatlos, James Espinoza, Daniel Feehan (Assistant Director),
Joseph Keener, Thomas Kingham, Matthew Lea, Mehrzad Nadji, Omari
Norman, Christopher Pacheco, Thomas Perry, and Carol Herrnstadt
Shulman made key contributions to this report.

GAO�s Mission

The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in
meeting its constitutional responsibilities and to help improve
the performance and accountability of the federal government for
the American people. GAO examines the use of public funds;
evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at
no cost is through GAO's Web site ( www.gao.gov ). Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of
newly posted products every afternoon, go to www.gao.gov and
select "Subscribe to Updates."

Order by Mail or Phone

The first copy of each printed report is free. Additional copies
are $2 each. A check or money order should be made out to the
Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are
discounted 25 percent. Orders should be sent to:

U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548

To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
(202) 512-6061

To Report Fraud, Waste, and Abuse in Federal Programs

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
[email protected] Automated answering system: (800) 424-5454 or
(202) 512-7470

Congressional Relations

Gloria Jarmon, Managing Director, [email protected] (202)
512-4400 U.S. Government Accountability Office, 441 G Street NW,
Room 7125 Washington, D.C. 20548

Public Affairs

Paul Anderson, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW,
Room 7149 Washington, D.C. 20548

(360652)

www.gao.gov/cgi-bin/getrpt?GAO-07-336 .

To view the full product, including the scope
and methodology, click on the link above.

For more information, contact Gene Aloise at (202) 512-3841 or
[email protected].

Highlights of [67]GAO-07-336 , a report to the Subcommittee on Energy and
Water Development, and Related Agencies, Committee on Appropriations,
House of Representatives

March 2007

DEPARTMENT OF ENERGY

Major Construction Projects Need a Consistent Approach for Assessing
Technology Readiness to Help Avoid Cost Increases and Delays

[68]What GAO Recommends

GAO recommends that DOE develop a consistent approach for measuring the
readiness of critical project technologies. DOE supports GAO's
recommendations but suggested revisions to allow it to first conduct a
pilot application on selected projects to better understand the process
and evaluate its potential use.

Of the 12 DOE major projects GAO reviewed, 9 exceeded their original cost
or schedule estimates, principally because of ineffective DOE project
oversight and poor contractor management. Specifically, 8 of the 12
projects experienced cost increases ranging from $79.0 million to $7.9
billion, and 9 of the 12 projects were behind schedule by 9 months to more
than 11 years. Project oversight problems included, among other things,
inadequate systems for measuring contractor performance, approval of
construction activities before final designs were sufficiently complete,
ineffective project reviews, and insufficient DOE staffing. Furthermore,
contractors poorly managed the development and integration of the
technology used in the projects by, among other things, not accurately
anticipating the cost and time that would be required to carry out the
highly complex tasks involved.

Even though DOE requires final project designs to be sufficiently complete
before beginning construction, it has not systematically ensured that the
critical technologies reflected in these designs have been demonstrated to
work as intended (technology readiness) before committing to construction
expenses. Specifically, only one of the five DOE project directors with
projects that have recently begun or are nearing construction had
systematically assessed technology readiness. The other four directors
also told us that they have or will have completed prior to construction,
85 to 100 percent of their projects' final design, but they had not
systematically assessed technology readiness. Proceeding into construction
without also demonstrating a technology's readiness can lead to cost
increases and delays. For example, one technology to be used in DOE's
Waste Treatment and Immobilization Plant was not sufficiently
demonstrated--that is, shown to be technologically ready for its intended
application--before construction began. Consequently, the technology did
not perform as expected, which resulted in about $225 million in redesign
costs and schedule delays of more than 1 year. To help avoid these
problems, the National Aeronautics and Space Administration (NASA)
pioneered and the Department of Defense (DOD) has adopted for its projects
a method for measuring and communicating technology readiness levels
(TRL). Using a scale from one (basic principles observed) through nine
(total system used successfully in project operations), TRLs show the
extent to which technologies have been demonstrated to work as intended in
the project. DOE project directors agreed that such an approach would help
make technology assessments more transparent and improve stakeholder
communication prior to making critical project decisions, such as
authorizing construction.

Technology Readiness Levels

References

Visible links
28. http://www.gao.gov/cgi-bin/getrpt?GAO-06-602T
29. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-99-162
30. http://www.gao.gov/cgi-bin/getrpt?GAO-06-391
31. http://www.gao.gov/cgi-bin/getrpt?GAO-04-759
32. http://www.gao.gov/cgi-bin/getrpt?GAO-05-123
33. http://www.gao.gov/cgi-bin/getrpt?GAO/RCED-97-146R
34. http://www.gao.gov/
35. http://www.gao.gov/cgi-bin/getrpt?GAO-03-570T
36. http://www.gao.gov/cgi-bin/getrpt?GAO-02-798
37. http://www.gao.gov/cgi-bin/getrpt?GAO-01-677R
38. http://www.gao.gov/cgi-bin/getrpt?GAO/RCED-00-141
39. http://www.gao.gov/cgi-bin/getrpt?GAO/RCED-00-271
40. http://www.gao.gov/cgi-bin/getrpt?GAO-06-602T
41. http://www.gao.gov/cgi-bin/getrpt?GAO-05-123
42. http://www.gao.gov/cgi-bin/getrpt?GAO-04-611
43. http://www.gao.gov/cgi-bin/getrpt?GAO-03-57T
44. http://www.gao.gov/cgi-bin/getrpt?GAO-02-798
45. http://www.gao.gov/cgi-bin/getrpt?GAO/RCED-93-99
46. http://www.gao.gov/cgi-bin/getrpt?GAO-03-570T
47. http://www.gao.gov/cgi-bin/getrpt?GAO-02-798
48. http://www.gao.gov/cgi-bin/getrpt?GAO/T-RCED-99-103
49. http://www.gao.gov/cgi-bin/getrpt?GAO-05-123
50. http://www.gao.gov/cgi-bin/getrpt?GAO-03-570T
51. http://www.gao.gov/cgi-bin/getrpt?GAO-02-798
52. http://www.gao.gov/cgi-bin/getrpt?GAO/RCED-98-75
53. http://www.gao.gov/cgi-bin/getrpt?GAO-03-570T
60. http://www.gao.gov/cgi-bin/getrpt?GAO-06-602T
61. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-99-162
62. http://www.gao.gov/cgi-bin/getrpt?GAO-02-39
63. http://www.gao.gov/cgi-bin/getrpt?GAO/T-RCED-00-248
64. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-99-162
65. http://www.gao.gov/cgi-bin/getrpt?GAO-05-123
67. http://www.gao.gov/cgi-bin/getrpt?GAO-07-336
*** End of document. ***