Department of Energy: Major Construction Projects Need a	 
Consistent Approach for Assessing Technology Readiness to Help	 
Avoid Cost Increases and Delays (27-MAR-07, GAO-07-336).	 
                                                                 
The Department of Energy (DOE) spends billions of dollars on	 
major construction projects that help maintain the nuclear	 
weapons stockpile, conduct research and development, and process 
nuclear waste so that it can be disposed of. Because of DOE's	 
long-standing project management problems, GAO determined the	 
extent to which (1) DOE's major construction projects are having 
cost increases and schedule delays and the major factors	 
contributing to these problems and (2) DOE ensures that project  
designs are sufficiently complete before construction begins to  
help avoid cost increases and delays. We examined 12 DOE major	 
projects with total costs of about $27 billion, spoke with	 
federal and contractor officials, and reviewed project management
documents.							 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-07-336 					        
    ACCNO:   A67345						        
  TITLE:     Department of Energy: Major Construction Projects Need a 
Consistent Approach for Assessing Technology Readiness to Help	 
Avoid Cost Increases and Delays 				 
     DATE:   03/27/2007 
  SUBJECT:   Contract oversight 				 
	     Cost overruns					 
	     Critical technologies				 
	     Design specifications				 
	     Facility construction				 
	     Performance measures				 
	     Program evaluation 				 
	     Program management 				 
	     Research and development facilities		 
	     Schedule slippages 				 
	     Strategic planning 				 
	     Technology assessment				 
	     Work measurement					 
	     Contract mismanagement				 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-07-336

   

     * [1]Results in Brief
     * [2]Background
     * [3]Most Major Projects Have Exceeded Original Costs and Are Yea
     * [4]DOE Does Not Consistently Measure Technology Readiness to En

          * [5]DOE Does Not Consistently Assess Technology Readiness
          * [6]Other Federal Agencies Use a Standard Method for Measuring a

     * [7]Conclusions
     * [8]Recommendations for Executive Action
     * [9]Agency Comments and Our Evaluation
     * [10]National Ignition Facility
     * [11]Mixed Oxide Fuel Fabrication Facility
     * [12]Pit Disassembly and Conversion Facility
     * [13]Waste Treatment and Immobilization Plant
     * [14]Spallation Neutron Source
     * [15]Salt Waste Processing Facility
     * [16]Tritium Extraction Facility
     * [17]Highly Enriched Uranium Materials Facility
     * [18]Depleted Uranium Hexafluoride 6 Conversion Facility
     * [19]Chemistry and Metallurgy Research Facility Replacement
     * [20]GAO Contact
     * [21]Staff Acknowledgments
     * [22]GAO's Mission
     * [23]Obtaining Copies of GAO Reports and Testimony

          * [24]Order by Mail or Phone

     * [25]To Report Fraud, Waste, and Abuse in Federal Programs
     * [26]Congressional Relations
     * [27]Public Affairs

Report to the Subcommittee on Energy and Water Development, and Related
Agencies, Committee on Appropriations, House of Representatives

United States Government Accountability Office

GAO

March 2007

DEPARTMENT OF ENERGY

Major Construction Projects Need a Consistent Approach for Assessing
Technology Readiness to Help Avoid Cost Increases and Delays

GAO-07-336

Contents

Letter 1

Results in Brief 4
Background 7
Most Major Projects Have Exceeded Original Costs and Are Years Late,
Principally Because of Ineffective DOE Project Oversight and Contractor
Management 9
DOE Does Not Consistently Measure Technology Readiness to Ensure That
Critical Technologies Will Work as Intended before Construction Begins 18
Conclusions 26
Recommendations for Executive Action 27
Agency Comments and Our Evaluation 28
Appendix I Scope and Methodology 31
Appendix II Information on the 12 Department of Energy Major Projects
Reviewed 34
Appendix III Independent Studies Reviewed 36
Appendix IV Survey Results for Primary Factors Affecting Cost and Schedule
on Nine Projects with Cost or Schedule Changes 42
Appendix V Definitions of Technology Readiness Levels 44
Appendix VI Comparison of DOD's Product Development Process with DOE's
Project Management Process 47
Appendix VII Comments from the Department of Energy 48
Appendix VIII GAO Contact and Staff Acknowledgments 50

Tables

Table 1: Changes in Estimated Total Project Cost for DOE Major
Construction Projects 10
Table 2: Changes in Estimated Project Schedules for DOE Major Construction
Projects 11
Table 3: Reasons for Cost Increases and Schedule Delays 12

Abbreviations

DOD Department of Defense
DOE Department of Energy
EM Office of Environmental Management
ITP In-Tank Precipitation
NASA National Aeronautics and Space Administration
NNSA National Nuclear Security Administration
PDRI Product Definition Rating Index
TPC total project cost
TRL technology readiness level

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office
Washington, DC 20548

March 27, 2007

The Honorable Peter J. Visclosky
Chairman
The Honorable David L. Hobson
Ranking Member
Subcommittee on Energy and Water Development, and Related Agencies
Committee on Appropriations
House of Representatives

The Department of Energy (DOE) spends billions of dollars on major
construction projects that, among other things, are used to help maintain
the nuclear weapons stockpile, conduct research and development in the
areas of high-energy physics and nuclear physics, and process nuclear
waste into forms suitable for longer-term storage or permanent disposal.
DOE oversees the construction of facilities primarily at government-owned,
contractor-operated sites throughout the nation. In July 2006, DOE revised
its dollar threshold defining a construction project as "major"; it is now
at $750 million, up from $400 million when we began our review in December
2005. The following 12 projects included in our review had estimated
project costs exceeding the original threshold.^1 The total cost of these
projects is currently estimated at about $27 billion.^2 These 12 projects
and their locations are as follows:

           o Chemistry and Metallurgy Research Facility Replacement--Los
           Alamos National Laboratory, Los Alamos, New Mexico.
           o Depleted Uranium Hexafluoride 6 Conversion Facility--Portsmouth,
           Ohio, and Paducah, Kentucky.
           o Highly Enriched Uranium Materials Facility--Y-12 National
           Security Complex, Oak Ridge, Tennessee.
           o Linac Coherent Light Source--Stanford Linear Accelerator Center,
           Menlo Park, California.
           o Microsystems and Engineering Sciences Applications--Sandia
           National Laboratories, Albuquerque, New Mexico.
           o Mixed Oxide Fuel Fabrication Facility--Savannah River Site,
           Aiken, South Carolina.
           o National Ignition Facility--Lawrence Livermore National
           Laboratory, Livermore, California.
           o Pit Disassembly and Conversion Facility--Savannah River Site,
           Aiken, South Carolina.
           o Salt Waste Processing Facility--Savannah River Site, Aiken,
           South Carolina.
           o Spallation Neutron Source--Oak Ridge National Laboratory, Oak
           Ridge, Tennessee.
           o Tritium Extraction Facility--Savannah River Site, Aiken, South
           Carolina.
           o Waste Treatment and Immobilization Plant--Hanford Site,
           Richland, Washington.

           These major projects require the construction of large building
           complexes and the development of innovative cleanup and other
           technologies. Many of these technologies are developed for the
           project or are applied in a new way. DOE project directors are
           responsible for managing these major projects and overseeing the
           contractors that design and construct the facilities. In doing so,
           project directors follow specific departmental directives,
           policies, and guidance for project management. Among these are DOE
           Order 413.3A and Manual 413.3-1, which establish protocols for
           planning and executing a project. The protocols require DOE
           projects to go through a series of five critical decisions as they
           enter each new phase of work. Two of the decisions made before
           construction begins are key: (1) formally approving the project's
           definitive cost and schedule estimates as accurate and
           complete--an approval that is to be based on a review of the
           project's completed preliminary design, and (2) reaching agreement
           that the project's final design is sufficiently complete and that
           resources can be committed toward procurement and construction. To
           oversee projects and approve these critical decisions, DOE
           conducts its own reviews, often with the help of independent
           technical experts. In addition, projects are regularly subject to
           reviews by DOE's Office of Engineering and Construction Management
           and it's Office of Inspector General, the Department of Defense's
           (DOD) U.S. Army Corps of Engineers, the Defense Nuclear Facilities
           Safety Board, and the National Research Council, among others.

           We and others have reported over the past decade that project
           management weaknesses have impaired these projects. For example,
           for the Waste Treatment and Immobilization Plant, we reported that
           DOE's use of a "fast-track, design-build" approach to
           construction--in which design and construction activities
           overlap--has been problematic for highly complex, first-of-a-kind
           facilities. We found that the designs for these facilities were
           not sufficiently complete for construction to begin, which has
           resulted in significant cost increases and schedule delays.^3

           In this context, we determined the extent to which (1) DOE's
           active major construction projects are experiencing cost increases
           and schedule delays and the key factors contributing to these
           problems and (2) DOE ensures that project designs are sufficiently
           complete before construction begins to help avoid cost increases
           and schedule delays.^4

           To determine the extent to which DOE projects are experiencing
           cost increases or schedule delays and factors contributing to
           these problems, we sent a survey to the 12 DOE directors of major
           projects and reviewed the project management documents for these
           projects--6 projects that were above the $750 million threshold, 2
           estimated to cost between the previous $400 million threshold and
           $750 million, and 4 estimated to cost between $300 million and
           $400 million. (App. II describes these projects.) These 12
           projects are managed by DOE's Office of Science, Office of
           Environmental Management, or National Nuclear Security
           Administration (NNSA). We conducted site visits and analyzed
           independent project studies for the 9 projects that experienced
           cost increases or schedule delays. During the course of our
           review, we identified a method used by the DOE project director
           for the Pit Disassembly and Conversion Facility to systematically
           assess the extent to which a technology is sufficiently developed
           for its intended purpose. The project director based this method
           on a system developed by the National Aeronautics and Space
           Administration (NASA). We had previously reported on the use of a
           similar assessment system--technology readiness levels (TRL),
           which DOD has adopted for its major projects.^5 We obtained and
           reviewed documents regarding these two assessment systems.

           To determine the extent to which DOE ensures that project designs
           are sufficiently complete before construction, we reviewed in
           detail 5 of the 12 projects that were approaching or had recently
           begun construction. Specifically, we obtained information on the
           extent to which project designs were, or are expected to be,
           complete before beginning construction, and the actions DOE has
           taken to ensure that technologies used in these designs are
           sufficiently ready to begin construction.

           Because we and others have previously expressed concern about the
           data reliability of a key DOE project management tracking
           database--the Project Assessment and Reporting System--we did not
           develop conclusions or findings on the basis of information
           generated through that system.^6 Instead, we collected information
           directly from project site offices. In addition, we spoke with
           officials from DOE program offices and DOE's Office of Engineering
           and Construction Management in Washington, D.C. We provided
           interim briefings to the Subcommittee on the status of our work in
           May and September, 2006. We performed our work between December
           2005 and January 2007, in accordance with generally accepted
           government auditing standards. Appendix I contains a detailed
           description of our scope and methodology.
			  
			  Results in Brief

           Nine of the 12 DOE major projects we reviewed have exceeded their
           original cost estimates and/or experienced schedule delays,
           principally because of ineffective DOE project oversight and poor
           contractor management, according to independent studies we
           reviewed and interviews we conducted with DOE and contractor
           project officials. Specifically, 8 of the 12 projects experienced
           cost increases ranging from $79.0 million to $7.9 billion, and 9
           of the 12 projects are behind schedule by 9 months to more than 11
           years. Major factors cited for these cost increases and delays
           included the following:

           o Ineffective DOE project oversight. For all 9 projects
           experiencing cost increases or schedule delays, poor DOE oversight
           was a key contributing factor. Project oversight problems included
           inadequate systems for measuring contractor performance, approval
           of construction activities before final designs were sufficiently
           complete, ineffective project reviews, and insufficient DOE
           staffing and project management experience.
           o Poor contractor management. Eight of the 9 major projects
           experienced cost increases and/or schedule delays, in part because
           contractors did not effectively manage the development and
           integration of the technology used in the projects, including not
           accurately anticipating the cost and time that would be required
           to carry out the highly complex tasks involved. For example, the
           National Ignition Facility has had over $1 billion in cost
           increases and years of schedule delays owing in part to technology
           integration problems, according to the DOE project director. Other
           examples of poor contractor performance included inadequate
           quality assurance for the Highly Enriched Uranium Materials
           Facility, which resulted in concrete work that did not meet design
           specifications. The subsequent suspension of construction
           activities and rework added to the project's estimated cost and
           schedule.

           DOE officials also explained that a now-defunct policy may have
           contributed to increased costs and delays for several projects we
           examined. Until 2000, DOE required contractors to prepare cost and
           schedule estimates early in the project, before preliminary
           designs were completed. These estimates were used to establish a
           baseline for measuring contractor performance and tracking any
           cost increases or schedule delays. However, these estimates often
           were based on early conceptual designs and, thus, were subject to
           significant change as more detailed designs were developed. To
           improve the reliability of these estimates, DOE issued a new order
           in October 2000 that required the preparation of a cost estimate
           range at the start of preliminary design, and delayed the
           requirement for a definitive cost and schedule baseline estimate
           until after preliminary design was completed. Consequently, DOE
           officials explained, the new policy should result in improved
           estimates and a more accurate measure of cost and schedule
           performance.

           Even though DOE requires final project designs to be sufficiently
           complete before beginning construction, it has not systematically
           ensured that the critical technologies reflected in these designs
           have been demonstrated to work as intended (technology readiness)
           before committing to construction expenses. Only one of the five
           DOE directors with projects that have recently begun, or are
           nearing construction, had systematically assessed technology
           readiness. The other four directors also told us that they have or
           will have completed prior to construction, 85 to 100 percent of
           their projects' final design, but they had not systematically
           assessed for technology readiness. Lack of technology readiness
           can result in cost overruns and schedule delays. For example,
           technology used in a subsystem intended to prepare radioactive
           material for processing in DOE's Waste Treatment and
           Immobilization Plant was not fully developed and did not work as
           expected after construction had already begun, resulting in
           redesign costs of about $225 million and over 1 year in schedule
           delays.

           To effectively assess technology readiness, NASA pioneered and DOD
           has adopted a process for measuring and communicating technology
           readiness for first-of-a-kind technology applications. This
           process uses a nine-point scale for assessing TRLs. Using this
           scale, a technology would receive a higher TRL value (e.g., TRL 7)
           if it has been successfully demonstrated in an operational
           environment, compared with a technology that has been demonstrated
           only in a laboratory test (e.g., TRL 4). Several DOE project
           directors we spoke with agreed that a consistent, systematic
           method for assessing technology readiness would help standardize
           terminology, make technology assessments more transparent, and
           help improve communication among project stakeholders before they
           make critical project decisions.

           To improve oversight and decision making for DOE's major
           construction projects, we are recommending that the Secretary of
           Energy evaluate and consider adopting a disciplined and consistent
           approach to assessing TRLs for projects with critical
           technologies.

           DOE provided comments to us based on a draft of the report. DOE
           agreed with our recommendations but suggested revisions that would
           first allow them to conduct a pilot application on selected
           projects to better understand the technology readiness assessment
           process and evaluate its potential use. We revised our
           recommendations as appropriate. DOE suggested that our report is
           too narrowly focused on technology assessment, and that we
           inappropriately calculated cost increases and schedule delays
           using preliminary estimates that were only intended for internal
           DOE planning. We believe that our recommendations were justifiably
           based on our finding that DOE has not systematically ensured that
           project designs, including critical technologies reflected in
           these designs, have been demonstrated to work as intended prior to
           construction. We also believe it was appropriate, when necessary,
           to measure cost and schedule changes using the initial estimates
           that were developed at the end of conceptual design, as specified
           in DOE's project management policy in effect prior to 2000. We
           note that these estimates were, in some instances, the only
           initial estimates available and had been used by DOE to inform the
           Congress of the estimated cost and schedule of the projects while
           it was seeking initial project funding. We also incorporated
           technical changes in this report where appropriate on the basis of
           detailed comments provided by DOE.
			  
			  Background

           To meet its diverse missions, DOE pays its contractors billions of
           dollars each year to implement hundreds of projects, ranging from
           hazardous waste cleanups at sites in the weapons complex to the
           construction of scientific facilities. Many of these complex,
           unique projects are designed to meet defense, energy research,
           environmental, and fissile materials disposition goals. They often
           rely on technologies that are unproven in operational conditions.
           In recent years, DOE's budget has been dominated by the monumental
           task of environmental restoration and waste management to repair
           damage caused by the past production of nuclear weapons.

           DOE has long had a poor track record for developing designs and
           cost estimates and managing projects. We reported in 1997 that
           from 1980 to 1996, 31 of DOE's 80 major projects were terminated
           prior to completion, after expenditures of over $10 billion; 15 of
           the projects were completed, but most of them were finished behind
           schedule and with cost overruns; and the remaining 34 ongoing
           projects also were experiencing schedule slippage or cost
           overruns.^7 In addition, for over a decade, DOE's Office of
           Inspector General, the National Academy of Sciences, and others
           have identified problems with DOE's management of major
           construction projects. Projects were late or never finished;
           project costs increased by millions and sometimes billions of
           dollars; and environmental conditions at the sites did not
           significantly improve. According to the National Research
           Council,^8 DOE's construction and environmental remediation
           projects take much longer and cost about 50 percent more than
           comparable projects by other federal agencies or projects in the
           private sector.^9 A 2004 assessment of departmental project
           management completed by the Civil Engineering Research Foundation
           recommended, among other things, that DOE develop a core group of
           highly qualified project directors and require peer reviews for
           first-of-a-kind and technically complex projects when the
           projects' preliminary baselines are approved.^10

           To address project management issues, DOE began a series of
           reforms in the 1990s that included efforts to strengthen project
           management practices. To guide these reforms, the department
           formed the Office of Engineering and Construction Management in
           1999. The reforms instituted to date have included planning,
           organizing, and tracking project activities, costs, and schedules;
           training to ensure that federal project managers had the required
           expertise to manage projects; increasing emphasis on independent
           reviews; and strengthening project reporting and oversight.
			  
			  Most Major Projects Have Exceeded Original Costs and Are Years
			  Late, Principally Because of Ineffective DOE Project Oversight
			  and Contractor Management

           The estimated costs of many of the DOE major construction projects
           we reviewed have significantly exceeded original estimates and
           schedules have slipped. On the basis of our analysis of
           independent project studies and interviews with project directors,
           cost growth and schedule slippage occurred principally because of
           ineffective DOE project oversight and poor contractor project
           management. Furthermore, unreliable initial cost and schedule
           estimates resulting from a now-defunct policy may have been a
           contributing factor, according to DOE project officials. Although
           external factors, such as additional security and safety
           requirements, contributed to cost growth and delays, the
           management of these requirements was complicated by ineffective
           and untimely DOE communication and decision making.

           Eight of the 12 DOE projects we reviewed had increases in
           estimates of total project cost (TPC) ranging from $79.0 million
           to $7.9 billion. As table 1 shows, the percentage of cost increase
           for these 8 projects ranged from 2 percent to over 200 percent.

           Table 1: Changes in Estimated Total Project Cost for DOE Major
           Construction Projects

           Source: GAO analysis of DOE data.

           aIn 2000, DOE changed its requirements for establishing initial
           cost and schedule estimates. Prior to 2000, these estimates were
           established at the end of conceptual design. After 2000, DOE
           required initial estimates to be completed later in the
           project--at the end of preliminary design. For projects beginning
           prior to 2000, and for projects beginning after 2000 that had not
           yet completed preliminary design, we used the TPC estimates
           prepared after conceptual design. For additional details on our
           methodology, see appendix I.

           bWe calculated the percentages of cost increases on the basis of
           constant 2007 dollars to make them comparable across projects and
           to show real increases in cost while excluding increases due to
           inflation.

           cNNSA officials, in commenting on our draft report, stated that
           initial and current cost estimates for the Mixed Oxide Fuel
           Fabrication Facillity and the Pit Disassembly and Conversion
           Facility should not be used in this analysis because neither
           project has an approved budget quality baseline. Nevertheless, we
           included the estimates in this analysis because both projects have
           been in an extended period of project design, without an approved
           budget-quality baseline, for about 10 years, and the estimates
           provided here are the only estimates available.

           dEstimate may change when DOE approves contractor's revised TPC in
           2007.

           In addition, as shown in table 2, 9 of the 12 projects experienced
           schedule delays ranging from 9 months to more than 11 years. Of
           the 9 projects, 7 had schedule delays of at least 2 years or more.

^1For this review, we lowered the threshold to $300 million out of concern
that some projects not considered major could later be defined as major
because of cost increases.

^2This estimate includes design and construction costs, but does not
reflect the total life-cycle costs of the projects, such as operating and
maintenance costs.

^3GAO, Hanford Waste Treatment Plant: Contractor and DOE Management
Problems Have Led to Higher Costs, Construction Delays, and Safety
Concerns, [28]GAO-06-602T (Washington, D.C.: Apr. 6, 2006).

^4A forthcoming GAO report will address actions taken by DOE to improve
overall project management.

^5GAO, Best Practices: Better Management of Technology Development Can
Improve Weapon System Outcomes, [29]GAO/NSIAD-99-162 (Washington, D.C.:
July 30, 1999); Defense Acquisitions: Assessments of Selected Major
Weapons Programs, [30]GAO-06-391 (Washington, D.C.: Mar. 31, 2006); and
Defense Acquisitions: Space-Based Radar Effort Needs Additional Knowledge
before Starting Development, [31]GAO-04-759 (Washington, D.C.: July 23,
2004).

^6GAO, Department of Energy: Further Actions Are Needed to Strengthen
Contract Management for Major Projects, [32]GAO-05-123 (Washington, D.C.:
Mar. 18, 2005); and Civil Engineering Research Foundation, Independent
Research Assessment of Project Management Factors Affecting Department of
Energy Project Success (Washington, D.C: July 12, 2004).

^7GAO, Oversight of DOE's Major Systems, [33]GAO/RCED-97-146R (Washington,
D.C.: Apr. 30, 1997).

^8The National Research Council was organized by the National Academy of
Sciences to advise the federal government on matters related to science
and technology.

^9National Research Council, Improving Project Management in the
Department of Energy (Washington, D.C.: July 1999).

^10Civil Engineering Research Foundation, Independent Research Assessment
of Project Management Factors Affecting Department of Energy Project
Success (Washington, D.C.: July 12, 2004).

Dollars in millions                                                        
                                         Initial total                        
                                    project cost (TPC) Current TPC Percentage 
Project                                  estimate^a    estimate increase^b 
Mixed Oxide Fuel Fabrication                                               
Facility^c                                   $1,400      $4,699       205% 
Waste Treatment and                                                        
Immobilization Plant                          4,350      12,263        143 
Highly Enriched Uranium                                                    
Materials Facility                              251         549        102 
National Ignition Facility                    1,199      $2,248         59 
Salt Waste Processing Facility                  440       680^d         50 
Pit Disassembly and Conversion                                             
Facility^c                                    1,700       2,694         40 
Tritium Extraction Facility                     384         506         15 
Spallation Neutron Source                     1,333       1,412          2 
Depleted Uranium Hexafluoride 6                                            
Conversion Facility                             346         346          0 
Chemistry and Metallurgy                                                   
Research Facility Replacement                   837         837          0 
Microsystems and Engineering                                               
Sciences Applications                           518         518          0 
Linac Coherent Light Source                     379         379          0 

Table 2: Changes in Estimated Project Schedules for DOE Major Construction
Projects

                       Year mission Initial       Current       Schedule      
                       need was     completion    completion    delay as of   
Project             approved     date estimate date estimate February 2007 
Pit Disassembly and 1997         06/2005       03/2017       11 years, 9   
Conversion Facility                                          months        
Mixed Oxide Fuel    1997         09/2004       03/2016       11 years, 6   
Fabrication                                                  months        
Facility                                                                   
Waste Treatment and 1995         07/2011       11/2019       8 years, 4    
Immobilization                                               months        
Plant                                                                      
National Ignition   1993         10/2003       03/2009       5 years, 5    
Facility                                                     months        
Depleted Uranium    2000         03/2006       06/2008       2 years, 3    
Hexafluoride 6                                               months        
Conversion                                                                 
Facility^a                                                                 
Salt Waste          2001         07/2009       09/2011^b     2 years, 2    
Processing Facility                                          months        
Tritium Extraction  1995         06/2005       07/2007       2 years, 1    
Facility                                                     month         
Highly Enriched     1999         04/2008       03/2010       1 year, 11    
Uranium Materials                                            months        
Facility                                                                   
Spallation Neutron  1996         09/2005       06/2006^c     9 months      
Source                                                                     
Chemistry and       2002         03/2014       03/2014       Not           
Metallurgy Research                                          applicable    
Facility                                                                   
Replacement                                                                
Microsystems and    1999         01/2009       01/2009       Not           
Engineering                                                  applicable    
Sciences                                                                   
Applications                                                               
Linac Coherent      2001         03/2009       03/2009       Not           
Light Source                                                 applicable    

Source: GAO analysis of DOE data.

aThis project reported a schedule delay but did not report an increase in
the estimated total project cost (TPC). According to the DOE project
director, the original cost estimate was probably too high and was not
well supported.

bAccording to DOE officials, schedule may slip further when the contractor
submits its revised TPC to DOE in July 2007.

cProject was completed on this date. Transition to operations has begun.

As table 3 shows, ineffective DOE project oversight and poor contractor
management were frequently cited reasons for cost increases and schedule
delays for the projects we reviewed, according to our review of
independent studies of the 9 projects experiencing cost growth and
schedule delays and our follow-up interviews with DOE project directors.
Project officials, in commenting on our draft report, were concerned that
table 3 might misrepresent the overall successful execution and completion
of some projects, such as the Spallation Neutron Source, and that some
problems may have already been addressed. Nevertheless, to clarify our
main purpose for table 3, our intent is to show broad categories of major
reasons for cost increases and schedule delays, regardless of when they
occurred or whether they have been adequately addressed.

Table 3: Reasons for Cost Increases and Schedule Delays

                                                        Poor External factors 
                                      DOE project contractor           (e.g., 
Project                              oversight management safety/security) 
Depleted Uranium Hexafluoride 6                                            
Conversion Facility                          X          X                X 
Highly Enriched Uranium Materials                                          
Facility                                     X          X                X 
Mixed Oxide Fuel Fabrication                                               
Facility                                     X          X                X 
National Ignition Facility                   X          X                  
Pit Disassembly and Conversion                                             
Facility                                     X          X                X 
Salt Waste Processing Facility               X                           X 
Spallation Neutron Source                    X          X                  
Tritium Extraction Facility                  X          X                X 
Waste Treatment and Immobilization                                         
Plant                                        X          X                X 
Total                                        9          8                7 

Source: GAO analysis of independent project studies and interviews with
DOE project directors (a list of the project studies we reviewed is
included in app. III).

The DOE project oversight issues mentioned in table 3 include the
following:

           o inadequate systems for measuring contractor performance;
           o approval of construction activities before final designs were
           sufficiently complete;
           o ineffective project reviews;
           o insufficient DOE staffing and experience;
           o inadequate use of project management controls;
           o lack of headquarters assistance and oversight support of field
           project directors;
           o failure to detect contractor performance problems, including
           inadequate federal inspection activities; and
           o poor government cost estimates, including inadequate funding for
           contingencies.

           DOE's lack of adequate systems to measure contractor performance
           was cited in a December 2005 DOE Inspector General review of the
           Mixed Oxide Fuel Fabrication Facility. The Inspector General
           criticized DOE's NNSA for failing to approve a baseline against
           which to measure contractor performance and relying on outdated
           cost plans.^11 According to the report, NNSA relied on confusing
           and misleading information detailed in the monthly project reports
           to monitor progress and track costs--reports that the contractor
           acknowledged as being "useless for evaluating performance or
           managing the project." Furthermore, although the contractor
           reported unfavorable cost and schedule variances for months, these
           variances were inaccurate and meaningless because performance was
           being compared against a 2-year-old plan. NNSA, in commenting on
           our draft report, stated that project oversight and contractor
           management problems identified in previous GAO, Inspector General,
           and other independent assessments have led to extensive
           improvements to the project, and that major findings identified
           during a recent independent review have been successfully
           addressed.

           Similarly, DOE's approval of construction activities before final
           designs were sufficiently complete has contributed significantly
           to project cost growth and schedule delays. As we have previously
           reported, the accelerated fast-track, design-build approach used
           for the Waste Treatment and Immobilization Plant, a highly complex
           first-of-a-kind nuclear facility, resulted in significant cost
           increases and schedule delays.^12 DOE also allowed the contractor
           on another project, the Tritium Extraction Facility, to begin
           construction before the final design was completed to meet
           schedule commitments. According to a 2002 DOE Inspector General
           report on the project,^13 this revised acquisition strategy of
           simultaneous design and construction directly resulted in at least
           $12 million in project overruns.

           The contractor management issues mentioned in table 3 include the
           poor management of technological challenges, among other
           contractor performance issues, according to DOE project directors.
           Cost increases and schedule delays for 6 of the 9 projects were
           due in part to contractors' poor management of the development and
           integration of technologies used in project designs by, among
           other things, not accurately anticipating the cost and time that
           would be required to carry out the highly complex tasks
           involved.^14 For example:

           o The National Ignition Facility had over $1 billion in cost
           overruns and years of schedule delays, in large part because of
           technology integration problems. The requirements for the National
           Ignition Facility--the use of 192 high-power laser beams focused
           on a single target in a "clean room" environment--had not been
           attempted before on such a large scale. According to the DOE
           project director, early incorrect assumptions about the original
           facility design and the amount of work necessary to integrate the
           technologies and assemble the technical components contributed to
           about half of the project's cost increases and schedule delays.
           o The design of the Mixed Oxide Fuel Fabrication Facility has
           presented technical challenges in adapting the design of a similar
           plant in France to the design needs of this project. Although the
           technological challenge related to adopting the process designs
           from the French designs was not the primary contributor to the
           project's cost increases and schedule delays, according to NNSA
           officials, it has affected the project's complexity. The basic
           technology--combining plutonium oxide with depleted uranium to
           form fuel assemblies for use in commercial power reactors--has
           been previously demonstrated in France. However, the DOE project
           director told us that the DOE facility design must, among other
           things, account for processing surplus weapon-grade plutonium, a
           different type of material than processed in the French facility,
           and must be adapted to satisfy U.S. regulatory and other local
           requirements. In addition, the DOE facility faced the
           technological challenge of reducing the scale of components used
           in the French facility. Although definitive cost estimates are not
           yet available, expected costs for completing this project have
           grown by about $3.3 billion since 2002, and the schedule has been
           extended by more than 11 years, in part because the contractor did
           not initially understand the project's complexity and
           underestimated the level of effort needed to complete the work.
           NNSA explained that the capability of the reference plants
           currently in operation in France, and by extension, the Mixed
           Oxide Fuel Fabrication Facility process design, is currently being
           demonstrated by several prototype fuel assemblies manufactured
           with weapon-grade plutonium oxide, which are currently being
           successfully used in a reactor in South Carolina.
           o For the Waste Treatment and Immobilization Plant, a technology
           application used on the project had not been tested before
           construction. Filters, widely used in the water treatment
           industry, were being designed for the project to concentrate and
           remove radioactive particles in liquid waste, a new application
           for the filters. Although tests are currently under way to
           demonstrate the effectiveness of this application, project
           officials conceded that these filters may still not be appropriate
           for the project.

           Other contractor performance problems are illustrated by two
           examples. First, DOE cited the contractor working on the Highly
           Enriched Uranium Materials Facility for inadequate quality
           assurance that resulted in concrete work that did not meet design
           specifications. The subsequent suspension of construction
           activities and rework added to the project's estimated cost and
           schedule. Second, the DOE project director of the Depleted Uranium
           Hexafluoride 6 Conversion Facility told us that the project was
           delayed 2 years because the contractor (1) did not have experience
           in government contracts, (2) underestimated the design effort
           needed, and (3) failed to properly integrate the operations of
           three separate organizations it managed.

           As table 3 shows, external factors were cited as also contributing
           to cost growth and schedule delays, such as additional work to
           implement requirements for higher levels of safety and security in
           project operations, among other things. For example, design rework
           for 4 of the projects occurred in response to external safety
           oversight recommendations by the Defense Nuclear Facilities Safety
           Board that large DOE construction projects meet a certain level of
           personnel safety, and that their designs be robust enough to
           withstand certain seismic events. In addition, owing to new
           security requirements implemented after September 11, 2001,
           project officials on the Highly Enriched Uranium Materials
           Facility had to redesign some aspects of the project to ensure
           that heightened security measures were addressed.

           While DOE faced additional requirements for safety and security,
           it did not always reach timely decisions on how to implement these
           requirements, which contributed significantly to cost increases
           and schedule delays for the Salt Waste Processing Facility. The
           DOE project director for this project told us the Defense Nuclear
           Facilities Safety Board had expressed concerns in June 2004, 5
           months after the preliminary design was started, that the facility
           design might not ensure nuclear wastes would be adequately
           contained in the event of earthquakes. However, DOE did not decide
           how to address this concern until 17 months later, as the project
           continued to move forward with the existing project design.
           According to the project director, better and more timely
           discussions between site officials and headquarters to decide on
           the actions needed to adequately address these safety and security
           requirements might have hastened resolution of the problem, and up
           to 1 year of design rework might have been avoided. The delay, the
           director told us, added $180 million to the total project cost and
           extended the schedule by 26 months. In commenting on our draft
           report, EM officials noted that it is now requiring a more
           rigorous safety analysis earlier in the decision-making process.

           Other external factors also contributed significantly to cost
           increases and delays for 2 interrelated projects we reviewed--the
           Mixed Oxide Fuel Fabrication Facility and the Pit Disassembly and
           Conversion Facility. Project officials for these projects told us
           that 25 to 50 percent of the cost increases and over 70 percent of
           the schedule delays they experienced were the direct result of
           Office of Management and Budget funding constraints and
           restrictions resulting from international agreements with Russia.
           That is, work that is delayed to a subsequent year because of
           funding constraints and other work restrictions can delay project
           completion, which likely increases total project costs. Similarly,
           Office of Science officials, commenting on our draft report,
           stated that external factors caused the largest percentage cost
           increase and schedule delay for the Spallation Neutron Source,
           including a reduced level of funding appropriated at a time when
           project activities and costs were increasing considerably.
           However, congressional funding was reduced in fiscal year 2000
           because of concerns about poor project oversight and management in
           the early stages of this project.

           DOE officials also explained that a now-defunct policy may have
           contributed to increased costs and delays for several projects we
           examined. Until 2000, DOE required contractors to prepare cost and
           schedule estimates early in the project, before preliminary
           designs were completed. These estimates were used to establish a
           baseline for measuring contractor performance and tracking any
           cost increases or schedule delays. However, these estimates often
           were based on early conceptual designs and, thus, were subject to
           significant change as more detailed designs were developed. To
           improve the reliability of these estimates, DOE issued a new order
           in October 2000 that required the preparation of a cost estimate
           range at the start of preliminary design, and delayed the
           requirement for a definitive cost and schedule baseline estimate
           until after the preliminary design was completed. Consequently,
           DOE officials explained, the new policy should result in improved
           estimates and a more accurate measure of cost and schedule
           performance.

           We also sent a survey to DOE project directors for all 12 projects
           asking them to identify key events that led to the greatest cost
           increases or schedule delays, and the major factors contributing
           to these key events. However, no individual factors were
           identified as being major contributors to the cost increases or
           schedule delays. In responding to our survey, DOE project
           directors cited several factors that affected changes in cost and
           schedule. However, when asked to rate the relative significance of
           these factors for their impact on cost and schedule changes, the
           project directors generally did not judge them to be significant
           contributors to the changes. The most frequently cited factors
           were

           o an absence of open communication, mutual trust, and close
           coordination;
           o changes in "political will" during project execution (e.g.,
           project changes resulting from political decisions, both internal
           and external to the project);
           o interruptions in project funding; and
           o project managers' lack of adequate professional experience.

           (For detailed survey results covering these four factors, see app.
           IV.)

           In contrast to the cost increases and schedule delays incurred on
           most of the projects we reviewed, 3 projects had not yet
           experienced cost increases or schedule delays--Microsystems and
           Engineering Sciences Applications, the Linac Coherent Light
           Source, and the Chemistry and Metallurgy Research Facility
           Replacement. DOE project officials identified key conditions that
           they believed helped avoid those cost increases and delays. These
           conditions included

           o active oversight--that is, the DOE project directors were never
           "blindsided" by contractor issues;
           o a lack of technological complexity;
           o an effective system to measure contractor performance;
           o reliable cost estimates;
           o effective communication with and integration of all
           stakeholders; and
           o sustained leadership.

           However, we observed that the Linac Coherent Light Source and the
           Chemistry and Metallurgy Research Facility Replacement facilities
           are still in a relatively early stage in the project development
           process, and thus it may be too early to gauge the overall success
           of either project. Additionally, because none of these 3 projects
           are highly technologically complex, they may be less susceptible
           to the types of problems associated with other projects we
           reviewed that experienced cost increases and delays.
			  
			  DOE Does Not Consistently Measure Technology Readiness to Ensure
			  That Critical Technologies Will Work as Intended before
			  Construction Begins

           Although DOE requires its final designs to be sufficiently
           complete before beginning construction, it has not systematically
           ensured that the critical technologies reflected in project
           designs are technologically ready. Recognizing that a lack of
           technology readiness can result in cost overruns and schedule
           delays, other federal agencies, such as NASA and DOD, have issued
           guidance for measuring and communicating technology readiness.
			  
			    DOE Does Not Consistently Assess Technology Readiness

           Only 1 of the 5 projects we reviewed to determine how DOE ensures
           that project designs are sufficiently complete before
           construction--projects that were approaching or had recently begun
           construction--had a systematic assessment of technology readiness
           to determine whether the project components would work
           individually or collectively as expected in the intended
           design.^15 Specifically, the DOE project director for the Pit
           Disassembly and Conversion Facility systematically measured and
           assessed readiness levels for each critical component of the
           overall project.^16 The assessment was based on a method developed
           by NASA, that is, rating each technology from 0 to 10 in terms of
           relative maturity. Because the project has not yet begun
           construction, we could not determine whether the technology
           readiness assessment has helped project managers to avoid cost
           increases or schedule delays during construction. However,
           according to DOE and contractor officials responsible for the
           project, the assessment helped focus management attention during
           project design on critical technologies that may require
           additional resources to ensure that they are sufficiently ready
           before construction begins. In reviewing the assessment, however,
           we noted that project officials had not updated the assessment
           tool for this project for over 3 years. DOE's project director
           acknowledged the delay in updating the assessment and responded
           that he plans to begin updating the assessment annually.

           The other 4 projects did not have systematic assessments of
           technological readiness. Therefore, the risk associated with the
           technology may not be clearly and consistently understood across
           all levels of management. Formally approving the project's cost
           and schedule estimates as accurate and complete, or proceeding
           into construction, without having clearly assessed evidence of
           technology readiness can result in cost overruns and schedule
           delays.

           DOE's experience with the Waste Treatment and Immobilization Plant
           is a case in point. Specifically, technology known as "pulse jet
           mixers"^17 was used in the design of a subsystem intended to
           prepare radioactive material for processing. However, this
           technology had not been used previously in this application, and
           it did not work in tests as expected, even after construction had
           already begun. Consequently, DOE incurred about $225 million in
           redesign costs and over 1 year in schedule delays, according to
           the DOE project director.

           Over the past several years, we and others have stressed the
           importance of assessing technology readiness to complete projects
           successfully, while avoiding cost increases and schedule delays.
           Specifically, by 1999, we reported that organizations using best
           practices recognize that delaying the resolution of technology
           problems until production or construction can result in at least a
           10-fold cost increase.^18 Furthermore, we reported that delaying
           the resolution until after the start of production could increase
           costs by 100-fold. Reporting on similar concerns, the National
           Research Council has identified factors common to large
           construction projects--in the areas of cost, schedule, and
           scope--that help to ensure projects are completed successfully.^19
           Among key technical conditions for defining project scope, the
           council stated, is a project plan that is based on employing the
           best available, state-of-the-art technology, but not experimental
           or unproven technology. As such, employing a consistent,
           systematic method for measuring the extent to which technology is
           still experimental or unproven is of critical importance.

           An assessment of technology readiness is even more crucial at
           certain points in the life of a project--particularly as DOE
           decides to accept a project's (1) preliminary design and formally
           approve the project's cost and schedule estimates as accurate and
           complete and (2) final design as sufficiently complete so that
           resources can be committed toward procurement and construction.
           Proceeding through these critical decision points without a
           credible and complete technology readiness assessment can lead to
           problems later in the project. Specifically, if DOE proceeds with
           the project when technologies are not yet ready, there is less
           certainty that the technologies specified in the preliminary or
           final designs will work as intended. Project managers may then
           need to modify or replace these technologies to make them work
           properly, which can result in costly and time-consuming redesign
           work.

           Moreover, modifying the design of a facility after construction
           has already begun can be expensive and time consuming. First,
           changes to an already designed work plan are not necessarily
           subject to competition because the new work can occur through
           "change orders"--that is, modifications to existing contracts.
           These change orders can be expensive, according to DOE project
           directors. Second, worker productivity can be lost if, for
           example, extra downtime results from delays to interrelated
           construction work. Finally, tearing down and rebuilding items
           already constructed, such as concrete floors, walls, and doors,
           might be necessary to accommodate a design change.

           DOE's experience in the predecessor project to the Salt Waste
           Processing Facility--the In-Tank Precipitation (ITP) project
           process--at the Savannah River Site illustrates the potential
           consequences of proceeding with technology that is not
           sufficiently ready. As we reported in 2000, the ITP project was
           selected in 1983 as the preferred method for separating highly
           radioactive material from 34 million gallons of liquid stored at
           the Savannah River site--a step considered necessary to
           effectively handle this large quantity of waste.^20 A 1983 test
           using the ITP technology on a tank containing 500,000 gallons of
           waste resulted in a significant buildup of benzene--a highly
           explosive and hazardous compound. The buildup of benzene was more
           than the tank instruments could register. Nevertheless, project
           managers decided to proceed with the project. In 1985, DOE
           estimated that it would take about 3 years and $32 million to
           construct the ITP facility. After a number of delays, the ITP
           facility was constructed and began start-up operations in 1995,
           which were halted because of safety concerns about the amount of
           benzene that the facility generated. In 1998, after about a decade
           of delays and costs of almost $500 million, DOE suspended the
           project because it did not work as safely and efficiently as
           designed. This suspension put an effective remedy for treating
           high-level waste at the Savannah River Site years behind schedule.
           DOE then directed its contractor to begin a process to identify
           and select an alternative technology, which has developed into the
           current project intended to treat this waste--the Salt Waste
           Processing Facility project.

           In response to our concerns about the 4 projects without
           systematic assessments of technology readiness, DOE project
           directors explained that they have alternative methods for
           assessing readiness. They are required to submit a project
           execution plan, which includes an assessment of risks, including
           technological risks, and a plan for mitigating risks. They also
           rely upon independent reviews, including extensive design reviews,
           before making critical decisions to accept designs, and cost and
           schedule estimates, or to proceed with construction. For example,
           DOE's Office of Engineering and Construction Management formally
           reviews major projects in an effort to ensure that the designs are
           sufficiently complete to begin construction. Specifically, an
           external independent readiness review is performed, often using
           the services of various independent technical experts, that, at a
           minimum, is intended to verify the readiness of the project to
           proceed into construction or to identify remedial action. Finally,
           several DOE project directors stated that they intentionally have
           avoided using fast-track, design-build approaches because of the
           many problems it posed for the Waste Treatment and Immobilization
           Plant project. The DOE project directors of the 5 DOE projects
           that are nearing, or have recently begun construction, told us
           they have completed, or expect to complete prior to construction,
           85 to 100 percent of their projects' final design.

           In addition to following the more standard approaches for managing
           projects, such as preparing risk assessment plans, some DOE
           offices have developed their own tools for assessing the readiness
           of projects. For example, DOE's Office of Environmental Management
           (EM) uses a Product Definition Rating Index (PDRI) as a tool to
           assess how well a project is planned, and whether it is ready to
           proceed to the next project phase. Project elements rated include
           cost, schedule, scope/technical, management planning and control,
           and external factors. Among the 77 project elements rated, 2
           involve technology--the identification of technology development
           requirements, and the testing and evaluation of the technology to
           be used. While the project technologies are collectively given a
           ranking with this tool, the PDRI does not represent a rigorous
           examination of the demonstrated readiness of each critical
           technology for its application in the project. Furthermore, not
           all EM projects we examined were using this tool.

           DOE's design reviews, risk assessments, and other actions to
           monitor design completion are extensive and certainly have merit.
           However, we found that these actions alone do not provide
           consistent and transparent assurance that all technologies are
           sufficiently ready because they do not use a consistent and
           systematic method of measurement. DOE's project design reviews,
           for example, do not always clearly distinguish between technology
           that has been demonstrated to work as expected in the intended
           design versus a judgment that the technology has potential for
           reaching a specific level of readiness.

           The external review of the technologies for the Mixed Oxide Fuel
           Fabrication Facility illustrates the shortfalls in DOE's current
           approach to assessing technology readiness and communicating the
           results of those assessments.^21 The report concluded, among other
           things, that the method chosen by the contractor is the most
           rigorous and comprehensive, and should result in the most
           successful technology transfer possible. Furthermore, the review
           team was very impressed with the rigor with which designs and
           design changes were being managed, finding ample evidence
           verifying that the exact design process used by the French was
           being transferred to the United States facility. Although the
           external reviewers seemed to be impressed with many aspects of the
           design transfer, and concluded that the technologies should not be
           problematic, they had identified some key concerns about
           technology readiness in the body of their final report. The
           reviewers did not explain how they reconciled their conclusion
           with their concerns. To reconcile these differences, we obtained
           several clarifying statements from DOE's project director,
           technical experts, and one of the study's authors. These
           clarifying statements appear to support the reviewers'
           conclusions. However, without these statements, the level of
           technological readiness was not readily evident because the
           independent review lacked consistent, systematic criteria and a
           method for measuring the degree of readiness or clearly
           communicating assessment results, and the review was not
           transparent.

           DOE does not consistently assess technology readiness of project
           technologies because its project management guidance lacks
           comprehensive standards for systematically measuring and
           communicating the readiness of project technologies. Specifically,
           DOE lacks consistent metrics for determining technology readiness
           departmentwide, terminology to facilitate effective communication,
           and oversight protocols for reporting and reviewing technology
           readiness levels. DOE project management guidance is contained in
           two key documents--DOE Order 413.3A and Manual 413.3-1. Although
           the manual requires final designs to be sufficiently complete
           before beginning construction, it does not specify how
           technologies reflected in project designs are to be assessed for
           readiness--to determine that they have been sufficiently
           demonstrated to work as intended. Consequently, critical decisions
           made without standard measures are susceptible to varying
           interpretations of the actual technology readiness attained and
           the level needed for a project to proceed, which can easily vary
           among projects and among officials within a single project.
			  
			    Other Federal Agencies Use a Standard Method for Measuring and
				 Communicating Technology Readiness

           Other federal agencies have recognized the importance of ensuring
           that technologies have been sufficiently demonstrated for their
           intended purpose and have issued standard guidance for measuring
           and communicating TRLs. In particular, recognizing the need to
           measure the readiness level of project technologies, NASA began
           using a systematic method of measurement in the mid-1990s. NASA
           incorporated a structured TRL approach into guidance on integrated
           technology planning.

           Similarly, to improve DOD management of risk and technology
           development, the Deputy Under Secretary of Defense (Science and
           Technology) officially endorsed, in a July 2001 memorandum, the
           use of TRLs in new major programs. In 2002, DOD issued mandatory
           procedures for major defense acquisition programs and major
           automated information system acquisition programs, which
           identified technology readiness as a principal element of program
           risk. The procedures require the military services' science and
           technology officials to conduct a systematic assessment of
           critical technologies that are identified in major weapon systems
           programs before starting engineering and manufacturing development
           and production. Using TRLs is the preferred method, and approval
           must be obtained from the Deputy Under Secretary if an equivalent
           alternative method is used, according to the Deputy Under
           Secretary's memorandum. Importantly, the procedures stated that
           TRLs are a measure of demonstrated technical maturity--they do not
           discuss the probability of occurrence (i.e., the likelihood of
           attaining required maturity) or the impact of not achieving
           technology maturity.

           Both NASA and DOD use a nine-point scale to measure technology
           readiness, from a low of TRL 1 (basic principles observed) to a
           high of TRL 9 (total system used successfully in project
           operations). (App. V contains the definitions of these nine TRLs.)
           For example, a subsystem prototype that has been successfully
           demonstrated in an operational environment would receive a higher
           TRL value (i.e., TRL 7) than a technological component that has
           been demonstrated in a laboratory test (i.e., TRL 4). In our
           previous work, we recommended to the Secretary of Defense that key
           project technologies used in weapons systems be demonstrated in an
           operational environment, reaching a high maturity level--analogous
           to TRL 7--before deciding to commit to a cost, schedule, and
           performance baseline for development and production of the weapon
           system.^22 In response to our recommendation, DOD has agreed that
           if a technology does not achieve a score of TRL 6 or 7, project
           managers must develop a plan to bring the technology to the
           required readiness level before proceeding to the next project
           phase.

           Use of TRLs is not by itself a cure-all for managing critical
           technologies, but TRLs can be used in conjunction with other
           measures to improve the way projects are managed. For example,
           according to studies by NASA, DOD, and others, TRLs can

           o provide a common language among the technology developers,
           engineers who will adopt/use the technology, and other
           stakeholders;
           o improve stakeholder communication regarding technology
           development--a by-product of the discussion among stakeholders
           that is needed to negotiate a TRL value;
           o reveal the gap between a technology's current readiness level
           and the readiness level needed for successful inclusion in the
           intended product;
           o identify at-risk technologies that need increased management
           attention or additional resources for technology development to
           initiate risk-reduction measures; and
           o increase transparency of critical decisions by identifying key
           technologies that have been demonstrated to work or by
           highlighting still immature or unproven technologies that might
           result in high project risk.

           Two DOE headquarters offices have attempted to systematically
           assess technology readiness. First, under the Office of Nuclear
           Energy, a DOE contractor preparing a congressional report used a
           TRL method to compare the maturity of advanced fuel cycle
           technologies. In addition, in 2000, DOE's Office of Science and
           Technology, under EM, issued a report that defined a process for
           assessing technology maturity of EM projects.^23 However,
           according to an EM official, the office decided to discontinue
           using this assessment process because it was considered overly
           burdensome. As a result, DOE devolved responsibility for managing
           technology readiness to the contractor level.

           According to several DOE project directors we spoke with, a
           consistent, systematic method for assessing technology readiness
           would help achieve a number of objectives: that is, standardize
           terminology, make technology assessments more transparent, and
           improve communication among project stakeholders before they make
           critical project decisions. DOE project managers also acknowledged
           that TRLs could improve project management departmentwide, and
           some managers are now attempting to use this tool to assess
           technology maturity. The DOE project director for the Waste
           Treatment and Immobilization Plant told us that a senior DOE
           official encouraged him to begin using TRLs. He is consulting with
           DOD officials knowledgeable about using the TRL method and expects
           to develop a TRL tool and have TRL determinations for major parts
           of the project in 2007. (App. VI compares DOD's product
           development process with DOE's project management process for
           major projects.)
			  
			  Conclusions

           The magnitude of the cost increases and schedule delays for DOE's
           major projects is cause for serious rethinking of how DOE manages
           them. To its credit, DOE has completed, or expects to complete
           prior to construction, 85 to 100 percent of project design work
           for the 5 projects we reviewed that have recently begun or are
           nearing construction. However, DOE has not systematically
           addressed another key factor--the readiness level of the
           technologies it expects to use in these projects. DOE lacks
           comprehensive standards in DOE Order 413.3A and Manual 413.3-1 for
           systematically measuring and communicating the readiness of
           project technologies. Specifically, the department lacks
           consistent metrics for determining technology readiness
           departmentwide, terminology, and oversight protocols for reporting
           and reviewing TRLs. Without consistent measurement and
           communication of the readiness of technologies, DOE does not have
           a basis for defining the acceptable level of technological risk
           for each project, making critical decisions on accepting the
           validity of a project's total estimated cost and schedule, or
           proceeding with construction.

           Other federal agencies have recognized the need to consistently
           measure and communicate technology readiness to help avoid cost
           increases and delays that result from relying on immature
           technologies. DOD, for example, requires its managers to use a TRL
           process to measure technology readiness and generally requires a
           TRL 7 (as we had recommended) before system development and
           demonstration. In contrast, as DOE's poor track record for
           managing the technological complexity of major projects shows, DOE
           has not systematically measured the readiness of critical project
           technologies before it approves definitive cost and schedule
           estimates or begins construction. Furthermore, without a
           systematic method for measuring technological readiness, DOE
           cannot effectively communicate within the department and to the
           Congress whether projects are at risk of experiencing cost
           increases and schedule delays associated with technology problems.
			  
			  Recommendations for Executive Action

           To improve decision making and oversight for major DOE
           construction projects, including how project technology readiness
           is measured and reported, we recommend that the Secretary of
           Energy evaluate and consider adopting a disciplined and consistent
           approach to assessing TRLs for projects with critical technologies
           that includes the following three actions:

           o Develop comprehensive standards for systematically measuring and
           communicating the readiness of project technologies. At a minimum,
           these standards should (1) specify consistent metrics for
           determining technology readiness departmentwide, (2) establish
           terminology that can be consistently applied across projects, and
           (3) detail the oversight protocols to be used in reporting and
           reviewing TRLs. In preparing these standards, DOE should consider
           lessons learned from NASA and DOD, and its own experience in
           measuring technology readiness. If DOE's evaluation results in the
           decision to adopt these standards, it should incorporate them into
           DOE Order 413.3A and Manual 413.3-1, and provide the appropriate
           training to ensure their proper implementation.
           o Direct DOE Acquisition Executives to ensure that projects with
           critical technologies reach a level of readiness commensurate with
           acceptable risk--analogous to TRL 7--before deciding to approve
           the preliminary design and commit to definitive cost and schedule
           estimates, and at least TRL 7 or, if possible, TRL 8 before
           committing to construction expenses.
           o Inform the appropriate committees and Members of Congress of any
           DOE decision to approve definitive cost and schedule estimates, or
           to begin construction, without first having ensured that project
           technologies are sufficiently ready (at TRL 7 or 8). This
           information should include specific plans for mitigating
           technology risks, such as developing backup technologies to offset
           the effects of a potential technology failure, and appropriate
           justification for accepting higher technological risk.
			  
			  Agency Comments and Our Evaluation

           We provided a draft of this report to DOE for its review and
           comment. DOE's written comments are reproduced in appendix VII.
           DOE agreed with our recommendations but suggested revisions that
           would allow it to first conduct a pilot application on selected
           projects to better understand the technology readiness assessment
           process and evaluate its potential use. We revised our
           recommendations to give DOE this flexibility. DOE also provided
           detailed technical comments, which we have incorporated into our
           report as appropriate.

           DOE also expressed several specific concerns with our draft
           report. First, DOE stated that while our draft broadly asserts
           that DOE project management has led to increases in cost and
           schedule, our recommendations are narrowly focused on technology
           assessment. We agree that our draft states that DOE project
           management has led to cost increases and schedule delays, a
           conclusion we reached on the basis of our contact with DOE project
           directors and our review of numerous studies and reports on DOE
           major projects. Our recommendations address technology assessment,
           a critical project management activity, because they were
           developed primarily on the basis of our specific finding that DOE
           lacks a systematic approach to ensure that final project designs,
           including critical technologies reflected in these designs, have
           been demonstrated to work as intended prior to construction. This
           report explains that delaying resolution of technology problems
           until construction can potentially lead to significant cost
           increases and schedule delays.

           Second, DOE stated that our draft report inappropriately
           characterizes cost and schedule growth from a small sample of
           projects by using preliminary cost and schedule estimates that are
           intended for internal DOE planning. To clarify, the scope of our
           review included an evaluation of DOE's major construction
           projects. In addition, our report explains that DOE changed its
           project management policy in 2000 to allow cost and schedule
           estimates to be prepared later in the project--at the end of
           preliminary design. Prior to this new policy, project directors
           submitted cost and schedule estimates earlier in the project
           development phase--at the end of conceptual design. For projects
           under way prior to the policy in 2000, we used post-2000 validated
           baseline estimates, if available. Otherwise, we used earlier
           estimates since these were the only estimates available and had
           been previously used by DOE to inform Congress of the total
           expected project cost and schedule while seeking initial project
           funding. We also note that for the five projects that were started
           after the new policy in 2000, we used the validated project
           baseline estimates recommended by DOE, if available.

           Third, DOE suggested we revise table 3 in our report to more
           clearly identify the correlation between cost and schedule growth
           and technology maturity. As our report states, the information in
           table 3 was drawn from the results of our review of independent
           studies involving the projects we reviewed and the results of our
           interviews with DOE project directors. Our report explains that
           cost increases and schedule delays for 6 of the 9 projects shown
           in the table were due in part to contractors' poor management of
           the development and integration of technologies used in the
           project designs.

           Finally, DOE stated that it is unclear how the factors cited in
           appendix IV, such as communication, and changes in "political
           will," among other things, led to our recommendation to assess
           technology readiness. Although not all of the factors cited in our
           survey have a link to our recommendation on technology readiness,
           one factor in particular--absence of communication--is addressed
           in our recommendation. Specifically, we recommended that the
           Secretary of Energy consider developing comprehensive standards
           for systematically measuring and communicating the readiness of
           project technologies, including the establishment of terminology
           that is to be consistently applied across projects.
			  
^11Department of Energy, Office of Inspector General, Audit Report: Status
of the Mixed Oxide Fuel Fabrication Facility, DOE/IG-0713 (Washington,
D.C.: December 2005).

^12 [60]GAO-06-602T .

^13Department of Energy, Office of Inspector General, Audit Report: The
Department of Energy's Tritium Extraction Facility, DOE/IG-0560
(Washington, D.C.: June 2002).

^14These 6 projects are the Mixed Oxide Fuel Fabrication Facility,
National Ignition Facility, Pit Disassembly and Conversion Facility,
Spallation Neutron Source, Tritium Extraction Facility, and Waste
Treatment and Immobilization Plant.

^15These 5 projects are the Chemistry and Metallurgy Research Facility
Replacement, Depleted Uranium Hexafluoride 6 Conversion Facility, Mixed
Oxide Fuel Fabrication Facility, Pit Disassembly and Conversion Facility,
and Salt Waste Processing Facility.

^16Los Alamos National Laboratory, Options for the Development and Testing
of the Pit Disassembly and Conversion Facility Government-Furnished
Design, LA-UR-03-3926 (Los Alamos, New Mexico: June 11, 2003).

^17Pulse jet mixers, which do not have moving parts, use compressed air to
continuously mix tank waste so that it can be properly prepared for
further processing. While such devices have previously been used
successfully in other applications, they have never been used for mixing
wastes with high-solid content like those at the Waste Treatment and
Immobilization Plant.

^18 [61]GAO/NSIAD-99-162 and GAO, Joint Strike Fighter Acquisition: Mature
Critical Technologies Needed to Reduce Risks, [62]GAO-02-39 (Washington,
D.C.: Oct. 19, 2001).

^19Improving Project Management.

^20GAO, Department of Energy: Uncertainties and Management Problems Have
Hindered Cleanup at Two Nuclear Waste Sites, [63]GAO/T-RCED-00-248
(Washington, D.C.: July 12, 2000).

^21Burns and Roe Enterprises, Inc., External Independent Review of the
Basis of Design for the Aqueous Polishing Process for the Mixed Oxide Fuel
Fabrication Facility at The Savannah River Site for the U.S. Department of
Energy Office of Engineering and Construction Management and National
Energy Technology Laboratory Report, BREI-LSP-R-06-01 (Oradell, New
Jersey: March 2006).

^22 [64]GAO/NSIAD-99-162 .

^23Department of Energy, Tracking Technology Maturity in DOE's
Environmental Management Science and Technology Program; Revision 1
(Washington, D.C.: Jan. 1, 2001).

           We are sending copies of the report to interested congressional
           committees, the Secretary of Energy, and the Director of the
           Office of Management and Budget. We will make copies available to
           others on request. In addition, the report will also be available
           at no charge on the GAO Web site at http://www.gao.gov .

           If you or your staffs have any questions about this report, please
           contact me at (202) 512-3841 or [email protected]. Contact points
           for our Offices of Congressional Relations and Public Affairs may
           be found on the last page of this report. Other staff contributing
           to the report are listed in appendix VIII.

           Gene Aloise
			  Director, Natural Resources and Environment
			  
			  Appendix I: Scope and Methodology

           To determine the extent to which the Department of Energy's (DOE)
           major construction projects have experienced cost increases and
           schedule delays and the factors that have contributed to these
           problems, we identified (1) active DOE major line-item
           construction projects that have current total project cost
           estimates above the $750 million threshold--DOE's criteria for
           "major construction projects," and (2) the projects with estimates
           above $400 million--the DOE threshold for major projects until
           July 2006. We also identified those projects above $300 million to
           account for any projects that may pass the $400 million
           threshold.^1 In all, we identified the following 12 projects:

           o Five of these 12 projects began before DOE moved its requirement
           for firm cost and schedule estimates to later in the project: the
           National Ignition Facility, the Mixed Oxide Fuel Fabrication
           Facility, the Pit Disassembly and Conversion Facility, the
           Spallation Neutron Source, and the Tritium Extraction Facility. We
           used the estimates at the end of conceptual design, as reported by
           project directors, for the initial project cost and schedule
           estimates.
           o Four of the remaining 7 projects had cost and schedule estimates
           completed at the end of preliminary design, according to the new
           DOE guidelines: the Highly Enriched Uranium Materials Facility,
           Microsystems and Engineering Sciences Applications, the Depleted
           Uranium Hexafluoride 6 Conversion Facilities, and the Linac
           Coherent Light Source. For these projects, we considered the
           estimates as reported by project directors to be the initial cost
           and schedule estimates.
           o One project, the Waste Treatment and Immobilization Plant, began
           after DOE moved the requirement for firm cost and schedule
           estimates to later in the project. However, DOE initially exempted
           the contractor from submitting firm cost and schedule estimates.
           Therefore, we used the estimates reported by the project director
           to be the initial cost and schedule estimates.
           o The final 2 projects, although falling under the new DOE
           requirements, had yet to complete their preliminary design at the
           time of our review: the Chemistry and Metallurgy Research Facility
           Replacement and the Salt Waste Processing Facility. For these
           projects, we considered the cost and schedule estimates at the end
           of conceptual design reported by project directors to be the
           initial project cost and schedule estimates.

           Because we and others have previously expressed concern about the
           data reliability of a key DOE project management tracking
           database--the Project Assessment and Reporting System--we did not
           develop conclusions or findings based on information generated
           through that system.^2 Instead, we collected information directly
           through surveys and interviews with project site officials.

           To identify cost increases and schedule delays, and the factors
           that may have contributed to these changes, we surveyed DOE
           project directors, interviewed DOE and contractor project
           personnel, and reviewed project management documents for 12 major
           projects. These 12 projects are managed by DOE's Office of
           Science, Office of Environmental Management (EM), or National
           Nuclear Security Administration (NNSA). (See app. II for
           information on these projects.)

           Our survey asked DOE project directors of the 12 projects to
           identify the degree to which cost and schedule estimates may have
           changed and the reasons for these changes, and to describe the
           events and conditions that led to any changes. Eight of the 12
           project directors responded that their projects had experienced
           cost increases and schedule delays, and 1 project director
           reported only a schedule delay. For these 9 projects, we asked
           project directors to (1) identify the top three events that led to
           the cost and schedule delays and (2) indicate to what extent
           certain factors may have contributed to the event that led to the
           largest percentage cost increase or schedule delay. The factors
           included in the survey instrument were based on the results of a
           National Research Council study that listed essential or important
           conditions needed for the successful completion of major
           projects.^3 We asked project directors to identify the extent to
           which the lack of these conditions may have contributed to any
           cost and schedule delays. (App. IV shows key survey results for
           these 9 projects.)

           In addition to reviewing project documentation, we conducted site
           visits for the 9 projects that had experienced cost and schedule
           changes, and we analyzed (1) studies of these projects completed
           by DOE's Office of Inspector General and (2) external independent
           project reviews conducted under the direction of DOE's Office of
           Engineering and Construction Management in Washington, D.C. We
           interviewed federal project directors of the 3 projects that had
           not experienced cost increases or schedule delays to obtain
           information on factors they believe are important in avoiding such
           increases.

           To determine the extent to which DOE ensures that project designs
           are sufficiently complete before construction, we obtained
           additional information from project directors on 5 projects that
           were approaching, or had recently begun, construction. During our
           review, we obtained information on the extent project designs
           were, or are expected to be, complete before beginning
           construction, and the actions DOE had taken to ensure technologies
           used in these designs are sufficiently ready to begin
           construction. For 2 of these 5 projects, we applied a tool we
           previously had used to assess DOD programs--the tool enables
           project directors to characterize the readiness level of each
           technology being developed for use in aircraft and other military
           applications. In addition, we spoke with officials from DOE
           program offices and DOE's Office of Engineering and Construction
           Management in Washington, D.C.

           We provided interim briefings to the Subcommittee on Energy and
           Water Development, House Committee on Appropriations, on the
           status of our work in May and September, 2006. We performed our
           work between December 2005 and January 2007, in accordance with
           generally accepted government auditing standards.
			  
^1We excluded the Yucca Mountain Repository project, with a total
estimated cost of $23 billion, from our review due to its uniqueness and
the fact that we have recently reported on the project and currently have
an ongoing review. Also, to review projects with sufficient maturity, we
included only the projects that were at least 1 year past completion of
conceptual design.

^2GAO, Department of Energy: Further Actions Are Needed to Strengthen
Contract Management for Major Projects, [65]GAO-05-123 (Washington, D.C.:
Mar. 18, 2005); and Civil Engineering Research Foundation, Independent
Research Assessment of Project Management Factors Affecting Department of
Energy Project Success (Washington, D.C.: July 12, 2004).

^3National Research Council, Improving Project Management in the
Department of Energy (Washington, D.C.: July 1999).
			  
			  Appendix II: Information on the 12 Department of Energy Major
			  Projects Reviewed

Project              DOE program office Project purpose/objectives         
Chemistry and        National Nuclear   Relocate and consolidate           
Metallurgy Research  Security           mission-critical analytical        
Facility Replacement Administration     chemistry, material                
                                           characterization, and research and 
                                           development capabilities to ensure 
                                           continuous national security       
                                           mission support beyond 2010.       
Depleted Uranium     Office of          Design and construct facilities at 
Hexafluoride 6       Environmental      Portsmouth, Ohio, and Paducah,     
Conversion Facility  Management         Kentucky, to convert the           
                                           Department of Energy's existing    
                                           inventory of depleted uranium      
                                           hexafluoride into a more stable    
                                           form for disposal or beneficial    
                                           reuse.                             
Highly Enriched      National Nuclear   Project will construct a highly    
Uranium Materials    Security           secure, state-of-the-art facility  
Facility             Administration     for consolidating and storing      
                                           highly enriched uranium, resulting 
                                           in cost savings and an increased   
                                           security posture.                  
Linac Coherent Light Science            Provide laser-like radiation in    
Source                                  the X-ray region of the spectrum   
                                           that is 10 billion times greater   
                                           in peak brightness than any        
                                           existing X-ray light source. The   
                                           project will apply these           
                                           high-brightness X-rays to          
                                           experiments in the chemical,       
                                           material, and biological sciences. 
Microsystems and     National Nuclear   Provide state-of-the-art national  
Engineering Sciences Security           complex that will provide for the  
Applications         Administration     design, integration, prototyping,  
                                           and qualification of microsystems  
                                           into components, subsystems, and   
                                           systems within the nuclear weapons 
                                           stockpile.                         
Mixed Oxide Fuel     National Nuclear   Facility will combine surplus      
Fabrication Facility Security           weapon-grade plutonium oxide with  
                        Administration     depleted uranium to form mixed     
                                           oxide fuel assemblies that will be 
                                           irradiated in United States        
                                           commercial nuclear reactors. Once  
                                           irradiated and converted into      
                                           spent fuel, the resulting          
                                           plutonium can no longer be readily 
                                           used for nuclear weapons.          
National Ignition    National Nuclear   Provide experimental capability to 
Facility             Security           assess nuclear weapons physics,    
                        Administration     providing critical data that will  
                                           allow the United States to         
                                           maintain its technical             
                                           capabilities in nuclear weapons in 
                                           the absence of underground         
                                           testing, and to advance fusion as  
                                           an energy source.                  
Pit Disassembly and  National Nuclear   Eliminate surplus Russian and      
Conversion Facility  Security           United States plutonium and highly 
                        Administration     enriched uranium by disassembling  
                                           surplus nuclear weapons pits and   
                                           converting the resulting plutonium 
                                           metal to a powder form that can    
                                           later be fabricated into mixed     
                                           oxide fuel to produce nuclear fuel 
                                           assemblies for use in commercial   
                                           nuclear reactors.                  
Salt Waste           Office of          Meet site cleanup goals and reduce 
Processing Facility  Environmental      significant environmental and      
                        Management         health/safety risk by construction 
                                           of a facility to treat large       
                                           quantities of waste from           
                                           reprocessing and nuclear materials 
                                           production operations at the       
                                           Savannah River Site. Process will  
                                           separate waste, solidify it in     
                                           glass, and send it to federal      
                                           repositories for disposal.         
Spallation Neutron   Science            Provide next generation,           
Source                                  short-pulse spallation neutron     
                                           source for neutron scattering, to  
                                           be used by researchers from        
                                           academia, national and federal     
                                           labs, and industry for basic and   
                                           applied research and technology    
                                           development in the fields of       
                                           condensed matter physics,          
                                           materials sciences, magnetic       
                                           materials, polymers and complex    
                                           fluids, chemistry, biology, earth  
                                           sciences, and engineering.         
Tritium Extraction   National Nuclear   To replenish the tritium needs of  
Facility             Security           the nuclear weapons stockpile, the 
                        Administration     facility will extract tritium      
                                           produced in a commercial nuclear   
                                           reactor for use in nuclear weapons 
                                           development.                       
Waste Treatment and  Office of          The plant will separate high-level 
Immobilization Plant Environmental      from low-level radioactive waste   
                        Management         currently stored in underground    
                                           tanks, processing and solidifying  
                                           all high-level waste and a         
                                           substantial portion of the         
                                           low-level waste, and will treat    
                                           the remaining low-level waste.     

           Source: DOE.
			  
			  Appendix III: Independent Studies Reviewed
			  
			  National Ignition Facility

           Department of Energy, Office of Inspector General. Audit Report:
           Status of the National Ignition Facility Project. DOE/IG-0598.
           Washington, D.C.: April 28, 2003.

           GAO. Department of Energy: Status of Contract and Project
           Management Reforms. [35]GAO-03-570T . Washington, D.C.: March 20,
           2003.

           GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to
           Ensure Initiatives Have Improved Results. [36]GAO-02-798 .
           Washington, D.C.: September 13, 2002.

           GAO. Department of Energy: Follow-up Review of DOE's National
           Ignition Facility. [37]GAO-01-677R . Washington, D.C.: June 1,
           2001.

           GAO. National Ignition Facility: Management and Oversight Failures
           Caused Major Cost Overruns and Schedule Delays.
           [38]GAO/RCED-00-141 and [39]GAO/RCED-00-271 . Washington, D.C.:
           August 8, 2000.

           The Mitre Corporation. NIF Ignition. JSR-05-340. McLean, VA: June
           29, 2005.
			  
			  Mixed Oxide Fuel Fabrication Facility

           Burns and Roe Enterprises, Inc. External Independent Review of the
           Mixed Oxide Fuel Fabrication Facility (MFFF) Project Critical
           Decision (CD) 2/3 Baseline: Performance Baseline (CD-2) and Start
           of Construction (CD-3) Review. BREI-L-R-06-03. Oradell, NJ: July
           7, 2006.

           Burns and Roe Enterprises, Inc. External Independent Review of the
           Basis of Design for the Aqueuous Polishing Process.
           BREI-SLP-R-06-01. Oradell, NJ: March 27, 2006.

           Civil Engineering Research Foundation. Independent Research
           Assessment of Project Management Factors Affecting Department of
           Energy Project Success. Washington, D.C.: July 12, 2004.

           Department of Energy, Office of Inspector General. Audit Report:
           Status of the Mixed Oxide Fuel Fabrication Facility. DOE/IG-0713.
           Washington, D.C.: December 21, 2005.
			  
			  Pit Disassembly and Conversion Facility

           Department of Energy, Office of Inspector General. Audit Report:
           National Nuclear Security Administration's Pit Disassembly and
           Conversion Facility. DOE/IG-0688. Washington, D.C.: May 3, 2005.

           Los Alamos National Laboratory. Options for the Development and
           Testing of the Pit Disassembly and Conversion Facility
           Government-Furnished Design. LA-UR-03-3926. Los Alamos, NM: June
           11, 2003.
			  
			  Waste Treatment and Immobilization Plant

           Bechtel National, Inc. Hanford Tank Waste Treatment and
           Immobilization Plant, May 2006 Estimate at Completion. Hanford
           Site, WA: May 31, 2006.

           Bechtel National, Inc. Comprehensive Review of the Hanford Tank
           Waste Treatment and Immobilization Plant Estimate at Completion.
           CCN 132848. Hanford Site, WA: March 31, 2006.

           Bechtel National, Inc. Comprehensive Review of the Hanford Waste
           Treatment Plant Flowsheet and Throughput. CCN132846. Hanford Site,
           WA: March 17, 2006.

           Bechtel National, Inc. Hanford Tank Waste Treatment and
           Immobilization Plant, December 2005 Estimate at Completion
           Executive Summary. Hanford Site, WA: January 30, 2006.

           Department of the Army Corp of Engineers. Complete Statement of
           Kim Callan, to the Subcommittee on Energy and Water Development,
           Committee on Appropriations, United States House of
           Representatives. Washington, D.C.: April 6, 2006.

           Department of Energy. External Independent Review, Independent
           Cost Review, CD-3C Review of the Waste Treatment and
           Immobilization Plant Project. Hanford Site, WA: September 2002.

           Department of Energy. External Independent Review CD-3B Review of
           the Waste Treatment and Immobilization Plant Project. Hanford
           Site, WA: April 2002.

           GAO. Hanford Waste Treatment Plant, Contractor and DOE Management
           Problems Have Led to Higher Costs, Construction Delays, and Safety
           Concerns. [40]GAO-06-602T . Washington, D.C.: April 6, 2006.

           GAO. Further Actions Are Needed to Strengthen Contract Management
           for Major Projects. [41]GAO-05-123 . Washington, D.C.: March 18,
           2005.

           GAO. Nuclear Waste: Absence of Key Management Reforms on Hanford's
           Cleanup Project Adds to Challenges of Achieving Cost and Schedule
           Goals. [42]GAO-04-611 . Washington, D.C.: June 9, 2004.

           GAO. Status of Contract and Project Management Reforms.
           [43]GAO-03-57T . Washington, D.C.: March 20, 2003.

           GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to
           Ensure Initiatives Have Improved Results. [44]GAO-02-798 .
           Washington, D.C.: September 13, 2002.

           GAO. Nuclear Waste: Hanford Tank Waste Program Needs Cost,
           Schedule, and Management Changes. [45]GAO/RCED-93-99 . Washington,
           D.C.: March 8, 1993.

           LMI Government Consulting. Hanford Waste Treatment and
           Immobilization Plant After-Action Fact-Finding Review. DE535T1.
           McLean, VA: January 2006.

           LMI Government Consulting. External Independent Review, Follow-up
           Review, Waste Treatment and Immobilization Plant (WTP) Out
           Briefing. Washington, D.C.: March 14, 2003.
			  
			  Spallation Neutron Source

           Civil Engineering Research Foundation. Independent Research
           Assessment of Project Management Factors Affecting Department of
           Energy Project Success. Washington, D.C.: July 12, 2004.

           Department of Energy, Office of Inspector General. Audit Report:
           Progress of the Spallation Neutron Source Project. DOE/IG-0532.
           Washington, D.C.: November 19, 2001.

           Department of Energy. Review Committee Report on the Baseline
           Review of the Spallation Neutron Source (SNS) Project. Washington,
           D.C.: July 15, 1999.

           Department of Energy. Technical, Cost, Schedule, and Management
           Review of the Spallation Neutron Source Project. Washington, D.C.:
           January 28, 1999.

           GAO. Department of Energy: Status of Contract and Project
           Management Reforms. [46]GAO-03-570T . Washington, D.C.: March 20,
           2003.

           GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to
           Ensure Initiatives Have Improved Results. [47]GAO-02-798 .
           Washington, D.C.: September 13, 2002.

           GAO. Department of Energy: Challenges Exist in Managing the
           Spallation Neutron Source Project. [48]GAO/T-RCED-99-103 .
           Washington, D.C.: March 3, 1999.
			  
			  Salt Waste Processing Facility

           Department of Energy, Office of Inspector General. Audit Report:
           Salt Processing Project at the Savannah River Site. DOE/IG-0565.
           Washington, D.C.: August 27, 2002.

           Institute for Regulatory Science. Technical Peer Review Report of
           the Review Panel on Salt Waste Processing Facility Technology
           Readiness. CRTD-Vol. 75. Danvers, MA: October 31, 2003.
			  
			  Tritium Extraction Facility

           Civil Engineering Research Foundation. Independent Research
           Assessment of Project Management Factors Affecting Department of
           Energy Project Success. Washington, D.C.: July 12, 2004.

           Department of Energy, Office of Inspector General. Audit Report:
           The Department of Energy's Tritium Extraction Facility.
           DOE/IG-0560. Washington, D.C.: June 24, 2002.

           GAO. Department of Energy: Further Actions Are Needed to
           Strengthen Contract Management for Major Projects. [49]GAO-05-123
           . Washington, D.C.: March 18, 2005.

           GAO. Department of Energy: Status of Contract and Project
           Management Reforms. [50]GAO-03-570T . Washington, D.C.: March 20,
           2003.

           GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to
           Ensure Initiatives Have Improved Results. [51]GAO-02-798 .
           Washington, D.C.: September 13, 2002.

           GAO. Nuclear Weapons: Design Reviews of DOE's Tritium Extraction
           Facility. [52]GAO/RCED-98-75 . Washington, D.C.: March 31, 1998.

           National Nuclear Security Administration. Program Review of the
           Estimate to Complete Tritium Extraction Facility (TEF) at Savannah
           River Site. Washington, D.C.: August 29, 2002.
			  
			  Highly Enriched Uranium Materials Facility

           BWXT Y-12. Highly Enriched Uranium Materials Facility Project
           Causal Analysis Report. Oak Ridge, TN: March 6, 2006.

           Department of Energy. Limited External Independent Review for
           Baseline Change Proposal Review. Oak Ridge, TN: August 31, 2004.

           Department of Energy, Office of Inspector General. Audit Report,
           Design of the Uranium Storage Facility at the Y-12 National
           Security Complex. DOE/IG-0643. Washington, D.C.: March 19, 2004.

           Department of Energy, Office of Inspector General. Audit Report,
           Reestablishment of Enriched Uranium Operations at the Y-12
           National Security Complex. DOE/IG-0640. Washington, D.C.: February
           24, 2004.

           Department of Energy. External Independent Review - Performance
           Baseline Review of the Highly Enriched Uranium Materials Facility
           Project. Oak Ridge, TN: June 2003.
			  
			  Depleted Uranium Hexafluoride 6 Conversion Facility

           Department of Energy. Report on the Independent Project Review of
           the Depleted Uranium Hexafluoride Conversion Project. Washington,
           D.C.: October 8, 2004.

           Department of Energy, Office of Inspector General. Audit Report:
           Depleted Uranium Hexafluoride Conversion. DOE/IG-0642. Washington,
           D.C.: March 18, 2004.

           GAO. Department of Energy: Status of Contract and Project
           Management Reforms. [53]GAO-03-570T . Washington, D.C.: March 20,
           2003.

           LMI Government Consulting. DUF6 Conversion Project CD-3 Corrective
           Action Plan Review. DE538T1. McLean, VA: October 2005.

           LMI Government Consulting. Construction Readiness EIR (for CD-3)
           of the Depleted Uranium Hexafluoride Conversion Project. DE534T1.
           McLean, VA: June 2005.

           LMI Government Consulting. DUF6 Conversion Project CD-3C
           Construction Readiness Review Preliminary Draft. Washington, D.C.:
           May 20, 2005.

           LMI Government Consulting. DUF6 Limited Conversion Plan Project
           External Independent Review for the Office of Engineering and
           Construction Management. DE428T1. McLean, VA: June 2004.
			  
			  Chemistry and Metallurgy Research Facility Replacement

           Jupiter Corporation. External Independent Review of the Chemistry
           and Metallurgy Research Building Replacement Project. Approve
           Performance Baseline and Approve Start of Construction.
           CD-2A/CD-3A. Wheaton, MD: October 14, 2005.
			  
			  Appendix IV: Survey Results for Primary Factors Affecting Cost
			  and Schedule on Nine Projects with Cost or Schedule Changes

           Source: GAO.
			  
                                   Survey results for primary factors
                                                                  To a        
                                        To a      To a    To a    very        
                              To no  limited  moderate   great   great     No 
Factor/Project            extent   extent    extent  extent  extent answer 
Absence of open communication, mutual trust, and close coordination
Depleted Uranium                                                           
Hexafluoride 6                                                             
Conversion Facility   X                                                    
Highly Enriched                                                            
Uranium Materials                                                          
Facility              X                                                    
Mixed Oxide Fuel                                                           
Fabrication                                                                
Facility                       X                                           
National Ignition                                                          
Facility              X                                                    
Pit Disassembly and                                                        
Conversion Facility            X                                           
Salt Waste                                                                 
Processing Facility   X                                                    
Spallation Neutron                                                         
Source                X                                                    
Tritium Extraction                                                         
Facility              X                                                    
Waste Treatment and                                                        
Immobilization                                                             
Plant                 X                                                    
Total                 0        4        2         1       0       2        
Changes in "political will" during project execution (e.g., project
changes resulting from political decisions--includes politics internal and
external to the project )
Depleted Uranium                                                           
Hexafluoride 6                                                             
Conversion Facility            X                                           
Highly Enriched                                                            
Uranium Materials                                                          
Facility                       X                                           
Mixed Oxide Fuel                                                           
Fabrication                                                                
Facility                                X                                  
National Ignition                                                          
Facility                       X                                           
Pit Disassembly and                                                        
Conversion Facility                     X                                  
Salt Waste                                                                 
Processing Facility            X                                           
Spallation Neutron                                                         
Source                                  X                                  
Tritium Extraction                                                         
Facility                       X                                           
Waste Treatment and                                                        
Immobilization                                                             
Plant                          X                                           
Total                          2        2         2       0       2      1 
Interruptions in planning and committing budget funds
Depleted Uranium                                                           
Hexafluoride 6                                                             
Conversion                     X                                           
Highly Enriched                                                            
Uranium Materials                                                          
Facility                                X                                  
Mixed Oxide Fuel                                                           
Fabrication                                                                
Facility                       X                                           
National Ignition                                                          
Facility                       X                                           
Pit Disassembly and                                                        
Conversion Facility                     X                                  
Salt Waste                                                                 
Processing Facility            X                                           
Spallation Neutron                                                         
Source                                  X                                  

                                      Survey results for primary factors
                                                                  To a        
                                           To a     To a   To a   very        
                                  To no limited moderate  great  great     No 
Factor/Project                extent  extent   extent extent extent answer 
Tritium Extraction Facility                         X                      
Waste Treatment and                                                        
Immobilization Plant                                       X               
Total                              3       0        1      2      2      1 
Project managers did not                                                   
have adequate professional                                                 
experience                                                                 
Depleted Uranium                                                           
Hexafluoride 6 Conversion                           X                      
Highly Enriched Uranium                                                    
Materials Facility                                  X                      
Mixed Oxide Fuel Fabrication                                               
Facility                                                                 X 
National Ignition Facility                          X                      
Salt Waste Processing                                                      
Facility                           X                                       
Spallation Neutron Source                           X                      
Tritium Extraction Facility                X                               
Pit Disassembly and                                                        
Conversion Facility                                                      X 
Waste Treatment and                                                        
Immobilization Plant               X                                       
Total                              2       1        4      0      0      2 

           Source: GAO analysis of DOD data.
			  
			  Appendix V: Definitions of Technology Readiness Levels

Technology                                                                                                     
readiness level                          Basic objective of                                   Tests and        
(TRL)             Level involved         TRLs               Components          Integration   environment      
1. Basic       Studies.               Research to prove  None.               None.         Desktop, "back   
principles                            feasibility.                                         of envelope"     
observed and                                                                               environment.     
reported.                                                                                                   
2. Technology  Studies.               Research to prove  None.               Paper studies Academic         
concept and/or                        feasibility.                           indicate      environment. The 
application                                                                  components    emphasis here is 
formulated.                                                                  ought to work still on         
                                                                                together.     understanding    
                                                                                              the science but  
                                                                                              beginning to     
                                                                                              think about      
                                                                                              possible         
                                                                                              applications of  
                                                                                              the scientific   
                                                                                              principles.      
3. Analytical  Pieces of components.  Research to prove  No system           No attempt at Uses of the      
and                                   feasibility.       components, just    integration;  observed         
experimental                                             basic laboratory    still trying  properties are   
critical                                                 research equipment  to see        postulated and   
function                                                 to verify physical  whether       experimentation  
and/or                                                   principles.         individual    with potential   
characteristic                                                               parts of the  elements of      
proof of                                                                     technology    subsystem        
concept.                                                                     work. Lab     begins. Lab work 
                                                                                experiments   to validate      
                                                                                with          pieces of        
                                                                                available     technology       
                                                                                components    without trying   
                                                                                show they     to integrate.    
                                                                                will work.    Emphasis is on   
                                                                                              validating the   
                                                                                              predictions made 
                                                                                              during earlier   
                                                                                              analytical       
                                                                                              studies to       
                                                                                              ensure that the  
                                                                                              technology has a 
                                                                                              firm scientific  
                                                                                              underpinning.    
4. Component   Low fidelity           Demonstrate        Ad Hoc and          Available     Tests in         
and/or         breadboard.            technical          available           components    controlled       
breadboard                            feasibility and    laboratory          assembled     laboratory       
validation in                         functionality.     components are      into          environment. Lab 
lab                                                      surrogates for      subsystem     work at less     
environment.                                             system components   breadboard.   than full        
                                                            that may require    Interfaces    subsystem        
                                                            special handling,   between       integration,     
                                                            calibration, or     components    although         
                                                            alignment to get    are           starting to see  
                                                            them to function.   realistic.    if components    
                                                            Not fully                         will work        
                                                            functional but                    together.        
                                                            representative of                                  
                                                            technically                                        
                                                            feasible approach.                                 
5. Component   High fidelity          Demonstrate        Fidelity of         Fidelity of   Laboratory       
and/or         breadboard/brass-board technical          components and      subsystem     environment      
breadboard     (e.g., nonscale or     feasibility and    interfaces are      mock up       modified to      
validation in  form components).      functionality.     improved from TRL   improves      approximate      
relevant                                                 4. Some special     (e.g., from   operational      
environment.                                             purpose components  breadboard to environment.     
                                                            combined with       brassboard).  Increases in     
                                                            available           Integration   accuracy of the  
                                                            laboratory          issues become controlled       
                                                            components.         defined.      environment in   
                                                            Functionally                      which it is      
                                                            equivalent but not                tested.          
                                                            of same material or                                
                                                            size. May include                                  
                                                            integration of                                     
                                                            several components                                 
                                                            with reasonably                                    
                                                            realistic support                                  
                                                            elements to                                        
                                                            demonstrate                                        
                                                            functionality.                                     
6. System/     Subsystem closely      Demonstrate        Subsystem is high   Components    Relevant         
Subsystem      configured for         applicability to   fidelity functional are           environment      
model or       intended project       intended project   prototype with      functionally  inside or        
prototype      application.           and subsystem      (very near same     compatible    outside the      
demonstration  Demonstrated in        integration.       material and size   (and very     laboratory, but  
in relevant    relevant environment.                     of operational      near same     not the eventual 
environment.   (Shows will work in    (Specific to       system). Probably   material and  operating        
                  desired                intended           includes the        size of       environment. The 
                  configuration).        application in     integration of many operational   testing          
                                         project.)          new components and  system).      environment does 
                                                            realistic           Component     not reach the    
                                                            supporting          integration   level of an      
                                                            elements/subsystems into system   operational      
                                                            if needed to        is            environment,     
                                                            demonstrate full    demonstrated. although moving  
                                                            functionality.                    out of           
                                                            Partially                         controlled       
                                                            integrated with                   laboratory       
                                                            existing systems.                 environment into 
                                                                                              something more   
                                                                                              closely          
                                                                                              approximating    
                                                                                              the realities of 
                                                                                              technology's     
                                                                                              intended use.    
7. Subsystem   Subsystem configured   Demonstrate        Prototype improves  Prototype not Operational      
prototype      for intended project   applicability to   to preproduction    integrated    environment, but 
demonstration  application.           intended project   quality. Components into intended not the eventual 
in an          Demonstrated in        and subsystem      are representative  system but    environment.     
operational    operational            integration.       of project          onto          Operational      
environment.   environment.                              components          surrogate     testing of       
                                         (Specific to       (material, size,    system.       system in        
                                         intended           and function) and                 representational 
                                         application in     integrated with                   environment.     
                                         project.)          other key                         Prototype will   
                                                            supporting                        be exposed to    
                                                            elements/subsystems               the true         
                                                            to demonstrate full               operational      
                                                            functionality.                    environment on a 
                                                            Accurate enough                   surrogate        
                                                            representation to                 platform,        
                                                            expect only minor                 demonstrator, or 
                                                            design changes.                   test bed.        
8. Total       Full integration of    Applied/Integrated Components are      Subsystem     Demonstration,   
system         subsystems to show     into intended      right material,     performance   test, and        
completed,     total system will meet project            size, and function  meets         evaluation       
tested, and    requirements.          application.       compatible with     intended      completed.       
fully                                                    operational system. application   Demonstrates     
demonstrated.                                                                and is fully  system meets     
                                                                                integrated    procurement      
                                                                                into total    specifications.  
                                                                                system.       Demonstrated in  
                                                                                              eventual         
                                                                                              environment.     
9. Total       System meeting         Applied/Integrated Components are      Subsystem has Operational      
system used    intended operational   into intended      successfully        been          testing and      
successfully   requirements.          project            performing in the   installed and evaluation       
in project                            application.       actual              successfully  completed.       
operations.                                              environment--proper deployed in   Demonstrates     
                                                            size, material, and project       that system is   
                                                            function.           systems.      capable of       
                                                                                              meeting all      
                                                                                              mission          
                                                                                              requirements.    
			  
			  Appendix VI: Comparison of DODï¿½s Product Development Process
			  with DOEï¿½s Project Management Process
			  
			  Appendix VII: Comments from the Department of Energy
			  
			  Appendix VIII: GAO Contact and Staff Acknowledgments
			  
			  GAO Contact

           Gene Aloise, (202) 512-3841
			  
			  Staff Acknowledgments 

           In addition to the individual named above, Michaela Brown, Rudy
           Chatlos, James Espinoza, Daniel Feehan (Assistant Director),
           Joseph Keener, Thomas Kingham, Matthew Lea, Mehrzad Nadji, Omari
           Norman, Christopher Pacheco, Thomas Perry, and Carol Herrnstadt
           Shulman made key contributions to this report.
			  
			  GAOï¿½s Mission

           The Government Accountability Office, the audit, evaluation and
           investigative arm of Congress, exists to support Congress in
           meeting its constitutional responsibilities and to help improve
           the performance and accountability of the federal government for
           the American people. GAO examines the use of public funds;
           evaluates federal programs and policies; and provides analyses,
           recommendations, and other assistance to help Congress make
           informed oversight, policy, and funding decisions. GAO's
           commitment to good government is reflected in its core values of
           accountability, integrity, and reliability.
			  
			  Obtaining Copies of GAO Reports and Testimony

           The fastest and easiest way to obtain copies of GAO documents at
           no cost is through GAO's Web site ( www.gao.gov ). Each
           weekday, GAO posts newly released reports, testimony, and
           correspondence on its Web site. To have GAO e-mail you a list of
           newly posted products every afternoon, go to www.gao.gov and
           select "Subscribe to Updates."
			  
			  Order by Mail or Phone

           The first copy of each printed report is free. Additional copies
           are $2 each. A check or money order should be made out to the
           Superintendent of Documents. GAO also accepts VISA and Mastercard.
           Orders for 100 or more copies mailed to a single address are
           discounted 25 percent. Orders should be sent to:

           U.S. Government Accountability Office 441 G Street NW, Room LM
           Washington, D.C. 20548

           To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
           (202) 512-6061
			  
			  To Report Fraud, Waste, and Abuse in Federal Programs

           Contact:

           Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
           [email protected] Automated answering system: (800) 424-5454 or
           (202) 512-7470
			  
			  Congressional Relations

           Gloria Jarmon, Managing Director, [email protected] (202)
           512-4400 U.S. Government Accountability Office, 441 G Street NW,
           Room 7125 Washington, D.C. 20548
			  
			  Public Affairs

           Paul Anderson, Managing Director, [email protected] (202)
           512-4800 U.S. Government Accountability Office, 441 G Street NW,
           Room 7149 Washington, D.C. 20548

(360652)

www.gao.gov/cgi-bin/getrpt?GAO-07-336 .

To view the full product, including the scope
and methodology, click on the link above.

For more information, contact Gene Aloise at (202) 512-3841 or
[email protected].

Highlights of [67]GAO-07-336 , a report to the Subcommittee on Energy and
Water Development, and Related Agencies, Committee on Appropriations,
House of Representatives

March 2007

DEPARTMENT OF ENERGY

Major Construction Projects Need a Consistent Approach for Assessing
Technology Readiness to Help Avoid Cost Increases and Delays

The Department of Energy (DOE) spends billions of dollars on major
construction projects that help maintain the nuclear weapons stockpile,
conduct research and development, and process nuclear waste so that it can
be disposed of. Because of DOE's long-standing project management
problems, GAO determined the extent to which (1) DOE's major construction
projects are having cost increases and schedule delays and the major
factors contributing to these problems and (2) DOE ensures that project
designs are sufficiently complete before construction begins to help avoid
cost increases and delays. We examined 12 DOE major projects with total
costs of about $27 billion, spoke with federal and contractor officials,
and reviewed project management documents.

[68]What GAO Recommends

GAO recommends that DOE develop a consistent approach for measuring the
readiness of critical project technologies. DOE supports GAO's
recommendations but suggested revisions to allow it to first conduct a
pilot application on selected projects to better understand the process
and evaluate its potential use.

Of the 12 DOE major projects GAO reviewed, 9 exceeded their original cost
or schedule estimates, principally because of ineffective DOE project
oversight and poor contractor management. Specifically, 8 of the 12
projects experienced cost increases ranging from $79.0 million to $7.9
billion, and 9 of the 12 projects were behind schedule by 9 months to more
than 11 years. Project oversight problems included, among other things,
inadequate systems for measuring contractor performance, approval of
construction activities before final designs were sufficiently complete,
ineffective project reviews, and insufficient DOE staffing. Furthermore,
contractors poorly managed the development and integration of the
technology used in the projects by, among other things, not accurately
anticipating the cost and time that would be required to carry out the
highly complex tasks involved.

Even though DOE requires final project designs to be sufficiently complete
before beginning construction, it has not systematically ensured that the
critical technologies reflected in these designs have been demonstrated to
work as intended (technology readiness) before committing to construction
expenses. Specifically, only one of the five DOE project directors with
projects that have recently begun or are nearing construction had
systematically assessed technology readiness. The other four directors
also told us that they have or will have completed prior to construction,
85 to 100 percent of their projects' final design, but they had not
systematically assessed technology readiness. Proceeding into construction
without also demonstrating a technology's readiness can lead to cost
increases and delays. For example, one technology to be used in DOE's
Waste Treatment and Immobilization Plant was not sufficiently
demonstrated--that is, shown to be technologically ready for its intended
application--before construction began. Consequently, the technology did
not perform as expected, which resulted in about $225 million in redesign
costs and schedule delays of more than 1 year. To help avoid these
problems, the National Aeronautics and Space Administration (NASA)
pioneered and the Department of Defense (DOD) has adopted for its projects
a method for measuring and communicating technology readiness levels
(TRL). Using a scale from one (basic principles observed) through nine
(total system used successfully in project operations), TRLs show the
extent to which technologies have been demonstrated to work as intended in
the project. DOE project directors agreed that such an approach would help
make technology assessments more transparent and improve stakeholder
communication prior to making critical project decisions, such as
authorizing construction.

Technology Readiness Levels

References

Visible links
  28. http://www.gao.gov/cgi-bin/getrpt?GAO-06-602T
  29. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-99-162
  30. http://www.gao.gov/cgi-bin/getrpt?GAO-06-391
  31. http://www.gao.gov/cgi-bin/getrpt?GAO-04-759
  32. http://www.gao.gov/cgi-bin/getrpt?GAO-05-123
  33. http://www.gao.gov/cgi-bin/getrpt?GAO/RCED-97-146R
  34. http://www.gao.gov/
  35. http://www.gao.gov/cgi-bin/getrpt?GAO-03-570T
  36. http://www.gao.gov/cgi-bin/getrpt?GAO-02-798
  37. http://www.gao.gov/cgi-bin/getrpt?GAO-01-677R
  38. http://www.gao.gov/cgi-bin/getrpt?GAO/RCED-00-141
  39. http://www.gao.gov/cgi-bin/getrpt?GAO/RCED-00-271
  40. http://www.gao.gov/cgi-bin/getrpt?GAO-06-602T
  41. http://www.gao.gov/cgi-bin/getrpt?GAO-05-123
  42. http://www.gao.gov/cgi-bin/getrpt?GAO-04-611
  43. http://www.gao.gov/cgi-bin/getrpt?GAO-03-57T
  44. http://www.gao.gov/cgi-bin/getrpt?GAO-02-798
  45. http://www.gao.gov/cgi-bin/getrpt?GAO/RCED-93-99
  46. http://www.gao.gov/cgi-bin/getrpt?GAO-03-570T
  47. http://www.gao.gov/cgi-bin/getrpt?GAO-02-798
  48. http://www.gao.gov/cgi-bin/getrpt?GAO/T-RCED-99-103
  49. http://www.gao.gov/cgi-bin/getrpt?GAO-05-123
  50. http://www.gao.gov/cgi-bin/getrpt?GAO-03-570T
  51. http://www.gao.gov/cgi-bin/getrpt?GAO-02-798
  52. http://www.gao.gov/cgi-bin/getrpt?GAO/RCED-98-75
  53. http://www.gao.gov/cgi-bin/getrpt?GAO-03-570T
  60. http://www.gao.gov/cgi-bin/getrpt?GAO-06-602T
  61. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-99-162
  62. http://www.gao.gov/cgi-bin/getrpt?GAO-02-39
  63. http://www.gao.gov/cgi-bin/getrpt?GAO/T-RCED-00-248
  64. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-99-162
  65. http://www.gao.gov/cgi-bin/getrpt?GAO-05-123
  67. http://www.gao.gov/cgi-bin/getrpt?GAO-07-336
*** End of document. ***