Nuclear Weapons: DOE Needs to Improve Oversight of the $5 Billion
Strategic Computing Initiative (Chapter Report, 06/28/1999,
GAO/RCED-99-195).
Historically, the United States has detonated nuclear weapons to
determine their safety and reliability. Since 1992, however, there has
been a moratorium on testing. As a substitute for actual testing, the
Department of Energy (DOE) developed the Stockpile Stewardship and
Management Program in 1995. The program uses various methods, including
computer modeling, to ensure weapon safety and reliability. The
strategic computing initiative seeks to develop advanced computer models
that will simulate nuclear explosions in three dimensions with higher
resolution than previous models and with a more complete treatment of
the underlying basic physics. The initiative is also developing the
world's largest and fastest computers. GAO found that weak management
and information processes have hampered oversight of this $5-billion
initiative. For example, the initiative's strategic plan is out of date,
annual plans have been prepared only sporadically, and milestones are
not well defined. Moreover, the lack of information has made it
difficult to determine which of the hundreds of milestones have been
met, which are behind schedule, or even which are still relevant.
Program cost estimates have risen substantially. Finally, significant
technical challenges will have to be overcome before a computer
simulation can determine, in the absence of nuclear testing, whether a
weapon system will work as expected.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: RCED-99-195
TITLE: Nuclear Weapons: DOE Needs to Improve Oversight of the $5
Billion Strategic Computing Initiative
DATE: 06/28/1999
SUBJECT: Information resources management
Computer software
Computer modeling
Future budget projections
Management information systems
Nuclear weapons testing
ADP procurement
Cost analysis
Weapons research and development
Strategic information systems planning
IDENTIFIER: DOE Accelerated Strategic Computing Initiative
DOE Stockpile Stewardship and Management Program
Comprehensive Test Ban Treaty of 1996
DOE Academic Strategic Alliances Program
DOE Financial Information Variance Reporting System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. This text was extracted from a PDF file. **
** Delineations within the text indicating chapter titles, **
** headings, and bullets have not been preserved, and in some **
** cases heading text has been incorrectly merged into **
** body text in the adjacent column. Graphic images have **
** not been reproduced, but figure captions are included. **
** Tables are included, but column deliniations have not been **
** preserved. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
United States General Accounting Office GAO Report
to the Chairman, Subcommittee on Military Procurement, House
Committee on Armed Services June 1999 NUCLEAR WEAPONS DOE
Needs to Improve Oversight of the $5 Billion Strategic Computing
Initiative GAO/RCED-99-195 GAO United States General Accounting
Office Washington, D.C. 20548 Resources, Community, and Economic
Development Division B-282727 June 28, 1999 The Honorable Duncan
Hunter Chairman, Subcommittee on Military Procurement Committee on
Armed Services House of Representatives Dear Mr. Chairman: As
requested, this report examines the Department of Energy's (DOE)
Accelerated Strategic Computing Initiative. Specifically, it
discusses the management of the program, including (1) whether the
program is meeting its key milestones and whether hardware and
software developments are adequate to date, (2) whether the
program is within its projected budget, and (3) what key technical
risks the program faces. As arranged with your office, we plan to
distribute copies of this report to the appropriate congressional
committees; the Honorable Bill Richardson, Secretary of Energy;
the Honorable Jacob Lew, Director, Office of Management and
Budget; and other interested parties. We will also make copies
available to others on request. If you have any questions
regarding this report, please contact me or Edward Zadjura at
(202) 512-3841. Key contributors to this assignment were Linda
Chu, Daniel Feehan, and Anne McCaffrey. Sincerely yours, Susan D.
Kladiva, Associate Director, Energy, Resources, and Science Issues
Executive Summary Purpose Historically, the United States
detonated nuclear weapons as the primary method of validating
designs and certifying the weapons as safe and reliable. Since
September 1992, there has been a moratorium on testing. To ensure
the continued safety and reliability of nuclear weapons, the
Department of Energy (DOE), which is responsible for designing and
building nuclear weapons, developed the 15-year Stockpile
Stewardship and Management Program in 1995 as a substitute for
actual testing. The stockpile stewardship program employs a
variety of means to ensure weapon safety and reliability,
including examining weapons, conducting laboratory experiments and
tests, and conducting computer modeling and simulation. The
computer modeling and simulation part of the program is known as
the Accelerated Strategic Computing Initiative. The strategic
computing initiative aims to develop advanced computer models that
will simulate nuclear explosions in three dimensions with higher
resolution than previous models and with a more complete treatment
of the underlying basic physics. The initiative is also developing
the world's largest and fastest computers, which may ultimately be
able to calculate more than 100-trillion mathematical operations
per second. The initiative is expected to cost about $5.2 billion
for fiscal years 1996 through 2004. Concerned about the status of
the strategic computing initiative, the Chairman, Subcommittee on
Military Procurement, House Committee on Armed Services, requested
that GAO review the management of the strategic computing
initiative, including (1) whether the program is meeting its key
milestones and whether its hardware and software developments are
adequate to date; (2) whether the program is within its projected
budget; and (3) what key technical risks the program faces.
Background Since the dawn of the nuclear era in 1945, the
testing of nuclear weapons and state-of-the-art computing have
been used together to ensure the performance, reliability, and
safety of the weapons. Testing was the ultimate judge of whether a
weapon worked and met its design requirements and provided data
needed for computer models. Computers were used to perform the
massive calculations needed to understand the basic physical
processes that take place at the heart of a nuclear explosion and
to interpret the results of nuclear experiments and tests, thus
providing feedback in the process of designing, building, and
testing nuclear weapons. The practical result of the Comprehensive
Test Ban Treaty of 1996 is that existing nuclear weapons will be
kept longer than planned because new weapon designs cannot be
tested and certified as safe and reliable. Faced with these
testing restrictions, DOE developed a Page 2
GAO/RCED-99-195 Strategic Computing Initiative Executive Summary
new approach to certifying the safety and reliability of weapons
in the U.S. stockpile. The computer models and hardware, developed
as part of the strategic computing initiative, will be used to
identify potential stockpile problems by predicting the effects of
aging and the need to replace components or even to retire weapons
systems if they become unsafe or unreliable. The existing
stockpile of weapons is aging, and many of the designers of those
weapons have retired or are approaching retirement. For these
reasons, DOE wants to have the computers and models available by
2004 so that the existing cadre of experienced weapons designers
will be available to help verify the results of the models.
Results in Brief Weak management and information processes
hamper oversight of the strategic computing initiative. Although
initiative managers report that many milestones have been met, the
lack of comprehensive planning and progress tracking systems make
assessment of the initiative's progress difficult and subjective.
Currently, the initiative's strategic plan is out of date, annual
plans have been prepared only sporadically, and milestones are not
well defined. Furthermore, little information exists to track the
initiative's progress or to compare its accomplishments with its
milestones. Consequently, it is difficult to determine which of
the hundreds of milestones have been met, which are behind
schedule, or even which are still relevant, given changes in the
initiative. Program cost estimates have increased substantially.
In 1995, DOE estimated that costs for the first 5 years of the
initiative (fiscal year 1996 through fiscal year 2001) would be
$1.7 billion. By 1999, estimated costs for that same 5-year period
increased to $2.9 billion. DOE currently estimates that the
program will cost about $5.2 billion for fiscal years 1996 through
2004.1 Some of the cost increases result from the shift to
computer-based simulations, while some reflect weaknesses in DOE's
cost estimation. Developing a computer simulation, or "virtual
test" capability, that, in the absence of nuclear testing, can be
used to determine whether a weapon system will perform as intended
requires overcoming significant technical challenges. These
challenges range from developing state-of-the-art hardware and
software technologies, to integrating scientific data from weapons
physics experiments, to recruiting and retaining staff with the
needed technical expertise. 1The strategic computing initiative is
a 15-year program, but because of the 5-year budget cycle, no cost
estimates are available beyond fiscal year 2004. Page 3
GAO/RCED-99-195 Strategic Computing Initiative Executive Summary
This report makes recommendations directed at improving the
oversight and management of the strategic computing program.
Principal Findings A Comprehensive Planning The strategic
computing initiative's planning efforts have been and Tracking
System Is inconsistent and incomplete. Strategic planning
documents have not been Needed to Assess Program updated, and
annual implementation plans were prepared inconsistently Progress
or, in some cases, not at all. The long-term milestones presented
in various plans are inconsistent, and no information exists to
link annual activities to these milestones. Strategic plans also
do not identify the multiple research strategies currently
employed to meet many long-term milestones, nor do the plans
include key decision points for managing these strategies.
Performance criteria for most milestones have also not been
defined. The efforts of DOE and laboratory managers to track the
progress of the strategic computing initiative have been limited
primarily to reporting annual accomplishments, without any
systematic tracking of progress towards long-term milestones. As a
result, it is not possible to determine whether annual milestones
were achieved or to what extent annual efforts contribute to long-
term milestones. In response to GAO's request for tracking
information, program officials have decided to track and report
the program's progress more systematically. The lack of a system
for tracking progress, combined with the lack of defined
performance criteria, make it difficult to assess whether the
strategic computing initiative is proceeding on schedule and
delivering the performance expected. However, it is possible to
gain some limited insights through discussions with laboratory
officials on individual projects or areas. For example, in the
area of hardware development, most contract milestones to date
relating to the delivery and installation of computers and related
hardware have been met, although not all acceptance tests have
been passed. DOE is not managing the strategic computing
initiative as a strategic system. To be designated as a strategic
system, under DOE criteria, a project must cost over $400 million,
be an urgent national priority, be high-risk, have international
implications, or be vital to national security. The purpose of
designating strategic systems is to ensure informed, objective,
and well-documented decisions for key events, such as changes Page
4 GAO/RCED-99-195 Strategic
Computing Initiative Executive Summary to baseline cost or
schedule and to ensure oversight at the highest departmental
level. The strategic computing initiative meets all these
criteria, has experienced delays in some areas, has had its
projected costs increase, and depends, in some cases, on as-yet
unknown technologies for success. These characteristics, coupled
with demonstrated weaknesses in program management and oversight,
make the strategic computing initiative a clear candidate for
being designated as a strategic system. According to DOE, it has
not designated the initiative as a strategic system because the
program is already subject to high-level departmental oversight.
However, as discussed above, GAO found serious weaknesses in the
program's management and information processes that make it
difficult to determine if the program is performing as expected.
The Management and DOE's cost estimates for the
strategic computing initiative have increased Tracking of Costs
Need to substantially since 1995, when early budget projections
were made. Costs Be Improved for fiscal years
1996 through 2001 have increased from an original estimate of $1.7
billion to the current $2.9 billion. DOE's fiscal year 2000 budget
request for the strategic computing initiative, which totals $692
million, is more than double the original fiscal year 2000
estimate made in 1995. Some of the cost increases result from the
shift from test-based experiments to computer-based simulations,
while some increases are the result of weaknesses in DOE's cost
estimation. Although DOE monitors month-by-month spending at the
laboratories, it does not track costs for specific projects. As a
result, DOE cannot determine which projects, if any, may be
costing more or less than originally planned. GAO has previously
noted DOE's difficulty in managing costs and schedules in large
projects. Technical Challenges Are The development of hardware
and software technologies and the Present in All Aspects of
necessary infrastructure to support these technologies are
critical to the Strategic Computing achieving the simulation
and modeling goals of the strategic computing Initiative
initiative. The program faces significant technical challenges in
all of these areas. For example, increasingly large and complex
computers using thousands of processors must be developed and made
to operate as a single integrated system at speeds far beyond any
achieved to date. The effort to develop software for simulation
models on the scale needed to model nuclear weapons requires
incorporating massive amounts of data, utilizing increasingly
sophisticated problem-solving techniques, and using increasingly
larger and faster computers. The President's Information Page 5
GAO/RCED-99-195 Strategic Computing Initiative Executive Summary
Technology Advisory Committee recently described software of this
scale as being "among the most complex of human-engineered
structures." Furthermore, developing the needed infrastructure,
including data storage and visualization technology, will require
significant technological improvements. The successful integration
of data from laboratory experiments conducted outside the
strategic computing initiative into software models being
developed as part of the initiative has been noted by the DOE-
chartered Blue Ribbon Panel as another important technical
challenge. Data from these experiments and past nuclear tests are
critical for demonstrating that the results of the software
simulations are accurate. According to program officials, a recent
reorganization of DOE offices and the creation of a formal
software validation program aim to address this challenge.
Finally, recruiting and retaining qualified personnel is a
continuing area of risk, according to strategic computing
initiative officials and outside program reviews such as the
Chiles Commission.2 DOE and laboratory officials have efforts
ongoing in many areas to improve the recruitment of staff with the
required expertise. Nonetheless, as noted by the Chiles
Commission, there is no certainty that DOE's efforts will succeed.
Recommendations DOE has chosen not to designate the strategic
computing program as a strategic system. Given the strategic
computing program's estimated cost of over $5 billion; the lack of
a comprehensive planning, tracking, and reporting system; and the
importance of the program to maintaining the stockpile of nuclear
weapons; it is important that DOE improve its oversight and
management of this program. Therefore, we recommend that the
Secretary of Energy require the establishment of a comprehensive
planning, progress tracking, and reporting system for the program
and designate the program as a strategic system warranting
oversight at the highest departmental level. Given the substantial
increases in the cost estimates for the strategic computing
initiative to date, DOE's weaknesses in estimating costs for the
unprecedented scale of development efforts, and the lack of a
cost-tracking process for the projects under the initiative, we
also recommend that the Secretary of Energy require the strategic
computing initiative to adopt systematic cost tracking procedures
that will allow DOE managers to determine if specific projects are
within budget. 2Report of the Commission on Maintaining United
States Nuclear Weapons Expertise, Mar. 1, 1999. Page 6
GAO/RCED-99-195 Strategic Computing Initiative Executive Summary
Agency Comments GAO provided DOE with a draft of this report
for its review and comment. and Our Evaluation DOE concurred
with part but not all of the report's recommendations.
Specifically, DOE concurred with the recommendation on the need to
improve its oversight and management of this program and cited
changes that it has made or is in the process of making. DOE did
not concur with the recommendation to designate the program as a
strategic system or on the need to adopt systematic cost tracking
procedures. In agreeing with the recommendation to improve the
oversight and management of the program, DOE cited several changes
it was making. Specifically, DOE stated that it would soon issue
an updated Program Plan that will include detailed specifications
for all of the critical program milestones. In addition, FY 2000
Implementation Plans will be issued by September 30, 1999, that
will include descriptions of all program elements and complete
lists of all milestones. The Department also cited the creation of
a quarterly progress tracking mechanism to track program
milestones. However, in addition to tracking the program's
progress against established calendar milestones, it is also
necessary to establish specific technical criteria for what
constitutes the successful completion of those milestones. Until
DOE completes and publishes its revised Program Plan and FY 2000
Implementation Plans, GAO cannot determine whether the Department
has fully complied with this recommendation. DOE disagreed with
the recommendation to designate this program as a strategic
system, stating that to do so would duplicate the existing
planning, progress tracking, and reporting system. GAO agrees that
creating a duplicate tracking system that mirrors the requirements
set out by DOE for strategic systems would not be worthwhile.
However, as discussed in detail in this report, DOE has not shown
that it has an adequate planning, progress tracking, and reporting
system in place for the strategic computing initiative. While DOE
is making some positive improvements in these areas, the changes
are not yet fully in place, and their adequacy cannot be judged at
this time. Furthermore, if the changes that DOE is making are
adequate to meet the requirements for tracking and monitoring of a
strategic system, then GAO cannot understand DOE's reluctance to
designate this large and costly program as a strategic system. DOE
stated that it has a review process that meets the intent of the
Clinger-Cohen Act of 1996. However, GAO reported in July 1998 that
the Department's process effectively excluded scientific computers
like those being acquired through this program from DOE's normal
review channels and places them Page 7
GAO/RCED-99-195 Strategic Computing Initiative Executive Summary
within the program offices3. GAO stated that all computers should
be included as part of the normal DOE Clinger-Cohen review
process. DOE also did not agree with the recommendation to adopt
systematic cost-tracking procedures for the strategic computing
initiative, noting that costs are tracked by budget and reporting
codes in the Department's Financial Information System. DOE stated
that these systems are extended down to individual projects with
other funding and cost-monitoring tools that gather more detailed
information. As an example, DOE cited a March 1999 analysis of
selected projects that identified the commitments and cost status
for specific procurements at the project level. GAO does not agree
that DOE has an adequate level of tracking at the project level or
that the changes it is making will rectify this problem. DOE's
current system tracks costs only at the aggregate level and does
not allow DOE managers to determine which projects at the
laboratories are under or over budget. Furthermore, the "other
funding and cost monitoring tools" that DOE uses do not allow the
systematic tracking of project costs. DOE also stated that some
budgeting flexibility is necessary to capitalize on changes within
the high-computing industry. While some budgeting flexibility is
necessary in a project of this size and complexity, GAO does not
believe that this flexibility should preclude effective oversight
of a multiyear program costing over $5 billion. DOE's written
comments are included in appendix II, and GAO's responses are
discussed in chapters 2 and 3 and in appendix II. 3Information
Technology: Department of Energy Does Not Effectively Manage Its
Supercomputers (GAO/RCED-98-208, Jul. 17, 1998). Page 8
GAO/RCED-99-195 Strategic Computing Initiative Page 9
GAO/RCED-99-195 Strategic Computing Initiative Contents Executive
Summary
2 Chapter 1
12 Introduction The Evolving Role of Computing in
Nuclear Weapons 12 Why Such Large
Computers Are Needed 14
ASCI Program Overview
15 Objectives, Scope, and Methodology
17 Chapter 2
19 A Comprehensive Comprehensive Planning Is Needed
19 Systematic Progress Tracking and Reporting Needed
25 Planning and Tracking Program Progress Difficult to Assess
26 System Is Needed to DOE May Need to Manage ASCI as a
Strategic System 28 Assess Program
Conclusions
29 Recommendations
29 Progress Agency Comments and Our Evaluation
30 Chapter 3
31 Management and Cost Estimates Increased
Substantially 31 Better
Oversight of Costs Needed
35 Tracking of Costs Conclusions
35 Need to Be Improved Recommendation
36 Agency Comments and Our Evaluation
36 Chapter 4
37 Technical Challenges Technology Development
37 Integration of Scientific Data
41 Are Present in All Technical Expertise
43 Aspects of the ASCI Program Appendixes
Appendix I: Program- and Laboratory-Level Software Milestones
46 From DOE Planning Documents, Fiscal Years 1996-2004 Appendix
II: Comments From the Department of Energy
49 Table Table 1.1. Requested ASCI Funding
by Program Component, 16 Fiscal Year 1999
Page 10 GAO/RCED-99-195 Strategic
Computing Initiative Contents Figures Figure 3.1 Original and
Current Cost Estimates for the ASCI 32
Program, Fiscal years 1996 Through 2004 Figure 3.2: Allocation of
Estimated ASCI Costs by Major Program 33 Areas,
Fiscal Years 1996 Through 2004 Abbreviations 3-D Three
dimension ASCI Accelerated Strategic Computing Initiative
DARHT Dual-Axis Radiograph Hydrodynamic Test Facility DOE
Department of Energy GAO General Accounting Office NIF
National Ignition Facility Page 11
GAO/RCED-99-195 Strategic Computing Initiative Chapter 1
Introduction Historically, the United States used actual nuclear
detonations as the primary method of validating designs and
certifying the weapons as safe and reliable. Since September 1992,
there has been a moratorium on testing. To ensure the continued
safety and reliability of nuclear weapons, the Department of
Energy (DOE), which is responsible for designing and building
nuclear weapons, developed the 15-year Stockpile Stewardship and
Management Program in 1995 as a substitute for actual testing. The
stockpile stewardship program employs a variety of means to ensure
weapons' safety and reliability, including examining weapons,
conducting laboratory experiments and tests, and conducting
computer modeling and simulation. The computer modeling and
simulation part of the program is known as the Accelerated
Strategic Computing Initiative (ASCI). The ASCI program aims to
replace actual testing with advanced computer models that will
simulate nuclear detonations. This effort requires modeling in 3-
dimensions (3-D), with higher resolution than previous models and
with better treatment of the underlying physical processes that
occur during an actual nuclear detonation. To run the models, DOE
is developing, as part of the ASCI program, the largest and
fastest computers, which may ultimately be able to perform 100
trillion mathematical operations per second-10,000 times more
powerful than those used to design the weapons originally. The
ASCI program is expected to cost about $5.2 billion for fiscal
years 1996 through 2004. The Evolving Role of Computers have
been used to design and build nuclear weapons almost Computing in
Nuclear from the dawn of the nuclear era. As early as 1945,
designers began using the ENIAC-the world's first computer, built
at the University of Weapons Pennsylvania with
government support-to perform calculations on the viability of a
hydrogen or thermonuclear bomb. A successor version, which was
fully electronic1 -the MANIAC-was built at Princeton in 1949, and
a duplicate was built at Los Alamos. From that time, computers,
and later so-called supercomputers, would play an increasing role
in the designing and building of the U.S. stockpile of nuclear
weapons. Computer models were used to design weapons and to
interpret data from actual nuclear weapons tests. Models and
computers were also used to identify and evaluate problems in the
nuclear weapons stockpile. In the end, however, the final arbiter
of a weapon's safety and reliability was usually an actual test or
series of tests. 1The ENIAC used vacuum tubes instead of gears to
perform calculations but had to be programmed for each new problem
by physically rearranging its circuit wires, which looked like
old-fashioned telephone switchboard cords. Page 12
GAO/RCED-99-195 Strategic Computing Initiative Chapter 1
Introduction Since the first nuclear weapon test, known as
Trinity, on July 16, 1945, the United States has conducted over
1,000 nuclear weapons tests. Testing was the principal method used
to certify the safety and reliability of nuclear weapons. Testing
was used to demonstrate that a particular weapon design actually
worked and yielded the expected power and to prove the safety and
reliability of components. For example, testing could be used to
demonstrate that older components were still functioning properly
after years of exposure to extremes of heat and cold and to
radiation. In addition to periodically testing stockpiled weapons,
the United States frequently developed new weapons to replace
older weapons in the stockpile, thus ensuring the continued
reliability and safety of its arsenal. In September 1992, the
Congress imposed a 9-month moratorium on underground nuclear
testing.2 This moratorium continued to be observed until September
1996, when President Clinton signed the Comprehensive Test Ban
Treaty.3 The test ban treaty has been interpreted by the
Administration to mean that no underground testing is allowed that
results in any nuclear yield-no matter how low. The practical
result of the test ban treaty is that existing nuclear weapons
will be kept longer than planned because new weapon designs cannot
be tested and certified as safe and reliable. The longer life span
of the existing stockpile of nuclear weapons increases the
possibility that they will decline in either performance or safety
because of age-related factors like extended exposure to heat,
vibration, and radiation. Faced with these testing restrictions,
DOE developed a new approach to certifying the safety and
reliability of weapons in the U.S. stockpile. A 1994 "Nuclear
Posture Review" charged DOE with maintaining the capability to
design, fabricate, and certify new weapons, if that ever became
necessary. DOE responded by developing the 15-year Stockpile
Stewardship and Management Program in 1995. The program is
intended to ensure the continued safety and reliability of
existing nuclear weapons using a variety of means, including
examining weapons to find possible problems, conducting
experiments to predict problems, and deciding on the basis of the
results of these efforts what, if anything, needs to be done to
ensure the continued reliability and safety of the weapons.
2Atmospheric testing was banned in 1963. 3Although the U.S. Senate
has not yet ratified the treaty, a statutory extension of the 1992
moratorium took effect on September 30, 1996, and continues
"unless a foreign state conducts a nuclear test" after that date,
in which case the moratorium is lifted. DOE continues to observe
the testing moratorium. Page 13
GAO/RCED-99-195 Strategic Computing Initiative Chapter 1
Introduction The ASCI component of the Stockpile Stewardship and
Management Program was intended to provide the modeling and
computers necessary to simulate in great detail the detonation of
a nuclear weapon. Related experimental facilities like the
National Ignition Facility (NIF) located at Lawrence Livermore
National Laboratory, and the Dual-Axis Radiograph Hydrodynamic
Test Facility (DARHT), located at Los Alamos National Laboratory,
are intended to provide the data needed to address basic physics
questions and to validate the accuracy of the ASCI computer
models. With this change to a science-based rather than a physical
test-based approach to addressing stockpile issues, the ASCI
program has become a critical link in certifying the safety and
reliability of nuclear weapons. The ASCI computer models and
hardware will be used to identify potential stockpile problems by
predicting the effects of aging and the need to replace components
or even to retire weapons systems if they become unsafe or
unreliable. In addition, the ASCI program will be used to design
and certify needed replacement parts as well as the entire weapons
system.4 The existing stockpile of weapons is aging, and many of
the designers of those weapons have retired or are approaching
retirement. For these reasons, DOE has decided that it is crucial
to have the ASCI program available by fiscal year 2004, including
the models and computers capable of performing 100-trillion
operations per second. The intent is to have the remaining
designers compare the output of the models against their actual
experience with nuclear weapons tests as one means of validating
the accuracy of computer models. Why Such Large The current
generation of nuclear weapons were designed on computers Computers
Are that were much smaller than those being developed for the
ASCI program-several million or a few billion operations per
second versus Needed 100-trillion operations per
second. A logical question rises as to why such vastly larger
computers are needed to ensure the safety and reliability of
existing weapons compared with those computers that were needed to
design and build these same weapons to the same safety and
reliability standards. The current stockpile of nuclear weapons
were designed and built using much less capable computers and far
simpler models than those 4Many of the manufacturing processes and
technologies that were used to build the current generation of
nuclear weapons and the components that they contain no longer
exist. As such, replacement components manufactured using new
processes, technologies, or materials need to be tested, in some
manner, and certified as to their performance and impact on the
weapons performance. Page 14
GAO/RCED-99-195 Strategic Computing Initiative Chapter 1
Introduction envisioned for the ASCI program. These less capable
computers could have been used for several reasons, including (1)
key components of the weapons were designed with a high level of
symmetry so that a one- or two-dimensional view of the component
would be fairly representative of the whole component, (2) weapons
were designed without a need to model all of the underlying
physics, (3) actual testing was used to resolve any uncertainties,
and (4) weapons were routinely replaced by newer, tested weapons
before they reached the end of their design life. With the loss of
testing opportunities and the aging of the current stockpile, this
approach is no longer feasible. Instead, DOE believes it is
necessary to provide detailed visual 3-D simulations of nuclear
weapons processes (that is, virtual testing capability). Virtual
testing requires far more complex and detailed models and much
greater computer capability to run these highly complex models in
a reasonable period of time. For example, to run certain two-
dimensional weapons calculations on a Cray YMP supercomputer (an
old generation of supercomputer but the type in use when some of
the existing weapons were designed) took up to 500 hours. By
comparison, moving from a two-dimensional to a 3-D model without
changing any other parts of the model results in a calculation
that is 1,000 times larger. At the same time, better detailed
physics calculations of what is happening at the time of the
nuclear detonation could require a calculation that is another
100,000 times larger. By extrapolating from these estimates, DOE
concluded that running such a calculation in a reasonable amount
of time (generally no more than several days for the largest
calculations) would require computers capable of calculating at
the rate of 100-trillion operations per second. Such machines were
far beyond those commercially available when the ASCI program was
started. Developing these increasingly powerful machines is one of
the main goals of the ASCI program. Building the highly complex 3-
D models is another. ASCI Program The ASCI program is comprised
of several components. As shown in table Overview 1.1, the
key components are Applications (software development), Platforms
(computers), and Infrastructure (peripheral technologies such as
networks, storage, and visualization). The program also includes
the Academic Strategic Alliances Program, which contracts with
universities for computing and scientific research to complement
ASCI efforts. In fiscal year 1999, the scope of the ASCI program
expanded when three new components were created. Numerical
Environment for Weapons Simulation will acquire the infrastructure
hardware needed for data Page 15
GAO/RCED-99-195 Strategic Computing Initiative Chapter 1
Introduction management and visualization. Distributed Distance
Computing will provide the infrastructure needed to extend the
advanced computing capabilities of the ASCI program to remote DOE
weapons facilities. Verification and Validation is intended to
verify that ASCI software is executing calculations as intended
and to validate the accuracy of ASCI software results. Table 1.1
shows funding for ASCI-related activities, including existing
computing facilities, verification and validation, and other
activities at each of the three laboratories to support ongoing
stockpile stewardship program requirements. Table 1.1. Requested
ASCI Funding by Program Component, Fiscal Year 1999 Dollars in
millions ASCI program component
Requested funding Applications (software)
$152 Platforms (computers)
70 Infrastructure Problem Solving Environments
46 Numerical Environment for Weapons Simulation
31 Distributed Distance Computing
28 Verification and Validation
13 Stockpile Computing
156 Academic Strategic Alliances Program
14 One Program/Three Labs (program coordination)
6 Total
$516 ASCI activities are carried out by DOE's three Defense
Program laboratories-Los Alamos, Lawrence Livermore, and Sandia
National Laboratories-with guidance from DOE's Office of Strategic
Computing and Simulation under the Assistant Secretary for Defense
Programs. For hardware development, the ASCI program intends to
build on economically viable computer vendor technologies and
thereby foster the health of the U.S. computing industry while
also stimulating competition in this industry to adopt new
technologies for advanced computing. ASCI computers of varying
sizes will be built and housed at each of the laboratories, and
their capabilities will be accessible to all three laboratories.
Infrastructure-related hardware will be procured from available
vendor technologies. For software development, the ASCI program is
relying on coordinated efforts at the three laboratories,
supplemented by university-based Page 16
GAO/RCED-99-195 Strategic Computing Initiative Chapter 1
Introduction research when possible. At Los Alamos and Livermore,
software development efforts focus on models that simulate the
performance of the nuclear components of weapons systems. In
contrast, software efforts at Sandia focus on models that simulate
the performance of nonnuclear weapons components like the arming,
firing, and guidance systems. Infrastructure-related software is
being developed in a joint effort by the three laboratories. The
schedules for hardware, software, and infrastructure development
are interdependent. For example, ASCI software must be able to
operate on a variety of increasingly large parallel computer
systems. The development of such "scalable" software requires the
availability of computers and peripheral technologies that are
sufficiently advanced to test and develop the software. Major
milestones for hardware, software, and infrastructure development
have been established for the program to 2004. Executing the ASCI
program and meeting these milestones with the involvement of three
laboratories will require close integration among programs and
across laboratories. Although the ASCI program's ultimate goal is
to provide 3-D weapons simulation capabilities by 2004, the ASCI
computers and software developed to date are already important
tools for addressing DOE's high-priority stockpile needs. Today's
ASCI computational capabilities, for example, are being used to
help design scientific experiments and to support the revalidation
and certification of certain weapons and/or their components in a
simulation environment. Objectives, Scope, As requested by the
Chairman, Subcommittee on Military Procurement, and Methodology
House Committee on Armed Services, we reviewed the management of
the ASCI program, including (1) whether the program was meeting
its key milestones and whether its hardware and software
developments are adequate to date; (2) whether the program was
within its projected budget; and (3) what key technical risks the
program faces. The scope of this review encompassed all aspects of
the ASCI program, which is conducted primarily by the Lawrence
Livermore, Los Alamos, and Sandia National Laboratories. To
determine whether the program was meeting its key milestones, we
obtained and reviewed planning and tracking documents and
interviewed ASCI program officials from the three weapons
laboratories and from DOE's Office of Defense Programs. We visited
each of the three laboratories and Page 17
GAO/RCED-99-195 Strategic Computing Initiative Chapter 1
Introduction DOE to discuss the program's progress in meeting its
key milestones and to obtain evidence to verify the statements
made by program officials. In July 1998 and January 1999, we
attended the semiannual ASCI "principal investigator" meetings to
learn more about the program's progress. We also reviewed studies
and reports that have assessed the status and progress of the ASCI
program. To determine whether the program is within its projected
budget, we examined cost and budget information provided to us by
DOE and the three laboratories. We also reviewed information from
DOE's Financial Information Variance Reporting System and
contracts. Furthermore, we examined budget information that was
included in DOE's budget request for fiscal years 1996 through
2000. We spoke to program officials to determine how costs were
estimated, why program costs have escalated, and how they review
and manage laboratory costs. We did not independently verify the
reliability of information contained in DOE's financial management
system, which we used in this report, because it is the basis for
DOE's financial statements, to which we have given an "unqualified
opinion" in our audit of the federal government's financial
statement. To identify the key technical risks facing the program,
we obtained and reviewed program-planning documents and
interviewed ASCI program managers from the three weapons
laboratories and from DOE's Office of Defense Programs. We also
reviewed studies and reports on the ASCI program and other
materials related to high-performance computing. We conducted our
review from July 1998 through June 1999 in accordance with
generally accepted government auditing standards. Key contributors
to this report were Linda Chu, Daniel Feehan, Anne McCaffrey, and
Edward Zadjura. Page 18 GAO/RCED-99-195
Strategic Computing Initiative Chapter 2 A Comprehensive Planning
and Tracking System Is Needed to Assess Program Progress Weak
management and information processes hamper oversight of the
strategic computing program and make assessing progress towards
program milestones difficult and subjective. Although program
managers report that many milestones have been met, the lack of
comprehensive planning and progress tracking systems make an
assessment of the program's short- and long-term progress
difficult and subjective. Current planning efforts include a
strategic plan that is out of date, annual plans that have been
prepared sporadically, and milestones that are not well defined.
Efforts to track the program's progress are not consistent, and no
clear record exists of program accomplishments compared with
milestones. Consequently, it is difficult to determine which of
the hundreds of milestones have been met, which are behind
schedule, or even which are still relevant, given changes in
program priorities, and how progress on individual projects
contributes to the program's overall goals. In response to our
requests for information, ASCI program officials have begun to
institute more systematic procedures for planning and for tracking
and reporting program progress. Comprehensive The
ASCI program's long- and short-term planning efforts thus far have
Planning Is Needed been inconsistent and incomplete.
Strategic planning documents have not been updated since the
program's inception. During the program's first 3 years, annual
implementation plans were prepared inconsistently or, in some
cases, not at all, resulting in an incomplete program baseline.
The long-term ASCI milestones presented in various plans are
inconsistent, and no information exists to link annual activities
to these milestones. Strategic plans also do not identify the
multiple research strategies currently employed to meet many long-
term milestones, nor do the plans include decision points for
managing these strategies. Performance criteria for most
milestones have not been defined in the planning process.
Strategic and Annual Plans The plans used to manage the ASCI
program to date have numerous Are Outdated,
limitations. The ASCI Program Plan is the program's primary
strategic plan. Inconsistent, and Published in 1996,
the plan included a list of long-term (program-level) Incomplete
milestones for hardware, software, and infrastructure development.
Although hardware and infrastructure milestones have not changed
much, program officials have revised software milestones numerous
times but have not published an updated strategic plan. Despite a
proliferation of program planning documents showing software
milestones, there is little consistency among these documents, and
no clear record of when and why milestones were changed and which
are the most current. Some of Page 19
GAO/RCED-99-195 Strategic Computing Initiative Chapter 2 A
Comprehensive Planning and Tracking System Is Needed to Assess
Program Progress the original milestones are intact or have
changed little over time, others have changed considerably, and
still others have been replaced with new milestones. In addition,
the dates for certain milestones are inconsistent among documents.
For example, the "Prototype 3-D Primary Simulations" milestone for
fiscal year 1999 has been shown consistently, although the
milestone name has changed slightly. By contrast, several
milestones do not appear consistently in different documents. For
example, the "abnormal environment thermal assessment" milestone
is listed as occurring in fiscal year 1999 in one planning
document and in fiscal year 2000 in another. (App. I lists ASCI
program milestones as identified in various planning documents).
This lack of consistency is also found in another strategic
planning document-the Stockpile Stewardship Plan (the "Green
Book"). The Green Book is the strategic plan for DOE's Office of
Defense Programs and is supported by more detailed planning
documents, including the ASCI Program Plan. The ASCI milestones
presented in the Green Book, however, are not always consistent
with those included in ASCI program documents. For example, the
Green Book lists the "crash/fire safety" and the "full physics,
full system prototype" as milestones, although they are not listed
among the most critical milestones compiled by the program office.
Conversely, the program office's most critical milestones list
includes the "hostile environment electronics assessment" and the
"abnormal environment thermal assessment" milestones, but the
Green Book does not list either. Program officials explained that
the Green Book is prepared at different times and for different
purposes than ASCI planning documents. Nonetheless, ASCI
milestones in DOE's Green Book and the ASCI Program Plan should be
consistent. In another example of inconsistent long-term planning,
the Simulation Development Roadmap has never been updated. This
document was intended to ensure that the ASCI program's simulation
and modeling activities would be clearly identified and that
priorities would be set on the basis of stockpile needs and
current capabilities. DOE intended to update this plan
periodically as the program progressed. Neither DOE nor the
laboratories, however, have developed this document beyond the
initial planning level since 1996. No other equivalent source of
information exists that provides a long-term, needs-based
perspective on the ASCI program's modeling and simulation
activities. DOE program officials said that there is no
programmatic requirement to update the Simulation Development
Roadmap and that some information about program needs is contained
in annual plans. This statement contradicts language in the Page
20 GAO/RCED-99-195 Strategic
Computing Initiative Chapter 2 A Comprehensive Planning and
Tracking System Is Needed to Assess Program Progress Stockpile
Stewardship Green Book, which states that the Simulation
Development Roadmap is to be used in conjunction with annual
planning documents to define computing requirements. Furthermore,
as discussed below, annual planning documents have not been
consistently prepared. The three laboratories in this program have
also developed software milestones (laboratory-level milestones)
and published various compilations of these. A consolidated set of
these milestones, known as the "Consolidated Roadmap" is supposed
to show the software milestones for all three laboratories.
Another listing, known as the "Nuclear Roadmap," shows combined
software milestones for two laboratories-Los Alamos and Livermore-
whose modeling efforts focus on the nuclear components of weapons
systems. A third listing, known as the "Non-Nuclear Roadmap,"
shows software milestones for Sandia Laboratories, whose modeling
efforts focus on nonnuclear weapons components, such as those for
arming, firing, and guidance. As with other ASCI program
documents, these sources report milestones inconsistently, and
laboratory-level milestones may or may not match program-level
milestones. For example, the consolidated roadmap lists milestones
such as "3-D forging/welding microstructure," "full physics burn
code prototype," and "burn code with aging" that do not appear in
either the nuclear or nonnuclear roadmaps. Conversely, the nuclear
and nonnuclear roadmaps list milestones such as the "3-D nuclear
safety simulation" and the "B61 penetrator" that do not appear in
the consolidated roadmap. The laboratories' long-term planning
efforts also include multiyear plans for some of the individual
software projects that contribute to laboratory-level milestones,
but such plans are not required, and their format has varied from
laboratory to laboratory. At Sandia, for example, project plans
spanning 5 years have been prepared for key software projects. At
Los Alamos and Livermore, project plans also have been developed
for key software projects, but their time frames are shorter and
variable (that is, 2 or 3 years). Short-term planning, as
represented by annual implementation plans, has also been
inconsistent and incomplete. The purpose of these plans is to
specify project tasks and milestones for the current year (annual
milestones). However, during the first 3 years of the program,
annual implementation plans were prepared by the three
laboratories for some, but not all, components of the ASCI program
and for only some of those years. DOE's failure to ensure
comprehensive and consistent planning during those years has
resulted in an incomplete program baseline for Page 21
GAO/RCED-99-195 Strategic Computing Initiative Chapter 2 A
Comprehensive Planning and Tracking System Is Needed to Assess
Program Progress fiscal years 1996 through 1998. In addition, the
annual plans do not explain changes in the scope of the work or
milestones that occur from year to year. Fiscal year 1999 was the
first year that DOE required the laboratories to submit
implementation plans for all the components of the program and
consolidated these documents into a program-wide ASCI
Implementation Plan. The Linkages Between To date, DOE has not
required that program documents show linkages Annual and Long-Term
between annual milestones and long-term program- or laboratory-
level Milestones Need to Be milestones. With the exception of
information provided in response to our Identified
request, these linkages have never been documented. Without clear
information to identify such links, it is impossible to determine
how annual progress contributes to meeting those long-term
milestones. Program officials maintain that such links do exist
but that they are not explicit in program documents. They also
acknowledged that technical expertise is needed to identify links
between annual milestones and long-term milestones. In our
discussions with ASCI laboratory staff, however, we found that
such links do not always exist and sometimes could not be
identified even by laboratory personnel. At Sandia, for example,
laboratory officials identified ASCI software activities that are
needed to meet stockpile requirements but that are not tied
directly to program-level milestones. In another case, it was not
apparent what laboratory activities contributed to meeting the
"macro-micro aging" milestone. DOE and laboratory officials
identified Sandia as the laboratory responsible for this
milestone, planned for completion in early 1999. While Sandia
officials identified some activities that they believed were
relevant to meeting this milestone, they were unsure about whether
Sandia's activities were all that was needed to meet this
milestone. They said it was possible that ongoing projects at the
other two laboratories contributed to meeting this milestone.
Program and laboratory officials agreed that such links should be
made more apparent and, in trying to respond to our request for
information about these links, they attempted to identify and
document linkages. Although complete information was not provided
by all of the laboratories, the information received shows that
annual milestones are not always directly linked to long-term
milestones. DOE and laboratory managers also told us that the
process of developing this information was helpful for tracking
the progress of the program and that they plan to refine this
process and update the information on a quarterly basis. Page 22
GAO/RCED-99-195 Strategic Computing Initiative Chapter 2 A
Comprehensive Planning and Tracking System Is Needed to Assess
Program Progress Plans Should Clearly ASCI plans do not
identify all of their multiple research strategies nor do Identify
Research they establish decision points for
identifying which of these strategies are Strategies, Critical
Paths, critical for meeting key program milestones. Because of
the complex and Decision Points technological
challenges involved in developing software and hardware to model
nuclear weapons, the laboratories have undertaken multiple
research strategies in an effort to mitigate risk and achieve
their laboratory- and program-level milestones. According to
laboratory officials, they will eventually have to reassess these
strategies to see which are working and which are not and decide
how they can focus their efforts to best achieve their milestones.
As a result, individual projects could be enhanced, scaled back,
or eliminated as "critical paths" are defined. In the area of
infrastructure development, for example, one overall goal is to
develop ways for scientists to examine massive amounts of weapons
simulation data. To achieve this goal, the development effort has
at least seven simultaneous lines of effort, including developing
"common data formats" and "distributed file systems." Within those
lines of effort, multiple research approaches are being pursued. A
program official explained that all of these lines of effort are
needed but that it is not clear at this time which efforts will be
critical to meeting the overall goal. It is also not known when a
choice among these options needs to be made. Another development
effort is focused on specialized software that would help ASCI
software developers understand and improve the performance of
their weapons software programs, which ultimately will help to
reduce the time needed to solve such problems. Currently, there is
no such specialized software to solve such problems. Commercial
vendors are developing such software, but only to a certain extent
because there is limited demand for such products outside the ASCI
program. As a result, while ASCI program officials are considering
vendor products, the program is also funding development efforts
at universities and collaborating with industry to develop the
needed software. The program official explained that the software
needed for the ASCI program may be available from at least one of
these sources, but, again no decision points have been established
for when a choice among these options needs to be made. According
to this official, infrastructure research strategies are managed
and decisions about them are made on a yearly basis by a team
assembled from the three laboratories. In the area of software
development, ASCI software developers at Los Alamos and Livermore
laboratories use multiple research and risk mitigation strategies
in developing their weapons simulation software. Los Alamos and
Livermore have multiple software teams competing to develop Page
23 GAO/RCED-99-195 Strategic
Computing Initiative Chapter 2 A Comprehensive Planning and
Tracking System Is Needed to Assess Program Progress weapons
simulation software. Program officials explain that having several
teams is advantageous because the arrangement allows the
laboratories to explore different simulation approaches, cross-
check/validate each other's work, and mitigate the risk of
selecting one approach before all approaches are explored
adequately. These different research and risk mitigation efforts,
however, are not identified clearly in planning documents. In
addition, although laboratory officials acknowledge that they will
need to eventually scale back or restructure their software
efforts, no decision points have been established for doing this.
Plans Should Define Measurable performance criteria for
most ASCI milestones have not been Performance Criteria for
defined. Laboratory officials were, in many cases, unable to
specify what Milestones needed to be done to
complete a milestone, which laboratory(ies) bore responsibility
for meeting it, or what their own particular contributions to
meeting that milestone were. Laboratory officials said that no
objective and specific measures exist, in most cases, to determine
whether milestones have been successfully completed. Instead, they
make subjective judgments about when a body of work meets a given
milestone. For example, the "micro-aging" milestone, which appears
in plans as a critical program milestone, was scheduled for
completion in 1997. Los Alamos officials said they were unfamiliar
with this milestone but believed it was Sandia's responsibility.
Sandia officials agreed that they were responsible for this
milestone and said that, in their opinion, the milestone had been
met. However, while they identified relevant tasks, they were
unable to identify precisely and completely what was supposed to
be done to meet the milestone. Sandia officials speculated that
the other two laboratories also had a responsibility for meeting
this milestone. On the other hand, we found one example of a
milestone for which specific performance criteria have been
established. The "Prototype 3-D Primary Simulations" milestone,
scheduled for completion by the end of calendar year 1999, was
subject to a review by program officials in June 1998. The review
established technical specifications for successfully completing
the milestone and assessed the progress of the Los Alamos and
Livermore software development activities contributing to this
milestone. A progress review for this milestone is planned for the
summer of 1999 and a follow-up review is planned upon its
completion. Program officials acknowledge that they need to
perform similar reviews for other milestones. In May 1999, DOE
issued a report on its review, entitled "Codes for the Complex,"
of the nonnuclear mechanics software development Page 24
GAO/RCED-99-195 Strategic Computing Initiative Chapter 2 A
Comprehensive Planning and Tracking System Is Needed to Assess
Program Progress efforts. At this juncture, however, most program-
level milestones remain undefined. Systematic Progress The
efforts of DOE and laboratory managers to track ASCI progress thus
far Tracking and have been inadequate. Progress tracking
has been limited primarily to the reporting of annual
accomplishments without any systematic tracking of Reporting
Needed progress towards long-term milestones. Current
tracking efforts occur through a variety of formal and informal
methods, and the format and organization of these tracking efforts
has varied from year to year. As a result, it is not possible to
determine whether annual milestones were achieved or the extent to
which annual efforts were contributing to the laboratory- and
program-level milestones. In response to our request for tracking
information, program officials decided to track and report the
program's progress more systematically. The ASCI Program Plan
described the semiannual principal investigators meetings as the
primary forum for reporting program progress. The plan also noted
that performance metrics would be developed and used at these
meetings to compare actual output with planned output. However,
these metrics were never developed. While these meetings are a
forum for ASCI researchers to exchange ideas, there are few
reporting requirements, and reports on program accomplishments
have generally not been related to established milestones.
Furthermore, the meetings focus on presentations of individual
projects, with no effort to pull together a systematic and
comprehensive assessment of how the ASCI program is progressing
towards its overall goals. Laboratory officials met with DOE
officials in fiscal years 1997 and 1998 to report their annual
accomplishments. These meetings were focused solely on
accomplishments and not on reporting the status of all milestones.
As a result of this limited reporting process, some key
information about the program's progress was not divulged. For
example, accomplishments reported for fiscal year 1998 for one key
software project did not reveal that the project was actually 6
months behind schedule, which affected the schedule of other
related projects. The Annual Performance Report is published at
the end of each fiscal year to report on the progress of the
Stockpile Stewardship Program, including the ASCI efforts. The
report discusses various ASCI milestones met during the year, but
these accomplishments are not all tied to the program-level
milestones, and no assessment is made of how the program is
progressing Page 25 GAO/RCED-99-
195 Strategic Computing Initiative Chapter 2 A Comprehensive
Planning and Tracking System Is Needed to Assess Program Progress
in terms of meeting its most critical milestones. As such, the
report did not provide a comprehensive assessment of the ASCI
program. However, according to the Green Book, the ASCI program is
critical to the success of the overall Stockpile Stewardship
Program. To varying degrees, the laboratories used the annual
implementation plans to report accomplishments during fiscal years
1996 through 1998, although this practice was discontinued in
fiscal year 1999. In fiscal year 1997, the implementation plan for
infrastructure work at the three laboratories did not report any
accomplishments for 1996, while the Sandia plan for software
development reported several. However, the reported
accomplishments cannot always be correlated easily with
established milestones. In some cases, a connection was apparent,
while in other cases it was not. In response to our request for
information, the laboratories agreed to prepare material showing
progress in meeting the milestones established in recent annual
implementation plans. Because implementation plans were not
prepared for all ASCI program components in 1998, however, draft
documents and other plans are being used as a baseline to track
progress in certain areas. Laboratory and DOE officials said that
the information they developed at our request was very helpful for
tracking program progress. DOE plans to have the laboratories
refine their efforts and require that this information be updated
on a quarterly basis. Program Progress Although program
managers have reported that many milestones have Difficult to
Assess been met, it is difficult to gauge the ASCI program's
overall progress because of weaknesses in program management and
information processes. The lack of a systematic progress tracking
system, combined with the lack of defined performance criteria,
make it difficult to assess whether the ASCI program is proceeding
on schedule and delivering the performance expected. However, it
is possible to gain some limited insights by discussing individual
projects or areas with laboratory officials. In an attempt to
determine the adequacy of hardware and software development to
date, we discussed program progress in these areas with program
officials. The insights we gained are discussed below. Hardware
Performance In the area of hardware development, most contract
milestones to date that relate to the delivery and installation of
the computers and related hardware have been met, although not all
acceptance tests have been met. Currently the two 3-trillion
operations per second systems at Los Alamos Page 26
GAO/RCED-99-195 Strategic Computing Initiative Chapter 2 A
Comprehensive Planning and Tracking System Is Needed to Assess
Program Progress and Livermore are operational. However, Los
Alamos is still working to address problems affecting the
reliability and stability of its system, which, as a result of
these problems, did not pass all its contractually required
acceptance tests on time. Progress in hardware development has
been reported in annual implementation plans, at semiannual
principal investigator meetings, and in press releases to the
media. DOE and the laboratories, for example, have issued press
releases that emphasize how fast these systems have performed on
tests of their sustained speed. These releases may be misleading
in some cases because they are reporting on only selected aspects
of performance. For example, DOE reported that the Los Alamos
system established a world record for sustained speed. However, it
was not reported that the test used was substantially easier than
the test specified in the contract or that the system has not yet
met other acceptance test criteria (such as, mean time between
failures). The sustained speed of this machine would be one-half
the speed reported in press releases had the test specified in the
contract been used. In addition, the Livermore computer continues
to have problems with parallel input/output file operations. DOE's
high-performance computers are expected to incorporate state-of-
the-art hardware and software technologies. These computers can
process multiple parts of one program at the same time, using
parallel-processing techniques. According to the ASCI Program
Plan, of the 100,000-fold increase in computing performance needed
by the program, DOE expects a 10-fold increase from improving
software to take advantage of parallel-processing techniques.
However, the management of the input and output data during such
processing continues to be an issue. For example, the data that
result from running a model on 1,000 processors currently have to
be saved to 1,000 separate files, making the data more difficult
to manage and use. The laboratory is working with the vendor to
address this problem. Software Development Documentation of
ASCI's software performance is limited. With the exception of the
"Prototype 3-D Primary Simulations" milestone review discussed
previously, little documentation exists that compares software
development progress against established milestones. This review,
also known as the "burn code review," is the only formal review of
a milestone done to date. The numerous other software projects
related to other milestones, such as 3-D secondary burn code
projects at Los Alamos and Livermore and all software projects at
Sandia, have not been reviewed. Page 27
GAO/RCED-99-195 Strategic Computing Initiative Chapter 2 A
Comprehensive Planning and Tracking System Is Needed to Assess
Program Progress According to our analysis of planning documents,
accomplishment reports, and other related material on software
activities at Sandia, the ASCI program's current planning and
tracking system does not always provide an accurate picture of
progress. For example, in reviewing the documentation for a major
software development effort that Sandia officials told us was 6
months behind schedule, we found no indication that delays had
occurred. The documentation showed that most fiscal year 1998
milestones had been met, and all milestones for the first quarter
of fiscal year 1999 also were met. Sandia officials explained
that, although the reported information was accurate, milestones
had not been established for all of the important tasks associated
with that project. Since the purpose of this project is to provide
a common framework for all Sandia ASCI software models, delays in
this project are hindering the performance of other software
projects that require integration into the common framework.
Sandia also did not report these delays in the list of
accomplishments for fiscal year 1998 that it submitted to DOE.
Sandia officials told us that, as of March 1999, 50 percent of
their software development projects were experiencing delays
because of funding or program changes or are behind schedule for
other reasons. In general, detailed performance requirements have
not been established for most software milestones, so it is
difficult to develop an objective assessment of performance in
this area. In effect, the judgment of whether software tests or
demonstrations have achieved the desired level of performance is
based on the subjective opinion of the laboratory and DOE program
managers. DOE May Need to DOE may not be appropriately
managing the ASCI program by not Manage ASCI as a designating
it as a strategic system. DOE has established criteria for
designating its most important projects as strategic systems to
ensure Strategic System oversight at the highest departmental
level. The criteria are that the project costs more than $400
million, is an urgent national priority, be high-risk, have
international implications, or be vital to national security. The
purpose of designating strategic systems is to ensure informed,
objective, and well-documented decisions for key events, such as
changes to baseline cost or schedule. The ASCI program meets the
criteria for being treated as a strategic system. The ASCI program
will likely cost about $5.2 billion for fiscal years 1996 through
2004, is a critical part of the stockpile stewardship program, is
an urgent national priority on national security grounds, and has
international implications because it is a major factor in U.S.
support of the Comprehensive Test Ban Treaty. Finally, the ASCI
program is high risk because it seeks to advance the state of the
art in computers, modeling, and simulation well beyond current
capabilities, has Page 28
GAO/RCED-99-195 Strategic Computing Initiative Chapter 2 A
Comprehensive Planning and Tracking System Is Needed to Assess
Program Progress already experienced delays, has had its projected
cost increase, and depends on as-yet unknown technologies for
success. These characteristics, coupled with the demonstrated
weaknesses in program management and oversight, make the ASCI
program a clear candidate for being designated as a strategic
system. According to DOE, it has not designated this effort as a
strategic system because the program is already subject to high-
level department oversight. However, as discussed above, we found
serious weaknesses in the program's management and information
processes that make it difficult to determine if the program is
performing as expected. Conclusions DOE's oversight of the
ASCI program is hampered by weaknesses in management and
information processes. The program lacks a comprehensive planning
system-one that clearly establishes milestones; links short- and
long-term milestones; identifies research strategies, critical
paths, and decision points; and defines performance criteria for
milestones. Furthermore, the program lacks a progress tracking and
reporting system. Consequently, overall program progress is
difficult to assess. DOE has chosen not to designate the ASCI
program as a strategic system. The demonstrated weaknesses in the
ASCI program's management and information processes, coupled with
the program's critical role in DOE's mission to maintain the
nation's stockpile of nuclear weapons without testing, warrant
DOE's designating ASCI as a strategic system requiring the highest
levels of management attention. Recommendations Given the ASCI
program's lack of a comprehensive planning, tracking, and
reporting system and the importance of the program to maintaining
the stockpile of nuclear weapons, it is important that DOE improve
its oversight and management of this program. Therefore, we
recommend that the Secretary of Energy require the establishment
of a comprehensive planning, tracking, and reporting system. This
system should, at a minimum, establish clear milestones; identify
links between short- and long-term milestones; identify research
strategies, critical paths, and decision points; define
performance criteria for the successful completion of milestones;
and establish progress tracking and reporting requirements. We
further recommend that the Secretary of Energy designate the ASCI
program as a strategic system warranting oversight at the highest
departmental level. Page 29
GAO/RCED-99-195 Strategic Computing Initiative Chapter 2 A
Comprehensive Planning and Tracking System Is Needed to Assess
Program Progress Agency Comments DOE agreed with the
recommendation to improve the oversight and and Our Evaluation
management of the program and cited several changes that it was
making. Specifically, DOE stated that it would soon issue an
updated Program Plan that will include detailed specifications for
all of the critical program milestones. In addition, the FY 2000
Implementation Plans that will be issued by September 30, 1999,
will include descriptions of all program elements and complete
lists of all milestones. The Department also cited the creation of
a quarterly progress tracking mechanism to track program
milestones. However, in addition to tracking the program's
progress against established calendar milestones, it is necessary
to establish specific technical criteria for what constitutes the
successful completion of those milestones. Until DOE completes and
publishes its revised Program Plan and FY 2000 Implementation
Plans, we cannot determine whether it has fully complied with this
recommendation. DOE disagreed with the recommendation to designate
the ASCI program as a strategic system and stated that to do so
would duplicate the planning, progress tracking, and reporting
system. We agree that creating a duplicative tracking system that
mirrors the requirements set out by DOE for strategic systems
would not be worthwhile. However, as discussed in detail in this
report, DOE has not shown that it has an adequate planning,
progress tracking, and reporting system in place for the strategic
computing initiative. While DOE is making improvements in these
areas, the changes are not yet fully in place and their adequacy
cannot be judged at this time. Furthermore, if the changes that
DOE is making are adequate to meet the requirements for tracking
and monitoring a strategic system, then we cannot understand DOE's
reluctance to designate this large and costly program as a
strategic system. DOE stated that it has a review process that
meets the intent of the Clinger-Cohen Act of 1996. However, we
reported in July 1998 that the Department's process effectively
excludes scientific computers like those being acquired through
the ASCI program from DOE's normal review channels and places them
within the program offices.5 We stated that all computers should
be included as part of the normal DOE Clinger-Cohen review
process. 5Information Technology: Department of Energy Does Not
Effectively Manage Its Supercomputers (GAO/RCED-98-208, Jul. 17,
1998). Page 30 GAO/RCED-99-195
Strategic Computing Initiative Chapter 3 Management and Tracking
of Costs Need to Be Improved ASCI cost estimates have increased
substantially. In 1995, DOE estimated that program costs for
fiscal years 1996 through 2001 would be $1.7 billion.1 By 1999,
estimated costs for those years increased to $2.9 billion. DOE
currently estimates that the program will cost about $5.2 billion
for fiscal years 1996 through 2004. Some of the cost increases
result from changing program requirements. For example, in 1996,
the United States shifted from a program based on nuclear testing
to one based on computer simulations of weapon performance. For
the ASCI program, the shift to computer simulations resulted in
higher costs to acquire the latest and fastest computers and to
develop advanced simulation and modeling software. The cost
increases also reflect weaknesses in DOE's cost estimation and
management. For example, DOE has difficulty determining technical
requirements and then reliably estimating costs for state-of-the-
art computers and software. In addition, DOE limits its cost
oversight to reviews of aggregate laboratory spending and
consequently cannot determine if the costs of specific projects at
the laboratories are over or under budget. Cost Estimates DOE's
cost estimates for the ASCI program have increased substantially
Increased since 1995, when early budget projections were
made. Costs for fiscal years 1996 through 2001 have increased from
an original estimate of $1.7 Substantially billion to the
current $2.9 billion. DOE's actual fiscal year 2000 budget request
for ASCI, which totals $692 million, is more than double the
original fiscal year 2000 estimate made in 1995. Figure 3.1 shows
for each fiscal year the original and current budget estimates.
1The estimated cost figures provided in this report have not been
adjusted to constant dollars. Rather, they reflect DOE's budgeting
and planning process estimates, which were provided in current
dollars. Page 31 GAO/RCED-
99-195 Strategic Computing Initiative Chapter 3 Management and
Tracking of Costs Need to Be Improved Figure 3.1 Original and
Current Cost Estimates for the ASCI Program, Fiscal 900
Dollars in millions Years 1996 Through 2004 800 700 600 500 400
300 200 100 0 1996 1997 1998 1999 2000 2001
2002 2003 2004 Fiscal Year Original Current Source: GAO's
analysis of DOE's data. The current total estimated cost of the
ASCI program, for fiscal years 1996 through 2004, is about $5.2
billion. Although the program is scheduled to operate through
2010, estimates beyond 2004 have not been made. Figure 3.2 shows
how this $5.2 billion is allocated by program areas. Page 32
GAO/RCED-99-195 Strategic Computing Initiative Chapter 3
Management and Tracking of Costs Need to Be Improved Figure 3.2:
Allocation of Estimated ASCI Costs by Major Program Areas,
Dollars in millions Fiscal Years 1996 Through 2004
$60 Program coordination * $218 Academic
alliances * $251 New buildings * $776
Computers * $3,937 Software and
infrastructure Source: GAO's analysis of DOE's data The $776
million estimated for computers includes the costs of the three
computers currently installed at the laboratories and the planned
acquisition of computers that perform 10-, 30-, and 100-trillion
operations per second. The vast majority of ASCI funds, almost $4
billion, will be spent on the software development and program
infrastructure. Software and infrastructure include the
development of the 3-D simulation models and the multiple
peripheral technologies needed for, among other things,
visualization, networking, and data management. DOE estimates it
will spend $251 million to construct new buildings at the
laboratories that will house the computers, as well as offices and
visualization theaters. The Academic Strategic Alliances Program
with the universities will cost $218 million, and another $60
million is estimated for the coordinating laboratories' efforts.
Changing Requirements DOE officials told us that
ASCI costs have escalated since the original budget Account for
Some Cost projections because the program was
originally intended as a limited Increases
effort to improve the computer capability available for stockpile
Page 33 GAO/RCED-99-195 Strategic
Computing Initiative Chapter 3 Management and Tracking of Costs
Need to Be Improved stewardship. In the original budget estimate,
developed in 1995, DOE officials requested funding for a 1,000-
fold increase in computing capability. Since then, the ASCI
program has expanded because of changes in the U.S. nuclear
weapons policy, particularly, the U.S. decision in August 1995 to
pursue a "zero yield" Comprehensive Test Ban Treaty. This policy
change meant that the United States would need to maintain the
nuclear stockpile far beyond its design life and would have to
shift from a traditional nuclear test-based program to one based
on computer simulations to ensure the safety and reliability of
nuclear weapons. As a result, DOE has developed strategies that
focus on advanced modeling and simulation that require a 100,000-
fold increase in computer capability. Unreliable Cost Estimates
While the change in program requirements has affected DOE's budget
Also Contribute to Cost estimates, officials also acknowledge
their difficulty in estimating costs Increases
because of the unprecedented scale of the hardware and software
technologies needed by the ASCI program. For example, before DOE
began this effort, a computer with thousands of processors
operating as a fully integrated system had never been built. In
addition, software to run on systems of this size and high-
performance visualization technology to display the results of
simulations at this scale had never been developed. According to
one DOE official, the Department might lack the expertise to
anticipate future technical requirements for state-of-the-art
hardware and software and to reliably estimate their costs. For
example, when planning the current, expanded ASCI program, DOE
envisioned a single computer capable of 3 trillion operations per
second that could provide access and collaborative opportunities
to all three laboratories, using secure, high-speed networking
capabilities. However, DOE decided to procure a second such
computer in 1997 because the technical capabilities to support
such long-distance computing were not yet in place. Combined, the
two computer systems cost DOE almost $220 million. In addition,
DOE acknowledged that during early funding strategies, it did not
consider the difficulty and importance of the technology needed by
weapons designers to visualize the results of the 3-D weapons
simulations. Yet such visualization technologies are required to
graphically represent to weapons designers the results of 3-D ASCI
simulations. DOE currently estimates that more than $87 million is
needed for visualization activities for fiscal years 1999 through
2004. Page 34 GAO/RCED-99-195
Strategic Computing Initiative Chapter 3 Management and Tracking
of Costs Need to Be Improved Better Oversight of DOE's
oversight of costs is limited to a review of aggregate spending at
the Costs Needed laboratories. While DOE contends that
cost controls for ASCI are in place, it does not track costs to
determine which specific projects may be over or under budget. DOE
monitors monthly spending for each laboratory but does not compare
previously estimated costs for major projects with their actual
costs. For example, as part of the budget formulation process,
project costs are estimated and subsequently included as part of
the Department's fiscal year budget request. However, DOE tracks
only how much the laboratories have spent in broad categories that
lump together costs for many projects. As a result, DOE cannot
determine which projects, if any, may be costing more or less than
originally planned. DOE told us it relies on the laboratories to
determine whether projects are within their planned budget, but
one laboratory ASCI manager told us that the laboratory tracks
only the technical status of projects, not their costs. In
addition, DOE's limited oversight of the laboratories' activities
could result in DOE's underestimating ASCI program costs. DOE
estimates that ASCI program costs have increased by about $1.2
billion compared with its original estimate for fiscal years 1996
through 2001. However, delays in completing projects at the
laboratories could increase those costs. For example, as noted in
chapter 2, one laboratory estimates that 50 percent of its
software development projects are experiencing delays or are
behind schedule. DOE's lack of information about the progress of
projects, combined with its limited cost tracking, do not allow
DOE to determine how much longer it will take to complete those
projects or at what cost, thus limiting its ability to accurately
project ASCI program costs. In January 1999, we reported on the
significant management challenges at DOE, including the difficulty
completing large projects within budget.2 We noted that DOE often
requires large projects costing hundreds of millions of dollars
that are often the first of their kind and involve substantial
risk. ASCI is such a challenge. ASCI is critical to DOE's mission,
is estimated to cost about $5.2 billion, requires the development
of hardware and software on an unprecedented scale, and involves
substantial risks. Conclusions ASCI costs have
increased substantially because of changes in program requirements
and weaknesses in DOE's cost estimates. Because its tracking of
costs is limited, DOE cannot determine whether specific projects
are under or over budget. Historically, DOE has had difficulty
managing the costs of large programs. 2Department of Energy: Major
Management Challenges and Program Risks (GAO/OGC-99-6, Jan. 1999).
Page 35 GAO/RCED-99-195
Strategic Computing Initiative Chapter 3 Management and Tracking
of Costs Need to Be Improved Recommendation Given the
substantial increases in the ASCI program's cost estimates to
date, DOE's acknowledged problem in estimating costs for the
unprecedented scale of development efforts involved in the ASCI
program, and the lack of a cost-tracking process, it is important
that DOE improve its oversight of ASCI program costs. Therefore,
we recommend that the Secretary of Energy require that ASCI adopt
systematic cost-tracking procedures that will allow DOE managers
to determine if specific projects are within budget. Agency
Comments DOE did not agree with our recommendation to adopt
systematic and Our Evaluation cost-tracking procedures for the
strategic computing initiative, noting that funding and costs are
tracked by budget and reporting codes in the Department's
Financial Information System. DOE stated that these systems are
extended to individual projects using other funding and cost-
monitoring tools that gather more detailed information. As an
example, DOE cited an analysis performed in March 1999 of selected
projects that identified the commitments and cost status for
specific procurements at the project level. We do not agree that
DOE has an adequate system for tracking at the project level or
that the changes it is making will rectify this problem. DOE's
current system tracks cost only at the aggregate level and does
not allow DOE managers to determine which projects at the
laboratories are under or over budget. Furthermore, the "other
funding and cost monitoring tools" that DOE uses do not allow the
systematic tracking of project costs. DOE also stated that some
budgeting flexibility is necessary to capitalize on changes within
the high-computing industry. While we agree that some budgeting
flexibility is necessary in a project of this size and complexity,
we do not believe that that flexibility should preclude the
effective oversight of a multiyear program costing over $5
billion. Page 36 GAO/RCED-99-195
Strategic Computing Initiative Chapter 4 Technical Challenges Are
Present in All Aspects of the ASCI Program The primary challenge
facing the ASCI program is to develop a simulation capability
that, in the absence of nuclear testing, can be used to determine
whether a modified weapon system will perform as intended. The
need for this "virtual test" capability encompasses most of the
technical challenges associated with the ASCI program. These
challenges range from developing state-of-the-art hardware and
software technologies, to the integration of scientific data from
weapons physics experiments, to recruiting and retaining staff
with the technical expertise needed. The risks in failing to
address these challenges are intensified because the program is a
research-and-development effort with an accelerated schedule.
Program officials acknowledge the multitude of risks associated
with the program and point to the risk mitigation strategies they
have designed to address program risks. These strategies include
the use of several vendors to develop computers, overlapping
software development efforts, and partnerships with industry and
academia. Technology The development of hardware
and software technologies and of the Development
necessary infrastructure to support these technologies is critical
to achieving the ASCI program's simulation and modeling goals.
Hardware development must successfully increase computational
speeds to 100-trillion operations per second in 2004. Software
development efforts are extensive and must ultimately incorporate
massive amounts of data, solve progressively more difficult
problems, and be capable of running on increasingly larger and
faster computers. Developing the needed infrastructure, including
data storage and visualization technologies, will require
significant improvements. Computer Speed Must Developing
computers capable of processing complex 3-D nuclear weapons
Increase Dramatically by simulations is one of the primary
challenges facing the ASCI program. 2004
Increasing the computational speed to 100-trillion operations per
second by 2004, according to program officials, is essential to
meeting program goals. At Livermore, development is under way on a
10-trillion operations per second computer that is scheduled to be
installed during fiscal year 2000, and the acquisition of a 30-
trillion operations per second computer at Los Alamos is planned
for fiscal year 2001. A request for proposals for the 30-trillion
operations per second computer was sent out in May 1999, and
contracts have been signed with several vendors to work on the
related technology needed for a computer of this size. Page 37
GAO/RCED-99-195 Strategic Computing Initiative Chapter 4 Technical
Challenges Are Present in All Aspects of the ASCI Program Program
officials explained that their risk mitigation strategy includes
using competing computer vendors to independently develop
increasingly larger computers at the three weapons laboratories. A
1.8-trillion operations per second computer that was developed by
the Intel Corporation is in use at Sandia, while both Los Alamos
and Livermore are developing computers capable of 3-trillion
operations per second. The computer at Los Alamos is being
developed by Silicon Graphics, Incorporated, while Livermore is
working with the IBM Corporation to develop its computer.
According to laboratory officials, the experience gained at
Sandia, plus the competing efforts at Los Alamos and Livermore,
helps to ensure that at least one of these computers will be
generally available to carry out computational work. According to
a program official, the competing computers at Los Alamos and
Livermore are based on different technologies, which helps to
further mitigate risks. Currently, Livermore is developing the 10-
trillion operations per second computer with the IBM Corporation,
while Los Alamos sent out a request for proposal to select a
vendor to develop the 30-trillion operations per second computer
in May 1999. ASCI program officials explained that the ASCI
computers being constructed involve thousands of processors,
switches, disks, and related components that must work together as
a fully integrated system to run the largest simulations. These
officials explained that getting computer systems of this size to
operate as a fully integrated system has never before been
achieved and is one of the most difficult challenges facing the
program. An April 1998 review of the computing division at Los
Alamos by an external committee recognized this issue by pointing
out that users generally had access only to small parts of the
computer and rarely had access to the full system.1 The
Committee's report explained that operating the computer as a
fully integrated system was important because the ASCI computer
needs are based on running simulations that require the full
capability of the computer. The challenge continues today with the
3-trillion operations per second computer at Los Alamos, which has
experienced many failures when trying to run as a fully integrated
system. A March 1999 review by the ASCI Blue Ribbon Panel noted
another important risk in meeting the schedules for computers
operating in the range of 30- to 100-trillion operations per
second.2 The report explained that to meet the schedule for these
larger computers, it might be necessary for the laboratories to
write the system software necessary to enable the 1Los Alamos CIC
Division External Review Committee Report for the April 1998
Review. 2Report of the ASCI Blue Ribbon Panel, Mar. 2, 1999. Page
38 GAO/RCED-99-195
Strategic Computing Initiative Chapter 4 Technical Challenges Are
Present in All Aspects of the ASCI Program computer to operate as
a fully integrated system. The report noted that there is a risk
in such a course of action because laboratory personnel do not
have extensive experience in this area. Software Development Is
Developing software that incorporates all of the required science
to Critical to Program simulate nuclear weapons while
running on computers consisting of Success
thousands of processors is, according to ASCI program managers,
one of the most demanding tasks of the ASCI program. These
officials explained that developing such software has historically
taken approximately 5 years before it can be used with confidence.
Because of the complexity, these officials stated that ASCI
software may take longer to develop, and a key program goal is to
reduce the development time to the 5-year historic average. A
report by the President's Information Technology Advisory
Committee described current software development as "among the
most complex of human-engineered structures."3 The report noted
that the nation's ability to construct needed software systems and
to analyze and predict the performance of these systems is
painfully inadequate. Part of DOE's risk mitigation strategy in
ASCI software development includes competing software efforts at
Los Alamos and Livermore. In addition, to leverage their efforts
in software development, the laboratories have contracted with
several universities (through the ASCI Academic Strategic
Alliances Program) to conduct research in areas of high-
performance computing and physical science. The technical
challenges inherent in the development of ASCI-related software
are due in part to the complexity of the needed software. Program
officials describe the ASCI software development effort as a
hierarchy of development. At the lower level of the hierarchy are
software modeling efforts that include (1) modeling the
engineering features and the materials used in weapons systems,
(2) modeling the physics phenomena associated with weapons
systems, and (3) developing computational problem-solving
techniques that will allow calculations to take place at
increasingly higher processing speeds. At the top of the software
development hierarchy are the integrated software applications
that will eventually (as larger ASCI computers become available)
incorporate all the lower-level modeling efforts and computational
techniques into a single system. This integrated software is
expected to provide the ability to simulate weapons performance
ranging from individual components to full weapons systems,
including performance in 3Interim Report to the President. The
President's Information Technology Advisory Committee, Aug. 1998.
Page 39 GAO/RCED-99-195
Strategic Computing Initiative Chapter 4 Technical Challenges Are
Present in All Aspects of the ASCI Program hostile environments.
This software is also expected to provide the capability of
predicting the performance of weapons components and full weapons
systems in analyses of design, aging effects, and accident
scenarios. Program officials also expect that ASCI software will
be used to design efficient and environmentally acceptable
manufacturing processes. The report by the ASCI Blue Ribbon Panel
noted several concerns about the modeling efforts of the lower-
level software. Among the concerns raised was one about the
materials science area. The panel said that this area warrants
further review because it forms the basis of so much of the work
and involves issues of great complexity, some of which are not
understood at a fundamental level anywhere in the materials
science community. The reviewers also noted that the presentations
they heard did not indicate that the integration of experimental
data is tightly coupled to software development. The report notes
that a robust experimental program that is closely tied to
simulations is crucial to assess the adequacy of the scientific
input and to test the software. Another technical challenge in the
area of software development is the development and consistent use
of software quality assurance. In general, software quality
assurance involves reviewing and auditing software products and
activities to verify that they comply with the applicable
procedures and standards. An April 1998 review of the computing
facilities at Los Alamos concluded that software quality assurance
has not been addressed sufficiently in the ASCI program.4 The
report noted that the situation is exacerbated because of the
current shortage of expertise in the area of software quality
assurance. ASCI program officials have explained that efforts to
ensure software quality are part of their new software
verification and validation effort and that they have efforts
under way at each of the laboratories to address this issue. For
example, Livermore has established the Software Technology Center,
and a software quality assurance team has been formed using staff
from each of the three laboratories. This team recently conducted
a survey at each of the laboratories to develop an initial
inventory of software quality practices being used at the
laboratories. The team is preparing detailed reports on its survey
findings that will be provided to each of the laboratories.
Significant Technological The major technical risks in the
infrastructure area are associated with Improvements Needed in
(1) extracting optimum simulation performance from tens of
thousands of Some Infrastructure Areas processors and (2)
moving, storing, and displaying large, complex results 4Los Alamos
CIC Division External Review Committee Report for the April 1998
Review. Page 40 GAO/RCED-99-
195 Strategic Computing Initiative Chapter 4 Technical Challenges
Are Present in All Aspects of the ASCI Program for interpretation
by weapons designers. According to program officials, risk
mitigation strategies employed in this area include ongoing
research at the laboratories and through the Academic Strategic
Alliances Program and contracts with third-party partnerships to
develop a variety of advanced techniques and technologies. In
addition, the ASCI program has sponsored workshops with
universities, other government agencies, and industry to engage
them in a common approach to meeting these challenges. Significant
technological improvements are needed in several critical
infrastructure areas, including visualization and storage
technology and the technology that connects computers to other
components. For example, visualization is an essential analysis
tool for understanding the volumes of data that will be produced
by ASCI software. The laboratories have recently unveiled new data
visualization centers, but according to ASCI planning documents,
the defined user needs exceed industry visualization hardware
capabilities by 15 to 60 times. Achieving the needed improvements
is also challenging because there is currently only one vendor in
this area. Significant improvements will also be needed for data
storage technology and connection technology for the 30-trillion
operations per second system. Contracts have been signed and work
is under way with several vendors to address these issues. The
ASCI Blue Ribbon Panel also cited visualization technology as an
area of concern, particularly the level of involvement by weapons
designers in the planning of visualization capabilities and
facilities. The report also noted that the panel was concerned
about the accelerated pace of investment in visualization
technology because the basis for visualization needs was unclear.
Integration of Integrating the data from laboratory
experiments conducted outside the Scientific Data ASCI program
into ASCI software development efforts has been noted as another
important technical challenge. DOE's Green Book notes that to
achieve the modeling and simulation goals of the ASCI program, new
data will be needed from laboratory experiments to help verify the
accuracy of the ASCI software. These experiments are designed to
learn more about the physical processes that occur to a weapon
under normal and abnormal conditions. The Green Book also notes
that the schedule for future experiments and the computational
needs of the ASCI program must be closely and carefully
coordinated to ensure that the experimental data are useful to the
ASCI program. The facility plan for one of the stockpile Page 41
GAO/RCED-99-195 Strategic Computing Initiative Chapter 4 Technical
Challenges Are Present in All Aspects of the ASCI Program
stewardship program's experimental facilities illustrates the
connection between data from experiments and the ASCI program. The
plan for the National Ignition Facility (NIF) explains that much
of the program is designed to gather fundamental weapons-relevant
data and use these data to enhance and refine nuclear weapons
simulations.5 Recent reviews have commented on the issue of
physics data in the ASCI program. The ASCI Blue Ribbon Panel noted
that a robust experimental program, which is closely tied to ASCI
simulations, is crucial to assessing the adequacy of the
scientific input and to testing the accuracy of the software. The
panel reported that the presentations it was given by laboratory
officials did not indicate that the experimental data were tightly
integrated with software development. Although the panel did not
review the experimental program in depth, it recommended that
additional funding should be made available to produce the
physical data required to support ASCI software efforts. Two June
1998 reports on the software development efforts at Los Alamos
expressed concern that the issue of weapons physics had not
received the attention it deserved.6 One report explained that the
review panel wanted to learn more about the role of experiments in
validating the accuracy of ASCI software and that certain
laboratory staff should have a more prominent role in the
selection of experiments conducted at DOE facilities. Program
officials acknowledge the need for closer integration between
laboratory experiments and the ASCI program. They explained that
they have taken actions such as a reorganizing DOE management and
creating a formal software validation program that requires data
from experiments. In addition, officials explained that the 1998
review of the primary burn code milestone reported on the
effective integration of experiments with the ASCI program.
According to DOE officials, the reorganization of offices within
DOE's Office of Defense Programs, now in progress, will allow ASCI
program officials to set both the ASCI program schedule and the
schedule for needed laboratory experiments. Program officials also
explained that the ASCI "verification and validation" effort, new
for fiscal year 1999, would provide the framework for aligning the
needs of the ASCI program with the schedule for laboratory
experiments. The validation effort includes the use of laboratory
experiments to ensure that the simulations are consistent with
observed behavior. The June 1998 review of the 5Facility Use Plan
of the National Ignition Facility, Edition 1, April 1997. NIF is
planned as a multiple-beam, high-power laser system with the goal
of attaining the ignition of thermonuclear fuel in the laboratory.
6"Report of the X-Division Review Committee," May 18-20, 1998, and
"Organizational Self-Assessment for the Applied Theoretical and
Computational Physics Division," June 26, 1998. Page 42
GAO/RCED-99-195 Strategic Computing Initiative Chapter 4 Technical
Challenges Are Present in All Aspects of the ASCI Program primary
burn code milestone found ample evidence of integration among ASCI
software development, experiments, and the use of existing
information from previous nuclear tests. The review noted that
data from experiments are currently being incorporated into ASCI
software. Technical Expertise Recruiting and retaining
qualified personnel is a continuing area of risk, according to
ASCI program officials and outside program reviews. DOE and
laboratory officials have explained that they have efforts ongoing
in many areas to improve the recruiting of staff with the required
expertise. These efforts include exemptions from salary ceilings
and the ASCI Academic Strategic Alliance Program. The Chiles
Commission noted that there is no certainty that DOE will succeed
in maintaining future nuclear weapons expertise.7 ASCI program
officials are concerned about the availability of staff with the
necessary expertise. For example, Los Alamos officials noted that
several milestones during fiscal year 1998 were delayed because of
a shortage of staff with the needed expertise in software
development. They explained that such personnel are difficult to
recruit and that, once recruited, they need time to develop the
necessary weapons-related expertise. Furthermore, these officials
said that they have received a waiver from DOE to offer more
competitive salaries to recruit qualified staff. Livermore
officials explained that personnel with computer science and math
skills are in high demand, which makes it difficult to recruit
them into the ASCI program. Their risk mitigation strategy
includes using the Academic Strategic Alliance Program to attract
qualified students, offering competitive salaries, and using the
unique research and development aspects of the ASCI program to
attract potential candidates. The Chiles Commission report on
maintaining nuclear weapons expertise noted that it was difficult
to conclude that DOE will succeed in maintaining future nuclear
weapons expertise. Although the report found a great deal that is
healthy in the nuclear weapons complex, with many trends moving in
the right direction, it also found other matters that are
disturbing. These other matters included the aging workforce, the
tight market for talent, and the lack of a long-term hiring plan.
The report also concluded that steps need to be taken now to
ensure that the upcoming generation of designers is recruited and
trained while the more experienced designers remain at the
laboratory or are available through retiree programs. The 7"Report
of the Commission on Maintaining United States Nuclear Weapons
Expertise." Report to the Congress and the Secretary of Energy,
March 1, 1999. Page 43
GAO/RCED-99-195 Strategic Computing Initiative Chapter 4 Technical
Challenges Are Present in All Aspects of the ASCI Program report
explained that more than 60 percent of the nuclear weapons
designers at Los Alamos and Livermore are between 50 to 65 years
old and that 5 or more years of experience working with
experienced designers is required to develop a fully capable,
independent designer. The ASCI Blue Ribbon Panel report concluded
that the training of the next generation of technical staff is the
single largest problem facing not only the ASCI program but also
the entire weapons program. The report commended the ASCI program
for its Academic Strategic Alliances Program to attract high-
quality, technically trained personnel but also stated that its
fellowships and summer internships must be made more appealing and
competitive. The report also noted that ASCI computers could be
used to attract students into the field of computational science
and that, although this is being done in the Academic Strategic
Alliance Program, it could be done more broadly. In addition, the
report questioned whether the laboratories are able to compete in
the market for the best personnel. The report concluded that ties
to the universities would be vital to convince the best students
to make a career at the laboratories. Program officials
acknowledge the problem of recruiting and retaining staff
expertise. They cite efforts such as a DOE fellowship program as a
means of attracting needed expertise. Under the fellowship
program, ASCI will support eight students, a number that may
double in the next 2 years. Students must serve a "practicum" at
DOE or the laboratories as a condition of support. Program
officials hope that this experience will interest the students in
working at the laboratories when they complete their education.
Page 44 GAO/RCED-99-195
Strategic Computing Initiative Page 45 GAO/RCED-99-195
Strategic Computing Initiative Appendix I Program- and Laboratory-
Level Software Milestones From DOE Planning Documents, Fiscal
Years 1996-2004 Program-level documents
Laboratory-level documents Program Most critical
Consolidated Nuclear Nonnuclear plan
milestones Green Booka roadmapb roadmapc
Roadmapd Fiscal year Milestone description (Sept. 96)
(July 98) (Apr. 98) (Feb. 98) (Mar. 99)
(Mar. 99) 96 Microaging
X X 97 Microaging X
X X High-fidelity safety X
X X X calculation Neutron generator
standoff
X 3-D casting microstructure
X X 98 3-D casting microstructure X
X X 3-D nuclear safety
X X simulation/
Prototype nuclear safety simulatione Neutron generator hostile
X X
certification/ Neutron generator radiation hardness mechanicale
Prototype 3-D physics
X Prototype 3-D
X hydrodynamics/ radiation-hydrodynamics System/composition
X thermal B61 penetrator
X Macro/micro aging
X 99 Macro/micro aging X
X X Prototype 3-D physics/ X
X X X X Prototype 3-D
primary simulations/ 3-D burn code/ 3-D primary burn codes/ 3-D
primary burn prototypee 3-D forging/welding X
X X X microstructure Crash/fire safety
X X X 2-D
deterministic radiation
X transport Abnormal environment
X thermal assessment Neutron generator
X performance code (continued) Page 46
GAO/RCED-99-195 Strategic Computing Initiative Appendix I Program-
and Laboratory-Level Software Milestones From DOE Planning
Documents, Fiscal Years 1996-2004 Program-level documents
Laboratory-level documents Program Most critical
Consolidated Nuclear Nonnuclear plan
milestones Green Booka roadmapb roadmapc
Roadmapd Fiscal year Milestone description (Sept.
96) (July 98) (Apr. 98) (Feb. 98) (Mar.
99) (Mar. 99) Full system (Salinas)
X 00 Full physics, full system X
X prototype Hostile environment X
electronics certification Abnormal environment
X thermal assessment Prototype 3-D secondary
X X simulations/ 3-D
secondary burn prototypee Pit casting and
X manufacturing code 3-D prototype radiation
X flow simulation Full system microaging
X simulation Parachute
X Reentry vehicle
X aerodynamics B61 laydown
X Electrical circuit simulation
X capability Component deterioration
X model 01 Component deterioration X
X X model Initial operating code/ X
X X X Prototype 3-D coupled simulation/
3-D secondary burn codee Stockpile-to-target-
X sequence certification demonstration System composition burn
X 3-D prototype full system
X coupled simulation 02 Full physics burn code
X prototype Full system radiation
X hardness & hostile (continued) Page 47
GAO/RCED-99-195 Strategic Computing Initiative Appendix I Program-
and Laboratory-Level Software Milestones From DOE Planning
Documents, Fiscal Years 1996-2004 Program-level documents
Laboratory-level documents Program Most critical
Consolidated Nuclear Nonnuclear plan
milestones Green Booka roadmapb roadmapc
Roadmapd Fiscal year Milestone description (Sept.
96) (July 98) (Apr. 98) (Feb. 98)
(Mar. 99) (Mar. 99) Abnormal
X stockpile-to-target- sequence Burn code with aging
X 03 Integrated full physics
X burn code prototype 3-D electrical device
X physics Abnormal
X stockpile-to-target- sequence with aging Normal stockpile-to-
target-
X sequence 04 Normal stockpile-to-target-
X sequence with aging Complete physics full
X X X system
prototype/ Initial full system simulation code/ 3-D high fidelity
physics full system initial capabilitye aThe full title of this
document is Stockpile Stewardship Plan - 2nd Annual Update. bThe
full title of this document is Consolidated Applications Roadmap.
cThe full title of this document is Consolidated Nuclear Component
Applications Roadmap. dThe full title of this document is Non-
Nuclear Codes Roadmap. eThis represents a single milestone that is
titled differently in multiple sources. Sources: ASCI Program Plan
(1996), Stockpile Stewardship Plan (1998), and other information
provided by the Department of Energy, and the Los Alamos,
Livermore, and Sandia National Laboratories. Page 48
GAO/RCED-99-195 Strategic Computing Initiative Appendix II
Comments From the Department of Energy Note: GAO comments
supplementing those in the report text appear at the end of this
appendix. Page 49 GAO/RCED-99-195 Strategic Computing
Initiative Appendix II Comments From the Department of Energy Page
50 GAO/RCED-99-195 Strategic
Computing Initiative Appendix II Comments From the Department of
Energy Now on p. 6. See comment 1. Page 51
GAO/RCED-99-195 Strategic Computing Initiative Appendix II
Comments From the Department of Energy Page 52
GAO/RCED-99-195 Strategic Computing Initiative Appendix II
Comments From the Department of Energy Now on p. 6. See comment 2.
Now on p. 19. See comment 3. Page 53
GAO/RCED-99-195 Strategic Computing Initiative Appendix II
Comments From the Department of Energy Now on p. 19. See comment
4. Now on p. 20. See comment 5. Now on p. 21. See comment 6. Page
54 GAO/RCED-99-195 Strategic
Computing Initiative Appendix II Comments From the Department of
Energy Now on p. 22. See comment 7. Page 55
GAO/RCED-99-195 Strategic Computing Initiative Appendix II
Comments From the Department of Energy Now on p. 22. See comment
8. Now on p. 23. See comment 9. Page 56
GAO/RCED-99-195 Strategic Computing Initiative Appendix II
Comments From the Department of Energy Now on pp. 24-25 See
comment 10. Now on p. 25. See comment 11. Page 57
GAO/RCED-99-195 Strategic Computing Initiative Appendix II
Comments From the Department of Energy Now on p. 25. See comment
12. Now on p. 26. See comment 13. Now on p. 27. See comment 14.
Page 58 GAO/RCED-99-195
Strategic Computing Initiative Appendix II Comments From the
Department of Energy Now on p. 27. See comment 15. Page 59
GAO/RCED-99-195 Strategic Computing Initiative Appendix II
Comments From the Department of Energy Now on p. 27. See comment
16. Now on p. 27. See comment 17. Page 60
GAO/RCED-99-195 Strategic Computing Initiative Appendix II
Comments From the Department of Energy Now on p. 28. See comment
18. Now on p. 28. See comment 12. Now on p. 28. See comment 12.
Page 61 GAO/RCED-99-195
Strategic Computing Initiative Appendix II Comments From the
Department of Energy Now on p. 28. See comment 12. Now on p. 28.
See comment 19. Now on p. 28. See comment 20. Page 62
GAO/RCED-99-195 Strategic Computing Initiative Appendix II
Comments From the Department of Energy Now on p. 31. See comment
14. Now on p. 35. See comment 19. Page 63
GAO/RCED-99-195 Strategic Computing Initiative Appendix II
Comments From the Department of Energy Now on p. 35. See comment
21. Page 64 GAO/RCED-99-195
Strategic Computing Initiative Appendix II Comments From the
Department of Energy Now on p. 38. See comment 22. Now on p. 38.
See comment 23. Now on p. 40. See comment 24. Page 65
GAO/RCED-99-195 Strategic Computing Initiative Appendix II
Comments From the Department of Energy Now on p. 40. See comment
25. Page 66 GAO/RCED-99-195
Strategic Computing Initiative Appendix II Comments From the
Department of Energy Now on p. 41. See comment 14. Page 67
GAO/RCED-99-195 Strategic Computing Initiative Appendix II
Comments From the Department of Energy The following are GAO's
comments on the Department of Energy's letter, dated June 7, 1999.
GAO's Comments 1. DOE's comments relating to our
recommendations and our responses are discussed in the executive
summary, and chapters 2 and 3, where the recommendations appear.
2. The report text was revised to note that the Chiles Commission
made this comment. The Commission was charged by the National
Defense Authorization Acts of 1997 and 1998 to address the issue
of maintaining nuclear weapons expertise. 3. We believe that the
evidence presented in our report clearly demonstrates that long-
and short-term planning efforts have been inconsistent and
incomplete. For example, during the first 3 years of the program,
annual implementation plans were prepared for some, but not all,
components for only some of those years. In fact, in commenting on
the draft of this report, DOE concurred with our recommendation
that comprehensive planning and progress tracking systems were
needed and cited improvements that they were making. 4. We agree
that lower-level milestones can change for the reasons cited by
DOE. However, the milestones discussed in this section of our
report are the long-term, high-level program milestones.
Furthermore, we believe that the examples cited in the report text
and those shown in the chart in appendix I clearly demonstrate the
many inconsistencies in DOE's strategic planning documents and the
need to better document the many changes that have taken place. 5.
The example referred to is just one of several cases in which DOE
planning documents inconsistently reported the target completion
date for a milestone. For example, the dates for the following
milestones were reported differently in various documents:
"Microaging," "3-D Casting Microstructure," "Macro/Micro Aging,"
"Prototype 3-D Physics," and "Component Deterioration Model". In
addition, DOE provided us with many versions of DOE planning
documents and briefings that contained similar inconsistencies.
Furthermore, in some of these documents, the dates were reported
in calendar years and in others in fiscal years. In still others,
program and laboratory officials were uncertain of whether the
dates were in calendar or fiscal years. Page 68
GAO/RCED-99-195 Strategic Computing Initiative Appendix II
Comments From the Department of Energy 6. The DOE response to our
referenced statement does not address the point of our statement.
We state that laboratory milestones are reported inconsistently in
the sources provided by DOE. The examples we use in our report
illustrate this fact. DOE contends that transcription and printing
errors caused labels and dates to change and that this problem
"explains virtually all the contradictory information" about
milestones detailed in the appendix to our report. We do not agree
that these types of errors explain most of the problems we found
in reviewing the documents provided to us. The point of our
statement was that there is inconsistency among milestones found
on the consolidated laboratory-level milestone chart and the two
laboratory-level milestone charts used to support the consolidated
chart. The examples we used show that milestones appear on the
consolidated chart that are not found on the laboratory-level
nuclear and nonnuclear milestone charts, which form the basis of
the consolidated chart. Correspondingly, there are milestones on
the laboratory-level nuclear and nonnuclear milestone charts that
are not found on the consolidated chart. DOE explains that these
charts were not meant to provide details about milestones and that
they are visual supplements to written documentation that provides
detailed information about the milestones. This statement is
surprising to us. One of the largest problems we had in evaluating
this program stemmed from the fact that there was almost no
detailed information about these milestones. As we state in our
report, the only milestone for which we found a detailed
description was for the primary burn code scheduled for completion
in 1999. Our review of implementation plans, strategic plans,
reports from principal investigators meetings, and other documents
did not reveal any of the written documentation DOE claims to
support the milestones found on these charts. 7. DOE and
laboratory officials told us that the linkages referred to here
are not obvious and that only those with technical expertise could
understand them. At our request, program officials tried for over
4 months to prepare documents showing the linkages between
projects and laboratory-level and higher-level milestones.
Subsequently, program and laboratory officials told us that
developing this information was a very useful exercise for them
because the linkages were not always evident. DOE's comments about
the principal investigators' meetings are not relevant to this
point. However, as noted in another part of the report, there are
few detailed reporting requirements for presentations at these
meetings. In fact, we found that many of the accomplishments
reported for individual projects Page 69
GAO/RCED-99-195 Strategic Computing Initiative Appendix II
Comments From the Department of Energy at the meetings were
generally not tied back to the annual milestones established for
those projects. In addition, there was no systematic assessment of
how the results on individual projects are contributing to meeting
either the laboratories' annual or program milestones. 8. We
understand that project-level milestones and program-level
milestones are not the same. However, we believe that DOE and the
laboratories should be able to demonstrate how individual projects
funded under the ASCI effort contribute to achieving annual
laboratory-level and overall program-level milestones. This is
essentially the "link" that we expected to see but did not find in
program plans and related documents. In fact, in some cases
program officials were unable to explain what that link was,
although they stated that the projects were needed for the overall
stockpile stewardship program. 9. DOE explained that they employ
multiple research efforts and risk mitigation strategies to
achieve program goals. We modified the report to clarify the use
of multiple research approaches in the area of infrastructure
development. We noted during our review, however, that these
research efforts and risk mitigation strategies were poorly
documented. DOE also explained the need to eventually select the
most promising of these research efforts. However, DOE's claim
that there are no set timetables for doing this seems inconsistent
with the need to maintain the accelerated pace of the program. As
stated in our report, program-planning documents should clearly
identify research strategies, critical paths, and decision points.
10. We have added wording to the body of the report to recognize
the completion of DOE's review of the nonnuclear mechanics codes.
However, as stated in the report, at this juncture, most program-
level milestones remain undefined. 11. As noted in our report,
there are few detailed reporting requirements for presentations at
these meetings. In fact, we found that the accomplishments
reported for individual projects at the meetings were generally
not tied to the annual milestones established for those projects.
In addition, there is no systematic assessment of how the results
on individual projects are contributing to meeting either the
laboratories' annual or program milestones. 12. We do not believe
that we misinterpreted the information provided to us.
Documentation provided to us by Sandia officials and the
statements Page 70 GAO/RCED-99-
195 Strategic Computing Initiative Appendix II Comments From the
Department of Energy they made during our meeting with them on
March 15, 1999, showed that the Sierra software development effort
was 6 months or more behind schedule and that this slippage was
affecting other software development efforts. Specifically, they
cited the Fuego software development effort, which supports the
"Abnormal Environment Thermal Assessment" milestone. They
acknowledged that if any milestone in fiscal year 1999 is missed,
it would most likely be this one because of the delays in Sierra.
Sandia officials also stated that they had to redirect the Fuego,
Coyote, Jas, and Pronto software development efforts until Sierra
is ready. Whether the Sierra effort is 6 months behind schedule or
whether the schedule itself has been slipped or, in DOE parlance,
"rebaselined" by 6 months to recognize this slippage is irrelevant
to the fact that the program schedule has been altered by delays
in the Sierra project and that those delays are affecting other
software efforts. 13. We believe that the statement in the report
is correct as stands. We agree that implementation plans have
improved over time. With respect to the quarterly reporting
requirement for fiscal year 1999, it was added halfway through the
fiscal year following our requests for information about progress
towards meeting program milestones. We believe that this new
quarterly reporting requirement is a positive improvement in the
program's progress tracking system. 14. DOE provided additional
information that did not require any response or changes to the
report. 15. The actual sustained speed achieved on the sPPM code
was 800 billion operations per second or one-half that reported on
the Linpack code. While we recognize that performance on the sPPM
was a "best effort" in the contract, we believe that it is
misleading to repeatedly cite the higher performance on the
Linpack code without recognizing the fact that the computer has
not yet passed its contractually required acceptance test. With
respect to our statement that the Linpack code was substantially
easier than the sPPM code, our basis was comments made to us by
program officials at a meeting on September 3, 1998. At that
meeting, program officials stated that they were using the sPPM
code instead of the Linpack scale because the Linpack is a "toy
program" not useful at all for measuring the capabilities needed
by the weapons program. For example, Linpack is not an industry
standard, contains only a few lines of code, and does not measure
important capabilities such as the use of input/output devices. On
the other hand, sPPM, using hydrodynamic calculations, fully tests
the platform regarding communications devices, as well as how Page
71 GAO/RCED-99-195 Strategic
Computing Initiative Appendix II Comments From the Department of
Energy efficiently the software scales to 6144 processors, and the
ability to incorporate I/O devices. 16. Wording was added to the
report to clarify that, according to the ASCI Program Plan, a 10-
fold increase was expected from improving software to take
advantage of parallel processing techniques. 17. We revised the
report text to clarify that we found little documentation of
software development progress as measured against established
milestones. 18. As pointed out in comment 12, the Sierra project
is 6 months or more behind schedule, is affecting other software
development efforts, and could potentially result in a milestone
being missed during fiscal year 1999. Simply changing the baseline
used to measure progress does not change these facts. 19. We were
provided with this information at a meeting with senior program
officials at Sandia on March 15, 1999. Also, see comment 12. 20.
DOE's comment is discussed in the agency comments section of
chapter 21. The National Academy of Sciences information, as
referenced by DOE, does not address the tracking of costs on
research programs and thus does not support DOE's management
approach as DOE contends. 22. The report does recognize that the
ASCI Red computer at Sandia is capable of operating at a
theoretical peak speed of 1.8 trillion operations per second.
While this is a significant accomplishment, DOE fails to mention
that the ASCI Red computer was developed by a different vendor and
uses an architecture that is different from the architecture in
the computer being developed at Los Alamos. 23. Although things
may have changed since 1998 as DOE states, our point is still
valid. Operating each new system constructed (including the new 3
trillion operations per second system at Los Alamos) as a fully
integrated system poses challenges for the ASCI program. For
example, ASCI's fiscal year 1999 implementation plan states, "The
most critical issues affecting the successful implementation of
the [Blue Mountain] system include hardware reliability and
stability." Recognizing this fact, the contract statement of work
for the Los Alamos computer included requirements to measure the
stability of the system. However, as of March 1999, the Page 72
GAO/RCED-99-195 Strategic Computing Initiative Appendix II
Comments From the Department of Energy computer had not met these
requirements, which clearly demonstrates this point. 24. We have
revised the text to more accurately cite the Blue Ribbon Panel
report. 25. We have revised the report text to more fully describe
this survey effort. (141213) Page 73
GAO/RCED-99-195 Strategic Computing Initiative Ordering
Information The first copy of each GAO report and testimony is
free. Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order made out
to the Superintendent of Documents, when necessary. VISA and
MasterCard credit cards are accepted, also. Orders for 100 or more
copies to be mailed to a single address are discounted 25 percent.
Orders by mail: U.S. General Accounting Office P.O. Box 37050
Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner
of 4th and G Sts. NW) U.S. General Accounting Office Washington,
DC Orders may also be placed by calling (202) 512-6000 or by using
fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO
issues a list of newly available reports and testimony. To
receive facsimile copies of the daily list or any list from the
past 30 days, please call (202) 512-6000 using a touchtone phone.
A recorded menu will provide information on how to obtain these
lists. For information on how to access GAO reports on the
INTERNET, send an e-mail message with "info" in the body to:
info@www.gao.gov or visit GAO's World Wide Web Home Page at:
http://www.gao.gov PRINTED ON RECYCLED PAPER United States
General Accounting Office Bulk Rate Washington, D.C.
20548-0001 Postage & Fees Paid GAO Permit No. G100 Official
Business Penalty for Private Use $300 Address Correction Requested
*** End of document. ***