Nuclear Weapons: DOE Needs to Improve Oversight of the $5 Billion
Strategic Computing Initiative (Chapter Report, 06/28/1999,
GAO/RCED-99-195).

Historically, the United States has detonated nuclear weapons to
determine their safety and reliability. Since 1992, however, there has
been a moratorium on testing. As a substitute for actual testing, the
Department of Energy (DOE) developed the Stockpile Stewardship and
Management Program in 1995. The program uses various methods, including
computer modeling, to ensure weapon safety and reliability. The
strategic computing initiative seeks to develop advanced computer models
that will simulate nuclear explosions in three dimensions with higher
resolution than previous models and with a more complete treatment of
the underlying basic physics. The initiative is also developing the
world's largest and fastest computers. GAO found that weak management
and information processes have hampered oversight of this $5-billion
initiative. For example, the initiative's strategic plan is out of date,
annual plans have been prepared only sporadically, and milestones are
not well defined. Moreover, the lack of information has made it
difficult to determine which of the hundreds of milestones have been
met, which are behind schedule, or even which are still relevant.
Program cost estimates have risen substantially. Finally, significant
technical challenges will have to be overcome before a computer
simulation can determine, in the absence of nuclear testing, whether a
weapon system will work as expected.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  RCED-99-195
     TITLE:  Nuclear Weapons: DOE Needs to Improve Oversight of the $5
	     Billion Strategic Computing Initiative
      DATE:  06/28/1999
   SUBJECT:  Information resources management
	     Computer software
	     Computer modeling
	     Future budget projections
	     Management information systems
	     Nuclear weapons testing
	     ADP procurement
	     Cost analysis
	     Weapons research and development
	     Strategic information systems planning
IDENTIFIER:  DOE Accelerated Strategic Computing Initiative
	     DOE Stockpile Stewardship and Management Program
	     Comprehensive Test Ban Treaty of 1996
	     DOE Academic Strategic Alliances Program
	     DOE Financial Information Variance Reporting System

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  This text was extracted from a PDF file.        **
** Delineations within the text indicating chapter titles,      **
** headings, and bullets have not been preserved, and in some   **
** cases heading text has been incorrectly merged into          **
** body text in the adjacent column.  Graphic images have       **
** not been reproduced, but figure captions are included.       **
** Tables are included, but column deliniations have not been   **
** preserved.                                                   **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************

    United States General Accounting Office GAO                Report
    to the Chairman, Subcommittee on Military Procurement, House
    Committee on Armed Services June 1999          NUCLEAR WEAPONS DOE
    Needs to Improve Oversight of the $5 Billion Strategic Computing
    Initiative GAO/RCED-99-195 GAO United States General Accounting
    Office Washington, D.C. 20548 Resources, Community, and Economic
    Development Division B-282727 June 28, 1999 The Honorable Duncan
    Hunter Chairman, Subcommittee on Military Procurement Committee on
    Armed Services House of Representatives Dear Mr. Chairman: As
    requested, this report examines the Department of Energy's (DOE)
    Accelerated Strategic Computing Initiative. Specifically, it
    discusses the management of the program, including (1) whether the
    program is meeting its key milestones and whether hardware and
    software developments are adequate to date, (2) whether the
    program is within its projected budget, and (3) what key technical
    risks the program faces. As arranged with your office, we plan to
    distribute copies of this report to the appropriate congressional
    committees; the Honorable Bill Richardson, Secretary of Energy;
    the Honorable Jacob Lew, Director, Office of Management and
    Budget; and other interested parties. We will also make copies
    available to others on request. If you have any questions
    regarding this report, please contact me or Edward Zadjura at
    (202) 512-3841. Key contributors to this assignment were Linda
    Chu, Daniel Feehan, and Anne McCaffrey. Sincerely yours, Susan D.
    Kladiva, Associate Director, Energy, Resources, and Science Issues
    Executive Summary Purpose       Historically, the United States
    detonated nuclear weapons as the primary method of validating
    designs and certifying the weapons as safe and reliable. Since
    September 1992, there has been a moratorium on testing. To ensure
    the continued safety and reliability of nuclear weapons, the
    Department of Energy (DOE), which is responsible for designing and
    building nuclear weapons, developed the 15-year Stockpile
    Stewardship and Management Program in 1995 as a substitute for
    actual testing. The stockpile stewardship program employs a
    variety of means to ensure weapon safety and reliability,
    including examining weapons, conducting laboratory experiments and
    tests, and conducting computer modeling and simulation. The
    computer modeling and simulation part of the program is known as
    the Accelerated Strategic Computing Initiative. The strategic
    computing initiative aims to develop advanced computer models that
    will simulate nuclear explosions in three dimensions with higher
    resolution than previous models and with a more complete treatment
    of the underlying basic physics. The initiative is also developing
    the world's largest and fastest computers, which may ultimately be
    able to calculate more than 100-trillion mathematical operations
    per second. The initiative is expected to cost about $5.2 billion
    for fiscal years 1996 through 2004. Concerned about the status of
    the strategic computing initiative, the Chairman, Subcommittee on
    Military Procurement, House Committee on Armed Services, requested
    that GAO review the management of the strategic computing
    initiative, including (1) whether the program is meeting its key
    milestones and whether its hardware and software developments are
    adequate to date; (2) whether the program is within its projected
    budget; and (3) what key technical risks the program faces.
    Background    Since the dawn of the nuclear era in 1945, the
    testing of nuclear weapons and state-of-the-art computing have
    been used together to ensure the performance, reliability, and
    safety of the weapons. Testing was the ultimate judge of whether a
    weapon worked and met its design requirements and provided data
    needed for computer models. Computers were used to perform the
    massive calculations needed to understand the basic physical
    processes that take place at the heart of a nuclear explosion and
    to interpret the results of nuclear experiments and tests, thus
    providing feedback in the process of designing, building, and
    testing nuclear weapons. The practical result of the Comprehensive
    Test Ban Treaty of 1996 is that existing nuclear weapons will be
    kept longer than planned because new weapon designs cannot be
    tested and certified as safe and reliable. Faced with these
    testing restrictions, DOE developed a Page 2
    GAO/RCED-99-195 Strategic Computing Initiative Executive Summary
    new approach to certifying the safety and reliability of weapons
    in the U.S. stockpile. The computer models and hardware, developed
    as part of the strategic computing initiative, will be used to
    identify potential stockpile problems by predicting the effects of
    aging and the need to replace components or even to retire weapons
    systems if they become unsafe or unreliable. The existing
    stockpile of weapons is aging, and many of the designers of those
    weapons have retired or are approaching retirement. For these
    reasons, DOE wants to have the computers and models available by
    2004 so that the existing cadre of experienced weapons designers
    will be available to help verify the results of the models.
    Results in Brief    Weak management and information processes
    hamper oversight of the strategic computing initiative. Although
    initiative managers report that many milestones have been met, the
    lack of comprehensive planning and progress tracking systems make
    assessment of the initiative's progress difficult and subjective.
    Currently, the initiative's strategic plan is out of date, annual
    plans have been prepared only sporadically, and milestones are not
    well defined. Furthermore, little information exists to track the
    initiative's progress or to compare its accomplishments with its
    milestones. Consequently, it is difficult to determine which of
    the hundreds of milestones have been met, which are behind
    schedule, or even which are still relevant, given changes in the
    initiative. Program cost estimates have increased substantially.
    In 1995, DOE estimated that costs for the first 5 years of the
    initiative (fiscal year 1996 through fiscal year 2001) would be
    $1.7 billion. By 1999, estimated costs for that same 5-year period
    increased to $2.9 billion. DOE currently estimates that the
    program will cost about $5.2 billion for fiscal years 1996 through
    2004.1 Some of the cost increases result from the shift to
    computer-based simulations, while some reflect weaknesses in DOE's
    cost estimation. Developing a computer simulation, or "virtual
    test" capability, that, in the absence of nuclear testing, can be
    used to determine whether a weapon system will perform as intended
    requires overcoming significant technical challenges. These
    challenges range from developing state-of-the-art hardware and
    software technologies, to integrating scientific data from weapons
    physics experiments, to recruiting and retaining staff with the
    needed technical expertise. 1The strategic computing initiative is
    a 15-year program, but because of the 5-year budget cycle, no cost
    estimates are available beyond fiscal year 2004. Page 3
    GAO/RCED-99-195 Strategic Computing Initiative Executive Summary
    This report makes recommendations directed at improving the
    oversight and management of the strategic computing program.
    Principal Findings A Comprehensive Planning    The strategic
    computing initiative's planning efforts have been and Tracking
    System Is      inconsistent and incomplete. Strategic planning
    documents have not been Needed to Assess Program    updated, and
    annual implementation plans were prepared inconsistently Progress
    or, in some cases, not at all. The long-term milestones presented
    in various plans are inconsistent, and no information exists to
    link annual activities to these milestones. Strategic plans also
    do not identify the multiple research strategies currently
    employed to meet many long-term milestones, nor do the plans
    include key decision points for managing these strategies.
    Performance criteria for most milestones have also not been
    defined. The efforts of DOE and laboratory managers to track the
    progress of the strategic computing initiative have been limited
    primarily to reporting annual accomplishments, without any
    systematic tracking of progress towards long-term milestones. As a
    result, it is not possible to determine whether annual milestones
    were achieved or to what extent annual efforts contribute to long-
    term milestones. In response to GAO's request for tracking
    information, program officials have decided to track and report
    the program's progress more systematically. The lack of a system
    for tracking progress, combined with the lack of defined
    performance criteria, make it difficult to assess whether the
    strategic computing initiative is proceeding on schedule and
    delivering the performance expected. However, it is possible to
    gain some limited insights through discussions with laboratory
    officials on individual projects or areas. For example, in the
    area of hardware development, most contract milestones to date
    relating to the delivery and installation of computers and related
    hardware have been met, although not all acceptance tests have
    been passed. DOE is not managing the strategic computing
    initiative as a strategic system. To be designated as a strategic
    system, under DOE criteria, a project must cost over $400 million,
    be an urgent national priority, be high-risk, have international
    implications, or be vital to national security. The purpose of
    designating strategic systems is to ensure informed, objective,
    and well-documented decisions for key events, such as changes Page
    4                               GAO/RCED-99-195 Strategic
    Computing Initiative Executive Summary to baseline cost or
    schedule and to ensure oversight at the highest departmental
    level. The strategic computing initiative meets all these
    criteria, has experienced delays in some areas, has had its
    projected costs increase, and depends, in some cases, on as-yet
    unknown technologies for success. These characteristics, coupled
    with demonstrated weaknesses in program management and oversight,
    make the strategic computing initiative a clear candidate for
    being designated as a strategic system. According to DOE, it has
    not designated the initiative as a strategic system because the
    program is already subject to high-level departmental oversight.
    However, as discussed above, GAO found serious weaknesses in the
    program's management and information processes that make it
    difficult to determine if the program is performing as expected.
    The Management and           DOE's cost estimates for the
    strategic computing initiative have increased Tracking of Costs
    Need to    substantially since 1995, when early budget projections
    were made. Costs Be Improved                  for fiscal years
    1996 through 2001 have increased from an original estimate of $1.7
    billion to the current $2.9 billion. DOE's fiscal year 2000 budget
    request for the strategic computing initiative, which totals $692
    million, is more than double the original fiscal year 2000
    estimate made in 1995. Some of the cost increases result from the
    shift from test-based experiments to computer-based simulations,
    while some increases are the result of weaknesses in DOE's cost
    estimation. Although DOE monitors month-by-month spending at the
    laboratories, it does not track costs for specific projects. As a
    result, DOE cannot determine which projects, if any, may be
    costing more or less than originally planned. GAO has previously
    noted DOE's difficulty in managing costs and schedules in large
    projects. Technical Challenges Are     The development of hardware
    and software technologies and the Present in All Aspects of
    necessary infrastructure to support these technologies are
    critical to the Strategic Computing      achieving the simulation
    and modeling goals of the strategic computing Initiative
    initiative. The program faces significant technical challenges in
    all of these areas. For example, increasingly large and complex
    computers using thousands of processors must be developed and made
    to operate as a single integrated system at speeds far beyond any
    achieved to date. The effort to develop software for simulation
    models on the scale needed to model nuclear weapons requires
    incorporating massive amounts of data, utilizing increasingly
    sophisticated problem-solving techniques, and using increasingly
    larger and faster computers. The President's Information Page 5
    GAO/RCED-99-195 Strategic Computing Initiative Executive Summary
    Technology Advisory Committee recently described software of this
    scale as being "among the most complex of human-engineered
    structures." Furthermore, developing the needed infrastructure,
    including data storage and visualization technology, will require
    significant technological improvements. The successful integration
    of data from laboratory experiments conducted outside the
    strategic computing initiative into software models being
    developed as part of the initiative has been noted by the DOE-
    chartered Blue Ribbon Panel as another important technical
    challenge. Data from these experiments and past nuclear tests are
    critical for demonstrating that the results of the software
    simulations are accurate. According to program officials, a recent
    reorganization of DOE offices and the creation of a formal
    software validation program aim to address this challenge.
    Finally, recruiting and retaining qualified personnel is a
    continuing area of risk, according to strategic computing
    initiative officials and outside program reviews such as the
    Chiles Commission.2 DOE and laboratory officials have efforts
    ongoing in many areas to improve the recruitment of staff with the
    required expertise. Nonetheless, as noted by the Chiles
    Commission, there is no certainty that DOE's efforts will succeed.
    Recommendations    DOE has chosen not to designate the strategic
    computing program as a strategic system. Given the strategic
    computing program's estimated cost of over $5 billion; the lack of
    a comprehensive planning, tracking, and reporting system; and the
    importance of the program to maintaining the stockpile of nuclear
    weapons; it is important that DOE improve its oversight and
    management of this program. Therefore, we recommend that the
    Secretary of Energy require the establishment of a comprehensive
    planning, progress tracking, and reporting system for the program
    and designate the program as a strategic system warranting
    oversight at the highest departmental level. Given the substantial
    increases in the cost estimates for the strategic computing
    initiative to date, DOE's weaknesses in estimating costs for the
    unprecedented scale of development efforts, and the lack of a
    cost-tracking process for the projects under the initiative, we
    also recommend that the Secretary of Energy require the strategic
    computing initiative to adopt systematic cost tracking procedures
    that will allow DOE managers to determine if specific projects are
    within budget. 2Report of the Commission on Maintaining United
    States Nuclear Weapons Expertise, Mar. 1, 1999. Page 6
    GAO/RCED-99-195 Strategic Computing Initiative Executive Summary
    Agency Comments       GAO provided DOE with a draft of this report
    for its review and comment. and Our Evaluation    DOE concurred
    with part but not all of the report's recommendations.
    Specifically, DOE concurred with the recommendation on the need to
    improve its oversight and management of this program and cited
    changes that it has made or is in the process of making. DOE did
    not concur with the recommendation to designate the program as a
    strategic system or on the need to adopt systematic cost tracking
    procedures. In agreeing with the recommendation to improve the
    oversight and management of the program, DOE cited several changes
    it was making. Specifically, DOE stated that it would soon issue
    an updated Program Plan that will include detailed specifications
    for all of the critical program milestones. In addition, FY 2000
    Implementation Plans will be issued by September 30, 1999, that
    will include descriptions of all program elements and complete
    lists of all milestones. The Department also cited the creation of
    a quarterly progress tracking mechanism to track program
    milestones. However, in addition to tracking the program's
    progress against established calendar milestones, it is also
    necessary to establish specific technical criteria for what
    constitutes the successful completion of those milestones. Until
    DOE completes and publishes its revised Program Plan and FY 2000
    Implementation Plans, GAO cannot determine whether the Department
    has fully complied with this recommendation. DOE disagreed with
    the recommendation to designate this program as a strategic
    system, stating that to do so would duplicate the existing
    planning, progress tracking, and reporting system. GAO agrees that
    creating a duplicate tracking system that mirrors the requirements
    set out by DOE for strategic systems would not be worthwhile.
    However, as discussed in detail in this report, DOE has not shown
    that it has an adequate planning, progress tracking, and reporting
    system in place for the strategic computing initiative. While DOE
    is making some positive improvements in these areas, the changes
    are not yet fully in place, and their adequacy cannot be judged at
    this time. Furthermore, if the changes that DOE is making are
    adequate to meet the requirements for tracking and monitoring of a
    strategic system, then GAO cannot understand DOE's reluctance to
    designate this large and costly program as a strategic system. DOE
    stated that it has a review process that meets the intent of the
    Clinger-Cohen Act of 1996. However, GAO reported in July 1998 that
    the Department's process effectively excluded scientific computers
    like those being acquired through this program from DOE's normal
    review channels and places them Page 7
    GAO/RCED-99-195 Strategic Computing Initiative Executive Summary
    within the program offices3. GAO stated that all computers should
    be included as part of the normal DOE Clinger-Cohen review
    process. DOE also did not agree with the recommendation to adopt
    systematic cost-tracking procedures for the strategic computing
    initiative, noting that costs are tracked by budget and reporting
    codes in the Department's Financial Information System. DOE stated
    that these systems are extended down to individual projects with
    other funding and cost-monitoring tools that gather more detailed
    information. As an example, DOE cited a March 1999 analysis of
    selected projects that identified the commitments and cost status
    for specific procurements at the project level. GAO does not agree
    that DOE has an adequate level of tracking at the project level or
    that the changes it is making will rectify this problem. DOE's
    current system tracks costs only at the aggregate level and does
    not allow DOE managers to determine which projects at the
    laboratories are under or over budget. Furthermore, the "other
    funding and cost monitoring tools" that DOE uses do not allow the
    systematic tracking of project costs. DOE also stated that some
    budgeting flexibility is necessary to capitalize on changes within
    the high-computing industry. While some budgeting flexibility is
    necessary in a project of this size and complexity, GAO does not
    believe that this flexibility should preclude effective oversight
    of a multiyear program costing over $5 billion. DOE's written
    comments are included in appendix II, and GAO's responses are
    discussed in chapters 2 and 3 and in appendix II. 3Information
    Technology: Department of Energy Does Not Effectively Manage Its
    Supercomputers (GAO/RCED-98-208, Jul. 17, 1998). Page 8
    GAO/RCED-99-195 Strategic Computing Initiative Page 9
    GAO/RCED-99-195 Strategic Computing Initiative Contents Executive
    Summary
    2 Chapter 1
    12 Introduction                 The Evolving Role of Computing in
    Nuclear Weapons                              12 Why Such Large
    Computers Are Needed                                            14
    ASCI Program Overview
    15 Objectives, Scope, and Methodology
    17 Chapter 2
    19 A Comprehensive              Comprehensive Planning Is Needed
    19 Systematic Progress Tracking and Reporting Needed
    25 Planning and Tracking Program Progress Difficult to Assess
    26 System Is Needed to          DOE May Need to Manage ASCI as a
    Strategic System                              28 Assess Program
    Conclusions
    29 Recommendations
    29 Progress                     Agency Comments and Our Evaluation
    30 Chapter 3
    31 Management and               Cost Estimates Increased
    Substantially                                         31 Better
    Oversight of Costs Needed
    35 Tracking of Costs            Conclusions
    35 Need to Be Improved          Recommendation
    36 Agency Comments and Our Evaluation
    36 Chapter 4
    37 Technical Challenges         Technology Development
    37 Integration of Scientific Data
    41 Are Present in All           Technical Expertise
    43 Aspects of the ASCI Program Appendixes
    Appendix I: Program- and Laboratory-Level Software Milestones
    46 From DOE Planning Documents, Fiscal Years 1996-2004 Appendix
    II: Comments From the Department of Energy
    49 Table                        Table 1.1. Requested ASCI Funding
    by Program Component,                        16 Fiscal Year 1999
    Page 10                             GAO/RCED-99-195 Strategic
    Computing Initiative Contents Figures    Figure 3.1 Original and
    Current Cost Estimates for the ASCI                     32
    Program, Fiscal years 1996 Through 2004 Figure 3.2: Allocation of
    Estimated ASCI Costs by Major Program                 33 Areas,
    Fiscal Years 1996 Through 2004 Abbreviations 3-D          Three
    dimension ASCI         Accelerated Strategic Computing Initiative
    DARHT        Dual-Axis Radiograph Hydrodynamic Test Facility DOE
    Department of Energy GAO          General Accounting Office NIF
    National Ignition Facility Page 11
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 1
    Introduction Historically, the United States used actual nuclear
    detonations as the primary method of validating designs and
    certifying the weapons as safe and reliable. Since September 1992,
    there has been a moratorium on testing. To ensure the continued
    safety and reliability of nuclear weapons, the Department of
    Energy (DOE), which is responsible for designing and building
    nuclear weapons, developed the 15-year Stockpile Stewardship and
    Management Program in 1995 as a substitute for actual testing. The
    stockpile stewardship program employs a variety of means to ensure
    weapons' safety and reliability, including examining weapons,
    conducting laboratory experiments and tests, and conducting
    computer modeling and simulation. The computer modeling and
    simulation part of the program is known as the Accelerated
    Strategic Computing Initiative (ASCI). The ASCI program aims to
    replace actual testing with advanced computer models that will
    simulate nuclear detonations. This effort requires modeling in 3-
    dimensions (3-D), with higher resolution than previous models and
    with better treatment of the underlying physical processes that
    occur during an actual nuclear detonation. To run the models, DOE
    is developing, as part of the ASCI program, the largest and
    fastest computers, which may ultimately be able to perform 100
    trillion mathematical operations per second-10,000 times more
    powerful than those used to design the weapons originally. The
    ASCI program is expected to cost about $5.2 billion for fiscal
    years 1996 through 2004. The Evolving Role of    Computers have
    been used to design and build nuclear weapons almost Computing in
    Nuclear    from the dawn of the nuclear era. As early as 1945,
    designers began using the ENIAC-the world's first computer, built
    at the University of Weapons                 Pennsylvania with
    government support-to perform calculations on the viability of a
    hydrogen or thermonuclear bomb. A successor version, which was
    fully electronic1 -the MANIAC-was built at Princeton in 1949, and
    a duplicate was built at Los Alamos. From that time, computers,
    and later so-called supercomputers, would play an increasing role
    in the designing and building of the U.S. stockpile of nuclear
    weapons. Computer models were used to design weapons and to
    interpret data from actual nuclear weapons tests. Models and
    computers were also used to identify and evaluate problems in the
    nuclear weapons stockpile. In the end, however, the final arbiter
    of a weapon's safety and reliability was usually an actual test or
    series of tests. 1The ENIAC used vacuum tubes instead of gears to
    perform calculations but had to be programmed for each new problem
    by physically rearranging its circuit wires, which looked like
    old-fashioned telephone switchboard cords. Page 12
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 1
    Introduction Since the first nuclear weapon test, known as
    Trinity, on July 16, 1945, the United States has conducted over
    1,000 nuclear weapons tests. Testing was the principal method used
    to certify the safety and reliability of nuclear weapons. Testing
    was used to demonstrate that a particular weapon design actually
    worked and yielded the expected power and to prove the safety and
    reliability of components. For example, testing could be used to
    demonstrate that older components were still functioning properly
    after years of exposure to extremes of heat and cold and to
    radiation. In addition to periodically testing stockpiled weapons,
    the United States frequently developed new weapons to replace
    older weapons in the stockpile, thus ensuring the continued
    reliability and safety of its arsenal. In September 1992, the
    Congress imposed a 9-month moratorium on underground nuclear
    testing.2 This moratorium continued to be observed until September
    1996, when President Clinton signed the Comprehensive Test Ban
    Treaty.3 The test ban treaty has been interpreted by the
    Administration to mean that no underground testing is allowed that
    results in any nuclear yield-no matter how low. The practical
    result of the test ban treaty is that existing nuclear weapons
    will be kept longer than planned because new weapon designs cannot
    be tested and certified as safe and reliable. The longer life span
    of the existing stockpile of nuclear weapons increases the
    possibility that they will decline in either performance or safety
    because of age-related factors like extended exposure to heat,
    vibration, and radiation. Faced with these testing restrictions,
    DOE developed a new approach to certifying the safety and
    reliability of weapons in the U.S. stockpile. A 1994 "Nuclear
    Posture Review" charged DOE with maintaining the capability to
    design, fabricate, and certify new weapons, if that ever became
    necessary. DOE responded by developing the 15-year Stockpile
    Stewardship and Management Program in 1995. The program is
    intended to ensure the continued safety and reliability of
    existing nuclear weapons using a variety of means, including
    examining weapons to find possible problems, conducting
    experiments to predict problems, and deciding on the basis of the
    results of these efforts what, if anything, needs to be done to
    ensure the continued reliability and safety of the weapons.
    2Atmospheric testing was banned in 1963. 3Although the U.S. Senate
    has not yet ratified the treaty, a statutory extension of the 1992
    moratorium took effect on September 30, 1996, and continues
    "unless a foreign state conducts a nuclear test" after that date,
    in which case the moratorium is lifted. DOE continues to observe
    the testing moratorium. Page 13
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 1
    Introduction The ASCI component of the Stockpile Stewardship and
    Management Program was intended to provide the modeling and
    computers necessary to simulate in great detail the detonation of
    a nuclear weapon. Related experimental facilities like the
    National Ignition Facility (NIF) located at Lawrence Livermore
    National Laboratory, and the Dual-Axis Radiograph Hydrodynamic
    Test Facility (DARHT), located at Los Alamos National Laboratory,
    are intended to provide the data needed to address basic physics
    questions and to validate the accuracy of the ASCI computer
    models. With this change to a science-based rather than a physical
    test-based approach to addressing stockpile issues, the ASCI
    program has become a critical link in certifying the safety and
    reliability of nuclear weapons. The ASCI computer models and
    hardware will be used to identify potential stockpile problems by
    predicting the effects of aging and the need to replace components
    or even to retire weapons systems if they become unsafe or
    unreliable. In addition, the ASCI program will be used to design
    and certify needed replacement parts as well as the entire weapons
    system.4 The existing stockpile of weapons is aging, and many of
    the designers of those weapons have retired or are approaching
    retirement. For these reasons, DOE has decided that it is crucial
    to have the ASCI program available by fiscal year 2004, including
    the models and computers capable of performing 100-trillion
    operations per second. The intent is to have the remaining
    designers compare the output of the models against their actual
    experience with nuclear weapons tests as one means of validating
    the accuracy of computer models. Why Such Large    The current
    generation of nuclear weapons were designed on computers Computers
    Are     that were much smaller than those being developed for the
    ASCI program-several million or a few billion operations per
    second versus Needed            100-trillion operations per
    second. A logical question rises as to why such vastly larger
    computers are needed to ensure the safety and reliability of
    existing weapons compared with those computers that were needed to
    design and build these same weapons to the same safety and
    reliability standards. The current stockpile of nuclear weapons
    were designed and built using much less capable computers and far
    simpler models than those 4Many of the manufacturing processes and
    technologies that were used to build the current generation of
    nuclear weapons and the components that they contain no longer
    exist. As such, replacement components manufactured using new
    processes, technologies, or materials need to be tested, in some
    manner, and certified as to their performance and impact on the
    weapons performance. Page 14
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 1
    Introduction envisioned for the ASCI program. These less capable
    computers could have been used for several reasons, including (1)
    key components of the weapons were designed with a high level of
    symmetry so that a one- or two-dimensional view of the component
    would be fairly representative of the whole component, (2) weapons
    were designed without a need to model all of the underlying
    physics, (3) actual testing was used to resolve any uncertainties,
    and (4) weapons were routinely replaced by newer, tested weapons
    before they reached the end of their design life. With the loss of
    testing opportunities and the aging of the current stockpile, this
    approach is no longer feasible. Instead, DOE believes it is
    necessary to provide detailed visual 3-D simulations of nuclear
    weapons processes (that is, virtual testing capability). Virtual
    testing requires far more complex and detailed models and much
    greater computer capability to run these highly complex models in
    a reasonable period of time. For example, to run certain two-
    dimensional weapons calculations on a Cray YMP supercomputer (an
    old generation of supercomputer but the type in use when some of
    the existing weapons were designed) took up to 500 hours. By
    comparison, moving from a two-dimensional to a 3-D model without
    changing any other parts of the model results in a calculation
    that is 1,000 times larger. At the same time, better detailed
    physics calculations of what is happening at the time of the
    nuclear detonation could require a calculation that is another
    100,000 times larger. By extrapolating from these estimates, DOE
    concluded that running such a calculation in a reasonable amount
    of time (generally no more than several days for the largest
    calculations) would require computers capable of calculating at
    the rate of 100-trillion operations per second. Such machines were
    far beyond those commercially available when the ASCI program was
    started. Developing these increasingly powerful machines is one of
    the main goals of the ASCI program. Building the highly complex 3-
    D models is another. ASCI Program    The ASCI program is comprised
    of several components. As shown in table Overview        1.1, the
    key components are Applications (software development), Platforms
    (computers), and Infrastructure (peripheral technologies such as
    networks, storage, and visualization). The program also includes
    the Academic Strategic Alliances Program, which contracts with
    universities for computing and scientific research to complement
    ASCI efforts. In fiscal year 1999, the scope of the ASCI program
    expanded when three new components were created. Numerical
    Environment for Weapons Simulation will acquire the infrastructure
    hardware needed for data Page 15
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 1
    Introduction management and visualization. Distributed Distance
    Computing will provide the infrastructure needed to extend the
    advanced computing capabilities of the ASCI program to remote DOE
    weapons facilities. Verification and Validation is intended to
    verify that ASCI software is executing calculations as intended
    and to validate the accuracy of ASCI software results. Table 1.1
    shows funding for ASCI-related activities, including existing
    computing facilities, verification and validation, and other
    activities at each of the three laboratories to support ongoing
    stockpile stewardship program requirements. Table 1.1. Requested
    ASCI Funding by Program Component, Fiscal Year 1999     Dollars in
    millions ASCI program component
    Requested funding Applications (software)
    $152 Platforms (computers)
    70 Infrastructure Problem Solving Environments
    46 Numerical Environment for Weapons Simulation
    31 Distributed Distance Computing
    28 Verification and Validation
    13 Stockpile Computing
    156 Academic Strategic Alliances Program
    14 One Program/Three Labs (program coordination)
    6 Total
    $516 ASCI activities are carried out by DOE's three Defense
    Program laboratories-Los Alamos, Lawrence Livermore, and Sandia
    National Laboratories-with guidance from DOE's Office of Strategic
    Computing and Simulation under the Assistant Secretary for Defense
    Programs. For hardware development, the ASCI program intends to
    build on economically viable computer vendor technologies and
    thereby foster the health of the U.S. computing industry while
    also stimulating competition in this industry to adopt new
    technologies for advanced computing. ASCI computers of varying
    sizes will be built and housed at each of the laboratories, and
    their capabilities will be accessible to all three laboratories.
    Infrastructure-related hardware will be procured from available
    vendor technologies. For software development, the ASCI program is
    relying on coordinated efforts at the three laboratories,
    supplemented by university-based Page 16
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 1
    Introduction research when possible. At Los Alamos and Livermore,
    software development efforts focus on models that simulate the
    performance of the nuclear components of weapons systems. In
    contrast, software efforts at Sandia focus on models that simulate
    the performance of nonnuclear weapons components like the arming,
    firing, and guidance systems. Infrastructure-related software is
    being developed in a joint effort by the three laboratories. The
    schedules for hardware, software, and infrastructure development
    are interdependent. For example, ASCI software must be able to
    operate on a variety of increasingly large parallel computer
    systems. The development of such "scalable" software requires the
    availability of computers and peripheral technologies that are
    sufficiently advanced to test and develop the software. Major
    milestones for hardware, software, and infrastructure development
    have been established for the program to 2004. Executing the ASCI
    program and meeting these milestones with the involvement of three
    laboratories will require close integration among programs and
    across laboratories. Although the ASCI program's ultimate goal is
    to provide 3-D weapons simulation capabilities by 2004, the ASCI
    computers and software developed to date are already important
    tools for addressing DOE's high-priority stockpile needs. Today's
    ASCI computational capabilities, for example, are being used to
    help design scientific experiments and to support the revalidation
    and certification of certain weapons and/or their components in a
    simulation environment. Objectives, Scope,    As requested by the
    Chairman, Subcommittee on Military Procurement, and Methodology
    House Committee on Armed Services, we reviewed the management of
    the ASCI program, including (1) whether the program was meeting
    its key milestones and whether its hardware and software
    developments are adequate to date; (2) whether the program was
    within its projected budget; and (3) what key technical risks the
    program faces. The scope of this review encompassed all aspects of
    the ASCI program, which is conducted primarily by the Lawrence
    Livermore, Los Alamos, and Sandia National Laboratories. To
    determine whether the program was meeting its key milestones, we
    obtained and reviewed planning and tracking documents and
    interviewed ASCI program officials from the three weapons
    laboratories and from DOE's Office of Defense Programs. We visited
    each of the three laboratories and Page 17
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 1
    Introduction DOE to discuss the program's progress in meeting its
    key milestones and to obtain evidence to verify the statements
    made by program officials. In July 1998 and January 1999, we
    attended the semiannual ASCI "principal investigator" meetings to
    learn more about the program's progress. We also reviewed studies
    and reports that have assessed the status and progress of the ASCI
    program. To determine whether the program is within its projected
    budget, we examined cost and budget information provided to us by
    DOE and the three laboratories. We also reviewed information from
    DOE's Financial Information Variance Reporting System and
    contracts. Furthermore, we examined budget information that was
    included in DOE's budget request for fiscal years 1996 through
    2000. We spoke to program officials to determine how costs were
    estimated, why program costs have escalated, and how they review
    and manage laboratory costs. We did not independently verify the
    reliability of information contained in DOE's financial management
    system, which we used in this report, because it is the basis for
    DOE's financial statements, to which we have given an "unqualified
    opinion" in our audit of the federal government's financial
    statement. To identify the key technical risks facing the program,
    we obtained and reviewed program-planning documents and
    interviewed ASCI program managers from the three weapons
    laboratories and from DOE's Office of Defense Programs. We also
    reviewed studies and reports on the ASCI program and other
    materials related to high-performance computing. We conducted our
    review from July 1998 through June 1999 in accordance with
    generally accepted government auditing standards. Key contributors
    to this report were Linda Chu, Daniel Feehan, Anne McCaffrey, and
    Edward Zadjura. Page 18                            GAO/RCED-99-195
    Strategic Computing Initiative Chapter 2 A Comprehensive Planning
    and Tracking System Is Needed to Assess Program Progress Weak
    management and information processes hamper oversight of the
    strategic computing program and make assessing progress towards
    program milestones difficult and subjective. Although program
    managers report that many milestones have been met, the lack of
    comprehensive planning and progress tracking systems make an
    assessment of the program's short- and long-term progress
    difficult and subjective. Current planning efforts include a
    strategic plan that is out of date, annual plans that have been
    prepared sporadically, and milestones that are not well defined.
    Efforts to track the program's progress are not consistent, and no
    clear record exists of program accomplishments compared with
    milestones. Consequently, it is difficult to determine which of
    the hundreds of milestones have been met, which are behind
    schedule, or even which are still relevant, given changes in
    program priorities, and how progress on individual projects
    contributes to the program's overall goals. In response to our
    requests for information, ASCI program officials have begun to
    institute more systematic procedures for planning and for tracking
    and reporting program progress. Comprehensive                 The
    ASCI program's long- and short-term planning efforts thus far have
    Planning Is Needed            been inconsistent and incomplete.
    Strategic planning documents have not been updated since the
    program's inception. During the program's first 3 years, annual
    implementation plans were prepared inconsistently or, in some
    cases, not at all, resulting in an incomplete program baseline.
    The long-term ASCI milestones presented in various plans are
    inconsistent, and no information exists to link annual activities
    to these milestones. Strategic plans also do not identify the
    multiple research strategies currently employed to meet many long-
    term milestones, nor do the plans include decision points for
    managing these strategies. Performance criteria for most
    milestones have not been defined in the planning process.
    Strategic and Annual Plans    The plans used to manage the ASCI
    program to date have numerous Are Outdated,
    limitations. The ASCI Program Plan is the program's primary
    strategic plan. Inconsistent, and             Published in 1996,
    the plan included a list of long-term (program-level) Incomplete
    milestones for hardware, software, and infrastructure development.
    Although hardware and infrastructure milestones have not changed
    much, program officials have revised software milestones numerous
    times but have not published an updated strategic plan. Despite a
    proliferation of program planning documents showing software
    milestones, there is little consistency among these documents, and
    no clear record of when and why milestones were changed and which
    are the most current. Some of Page 19
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 2 A
    Comprehensive Planning and Tracking System Is Needed to Assess
    Program Progress the original milestones are intact or have
    changed little over time, others have changed considerably, and
    still others have been replaced with new milestones. In addition,
    the dates for certain milestones are inconsistent among documents.
    For example, the "Prototype 3-D Primary Simulations" milestone for
    fiscal year 1999 has been shown consistently, although the
    milestone name has changed slightly. By contrast, several
    milestones do not appear consistently in different documents. For
    example, the "abnormal environment thermal assessment" milestone
    is listed as occurring in fiscal year 1999 in one planning
    document and in fiscal year 2000 in another. (App. I lists ASCI
    program milestones as identified in various planning documents).
    This lack of consistency is also found in another strategic
    planning document-the Stockpile Stewardship Plan (the "Green
    Book"). The Green Book is the strategic plan for DOE's Office of
    Defense Programs and is supported by more detailed planning
    documents, including the ASCI Program Plan. The ASCI milestones
    presented in the Green Book, however, are not always consistent
    with those included in ASCI program documents. For example, the
    Green Book lists the "crash/fire safety" and the "full physics,
    full system prototype" as milestones, although they are not listed
    among the most critical milestones compiled by the program office.
    Conversely, the program office's most critical milestones list
    includes the "hostile environment electronics assessment" and the
    "abnormal environment thermal assessment" milestones, but the
    Green Book does not list either. Program officials explained that
    the Green Book is prepared at different times and for different
    purposes than ASCI planning documents. Nonetheless, ASCI
    milestones in DOE's Green Book and the ASCI Program Plan should be
    consistent. In another example of inconsistent long-term planning,
    the Simulation Development Roadmap has never been updated. This
    document was intended to ensure that the ASCI program's simulation
    and modeling activities would be clearly identified and that
    priorities would be set on the basis of stockpile needs and
    current capabilities. DOE intended to update this plan
    periodically as the program progressed. Neither DOE nor the
    laboratories, however, have developed this document beyond the
    initial planning level since 1996. No other equivalent source of
    information exists that provides a long-term, needs-based
    perspective on the ASCI program's modeling and simulation
    activities. DOE program officials said that there is no
    programmatic requirement to update the Simulation Development
    Roadmap and that some information about program needs is contained
    in annual plans. This statement contradicts language in the Page
    20                                  GAO/RCED-99-195 Strategic
    Computing Initiative Chapter 2 A Comprehensive Planning and
    Tracking System Is Needed to Assess Program Progress Stockpile
    Stewardship Green Book, which states that the Simulation
    Development Roadmap is to be used in conjunction with annual
    planning documents to define computing requirements. Furthermore,
    as discussed below, annual planning documents have not been
    consistently prepared. The three laboratories in this program have
    also developed software milestones (laboratory-level milestones)
    and published various compilations of these. A consolidated set of
    these milestones, known as the "Consolidated Roadmap" is supposed
    to show the software milestones for all three laboratories.
    Another listing, known as the "Nuclear Roadmap," shows combined
    software milestones for two laboratories-Los Alamos and Livermore-
    whose modeling efforts focus on the nuclear components of weapons
    systems. A third listing, known as the "Non-Nuclear Roadmap,"
    shows software milestones for Sandia Laboratories, whose modeling
    efforts focus on nonnuclear weapons components, such as those for
    arming, firing, and guidance. As with other ASCI program
    documents, these sources report milestones inconsistently, and
    laboratory-level milestones may or may not match program-level
    milestones. For example, the consolidated roadmap lists milestones
    such as "3-D forging/welding microstructure," "full physics burn
    code prototype," and "burn code with aging" that do not appear in
    either the nuclear or nonnuclear roadmaps. Conversely, the nuclear
    and nonnuclear roadmaps list milestones such as the "3-D nuclear
    safety simulation" and the "B61 penetrator" that do not appear in
    the consolidated roadmap. The laboratories' long-term planning
    efforts also include multiyear plans for some of the individual
    software projects that contribute to laboratory-level milestones,
    but such plans are not required, and their format has varied from
    laboratory to laboratory. At Sandia, for example, project plans
    spanning 5 years have been prepared for key software projects. At
    Los Alamos and Livermore, project plans also have been developed
    for key software projects, but their time frames are shorter and
    variable (that is, 2 or 3 years). Short-term planning, as
    represented by annual implementation plans, has also been
    inconsistent and incomplete. The purpose of these plans is to
    specify project tasks and milestones for the current year (annual
    milestones). However, during the first 3 years of the program,
    annual implementation plans were prepared by the three
    laboratories for some, but not all, components of the ASCI program
    and for only some of those years. DOE's failure to ensure
    comprehensive and consistent planning during those years has
    resulted in an incomplete program baseline for Page 21
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 2 A
    Comprehensive Planning and Tracking System Is Needed to Assess
    Program Progress fiscal years 1996 through 1998. In addition, the
    annual plans do not explain changes in the scope of the work or
    milestones that occur from year to year. Fiscal year 1999 was the
    first year that DOE required the laboratories to submit
    implementation plans for all the components of the program and
    consolidated these documents into a program-wide ASCI
    Implementation Plan. The Linkages Between     To date, DOE has not
    required that program documents show linkages Annual and Long-Term
    between annual milestones and long-term program- or laboratory-
    level Milestones Need to Be    milestones. With the exception of
    information provided in response to our Identified
    request, these linkages have never been documented. Without clear
    information to identify such links, it is impossible to determine
    how annual progress contributes to meeting those long-term
    milestones. Program officials maintain that such links do exist
    but that they are not explicit in program documents. They also
    acknowledged that technical expertise is needed to identify links
    between annual milestones and long-term milestones. In our
    discussions with ASCI laboratory staff, however, we found that
    such links do not always exist and sometimes could not be
    identified even by laboratory personnel. At Sandia, for example,
    laboratory officials identified ASCI software activities that are
    needed to meet stockpile requirements but that are not tied
    directly to program-level milestones. In another case, it was not
    apparent what laboratory activities contributed to meeting the
    "macro-micro aging" milestone. DOE and laboratory officials
    identified Sandia as the laboratory responsible for this
    milestone, planned for completion in early 1999. While Sandia
    officials identified some activities that they believed were
    relevant to meeting this milestone, they were unsure about whether
    Sandia's activities were all that was needed to meet this
    milestone. They said it was possible that ongoing projects at the
    other two laboratories contributed to meeting this milestone.
    Program and laboratory officials agreed that such links should be
    made more apparent and, in trying to respond to our request for
    information about these links, they attempted to identify and
    document linkages. Although complete information was not provided
    by all of the laboratories, the information received shows that
    annual milestones are not always directly linked to long-term
    milestones. DOE and laboratory managers also told us that the
    process of developing this information was helpful for tracking
    the progress of the program and that they plan to refine this
    process and update the information on a quarterly basis. Page 22
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 2 A
    Comprehensive Planning and Tracking System Is Needed to Assess
    Program Progress Plans Should Clearly           ASCI plans do not
    identify all of their multiple research strategies nor do Identify
    Research              they establish decision points for
    identifying which of these strategies are Strategies, Critical
    Paths,    critical for meeting key program milestones. Because of
    the complex and Decision Points            technological
    challenges involved in developing software and hardware to model
    nuclear weapons, the laboratories have undertaken multiple
    research strategies in an effort to mitigate risk and achieve
    their laboratory- and program-level milestones. According to
    laboratory officials, they will eventually have to reassess these
    strategies to see which are working and which are not and decide
    how they can focus their efforts to best achieve their milestones.
    As a result, individual projects could be enhanced, scaled back,
    or eliminated as "critical paths" are defined. In the area of
    infrastructure development, for example, one overall goal is to
    develop ways for scientists to examine massive amounts of weapons
    simulation data. To achieve this goal, the development effort has
    at least seven simultaneous lines of effort, including developing
    "common data formats" and "distributed file systems." Within those
    lines of effort, multiple research approaches are being pursued. A
    program official explained that all of these lines of effort are
    needed but that it is not clear at this time which efforts will be
    critical to meeting the overall goal. It is also not known when a
    choice among these options needs to be made. Another development
    effort is focused on specialized software that would help ASCI
    software developers understand and improve the performance of
    their weapons software programs, which ultimately will help to
    reduce the time needed to solve such problems. Currently, there is
    no such specialized software to solve such problems. Commercial
    vendors are developing such software, but only to a certain extent
    because there is limited demand for such products outside the ASCI
    program. As a result, while ASCI program officials are considering
    vendor products, the program is also funding development efforts
    at universities and collaborating with industry to develop the
    needed software. The program official explained that the software
    needed for the ASCI program may be available from at least one of
    these sources, but, again no decision points have been established
    for when a choice among these options needs to be made. According
    to this official, infrastructure research strategies are managed
    and decisions about them are made on a yearly basis by a team
    assembled from the three laboratories. In the area of software
    development, ASCI software developers at Los Alamos and Livermore
    laboratories use multiple research and risk mitigation strategies
    in developing their weapons simulation software. Los Alamos and
    Livermore have multiple software teams competing to develop Page
    23                                  GAO/RCED-99-195 Strategic
    Computing Initiative Chapter 2 A Comprehensive Planning and
    Tracking System Is Needed to Assess Program Progress weapons
    simulation software. Program officials explain that having several
    teams is advantageous because the arrangement allows the
    laboratories to explore different simulation approaches, cross-
    check/validate each other's work, and mitigate the risk of
    selecting one approach before all approaches are explored
    adequately. These different research and risk mitigation efforts,
    however, are not identified clearly in planning documents. In
    addition, although laboratory officials acknowledge that they will
    need to eventually scale back or restructure their software
    efforts, no decision points have been established for doing this.
    Plans Should Define         Measurable performance criteria for
    most ASCI milestones have not been Performance Criteria for
    defined. Laboratory officials were, in many cases, unable to
    specify what Milestones                  needed to be done to
    complete a milestone, which laboratory(ies) bore responsibility
    for meeting it, or what their own particular contributions to
    meeting that milestone were. Laboratory officials said that no
    objective and specific measures exist, in most cases, to determine
    whether milestones have been successfully completed. Instead, they
    make subjective judgments about when a body of work meets a given
    milestone. For example, the "micro-aging" milestone, which appears
    in plans as a critical program milestone, was scheduled for
    completion in 1997. Los Alamos officials said they were unfamiliar
    with this milestone but believed it was Sandia's responsibility.
    Sandia officials agreed that they were responsible for this
    milestone and said that, in their opinion, the milestone had been
    met. However, while they identified relevant tasks, they were
    unable to identify precisely and completely what was supposed to
    be done to meet the milestone. Sandia officials speculated that
    the other two laboratories also had a responsibility for meeting
    this milestone. On the other hand, we found one example of a
    milestone for which specific performance criteria have been
    established. The "Prototype 3-D Primary Simulations" milestone,
    scheduled for completion by the end of calendar year 1999, was
    subject to a review by program officials in June 1998. The review
    established technical specifications for successfully completing
    the milestone and assessed the progress of the Los Alamos and
    Livermore software development activities contributing to this
    milestone. A progress review for this milestone is planned for the
    summer of 1999 and a follow-up review is planned upon its
    completion. Program officials acknowledge that they need to
    perform similar reviews for other milestones. In May 1999, DOE
    issued a report on its review, entitled "Codes for the Complex,"
    of the nonnuclear mechanics software development Page 24
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 2 A
    Comprehensive Planning and Tracking System Is Needed to Assess
    Program Progress efforts. At this juncture, however, most program-
    level milestones remain undefined. Systematic Progress    The
    efforts of DOE and laboratory managers to track ASCI progress thus
    far Tracking and           have been inadequate. Progress tracking
    has been limited primarily to the reporting of annual
    accomplishments without any systematic tracking of Reporting
    Needed       progress towards long-term milestones. Current
    tracking efforts occur through a variety of formal and informal
    methods, and the format and organization of these tracking efforts
    has varied from year to year. As a result, it is not possible to
    determine whether annual milestones were achieved or the extent to
    which annual efforts were contributing to the laboratory- and
    program-level milestones. In response to our request for tracking
    information, program officials decided to track and report the
    program's progress more systematically. The ASCI Program Plan
    described the semiannual principal investigators meetings as the
    primary forum for reporting program progress. The plan also noted
    that performance metrics would be developed and used at these
    meetings to compare actual output with planned output. However,
    these metrics were never developed. While these meetings are a
    forum for ASCI researchers to exchange ideas, there are few
    reporting requirements, and reports on program accomplishments
    have generally not been related to established milestones.
    Furthermore, the meetings focus on presentations of individual
    projects, with no effort to pull together a systematic and
    comprehensive assessment of how the ASCI program is progressing
    towards its overall goals. Laboratory officials met with DOE
    officials in fiscal years 1997 and 1998 to report their annual
    accomplishments. These meetings were focused solely on
    accomplishments and not on reporting the status of all milestones.
    As a result of this limited reporting process, some key
    information about the program's progress was not divulged. For
    example, accomplishments reported for fiscal year 1998 for one key
    software project did not reveal that the project was actually 6
    months behind schedule, which affected the schedule of other
    related projects. The Annual Performance Report is published at
    the end of each fiscal year to report on the progress of the
    Stockpile Stewardship Program, including the ASCI efforts. The
    report discusses various ASCI milestones met during the year, but
    these accomplishments are not all tied to the program-level
    milestones, and no assessment is made of how the program is
    progressing Page 25                                  GAO/RCED-99-
    195 Strategic Computing Initiative Chapter 2 A Comprehensive
    Planning and Tracking System Is Needed to Assess Program Progress
    in terms of meeting its most critical milestones. As such, the
    report did not provide a comprehensive assessment of the ASCI
    program. However, according to the Green Book, the ASCI program is
    critical to the success of the overall Stockpile Stewardship
    Program. To varying degrees, the laboratories used the annual
    implementation plans to report accomplishments during fiscal years
    1996 through 1998, although this practice was discontinued in
    fiscal year 1999. In fiscal year 1997, the implementation plan for
    infrastructure work at the three laboratories did not report any
    accomplishments for 1996, while the Sandia plan for software
    development reported several. However, the reported
    accomplishments cannot always be correlated easily with
    established milestones. In some cases, a connection was apparent,
    while in other cases it was not. In response to our request for
    information, the laboratories agreed to prepare material showing
    progress in meeting the milestones established in recent annual
    implementation plans. Because implementation plans were not
    prepared for all ASCI program components in 1998, however, draft
    documents and other plans are being used as a baseline to track
    progress in certain areas. Laboratory and DOE officials said that
    the information they developed at our request was very helpful for
    tracking program progress. DOE plans to have the laboratories
    refine their efforts and require that this information be updated
    on a quarterly basis. Program Progress        Although program
    managers have reported that many milestones have Difficult to
    Assess     been met, it is difficult to gauge the ASCI program's
    overall progress because of weaknesses in program management and
    information processes. The lack of a systematic progress tracking
    system, combined with the lack of defined performance criteria,
    make it difficult to assess whether the ASCI program is proceeding
    on schedule and delivering the performance expected. However, it
    is possible to gain some limited insights by discussing individual
    projects or areas with laboratory officials. In an attempt to
    determine the adequacy of hardware and software development to
    date, we discussed program progress in these areas with program
    officials. The insights we gained are discussed below. Hardware
    Performance    In the area of hardware development, most contract
    milestones to date that relate to the delivery and installation of
    the computers and related hardware have been met, although not all
    acceptance tests have been met. Currently the two 3-trillion
    operations per second systems at Los Alamos Page 26
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 2 A
    Comprehensive Planning and Tracking System Is Needed to Assess
    Program Progress and Livermore are operational. However, Los
    Alamos is still working to address problems affecting the
    reliability and stability of its system, which, as a result of
    these problems, did not pass all its contractually required
    acceptance tests on time. Progress in hardware development has
    been reported in annual implementation plans, at semiannual
    principal investigator meetings, and in press releases to the
    media. DOE and the laboratories, for example, have issued press
    releases that emphasize how fast these systems have performed on
    tests of their sustained speed. These releases may be misleading
    in some cases because they are reporting on only selected aspects
    of performance. For example, DOE reported that the Los Alamos
    system established a world record for sustained speed. However, it
    was not reported that the test used was substantially easier than
    the test specified in the contract or that the system has not yet
    met other acceptance test criteria (such as, mean time between
    failures). The sustained speed of this machine would be one-half
    the speed reported in press releases had the test specified in the
    contract been used. In addition, the Livermore computer continues
    to have problems with parallel input/output file operations. DOE's
    high-performance computers are expected to incorporate state-of-
    the-art hardware and software technologies. These computers can
    process multiple parts of one program at the same time, using
    parallel-processing techniques. According to the ASCI Program
    Plan, of the 100,000-fold increase in computing performance needed
    by the program, DOE expects a 10-fold increase from improving
    software to take advantage of parallel-processing techniques.
    However, the management of the input and output data during such
    processing continues to be an issue. For example, the data that
    result from running a model on 1,000 processors currently have to
    be saved to 1,000 separate files, making the data more difficult
    to manage and use. The laboratory is working with the vendor to
    address this problem. Software Development    Documentation of
    ASCI's software performance is limited. With the exception of the
    "Prototype 3-D Primary Simulations" milestone review discussed
    previously, little documentation exists that compares software
    development progress against established milestones. This review,
    also known as the "burn code review," is the only formal review of
    a milestone done to date. The numerous other software projects
    related to other milestones, such as 3-D secondary burn code
    projects at Los Alamos and Livermore and all software projects at
    Sandia, have not been reviewed. Page 27
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 2 A
    Comprehensive Planning and Tracking System Is Needed to Assess
    Program Progress According to our analysis of planning documents,
    accomplishment reports, and other related material on software
    activities at Sandia, the ASCI program's current planning and
    tracking system does not always provide an accurate picture of
    progress. For example, in reviewing the documentation for a major
    software development effort that Sandia officials told us was 6
    months behind schedule, we found no indication that delays had
    occurred. The documentation showed that most fiscal year 1998
    milestones had been met, and all milestones for the first quarter
    of fiscal year 1999 also were met. Sandia officials explained
    that, although the reported information was accurate, milestones
    had not been established for all of the important tasks associated
    with that project. Since the purpose of this project is to provide
    a common framework for all Sandia ASCI software models, delays in
    this project are hindering the performance of other software
    projects that require integration into the common framework.
    Sandia also did not report these delays in the list of
    accomplishments for fiscal year 1998 that it submitted to DOE.
    Sandia officials told us that, as of March 1999, 50 percent of
    their software development projects were experiencing delays
    because of funding or program changes or are behind schedule for
    other reasons. In general, detailed performance requirements have
    not been established for most software milestones, so it is
    difficult to develop an objective assessment of performance in
    this area. In effect, the judgment of whether software tests or
    demonstrations have achieved the desired level of performance is
    based on the subjective opinion of the laboratory and DOE program
    managers. DOE May Need to     DOE may not be appropriately
    managing the ASCI program by not Manage ASCI as a    designating
    it as a strategic system. DOE has established criteria for
    designating its most important projects as strategic systems to
    ensure Strategic System    oversight at the highest departmental
    level. The criteria are that the project costs more than $400
    million, is an urgent national priority, be high-risk, have
    international implications, or be vital to national security. The
    purpose of designating strategic systems is to ensure informed,
    objective, and well-documented decisions for key events, such as
    changes to baseline cost or schedule. The ASCI program meets the
    criteria for being treated as a strategic system. The ASCI program
    will likely cost about $5.2 billion for fiscal years 1996 through
    2004, is a critical part of the stockpile stewardship program, is
    an urgent national priority on national security grounds, and has
    international implications because it is a major factor in U.S.
    support of the Comprehensive Test Ban Treaty. Finally, the ASCI
    program is high risk because it seeks to advance the state of the
    art in computers, modeling, and simulation well beyond current
    capabilities, has Page 28
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 2 A
    Comprehensive Planning and Tracking System Is Needed to Assess
    Program Progress already experienced delays, has had its projected
    cost increase, and depends on as-yet unknown technologies for
    success. These characteristics, coupled with the demonstrated
    weaknesses in program management and oversight, make the ASCI
    program a clear candidate for being designated as a strategic
    system. According to DOE, it has not designated this effort as a
    strategic system because the program is already subject to high-
    level department oversight. However, as discussed above, we found
    serious weaknesses in the program's management and information
    processes that make it difficult to determine if the program is
    performing as expected. Conclusions        DOE's oversight of the
    ASCI program is hampered by weaknesses in management and
    information processes. The program lacks a comprehensive planning
    system-one that clearly establishes milestones; links short- and
    long-term milestones; identifies research strategies, critical
    paths, and decision points; and defines performance criteria for
    milestones. Furthermore, the program lacks a progress tracking and
    reporting system. Consequently, overall program progress is
    difficult to assess. DOE has chosen not to designate the ASCI
    program as a strategic system. The demonstrated weaknesses in the
    ASCI program's management and information processes, coupled with
    the program's critical role in DOE's mission to maintain the
    nation's stockpile of nuclear weapons without testing, warrant
    DOE's designating ASCI as a strategic system requiring the highest
    levels of management attention. Recommendations    Given the ASCI
    program's lack of a comprehensive planning, tracking, and
    reporting system and the importance of the program to maintaining
    the stockpile of nuclear weapons, it is important that DOE improve
    its oversight and management of this program. Therefore, we
    recommend that the Secretary of Energy require the establishment
    of a comprehensive planning, tracking, and reporting system. This
    system should, at a minimum, establish clear milestones; identify
    links between short- and long-term milestones; identify research
    strategies, critical paths, and decision points; define
    performance criteria for the successful completion of milestones;
    and establish progress tracking and reporting requirements. We
    further recommend that the Secretary of Energy designate the ASCI
    program as a strategic system warranting oversight at the highest
    departmental level. Page 29
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 2 A
    Comprehensive Planning and Tracking System Is Needed to Assess
    Program Progress Agency Comments       DOE agreed with the
    recommendation to improve the oversight and and Our Evaluation
    management of the program and cited several changes that it was
    making. Specifically, DOE stated that it would soon issue an
    updated Program Plan that will include detailed specifications for
    all of the critical program milestones. In addition, the FY 2000
    Implementation Plans that will be issued by September 30, 1999,
    will include descriptions of all program elements and complete
    lists of all milestones. The Department also cited the creation of
    a quarterly progress tracking mechanism to track program
    milestones. However, in addition to tracking the program's
    progress against established calendar milestones, it is necessary
    to establish specific technical criteria for what constitutes the
    successful completion of those milestones. Until DOE completes and
    publishes its revised Program Plan and FY 2000 Implementation
    Plans, we cannot determine whether it has fully complied with this
    recommendation. DOE disagreed with the recommendation to designate
    the ASCI program as a strategic system and stated that to do so
    would duplicate the planning, progress tracking, and reporting
    system. We agree that creating a duplicative tracking system that
    mirrors the requirements set out by DOE for strategic systems
    would not be worthwhile. However, as discussed in detail in this
    report, DOE has not shown that it has an adequate planning,
    progress tracking, and reporting system in place for the strategic
    computing initiative. While DOE is making improvements in these
    areas, the changes are not yet fully in place and their adequacy
    cannot be judged at this time. Furthermore, if the changes that
    DOE is making are adequate to meet the requirements for tracking
    and monitoring a strategic system, then we cannot understand DOE's
    reluctance to designate this large and costly program as a
    strategic system. DOE stated that it has a review process that
    meets the intent of the Clinger-Cohen Act of 1996. However, we
    reported in July 1998 that the Department's process effectively
    excludes scientific computers like those being acquired through
    the ASCI program from DOE's normal review channels and places them
    within the program offices.5 We stated that all computers should
    be included as part of the normal DOE Clinger-Cohen review
    process. 5Information Technology: Department of Energy Does Not
    Effectively Manage Its Supercomputers (GAO/RCED-98-208, Jul. 17,
    1998). Page 30                                     GAO/RCED-99-195
    Strategic Computing Initiative Chapter 3 Management and Tracking
    of Costs Need to Be Improved ASCI cost estimates have increased
    substantially. In 1995, DOE estimated that program costs for
    fiscal years 1996 through 2001 would be $1.7 billion.1 By 1999,
    estimated costs for those years increased to $2.9 billion. DOE
    currently estimates that the program will cost about $5.2 billion
    for fiscal years 1996 through 2004. Some of the cost increases
    result from changing program requirements. For example, in 1996,
    the United States shifted from a program based on nuclear testing
    to one based on computer simulations of weapon performance. For
    the ASCI program, the shift to computer simulations resulted in
    higher costs to acquire the latest and fastest computers and to
    develop advanced simulation and modeling software. The cost
    increases also reflect weaknesses in DOE's cost estimation and
    management. For example, DOE has difficulty determining technical
    requirements and then reliably estimating costs for state-of-the-
    art computers and software. In addition, DOE limits its cost
    oversight to reviews of aggregate laboratory spending and
    consequently cannot determine if the costs of specific projects at
    the laboratories are over or under budget. Cost Estimates    DOE's
    cost estimates for the ASCI program have increased substantially
    Increased         since 1995, when early budget projections were
    made. Costs for fiscal years 1996 through 2001 have increased from
    an original estimate of $1.7 Substantially     billion to the
    current $2.9 billion. DOE's actual fiscal year 2000 budget request
    for ASCI, which totals $692 million, is more than double the
    original fiscal year 2000 estimate made in 1995. Figure 3.1 shows
    for each fiscal year the original and current budget estimates.
    1The estimated cost figures provided in this report have not been
    adjusted to constant dollars. Rather, they reflect DOE's budgeting
    and planning process estimates, which were provided in current
    dollars. Page 31                                         GAO/RCED-
    99-195 Strategic Computing Initiative Chapter 3 Management and
    Tracking of Costs Need to Be Improved Figure 3.1 Original and
    Current Cost Estimates for the ASCI Program, Fiscal    900
    Dollars in millions Years 1996 Through 2004 800 700 600 500 400
    300 200 100 0 1996      1997       1998    1999    2000    2001
    2002    2003    2004 Fiscal Year Original Current Source: GAO's
    analysis of DOE's data. The current total estimated cost of the
    ASCI program, for fiscal years 1996 through 2004, is about $5.2
    billion. Although the program is scheduled to operate through
    2010, estimates beyond 2004 have not been made. Figure 3.2 shows
    how this $5.2 billion is allocated by program areas. Page 32
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 3
    Management and Tracking of Costs Need to Be Improved Figure 3.2:
    Allocation of Estimated ASCI Costs by Major Program Areas,
    Dollars in millions Fiscal Years 1996 Through 2004
    $60 Program coordination *                     $218 Academic
    alliances *                $251 New buildings *           $776
    Computers *                           $3,937 Software and
    infrastructure Source: GAO's analysis of DOE's data The $776
    million estimated for computers includes the costs of the three
    computers currently installed at the laboratories and the planned
    acquisition of computers that perform 10-, 30-, and 100-trillion
    operations per second. The vast majority of ASCI funds, almost $4
    billion, will be spent on the software development and program
    infrastructure. Software and infrastructure include the
    development of the 3-D simulation models and the multiple
    peripheral technologies needed for, among other things,
    visualization, networking, and data management. DOE estimates it
    will spend $251 million to construct new buildings at the
    laboratories that will house the computers, as well as offices and
    visualization theaters. The Academic Strategic Alliances Program
    with the universities will cost $218 million, and another $60
    million is estimated for the coordinating laboratories' efforts.
    Changing Requirements                  DOE officials told us that
    ASCI costs have escalated since the original budget Account for
    Some Cost                  projections because the program was
    originally intended as a limited Increases
    effort to improve the computer capability available for stockpile
    Page 33                                 GAO/RCED-99-195 Strategic
    Computing Initiative Chapter 3 Management and Tracking of Costs
    Need to Be Improved stewardship. In the original budget estimate,
    developed in 1995, DOE officials requested funding for a 1,000-
    fold increase in computing capability. Since then, the ASCI
    program has expanded because of changes in the U.S. nuclear
    weapons policy, particularly, the U.S. decision in August 1995 to
    pursue a "zero yield" Comprehensive Test Ban Treaty. This policy
    change meant that the United States would need to maintain the
    nuclear stockpile far beyond its design life and would have to
    shift from a traditional nuclear test-based program to one based
    on computer simulations to ensure the safety and reliability of
    nuclear weapons. As a result, DOE has developed strategies that
    focus on advanced modeling and simulation that require a 100,000-
    fold increase in computer capability. Unreliable Cost Estimates
    While the change in program requirements has affected DOE's budget
    Also Contribute to Cost      estimates, officials also acknowledge
    their difficulty in estimating costs Increases
    because of the unprecedented scale of the hardware and software
    technologies needed by the ASCI program. For example, before DOE
    began this effort, a computer with thousands of processors
    operating as a fully integrated system had never been built. In
    addition, software to run on systems of this size and high-
    performance visualization technology to display the results of
    simulations at this scale had never been developed. According to
    one DOE official, the Department might lack the expertise to
    anticipate future technical requirements for state-of-the-art
    hardware and software and to reliably estimate their costs. For
    example, when planning the current, expanded ASCI program, DOE
    envisioned a single computer capable of 3 trillion operations per
    second that could provide access and collaborative opportunities
    to all three laboratories, using secure, high-speed networking
    capabilities. However, DOE decided to procure a second such
    computer in 1997 because the technical capabilities to support
    such long-distance computing were not yet in place. Combined, the
    two computer systems cost DOE almost $220 million. In addition,
    DOE acknowledged that during early funding strategies, it did not
    consider the difficulty and importance of the technology needed by
    weapons designers to visualize the results of the 3-D weapons
    simulations. Yet such visualization technologies are required to
    graphically represent to weapons designers the results of 3-D ASCI
    simulations. DOE currently estimates that more than $87 million is
    needed for visualization activities for fiscal years 1999 through
    2004. Page 34                                GAO/RCED-99-195
    Strategic Computing Initiative Chapter 3 Management and Tracking
    of Costs Need to Be Improved Better Oversight of    DOE's
    oversight of costs is limited to a review of aggregate spending at
    the Costs Needed           laboratories. While DOE contends that
    cost controls for ASCI are in place, it does not track costs to
    determine which specific projects may be over or under budget. DOE
    monitors monthly spending for each laboratory but does not compare
    previously estimated costs for major projects with their actual
    costs. For example, as part of the budget formulation process,
    project costs are estimated and subsequently included as part of
    the Department's fiscal year budget request. However, DOE tracks
    only how much the laboratories have spent in broad categories that
    lump together costs for many projects. As a result, DOE cannot
    determine which projects, if any, may be costing more or less than
    originally planned. DOE told us it relies on the laboratories to
    determine whether projects are within their planned budget, but
    one laboratory ASCI manager told us that the laboratory tracks
    only the technical status of projects, not their costs. In
    addition, DOE's limited oversight of the laboratories' activities
    could result in DOE's underestimating ASCI program costs. DOE
    estimates that ASCI program costs have increased by about $1.2
    billion compared with its original estimate for fiscal years 1996
    through 2001. However, delays in completing projects at the
    laboratories could increase those costs. For example, as noted in
    chapter 2, one laboratory estimates that 50 percent of its
    software development projects are experiencing delays or are
    behind schedule. DOE's lack of information about the progress of
    projects, combined with its limited cost tracking, do not allow
    DOE to determine how much longer it will take to complete those
    projects or at what cost, thus limiting its ability to accurately
    project ASCI program costs. In January 1999, we reported on the
    significant management challenges at DOE, including the difficulty
    completing large projects within budget.2 We noted that DOE often
    requires large projects costing hundreds of millions of dollars
    that are often the first of their kind and involve substantial
    risk. ASCI is such a challenge. ASCI is critical to DOE's mission,
    is estimated to cost about $5.2 billion, requires the development
    of hardware and software on an unprecedented scale, and involves
    substantial risks. Conclusions            ASCI costs have
    increased substantially because of changes in program requirements
    and weaknesses in DOE's cost estimates. Because its tracking of
    costs is limited, DOE cannot determine whether specific projects
    are under or over budget. Historically, DOE has had difficulty
    managing the costs of large programs. 2Department of Energy: Major
    Management Challenges and Program Risks (GAO/OGC-99-6, Jan. 1999).
    Page 35                                     GAO/RCED-99-195
    Strategic Computing Initiative Chapter 3 Management and Tracking
    of Costs Need to Be Improved Recommendation        Given the
    substantial increases in the ASCI program's cost estimates to
    date, DOE's acknowledged problem in estimating costs for the
    unprecedented scale of development efforts involved in the ASCI
    program, and the lack of a cost-tracking process, it is important
    that DOE improve its oversight of ASCI program costs. Therefore,
    we recommend that the Secretary of Energy require that ASCI adopt
    systematic cost-tracking procedures that will allow DOE managers
    to determine if specific projects are within budget. Agency
    Comments       DOE did not agree with our recommendation to adopt
    systematic and Our Evaluation    cost-tracking procedures for the
    strategic computing initiative, noting that funding and costs are
    tracked by budget and reporting codes in the Department's
    Financial Information System. DOE stated that these systems are
    extended to individual projects using other funding and cost-
    monitoring tools that gather more detailed information. As an
    example, DOE cited an analysis performed in March 1999 of selected
    projects that identified the commitments and cost status for
    specific procurements at the project level. We do not agree that
    DOE has an adequate system for tracking at the project level or
    that the changes it is making will rectify this problem. DOE's
    current system tracks cost only at the aggregate level and does
    not allow DOE managers to determine which projects at the
    laboratories are under or over budget. Furthermore, the "other
    funding and cost monitoring tools" that DOE uses do not allow the
    systematic tracking of project costs. DOE also stated that some
    budgeting flexibility is necessary to capitalize on changes within
    the high-computing industry. While we agree that some budgeting
    flexibility is necessary in a project of this size and complexity,
    we do not believe that that flexibility should preclude the
    effective oversight of a multiyear program costing over $5
    billion. Page 36                                GAO/RCED-99-195
    Strategic Computing Initiative Chapter 4 Technical Challenges Are
    Present in All Aspects of the ASCI Program The primary challenge
    facing the ASCI program is to develop a simulation capability
    that, in the absence of nuclear testing, can be used to determine
    whether a modified weapon system will perform as intended. The
    need for this "virtual test" capability encompasses most of the
    technical challenges associated with the ASCI program. These
    challenges range from developing state-of-the-art hardware and
    software technologies, to the integration of scientific data from
    weapons physics experiments, to recruiting and retaining staff
    with the technical expertise needed. The risks in failing to
    address these challenges are intensified because the program is a
    research-and-development effort with an accelerated schedule.
    Program officials acknowledge the multitude of risks associated
    with the program and point to the risk mitigation strategies they
    have designed to address program risks. These strategies include
    the use of several vendors to develop computers, overlapping
    software development efforts, and partnerships with industry and
    academia. Technology                  The development of hardware
    and software technologies and of the Development
    necessary infrastructure to support these technologies is critical
    to achieving the ASCI program's simulation and modeling goals.
    Hardware development must successfully increase computational
    speeds to 100-trillion operations per second in 2004. Software
    development efforts are extensive and must ultimately incorporate
    massive amounts of data, solve progressively more difficult
    problems, and be capable of running on increasingly larger and
    faster computers. Developing the needed infrastructure, including
    data storage and visualization technologies, will require
    significant improvements. Computer Speed Must         Developing
    computers capable of processing complex 3-D nuclear weapons
    Increase Dramatically by    simulations is one of the primary
    challenges facing the ASCI program. 2004
    Increasing the computational speed to 100-trillion operations per
    second by 2004, according to program officials, is essential to
    meeting program goals. At Livermore, development is under way on a
    10-trillion operations per second computer that is scheduled to be
    installed during fiscal year 2000, and the acquisition of a 30-
    trillion operations per second computer at Los Alamos is planned
    for fiscal year 2001. A request for proposals for the 30-trillion
    operations per second computer was sent out in May 1999, and
    contracts have been signed with several vendors to work on the
    related technology needed for a computer of this size. Page 37
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 4 Technical
    Challenges Are Present in All Aspects of the ASCI Program Program
    officials explained that their risk mitigation strategy includes
    using competing computer vendors to independently develop
    increasingly larger computers at the three weapons laboratories. A
    1.8-trillion operations per second computer that was developed by
    the Intel Corporation is in use at Sandia, while both Los Alamos
    and Livermore are developing computers capable of 3-trillion
    operations per second. The computer at Los Alamos is being
    developed by Silicon Graphics, Incorporated, while Livermore is
    working with the IBM Corporation to develop its computer.
    According to laboratory officials, the experience gained at
    Sandia, plus the competing efforts at Los Alamos and Livermore,
    helps to ensure that at least one of these computers will be
    generally available to carry out computational work. According to
    a program official, the competing computers at Los Alamos and
    Livermore are based on different technologies, which helps to
    further mitigate risks. Currently, Livermore is developing the 10-
    trillion operations per second computer with the IBM Corporation,
    while Los Alamos sent out a request for proposal to select a
    vendor to develop the 30-trillion operations per second computer
    in May 1999. ASCI program officials explained that the ASCI
    computers being constructed involve thousands of processors,
    switches, disks, and related components that must work together as
    a fully integrated system to run the largest simulations. These
    officials explained that getting computer systems of this size to
    operate as a fully integrated system has never before been
    achieved and is one of the most difficult challenges facing the
    program. An April 1998 review of the computing division at Los
    Alamos by an external committee recognized this issue by pointing
    out that users generally had access only to small parts of the
    computer and rarely had access to the full system.1 The
    Committee's report explained that operating the computer as a
    fully integrated system was important because the ASCI computer
    needs are based on running simulations that require the full
    capability of the computer. The challenge continues today with the
    3-trillion operations per second computer at Los Alamos, which has
    experienced many failures when trying to run as a fully integrated
    system. A March 1999 review by the ASCI Blue Ribbon Panel noted
    another important risk in meeting the schedules for computers
    operating in the range of 30- to 100-trillion operations per
    second.2 The report explained that to meet the schedule for these
    larger computers, it might be necessary for the laboratories to
    write the system software necessary to enable the 1Los Alamos CIC
    Division External Review Committee Report for the April 1998
    Review. 2Report of the ASCI Blue Ribbon Panel, Mar. 2, 1999. Page
    38                                        GAO/RCED-99-195
    Strategic Computing Initiative Chapter 4 Technical Challenges Are
    Present in All Aspects of the ASCI Program computer to operate as
    a fully integrated system. The report noted that there is a risk
    in such a course of action because laboratory personnel do not
    have extensive experience in this area. Software Development Is
    Developing software that incorporates all of the required science
    to Critical to Program        simulate nuclear weapons while
    running on computers consisting of Success
    thousands of processors is, according to ASCI program managers,
    one of the most demanding tasks of the ASCI program. These
    officials explained that developing such software has historically
    taken approximately 5 years before it can be used with confidence.
    Because of the complexity, these officials stated that ASCI
    software may take longer to develop, and a key program goal is to
    reduce the development time to the 5-year historic average. A
    report by the President's Information Technology Advisory
    Committee described current software development as "among the
    most complex of human-engineered structures."3 The report noted
    that the nation's ability to construct needed software systems and
    to analyze and predict the performance of these systems is
    painfully inadequate. Part of DOE's risk mitigation strategy in
    ASCI software development includes competing software efforts at
    Los Alamos and Livermore. In addition, to leverage their efforts
    in software development, the laboratories have contracted with
    several universities (through the ASCI Academic Strategic
    Alliances Program) to conduct research in areas of high-
    performance computing and physical science. The technical
    challenges inherent in the development of ASCI-related software
    are due in part to the complexity of the needed software. Program
    officials describe the ASCI software development effort as a
    hierarchy of development. At the lower level of the hierarchy are
    software modeling efforts that include (1) modeling the
    engineering features and the materials used in weapons systems,
    (2) modeling the physics phenomena associated with weapons
    systems, and (3) developing computational problem-solving
    techniques that will allow calculations to take place at
    increasingly higher processing speeds. At the top of the software
    development hierarchy are the integrated software applications
    that will eventually (as larger ASCI computers become available)
    incorporate all the lower-level modeling efforts and computational
    techniques into a single system. This integrated software is
    expected to provide the ability to simulate weapons performance
    ranging from individual components to full weapons systems,
    including performance in 3Interim Report to the President. The
    President's Information Technology Advisory Committee, Aug. 1998.
    Page 39                                       GAO/RCED-99-195
    Strategic Computing Initiative Chapter 4 Technical Challenges Are
    Present in All Aspects of the ASCI Program hostile environments.
    This software is also expected to provide the capability of
    predicting the performance of weapons components and full weapons
    systems in analyses of design, aging effects, and accident
    scenarios. Program officials also expect that ASCI software will
    be used to design efficient and environmentally acceptable
    manufacturing processes. The report by the ASCI Blue Ribbon Panel
    noted several concerns about the modeling efforts of the lower-
    level software. Among the concerns raised was one about the
    materials science area. The panel said that this area warrants
    further review because it forms the basis of so much of the work
    and involves issues of great complexity, some of which are not
    understood at a fundamental level anywhere in the materials
    science community. The reviewers also noted that the presentations
    they heard did not indicate that the integration of experimental
    data is tightly coupled to software development. The report notes
    that a robust experimental program that is closely tied to
    simulations is crucial to assess the adequacy of the scientific
    input and to test the software. Another technical challenge in the
    area of software development is the development and consistent use
    of software quality assurance. In general, software quality
    assurance involves reviewing and auditing software products and
    activities to verify that they comply with the applicable
    procedures and standards. An April 1998 review of the computing
    facilities at Los Alamos concluded that software quality assurance
    has not been addressed sufficiently in the ASCI program.4 The
    report noted that the situation is exacerbated because of the
    current shortage of expertise in the area of software quality
    assurance. ASCI program officials have explained that efforts to
    ensure software quality are part of their new software
    verification and validation effort and that they have efforts
    under way at each of the laboratories to address this issue. For
    example, Livermore has established the Software Technology Center,
    and a software quality assurance team has been formed using staff
    from each of the three laboratories. This team recently conducted
    a survey at each of the laboratories to develop an initial
    inventory of software quality practices being used at the
    laboratories. The team is preparing detailed reports on its survey
    findings that will be provided to each of the laboratories.
    Significant Technological    The major technical risks in the
    infrastructure area are associated with Improvements Needed in
    (1) extracting optimum simulation performance from tens of
    thousands of Some Infrastructure Areas    processors and (2)
    moving, storing, and displaying large, complex results 4Los Alamos
    CIC Division External Review Committee Report for the April 1998
    Review. Page 40                                      GAO/RCED-99-
    195 Strategic Computing Initiative Chapter 4 Technical Challenges
    Are Present in All Aspects of the ASCI Program for interpretation
    by weapons designers. According to program officials, risk
    mitigation strategies employed in this area include ongoing
    research at the laboratories and through the Academic Strategic
    Alliances Program and contracts with third-party partnerships to
    develop a variety of advanced techniques and technologies. In
    addition, the ASCI program has sponsored workshops with
    universities, other government agencies, and industry to engage
    them in a common approach to meeting these challenges. Significant
    technological improvements are needed in several critical
    infrastructure areas, including visualization and storage
    technology and the technology that connects computers to other
    components. For example, visualization is an essential analysis
    tool for understanding the volumes of data that will be produced
    by ASCI software. The laboratories have recently unveiled new data
    visualization centers, but according to ASCI planning documents,
    the defined user needs exceed industry visualization hardware
    capabilities by 15 to 60 times. Achieving the needed improvements
    is also challenging because there is currently only one vendor in
    this area. Significant improvements will also be needed for data
    storage technology and connection technology for the 30-trillion
    operations per second system. Contracts have been signed and work
    is under way with several vendors to address these issues. The
    ASCI Blue Ribbon Panel also cited visualization technology as an
    area of concern, particularly the level of involvement by weapons
    designers in the planning of visualization capabilities and
    facilities. The report also noted that the panel was concerned
    about the accelerated pace of investment in visualization
    technology because the basis for visualization needs was unclear.
    Integration of     Integrating the data from laboratory
    experiments conducted outside the Scientific Data    ASCI program
    into ASCI software development efforts has been noted as another
    important technical challenge. DOE's Green Book notes that to
    achieve the modeling and simulation goals of the ASCI program, new
    data will be needed from laboratory experiments to help verify the
    accuracy of the ASCI software. These experiments are designed to
    learn more about the physical processes that occur to a weapon
    under normal and abnormal conditions. The Green Book also notes
    that the schedule for future experiments and the computational
    needs of the ASCI program must be closely and carefully
    coordinated to ensure that the experimental data are useful to the
    ASCI program. The facility plan for one of the stockpile Page 41
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 4 Technical
    Challenges Are Present in All Aspects of the ASCI Program
    stewardship program's experimental facilities illustrates the
    connection between data from experiments and the ASCI program. The
    plan for the National Ignition Facility (NIF) explains that much
    of the program is designed to gather fundamental weapons-relevant
    data and use these data to enhance and refine nuclear weapons
    simulations.5 Recent reviews have commented on the issue of
    physics data in the ASCI program. The ASCI Blue Ribbon Panel noted
    that a robust experimental program, which is closely tied to ASCI
    simulations, is crucial to assessing the adequacy of the
    scientific input and to testing the accuracy of the software. The
    panel reported that the presentations it was given by laboratory
    officials did not indicate that the experimental data were tightly
    integrated with software development. Although the panel did not
    review the experimental program in depth, it recommended that
    additional funding should be made available to produce the
    physical data required to support ASCI software efforts. Two June
    1998 reports on the software development efforts at Los Alamos
    expressed concern that the issue of weapons physics had not
    received the attention it deserved.6 One report explained that the
    review panel wanted to learn more about the role of experiments in
    validating the accuracy of ASCI software and that certain
    laboratory staff should have a more prominent role in the
    selection of experiments conducted at DOE facilities. Program
    officials acknowledge the need for closer integration between
    laboratory experiments and the ASCI program. They explained that
    they have taken actions such as a reorganizing DOE management and
    creating a formal software validation program that requires data
    from experiments. In addition, officials explained that the 1998
    review of the primary burn code milestone reported on the
    effective integration of experiments with the ASCI program.
    According to DOE officials, the reorganization of offices within
    DOE's Office of Defense Programs, now in progress, will allow ASCI
    program officials to set both the ASCI program schedule and the
    schedule for needed laboratory experiments. Program officials also
    explained that the ASCI "verification and validation" effort, new
    for fiscal year 1999, would provide the framework for aligning the
    needs of the ASCI program with the schedule for laboratory
    experiments. The validation effort includes the use of laboratory
    experiments to ensure that the simulations are consistent with
    observed behavior. The June 1998 review of the 5Facility Use Plan
    of the National Ignition Facility, Edition 1, April 1997. NIF is
    planned as a multiple-beam, high-power laser system with the goal
    of attaining the ignition of thermonuclear fuel in the laboratory.
    6"Report of the X-Division Review Committee," May 18-20, 1998, and
    "Organizational Self-Assessment for the Applied Theoretical and
    Computational Physics Division," June 26, 1998. Page 42
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 4 Technical
    Challenges Are Present in All Aspects of the ASCI Program primary
    burn code milestone found ample evidence of integration among ASCI
    software development, experiments, and the use of existing
    information from previous nuclear tests. The review noted that
    data from experiments are currently being incorporated into ASCI
    software. Technical Expertise    Recruiting and retaining
    qualified personnel is a continuing area of risk, according to
    ASCI program officials and outside program reviews. DOE and
    laboratory officials have explained that they have efforts ongoing
    in many areas to improve the recruiting of staff with the required
    expertise. These efforts include exemptions from salary ceilings
    and the ASCI Academic Strategic Alliance Program. The Chiles
    Commission noted that there is no certainty that DOE will succeed
    in maintaining future nuclear weapons expertise.7 ASCI program
    officials are concerned about the availability of staff with the
    necessary expertise. For example, Los Alamos officials noted that
    several milestones during fiscal year 1998 were delayed because of
    a shortage of staff with the needed expertise in software
    development. They explained that such personnel are difficult to
    recruit and that, once recruited, they need time to develop the
    necessary weapons-related expertise. Furthermore, these officials
    said that they have received a waiver from DOE to offer more
    competitive salaries to recruit qualified staff. Livermore
    officials explained that personnel with computer science and math
    skills are in high demand, which makes it difficult to recruit
    them into the ASCI program. Their risk mitigation strategy
    includes using the Academic Strategic Alliance Program to attract
    qualified students, offering competitive salaries, and using the
    unique research and development aspects of the ASCI program to
    attract potential candidates. The Chiles Commission report on
    maintaining nuclear weapons expertise noted that it was difficult
    to conclude that DOE will succeed in maintaining future nuclear
    weapons expertise. Although the report found a great deal that is
    healthy in the nuclear weapons complex, with many trends moving in
    the right direction, it also found other matters that are
    disturbing. These other matters included the aging workforce, the
    tight market for talent, and the lack of a long-term hiring plan.
    The report also concluded that steps need to be taken now to
    ensure that the upcoming generation of designers is recruited and
    trained while the more experienced designers remain at the
    laboratory or are available through retiree programs. The 7"Report
    of the Commission on Maintaining United States Nuclear Weapons
    Expertise." Report to the Congress and the Secretary of Energy,
    March 1, 1999. Page 43
    GAO/RCED-99-195 Strategic Computing Initiative Chapter 4 Technical
    Challenges Are Present in All Aspects of the ASCI Program report
    explained that more than 60 percent of the nuclear weapons
    designers at Los Alamos and Livermore are between 50 to 65 years
    old and that 5 or more years of experience working with
    experienced designers is required to develop a fully capable,
    independent designer. The ASCI Blue Ribbon Panel report concluded
    that the training of the next generation of technical staff is the
    single largest problem facing not only the ASCI program but also
    the entire weapons program. The report commended the ASCI program
    for its Academic Strategic Alliances Program to attract high-
    quality, technically trained personnel but also stated that its
    fellowships and summer internships must be made more appealing and
    competitive. The report also noted that ASCI computers could be
    used to attract students into the field of computational science
    and that, although this is being done in the Academic Strategic
    Alliance Program, it could be done more broadly. In addition, the
    report questioned whether the laboratories are able to compete in
    the market for the best personnel. The report concluded that ties
    to the universities would be vital to convince the best students
    to make a career at the laboratories. Program officials
    acknowledge the problem of recruiting and retaining staff
    expertise. They cite efforts such as a DOE fellowship program as a
    means of attracting needed expertise. Under the fellowship
    program, ASCI will support eight students, a number that may
    double in the next 2 years. Students must serve a "practicum" at
    DOE or the laboratories as a condition of support. Program
    officials hope that this experience will interest the students in
    working at the laboratories when they complete their education.
    Page 44                                    GAO/RCED-99-195
    Strategic Computing Initiative Page 45      GAO/RCED-99-195
    Strategic Computing Initiative Appendix I Program- and Laboratory-
    Level Software Milestones From DOE Planning Documents, Fiscal
    Years 1996-2004 Program-level documents
    Laboratory-level documents Program           Most critical
    Consolidated     Nuclear        Nonnuclear plan
    milestones       Green Booka roadmapb           roadmapc
    Roadmapd Fiscal year    Milestone description          (Sept. 96)
    (July 98)        (Apr. 98)     (Feb. 98)        (Mar. 99)
    (Mar. 99) 96             Microaging
    X                X 97             Microaging                     X
    X                X High-fidelity safety           X
    X                X             X calculation Neutron generator
    standoff
    X 3-D casting microstructure
    X                X 98             3-D casting microstructure X
    X                X 3-D nuclear safety
    X                                               X simulation/
    Prototype nuclear safety simulatione Neutron generator hostile
    X                                                              X
    certification/ Neutron generator radiation hardness mechanicale
    Prototype 3-D physics
    X Prototype 3-D
    X hydrodynamics/ radiation-hydrodynamics System/composition
    X thermal B61 penetrator
    X Macro/micro aging
    X 99             Macro/micro aging              X
    X                X Prototype 3-D physics/         X
    X                X             X                X Prototype 3-D
    primary simulations/ 3-D burn code/ 3-D primary burn codes/ 3-D
    primary burn prototypee 3-D forging/welding            X
    X                X             X microstructure Crash/fire safety
    X                                  X             X 2-D
    deterministic radiation
    X transport Abnormal environment
    X thermal assessment Neutron generator
    X performance code (continued) Page 46
    GAO/RCED-99-195 Strategic Computing Initiative Appendix I Program-
    and Laboratory-Level Software Milestones From DOE Planning
    Documents, Fiscal Years 1996-2004 Program-level documents
    Laboratory-level documents Program             Most critical
    Consolidated     Nuclear        Nonnuclear plan
    milestones       Green Booka roadmapb           roadmapc
    Roadmapd Fiscal year    Milestone description            (Sept.
    96)          (July 98)        (Apr. 98)     (Feb. 98)        (Mar.
    99)      (Mar. 99) Full system (Salinas)
    X 00             Full physics, full system        X
    X prototype Hostile environment                                  X
    electronics certification Abnormal environment
    X thermal assessment Prototype 3-D secondary
    X                                               X simulations/ 3-D
    secondary burn prototypee Pit casting and
    X manufacturing code 3-D prototype radiation
    X flow simulation Full system microaging
    X simulation Parachute
    X Reentry vehicle
    X aerodynamics B61 laydown
    X Electrical circuit simulation
    X capability Component deterioration
    X model 01             Component deterioration          X
    X                X model Initial operating code/          X
    X                X             X Prototype 3-D coupled simulation/
    3-D secondary burn codee Stockpile-to-target-
    X sequence certification demonstration System composition burn
    X 3-D prototype full system
    X coupled simulation 02             Full physics burn code
    X prototype Full system radiation
    X hardness & hostile (continued) Page 47
    GAO/RCED-99-195 Strategic Computing Initiative Appendix I Program-
    and Laboratory-Level Software Milestones From DOE Planning
    Documents, Fiscal Years 1996-2004 Program-level documents
    Laboratory-level documents Program           Most critical
    Consolidated         Nuclear       Nonnuclear plan
    milestones       Green Booka roadmapb                   roadmapc
    Roadmapd Fiscal year    Milestone description              (Sept.
    96)        (July 98)        (Apr. 98)         (Feb. 98)
    (Mar. 99)     (Mar. 99) Abnormal
    X stockpile-to-target- sequence Burn code with aging
    X 03             Integrated full physics
    X burn code prototype 3-D electrical device
    X physics Abnormal
    X stockpile-to-target- sequence with aging Normal stockpile-to-
    target-
    X sequence 04             Normal stockpile-to-target-
    X sequence with aging Complete physics full
    X                                  X                    X system
    prototype/ Initial full system simulation code/ 3-D high fidelity
    physics full system initial capabilitye aThe full title of this
    document is Stockpile Stewardship Plan - 2nd Annual Update. bThe
    full title of this document is Consolidated Applications Roadmap.
    cThe full title of this document is Consolidated Nuclear Component
    Applications Roadmap. dThe full title of this document is Non-
    Nuclear Codes Roadmap. eThis represents a single milestone that is
    titled differently in multiple sources. Sources: ASCI Program Plan
    (1996), Stockpile Stewardship Plan (1998), and other information
    provided by the Department of Energy, and the Los Alamos,
    Livermore, and Sandia National Laboratories. Page 48
    GAO/RCED-99-195 Strategic Computing Initiative Appendix II
    Comments From the Department of Energy Note: GAO comments
    supplementing those in the report text appear at the end of this
    appendix. Page 49      GAO/RCED-99-195 Strategic Computing
    Initiative Appendix II Comments From the Department of Energy Page
    50                                   GAO/RCED-99-195 Strategic
    Computing Initiative Appendix II Comments From the Department of
    Energy Now on p. 6. See comment 1. Page 51
    GAO/RCED-99-195 Strategic Computing Initiative Appendix II
    Comments From the Department of Energy Page 52
    GAO/RCED-99-195 Strategic Computing Initiative Appendix II
    Comments From the Department of Energy Now on p. 6. See comment 2.
    Now on p. 19. See comment 3. Page 53
    GAO/RCED-99-195 Strategic Computing Initiative Appendix II
    Comments From the Department of Energy Now on p. 19. See comment
    4. Now on p. 20. See comment 5. Now on p. 21. See comment 6. Page
    54                                   GAO/RCED-99-195 Strategic
    Computing Initiative Appendix II Comments From the Department of
    Energy Now on p. 22. See comment 7. Page 55
    GAO/RCED-99-195 Strategic Computing Initiative Appendix II
    Comments From the Department of Energy Now on p. 22. See comment
    8. Now on p. 23. See comment 9. Page 56
    GAO/RCED-99-195 Strategic Computing Initiative Appendix II
    Comments From the Department of Energy Now on pp. 24-25 See
    comment 10. Now on p. 25. See comment 11. Page 57
    GAO/RCED-99-195 Strategic Computing Initiative Appendix II
    Comments From the Department of Energy Now on p. 25. See comment
    12. Now on p. 26. See comment 13. Now on p. 27. See comment 14.
    Page 58                                   GAO/RCED-99-195
    Strategic Computing Initiative Appendix II Comments From the
    Department of Energy Now on p. 27. See comment 15. Page 59
    GAO/RCED-99-195 Strategic Computing Initiative Appendix II
    Comments From the Department of Energy Now on p. 27. See comment
    16. Now on p. 27. See comment 17. Page 60
    GAO/RCED-99-195 Strategic Computing Initiative Appendix II
    Comments From the Department of Energy Now on p. 28. See comment
    18. Now on p. 28. See comment 12. Now on p. 28. See comment 12.
    Page 61                                   GAO/RCED-99-195
    Strategic Computing Initiative Appendix II Comments From the
    Department of Energy Now on p. 28. See comment 12. Now on p. 28.
    See comment 19. Now on p. 28. See comment 20. Page 62
    GAO/RCED-99-195 Strategic Computing Initiative Appendix II
    Comments From the Department of Energy Now on p. 31. See comment
    14. Now on p. 35. See comment 19. Page 63
    GAO/RCED-99-195 Strategic Computing Initiative Appendix II
    Comments From the Department of Energy Now on p. 35. See comment
    21. Page 64                                   GAO/RCED-99-195
    Strategic Computing Initiative Appendix II Comments From the
    Department of Energy Now on p. 38. See comment 22. Now on p. 38.
    See comment 23. Now on p. 40. See comment 24. Page 65
    GAO/RCED-99-195 Strategic Computing Initiative Appendix II
    Comments From the Department of Energy Now on p. 40. See comment
    25. Page 66                                   GAO/RCED-99-195
    Strategic Computing Initiative Appendix II Comments From the
    Department of Energy Now on p. 41. See comment 14. Page 67
    GAO/RCED-99-195 Strategic Computing Initiative Appendix II
    Comments From the Department of Energy The following are GAO's
    comments on the Department of Energy's letter, dated June 7, 1999.
    GAO's Comments    1. DOE's comments relating to our
    recommendations and our responses are discussed in the executive
    summary, and chapters 2 and 3, where the recommendations appear.
    2. The report text was revised to note that the Chiles Commission
    made this comment. The Commission was charged by the National
    Defense Authorization Acts of 1997 and 1998 to address the issue
    of maintaining nuclear weapons expertise. 3. We believe that the
    evidence presented in our report clearly demonstrates that long-
    and short-term planning efforts have been inconsistent and
    incomplete. For example, during the first 3 years of the program,
    annual implementation plans were prepared for some, but not all,
    components for only some of those years. In fact, in commenting on
    the draft of this report, DOE concurred with our recommendation
    that comprehensive planning and progress tracking systems were
    needed and cited improvements that they were making. 4. We agree
    that lower-level milestones can change for the reasons cited by
    DOE. However, the milestones discussed in this section of our
    report are the long-term, high-level program milestones.
    Furthermore, we believe that the examples cited in the report text
    and those shown in the chart in appendix I clearly demonstrate the
    many inconsistencies in DOE's strategic planning documents and the
    need to better document the many changes that have taken place. 5.
    The example referred to is just one of several cases in which DOE
    planning documents inconsistently reported the target completion
    date for a milestone. For example, the dates for the following
    milestones were reported differently in various documents:
    "Microaging," "3-D Casting Microstructure," "Macro/Micro Aging,"
    "Prototype 3-D Physics," and "Component Deterioration Model". In
    addition, DOE provided us with many versions of DOE planning
    documents and briefings that contained similar inconsistencies.
    Furthermore, in some of these documents, the dates were reported
    in calendar years and in others in fiscal years. In still others,
    program and laboratory officials were uncertain of whether the
    dates were in calendar or fiscal years. Page 68
    GAO/RCED-99-195 Strategic Computing Initiative Appendix II
    Comments From the Department of Energy 6. The DOE response to our
    referenced statement does not address the point of our statement.
    We state that laboratory milestones are reported inconsistently in
    the sources provided by DOE. The examples we use in our report
    illustrate this fact. DOE contends that transcription and printing
    errors caused labels and dates to change and that this problem
    "explains virtually all the contradictory information" about
    milestones detailed in the appendix to our report. We do not agree
    that these types of errors explain most of the problems we found
    in reviewing the documents provided to us. The point of our
    statement was that there is inconsistency among milestones found
    on the consolidated laboratory-level milestone chart and the two
    laboratory-level milestone charts used to support the consolidated
    chart. The examples we used show that milestones appear on the
    consolidated chart that are not found on the laboratory-level
    nuclear and nonnuclear milestone charts, which form the basis of
    the consolidated chart. Correspondingly, there are milestones on
    the laboratory-level nuclear and nonnuclear milestone charts that
    are not found on the consolidated chart. DOE explains that these
    charts were not meant to provide details about milestones and that
    they are visual supplements to written documentation that provides
    detailed information about the milestones. This statement is
    surprising to us. One of the largest problems we had in evaluating
    this program stemmed from the fact that there was almost no
    detailed information about these milestones. As we state in our
    report, the only milestone for which we found a detailed
    description was for the primary burn code scheduled for completion
    in 1999. Our review of implementation plans, strategic plans,
    reports from principal investigators meetings, and other documents
    did not reveal any of the written documentation DOE claims to
    support the milestones found on these charts. 7. DOE and
    laboratory officials told us that the linkages referred to here
    are not obvious and that only those with technical expertise could
    understand them. At our request, program officials tried for over
    4 months to prepare documents showing the linkages between
    projects and laboratory-level and higher-level milestones.
    Subsequently, program and laboratory officials told us that
    developing this information was a very useful exercise for them
    because the linkages were not always evident. DOE's comments about
    the principal investigators' meetings are not relevant to this
    point. However, as noted in another part of the report, there are
    few detailed reporting requirements for presentations at these
    meetings. In fact, we found that many of the accomplishments
    reported for individual projects Page 69
    GAO/RCED-99-195 Strategic Computing Initiative Appendix II
    Comments From the Department of Energy at the meetings were
    generally not tied back to the annual milestones established for
    those projects. In addition, there was no systematic assessment of
    how the results on individual projects are contributing to meeting
    either the laboratories' annual or program milestones. 8. We
    understand that project-level milestones and program-level
    milestones are not the same. However, we believe that DOE and the
    laboratories should be able to demonstrate how individual projects
    funded under the ASCI effort contribute to achieving annual
    laboratory-level and overall program-level milestones. This is
    essentially the "link" that we expected to see but did not find in
    program plans and related documents. In fact, in some cases
    program officials were unable to explain what that link was,
    although they stated that the projects were needed for the overall
    stockpile stewardship program. 9. DOE explained that they employ
    multiple research efforts and risk mitigation strategies to
    achieve program goals. We modified the report to clarify the use
    of multiple research approaches in the area of infrastructure
    development. We noted during our review, however, that these
    research efforts and risk mitigation strategies were poorly
    documented. DOE also explained the need to eventually select the
    most promising of these research efforts. However, DOE's claim
    that there are no set timetables for doing this seems inconsistent
    with the need to maintain the accelerated pace of the program. As
    stated in our report, program-planning documents should clearly
    identify research strategies, critical paths, and decision points.
    10. We have added wording to the body of the report to recognize
    the completion of DOE's review of the nonnuclear mechanics codes.
    However, as stated in the report, at this juncture, most program-
    level milestones remain undefined. 11. As noted in our report,
    there are few detailed reporting requirements for presentations at
    these meetings. In fact, we found that the accomplishments
    reported for individual projects at the meetings were generally
    not tied to the annual milestones established for those projects.
    In addition, there is no systematic assessment of how the results
    on individual projects are contributing to meeting either the
    laboratories' annual or program milestones. 12. We do not believe
    that we misinterpreted the information provided to us.
    Documentation provided to us by Sandia officials and the
    statements Page 70                                   GAO/RCED-99-
    195 Strategic Computing Initiative Appendix II Comments From the
    Department of Energy they made during our meeting with them on
    March 15, 1999, showed that the Sierra software development effort
    was 6 months or more behind schedule and that this slippage was
    affecting other software development efforts. Specifically, they
    cited the Fuego software development effort, which supports the
    "Abnormal Environment Thermal Assessment" milestone. They
    acknowledged that if any milestone in fiscal year 1999 is missed,
    it would most likely be this one because of the delays in Sierra.
    Sandia officials also stated that they had to redirect the Fuego,
    Coyote, Jas, and Pronto software development efforts until Sierra
    is ready. Whether the Sierra effort is 6 months behind schedule or
    whether the schedule itself has been slipped or, in DOE parlance,
    "rebaselined" by 6 months to recognize this slippage is irrelevant
    to the fact that the program schedule has been altered by delays
    in the Sierra project and that those delays are affecting other
    software efforts. 13. We believe that the statement in the report
    is correct as stands. We agree that implementation plans have
    improved over time. With respect to the quarterly reporting
    requirement for fiscal year 1999, it was added halfway through the
    fiscal year following our requests for information about progress
    towards meeting program milestones. We believe that this new
    quarterly reporting requirement is a positive improvement in the
    program's progress tracking system. 14. DOE provided additional
    information that did not require any response or changes to the
    report. 15. The actual sustained speed achieved on the sPPM code
    was 800 billion operations per second or one-half that reported on
    the Linpack code. While we recognize that performance on the sPPM
    was a "best effort" in the contract, we believe that it is
    misleading to repeatedly cite the higher performance on the
    Linpack code without recognizing the fact that the computer has
    not yet passed its contractually required acceptance test. With
    respect to our statement that the Linpack code was substantially
    easier than the sPPM code, our basis was comments made to us by
    program officials at a meeting on September 3, 1998. At that
    meeting, program officials stated that they were using the sPPM
    code instead of the Linpack scale because the Linpack is a "toy
    program" not useful at all for measuring the capabilities needed
    by the weapons program. For example, Linpack is not an industry
    standard, contains only a few lines of code, and does not measure
    important capabilities such as the use of input/output devices. On
    the other hand, sPPM, using hydrodynamic calculations, fully tests
    the platform regarding communications devices, as well as how Page
    71                                   GAO/RCED-99-195 Strategic
    Computing Initiative Appendix II Comments From the Department of
    Energy efficiently the software scales to 6144 processors, and the
    ability to incorporate I/O devices. 16. Wording was added to the
    report to clarify that, according to the ASCI Program Plan, a 10-
    fold increase was expected from improving software to take
    advantage of parallel processing techniques. 17. We revised the
    report text to clarify that we found little documentation of
    software development progress as measured against established
    milestones. 18. As pointed out in comment 12, the Sierra project
    is 6 months or more behind schedule, is affecting other software
    development efforts, and could potentially result in a milestone
    being missed during fiscal year 1999. Simply changing the baseline
    used to measure progress does not change these facts. 19. We were
    provided with this information at a meeting with senior program
    officials at Sandia on March 15, 1999. Also, see comment 12. 20.
    DOE's comment is discussed in the agency comments section of
    chapter 21. The National Academy of Sciences information, as
    referenced by DOE, does not address the tracking of costs on
    research programs and thus does not support DOE's management
    approach as DOE contends. 22. The report does recognize that the
    ASCI Red computer at Sandia is capable of operating at a
    theoretical peak speed of 1.8 trillion operations per second.
    While this is a significant accomplishment, DOE fails to mention
    that the ASCI Red computer was developed by a different vendor and
    uses an architecture that is different from the architecture in
    the computer being developed at Los Alamos. 23. Although things
    may have changed since 1998 as DOE states, our point is still
    valid. Operating each new system constructed (including the new 3
    trillion operations per second system at Los Alamos) as a fully
    integrated system poses challenges for the ASCI program. For
    example, ASCI's fiscal year 1999 implementation plan states, "The
    most critical issues affecting the successful implementation of
    the [Blue Mountain] system include hardware reliability and
    stability." Recognizing this fact, the contract statement of work
    for the Los Alamos computer included requirements to measure the
    stability of the system. However, as of March 1999, the Page 72
    GAO/RCED-99-195 Strategic Computing Initiative Appendix II
    Comments From the Department of Energy computer had not met these
    requirements, which clearly demonstrates this point. 24. We have
    revised the text to more accurately cite the Blue Ribbon Panel
    report. 25. We have revised the report text to more fully describe
    this survey effort. (141213)    Page 73
    GAO/RCED-99-195 Strategic Computing Initiative Ordering
    Information The first copy of each GAO report and testimony is
    free. Additional copies are $2 each. Orders should be sent to the
    following address, accompanied by a check or money order made out
    to the Superintendent of Documents, when necessary. VISA and
    MasterCard credit cards are accepted, also. Orders for 100 or more
    copies to be mailed to a single address are discounted 25 percent.
    Orders by mail: U.S. General Accounting Office P.O. Box 37050
    Washington, DC  20013 or visit: Room 1100 700 4th St. NW (corner
    of 4th and G Sts. NW) U.S. General Accounting Office Washington,
    DC Orders may also be placed by calling (202) 512-6000 or by using
    fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO
    issues a list of newly available reports and testimony.  To
    receive facsimile copies of the daily list or any list from the
    past 30 days, please call (202) 512-6000 using a touchtone phone.
    A recorded menu will provide information on how to obtain these
    lists. For information on how to access GAO reports on the
    INTERNET, send an e-mail message with "info" in the body to:
    [email protected] or visit GAO's World Wide Web Home Page at:
    http://www.gao.gov PRINTED ON     RECYCLED PAPER United States
    General Accounting Office           Bulk Rate Washington, D.C.
    20548-0001     Postage & Fees Paid GAO Permit No. G100 Official
    Business Penalty for Private Use $300 Address Correction Requested

*** End of document. ***