Department of Energy: Poor Management of Nuclear Materials Tracking
System Makes Success Unlikely (Letter Report, 08/03/95, GAO/AIMD-95-165).
The Energy Department (DOE) is attempting to develop a new tracking
system to replace its current system for monitoring U.S. imports and
exports of nuclear materials. DOE plans to discontinue the existing
system and begin operation of the replacement system in September 1995.
However, DOE's replacement system is being developed in an
undisciplined, poorly controlled manner that makes success unlikely.
Planning was inadequate and basic system development practices were not
followed. The upshot is that DOE has no guarantee that the replacement
system will produce accurate and timely reports before it accepts the
system and pays the subcontractor. DOE's disregard for basic system
development practices to ensure the accuracy and dependability of its
nuclear tracking system is inconsistent with the importance of the
Nuclear Materials Management and Safeguards System, which produces the
United States' officials records for tracking nuclear materials. It is
not in DOE's best interests, therefore, to disconnect the existing
system and replace it with an untested, undocumented new system. The
history of software development is littered with systems that failed
under similar circumstances.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: AIMD-95-165
TITLE: Department of Energy: Poor Management of Nuclear Materials
Tracking System Makes Success Unlikely
DATE: 08/03/95
SUBJECT: Systems design
Systems conversions
Strategic materials
Nuclear proliferation
Strategic information systems planning
Nuclear weapons
Testing
Exporting
Cost effectiveness analysis
Information systems
IDENTIFIER: DOE/NRC Nuclear Materials Management and Safeguards System
**************************************************************************
* This file contains an ASCII representation of the text of a GAO *
* report. Delineations within the text indicating chapter titles, *
* headings, and bullets are preserved. Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are *
* identified by double and single lines. The numbers on the right end *
* of these lines indicate the position of each of the subsections in the *
* document outline. These numbers do NOT correspond with the page *
* numbers of the printed product. *
* *
* No attempt has been made to display graphic images, although figure *
* captions are reproduced. Tables are included, but may not resemble *
* those in the printed version. *
* *
* A printed copy of this report may be obtained from the GAO Document *
* Distribution Facility by calling (202) 512-6000, by faxing your *
* request to (301) 258-4066, or by writing to P.O. Box 6015, *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time. *
**************************************************************************
Cover
================================================================ COVER
Report to the Ranking Minority Member, Committee on Governmental
Affairs, U.S. Senate
August 1995
DEPARTMENT OF ENERGY - POOR
MANAGEMENT OF NUCLEAR MATERIALS
TRACKING SYSTEM MAKES SUCCESS
UNLIKELY
GAO/AIMD-95-165
DOE Nuclear Tracking
Abbreviations
=============================================================== ABBREV
DOE - Department of Energy
NMMSS - Nuclear Materials Management and Safeguards System
Letter
=============================================================== LETTER
B-260569
August 3, 1995
The Honorable John Glenn
Ranking Minority Member
Committee on Governmental Affairs
United States Senate
Dear Senator Glenn:
This report responds to your February 1995 request that we review the
Department of Energy's (DOE) progress in developing a new nuclear
materials tracking system. This new system is intended to replace
the existing Nuclear Materials Management and Safeguards System
(NMMSS), which is the United States' official system for tracking
U.S. imports and exports of nuclear materials. In December 1994, we
reported on DOE's planning for the replacement NMMSS.\1 This report
discusses (1) what actions DOE has taken to implement the
recommendations in our previous report and (2) whether DOE is
adequately addressing key system development risks.
--------------------
\1 Nuclear Nonproliferation: U.S. International Nuclear Materials
Tracking Capabilities Are Limited (GAO/RCED/AIMD-95-5, December 27,
1994).
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
DOE has not implemented any of the recommendations contained in our
prior report and has no plans to do so. In December 1994, we
reported that DOE had not adequately planned the replacement NMMSS
and recommended that the Department determine users' requirements,
investigate alternatives, and conduct cost-benefit analyses before
proceeding further with the replacement system. However, DOE
continued with the system development without performing these steps
because it believed that its planning was sufficient and that it
would not be cost-effective to delay the replacement system. Due to
its lack of sound planning, DOE does not know if the system will
fulfill the needs of its major users or be cost-effective.
These planning risks are magnified by additional system development
risks that DOE is not adequately addressing. For example, the
subcontractor building the replacement NMMSS has not documented its
system development process. Because little system documentation
exists, and the contract does not require any interim deliverables
describing development progress before complete system delivery, DOE
cannot determine the status of the development effort. In addition,
the subcontractor did not place its software under configuration
management. Sound configuration management helps ensure that the
status of the system's software is known at all times and that, when
more than one programmer is making changes and updating the software,
all changes are consistent and are being written to the same software
version. Finally, DOE plans to pay for, install, and use the
replacement NMMSS without requiring that it pass acceptance testing.
Acceptance testing demonstrates that a system meets hardware,
software, and performance requirements and users' operational needs.
Without such testing, coupled with inadequate planning and the lack
of basic system development discipline and sound practices, DOE has
no assurance that the replacement NMMSS will ever perform as
intended.
BACKGROUND
------------------------------------------------------------ Letter :2
NMMSS is the United States' official nuclear materials tracking and
accounting system. NMMSS provides information on nuclear materials
to support both domestic programs and international nuclear policies.
Keeping track of the growing amount of nuclear materials is
especially important as a result of the breakdown of the Soviet Union
and increases in both domestic and international terrorism. Tracking
and accounting for the hundreds of tons of plutonium, highly enriched
uranium, and other nuclear materials that have accumulated are
important to help (1) ensure that nuclear materials are used only for
peaceful purposes, (2) protect nuclear materials from loss, theft, or
other diversion, (3) comply with international treaty obligations,
and (4) provide data to policymakers and other government officials
on the amount and location of nuclear materials.
The NMMSS database contains data on nuclear materials supplied and
controlled under international agreements, including U.S.-supplied
international nuclear materials transactions, foreign contracts,
import/export licenses, government-to-government approvals, and other
DOE authorizations, such as authorizations to retransfer
U.S.-supplied materials between foreign countries. NMMSS also
maintains and provides DOE with information on domestic production
and materials management, safeguards, physical accountability,
financial and cost accounting, and other data related to nuclear
materials accountability and safeguards for Nuclear Regulatory
Commission licensees.
DOE and the Nuclear Regulatory Commission cosponsor NMMSS, and it is
managed and operated by a DOE contractor--Martin Marietta Energy
Systems, Incorporated. NMMSS has been used to account for U.S.
imports and exports of nuclear materials since 1977.
Because the existing NMMSS is an older system, DOE decided to replace
and modernize it. The existing NMMSS is housed on a mainframe using
unstructured COBOL code. Performing modifications on the existing
NMMSS and designing custom reports are difficult because of the
volume and complexity of the code. Accordingly, DOE's Office of
Nonproliferation and National Security, which is responsible for
operating NMMSS, tasked the Lawrence Livermore National Laboratory
with developing a new NMMSS. Livermore hired a subcontractor to
perform this task in February 1994 and assigned a program manager to
oversee the effort. In April 1994, Livermore formed a technical
advisory committee, composed of senior computer scientists and
material control and accountability specialists, to assist the
program manager in overseeing the system development.
The replacement NMMSS is scheduled to become operational on September
1, 1995. Martin Marietta is scheduled to discontinue operation of
the existing NMMSS during September 1995.
SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :3
To address our objectives, we reviewed the replacement NMMSS
contract, transition plan, test plan, and various other draft system
documents. We requested documentation on the status of the system
coding and testing; however, none was available for our review. We
also analyzed documentation provided to us by Lawrence Livermore's
technical advisory committee on the subcontractor's development
efforts. In addition, we analyzed documentation from various user
groups on their concerns with the NMMSS development. We analyzed DOE
Order 1330.1D, Computer Software Management, to determine its
applicability to this project and whether or not it was being
followed.
We interviewed DOE officials in the Office of Nonproliferation and
National Security concerning actions taken to implement the
recommendations in our previous report and the status of the NMMSS
development. We also interviewed the NMMSS program manager, members
of Lawrence Livermore's technical advisory committee, contract
officials at DOE and Lawrence Livermore, and the NMMSS
subcontractor's lead programmers, system engineer, and project
managers to determine the status of the system development. In
addition, we interviewed officials in DOE's Defense Programs
Office--the biggest user of NMMSS information--on their concerns
about the replacement NMMSS development.
We performed our work between February 1995 and May 1995, in
accordance with generally accepted government auditing standards.
Our work was primarily done at DOE's headquarters in Washington,
D.C., and its offices in Germantown, Maryland; at Lawrence Livermore
National Laboratory in Livermore, California; and at the
subcontractor's facility in Norcross, Georgia. The Department of
Energy provided written comments on a draft of this report. These
comments are presented and evaluated in the report, and are reprinted
in appendix I.
DOE IS PURSUING THE REPLACEMENT
NMMSS WITHOUT ADDRESSING
CRITICAL PLANNING ISSUES
------------------------------------------------------------ Letter :4
In December 1994, we reported that DOE did not adequately plan the
development effort for the replacement NMMSS.\2 For example, DOE did
not follow sound system development practices such as identifying and
defining users' needs and adequately exploring design alternatives
that would best satisfy these needs in the most economic fashion.
Accordingly, we recommended that DOE determine users' requirements,
investigate alternatives, conduct cost-benefit analyses, and develop
a plan to meet identified needs before investing further resources in
the replacement NMMSS.
In its official response to the recommendations in our prior report,
DOE stated that it did not concur with our recommendations and that
it did not believe it would be cost-effective to delay its effort to
transition from the existing system to the new system. Further, in
commenting on a draft of this report, the Acting Director of the
Office of Nonproliferation and National Security stated that DOE's
planning was sufficient. However, because of DOE's lack of basic
planning, it does not know if the system will fulfill the needs of
its major users or be cost-effective.
--------------------
\2 GAO/RCED/AIMD-95-5, December 27, 1994.
SYSTEM DEVELOPMENT RISKS ARE
HIGH
------------------------------------------------------------ Letter :5
Adhering to generally accepted system development practices helps to
ensure that information systems perform as desired.\3
These practices include (1) generating clear, complete, and accurate
documentation throughout the system development process, (2) placing
the software development under configuration management, and (3)
ensuring that the system successfully completes acceptance testing
prior to becoming operational. However, because DOE has not required
the subcontractor to follow any of these practices for the
replacement NMMSS, the Department does not know how much of the
system development is completed and whether the part that is
completed performs as required. As a result, the risk of system
failure is inordinately high.
--------------------
\3 Such practices are discussed in Defense Acquisition Management
Policies and Procedures (DOD 5000.2, February 1991); Systems
Engineering Management Guide (Defense Systems Management College,
January 1990); Capability Maturity Model for Software
(CMU/SEI-91-TR-24, ESD-TR-91-24, August 1991); Key Practices of the
Capability Maturity Model (CMU/SEI-91-TR-25, ESD-TR-91-25, August
1991); and Defense System Software Development (DOD-STD-2167A,
February 1988).
LITTLE SYSTEM DEVELOPMENT
DOCUMENTATION EXISTS
---------------------------------------------------------- Letter :5.1
DOE Order 1330.1D, Computer Software Management, requires that the
development of a system be documented so that, among other things,
the status of the system is known at all times. Documentation, such
as the results of system testing and the tracking of source code as
it changes, allows program managers to review the development's
progress and determine if requirements are being met.
The subcontractor developing the replacement NMMSS could not provide
any system documentation--software specifications, system
requirements, results of formal reviews (e.g.,
system/preliminary/critical design) or informal system testing
reports, operational procedures, quality assurance checklists, or
project tracking reports. Because little system documentation
exists, and the contract does not require any interim deliverables
that measure system performance, DOE does not know the status of the
system development. In addition, members of Livermore's technical
advisory committee told us they have been unable to obtain the
documentation they needed to determine the status of the development
effort. As a result, the committee said it could not accurately
determine such factors as the number of lines of code in the system.
In fact, the advisory committee could only estimate system size in
very gross terms--between 10,000 and 100,000 lines of code.
DOE officials agreed that the development effort is largely
undocumented and stated that DOE historically has not enforced its
own regulations requiring system documentation. At the conclusion of
our review and in commenting on a draft of this report, DOE officials
told us that they will begin to require such documentation for the
replacement NMMSS.
CONFIGURATION MANAGEMENT WAS
NOT USED
---------------------------------------------------------- Letter :5.2
A successful system development project should include a software
configuration management plan that clearly defines the procedures for
identifying, accounting for, and reporting on changes to software
items that are under configuration control. Configuration management
is necessary throughout the life cycle of a software project because
it provides (1) a control mechanism to ensure that the software
status is accurately known at all times and (2) a baseline for system
developers and testers.
Although the subcontractor developed a software configuration
management plan for the replacement NMMSS, no software had been
placed under configuration control. As a result, DOE does not know
what version of the software is current, which versions of the
software have been tested, what problems were identified during
testing, and what corrections are being made. Developing software
without configuration management frequently results in projects that
are delivered late, exceed budget, and perform poorly.
Officials in DOE's Office of Nonproliferation and National Security
agreed that the replacement NMMSS software had not been placed under
configuration management at the time of our exit conference. The
officials stated that, until recently, they had not required the use
of configuration management on software development projects. In its
written comments on a draft of this report, DOE stated that the
replacement NMMSS is now being placed under configuration control.
DOE DOES NOT PLAN TO
ADEQUATELY TEST NMMSS
---------------------------------------------------------- Letter :5.3
During acceptance testing, tests are performed to determine if a
system will meet its hardware, software, performance, and user
operational requirements. Acceptance testing is usually performed by
the system developer and witnessed by an independent verification and
validation group, which includes system users. Such testing is
important to determine if the new system performs as required.
The previous implementation schedule for the replacement NMMSS called
for acceptance testing and 2 months of parallel operation with the
existing NMMSS. In addition, in a January 1994 memorandum, an
official from DOE's Office of Nonproliferation and National Security
stated that the replacement NMMSS would not be made operational until
"it has been demonstrated that the new system is capable of meeting
present and new customer needs and requirements." Adhering to this
position on testing the replacement NMMSS would have greatly reduced
system risks.
In January 1995, DOE changed its position and decided to make the
replacement NMMSS operational without performing acceptance testing.
DOE officials stated that this decision was made to avoid the cost of
simultaneously funding both the existing and replacement systems.
Instead, DOE plans to perform what it is calling "system testing" on
a subset of system reports--87 of approximately 500 reports. While
DOE stated that these 87 reports were selected based on users' needs,
it could not produce documentation to validate this statement.
The only system testing at the time of our review was the informal
testing that the subcontractor stated it had performed. However, the
subcontractor could not provide documentation on either its test
plans or the test results. In its written comments on a draft of
this report, DOE officials stated that system test procedures have
now been written and approved.
In addition, parallel operations with the existing NMMSS are not
scheduled. During parallel operations, both systems would perform
all required functions, and then outputs would be compared to ensure
that the replacement system is producing accurate reports. Because
the replacement NMMSS will replicate the functions of the existing
NMMSS, a period of parallel operations is especially important.
Without parallel processing, DOE is introducing additional risk that
the replacement system will not perform all functions of the existing
system or, more importantly, that the information produced by the
replacement system will not be accurate. As a result, DOE cannot
guarantee its users that the information they need from NMMSS to do
their jobs will continue to be available. NMMSS users told us that
information they get from the existing NMMSS within hours could take
weeks or months to gather if they cannot obtain it from the new NMMSS
or if they cannot be sure that the information in the new NMMSS is
accurate.
CONCLUSIONS
------------------------------------------------------------ Letter :6
DOE has stated that it will discontinue the existing system on
September 1, 1995, and begin operation of the replacement NMMSS
without acceptance testing. However, DOE's replacement NMMSS is
being developed in an undisciplined, poorly controlled manner that
makes success unlikely. Planning was inadequate and basic system
development practices are not being followed. As a result, DOE will
not know if the replacement NMMSS will produce the accurate and
timely reports needed to meet users' needs before it accepts the
system and pays the subcontractor.
DOE's disregard for basic system development practices necessary to
ensure the accuracy and dependability of its nuclear tracking system
is inconsistent with the importance of NMMSS, which provides the
United States' official record for tracking nuclear materials. It is
not in DOE's best interests, therefore, to disconnect the existing
NMMSS and replace it with an untested, undocumented new system. The
history of software development is littered with systems that failed
under similar circumstances.
RECOMMENDATIONS
------------------------------------------------------------ Letter :7
We recommend that the Secretary of Energy immediately terminate any
further development of the replacement NMMSS. Further, as we
recommended in our December 1994 report, the Secretary should direct
the Office of Nonproliferation and National Security to determine
users' requirements, investigate alternatives, and conduct
cost-benefit analyses before proceeding with any plan to develop a
replacement NMMSS.
If, after thorough planning, the Office proceeds with plans to
develop a new NMMSS, we recommend that it follow generally accepted
system development practices. In the interim, we recommend that DOE
continue using the existing NMMSS system until the above
recommendations are addressed.
AGENCY COMMENTS AND OUR
EVALUATION
------------------------------------------------------------ Letter :8
The Department of Energy provided written comments on a draft of this
report. Their comments are summarized below and reproduced in
appendix I.
The Department of Energy agreed with the need for systems development
documentation, configuration management, and adequate testing.
However, the Department did not concur with our assessment of its
analyses and planning for the system development effort, or with our
recommendation that it terminate the system development until users'
requirements, alternatives, and cost-benefit analyses have been
performed. DOE stated that its planning was adequate because it is
converting an existing system from an unstructured language to a
structured, fourth generation language, rather than developing a new
system.
We disagree. Without sound analyses or planning, DOE does not know
that "converting an existing system" is a cost-effective way to meet
its needs. Furthermore, as our report discusses, DOE is implementing
this unsupported approach in an unsatisfactory manner. Therefore,
DOE should discontinue its current effort and perform users'
requirements, alternatives, and cost-benefit analyses before
proceeding.
---------------------------------------------------------- Letter :8.1
As arranged with your office, unless you publicly announce the
contents of this report earlier, we plan no further distribution
until 30 days from the date of this letter. At that time, we will
provide copies of this report to the Secretary of Energy; the
Director, Office of Management and Budget; appropriate congressional
committees; and other interested parties. Copies will also be made
available to others upon request.
Please call me at (202) 512-6253 if you or your staff have any
questions. Major contributors to this report are listed in appendix
II.
Sincerely yours,
Joel C. Willemssen
Director, Information Resources
Management/Resources, Community,
and Economic Development
(See figure in printed edition.)Appendix I
COMMENTS FROM THE DEPARTMENT OF
ENERGY
============================================================== Letter
(See figure in printed edition.)
(See figure in printed edition.)
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II
ACCOUNTING AND INFORMATION
MANAGEMENT DIVISION, WASHINGTON,
D.C.
Valerie C. Melvin, Assistant Director
Keith A. Rhodes, Technical Assistant Director
Suzanne M. Burns, Evaluator-in-Charge
Linda J. Lambert, Senior Auditor
*** End of document. ***