Weather Forecasting: Recommendations to Address New Weather Processing
System Development Risks (Letter Report, 05/13/96, GAO/AIMD-96-74).
GAO testified in February 1996 on the National Weather Service's (NWS)
Advanced Weather Interactive Processing System, the cornerstone of NWS'
$4.5 billion modernization program. (See GAO/T-AIMD-96-47.) This report
discusses the recommendations GAO made during its testimony before
Congress. These recommendations, if fully implemented, will strengthen
NWS' ability to achieve a fair return on its investment in the system.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: AIMD-96-74
TITLE: Weather Forecasting: Recommendations to Address New Weather
Processing System Development Risks
DATE: 05/13/96
SUBJECT: ADP procurement
Systems design
Systems conversions
Information resources management
Weather forecasting
Computer software verification and validation
Requirements definition
Cost analysis
Concurrency
Information systems
IDENTIFIER: NWS Advanced Weather Interactive Processing System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Secretary of Commerce
May 1996
WEATHER FORECASTING -
RECOMMENDATIONS TO ADDRESS NEW
WEATHER PROCESSING SYSTEM
DEVELOPMENT RISKS
GAO/AIMD-96-74
Weather Forecasting Recommendations
(511413)
Abbreviations
=============================================================== ABBREV
AWIPS - Advanced Weather Interactive Processing System
NOAA - National Oceanic and Atmospheric Administration
NWS - National Weather Service
Letter
=============================================================== LETTER
B-271478
May 13, 1996
The Honorable Mickey Kantor
The Secretary of Commerce
Dear Mr. Secretary:
On February 29, 1996, we testified on the National Weather Service's
(NWS) Advanced Weather Interactive Processing System (AWIPS)--the
cornerstone of NWS' $4.5 billion modernization program--before the
Subcommittee on Energy and Environment, House Committee on Science.\1
This report officially transmits recommendations we made as part of
our testimony which, if implemented, will strengthen NWS' ability to
achieve a fair return on its AWIPS investment.
First, as we also recommended in our February 1996 report\2 on AWIPS,
NWS should (1) expand ongoing AWIPS requirements review activities to
include validation that proposed capabilities are justified on the
basis of mission impact and (2) implement only those capabilities
that are validated. These recommendations are grounded in NWS'
long-standing assertion that the modernization and its component
systems will produce significant public service improvements--namely,
better forecasts at reduced costs. To achieve these goals, NWS has
specified that AWIPS must provide about 450 high-order capabilities,
such as the ability to execute certain models or display data in
certain formats and colors. These high-order capabilities are
composed of about 22,000 individual system requirements.
We testified that while NWS has done a commendable job of
incorporating the views of the forecasting community in defining
AWIPS' capabilities, it has not demonstrated that all proposed
capabilities will produce expected mission impacts. We have found
that successful public and private organizations tie decisions on
information technology investments to explicit and quantifiable
mission improvements.\3 By failing to do the same, NWS risks building
either more or less into AWIPS than necessary, which, either way,
translates into lowering AWIPS' return-on-investment.
Second, we testified that NWS has wisely chosen to break AWIPS
software development into increments, which is a widely accepted
means of reducing software development risks. However, we believe
that NWS' approach to implementing this strategy makes AWIPS
development unnecessarily risky because NWS chose to overlap
development of these builds, beginning development of a new build
before the prior build is fully stabilized. In doing so, it
increased the chances of potential defects being passed along
module-to-module. Accordingly, we recommended that NWS ensure that
each build is fully tested and all material defects are corrected
before software development associated with the next build begins.
Third, we recommended in our testimony that NWS establish a software
quality assurance program to increase the probability of delivering
promised AWIPS capability on time and within budget. While NWS has
reported a number of software development process improvements,
including preparing a software development plan, we believe its
failure to establish a software quality assurance program to monitor
NWS' in-house development processes and products unnecessarily puts
AWIPS' current cost, schedule, and performance goals in jeopardy.
Such a program would provide independent assurance that software
development processes and products meet prescribed standards and that
any identified deficiencies in process or product are brought to
management's attention sooner rather than later, thus minimizing
costly system rework. Employing a software quality assurance program
is a common practice among public and private organizations with
solid reputations for producing quality software.
Finally, in our testimony, we recommended that NWS obtain an
independent assessment of the cost to develop and deploy AWIPS. We
testified that NWS currently does not have reliable cost information
on AWIPS development and deployment costs and that its current
estimate of $525 million, which has not changed for more than 1 year,
omits certain costs and is thus understated. For example:
-- The $525 million estimate includes neither known deployment
phase contract cost increases nor NWS internal program
management costs resulting from NWS' 1995 decision to extend
AWIPS deployment 1 year. In our discussions with NWS officials,
contract costs associated with the extension were estimated to
be about $9 million, while NWS program management costs for the
additional year had yet to be estimated.
-- The estimate omits a known cost increase in the development
phase contract of about $10 million. While NWS officials stated
that this increase will be offset by reductions in the
deployment phase contract, the terms of the deployment phase
have yet to be renegotiated.
-- The estimate omits the cost of AWIPS product improvements that
were recently identified and are planned for 1999. These
improvements are estimated to cost about $2 million.
-- The $525 million estimate, when originally derived, included
about $49 million for future development and deployment phase
contract modifications--a $34 million decrease from the amount
NWS included in its 1992 project cost estimate. Since 1994,
however, NWS officials told us that more than $42 million of the
$49 million in reserve has been allocated to known contract
costs, leaving relatively meager reserves for inevitable
modifications. Omitting adequate reserves at this stage in
AWIPS' development cycle is inconsistent with prudent software
engineering practices. According to software engineering
experts, the size of a system's software at this stage in its
development life cycle could grow by as much as 50 percent.
As we testified, NWS has an opportunity with AWIPS to substantially
improve its weather forecasting capability. If the recommendations
we made in our testimony and in this report are implemented, NWS can
position itself to take full advantage of this opportunity.
--------------------
\1 Weather Forecasting: New Processing System Faces Uncertainties
and Risks (GAO/T-AIMD-96-47, February 29, 1996).
\2 Weather Forecasting: NWS Has Not Demonstrated That New Processing
System Will Improve Mission Effectiveness (GAO/AIMD-96-29, February
29, 1996).
\3 Executive Guide: Improving Mission Performance Through Strategic
Information Management and Technology (GAO/AIMD-94-115, May 1994).
AGENCY COMMENTS AND OUR
EVALUATION
------------------------------------------------------------ Letter :1
In its written comments on a draft of this report, the National
Oceanic and Atmospheric Administration (NOAA) generally disagreed
with our conclusions and recommendations. A summary of NOAA's
specific comments and our evaluation of them follows. We have
incorporated other NOAA comments in the report where appropriate.
NOAA disagreed with our first recommendation to (1) expand ongoing
AWIPS validation activities to include justifying proposed
capabilities on the basis of mission impact and (2) implement only
mission-justified capabilities. According to NOAA, completed and
ongoing requirements reviews and risk reduction activities as well as
operational test and evaluation of each AWIPS release are sufficient
to ensure that unneeded AWIPS capabilities are revised or not
implemented. In addition, NOAA added its view that validating
capabilities to mission impact is not efficient or cost-effective and
would cause further delay and expense to the program. We disagree.
While we reiterate our commendation for AWIPS requirements reviews
and risk reduction activities to date and support plans to prototype
future system releases, the fact remains that none of these
activities were or are intended to demonstrate the mission impact of
AWIPS capabilities. Moreover, waiting until the operational test and
evaluation stage of each AWIPS release is too late to assess mission
impact because by this time considerable sums of money will have
already been invested in the capabilities' development.
Additionally, while validating capabilities to mission impact would
require some resource investment now, it could also prevent time and
money being spent unnecessarily on developing AWIPS capabilities that
promise little in terms of mission improvement. Thus, we continue to
support our recommendation.
NOAA did not comment specifically on our second recommendation to
fully test and correct all material defects in each AWIPS build
before software development associated with a succeeding build
begins. However, NOAA stated that its plans for overlapping AWIPS'
builds does not introduce unnecessary risks and stated that judicious
overlapping is efficient and appropriate for those builds that do not
need to wait for the previous build to be "frozen." Further, while
acknowledging the potential risks of AWIPS build overlaps, it noted
two factors that mitigate these risks. First, NOAA stated that there
is minimal overlap associated with builds 1 and 2 and that the two
areas in which overlap occurs (data modeling and communications
software) are inherently stable because of early development and
demonstration activities. Second, NOAA stated that development in
the later builds is primarily hydrometeorological applications and
that these applications will not interact directly with each other.
We agree that overlapping builds can be appropriate and efficient,
and that "freezing" one build before moving to the next is not always
efficient. Further, we are not questioning the degree of interaction
among hydrometeorological applications. Our concern is that AWIPS
plans call for extending or adding to baseline applications, such as
those that process radar or satellite products for display on AWIPS
monitors, in succeeding builds before these baselines have even begun
formal testing--much less before the build has been stabilized--by
removing all material defects. By not doing so, NWS runs the risk of
introducing defects on top of unresolved defects. Thus, we continue
to support our recommendation that NWS fully test each AWIPS build
and correct all material defects before beginning software
development associated with the next build.
Although NOAA did not directly address our third recommendation to
establish an AWIPS quality assurance program, it agreed that software
quality assurance is important for NWS in-house software development
(i.e., hydrometeorological applications), and it stated that NWS has
hired a computer specialist for the AWIPS program manager's staff to
oversee the government's software development activities. NOAA also
stated that NWS is following accepted software development practices
and that an independent review team draft report gives a "generally
favorable impression of the current development methodology." Despite
NWS' recent actions, it is important to implement our recommendation
for two reasons. First, hiring a computer specialist falls far short
of our recommendation because such action does not provide either the
structure or content of a disciplined, thorough, or independent
quality assurance program. Second, since NOAA provides no evidence
or details of actions taken to follow "accepted software development
practices" and no basis for the "generally favorable impression" of
its development methodology, these points are unconvincing. Thus, we
continue to support our recommendation.
Finally, NOAA disagreed with our fourth recommendation to obtain an
independent assessment of the cost to develop and deploy AWIPS,
adding that it plans to reevaluate its current cost estimate of $525
million in August 1996 when the deployment phase contract is
renegotiated and that it anticipates no significant increases in the
current estimate. NOAA further stated that developing a new estimate
at this time would add little value to program and budget planning.
In defending the validity of its current estimate of $525 million,
NOAA also dismissed each of the omitted cost items cited in our
report for reasons cited below. We disagree that a new estimate
would add little value, particularly in light of expressed interest
by members and staff of the Subcommittee on Energy and Environment,
House Committee on Science, in having an up-to-date, reliable
estimate upon which to base authorization decisions. Also, given
NOAA's dismissal of all the omitted cost items cited in our report,
our recommendation remains valid. NOAA's comments on each of the
omitted cost items cited in our report and our response follow.
-- NOAA stated that the $9 million increase associated with
extending the deployment phase by 1 year should not be added to
the AWIPS project cost estimate because NWS is committed to
completing the deployment phase in 24 months as originally
planned. This contradicts NWS documentation dated February 9,
1996, stating that the AWIPS deployment phase would be extended
1 year and would result in an additional $9 million, and NOAA
has provided no documentation rescinding this plan or
reassessing its cost.
-- NOAA repeats its claim that the omission of a $10 million cost
increase to the development phase contract will be offset by a
corresponding decrease in the deployment phase contract.
However, as stated in our report, since the deployment phase
contract has yet to be renegotiated, there is no basis for
NOAA's claim.
-- NOAA stated that the $2 million omission for recently identified
product improvements should not be included in the AWIPS cost
estimate because this AWIPS enhancement will be developed after
the deployment phase is completed. We disagree. Known system
development costs cannot rationally be excluded from cost
estimates. Moreover, the AWIPS project cost estimate already
includes numerous other AWIPS product improvements and thus
excluding the cost of this product improvement would be
inconsistent with NWS' own practice.
-- NOAA acknowledged that the $525 million estimate provides a
relatively small reserve for contract modifications needed to
address such things as code growth. However, NOAA contends that
inherent conservatism in the deployment phase component of the
estimate will ultimately produce more contingency money than is
currently apparent in the estimate. Also, NOAA stated that it
does not expect major modifications for the remainder of the
development phase. Despite NOAA's optimism, it admits that
contract costs (development and deployment) are the greatest
variable in the current cost estimate. Further, prudent
software engineering practices recommend planning for as much as
50 percent growth in software size for systems at AWIPS' stage
of development. Finally, federal agencies' past performance in
acquiring major, software intensive systems is routinely
characterized by high cost overruns and significant schedule
slippage. Thus, we find no basis for NOAA to conclude that
contingency money will appear before the deployment phase
contract is renegotiated.
The full text of NOAA's comments is provided in appendix I.
---------------------------------------------------------- Letter :1.1
This report contains recommendations to you. The head of a federal
agency is required by 31 U.S.C. 720 to submit a written statement on
actions taken on these recommendations to the Senate Committee on
Governmental Affairs and the House Committee on Government Reform and
Oversight not later than 60 days after the date of this report. A
written statement also must be sent to the House and Senate
Committees on Appropriations with the agency's first request for
appropriations made more than 60 days after the date of this report.
We are sending copies of this letter to each member of the
Subcommittee on Energy and Environment, House Committee on Science,
and other interested parties. We will send copies to others upon
request. If you have questions or wish to discuss the issues in this
report further, please contact me at (202) 512-6240. Major
contributors to this report are listed in appendix II.
Sincerely yours,
Jack L. Brock, Jr.
Director, Information Resources Management/
Resources, Community, and Economic Development
(See figure in printed edition.)Appendix I
COMMENTS FROM THE NATIONAL OCEANIC
AND ATMOSPHERIC ADMINISTRATION
============================================================== Letter
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
The following are GAO's comments on the National Oceanic and
Atmospheric Administration's letter dated April 17, 1996.
GAO COMMENTS
1. Discussed in the "Agency Comments and Our Evaluation" section of
the report.
2. Discussion of this point deleted.
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II
ACCOUNTING AND INFORMATION
MANAGEMENT DIVISION, WASHINGTON,
D.C.
Rona B. Stillman, Chief Scientist for Computers and
Telecommunications
Randolph C. Hite, Assistant Director
Keith A. Rhodes, Technical Assistant Director
David A. Powner, Evaluator-in-Charge
Robert C. Reining, Information Systems Analyst
Colleen M. Phillips, Information Systems Analyst
Cristina T. Chaplain, Communications Analyst
*** End of document. ***