Weather Forecasting: New Processing System Faces Uncertainties and Risks
(Testimony, 02/29/96, GAO/T-AIMD-96-47).

GAO discussed the National Weather Service's (NWS) Advanced Weather
Interactive Processing System (AWIPS). GAO found that: (1) NWS has
involved its forecasters in defining AWIPS requirements, organizing
AWIPS to respond to system changes, and strengthening its in-house
software development; (2) NWS risks wasting money on AWIPS because it
has not been able to reduce operational costs or improve its forecasts;
(3) NWS ability to meet its AWIPS commitments is being jeopardized by a
risky development approach that develops one software increment before
the next, complicates government and contractor accountability for
system integration and performance problems, and omits software quality
assurance; and (4) NWS has not realized expected staffing reductions as
a result of its modernization program because it underestimated the
staff needed to operate and maintain the new systems.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  T-AIMD-96-47
     TITLE:  Weather Forecasting: New Processing System Faces 
             Uncertainties and Risks
      DATE:  02/29/96
   SUBJECT:  Weather forecasting
             Systems design
             Systems architecture
             Requirements definition
             Reductions in force
             ADP procurement
             Computer software
             Quality assurance
             Cost control
             Systems conversions
IDENTIFIER:  NWS Advanced Weather Interactive Processing System
             NWS Modernization Program
             NWS Next Generation Weather Radar
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Before the Subcommittee on Energy and the Environment, Committee on
Science, House of Representatives

For Release on Delivery
Expected at
10:00 a.m.
Thursday,
February 29, 1996

WEATHER FORECASTING - NEW
PROCESSING SYSTEM FACES
UNCERTAINTIES AND RISKS

Statement of Jack L.  Brock, Jr.
Director, Information Resources Management/Resources, Community, and
Economic Development
Accounting and Information Management Division

GAO/T-AIMD-96-47

GAO/AIMD-96-47T


(511409)


Abbreviations
=============================================================== ABBREV


============================================================ Chapter 0

Mr.  Chairman and Members of the Subcommittee: 

Thank you for the opportunity to testify on the National Weather
Service's (NWS) Advanced Weather Interactive Processing System
(AWIPS)--the "linchpin" of NWS' $4.5 billion modernization program. 
The weather service modernization entails building and putting into
place vastly more capable weather observing systems, such as Doppler
radars, that will feed a network of sophisticated AWIPS workstations. 
This AWIPS network, in turn, is to use the observations in
combination with national weather modeling results to aid forecasters
in making and communicating localized weather predictions.  NWS
estimates that AWIPS will cost $525 million to fully deploy by 1999. 

Mr.  Chairman, today we are the bearers of both good and bad news,
some of which we pointed out last February during testimony before
this Subcommittee.  The good news is that NWS has done a stellar job
of involving its forecaster community in defining what AWIPS should
be.  Heeding the advice of a 1994 independent review team, NWS has
also reorganized the program to address fundamental management
impediments to establishing an acceptable system design and moving
the program forward.  Additionally, NWS has acted on some of our
recent recommendations to strengthen its in-house software
development capability. 

The bad news is that NWS runs the risk of wasting money on AWIPS
capabilities that may not be needed because it has yet to link all
planned capabilities to promised mission improvements, such as
cheaper operations and better forecasts.  Additionally, NWS' ability
to meet its AWIPS commitments is being jeopardized by a risky
development approach that (1) prematurely begins developing one
software increment before the previous increment is stabilized, (2)
complicates government versus contractor accountability for
inevitable system integration and performance problems, and (3) omits
a vital development process known as software quality assurance. 
Also, NWS expectations for staffing reductions from the modernization
continue to shrink. 


   AWIPS:  A BRIEF DESCRIPTION
---------------------------------------------------------- Chapter 0:1

AWIPS is to serve as both a weather decision support system and
communication system.  More specifically, AWIPS is to support
forecasters in graphically integrating and analyzing the volumes of
weather observations and products that form the basis for decisions
on each day's forecasts and warnings.  It is also to provide the
national communications infrastructure for NWS field offices and
national centers, connecting them not only to each other but also
linking them to NWS' diverse customer base.  Through AWIPS, NWS
expects to tap a reservoir of data from its new observing systems
that the aging processing and communication system currently in
place, known as AFOS, cannot.\1

AWIPS' progress to date has been uneven.  Despite early successes in
effectively involving forecasters in AWIPS requirements analysis and
definition activities and demonstrating the technical feasibility of
AWIPS functions, AWIPS made little progress during 1993 and 1994
because of an impasse with the development contractor over the AWIPS
design and shortcomings in NWS' program management.  Acting on the
recommendations of an independent review team, NWS was able to move
the program forward in 1995 by restructuring the program and
renegotiating the development phase of the AWIPS contract. 
Renegotiation of the deployment phase of the contract is ongoing. 

Under the restructuring, NWS assumed responsibility for developing
all AWIPS hydrology and meteorology application software.  Also,
development of the system was divided into a series of seven
increments of increasing functional capability.  Thus far, the first
increment has been installed at three sites to gain experience in
developing, testing, deploying, and operating a very limited version
of AWIPS.  Development of the second increment is underway. 

NWS' current project cost estimate for AWIPS, which was first
reported in December 1994 and according to NWS is still valid, is
$525 million or roughly $58 million more than its previous official
estimate done in October 1992.  The current schedule calls for AWIPS
deployment to be completed in 1999 or 1 year later than NWS projected
in 1992. 


--------------------
\1 AFOS stands for Automation of Field Operations and Services. 


   HOW MUCH AWIPS CAPABILITY IS
   ENOUGH? 
---------------------------------------------------------- Chapter 0:2

Since its inception in the early 1980s, NWS has justified the
modernization, and its component systems, on the grounds that it will
produce significant "service-to-the-public" improvements--namely,
better forecasts at less cost.  To facilitate attaining these goals,
NWS has specified that AWIPS must provide about 450 high-order
capabilities, such as the ability to execute certain models or
display data in certain formats and colors.  All told, these
high-order capabilities are composed of about 22,000 separate system
requirements. 

In defining AWIPS' capabilities, NWS effectively solicited and
incorporated the views of the forecaster community.  In this regard,
it performed multiple requirements analyses and reviews and it
effectively employed system prototyping to ensure that forecaster
perspectives were heard and understood. 

However, the true measure of AWIPS' value is not that forecasters
like it, but rather that it contributes significantly to improving
NWS' mission performance.  In studying the practices of leading
public and private sector organizations, we found that successful
organizations' system investments are based on explicit and
quantifiable mission improvements.  By doing so, these organizations
know that investing in system capabilities is justified and will make
a difference in mission outcomes, such as service delivery or product
quality.\2

Unfortunately, NWS has not demonstrated that the package of
capabilities it envisions for AWIPS will enable it to make better
forecasts, operate fewer field offices, and reduce staffing levels,
leaving the question wide open as to whether AWIPS, as envisioned, is
the "right thing." In our view, unless NWS takes advantage of ongoing
and planned AWIPS prototyping to validate that proposed capabilities
produce measurable mission improvements, it runs the risk of wasting
taxpayer money. 


--------------------
\2 Executive Guide:  Improving Mission Performance Through Strategic
Information Management and Technology (GAO/AIMD-94-115, May 1994). 


   AWIPS SOFTWARE DEVELOPMENT
   RISKS REMAIN
---------------------------------------------------------- Chapter 0:3

Our recent work on the NWS modernization has identified several AWIPS
development risks, particularly with regard to the system's software. 
Despite the fact that NWS has moved to mitigate some of these risks,
others remain that require careful management attention and action. 
The risks are (1) a development approach that is predicated on
overlapping software builds,\3 (2) lines of accountability between
the government and the contractor for the system's development that
are unclear, and (3) extensive software development that is occurring
without a software quality assurance program.  Each is discussed
below. 

  In its recent restructuring of the AWIPS program, NWS responded to
     the recommendation of a 1994 independent review team and broke
     the system's development into increments, thus employing a
     widely accepted risk reduction strategy of "build-a-little,
     test-a-little." Generally speaking, incremental development
     breaks a large, monolithic system development effort into
     several smaller, more manageable development pieces, thereby
     permitting earlier warning of significant system development
     problems, and avoiding expending the huge sums of money
     associated with developing a complete system before more basic
     capabilities are successfully developed.

     The key to effective incremental development, however, is to
     ensure that each software increment or build is stabilized
     (i.e., free of material defects) before adding new capabilities
     (i.e., software components) to it during succeeding builds. 
     Without build stabilization, new software problems are
     introduced on top of already existing, unresolved problems,
     greatly increasing the time and money needed to produce mature
     software.  In February 1995, we testified that NWS was not
     providing itself the opportunity for AWIPS build stabilization
     because it had chosen to overlap its software builds as a means
     of schedule compression to meet arbitrary deployment dates.\4
     Specifically, NWS plans to enhance and extend AWIPS software
     components (i.e., capabilities) before these more basic
     capabilities, upon which the enhancements and extensions will
     rely, are fully developed and tested.  Without a mature baseline
     to begin each build, existing software defects are likely to be
     compounded, causing the time and money needed to complete AWIPS
     to grow.

     NWS officials agree that overlapping AWIPS' software builds is a
     risk.  However, they stated that this risk will be mitigated by
     completely testing one build before moving on to the next.  Our
     analysis of the AWIPS build schedule does not support these
     statements, revealing that software builds are scheduled to
     begin before the previous build has been stabilized (i.e., fully
     tested and debugged).

     While we appreciate and share NWS' desire to field AWIPS
     capabilities as soon as possible, thereby allowing it to take
     full advantage of its new observing systems' data sets, we
     believe that overlapping AWIPS' software builds introduces an
     element of risk that could ultimately slow the system's
     completion rather than accelerate it, not to mention raise its
     price tag. 

  In January 1993, we reported on several AWIPS risks confronting
     NWS, including unclear roles and responsibilities between the
     government and the contractor.\5

Again last year, prior to AWIPS' restructuring, we testified that
this risk remained.\6

Under AWIPS' recent restructuring and the associated renegotiated
development contract, the government has maintained a large software
development responsibility, writing all the hydrology and meteorology
applications, while the contractor delivers the AWIPS' hardware,
systems software, and communications networks and integrates these
with the applications.  The government's exposure to risk, however,
still remains because it is uncertain whether the government or the
contractor will bear responsibility for resolving any software
defects discovered during system integration that are not readily
attributable to either party. 

  In December 1994, we reported that NWS' in-house software
     development processes were not adequate for anything more than
     the AWIPS prototyping activities that NWS planned for itself at
     that time.\7 As just mentioned, however, NWS has since assumed
     responsibility for developing over one-half of AWIPS' 1.5
     million lines of code, thus making its need for mature internal
     software development process capabilities absolutely vital.

     While NWS has reported taking a number of steps to strengthen
     its software development processes, such as establishing a
     software development plan, we are aware of at least one serious
     process weakness that remains.  Namely, NWS has not established
     a software quality assurance program for AWIPS.  In a nutshell,
     software quality assurance exists to address the management
     axiom of "what is not tracked is not done." Such a program
     independently (1) monitors whether the software and the
     processes used to develop it fully satisfy established standards
     and procedures and (2) ensures that any deficiencies in the
     software product, process, or their associated standards are
     swiftly brought to management's attention.  In our view, the
     absence of a software quality assurance program for AWIPS
     exposes the project to unacceptable cost, schedule, and
     performance risk. 


--------------------
\3 Software builds are a series of software increments, each with
increasing capabilities, that add to or build upon the capabilities
of the preceding increment. 

\4 Weather Service Modernization:  Despite Progress, Significant
Problems and Risks Remain (GAO/T-AIMD-95-87, Feb.  21, 1995). 

\5 Weather Forecasting:  Important Issues on Automated Weather
Processing System Need Resolution (GAO/IMTEC-93-12BR, Jan.  6, 1993). 

\6 Weather Service Modernization:  Despite Progress, Significant
Problems and Risks Remain (GAO/T-AIMD-95-87, Feb.  21, 1995). 

\7 Weather Forecasting:  Improvements Needed in Laboratory Software
Development Processes (GAO/AIMD-95-24, Dec.  14, 1994). 


   ORIGINALLY PROMISED STAFFING
   REDUCTIONS WILL NOT MATERIALIZE
---------------------------------------------------------- Chapter 0:4

Among the mission benefits to be derived from the NWS modernization
was a 21-percent reduction in its then current staffing levels--a
drop from 5,100 to 4,028.  However, in September 1995, we reported
that the 4,028 target staffing level had grown to 4,678--a decrease
in projected staff savings of 650 or 61 percent.\8 NWS attributed the
reduction in expected staff savings to underestimating the number of
staff needed to operate and maintain the new systems as well as
unexpected, external direction to add field offices and perform
specialized forecasting activities.  As a result of the Secretary of
Commerce's October 1995 decision for NWS to add one more field office
and three NEXRAD sites, expected staffing savings will decrease even
more, perhaps by as much as 60, according to NWS officials.  Such a
staffing change would increase NWS' target staffing level to 4,738,
which is 710 more than the original target levels. 


--------------------
\8 Weather Service Modernization Staffing (GAO/AIMD-95-239R, Sept. 
26, 1995). 


-------------------------------------------------------- Chapter 0:4.1

In conclusion, the inadequacies of AFOS and the potential utility of
incorporating new observing systems' data sets into forecast models
and analyses argue strongly for an AWIPS-like system to support NWS
decision-making and communications needs.  However, because NWS has
not linked AWIPS capabilities to explicit, measurable improvements in
mission performance, we do not know whether AWIPS as currently
defined with all its capabilities is a wise investment.  Furthermore,
because of continuing software development risks, it is uncertain
that NWS will deliver AWIPS as promised. 

Mr.  Chairman, this concludes my statement.  We will be happy to
respond to any questions you or Members of the Subcommittee might
have at this time. 

*** End of document. ***