Weather Forecasting: National Weather Service's Operations
Prototype Needs More Rigorous Planning (08-JUN-07, GAO-07-650).
Using advanced systems and trained specialists located in 122
weather forecast offices throughout the country, the National
Weather Service (NWS) provides storm and flood warnings and
weather forecasts to protect life and property and to enhance the
national economy. To improve the efficiency of its operations, in
November 2006, NWS approved an effort to develop a prototype of
an alternative way of operating. Under this prototype, weather
forecasting offices would share selected responsibilities. GAO
(1) determined the status of and plans for the prototype, (2)
evaluated whether the prototype's justification was sufficient,
(3) determined whether NWS's plans to evaluate the prototype are
adequate, (4) evaluated whether NWS is sufficiently involving
stakeholders in its prototype plans, and (5) determined how NWS
plans to ensure that there will be no degradation of service
during and after the prototype. To do so, GAO analyzed agency
documentation and interviewed program officials and stakeholders.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-650
ACCNO: A70499
TITLE: Weather Forecasting: National Weather Service's
Operations Prototype Needs More Rigorous Planning
DATE: 06/08/2007
SUBJECT: Concept of operations
Developmental testing
Evaluation methods
Federal agency reorganization
Operational testing
Systems analysis
Systems design
Systems evaluation
Weather
Weather forecasting
NWS Advanced Weather Interactive
Processing System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-650
* [1]Results in Brief
* [2]Background
* [3]NWS Office Structure: An Overview
* [4]NWS Relies on Key Systems and Technologies to Fulfill Its Mi
* [5]Automated Surface Observing System
* [6]Next Generation Weather Radar
* [7]Operational Environmental Satellites
* [8]Advanced Weather Interactive Processing System
* [9]Numerical Weather Models
* [10]Supercomputers
* [11]Weather Forecast Office Operations: An Overview
* [12]NWS Explored Options for a New Concept of Operations and Pro
* [13]NWS Developed a Prototype Plan for the Alternative Concept o
* [14]NWS Initiated a 2-Year Prototype and Then Suspended It
* [15]NWS Has Not Yet Sufficiently Justified Its Prototype
* [16]NWS Lacks a Rigorous Evaluation Plan for Assessing Its Proto
* [17]NWS Involved Internal Stakeholders in Planning the Prototype
* [18]NWS Planned to Ensure That the Prototype Would Not Degrade I
* [19]Conclusions
* [20]Recommendations for Executive Action
* [21]Agency Comments and Our Evaluation
* [22]Appendix I: Objectives, Scope, and Methodology
* [23]Appendix II: NWS Prototype Plans, as of March 2007
* [24]Appendix III: Comments from the Department of Commerce
* [25]Appendix IV: GAO Contact and Staff Acknowledgments
* [26]GAO Contact
* [27]Staff Acknowledgments
* [28]Order by Mail or Phone
Report to Congressional Requesters
United States Government Accountability Office
GAO
June 2007
WEATHER FORECASTING
National Weather Service's Operations Prototype Needs More Rigorous
Planning
GAO-07-650
Contents
Letter 1
Results in Brief 2
Background 4
NWS Initiated a 2-Year Prototype and Then Suspended It 14
NWS Has Not Yet Sufficiently Justified Its Prototype 16
NWS Lacks a Rigorous Evaluation Plan for Assessing Its Prototype 17
NWS Involved Internal Stakeholders in Planning the Prototype, but Did Not
Involve External Stakeholders or Establish a Plan for Stakeholder
Involvement 19
NWS Planned to Ensure That the Prototype Would Not Degrade Its Service 21
Conclusions 21
Recommendations for Executive Action 22
Agency Comments and Our Evaluation 22
Appendix I Objectives, Scope, and Methodology 24
Appendix II NWS Prototype Plans, as of March 2007 27
Appendix III Comments from the Department of Commerce 29
Appendix IV GAO Contact and Staff Acknowledgments 32
Table
Table 1: Clustered Peer Concept of Operations Prototype Costs, as of March
2007 17
Figures
Figure 1: NWS's 122 Weather Forecast Offices 6
Figure 2: Overview of Key Systems and Technologies Supporting NWS
Forecasts 7
Figure 3: An AWIPS Workstation 9
Figure 4: NWS Offices Involved in the Clustered Peer Concept of Operations
Prototype 14
Figure 5: Preliminary Schedule for NWS's Clustered Peer Concept of
Operations Prototype, as of November 14, 2006 15
Abbreviations
AWIPS Advanced Weather Interactive Processing System
FAA Federal Aviation Administration
NOAA National Oceanic and Atmospheric Administration
NWS National Weather Service
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office
Washington, DC 20548
June 8, 2007
The Honorable Nick Lampson
Chairman
The Honorable Bob Inglis
Ranking Member
Subcommittee on Energy and Environment
Committee on Science and Technology
House of Representatives
The Honorable David Wu
House of Representatives
The Honorable Vernon J. Ehlers
House of Representatives
The National Weather Service's (NWS) ability to forecast the weather
affects the life and property of every American. The agency's basic
mission is to provide storm and flood warnings and weather forecasts for
the United States, its territories, and adjacent oceans and waters, in
order to protect life and property and to enhance the national economy.
NWS also supports other federal agencies' operations by providing
aviation- and marine-related weather forecasts and warnings. To fulfill
its mission, NWS operates 122 weather forecasting offices and other
specialized weather centers across the country. In these offices, trained
meteorologists use advanced systems to provide forecasts and warnings for
local geographic areas.
In an effort to improve its operational efficiency, in August 2005, NWS
chartered a working group to evaluate the roles, responsibilities,
functions, and supporting technology of weather forecast offices
nationwide and to make a proposal for a more efficient concept of
operations. In December 2005, the working group proposed an alternative
way of operating in which weather offices could share some weather
forecasting functions--particularly during high-intensity weather events.
After receiving this proposal, NWS chartered another team to develop plans
for a prototype of this concept to more fully evaluate this approach.
Because of your interest in weather service operations, we (1) determined
the current status of and plans for the prototype, (2) evaluated whether
the justification for the prototype was sufficient, (3) determined whether
NWS's plans to evaluate the prototype are adequate, (4) evaluated whether
NWS is sufficiently involving stakeholders in its prototype plans, and (5)
determined how NWS plans to ensure that there will be no degradation of
service during and after the prototype.
In late March 2007, after our audit work was nearly completed, the
Department of Commerce's Under Secretary for Oceans and Atmosphere
directed the Director of the National Weather Service to place all
activities associated with the agency's concept of operations on hold
pending a review by the Deputy Under Secretary. NWS officials stated that
the prototype was suspended because of concerns raised about the agency's
approach by some Members of Congress and by National Oceanic and
Atmospheric Administration (NOAA) leadership, the private sector, NWS's
workforce, and its critical emergency management partners. Shortly after
this decision was made, we met with your staffs and agreed to continue
with plans to issue this report to offer insight and recommendations to
NWS, should it decide to proceed with the prototype.
To address our objectives, we reviewed prototype plans and presentations.
We compared program documents with best practices for justifying and
evaluating prototypes and for involving stakeholders. We also interviewed
internal and external stakeholders and agency officials. We conducted our
work at NWS headquarters in the Washington, D.C., metropolitan area; NWS
offices in Tampa, Florida, Detroit, Michigan, and Kansas City, Missouri;
and a NOAA office in Boulder, Colorado, because of the relevance of these
sites to the prototype. We performed our work from October 2006 to April
2007, in accordance with generally accepted government auditing standards.
Additional details on our objectives, scope, and methodology are provided
in appendix I.
Results in Brief
NWS had begun the first phase of its three-phased prototype to demonstrate
a new concept of operations in which weather offices would share selected
forecasting responsibilities, when the Under Secretary for Oceans and
Atmosphere suspended the prototype because of concerns about the agency's
approach. As of March 2007, NWS had established a program manager and
begun to document how the prototype would be executed and to develop
training for employees who would participate. During the second and third
phases, NWS planned to have 20 weather forecasting offices share
forecasting responsibilities in 2-office pairs and then in 4-office
clusters. At the conclusion of the 2-year prototype effort, NWS planned to
make a decision on whether to proceed to implement the new concept of
operations on a national basis. However, these efforts were halted when
the agency suspended the prototype pending a reevaluation on how best to
proceed.
Before the prototype was suspended, NWS had approved moving forward with
its prototype plan without conducting an analysis of the relative costs
and benefits of the initiative. NWS estimated that the prototype would
cost approximately $9.3 million and identified expected qualitative
benefits--including increased efficiency, improved ability to focus on
high-impact weather events, and the ability to switch to a backup site
more quickly when a weather forecast office is disabled due to loss of
power or communications. However, the agency has not conducted a
cost-benefit analysis or determined whether there would be a return on
investment for the prototype. If NWS were to proceed with the prototype
without a cost-benefit analysis, it would lack assurance that its approach
would be a cost-effective investment for the agency.
NWS identified goals and selected measures to evaluate during its
prototype activities, but it did not establish a rigorous evaluation plan.
NWS's prototype plan identified evaluation goals, such as to validate that
a group of weather forecast offices, operating as one unit, could be more
effective and efficient than each office on its own. Also, the prototype
plan identified quantitative and qualitative measures to be used in
evaluating the prototype, including the number of times that work was
transferred between offices to focus on high-impact events, the number of
additional hours made available during benign weather for nonforecast
work, and customer perceptions of service quality and timeliness. However,
NWS did not establish an evaluation plan that identified a comprehensive
set of applicable measures, what baseline performance its prototype would
be compared with, and how its measures supported the goals of the
prototype. If NWS were to proceed with its prototype without a rigorous
evaluation plan, it would run an increased risk of not sufficiently
measuring the impact of planned changes on its performance, and of
subsequently making decisions affecting the nation's weather service on
the basis of incomplete and flawed data.
NWS involved internal stakeholders in planning its prototype, including
having selected internal stakeholders review plans and provide feedback on
work products and laboratory activities. However, NWS did not involve
external stakeholders in its planning for the prototype or establish a
plan for stakeholder involvement during the prototype. Specifically, NWS
did not request input from external stakeholders, including emergency
managers, in the development and planning of the prototype. In addition,
the agency did not establish a plan that identified key external
stakeholders or stakeholder representatives, determined the
responsibilities of internal or external stakeholders, and established a
time line for involving stakeholders and addressing their comments. If NWS
were to proceed with its prototype without such a plan, the agency would
be unable to ensure that employee and customer interests are anticipated,
identified, and addressed.
Prior to suspending the prototype, NWS stated that it would ensure that
there would be no degradation of service resulting from the prototype, and
that it had plans in place to mitigate the risk of degradation. To these
ends, NWS planned to conduct multiple laboratory exercises to understand
the impact of the prototype. Also, NWS planned to conduct the prototype on
a dedicated workstation in each cluster office that was separate from
existing systems, so that it could revert to normal operations should the
need arise. In addition, NWS planned to monitor its systems and products
under the prototype, so that the agency could revert to baseline systems
if warranted. NWS officials stated that, should the agency decide to
implement the prototype on a national basis, it has standard procedures
for testing and validating systems and software to avoid any degradation
of service before they are put into use.
We are making recommendations to the Secretary of Commerce to ensure that
if NWS decides to proceed with its prototype, it undertake a more rigorous
approach to justifying, evaluating, and involving stakeholders in its
prototype effort. We also are recommending that the agency evaluate moving
forward with technology upgrades to allow offices to switch to backup
service more quickly during high-impact weather events.
Commerce provided written comments on a draft of this report in which it
agreed with our recommendations, and stated that our findings were
included in the Under Secretary's recent review of the concept of
operations prototype. Based on that review, the NOAA Administrator
directed NWS to cease all activities associated with the concept of
operations prototype, and to undertake a comprehensive and analytical
review to determine new and revised requirements for NWS products and
services. A reprint of the department's comments is provided in appendix
III.
Background
The mission of NWS--an agency within Commerce's NOAA--is to provide
weather, water, and climate forecasts and warnings for the United States,
its territories, and its adjacent waters and oceans to protect life and
property and to enhance the national economy. NWS is the official source
of aviation- and marine-related weather forecasts and warnings as well as
warnings about life-threatening weather situations.
NWS Office Structure: An Overview
The coordinated activities of weather facilities throughout the United
States allow NWS to deliver a broad spectrum of climate, weather, water,
and space weather services. These facilities include weather forecast
offices, river forecast centers, national centers, and aviation center
weather service units. The functions of these facilities are described in
the following text:
o 122 weather forecast offices are responsible for providing a
wide variety of weather, water, and climate services for their
local county warning areas, including advisories, warnings, and
forecasts. See figure 1 for a map of the locations of these
offices.
o 13 river forecast centers provide river, stream, and reservoir
information to a wide variety of government and commercial users
as well as to local weather forecast offices for use in flood
forecasts and warnings.
o 9 national centers constitute the National Centers for
Environmental Prediction, which provide nationwide computer model
output and manual forecast information to all NWS field offices
and to a wide variety of government and commercial users. These
centers include the Environmental Modeling Center, Storm
Prediction Center, Tropical Prediction Center/National Hurricane
Center, Climate Prediction Center, Aviation Weather Center, and
Space Environment Center, among others.
o 21 aviation center weather service units, which are colocated
with key Federal Aviation Administration (FAA) air traffic control
centers across the nation, provide meteorological support to air
traffic controllers.
Figure 1: NWS's 122 Weather Forecast Offices
NWS Relies on Key Systems and Technologies to Fulfill Its Mission
To fulfill its mission, NWS relies on a national infrastructure of systems
and technologies that gather and process data from the land, sea, and air.
NWS collects data from many sources, including ground-based Automated
Surface Observing Systems, the Next Generation Weather Radars, and
operational environmental satellites. These data are integrated by
advanced data processing workstations--called an Advanced Weather
Interactive Processing System (AWIPS)--and used by meteorologists to issue
local forecasts and warnings. Also, these data are fed into sophisticated
computer models running on high-speed supercomputers, which are then used
to help develop forecasts and warnings. Figure 2 depicts the integration
of the various systems and technologies and is followed by a description
of each.
Figure 2: Overview of Key Systems and Technologies Supporting NWS
Forecasts
Automated Surface Observing System
The Automated Surface Observing System is a system of sensors, computers,
display units, and communications equipment that automates the
ground-based observation and dissemination of weather information
nationwide. This system collects data on temperature, dew point,
visibility, wind direction and speed, pressure, cloud height and amount,
and types and amounts of precipitation. There are currently 1,001 units
deployed across the United States, with NWS, FAA, and the Department of
Defense operating 312, 571, and 118 units, respectively.
Next Generation Weather Radar
The Next Generation Weather Radar is a Doppler radar system that detects,
tracks, and determines the intensity of storms and other areas of
precipitation; determines wind velocities in and around detected storm
events; and generates data and imagery to help forecasters distinguish
hazards, such as severe thunderstorms and tornadoes. The radar system also
provides information about heavy precipitation that leads to warnings
about flash floods and heavy snow. A network of these radars is made up of
158 operational radar systems and 8 nonoperational systems that are used
for training and testing. Of these, NWS, the United States Air Force, and
FAA sponsor 120, 26, and 12 radars, respectively.
Operational Environmental Satellites
Although NWS does not own or operate satellites, geostationary and
polar-orbiting environmental satellites^1 are key sources of data for its
operations. These satellite systems continuously collect environmental
data about Earth's atmosphere, surface, and cloud cover and the
electromagnetic environment. These data are used by meteorologists to
develop weather forecasts and other services, and are critical to the
early and reliable prediction of severe storms, such as tornadoes and
hurricanes.
Advanced Weather Interactive Processing System
AWIPS is a data processing workstation that integrates and displays
hydrometeorological data at NWS weather forecast offices, river forecast
centers, and national centers. This system integrates data from a variety
of sources (including the ground systems, radars, and satellites
previously listed) to produce rich graphical displays to aid forecaster
analysis and decision making. AWIPS includes a graphical forecast editing
tool that allows forecasters to display and manipulate detailed graphical
depictions of expected weather, and to use these depictions to generate
text and graphical forecasts. AWIPS is used to disseminate weather
information to the national centers; weather offices; the media; and other
federal, state, and local government agencies. Figure 3 shows a standard
AWIPS workstation.
^1Geostationary environmental satellites orbit above Earth's surface at
the same speed as Earth rotates, so that each satellite remains over the
same location on Earth. Polar satellites constantly circle Earth in an
almost north-south orbit, providing global coverage of conditions that
affect the weather and climate.
Figure 3: An AWIPS Workstation
Numerical Weather Models
Numerical weather models are advanced software programs that assimilate
data from satellites and ground-based observing systems and provide short-
and long-term weather pattern predictions. Meteorologists typically use a
combination of models and their own experience to develop local forecasts
and warnings. In addition, numerical weather models are a critical source
for forecasting weather up to 2 weeks in advance and forecasting long-term
climate changes.
Supercomputers
NWS leases high-performance supercomputers to execute numerical
calculations supporting weather prediction and climate modeling. In 2002,
NWS awarded a $227 million contract to lease high-performance
supercomputers to run its environmental models from 2002 through September
2011. Included in this contract are an operational supercomputer used to
run numerical weather models, an identical backup supercomputer located at
a different site, and a research and development supercomputer on which
researchers can test new analyses and models.
Weather Forecast Office Operations: An Overview
NWS delivers a broad spectrum of climate, weather, water, and space
weather services through the coordinated activities of the weather
forecast offices; river forecast centers; and national centers, which
comprise the National Centers for Environmental Prediction, aviation
center weather service units, and other support offices around the
country. Each weather forecast office currently has a fixed geographic
area for which it provides local warnings and forecasts. These offices are
staffed by meteorologists who produce a suite of weather products,
including short-term and 3- to 5-day extended forecasts as well as
aviation and marine weather products, and monitor for severe weather
events. Predictive weather services (outlooks, forecasts, watches,
warnings, advisories, and other supporting information) are produced
through a process that incorporates observational data, numerical model
output and statistical guidance, and human expertise. These services are
delivered in many forms, including textual, digital, and graphical
formats. Local offices and regional and national centers coordinate as
needed in the preparation of these services.
When a severe weather event occurs, forecast office managers often ask
their staffs to work overtime so that there are enough personnel available
to do both the normal forecasting work and the watches and warnings
required by the severe event. If a weather forecast office is unable to
provide forecast and warning functions due to an interruption in
communications or power, an adjacent office will temporarily assume those
duties. For example, during Hurricane Katrina, several weather forecast
offices in Louisiana, Mississippi, and Alabama were damaged and
inoperable. These offices' responsibilities were transferred to backup
sites in Texas, Florida, Alabama, and Tennessee. During this transfer of
operations, called service backup, the backup office gains access to the
original office's system data and calls in extra personnel to handle the
workload of both offices.
According to NWS officials, due to software limitations in AWIPS,
switching to a backup service arrangement can take 60 to 90 minutes, and,
when invoked, the backup office must assume all products and services for
the defunct office. In addition, the current backup arrangement can cause
duplicate work. For example, one office could produce a forecast for its
own geographic area, and then would need to produce a separate forecast
for its backup partner.
NWS Explored Options for a New Concept of Operations and Proposed One Option for
Further Study
Over the past few years, NWS has evaluated options for an alternate
concept of operations. According to agency officials, in 2005, NWS's board
of executive advisors (called the Corporate Board)^2 noted that a
constrained budget, high labor costs, difficulty in training and
developing its employees, and a lack of flexibility in how the agency was
operating were making it more difficult for the agency to continue to
perform its mission. In August 2005, the board chartered a working group
to evaluate the roles, responsibilities, and functions of weather offices
nationwide and to make a proposal for a more efficient concept of
operations. The group was given a set of guiding principles, including
that the proposed concept should (1) be cost-effective, (2) ensure that
there would be no degradation of service, (3) ensure that weather services
nationwide were equitable, and (4) not reduce the number of forecast
offices nationwide.
The group gathered input from various agency stakeholders and other
partners within NOAA and considered multiple alternatives. They dismissed
all but one of the alternative concepts because they were not consistent
with the guiding principles. In its December 2005 proposal, the working
group proposed the remaining concept, in which several offices with the
same type of weather and warning responsibilities, climate, and customers
would be grouped together to share responsibilities, particularly when
there is a high-impact weather event affecting one of the offices. NWS
called this a "clustered peer concept of operations."
Under the clustered peer concept of operations, offices would share the
workload associated with routine services, such as 7-day forecasts. During
a high-impact weather event--such as a severe storm, flood, or
wildfire--the offices would redistribute the workload to allow the
impacted office to focus solely on the event, while the other offices in
the cluster would pick up the impacted office's routine services. In
addition, offices in a cluster could occasionally allocate the work so
that certain offices would take on forecasting responsibilities while
other offices focused on other tasks, such as training, customer outreach,
and research. In this manner, peer offices could help supplement staffing
needs, and the workload across multiple offices could be more efficiently
balanced.
^2NWS's Corporate Board is chaired by the Director of the National Weather
Service and composed of senior officials responsible for different aspects
of the agency's mission, including the Chief Information Officer and the
Directors of the Office of Climate, Water, and Weather Services; the
National Centers for Environmental Prediction; and the NWS regions. The
board meets at least twice annually to discuss NWS's budget and strategic
issues. It also holds meetings, as needed, to focus on special topics,
such as an assessment of weather services provided during Hurricane
Charley in 2004.
Some of these benefits were demonstrated in recent years by a pair of
weather forecasting offices. The Detroit and Grand Rapids offices
established a technology workaround to allow them to more efficiently
switch to backup operations between the two offices. This technology
improvement allows them to switch to backup mode instantaneously, rather
than taking 60 to 90 minutes using current AWIPS technology. To date, the
two offices have used this backup capability to share workload during
high-impact weather events; during system upgrades; and while conducting
officewide training, research, and public outreach.
NWS Developed a Prototype Plan for the Alternative Concept of Operations
In February 2006, NWS's Corporate Board chartered a team to develop a
prototype plan for the clustered peer concept of operations. Between
February and November 2006, the team established a preliminary concept for
what the prototype would entail, conducted a laboratory exercise to
determine AWIPS's ability to support the prototype, and--on the basis of
the findings from the laboratory exercise--developed a prototype plan.
The team's preliminary concept was for the prototype to consist of four
clusters ranging in size from four to eight weather forecast offices.
However, in September 2006, a laboratory exercise conducted between NOAA's
Global Systems Division in Boulder and NWS's Central Region Headquarters
in Kansas City showed that AWIPS was not capable of supporting clusters of
more than four offices. The laboratory results showed that with minor
modifications, AWIPS was capable of allowing a more efficient and timely
switch of operations between two weather forecast offices for backup
purposes, and that, in some cases, functions could be shared among four
offices. Officials in charge of the laboratory exercise estimated that
larger clusters would not be possible until the next version of AWIPS is
deployed--a milestone currently planned for 2010. The laboratory exercise
also identified several problems with system performance when weather
forecast offices work as a cluster of two or more offices. Specifically,
each weather forecast office has its own set of customized tools
supporting each office's graphical forecast editor. The laboratory
exercise found that running these tools when sharing functions among
multiple offices caused AWIPS to slow down or crash.
On the basis of the findings from the laboratory exercise, the prototype
team developed a plan to demonstrate opportunities for improved
efficiency. The goal of the prototype was to validate that a group of
weather forecast offices, operating as one unit, could be more efficient
than each office operating on its own. The plan entailed sharing
forecasting responsibilities between 2, and subsequently among 4, weather
forecast offices using a single workstation in each office. The prototype
was to be a hands-on demonstration conducted at 20 weather forecast
offices, clustered in 4 different areas of the country: the Pacific
Northwest, Great Plains, Great Lakes, and Southeast (see fig. 4). These
areas were selected because they present different forecasting challenges,
involve all of the regions in the contiguous United States, and engage a
variety of field offices. In November 2006, the Corporate Board approved
moving forward with the prototype plan.
Figure 4: NWS Offices Involved in the Clustered Peer Concept of Operations
Prototype
NWS Initiated a 2-Year Prototype and Then Suspended It
Following the Corporate Board's decision to proceed, NWS initiated the
first of three phases of its prototype, which it planned to continue over
the next 2 years. The first phase was expected to take place between
November 2006 and June 2007 and was to include planning activities. As of
March 2007, NWS had established a program manager and initiated efforts to
plan for the prototype, including documenting how the prototype would be
executed and developing training for employees who would participate in
the prototype.
During the second phase, which NWS expected to take place between August
2007 and February 2008, the agency planned to have 20 weather offices
share forecasting responsibilities in 2-office pairs. During the third
phase, which NWS expected to take place between April 2008 and April 2009,
the agency planned to have the same 20 weather offices share forecasting
responsibilities in 4-office clusters. Each phase was to be preceded by
both a laboratory exercise to test the new expanded functions and a board
decision on whether to move ahead, based on the results of the laboratory
exercise. NWS planned to conclude prototype operations and issue a report
on the prototype by July 2009. By August 2009, NWS intended to make a
decision on whether to implement the new concept of operations on a
national basis. NWS's schedule for the prototype is provided in figure 5.
A more detailed description of the phases of the prototype is provided in
appendix II.
Figure 5: Preliminary Schedule for NWS's Clustered Peer Concept of
Operations Prototype, as of November 14, 2006
These efforts were halted when the prototype was suspended. On March 23,
2007, Commerce's Under Secretary for Oceans and Atmosphere directed NWS's
director to place all activities associated with NWS's concept of
operations on hold pending a review by the Deputy Under Secretary. NWS's
director subsequently issued a memorandum noting that the prototype was
being suspended because of concerns about the agency's approach expressed
by some Members of Congress and by NOAA leadership, the private sector,
the agency's workforce, and the agency's emergency management partners.
The director noted that the prototype would be put on hold until a
reevaluation could be undertaken.
NWS Has Not Yet Sufficiently Justified Its Prototype
Before the prototype was suspended, NWS developed and approved a 2-year
plan for its prototype without conducting an analysis of the relative
costs and benefits of the initiative. Leading organizations have developed
best practices for justifying prototyping efforts--including analyzing the
expected costs and quantified benefits of alternatives before beginning
prototyping activities--and identifying plans, schedules, and risks.^3
Furthermore, federal guidance^4 notes that in conducting a cost-benefit
assessment, tangible and intangible benefits and costs should be
identified, assessed, and reported.
The prototype plan approved by the Corporate Board in November 2006
identifies the schedules, risks, and expected benefits of the prototype.
The prototype was estimated to cost $9.3 million, and the Corporate
Board's financial investment review committee approved spending $1.5
million on the prototype in the first half of fiscal year 2007. As of
March 2007, NWS had spent approximately $724,000 on the prototype (see
table 1). Moreover, the prototype plan identifies qualitative benefits of
the new concept of operations--including increased efficiency and time for
other activities (such as outreach and research), the ability to switch to
a backup site more quickly when a weather forecast office is disabled due
to loss of power or communications, and an improved ability to focus on
high-impact events.
^3Carnegie Mellon University Software Engineering Institute, Capability
Maturity Model^(R) Integration for Development, Version 1.2 (Pittsburgh,
PA: August 2006). The Carnegie Mellon University Software Engineering
Institute, recognized for its expertise in software and system processes,
has developed the Capability Maturity Model^(R) Integration for
Development to evaluate, improve, and manage system and software
development processes.
^4Office of Management and Budget, Guidelines and Discount Rates for
Benefit-Cost Analysis of Federal Programs, OMB Circular A-94 (Washington,
D.C.: Oct. 29, 1992).
Table 1: Clustered Peer Concept of Operations Prototype Costs, as of March
2007
Fiscal Approved
year expenditures Actualexpenditures Estimated expenditures^a Total
2005 $4,125 $4,125 -- $4,125
2006 430,879 430,879 -- 430,879
2007 1,500,000 288,907 $2,911,093 3,200,000
2008 -- 3,000,000 3,000,000
2009 -- 2,700,000 2,700,000
Total $1,935,004 $723,911 $8,611,093 $9,335,004
Source: GAO analysis of NWS data.
aNWS has not yet approved these estimates.
However, NWS did not conduct a cost-benefit analysis to justify its plans
for the prototype or the $1.9 million that has already been approved for
the prototype.
By not conducting a cost-benefit analysis before beginning the prototype,
NWS lacked assurance that its approach was cost-effective. If NWS decides
to proceed with the prototype without conducting a cost-benefit analysis,
the agency will have little basis to ensure that its investment is sound
or that the options it is undertaking provide the greatest return on
investment. Specifically, a cost-benefit analysis could help ascertain
which aspects of the prototype promise the most benefits and enable NWS to
adjust its plans accordingly. For example, a cost-benefit assessment might
show that more benefits could be gained for the least cost by moving to a
2-office cluster, rather than a 4-office cluster.
NWS Lacks a Rigorous Evaluation Plan for Assessing Its Prototype
In its prototype plan, NWS identified goals and selected measures to
evaluate during its prototype activities, but it did not establish a
rigorous evaluation plan. Specifically, the agency had not defined a
comprehensive set of measures, how it planned to compare prototype results
with baseline performance, or how its measures supported the goals of the
prototype.
According to best practices in leading organizations, well-developed
evaluation plans increase the likelihood that evaluations will yield sound
results, thereby supporting effective policy decisions. Key features of a
well-developed evaluation plan include well-defined, clear, and measurable
objectives or goals; quantitative measures for determining program
performance as compared with baseline performance; and a linkage between
the measures and the program goals. Such plans are essential to ensuring,
among other things, that programs are executed properly and that funding
requests are developed appropriately. In July 2006, we reported that NWS
had not yet established plans, time lines, and measures for evaluating its
prototype, and we recommended that NWS do so before beginning the
prototype.^5
NWS's prototype plan identifies evaluation goals and measures, and notes
that NWS plans to contract with an independent entity to perform the
evaluation. The prototype goals were to (1) validate that a group of
weather forecast offices, operating as one unit, could be more effective
and efficient than each office on its own; (2) increase focus during
high-impact events through more efficient resource allocation; (3) gain
efficiencies through the use of resource allocation during benign weather
that can be applied to training, outreach, and professional development;
and (4) use some of the efficiencies gained through the prototype to
further the NOAA mission. Quantitative and qualitative measures that were
to be used in evaluating the prototype included the number of times that
work was transferred to another forecast office to focus on high-impact
events, the number of additional hours made available during benign
weather for nonforecast work, and customer perceptions of service quality
and timeliness. The agency stated that system performance and
hardware/software functionality would have been evaluated using staff
feedback, performance monitoring software, and usage logs, while customer
perceptions would have been evaluated using customer satisfaction surveys.
However, NWS did not establish an evaluation plan that identifies a full
set of applicable measures, what baseline performance its prototype would
be compared with, and how its measures would support the goals of the
prototype. Specifically, while the agency listed various items that it
planned to measure in its prototype plan, it did not include quantitative
measures of product quality, timeliness, the impact on the efficiency of
the office that is receiving the increased workload, or plans to assess
the sensitivity of its results to weather-related situations. In addition,
the agency did not establish the baseline performance against which it
would compare its prototype, so it was not clear how prototype results
would be assessed. Furthermore, the measures that NWS identified did not
directly support the goals. For example, the number of times functions are
transferred to another site does not demonstrate increased efficiency. For
this measure to support the desired goal of increased efficiency, the
measure would need to be supplemented with an assessment of how the office
that transferred its workload used the newly available time and how the
receiving site handled its increased workload.
^5GAO, Weather Forecasting: National Weather Service Is Planning to
Improve Service and Gain Efficiency, but Impacts of Potential Changes Are
Not Yet Known, [29]GAO-06-792 (Washington, D.C.: July 14, 2006).
If NWS were to proceed with its prototype without a rigorous evaluation
plan, the agency would run an increased risk that it could not
sufficiently measure the impact of planned changes on its performance, and
would subsequently make decisions affecting the nation's weather service
on the basis of incomplete and flawed data.
NWS Involved Internal Stakeholders in Planning the Prototype, but Did Not
Involve External Stakeholders or Establish a Plan for Stakeholder Involvement
NWS involved internal stakeholders in planning its prototype, including
having selected stakeholders review plans and provide feedback on work
products and laboratory activities. However, NWS did not involve external
stakeholders or establish a plan for stakeholder involvement throughout
the prototype. Leading organizations routinely involve relevant
stakeholders when considering operational or process changes.^6 In
addition, NWS has agreed to negotiate with its employees' union, the
National Weather Service Employees Organization, whenever organizational
changes could affect working conditions.^7 Moreover, best practices call
for developing a plan for stakeholder involvement that includes
identifying relevant stakeholders, the roles and responsibilities of the
relevant stakeholders, and a schedule for stakeholder involvement
throughout the project.
In planning its prototype, NWS involved internal stakeholders, who were
identified by the NWS Director of Strategic Plans as forecasters, other
field and headquarters staff, and union representatives. For example, when
deciding which offices to group into clusters, meteorologists-in-charge
were asked for their input on cluster criteria and preferences.
Furthermore, one meteorologist from each of eight potential prototype
weather forecast offices participated in the laboratory test. Their
feedback on the new concept of operations was acquired through
questionnaires, electronic logs, and debriefings at the end of each week.
In addition, the employees' union provided input before the decision was
made to begin the prototype. Both the initial working group and the
prototype team included union representatives. Also, the union president
participated in Corporate Board meetings that included discussions of the
clustered peer prototype, and a union representative observed the final
week of the laboratory test.
^6GAO, Information Technology Investment Management: A Framework for
Assessing and Improving Process Maturity, [30]GAO-04-394G (Washington,
D.C.: March 2004); and Capability Maturity Model^(R)Integration for
Development, Version 1.2.
^7NWS, National Collective Bargaining Agreement Between The National
Weather Service and The National Weather Service Employees Organization
(Oct. 25, 2001). This agreement calls for NWS to negotiate with the union
on the impact and implementation of changes affecting working conditions
before they can be implemented. The agreement also allows the agency to
avoid the need for negotiations if the union has sufficient predecisional
involvement.
NWS also intended to work with external stakeholders (including emergency
managers and other public safety officials, the media and other users of
NWS information services, and the general public) after the prototype was
complete to assess the results. Specifically, NWS planned to identify
customer perceptions of service as one basis on which to evaluate the
prototype. Furthermore, in an effort to make its plans available to
employees and customers, NWS developed a public Web site that contains the
plans and reports for the prototype.^8
However, NWS did not request input from external stakeholders, including
emergency managers, during the past 20 months that it was planning and
initiating the prototype. NWS officials reported that they did not involve
external stakeholders because changes in the concept of operations during
the prototype should not be noticeable to outside stakeholders; that is,
external stakeholders should not notice any difference in service.
Although this goal has merit, external stakeholders need to be more
involved in the prototype for it to gain acceptance and succeed.
Specifically, external stakeholders need to understand what is being
tested, have input into the tests, be alert for differences under the
prototype, and have formal mechanisms for sharing their observations. In
addition, NWS did not have a comprehensive plan for involving stakeholders
in its prototyping activities. Specifically, it did not identify key
external stakeholders or stakeholder representatives, determine the
responsibilities of internal or external stakeholders, or develop a time
line for involving stakeholders and addressing their comments.
^8NWS, Focus on the Future (Silver Spring, MD: Apr. 5, 2007),
[31]http://www.weather.gov/com/digitalera/ (viewed on Apr. 10, 2007).
If NWS were to proceed with its prototype without a plan for involving
internal and external stakeholders, it would not be able to ensure that
employee and customer interests are anticipated, identified, and
addressed.
NWS Planned to Ensure That the Prototype Would Not Degrade Its Service
NWS stated that it would ensure that there would be no degradation of
service during or after its prototype, and that it had plans in place to
mitigate the risk of any service degradation. Specifically, the agency
conducted and planned to continue to conduct laboratory exercises to
assess the impact of the prototype before moving into each phase. In
addition, NWS planned to run the prototype software on a single, isolated
workstation in each office, while maintaining the existing AWIPS software
on all other workstations. Established procedures would allow offices to
return to the baseline software and operational practices should problems
have developed. Furthermore, NWS planned to monitor the performance of all
activities and the quality and timeliness of products.
Looking forward, an NWS official stated that they also would ensure that
there would be no degradation of service if the Corporate Board decided to
implement this alternative concept of operations on a national basis after
the prototype was completed. Agency officials explained that the agency
would use its standard procedures for testing and validating systems and
software to ensure that there would be no degradation of service before
moving them into operations.
Conclusions
Although there is merit in NWS's goal of improving its operational
efficiency and allowing faster backup service during severe weather, the
agency lacked the rigorous planning it needed to justify, evaluate, and
obtain stakeholder involvement in its planned--and now
suspended--prototype of a new concept of operations. Specifically, the
agency developed a prototype plan that identified activities, gates, and
decisions, but it did not perform a cost-benefit assessment to justify the
prototype. Also, NWS identified goals and selected measures for evaluating
its prototype, but it did not establish an evaluation plan that clearly
identifies a comprehensive set of measures or how these measures would be
compared with baseline performance. Furthermore, the agency involved
internal stakeholders in planning for its prototype, but it did not seek
input from external stakeholders or establish a plan for stakeholder
involvement throughout the prototype. These matters will be important to
consider should NWS decide to proceed with the prototype. Without
undertaking a more rigorous approach, NWS would run an increased risk that
its prototype will not be a sound investment, and that any resulting
decisions could be based on flawed data and analysis. Taking this more
rigorous approach to the prototype should not preclude NWS from
considering options for improving the ability to switch to a backup site
when a weather forecast office becomes disabled due to loss of power or
communications.
Recommendations for Executive Action
We are making four recommendations to the Secretary of Commerce. If NWS
decides to proceed with its prototype, we recommend the Secretary direct
the Assistant Administrator for the National Weather Service to
o develop a cost-benefit analysis for the clustered peer approach
before implementing the prototype in any weather forecasting
office;
o develop an evaluation plan that includes a comprehensive set of
measures that are linked to prototype goals and identifies the
baseline performance that the prototype will be compared with
before implementing the prototype in any weather forecasting
office; and
o develop a plan for internal and external stakeholder
involvement, which includes a list of relevant stakeholders, roles
and responsibilities for these stakeholders, and a schedule for
when stakeholders should be involved before implementing the
prototype in any weather forecasting office.
In addition, we recommend that the Secretary direct the Assistant
Administrator for the National Weather Service to evaluate moving
forward with technology upgrades to the AWIPS processing system to
allow weather forecast offices to switch to backup service more
quickly during high-impact weather situations; these upgrades
could be handled as an initiative that is separate and distinct
from broader changes in the agency's concept of operations.
Agency Comments and Our Evaluation
We received written comments on a draft of this report from
Commerce (see app. III). In the department's response, the Deputy
Secretary of Commerce agreed with our recommendations to conduct a
thorough cost-benefit analysis, develop a rigorous evaluation plan
with fully defined performance measures, and involve internal and
external stakeholders. The response stated that our findings were
included in the Under Secretary's recent review of the concept of
operations prototype. On the basis of that review, the NOAA
Administrator directed NWS to cease all activities associated with
the concept of operations prototype, and to undertake a
comprehensive and analytical review to determine new and revised
requirements for NWS products and services. This review is
expected to determine what, if any, changes are required. Should
NWS decide to proceed with a new concept of operations, we believe
that our recommendations to justify, evaluate, and involve
stakeholders in any such changes would still be relevant and
useful.
In addition, Commerce agreed with our recommendation to evaluate
moving forward with technology upgrades to the AWIPS processing
system to allow weather forecast offices to switch to backup
service more quickly during high-impact weather situations. The
department stated that NWS has established a proposal for
improving its service backup capability, expects to approve
requirements for the backup capability by September 2007, and
plans to deploy this capability by September 2008.
As we agreed with your offices, unless you publicly announce the
contents of this report earlier, we plan no further distribution
of it until 30 days from the report date. At that time, we will
send copies of this report to interested congressional committees,
the Secretary of Commerce, the Director of the Office of
Management and Budget, and other interested parties. In addition,
this report will be available at no charge on our Web site at
www.gao.gov.
If you have any questions about this report, please contact me at
(202) 512-9286 or by e-mail at [email protected]. Contact points for
our Offices of Congressional Relations and Public Affairs may be
found on the last page of this report. GAO staff who made major
contributions to this report are listed in appendix IV.
David A. Powner
Director, Information Technology Management Issues
Appendix I: Objectives, Scope, and Methodology
Our objectives were to (1) determine the current status of and
plans for the prototype, (2) evaluate whether the justification
for the prototype is sufficient, (3) determine whether the
National Weather Service's (NWS) plans to evaluate the prototype
are adequate, (4) evaluate whether NWS is sufficiently involving
stakeholders in its prototype plans, and (5) determine how NWS
plans to ensure that there will be no degradation of service
during and after the prototype.
To determine the current status of and plans for the prototype, we
reviewed agency documentation on the planned prototype's goals,
major components, and milestones, such as the Corporate Board's
meeting minutes, the prototype plan, and the preprototype
laboratory report. We also interviewed NWS officials to obtain
clarifications on agency plans.
^1Carnegie Mellon University Software Engineering Institute, Capability
Maturity Model^(R) Integration for Development, Version 1.2 (Pittsburgh,
PA: August 2006). Capability Maturity Model^(R) and Capability Maturity
Modeling are registered in the U.S. Patent and Trademark Office. CMM is a
service mark of Carnegie Mellon University.
^2Office of Management and Budget, Guidelines and Discount Rates for
Benefit-Cost Analysis of Federal Programs, OMB Circular A-94 (Washington,
D.C.: Oct. 29, 1992).
To evaluate whether the justification for the prototype is
sufficient, we reviewed program documents such as the prototype
plan and other NWS planning documents, presentations, and
communications about the goals and costs of the prototype. We
compared these planning documents with best practices for
justifying process-improvement proposals by the Carnegie Mellon
University Software Engineering Institute's Capability Maturity
Model^(R)Integration for Development^1 and federal guidance on
conducting cost-benefit analyses.^2 We also interviewed agency
officials and employees who were involved in the planning of the
prototype and laboratory exercise to determine the potential
costs, benefits, and risks of the prototype.
To determine whether NWS plans to evaluate the prototype are
adequate, we reviewed planning documents, such as the prototype
plan and NWS customer satisfaction scores, and compared them with
best practices for
evaluating prototypes from the Capability Maturity
Model^(R)Integration for Development and GAO's guidance for
developing and assessing evaluation plans.^3 In addition, we
interviewed NWS officials who were responsible for planning the
prototype activities.
To evaluate whether NWS is sufficiently involving stakeholders in
its prototype plans, we reviewed NWS's plans, presentations, and
other communications about the prototype, including an external
Web site that presented these documents. We compared NWS's plans
for involving stakeholders with best practices from the Capability
Maturity Model^(R) Integration for Development. We spoke with
members of stakeholder groups to determine their knowledge of the
prototype and their interaction with NWS. In addition, we
interviewed NWS officials and the president of the National
Weather Service Employees Organization to determine the level of
involvement of the employee union in planning for the prototype.
To determine how NWS plans to ensure that there will be no
degradation of service during and after the prototype, we reviewed
planning documents--including the prototype plan, presentations,
and communications regarding the prototype--and interviewed NWS
officials.
We performed our work at NWS headquarters in the Washington, D.C.,
metropolitan area; at NWS offices in Tampa, Florida, Detroit,
Michigan, and Kansas City, Missouri; and at a National Oceanic and
Atmospheric Administration (NOAA) office in Boulder, Colorado. We
visited the Tampa weather forecast office because the president of
the employee union and the vice president of the International
Association of Emergency Managers are located in the Tampa area.
The Detroit weather forecast office was included because of its
prior experiences in working in a paired backup with expanded
domains. We visited the NWS Central Region Headquarters in Kansas
City because of its participation in the first laboratory exercise
and because it is where the prototype team chair worked. The NOAA
Global Systems Division office in Boulder was chosen because of
its participation in the first laboratory exercise. We performed
our work from October 2006 to April 2007 in accordance with
generally accepted government auditing standards.
^3GAO, Equal Employment Opportunity: DOD's EEO Pilot Program Under Way,
but Improvements Needed to DOD's Evaluation Plan, [39]GAO-06-538
(Washington, D.C.: May 5, 2006); and Earned Income Tax Credit:
Implementation of Three New Tests Proceeded Smoothly, But Tests and
Evaluation Plans Were Not Fully Documented, [40]GAO-05-92 (Washington,
D.C.: Dec. 30, 2004).
Appendix II: NWS Prototype Plans, as of March 2007
Prior to the suspension of the prototype by the Director of the
National Weather Service, the prototype team developed a plan for
a three-phased prototype of the clustered peer concept of
operations. As described in the plan, phase 1 would include
planning activities, such as training; phase 2 was meant to
establish clustered peer operations in 10 pairs of weather
forecast offices across the country; and phase 3 would include
offices grouped into 4-office clusters. NWS planned to conclude
prototype operations and to issue a report on the prototype by
July 2009. NWS was to decide at that time if it would proceed to
implement the clustered peer concept of operations on a national
basis. Details about each of the planned prototype phases are
included in the following text.
Phase 1, the planning phase, began in November 2006 with the
Corporate Board's approval of the prototype plan. NWS established
a program manager to oversee the prototype. In addition, cluster
management teams^1 representing each cluster were to meet to
decide on the rules of engagement, common tools, and common
forecast methodologies. At the end of this phase, a second
laboratory exercise was planned to validate that software upgrades
and hardware additions to support the prototype were operationally
ready. The second laboratory exercise was scheduled for June 2007
and was to involve only weather forecast offices that are
currently service-backup pairs within the original four geographic
cluster areas. Upon completion of this exercise, the Corporate
Board was to decide on whether to initiate Phase 2 of the
prototype.
Phase 2 was planned to begin in August 2007. During this phase,
NWS planned to conduct training, develop guidance for how and when
offices would take over activities for other offices (called
playbooks), define the cluster methodology, and have 10 pairs of
offices operate as clustered peers. Also during this phase, NWS
planned to develop the software upgrades needed for expanded
operations in the next phase. Near the end of this phase, in
February 2008, NWS planned to conduct a third laboratory exercise
to test the software necessary for moving to clusters of 4
offices. After completion of this exercise, the Corporate Board
was to decide on whether to initiate Phase 3 of the prototype.
^1The cluster management team provides operational oversight to the
cluster. The team is composed of the meteorologists-in-charge from the
cluster weather forecast offices, hydrologists-in-charge from the river
forecast centers, and a regional representative.
Phase 3 was planned to begin in April 2008 and was to determine if
clusters of 4 offices can work as a multioffice team sharing
forecasting responsibilities. NWS planned to complete a
preliminary prototype report in July 2009 and present it to the
Corporate Board. At that time, the board would have decided
whether to implement the clustered peer concept of operations
across the country.
Appendix III: Comments from the Department of Commerce
Appendix IV: GAO Contact and Staff Acknowledgments
GAO Contact
David A. Powner, (202) 512-9286 or [32][email protected]
Staff Acknowledgments
In addition to the contact person named above, Colleen Phillips,
Assistant Director; Kate Agatone; Neil Doherty; Amos Tevelow; and
Jessica Waselkow made key contributions to this report.
GAO's Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in
meeting its constitutional responsibilities and to help improve
the performance and accountability of the federal government for
the American people. GAO examines the use of public funds;
evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at
no cost is through GAO's Web site ( [33]www.gao.gov ). Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of
newly posted products every afternoon, go to [34]www.gao.gov and
select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies
are $2 each. A check or money order should be made out to the
Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are
discounted 25 percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548
To order by Phone: Voice: (202) 512-6000
TDD: (202) 512-2537
Fax: (202) 512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: [35]www.gao.gov/fraudnet/fraudnet.htm
E-mail: [36][email protected]
Automated answering system: (800) 424-5454 or (202) 512-7470
Congressional Relations
Gloria Jarmon, Managing Director, [37][email protected] (202)
512-4400 U.S. Government Accountability Office, 441 G Street NW,
Room 7125 Washington, D.C. 20548
Public Affairs
Paul Anderson, Managing Director, [38][email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW,
Room 7149 Washington, D.C. 20548
(310824)
[41]transparent illustrator graphic
[42]www.gao.gov/cgi-bin/getrpt?GAO-07-650 .
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact David Powner at (202) 512-9286 or
[email protected].
Highlights of [43]GAO-07-650 , a report to congressional requesters
June 2007
WEATHER FORECASTING
National Weather Service's Operations Prototype Needs More Rigorous
Planning
Using advanced systems and trained specialists located in 122 weather
forecast offices throughout the country, the National Weather Service
(NWS) provides storm and flood warnings and weather forecasts to protect
life and property and to enhance the national economy. To improve the
efficiency of its operations, in November 2006, NWS approved an effort to
develop a prototype of an alternative way of operating. Under this
prototype, weather forecasting offices would share selected
responsibilities.
GAO (1) determined the status of and plans for the prototype,
(2) evaluated whether the prototype's justification was sufficient, (3)
determined whether NWS's plans to evaluate the prototype are adequate,
(4) evaluated whether NWS is sufficiently involving stakeholders in its
prototype plans, and
(5) determined how NWS plans to ensure that there will be no degradation
of service during and after the prototype. To do so, GAO analyzed agency
documentation and interviewed program officials and stakeholders.
[44]What GAO Recommends
GAO is recommending that if NWS decides to proceed with its prototype, it
undertake a more rigorous approach to justifying, evaluating, and
involving stake-holders in the prototype. In written comments, Commerce
agreed with GAO's recommendations.
NWS's prototype is currently on hold pending a reevaluation of the
agency's approach. The agency had just begun the first phase of its
three-phased prototype to demonstrate a new concept of operations over a
2-year period when, in late March 2007, the Department of Commerce's Under
Secretary suspended the prototype because of concerns about the agency's
approach. In the first phase, NWS established a program manager and began
planning for the next two phases. During the remaining phases, NWS planned
to have 20 weather forecasting offices share responsibilities in 2-office
pairs and then in 4-office clusters. NWS then planned to decide whether to
implement the new concept of operations on a national basis.
The justification for the prototype was not sufficient. Before the
prototype was suspended, the agency had approved moving forward with its
prototype without conducting a cost-benefit analysis. NWS estimated that
the prototype would cost approximately $9.3 million and would offer
qualitative benefits, such as increased efficiency and an improved ability
to focus on severe weather events, but did not quantify benefits or the
expected return on its investment. If NWS were to proceed with the
prototype without a cost-benefit analysis, it would lack assurance that
its approach would be a cost-effective investment for the agency.
NWS identified goals and selected measures to evaluate during its
prototype activities, but it did not establish a rigorous evaluation plan.
Specifically, NWS did not define a full set of needed measures, how it
planned to compare prototype results with baseline performance, or how its
selected measures supported the prototype goals. If NWS were to proceed
without a rigorous evaluation plan, the agency would run an increased risk
of not sufficiently measuring the impact of changes on its performance and
could make decisions affecting the nation's weather on the basis of
incomplete or flawed data.
Although NWS involved internal stakeholders in planning its prototype, it
did not (1) involve external stakeholders or (2) establish a plan that
identified key stakeholders, the stakeholders' responsibilities, and a
time line for involving stakeholders and addressing their comments. If NWS
were to proceed with its prototype without such a plan, it could not
ensure that stakeholder interests would be identified and addressed.
NWS planned to mitigate the risk of degradation during and after the
prototype by conducting laboratory exercises to understand the impact of
the prototype, using a dedicated workstation in each office so that the
office could switch to original systems if warranted, and monitoring its
systems and products during the prototype. NWS officials stated that
should the agency decide to implement the prototype on a national basis,
it has standard procedures for testing and validating systems and software
to avoid any degradation of service.
References
Visible links
29. http://www.gao.gov/cgi-bin/getrpt?GAO-06-792
30. http://www.gao.gov/cgi-bin/getrpt?GAO-04-394G
31. http://www.weather.gov/com/digitalera/
32. mailto:[email protected]
33. http://www.gao.gov/
34. http://www.gao.gov/
35. http://www.gao.gov/fraudnet/fraudnet.htm
36. mailto:[email protected]
37. mailto:[email protected]
38. mailto:[email protected]
39. http://www.gao.gov/cgi-bin/getrpt?GAO-06-538
40. http://www.gao.gov/cgi-bin/getrpt?GAO-05-92
42. http://www.gao.gov/cgi-bin/getrpt?GAO-07-650
43. http://www.gao.gov/cgi-bin/getrpt?GAO-07-650
*** End of document. ***