Aviation Weather: Services at Key Aviation Facilities Lack	 
Performance Measures, but Improvement Efforts Are Under Way	 
(26-FEB-08, GAO-08-491T).					 
                                                                 
The National Weather Service (NWS), an agency under the 	 
Department of Commerce's National Oceanic and Atmospheric	 
Administration (NOAA), provides staff on-site at each of the	 
Federal Aviation Administration's (FAA) en route centers--the	 
facilities that control high-altitude flight outside the airport 
tower and terminal areas. This group of NWS meteorologists	 
provides air traffic managers with forecasts and briefings on	 
regional conditions such as turbulence and icing. Over the last  
few years, FAA has been exploring options for enhancing the	 
efficiency of the aviation weather services provided by these NWS
meteorologists. In late December 2007, FAA delivered revised	 
requirements and associated performance measures to NWS to	 
improve these services. GAO was asked to summarize key segments  
of its report being released today, including its assessment of  
NWS and FAA efforts to ensure the quality of aviation weather	 
services at en route centers, and its recommendations to improve 
these efforts. In addition, GAO was asked to provide an update on
FAA's recent efforts to establish aviation weather requirements  
and performance measures, and NWS's plans for responding to these
requirements. To do so, GAO summarized segments of its report,	 
reviewed FAA's recently released requirements, and interviewed	 
the official responsible for NWS's response.			 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-08-491T					        
    ACCNO:   A81037						        
  TITLE:     Aviation Weather: Services at Key Aviation Facilities    
Lack Performance Measures, but Improvement Efforts Are Under Way 
     DATE:   02/26/2008 
  SUBJECT:   Air traffic control systems			 
	     Aircraft safety					 
	     Aviation						 
	     Aviation security					 
	     Commercial aviation				 
	     Cost effectiveness analysis			 
	     Evaluation criteria				 
	     Interagency relations				 
	     Meteorological research				 
	     Performance measures				 
	     Product evaluation 				 
	     Reporting requirements				 
	     Safety standards					 
	     Transportation safety				 
	     Weather						 
	     Weather forecasting				 
	     Weather modification				 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-08-491T

   

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Testimony: 

Before the Subcommittee on Energy and Environment, House Committee on 
Science and Technology: 

United States Government Accountability Office: 
GAO: 

For Release on Delivery: 
Expected at 1:00 p.m. EST: 
Tuesday, February 26, 2008: 

Aviation Weather: 

Services at Key Aviation Facilities Lack Performance Measures, but 
Improvement Efforts Are Under Way: 

Statement of David A. Powner, Director: Information Technology 
Management Issues: 

GAO-08-491T: 

GAO Highlights: 

Highlights of GAO-08-491T, a testimony before the Subcommittee on 
Energy and Environment, House Committee on Science and Technology. 

Why GAO Did This Study: 

The National Weather Service (NWS), an agency under the Department of 
Commerceï¿½s National Oceanic and Atmospheric Administration (NOAA), 
provides staff on-site at each of the Federal Aviation Administrationï¿½s 
(FAA) en route centersï¿½the facilities that control high-altitude flight 
outside the airport tower and terminal areas. This group of NWS 
meteorologists provides air traffic managers with forecasts and 
briefings on regional conditions such as turbulence and icing. Over the 
last few years, FAA has been exploring options for enhancing the 
efficiency of the aviation weather services provided by these NWS 
meteorologists. In late December 2007, FAA delivered revised 
requirements and associated performance measures to NWS to improve 
these services. 

GAO was asked to summarize key segments of its report being released 
today, including its assessment of NWS and FAA efforts to ensure the 
quality of aviation weather services at en route centers, and its 
recommendations to improve these efforts. In addition, GAO was asked to 
provide an update on FAAï¿½s recent efforts to establish aviation weather 
requirements and performance measures, and NWSï¿½s plans for responding 
to these requirements. To do so, GAO summarized segments of its report, 
reviewed FAAï¿½s recently released requirements, and interviewed the 
official responsible for NWSï¿½s response. 

What GAO Found: 

Although interagency agreements between NWS and FAA state that both 
agencies have responsibilities for assuring and controlling the quality 
of aviation weather observations, neither agency consistently does so 
for weather products and services produced at the en route centers. 
Specifically, neither agency has developed and implemented performance 
measures and metrics, regularly evaluated weather service unit 
performance, or provided feedback to improve these aviation weather 
products and services. Because of this lack of performance tracking and 
oversight, NWS cannot demonstrate the quality or value of its services, 
and FAA cannot ensure the quality of the services it funds. Until both 
agencies are able to measure and ensure the quality of the aviation 
weather products at the en route centers, FAA may not be getting the 
information it needs to effectively manage air traffic. 

In its report being issued today, GAO is making recommendations to the 
Secretaries of Commerce and Transportation to ensure that NWS and FAA 
develop performance measures, evaluate the services against those 
measures, and provide feedback to NWS. Commerce agreed with the 
recommendations and stated that NOAA would work with FAA to develop 
methods for performance monitoring and evaluation. Transportation did 
not agree or disagree with the recommendations, but stated that FAAï¿½s 
revised requirements would establish performance measures and 
evaluation procedures, and that FAA would negotiate with NWS to 
implement them. 

FAA has begun to address GAOï¿½s recommendations. In late December 2007, 
FAA finalized its new requirements, including performance measures and 
methods for evaluating performance and providing feedback to NWS. In 
doing so, FAA provides its overall vision for aviation weather 
services, reiterates its need for existing products and services, 
provides revisions to existing requirements, and defines a new product. 
FAA directed NWS to respond by May 2008 and include plans in its 
response for three operational conceptsï¿½in its existing configuration 
located at the 21 en route centers, through remote services provided by 
a reduced number of regional facilities, and through remote services 
provided by a single centralized facility. FAA stated that NWS should 
assume a transition time of 90 days for the existing configuration, 180 
days for regionalized services, and 1 year for a single facility. 

NWS plans to respond to FAA by the May 2008 deadline, but FAAï¿½s 
estimated time frames for transitioning to a new operational concept 
may be overly ambitious. Given the importance of accurate and timely 
weather information in air traffic control, it will be important for 
NWS to conduct a thorough evaluation before it transitions to a new 
operational concept in order to ensure that there are no impacts on the 
continuity of air traffic operations and no degradation of weather 
service. 

To view the full product, including the scope and methodology, click on 
[hyperlink, http://www.GAO-08-491T]. For more information, contact 
David A. Powner at 202-512-9286 or [email protected]. 

[End of section] 

Mr. Chairman and Members of the Subcommittee: 

We appreciate the opportunity to participate in today's hearing to 
discuss our work on the National Weather Service's (NWS) aviation 
weather services. NWS is responsible for providing storm and flood 
warnings and weather forecasts for the United States, its territories, 
and adjacent oceans and waters. NWS's weather products are also a vital 
component of the Federal Aviation Administration's (FAA) air traffic 
control program, providing weather information to local, regional, and 
national air traffic management, navigation, and surveillance systems. 
NWS aviation weather products include forecasts and warnings of 
meteorological conditions that could affect air traffic, including 
thunderstorms, air turbulence, and icing. In addition to providing 
aviation weather products developed at its own facilities, NWS also 
provides staff on-site at each of FAA's en route centers--the 
facilities responsible for controlling high-altitude air traffic 
outside the tower and terminal areas. 

Over the last few years, FAA has been exploring options for enhancing 
the efficiency of the aviation weather services provided at its en 
route centers. In September 2005, the agency asked NWS to restructure 
its services to be more efficient; in response, NWS conducted a 
prototype and proposed restructuring its offices to provide services 
remotely. FAA declined this proposal in favor of making its existing 
requirements more precise. In late December 2007, FAA delivered its 
revised requirements to NWS to improve these services. 

As requested, our testimony summarizes our report being released today 
on NWS's aviation weather services[Footnote 1] and provides an update 
on recent efforts to develop aviation weather requirements and 
performance measures. Specifically, we discuss both agencies' efforts 
to ensure the consistency and quality of these services, our 
recommendations to improve these services, FAA's recent efforts to 
establish requirements and performance measures, and NWS's plans for 
responding to these requirements. 

The information in this statement is based largely on our work 
supporting the report being released today. In addition, to provide an 
update on the agencies' recent efforts, we reviewed key documents 
completed in December 2007, including a new interagency agreement, 
FAA's requirements, and the accompanying quality assurance plan. We 
compared NWS's tentative next steps with best practices for validating 
requirements and interviewed the NWS official responsible for 
responding to the new requirements. We conducted our work on the report 
and testimony between May 2007 and February 2008, in accordance with 
generally accepted government auditing standards. Those standards 
require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings and 
conclusions based on our audit objectives. We believe that the evidence 
obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives. Additional details on our objectives, 
scope, and methodology are provided in appendix I. 

Results in Brief: 

Although interagency agreements between NWS and FAA state that both 
agencies have responsibilities for assuring and controlling the quality 
of aviation weather observations, neither agency consistently does so 
for weather products and services produced at the en route centers. 
Specifically, neither has implemented performance measures and metrics, 
regularly evaluated weather service unit performance, or provided 
feedback to improve these aviation weather products and services. 
Because of this lack of performance tracking and oversight, NWS cannot 
demonstrate the quality or value of its services, and FAA cannot ensure 
the quality of the services it funds. Until both agencies are able to 
measure and ensure the quality of the aviation weather products and 
services at the en route centers, FAA may not be getting the 
information it needs to effectively manage air traffic. 

In our report being released today, we are making recommendations to 
the Secretaries of Commerce and Transportation to ensure that NWS and 
FAA develop performance measures for aviation weather services provided 
at en route centers, evaluate the services against those measures, and 
provide feedback to the NWS staff on how to improve services. The 
Secretary of Commerce agreed with our recommendations and stated that 
the National Oceanic and Atmospheric Administration (NOAA) would work 
with FAA to develop methods for performance monitoring and evaluation. 
The Department of Transportation did not agree or disagree with our 
recommendations, but stated that FAA's revised requirements would 
establish performance measures and evaluation procedures, and that FAA 
would negotiate with NWS to implement them. 

FAA has already begun to address the recommendations noted in our 
report; specifically, in late December 2007, FAA finalized its new 
aviation weather requirements, which include proposed performance 
measures and methods for evaluation. In its requirements, FAA provides 
NWS with an overall vision for aviation weather services that are 
performance-based, standardized, continuous, and have a national scope. 
FAA reiterates its need for existing products and services (such as 
twice-daily briefings), provides revisions to some existing 
requirements, and defines a new graphical forecast product for terminal 
radar approach control facilities. In addition, FAA identifies 
performance measures--such as customer satisfaction and forecast 
accuracy--and processes for evaluating performance and providing 
feedback to NWS. FAA expects NWS to respond as to whether it is able to 
meet the requirements by early May 2008, and has directed NWS to 
include plans for three operational concepts to fulfill the 
requirements--in its existing configuration located at the 21 en route 
centers, through remote services provided by a reduced number of 
regional facilities, and through remote services provided by a single 
centralized facility. FAA plans to select one of the operational 
concepts and NWS will immediately begin to transition to the new 
concept. FAA required that NWS assume a transition time of 90 days if 
it selects the existing configuration, 180 days if it selects the 
regionalized remote services concept, and 1 year if it selects the 
single facility concept. 

NWS plans to respond to FAA by the May 2008 deadline, but FAA's 
estimated time frames for providing the revised services may be overly 
ambitious. NWS created a team to analyze FAA's requirements and to 
develop a response package for all three operational concepts. The NWS 
official responsible for aviation services reported that the agency is 
on track to respond by FAA's deadline of May 2008. However, FAA's 
estimated time frames for transitioning to a new operational concept 
may be overly ambitious. Given the importance of accurate and timely 
weather information in air traffic control, it will be important for 
NWS to conduct a thorough evaluation before it transitions to a new 
operational concept in order to ensure that there are no impacts on the 
continuity of air traffic operations and no degradation of weather 
service. 

Background: 

FAA is responsible for ensuring safe, orderly, and efficient air travel 
in the national airspace system. NWS supports FAA by providing aviation-
related forecasts and warnings at air traffic facilities across the 
country. Among other support and services, NWS provides four 
meteorologists at each of FAA's 21 en route centers to provide on-site 
aviation weather services. This arrangement is defined and funded under 
an interagency agreement. 

FAA's Mission and Organizational Structure: 

FAA's primary mission is to ensure safe, orderly, and efficient air 
travel in the national airspace system. The agency's ability to fulfill 
its mission depends on the adequacy and reliability of its air traffic 
control systems, as well as weather forecasts made available by NWS and 
automated systems. These resources reside at, or are associated with, 
several types of facilities: air traffic control towers, terminal radar 
approach control facilities, air route traffic control centers (en 
route centers), and the Air Traffic Control System Command Center. The 
number and functions of these facilities are as follows: 

* 517 air traffic control towers manage and control the airspace within 
about 5 miles of an airport. They control departures and landings, as 
well as ground operations on airport taxiways and runways. 

* 170 terminal radar approach control facilities provide air traffic 
control services for airspace within approximately 40 miles of an 
airport and generally up to 10,000 feet above the airport, where en 
route centers' control begins. Terminal controllers establish and 
maintain the sequence and separation of aircraft. 

* 21 en route centers control planes over the United States--in transit 
and during approaches to some airports. Each center handles a different 
region of airspace. En route centers operate the computer suite that 
processes radar surveillance and flight planning data, reformats them 
for presentation purposes, and sends them to display equipment that is 
used by controllers to track aircraft. The centers control the 
switching of voice communications between aircraft and the center, as 
well as between the center and other air traffic control facilities. 
Two en route centers also control air traffic over the oceans. 

* The Air Traffic Control System Command Center manages the flow of air 
traffic within the United States. This facility regulates air traffic 
when weather, equipment, runway closures, or other conditions place 
stress on the national airspace system. In these instances, traffic 
management specialists at the command center take action to modify 
traffic demands in order to keep traffic within system capacity. 

See figure 1 for a visual summary of the facilities that control and 
manage air traffic over the United States. 

Figure 1: FAA Facilities Involved in Air Traffic Control: 

[See PDF for image] 

This figure is an illustration of FAA Facilities Involved in Air 
Traffic Control. The following data is depicted: 

Air Traffic Control System Command Center: 
Preflight: Air traffic control tower; 
Takeoff: Air traffic control tower; 
Departure: Terminal radar approach control; 
En route: Air route traffic control center; 
Descent: Air route traffic control center; Terminal radar approach 
control; 
Approach: Terminal radar approach control; 
Landing: Air traffic control tower. 

Source: GAO analysis of FAA data. 

[End of figure] 

NWS's Mission and Organizational Structure: 

The mission of NWS--an agency within the Department of Commerce's NOAA--
is to provide weather, water, and climate forecasts and warnings for 
the United States, its territories, and its adjacent waters and oceans 
to protect life and property and to enhance the national economy. In 
addition, NWS is the official source of aviation- and marine-related 
weather forecasts and warnings, as well as warnings about life-
threatening weather situations. 

The coordinated activities of weather facilities throughout the United 
States allow NWS to deliver a broad spectrum of climate, weather, 
water, and space weather services in support of its mission. These 
facilities include 122 weather forecast offices located across the 
country that provide a wide variety of weather, water, and climate 
services for their local county warning areas, including advisories, 
warnings, and forecasts; 9 national prediction centers[Footnote 2] that 
provide nationwide computer modeling to all NWS field offices; and 21 
center weather service units that are located at FAA en route centers 
across the nation and provide meteorological support to air traffic 
controllers. 

NWS Provides Aviation Weather Services to FAA: 

As an official source of aviation weather forecasts and warnings, 
several NWS facilities provide aviation weather products and services 
to the FAA and aviation sector. These facilities include the aviation 
weather center, weather forecast offices located across the country, 
and center weather service units located at FAA en route centers. See 
table 1. 

Table 1: NWS Offices That Provide Aviation Weather Products and 
Services to FAA: 

Office: Aviation weather center; 
Description: The aviation weather center, located in Kansas City, 
Missouri, issues warnings, forecasts, and analyses of hazardous weather 
for aviation. Staffed by 65 personnel, the center develops warnings of 
hazardous weather for aircraft in flight and forecasts of weather 
conditions for the next 2 days that could affect both domestic and 
international aviation. The center also leads a collaborative effort to 
develop a forecast of expected convective events for the entire country 
every 2 hours. This is used by FAA to manage aviation traffic flow 
across the country. 

Office: Weather forecast offices; 
Description: NWS's 122 weather forecast offices issue terminal area 
forecasts for approximately 625 locations every 6 hours or when 
conditions change. These forecasts consist of the expected weather 
conditions significant to a given airport or terminal and are primarily 
used by commercial and general aviation pilots. 

Office: Center weather service units; 
Description: NWS's center weather service units are located at each of 
FAA's 21 en route centers and operate 16 hours a day, 7 days a week 
(see fig. 2). Each weather service unit usually consists of three 
meteorologists and a meteorologist-in-charge who provide strategic 
advice and aviation weather forecasts to FAA traffic management 
personnel. Governed by an interagency agreement, FAA currently 
reimburses NWS approximately $12 million annually for this support. 

Source: GAO analysis of NWS and FAA data. 

[End of table] 

Figure 2: Center Weather Service Unit Locations and Service Areas: 

[See PDF for image] 

This figure is a map of the United States, illustrating Center Weather 
Service Unit Locations and Service Areas. The following locations are 
depicted: 

Boston; 
New York; 
Washington, DC; 
Cleveland; 
Indianapolis; 
Memphis; 
Atlanta; 
Jacksonville; 
Miami; 
Houston; 
Fort Worth; 
Kansas City; 
Chicago; 
Minneapolis; 
Denver; 
Albuquerque; 
Salt Lake City; 
Los Angeles; 
Oakland; 
Seattle; 
Anchorage. 

Source: NWS (data). Map Resources (map). 

[End of figure] 

Center Weather Service Units: An Overview of Current Requirements: 

FAA's existing requirements for the center weather service units are 
broadly outlined in an interagency agreement that is updated every few 
years. The interagency agreement specifies that NWS is to provide 
meteorological advice and consultation to en route center operations 
personnel and other designated FAA air traffic facilities within the en 
route area of responsibility. This agreement establishes specific terms 
that govern the number of NWS staff, their working hours, and cost 
reimbursement details. It does not specify the contents, quality, or 
frequency of weather products. 

An NWS directive, signed in May 2006 and intended for NWS's weather 
forecast offices and center weather service units, provides more 
specific information regarding the content of weather products and 
services, including center weather advisories, daily briefings, on- 
demand consultations, and meteorological impact statements. These 
products and services are described in table 2. In addition, center 
weather service unit meteorologists can provide input every 2 hours to 
the Aviation Weather Center's creation of the Collaborative Convective 
Forecast Product; train FAA personnel on how to interpret weather 
information; and, if warranted, provide weather briefings to nearby 
terminal radar approach control facilities. 

Table 2: Key Products and Services Currently Provided by Center Weather 
Service Units: 

Product or service: Center weather advisory; 
Description: A short-term, unscheduled warning of hazardous weather 
conditions used primarily by air crews to anticipate and avoid adverse 
weather conditions in the en route and terminal environments. It 
describes current weather conditions or adverse weather conditions--
such as moderate to severe icing or turbulence, thunderstorms, and low 
ceilings and visibility-- beginning within the next 2 hours. 

Product or service: Briefings; 
Description: Short updates provided by en route center meteorologists 
to FAA supervisors twice a day; they include current weather 
advisories, a summary of the predicted weather in the en route area, 
terminal forecasts, and jet stream and freezing information. 

Product or service: On-demand consultation; 
Description: Unscheduled verbal presentations provided to traffic 
management controllers, supervisors, and other FAA facilities within 
the en route center area. Consultations may be about the expected 
weather conditions or interpretations of weather information from the 
satellite images. 

Product or service: Meteorological impact statement; 
Description: An unscheduled forecast of weather conditions that are 
expected to adversely impact the flow of air traffic in the en route 
center's area of responsibility within 4 to 12 hours. These statements 
detail weather conditions expected to adversely impact air traffic flow 
in the service unit area of responsibility and should include the 
location, height, extent, and movement of the weather conditions. 

Source: GAO analysis of FAA and NWS data. 

[End of table] 

FAA Sought to Improve Aviation Weather Services Provided at En Route 
Centers: 

In recent years, FAA has sought to assess and improve the performance 
of the center weather service units.[Footnote 3] For example, FAA 
performed multiple studies on the current services provided by the 
center weather service units that noted the lack of standardization of 
products and services. In addition, FAA conducted a study to determine 
if remote operations were feasible, and requested that NWS restructure 
its aviation weather services to provide improved services more 
efficiently. In response to this request, NWS conducted a prototype of 
remote operations in which center weather service unit products and 
services were prepared by the closest weather forecast office. NWS 
proposed expanding this prototype to FAA, but the agency declined this 
proposal. Instead, FAA stated that it would redefine its requirements 
for the functions provided by center weather service units. Table 3 
provides more information about the agency's assessment and improvement 
efforts. 

Table 3: Recent Assessment and Improvement Efforts by FAA and NWS: 

Time frame: November 2003; 
Activity: FAA performed a functional audit of center weather service 
units and found that the services provided at different en route 
locations were inconsistent, the products were not standardized, and 
there was little communication and collaboration between neighboring 
service units. 

Time frame: September 2005; 
Activity: FAA requested that NWS restructure its aviation weather 
services to provide improved services more efficiently. 

Time frame: January 2006; 
Activity: FAA initiated an analysis of the value of different 
activities performed by the center weather service units. Similar to 
the 2003 study, the results of this analysis noted the lack of 
standardization of products, services, tools, and procedures. In 
addition, the report found that quality assurance was provided on an 
informal basis, there was no formal feedback process for products and 
services, and meteorological training was not standardized. 

Time frame: August 2006; 
Activity: NWS conducted a prototype in which center weather service 
unit products and services were completed and delivered remotely from 
the closest weather forecast office. This prototype showed that remote 
operations were possible and effective, but that they would be 
difficult to implement because of the need for cultural change, 
technology upgrades, and communication stability. Specifically, 
forecasters in the prototype were not able to provide dedicated support 
for the aviation mission because their other duties-
-including forecasting severe weather at the weather forecast office-- 
took precedence. In addition, a collaboration technology used during 
the prototype was not operationally ready to use, servers were 
unstable, critical radar data were inconsistent with weather forecast 
office data, and communications lines were unstable throughout the 
prototype. 

Time frame: September 2006; 
Activity: An FAA study confirmed that it is possible to deliver weather 
information, products, and services from one or many remote locations 
with currently available state-of-the-art technology platforms. 

Time frame: October 2006; 
Activity: FAA administered a market survey to determine whether the 
private sector could provide remote weather services at a lower cost 
than currently provided. Ten organizations, including private sector 
firms and government-funded laboratories, responded that they could 
provide the services that FAA wanted; NWS presented its proposal for 
restructuring its aviation weather services to FAA. In this proposal, 
NWS suggested moving meteorologists from the en route centers to 
weather forecast offices, and providing remote aviation weather 
services from the weather forecast offices. 

Time frame: April 2007; 
Activity: FAA declined NWS's proposal. Instead, FAA reported that it 
would redefine its requirements for the functions provided by the 
center weather service units. 

Source: GAO analysis of NWS and FAA data. 

[End of table] 

FAA Found Its Requirements Were Not Sufficiently Precise and Worked to 
Develop New Ones: 

When FAA declined NWS's proposal for restructured aviation weather 
services, it did so in part because it considered its existing 
requirements governing NWS's center weather service units to be too 
broad to ensure the efficiency and cost-effectiveness of the services. 
FAA then worked for several months to redefine these requirements. In 
April 2007, FAA's Air Traffic Organization began refining its 
requirements for aviation weather services at the en route centers. To 
do this, FAA collected relevant NWS and FAA orders and directives and 
developed a list of over 100 products and services that the different 
service units provide. FAA then sent this list to traffic managers in 
each of the en route centers, asking them to specify the products and 
services that they need, the ones they do not need, and any new 
products or services that they would like. Traffic managers were also 
able to determine if they would want some of the more customized 
weather products that are currently available at selected en route 
centers. Using results from this survey, FAA developed a revised list 
of requirements and performance measures, which it provided to NWS in 
late December 2007. 

Neither NWS nor FAA Currently Ensures the Quality of Aviation Weather 
Services at En Route Centers: 

While interagency agreements between NWS and FAA state that both 
agencies have responsibilities for assuring and controlling the quality 
of aviation weather observations, neither NWS nor FAA consistently does 
so for weather products produced at the en route centers. Leading 
organizations use quality assurance to provide staff and management 
with objective insights into processes and associated work products 
[Footnote 4]. Generally, quality assurance includes objectively 
evaluating performed processes, work products, and services against 
applicable process descriptions, standards, and procedures; identifying 
and documenting noncompliance issues; providing feedback to project 
staff and managers on the results of quality assurance activities; and 
ensuring that noncompliance issues are addressed. However, neither NWS 
nor FAA has developed and implemented performance measures and metrics, 
regularly evaluated weather service unit performance, or provided 
feedback to improve these aviation weather products and services. 

Because of this lack of performance tracking and oversight, NWS cannot 
demonstrate the quality or value of its services, and FAA cannot ensure 
the value of the services it funds. As a result, it is not clear that 
FAA is getting the information it needs to effectively manage air 
travel. FAA officials stated that they intend to establish performance 
measures for their redefined requirements and to improve their 
oversight against these measures. However, FAA has not worked with NWS 
to define a comprehensive set of measures for its requirements, and it 
is unclear how the agency would develop a performance baseline for 
comparison to actual performance because many of the products and 
services have not previously been measured. 

NWS Does Not Measure or Evaluate Aviation Weather Products and Services 
at En Route Centers: 

NWS does not measure or evaluate the aviation weather services it 
provides at en route centers. Under existing interagency agreements, 
NWS is responsible for controlling the quality of its aviation weather 
observations. Specifically, the agency is responsible for monitoring 
and evaluating the quality and effectiveness of its aviation weather 
services, including the services provided at the weather forecast 
offices, the Aviation Weather Center, and the en route centers. 

While NWS has developed and continues to monitor performance measures 
for aviation weather forecasts provided by its weather forecast offices 
and the Aviation Weather Center, the agency has not done so for the 
weather products and services provided at the en route centers. 
Specifically, NWS has not developed performance measures for aviation 
weather products and services at en route centers, evaluated the 
aviation weather products and services developed at the en route 
centers, or provided feedback for those services. NOAA and NWS 
officials declined to explain why the agency does not have performance 
measures for aviation weather products or services at en route centers, 
but they noted that neither FAA nor NWS has required or funded such an 
effort. Further, the aviation services branch chief told us that he had 
planned to begin evaluations for weather unit services at the en route 
centers but decided to wait because of the potential for large-scale 
changes to the services. 

Until NWS establishes performance measures and evaluates the quality 
and effectiveness of its products against these measures, the agency 
will remain unable to ensure that it provides consistent quality 
products and to effectively demonstrate the value it provides to FAA. 

FAA Does Not Consistently Evaluate or Provide Feedback on Aviation 
Weather Services at En Route Centers: 

FAA has not consistently evaluated NWS services at its en route centers 
or adequately provided feedback on the results of its few evaluations. 
Under interagency agreements, FAA is responsible for ensuring that 
aviation weather services meet its requirements. In addition, it 
requires the traffic management officer within each traffic management 
unit to evaluate the aviation weather services at the en route centers 
annually and to provide feedback to the resident meteorologist-in- 
charge. 

FAA has not consistently ensured the quality of aviation weather 
services at en route centers. Specifically, it currently does not have 
any quantitative and objective performance measures--such as 
timeliness, accuracy, or false alarm rate--by which to evaluate these 
services. Agreements between the agencies broadly specify the types of 
aviation weather products to be developed at the en route centers but 
do not provide criteria by which these products can be evaluated. In 
addition, FAA has not consistently performed its annual evaluations of 
these products and services. According to the contracting officer's 
technical representative responsible for the evaluations, the last 
evaluation was performed in 2006, and its results were largely 
anecdotal. Specifically, the evaluation called for the traffic 
management officer to rate the weather unit on a scale of 0 to 4 in 
different categories, including quality and timeliness of products and 
services, knowledge of air traffic control, and participation in 
training. The technical representative could not find any evaluations 
in 2005, evaluations of only three service units in 2004, and 
evaluations of a similarly small number of service units in 2003. 

Further, FAA is not consistently providing feedback to weather staff at 
the en route centers. According to the technical representative, the 
evaluations from 2006 were not compiled or analyzed because the 
evaluations contained no glaring problems or issues that needed 
additional attention. In addition, the NWS aviation services chief told 
us that FAA had sent him copies of the evaluations from 2006 but did 
not offer analysis of these evaluations, express concerns about the 
services, or send the results to the individual center weather service 
units. This official also stated that he was not aware that FAA had 
performed any annual evaluations of the center weather service units 
prior to 2006. 

Because FAA has not established performance requirements or 
consistently and thoroughly evaluated the aviation weather services at 
en route centers, the agency cannot be sure that the products and 
services provided by the weather unit meteorologists are adding value, 
and it cannot provide feedback to NWS meteorologists in order to 
improve the services. To address this shortfall, FAA officials stated 
that they intend to establish performance measures for aviation weather 
services at en route centers when they revise their requirements and to 
improve their oversight of NWS against these measures. However, FAA has 
not worked with NWS to develop measures for the products and services 
it will require from NWS, and it is unclear how the agency would 
develop a performance baseline for comparison to actual performance 
because many of the products and services have not previously been 
measured. 

Implementation of GAO Recommendations Should Help Address Performance 
Measurement Shortfalls: 

While many steps remain in defining the future of aviation weather 
services at en route centers--including negotiations between FAA and 
NWS on the provision of these services and FAA's subsequent decision on 
whether to obtain selected services from alternative sources--there are 
steps both agencies can take to ensure that the quality of future 
aviation weather products and services is measured and evaluated. In 
our accompanying report released today,[Footnote 5] we made two 
recommendations to the Secretary of Commerce and three recommendations 
to the Secretary of Transportation to improve the quality of aviation 
weather products and services at en route centers. 

We recommended that the Secretary of Commerce direct the Assistant 
Administrator for the National Weather Service to: 

* assist FAA in developing performance measures and metrics for the 
products and services to be provided by center weather service units, 
and: 

* perform annual evaluations of aviation weather services provided at 
en route centers and provide feedback to the center weather service 
units. 

Further, we recommended that the Secretary of the Department of 
Transportation direct the FAA Administrator to: 

* work with NWS to define performance measures and metrics for aviation 
weather services provided by meteorologists, 

* evaluate the services it receives against those measures and metrics, 
and: 

* ensure that results of these evaluations are provided to staff 
stationed at center weather service units so that they can improve 
performance, where applicable. 

In written comments on a draft of our report, the Secretary of Commerce 
agreed with our recommendations to assist FAA in developing performance 
measures and metrics, and to perform annual evaluations of aviation 
weather services and provide feedback to the center weather service 
units. The department stated that after FAA provides its revised 
requirements, it would work with FAA to develop methods for performance 
monitoring and evaluation. Subsequently, on December 19, 2007, FAA 
provided its revised requirements to NWS. 

The Department of Transportation's Director of Audit Relations provided 
comments on a draft of the report via e-mail. In those comments, the 
department did not agree or disagree with our recommendations. The 
department stated that FAA's revised requirements are consistent with 
our recommendations in that they establish performance measures and 
evaluation procedures, and that FAA would begin to negotiate with NWS 
to implement them. In addition, in late December 2007, after reviewing 
our draft report, FAA and NWS signed a new interagency agreement that 
requires FAA to develop performance standards and measures for the 
assessment of center weather service units, and requires NWS to develop 
and track metrics to support FAA's performance measures. 

FAA Identified New Aviation Weather Requirements and Performance 
Measures: 

FAA has already begun to address our recommendations; specifically, in 
late December 2007, FAA finalized its new requirements for the aviation 
weather services to be provided by center weather service units, which 
include proposed performance measures and methods for evaluation. In 
its requirements, FAA provides NWS with its overall vision for aviation 
weather services, revises existing requirements, and defines a 
requirement for a new product for terminal radar approach control 
facilities. In addition, FAA identifies performance measures and 
processes for evaluating performance and providing feedback to NWS. 

FAA envisions services that are performance-based, standardized, 
continuous, and have a national scope. Specifically, FAA requires 
performance-based services that are measurable and allow for 
identifying both successful performance and any performance problems. 
In addition, FAA requests that the center weather service units provide 
standardized services to all en route centers and increase their 
service coverage from 16 hours a day to 24 hours a day. Finally, FAA 
calls for transitioning the scope of the center weather service units 
to monitor the entire national airspace system, rather than the 
respective en route center regions. This national scope is expected to 
allow more integrated decision making at the national level while 
continuing to provide specialized products at the regional and local 
levels. 

In its new requirements, FAA also reiterates its need for existing 
products and services and provides revisions to some of these 
requirements. Specifically, FAA continues to require products such as 
twice-daily briefings, center weather advisories, and the Collaborative 
Convective Forecast Product. In addition, center weather service units 
will continue to provide forecast coordination with other NWS offices, 
on-demand advice and consultation, emergency planning, training, and 
dissemination of a number of weather advisories into both NWS and FAA 
systems. Daily briefings, however, will now be recorded, verified, and 
disseminated to other facilities that do not receive an in-person 
briefing. In addition, on-demand consultation will be provided to en 
route centers, terminal radar approach control facilities, towers, and 
the Air Traffic System Command Center. According to the aviation 
services branch chief, this consultation is currently provided only to 
en route centers, selected terminal radar approach control facilities, 
and a small number of towers. 

Further, the revised requirements define a new product: the terminal 
radar approach control forecast product. This product is based in part 
on decision aids currently used in select center weather service units, 
and on requirements developed by a team consisting of aviation 
meteorological stakeholders from industry and FAA. This forecast, which 
describes the next 6 hours and is updated at least every 2 hours, will 
be presented in a graphical format and include convection, winds, 
ceilings, and visibilities for the area around terminal radar approach 
control facilities. FAA also expects this product to include methods 
for verification and the systematic collection of user feedback. 

In addition to these requirements, FAA identifies performance measures 
as well as processes for evaluating performance and providing feedback 
to the forecasters. These performance measures include customer 
satisfaction, forecast accuracy, and the aggregate of aircraft 
incidents attributed to inaccurate aviation weather forecasts. 
Baselines for all of these measures have not yet been developed. 
According to the chief of the aviation services branch, NWS will 
propose additional performance measures and develop baselines as it is 
able. To measure against these performance measures, FAA has identified 
methods by which to evaluate NWS. For example, to determine customer 
satisfaction, FAA plans to develop a questionnaire for traffic 
management unit staff to be filled out quarterly; to determine the 
aggregate of aircraft incidents attributed to inaccurate aviation 
weather forecasts, FAA is to use safety statistics currently tracked by 
the Air Traffic Organization. In addition, FAA is planning to draw on 
NWS's subject matter expertise to record and analyze information to 
determine the accuracy of forecasts. 

FAA expects NWS to respond as to whether it is able to meet the 
requirements by early May 2008. In addition, FAA directed NWS to 
include plans for three operational concepts (including technical and 
cost information) for fulfilling the requirements--in its existing 
configuration located at the 21 en route centers, through remote 
services provided by a reduced number of regional facilities, and 
through remote services provided by a single centralized facility. 
According to the requirements, NWS's response should assume a 
transition time of 90 days for the existing configuration, 180 days for 
regionalized remote services, and 1 year for a single facility. For 
each of these concepts, FAA has requested that NWS include a technical 
plan, a facilities plan, a quality management plan, and a plan for 
transitioning to the new approach. In addition, FAA asked that NWS 
include cost plans for each of the concepts with the assumption of 1 
base year and four 1-year options thereafter. The cost plan is also 
expected to include estimated annual cost savings over this 5-year 
period. FAA plans to select one of these operational concepts. 

NWS Plans to Respond to FAA's Requirements, but Proposed Transition 
Time Frames May Be Overly Ambitious: 

NWS plans to respond to FAA by the May 2008 deadline, but FAA's 
estimated time frames for providing the revised services may be overly 
ambitious. NWS plans to submit its proposals for the three operational 
concepts--the existing configuration located at the 21 en route 
centers, remote services provided by a reduced number of regional 
facilities, and remote services provided by a single centralized 
facility. FAA directed NWS to assume a transition time of 90 days for 
the existing configuration, 180 days for regionalized remote services, 
and 1 year for a single facility. To respond, NWS management created a 
team to analyze the requirements, gain clarification on the 
requirements from FAA, and develop the response. The NWS aviation 
services branch chief reported that the agency is on track to respond 
by May 2008. 

However, FAA's proposed time frames for transitioning to a new 
operational concept may be overly ambitious because of the activities 
that NWS should perform before any transition. In accordance with best 
practices and the opinion of the National Transportation Safety Board, 
NWS intends to validate that the organization can provide these new 
requirements--through a prototype or similar demonstration--before 
transitioning to a new approach. Leading organizations validate 
requirements to determine what impact the intended operational 
environment will have on the ability to satisfy the stakeholders' 
needs, expectations, constraints, and interfaces. As part of this 
validation, organizations explore the adequacy and completeness of 
requirements by developing prototypes or simulations and by obtaining 
feedback about them from relevant stakeholders. Given the importance of 
accurate and timely weather information in air traffic control, it will 
be important for NWS to conduct a thorough evaluation before it 
transitions to a new operational concept in order to ensure that there 
are no impacts on the continuity of air traffic operations and no 
degradation of weather service. 

In addition, NWS has agreed to negotiate with its employees' union, the 
National Weather Service Employees Organization, whenever 
organizational changes could affect working conditions--unless the 
union has sufficient predecisional involvement. NWS's employees' union 
has a representative on the team that is responding to FAA's 
requirements, so that later negotiations may be unnecessary. However, 
it is too soon to determine whether negotiations will be needed and how 
long they will take. 

NWS's aviation services branch chief agreed that FAA's transition time 
lines would be challenging. This official estimated that it could take 
over 2 years to transition to a new operational concept. To address 
this disconnect between NWS's capabilities and FAA's expectations, NWS 
plans to propose more feasible time frames in its response to FAA. FAA 
officials reported that the agency will be open to NWS's proposal. 

In summary, even though center weather service units have been in 
operation for over two decades, neither FAA nor NWS has implemented 
performance measures and metrics, regularly evaluated weather service 
unit performance, or provided feedback to improve these aviation 
weather products and services. Until the agencies establish a system of 
performance tracking and oversight, NWS will not be able to demonstrate 
the quality or value of its services, and FAA will not be able to 
ensure the value of the services it funds. To address these shortfalls, 
FAA has defined more precise requirements that include performance 
measures and evaluation methods, and NWS is working to respond to these 
requirements. In its response, NWS is expected to prepare plans for 
three alternative operational concepts for fulfilling these 
requirements. FAA will choose one of the concepts. However, FAA's 
proposed time frames for transitioning to a new operational concept may 
be overly ambitious. Given the potential for major changes to services 
and structure, NWS may require additional time to properly validate the 
requirements, plan for any necessary operational transitions, and 
ensure that aviation weather services are not degraded by any potential 
changes. 

Moving forward, it is important that FAA and NWS work together to 
ensure that NWS's aviation weather services address requirements and 
that effective performance measures and evaluation methods are 
established. This collaboration will help both agencies ensure the 
quality and consistency of these services, and ensure that FAA has the 
information it needs to effectively manage air traffic. 

Mr. Chairman, this concludes my statement. I would be happy to answer 
any questions that you or members of the subcommittee may have at this 
time. 

If you have any questions on matters discussed in this testimony, 
please contact me at (202) 512-9286 or by e-mail at [email protected]. 
Other key contributors to this testimony include Colleen Phillips 
(Assistant Director), Kate Agatone, Amos Tevelow, and Jessica Waselkow. 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

The objectives of this statement were to summarize selected sections of 
our report[Footnote 6] being released today, including National Weather 
Service (NWS) and Federal Aviation Administration (FAA) efforts to 
ensure the consistency and quality of aviation weather services at en 
route centers, and our recommendations to improve these services. In 
addition, we were asked to provide an update on FAA's recent efforts to 
develop aviation weather requirements and performance measures, and 
NWS's plans for responding to these requirements. 

To summarize selected sections of our report, we relied on the work 
supporting that report. That report contains a detailed overview of our 
scope and methodology. 

In addition, to provide an update on FAA's recent efforts to establish 
requirements, we reviewed the new interagency agreement signed by both 
agencies in late December 2007; FAA's requirements sent to NWS on 
December 19, 2007; and the accompanying quality assurance plan. We also 
interviewed NWS's aviation services branch chief to clarify which of 
these requirements were new and which were revisions of current 
requirements. 

To determine NWS's plans for responding to these requirements, we 
reviewed the new interagency agreement, FAA's requirements, and the 
accompanying quality assurance plan. We also interviewed the aviation 
services branch chief, who is serving in an oversight role for NWS's 
response to FAA. We compared NWS's tentative next steps with best 
practices for validating requirements from the Capability Maturity 
Modelï¿½ Integration for Development. 

We performed our work on the report and testimony from May 2007 to 
February 2008. All of the work on which this testimony is based was 
performed in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

[End of section] 

Footnotes: 

[1] GAO, Aviation Weather: FAA Is Reevaluating Services at Key Centers; 
Both FAA and the National Weather Service Need to Better Ensure Product 
Quality, GAO-08-258 (Washington, D.C.: Jan. 11, 2008). 

[2] These centers include the National Centers for Environmental 
Prediction Central Operations, Aviation Weather Center, Environmental 
Modeling Center, Hydrometeorological Prediction Center, Ocean 
Prediction Center, Storm Prediction Center, Tropical Prediction Center/ 
National Hurricane Center, Climate Prediction Center, and Space 
Environment Center. 

[3] FAA is also involved in a longer-term initiative to increase the 
efficiency of the national airspace system and to improve its overall 
safety. This initiative, called the Next Generation Air Transportation 
System, is a joint effort of the Department of Transportation, the 
National Aeronautics and Space Administration, the White House Office 
of Science and Technology Policy, and the Departments of Homeland 
Security, Defense, and Commerce. FAA anticipates that this initiative 
may lead to major changes in the aviation weather program that would 
supersede its current efforts. 

[4] The Carnegie Mellon University's Software Engineering Institute is 
recognized for its expertise in software and system processes. See 
Carnegie Mellon University Software Engineering Institute, Capability 
Maturity Modelï¿½ Integration for Development Version 1.2 (Pittsburgh, 
PA: August 2006). 

[5] GAO-08-258. 

[6] GAO, Aviation Weather: FAA Is Reevaluating Services at Key Centers; 
Both FAA and the National Weather Service Need to Better Ensure Product 
Quality, GAO-08-258 (Washington, D.C.: Jan. 11, 2008). 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "Subscribe to Updates." 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office: 
441 G Street NW, Room LM: 
Washington, D.C. 20548: 

To order by Phone: 
Voice: (202) 512-6000: 
TDD: (202) 512-2537: 
Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: [email protected]: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, [email protected]: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, [email protected]: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: 

*** End of document. ***