Air Traffic Control: FAA's Modernization Investment Management Approach
Could Be Strengthened (Chapter Report, 04/30/99, GAO/RCED/AIMD-99-88).
Pursuant to a congressional request, GAO reviewed the Federal Aviation
Administration's (FAA) modernization investment management approach as
carried out through the Acquisition Management System (AMS), focusing on
the extent to which FAA, through AMS: (1) has established a structured
approach for selecting and controlling its investments; (2) incorporates
all investments, including those in operation, in the agency's
portfolio; and (3) selects, controls, and evaluates its investments with
complete and reliable information.
GAO noted that: (1) AMS is a good first step in establishing a
structured investment management approach for selecting and controlling
the agency's investments; (2) the system contains a set of policies,
procedures, and reporting requirements to analyze mission needs, assess
the affordability of proposed projects, and establish life-cycle costs,
schedules, benefits, and performance baselines to control the
performance of the projects that are selected; (3) under this system, a
senior management investment review group makes key decisions about
which investments best meet the agency's needs and are to be funded; (4)
however, the system is not comprehensive in that it does not incorporate
all of FAA's projects into a complete strategic investment portfolio;
(5) key decisionmaking processes and requirements of AMS are applied
only to proposed projects and those under development but not to
projects already in operation; (6) agency officials have not yet
developed a sound estimate of the costs to operate projects and these
costs are not included in the agency's financial plan for modernization;
(7) because FAA does not apply the same scrutiny to all of its projects,
senior officials are unable to fully assess and make trade-offs about
the relative merits of spending funds to develop new systems, to enhance
current systems, or to continue operating and maintaining existing
systems; (8) AMS does not provide complete and reliable information for
selecting, controlling, and evaluating the agency's investments; (9) the
cost data used to select projects are of questionable reliability
because of weaknesses in FAA's cost estimating practices and processes
and the lack of a cost accounting system; (10) the information used to
control projects is incomplete since FAA has not fully implemented an
effective process for controlling the baselines for the costs,
schedules, benefits, performance, and risks of its investments; (11) FAA
has approved the baseline information for only half of the required
universe of projects, and the agency's processes for tracking actual
performance against estimates frequently has provided incomplete
information; and (12) FAA lacks information needed to evaluate its
investments since AMS does not have a post-implementation evaluation
process for assessing projects' outcomes and feeding lessons learned
back into the selection and control phases to help improve its
management of future projects.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: RCED/AIMD-99-88
TITLE: Air Traffic Control: FAA's Modernization Investment
Management Approach Could Be Strengthened
DATE: 04/30/99
SUBJECT: Air traffic control systems
Strategic information systems planning
Performance measures
Information resources management
Cost analysis
ADP procurement
Procurement planning
Evaluation methods
Mission budgeting
IDENTIFIER: FAA Acquisition Management System
FAA National Airspace System Plan
FAA Wide Area Augmentation System
FAA Standard Terminal Automation Replacement System
FAA Oceanic Automation System
FAA Voice Switching and Control System
FAA Operational Supportability and Implementation System
FAA Simplified Program Information Reporting and
Evaluation System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. This text was extracted from a PDF file. **
** Delineations within the text indicating chapter titles, **
** headings, and bullets have not been preserved, and in some **
** cases heading text has been incorrectly merged into **
** body text in the adjacent column. Graphic images have **
** not been reproduced, but figure captions are included. **
** Tables are included, but column deliniations have not been **
** preserved. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
AIR TRAFFIC CONTROL: FAs Modernization Investment Management
Approach Could Be Strengthened GAO/RCED/AIMD-99-88 United States
General Accounting Office
GAO Report to Congressional Requesters
April 1999 AIR TRAFFIC CONTROL FAA's Modernization Investment
Management Approach Could Be Strengthened
GAO/ RCED/ AIMD- 99- 88
GAO United States General Accounting Office
Washington, D. C. 20548 Resources, Community, and Economic
Development Division
B-279992 April 30, 1999
The Honorable Slade Gorton Chairman The Honorable John D.
Rockefeller IV Ranking Minority Member Subcommittee on Aviation
Committee on Commerce, Science, and Transportation United States
Senate
The Honorable John J. Duncan Chairman The Honorable William O.
Lipinski Ranking Democratic Member Subcommittee on Aviation
Committee on Transportation and Infrastructure House of
Representatives
In response to your request, this report addresses the extent to
which the Federal Aviation Administration's (FAA) Acquisition
Management System provides a comprehensive approach for managing
the agency's investments in air traffic control information
technology. FAA plans to spend billions of dollars to replace
data- processing, navigation, communications, and other systems
under its air traffic control modernization program but has a
history of poor performance in delivering systems on time and
within budget and performance parameters. We found that FAA has
established a structured approach for managing its modernization
investments, but weaknesses in this approach limit its
effectiveness. We are making recommendations to the Secretary of
Transportation to strengthen FAA's investment management approach.
We are sending copies of this report to Senator Frank R.
Lautenberg, Senator Richard C. Shelby, Representative James A.
Barcia, Representative Stephen Horn, Representative Jim Turner,
Representative Constance A. Morella, Representative Martin Olav
Sabo, and Representative Frank R. Wolf in their capacities as
Chair or Ranking Minority Member of Senate and House
Subcommittees. We are also sending copies of this report to the
Honorable Rodney E. Slater, Secretary of Transportation; the
Honorable Jane F. Garvey, Administrator of the Federal Aviation
Administration; and the Honorable Jacob Lew, Director, Office of
Management and Budget. Copies will also be made available to
others on request.
B-279992 Please call me at (202) 512- 2834 if you have questions
about the report. Major contributors to this report are listed in
appendix II.
Gerald L. Dillingham Associate Director, Transportation
Issues
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 2
B-279992
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 3
Executive Summary Purpose The Federal Aviation Administration
(FAA) has undertaken an ambitious
and costly program to modernize its air traffic control system.
Under this program, FAA is acquiring new surveillance, data-
processing, navigation, and communications equipment in addition
to new facilities and support equipment. Totaling 126 active
projects, the modernization effort is estimated to cost $26.5
billion from fiscal year 1982 through fiscal 2004. 1 Of this
total, FAA estimates that it will need $12.9 billion for 59
information technology projects the software- intensive and
complex information and communications systems supporting the air
traffic control system.
Given the large expenditures required to carry out FAA's
modernization effort, the past problems, and the continuing
concerns about key projects funded under the program, the Chairmen
and Ranking Members of the Senate Committee on Commerce, Science,
and Transportation's Subcommittee on Aviation and the House
Committee on Transportation and Infrastructure's Subcommittee on
Aviation asked GAO to review FAA's investment management approach
as carried out through the Acquisition Management System (AMS),
which was implemented in April 1996. GAO evaluated the processes,
data, and decisions that FAA uses to select, control, and evaluate
its investments. This report addresses the extent to which FAA,
through AMS, (1) has established a structured approach for
selecting and controlling its investments; (2) incorporates all
investments, including those currently in operation, in the
agency's portfolio; and (3) selects, controls, and evaluates its
investments with complete and reliable information.
Results in Brief FAA's Acquisition Management System is a good
first step in establishing a structured investment management
approach for selecting and controlling
the agency's investments. The system contains a set of policies,
procedures, and reporting requirements to analyze mission needs;
assess the affordability of proposed projects; and establish life-
cycle costs, schedules, benefits, and performance baselines
(boundaries) to control the performance of the projects that are
selected. Additionally, under this system, a senior management
investment review group makes key decisions about which
investments best meet the agency's needs and are to be funded.
However, the system is not comprehensive in that it does not
incorporate all of FAA's projects into a complete strategic
investment portfolio. Key
1 The total cost of the modernization program which includes
completed, canceled, and restructured projects as well as the
active projects is estimated to be $41 billion from fiscal year
1982 through fiscal 2004. In this report, all dollars are
expressed as current- year dollars.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 4
Executive Summary
decision- making processes and requirements of the Acquisition
Management System are applied only to proposed projects and those
under development but not to projects already in operation. In
particular, agency officials have not yet developed a sound
estimate of the costs to operate projects and these costs are not
included in the agency's financial plan for modernization. Because
FAA does not apply the same scrutiny to all of its projects,
senior officials are unable to fully assess and make trade- offs
about the relative merits of spending funds to develop new
systems, to enhance current systems, or to continue operating and
maintaining existing systems. This report makes a recommendation
designed to strengthen FAA's investment management by directing
the agency to establish and control a complete portfolio of all
information technology investments, including those projects
already in operation.
FAA's Acquisition Management System currently does not provide
complete and reliable information for selecting, controlling, and
evaluating the agency's investments. First, the cost data used to
select projects are of questionable reliability because of
weaknesses in FAA's cost estimating practices and processes and
the lack of a cost accounting system. Second, the information used
to control projects is incomplete since FAA has not fully
implemented an effective process for controlling the baselines for
the costs, schedules, benefits, performance, and risks of its
investments. FAA has approved the baseline information for only
half of the required universe of projects, and the agency's
processes for tracking actual performance against estimates
frequently has provided incomplete information. Third, FAA lacks
information needed to evaluate its investments since the
Acquisition Management System does not have a post- implementation
evaluation process for assessing projects' outcomes and feeding
lessons learned back into the selection and control phases to help
improve its management of future projects. This report makes
several recommendations to improve FAA's selection, control, and
evaluation of its information technology investments.
Background Over the past 17 years, FAA's modernization projects
have experienced substantial cost overruns, lengthy delays, and
significant performance
shortfalls. Because of FAA's contention that some of its
modernization problems were caused by federal acquisition
regulations, the Congress enacted legislation in November 1995
that exempted the agency from most federal procurement laws and
regulations and directed FAA to develop a new acquisition
management system. In response, FAA implemented AMS on April 1,
1996. AMS provides high- level acquisition policy and guidance for
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 5
Executive Summary
selecting and controlling investments throughout all phases of the
acquisition life cycle.
Funding for FAA's modernization investments is primarily provided
through two of its budget accounts: (1) facilities and equipment
and (2) operations. The facilities and equipment account covers
the costs to develop, procure, and place the new equipment or
facility in operation. Once the project goes into full operation,
it is funded by the operations account, which covers the costs to
support and maintain the new equipment or facility.
Using the methodology described in Assessing Risks and Returns: A
Guide for Evaluating Federal Agencies' IT Investment Decision-
Making, 2 GAO evaluated how FAA selects, controls, and evaluates
its investments. This guide incorporates GAO's analysis of the
management practices of leading private and public sector
organizations as well as provisions of major federal legislation
(e. g., the Clinger- Cohen Act of 1996) and executive branch
guidance that address investment decision- making. As part of its
evaluation, GAO examined five of FAA's projects to determine how
AMS is implemented at the project level. (The five projects are
described in app. I.)
Principal Findings FAA's AMS Is Designed to Provide a Disciplined,
Structured Process for Selecting and Controlling Investments
Through AMS, FAA has designed and implemented processes that
provide many of the key elements leading organizations follow to
select and control investments the agency funds through its
facilities and equipment budget account. In the selection phase,
leading organizations take a structured approach to determining
priorities, screening and analyzing the relative merits of the
projects, and making decisions about which projects will be funded
during the year. Through AMS, FAA has established two processes
mission analysis and investment analysis that together constitute
a set of policies, procedures, and guidance that enhances the
agency's ability to screen projects submitted for funding; assess
and rank each project based on its relative costs, benefits,
risks, and contribution to FAA's mission; and utilize a senior,
corporate- level decision- making group to select projects for
funding. Once a project is selected, AMS requires FAA officials to
formally establish the life- cycle cost, schedule, benefits, and
performance baselines that are used to monitor the project's
status. FAA
2 GAO/ AIMD- 10.1.13 (Feb. 1997).
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 6
Executive Summary
has developed a number of mechanisms for monitoring projects'
estimated versus actual baseline performance and reporting any
variances from the established baselines.
Lack of Oversight of the Operations Portion of Projects Prevents
FAA From Managing Investments as a Complete Portfolio
FAA lacks oversight of the operations portion of its investments
under AMS. For example, FAA's process for scoring and ranking
projects prior to selection is applied to proposed projects and
those under development that will receive facilities and equipment
funding, not to existing systems that are funded from the
operations budget account. In contrast, leading organizations
include all types of information technology projects (i. e., new
and existing systems) in their selection process to create a
complete strategic investment portfolio. FAA also has not yet
developed a sound estimate of the operations cost baseline for
each of its projects and the agency's financial plan for the
modernization program reports only costs funded by the facilities
and equipment budget account, omitting the operations costs
associated with its investments. Although FAA has developed
operations cost projections for 26 of the 70 projects or segments
of projects identified as requiring baselines, officials
throughout the agency told us that these estimates are not
reliable. Finally, while FAA's budget provides detailed analyses
of actual and projected costs for each of the projects funded by
the facilities and equipment budget account, it provides very
little project- level detail in its justification for the
operations budget account. FAA has two initiatives that it
believes will improve the data on operations costs. First, it is
developing a cost accounting system, although operations data from
that system will not be available for at least a year, given the
schedule for implementing the system. Second, FAA has a team that
is addressing the agency's concerns about the quality of
operations data and developing operations cost baselines for
selected projects. This team is expected to report its findings in
May 1999.
Weaknesses in the Selection, Control, and Evaluation Phases Limit
FAA's Effectiveness in Managing Its Investments
AMS has weaknesses in all three investment management phases
selection, control, and evaluation that limit FAA's ability to
manage its investments effectively. First, the cost information
used to make selection decisions is of questionable reliability,
and there is little evidence that the data or underlying analyses
used in the selection process are validated to ensure accuracy and
completeness. While the agency has improvements under way, FAA's
cost estimating techniques do not yet satisfy recognized standards
that call for organizing and retaining projects' cost information
in a historical database and using cost models that are calibrated
and validated on the basis of actual experience. Instead, FAA's
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 7
Executive Summary
processes allow each project to approach cost estimating in
whatever manner its estimators choose. Moreover, the data used to
support FAA's selection decisions are not validated because FAA
does not require that all project information such as that
pertaining to costs, schedules, performance, benefits, or risks
used in making selection decisions be validated under AMS.
Furthermore, AMS guidance does not specify what types of
validation steps should be taken nor does it require documentation
of the results.
Second, FAA has not fully implemented its procedures for
controlling key projects' baselines. To control its projects at
the agencywide level, FAA relies on periodic reviews of each
project's acquisition program baseline, a document that
establishes a project's cost, schedule, benefits, and performance
boundaries and that is intended to be used to monitor a project's
status in achieving those boundaries. This baseline document is
incomplete, however, because its schedule baseline does not
include any milestones for project reviews during the operations
phase of the project and because it does not address the project's
risks. In addition, FAA has completed about half of the baselines
for its universe of projects or project segments that require
them, and agency- level processes for tracking actual baseline
performance against estimates frequently provided incomplete
information on projects' costs, schedules, benefits, and
performance. For example, for the five projects GAO reviewed, none
of the monthly baseline status reports analyzed the projects'
estimated operations costs, assessed estimated versus actual
benefits, or contained information on the performance requirements
outlined in the projects' baseline documentation. Moreover, FAA's
investment control group made up of senior managers, the Joint
Resources Council, is not actively involved in monitoring all
projects after the investment decisions are made.
Third, FAA does not have a defined, documented process for
conducting post- implementation reviews of projects for the
purpose of assessing project performance as well as improving the
selection and control of its investments. FAA performs some
elements of a post- implementation review in its life- cycle
management review process, including such tasks as the independent
operational test and evaluation of some projects prior to
deployment, operational performance monitoring, customer
satisfaction surveys, and periodic reviews throughout an
investment's life cycle. However, this process is not standardized
and is not required for all projects. As a result, there is no
evidence that changes, especially to the selection and control
phases, are being implemented based on lessons learned.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 8
Executive Summary
Finally, FAA's recently implemented agencywide management
information system for tracking information about projects under
AMS contains data related to FAA's processes for managing
baselines but excludes key selection data, such as mission need
statements, cost- benefit analyses, risk assessments, and other
required reports. Informed management decisions can only be made
if information from all phases of the investment management
process is included in the decision- making process and made
easily available through a management information system.
Recommendations GAO recommends that the Secretary of
Transportation direct the Administrator of FAA to implement a
comprehensive investment
management approach through the Acquisition Management System.
Specifically, the Administrator should take the following actions:
Establish a complete portfolio of investments including existing
systems funded by the operations budget account as well as
projects funded by the facilities and equipment account and
require the Joint Resources Council to periodically review the
baseline status and merits of each of these investments throughout
their entire life cycle. As part of this portfolio, cost baselines
for operating and maintaining all projects should be developed,
and this information should be included in the agency's financial
plan for its investments and in its annual budget request to the
Congress. Improve the selection process by (1) establishing
clearly defined
procedures for validating each project's cost, schedule, benefit,
performance, and risk information and (2) requiring documentation
of the results of the validation procedures applied to each
project. Strengthen control over investments by (1) revising the
acquisition
program baseline requirements to include project risks and to add
milestones for project reviews during the operations phase and (2)
ensuring that project officials fully track and document estimated
versus actual results on all the elements (i. e., cost, schedule,
benefit, performance, and risk) contained in the baseline
documentation. Initiate post- implementation evaluations for
projects within 3 to 12 months
of deployment or cancellation to compare the completed projects'
cost, schedule, performance, and mission improvement outcomes with
the original estimates. Incorporate key information from the
selection process (e. g., mission need
statements, cost- benefit analyses, and risk assessments) into
FAA's management information system for investments.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 9
Executive Summary
Agency Comments and Our Evaluation
We provided copies of a draft of this report to the Department of
Transportation and FAA for their review and comment. We met with
FAA officials, including the Associate Administrator for Research
and Acquisitions, who is also FAA's Acquisition Executive. These
officials generally agreed with the recommendations in this report
and made clarifying comments, which have been incorporated as
appropriate. FAA officials noted that the Acquisition Management
System represents a substantial revision to the way that FAA
contracts for large and complex systems and that it is important
to maintain perspective on how far the agency has come in a
relatively short time. These officials were concerned that the use
of GAO's guide, Assessing Risks and Returns: A Guide for
Evaluating Federal Agencies' IT Investment Decision- Making,
unduly diminished FAA's accomplishments by comparing the agency to
an ideal end- state that may not exist in any single organization.
They also asserted that the Acquisition Management System would
compare favorably with acquisition systems in other federal
government or private sector organizations.
As we stated in this report, FAA's Acquisition Management System
is a good first step in establishing a structured investment
management approach, but it has weaknesses that limit its
effectiveness. The GAO guide was developed to provide a structure
for evaluating and assessing how well a federal agency is managing
its information technology resources and to identify specific
areas where improvements can be made. The concepts and practices
contained in the guide are based on the practices followed by
leading private and public sector organizations as well as on
provisions of major federal legislation and executive branch
guidance that address investment decision- making. While
acknowledging the wide variance among organizations and the
complexity of the investment management process, the guide focuses
on the common elements that should be present in any
organization's investment management process. Our review evaluated
the extent to which FAA's Acquisition Management System contains
these elements, not how well FAA's system compares with other
federal acquisition systems. Therefore, we did not make changes to
the report based on these comments.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 10
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 11
Contents Executive Summary 4 Chapter 1 Introduction
14 The ATC System Modernization Program Is Complex, Costly, and
Historically Problematic 15
FAA Has Developed the Acquisition Management System to Manage Its
Modernization Investments
16 FAA's Acquisitions Are Funded From Two Major Budget
Accounts 18
Guidance Provides a Framework for Assessing Federal Agencies'
Information Technology Investment Decision- Making
19 Objectives, Scope, and Methodology 20
Chapter 2 FAA's AMS Is Designed to Provide a Disciplined,
Structured Process for Selecting and Controlling Investments
22 FAA Has Established a Defined Process for Selecting Investments
Funded Through the Facilities and Equipment Account 22
AMS Requires That Projects' Baselines Be Established and
Controlled
24 Conclusions 26
Chapter 3 Lack of Oversight of the Operations Portion of Projects
Prevents FAA From Managing Investments as a Complete Portfolio
27 FAA's Selection Process Does Not Address Operations- Funded
Investment Projects 27
FAA Does Not Have a Complete and Sound Operations Cost Baseline
for Its Investments
28 FAA's Operations Budget Justifies Only a Small Portion of Its
Spending for Investments 29
Conclusions 30 Recommendation 30 Agency Comments 30
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 12
Contents
Chapter 4 Weaknesses in the Selection, Control, and Evaluation
Phases Limit FAA's Effectiveness in Managing Its Investments
31 Weaknesses in Some Supporting Data Limit the Effectiveness of
the Selection Process 31
FAA Has Not Fully Implemented Its Process for Establishing and
Tracking Key Project Baselines
33 FAA's AMS Lacks a Post- Implementation Review Process for
Evaluating Investments 41
FAA's Efforts to Implement an Agencywide Management Information
System Do Not Include Key Selection Data
42 Conclusions 44 Recommendations 44 Agency Comments 45
Appendixes Appendix I: Background and Status of Five Projects 46
Appendix II: Major Contributors to This Report 53
Tables Table 2.1: Information on the Baseline Elements Required in
the Acquisition Program Baseline and the Project Parameter Sheet
25 Table 4.1: Status of FAA's Efforts to Develop AMS Baseline
Documentation on 60 Ongoing Projects and Segments for Which
Investment Decisions Have Been Made
37 Table 4.2: Analysis of the Missing Baseline Information
Reported
in Acquisition Reviews of Five Projects 39
Figure Figure 1.1: FAA's Life- Cycle Acquisition Management
Process 18
Abbreviations
AMS Acquisition Management System ATC air traffic control FAA
Federal Aviation Administration GAO General Accounting Office I-
BEAM Integrated Baseline Establishment and Management NAS National
Airspace System SPIRE Simplified Program Information Reporting and
Evaluation
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 13
Chapter 1 Introduction
The Federal Aviation Administration's (FAA) mission is to promote
the safe, orderly, and expeditious flow of air traffic in the
United States through what is commonly referred to as the National
Airspace System (NAS). FAA's ability to fulfill its mission
depends on the adequacy and reliability of the nation's air
traffic control (ATC) system the principal component of the NAS
which comprises a vast network of radars; automated data-
processing, navigation, and communications equipment; and ATC
facilities. It is through the ATC system that FAA provides
services such as controlling takeoffs and landings and managing
the flow of traffic between airports. 3 Sustained growth in air
traffic and FAA's aging equipment have strained the current ATC
system. This growth in traffic is predicted to continue as the
number of passengers traveling on U. S. airlines is expected to
grow from about 674 million in 1998 to 1.055 billion by 2010, an
increase of about 57 percent.
To relieve the problems of aging equipment and to accommodate the
predicted growth in air traffic, FAA initiated a multibillion-
dollar modernization effort in December 1981. Our work over the
years has chronicled many of FAA's failures in meeting projects'
cost, schedule, and performance goals. Because of the size,
complexity, cost, and problem- plagued past of FAA's modernization
program, we have designated it a high- risk information technology
investment since 1995. 4
3 FAA uses three types of facilities to manage and control
traffic. Airport towers direct aircraft on the ground, before
landing, and after takeoff when they are about 5 nautical miles
from the airport and up to about 3,000 feet above the airport.
Terminal radar approach control facilities sequence and separate
aircraft as they approach and leave busy airports, beginning about
5 nautical miles and ending about 50 nautical miles from the
airport and generally up to 10,000 feet above the ground. Air
route traffic control centers, called en route centers, control
planes in transit and during approaches to some airports. Most of
the en route centers' controlled airspace extends above 18,000
feet for commercial aircraft. En route centers also handle
aircraft at lower altitudes when dealing directly with a tower or
when agreed upon with a terminal facility. FAA provides additional
services, such as weather and pilot briefings, through a network
of flight service stations.
4 FAA's modernization program is one of four high- risk system development and modernization efforts in the federal government. See High- Risk Series: An Overview (
GAO/HR-95-1
, Feb. 1995), High- Risk Series: Information Management and Technology (GAO/ HR- 97- 9, Feb. 1997), and High- Risk Series: An Update (GAO/ HR- 99- 1, Jan. 1999).
4 FAA's modernization program is one of four high- risk system
development and modernization efforts in the federal government.
See High- Risk Series: An Overview ( GAO/HR-95-1 , Feb. 1995),
High- Risk Series: Information Management and Technology (GAO/HR-
97-9, Feb. 1997), and High- Risk Series: An Update (GAO/HR-99-1,
Jan. 1999).
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 14
Chapter 1 Introduction
The ATC System Modernization Program Is Complex, Costly, and
Historically Problematic
Under its ambitious modernization program, FAA is acquiring new
surveillance, data- processing, navigation, and communications
equipment in addition to new facilities and support equipment.
Totaling 126 active projects, the modernization is estimated to
cost $26.5 billion from fiscal year 1982 through fiscal 2004. 5 Of
this total, FAA estimates that it will need $12.9 billion from
fiscal year 1982 through 2004 for 59 information technology
projects the software- intensive and complex information and
communications systems supporting the ATC system. These projects
range from those designed to replace equipment used by controllers
to communicate with aircraft and with each other to radars that
provide controllers with surveillance information for separating
aircraft. An example of an information technology project is the
Display System Replacement project that will modernize equipment
in FAA's en route facilities by replacing 20- to 30- year- old
display channels, controllers' workstations, and network
infrastructure.
Over the past 17 years, FAA's modernization projects have
experienced substantial cost overruns, lengthy delays, and
significant performance shortfalls. To illustrate, the longtime
centerpiece of the modernization program the Advanced Automation
System was restructured in 1994 after estimated costs to develop
the system tripled from $2.5 billion to $7.6 billion and delays in
putting significantly less- than- promised system capabilities
into operation were expected to run 8 years or more over the
original estimates. 6 These problems have persisted. For example,
two key projects in the modernization effort the Wide Area
Augmentation System and the Standard Terminal Automation
Replacement System have encountered significant cost increases,
delays, and changes in requirements. 7 In the case of the Wide
Area Augmentation System, between September 1997 and January 1998,
total estimated costs increased by $600 million, or 25 percent,
from $2.4 billion to $3 billion. The increased costs were
attributable to FAA's including previously overlooked costs for
periodically updating the project's equipment and to higher- than-
expected operations and maintenance costs. In the case of the
Standard Terminal Automation Replacement System, although FAA has
not officially changed the project's baseline approved in February
1996, the
5 The total cost of the modernization program which includes
completed, canceled, and restructured projects as well as active
projects is estimated to be $41 billion from fiscal year 1982
through fiscal 2004. In this report, all dollars are expressed as
current year dollars.
6 See Advanced Automation System: Implications of Problems and
Recent Changes (GAO/T-RCED-94-188, Apr. 13, 1994). 7 See Air
Traffic Control: Observations on FAA's Modernization Program (GAO/
T- RCED/ AIMD- 98- 93, Feb. 26, 1998).
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 15
Chapter 1 Introduction
baseline is in jeopardy of being breached because of unions'
concerns surrounding human- factor and design issues, 8 the
refinement of requirements, and the interjection of a new project
phase. 9 FAA estimates that these issues have the potential to
increase the project's costs from $294 million to $410 million
above the approved baseline. FAA also estimates that the project's
initial completion could be delayed by almost 2- 1/ 2 years.
FAA Has Developed the Acquisition Management System to Manage Its
Modernization Investments
Because of FAA's contention that some of its modernization
problems were caused by federal acquisition regulations, the
Congress enacted legislation in November 1995 that exempted the
agency from most federal procurement laws and regulations and
directed FAA to develop and implement a new acquisition management
system that would address the unique needs of the agency. 10 On
April 1, 1996, in response to the Congress's action, FAA
implemented a new acquisition management system. The system is
intended to reduce the time and cost to field new products and
services by introducing (1) a new investment management system
that spans the entire life cycle of an acquisition, (2) a new
procurement system that provides flexibility in selecting and
managing contractors, and (3) organizational and cultural reform
that supports the new investment and procurement systems.
The Acquisition Management System (AMS) provides high- level
acquisition policy and guidance for selecting and controlling
FAA's investments throughout all phases of the acquisition life
cycle, which is organized into a series of phases and decision
points, including (1) mission analysis, (2) investment analysis,
(3) solution implementation, (4) in- service management, and (5)
service life extension. AMS provides guidance on the documents and
decisions that result from each of these phases. For example,
through the mission analysis process, FAA identifies critical
needs that the agency must meet for improving the safety,
security, capacity, efficiency, and effectiveness of the NAS.
Approval of a mission need statement by the Joint Resources
Council FAA's corporate decision- making body signifies that the
agency agrees that the need is critical enough to proceed to the
next phase, investment analysis. During the investment analysis
phase, teams of acquisition and program specialists (1) identify
and analyze alternatives, (2) develop baselines and
8 Concerns were raised by two unions, the National Air Traffic
Controllers Association and the Professional Airways Systems
Specialists. 9 See Air Traffic Control: Status of FAA's
Modernization Program (GAO/RCED-99-25, Dec. 3, 1998). 10
Department of Transportation and Related Agencies Appropriations
Act of 1996 (P. L. 104- 50).
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 16
Chapter 1 Introduction
assesses affordability, (3) prepare an investment analysis report,
and (4) recommend a preferred solution to the mission need. FAA
then scores and ranks each proposed project based on a number of
factors, including how well, relative to other projects, it meets
mission needs and has a favorable cost- benefit ratio. Once a
project is selected, life- cycle cost, schedule, benefits, and
performance baselines are established in a formal document called
the acquisition program baseline. The acquisition program
baseline, which must be approved by the Joint Resources Council,
is used to monitor a project's status in achieving those baselines
throughout the remaining phases of the acquisition management life
cycle.
During the solution implementation phase, a multidisciplinary team
develops and carries out an acquisition strategy for implementing
the project. Once the project has been implemented and is in
operation (the in- service management phase), the team monitors
and assesses its performance, costs, and support trends; proposes
fixes for any defects or other problems; incorporates product
improvements; seeks new technology to enhance the capability or
reduce costs; and identifies and prepares for decisions to correct
capability shortfalls at the end of the project's service- life.
Finally, during the service- life extension phase, a determination
is made about whether the current capability satisfies the demand
for services or whether another solution offers the potential for
improving safety or effectiveness or for significantly lowering
costs. The team initiates a process whereby the mission need would
be revalidated and the investment analysis process begun again,
which could lead to a new investment decision. See figure 1.1 for
a graphic depiction of FAA's life- cycle acquisition management
process.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 17
Chapter 1 Introduction
Figure 1.1: FAA's Life- Cycle Acquisition Management Process
INVESTMENTDECISION IN- SERVICEDECISION END
PROGRAM MISSION NEEDDECISION
MISSION ANALYSIS
INVESTMENT ANALYSIS
SOLUTION IMPLEMENTATION SERVICE
LIFE EXTENSION
FAA LIFE CYCLE ACQUISITION MANAGEMENT SYSTEM
IN- SERVICE MANAGEMENT
Source: FAA.
FAA's Acquisitions Are Funded From Two Major Budget Accounts
Funding for FAA's investments is primarily provided through two of
its budget accounts: (1) facilities and equipment and (2)
operations. The facilities and equipment account covers the costs
to develop, procure, and place the new equipment or facility in
operation. Once the project goes into full operation, it is funded
by the operations account, which covers the costs to support and
maintain the new equipment or facility. Costs for planned product
improvements and upgrades to the technology can be funded from
either the facilities and equipment or the operations accounts.
Some investment funding is also provided through the research,
engineering, and development account. This account is relatively
small compared with the facilities and equipment and operations
accounts. For example, in fiscal year 1999, $150 million was
appropriated for research, engineering, and development;
approximately $2.1 billion for facilities and equipment; and
approximately $5.6 billion for operations.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 18
Chapter 1 Introduction
Guidance Provides a Framework for Assessing Federal Agencies'
Information Technology Investment Decision- Making
Several recent management reforms including the revision of the
Paperwork Reduction Act and the passage of the Clinger- Cohen Act
of 1996, the Government Performance and Results Act of 1993, and
the Chief Financial Officers Act of 1990 have introduced
requirements emphasizing the need for federal agencies to improve
their management processes for selecting and managing information
technology resources. GAO and the Office of Management and Budget
have developed guidance to assist federal agencies in evaluating
information technology investments. One such guide, Assessing
Risks and Returns: A Guide for Evaluating Federal Agencies' IT
Investment Decision- Making, incorporates our analysis of the
management practices of leading private and public sector
organizations as well as of the provisions of major federal
legislation (e. g., the Clinger- Cohen Act) and executive branch
guidance that address investment decision- making. 11 The guide
outlines three phases of a successful investment management
approach selection, control, and evaluation. To help ensure that
real, positive change is produced as agencies seek to improve
their decision- making about their information technology
investments, agencies need to (1) institutionalize management
processes; (2) regularly validate the cost, benefit, and risk data
used to support information technology decisions; and (3) focus on
measuring and evaluating results. The guide provides a framework
for evaluating and assessing how well a federal agency is
achieving these goals and identifies specific areas where
improvements can be made.
Many of the concepts in our Assessing Risks and Returns have been
incorporated into the Office of Management and Budget's Capital
Programming Guide, which provides guidance on the planning,
budgeting, acquisition, and management of different kinds of
capital assets, including information technology. 12 Our guide has
also been endorsed by the federal Chief Information Officers
Council. A third guide we prepared, Executive Guide: Leading
Practices in Capital Decision- Making, summarizes 12 fundamental
practices that have been successfully implemented by organizations
recognized for their outstanding capital decision- making. 13
Since information technology investments are a form of capital
asset, this guide emphasizes many of the same concepts as the
aforementioned guides, such as evaluating alternative approaches
to achieving results;
11 GAO/ AIMD- 10.1.13 (Feb. 1997). 12 Capital Programming Guide,
Version 1.0, Supplement to Office of Management and Budget
Circular A- 11, Executive Office of the President, Office of
Management and Budget (July 1997). 13 Executive Guide: Leading
Practices in Capital Decision- Making (GAO/AIMD-99-32, Dec. 1998).
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 19
Chapter 1 Introduction
assessing investments as a portfolio; tracking projects' costs,
schedules, and performance; and conducting post- implementation
reviews.
Objectives, Scope, and Methodology
Given the large expenditures required to carry out FAA's
modernization program, the past problems, and the continuing
concerns about key projects funded under the program, the Chairmen
and Ranking Minority Members of the Senate Commerce, Science, and
Transportation Committee's Subcommittee on Aviation and the House
Transportation and Infrastructure Committee's Subcommittee on
Aviation asked us to evaluate the extent to which AMS provides a
comprehensive approach for managing FAA's modernization
investments. This report addresses the extent to which FAA,
through AMS, (1) has established a structured approach for
selecting and controlling its investments; (2) incorporates all
investments, including those in operation, in the agency's
portfolio; and (3) selects, controls, and evaluates its
investments with complete and reliable information.
To address our objectives, we reviewed FAA's overall approach for
managing investments, as carried out through AMS, including the
policies, procedures, and guidance for managing the life- cycle
acquisition process. AMS is designed to include all of FAA's
acquisition projects, including those related to facilities,
mission support, and information technology. Our review focused
primarily on information technology projects, and we worked with
FAA to identify the universe of such projects. Then, in concert
with FAA, we selected five projects for review to obtain a more
detailed understanding of how the investment management process is
implemented at the project level. The five projects we reviewed
were (1) the NAS Infrastructure Management System, (2) the Oceanic
Automation Program, (3) the Operational and Supportability
Implementation System, (4) the Standard Terminal Automation
Replacement System, and (5) the Voice Switching and Control
System. We selected these projects because they were in various
stages of implementation when AMS was established in April 1996,
were key to replacing the aging NAS infrastructure or improving
its capacity and effectiveness, and represented significant
expenditures. Total estimated facilities and equipment funding for
each of these projects exceeds $100 million. Two of the five
projects the NAS Infrastructure Management System and the
Operational and Supportability Implementation System are subject
to all AMS requirements because they were selected after AMS went
into effect in April 1996. The other three projects, which were
ongoing when AMS was established, are subject to AMS
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 20
Chapter 1 Introduction
requirements for the remaining stages in their development and
implementation.
As part of our effort, we applied the methodology prescribed in
Assessing Risks and Returns. We used the questions in this guide
to determine the extent to which FAA has decision- making and
management processes and data in place to select information
technology projects and systems, control these projects throughout
their life cycles, and evaluate results and revise the processes
based on lessons learned. We provided the questions in the guide
to FAA officials and assessed their responses, along with
supporting documentation, as a basis for determining whether FAA's
approach provided the necessary elements for managing its
investments. We also incorporated the guidance from the Executive
Guide: Leading Practices in Capital Decision- Making in
determining whether FAA managed its investments as a portfolio;
tracked projects' costs, schedules, and performance; and conducted
post- implementation reviews.
To gain an overall perspective on FAA's investment management
process, we interviewed FAA officials responsible for implementing
and managing AMS. We also interviewed FAA officials responsible
for preparing the facilities and equipment and operations budgets
and reviewed the agency's fiscal year 1999 budget justification
documentation to understand how data on FAA's investments are used
to support FAA's budget request to the Congress. Finally, we
interviewed officials responsible for managing the five projects
we reviewed to obtain their views on how AMS is applied at the
project level as well as potential areas for improvement. We
performed our work from April 1998 through April 1999 in
accordance with generally accepted government auditing standards.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 21
Chapter 2 FAA's AMS Is Designed to Provide a Disciplined,
Structured Process for Selecting and Controlling Investments
Through AMS, FAA has designed and implemented processes that
provide many of the key elements followed by leading organizations
for selecting and controlling investments. AMS' mission analysis
and investment analysis processes are meant to provide FAA's
senior management with a basis for screening proposed projects;
evaluating their relative costs, benefits, and risks; and
selecting projects for funding based on their relative merits.
Additionally, AMS policy requires each acquisition project to have
an approved baseline that establishes the project's life- cycle
cost, schedule, benefits, and performance boundaries and that is
intended to be used to monitor the project's status in achieving
those baselines.
FAA Has Established a Defined Process for Selecting Investments
Funded Through the Facilities and Equipment Account
During the selection phase, leading organizations take a
structured approach to determining priorities, screening and
analyzing the relative merits of projects, and making decisions
about which projects will be funded during the year. Such an
approach builds on an organization's assessment of where it should
invest its resources for the greatest benefit over the long term.
The Clinger- Cohen Act requires federal agencies to apply this
sort of structured approach in deciding whether to undertake
particular investments in information technology systems.
A starting point for the selection phase is the screening process
in which projects being submitted for funding are compared against
a uniform set of screening criteria and thresholds to determine
whether the projects meet minimal requirements and to identify at
what organizational level the projects should be reviewed. Next,
the costs, benefits, risks, and mission focus of all the projects
are assessed, and the projects are compared against each other and
ranked or prioritized. In conducting their selection processes,
leading organizations require all projects to have complete and
accurate project proposals and justification information. Finally,
a decision- making body of senior managers makes decisions about
which projects to select for funding on the basis of mission needs
and organizational priorities. The selection phase helps ensure
that the organization selects those projects that will best
support mission needs and identifies and analyzes each project's
risks and proposed benefits before a significant amount of funds
is spent.
Through AMS, FAA has established two processes mission analysis
and investment analysis that together constitute a set of
policies, procedures, and guidance that are designed to enhance
the agency's ability to select
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 22
Chapter 2 FAA's AMS Is Designed to Provide a Disciplined,
Structured Process for Selecting and Controlling Investments
investments. 14 These two processes define what should be done and
who should do it, what reports are required, who reviews and
approves reports and processes, who makes corporate- level
decisions, and the roles and responsibilities of those involved.
AMS policy and guidance which is on an easily accessible Internet
Website contain the procedures, process flowcharts, document
templates, checklists, and other acquisition- related information
needed for FAA's project officials and senior management to
understand and implement the selection processes.
Although AMS does not include an explicit screening step,
screening activities are part of the agency's mission analysis
process. This process culminates in a mission need statement
reflecting the Joint Resources Council's decision that a high-
priority, critical need exists and that the agency should go
forward with a detailed investment analysis of proposed solutions
to meet that need. During mission analysis, FAA's operating
divisions identify and quantify projected demand for and supply of
services, capability shortfalls, and technological opportunities
to meet those shortfalls, and summarize the major decision factors
that the Joint Resources Council should evaluate in considering
the need. The Joint Resources Council approves mission need
statements.
Once a mission need statement is approved, AMS' investment
analysis process provides a set of detailed steps for evaluating
the costs, benefits, and risks of alternative solutions and for
selecting the best solution to meet the need. Under AMS,
multidisciplinary investment analysis teams comprising officials
from the operating divisions and other acquisition and engineering
specialists assess each project proposed for funding to define the
technical requirements; estimate the life- cycle costs, benefits,
schedule, and risks; and determine the project's affordability
relative to other projects. As part of the investment analysis
process, FAA scores and ranks each proposed project on the basis
of defined criteria for how well relative to other projects it
meets the agency's mission objectives, whether it is of high
priority to the organization sponsoring the project, whether it is
consistent with the NAS architecture 15 or provides critical
administrative capacity for the agency, and whether it has a
favorable cost- benefit ratio. AMS requires that the mission need
be revalidated during the investment analysis and that the
underlying analyses be documented through cost- benefit studies,
risk assessments, and other documents.
14 As we discuss in chapter 3, key aspects of AMS' selection
process are limited to projects funded by the facilities and
equipment account and exclude investments funded by the operations
account. 15 The architecture is FAA's blueprint for defining the
long- range needs of the NAS.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 23
Chapter 2 FAA's AMS Is Designed to Provide a Disciplined,
Structured Process for Selecting and Controlling Investments
After considering the results of the investment analysis process,
the Joint Resources Council decides whether to proceed with a
proposed project. This step is called the investment decision, and
if the project is selected, the Council commits FAA to fully
funding it. The project is then incorporated into the agency's
financial plan for projects funded by the facilities and equipment
budget account.
AMS Requires That Projects' Baselines Be Established and
Controlled
Under AMS, FAA's primary mechanism for controlling a project is
the acquisition program baseline, which establishes a project's
life- cycle cost, schedule, benefits, and performance baselines
and which is intended to be used to monitor a project's status in
achieving those baselines. The acquisition program baseline is
supposed to be established when the investment decision is made
for a project. The baseline, which is the agreement between the
organizations within FAA that are acquiring and will use the
project, sets the cost and schedule boundaries within which the
project is authorized to proceed, defines the performance and
benefits the project must achieve, and establishes the performance
measurements for assessing the project's success as it advances
through its life cycle.
FAA's ultimate goal is to establish an acquisition program
baseline document for every acquisition project. When AMS was
established in April 1996, however, the agency decided to
establish two standards for baseline documentation to facilitate
the preparation of baselines during FAA's transition to AMS. One
standard applies to those projects initiated after AMS and
another, less detailed standard applies to those projects ongoing
at the time that AMS was established. The post- AMS projects must
have an acquisition program baseline that provides a full set of
detailed information on each baseline element as well as a
document called a parameter sheet that summarizes a subset of
baseline information that is designated for the Joint Resources
Council's control and that is considered critical to assessing the
project's ability to satisfy mission need, achieve needed
operational capability, achieve benefits, and meet the schedule
requirements of interdependent programs. Projects begun before AMS
was established are required only to have the parameter sheet.
Table 2.1 summarizes the full set of baseline information
contained in the acquisition program baseline and the subset of
information contained in the parameter sheet.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 24
Chapter 2 FAA's AMS Is Designed to Provide a Disciplined,
Structured Process for Selecting and Controlling Investments
Table 2.1: Information on the Baseline Elements Required in the
Acquisition Program Baseline and the Project Parameter Sheet
Baseline elements Full set of baseline information contained in
the acquisition
program baseline Subset of baseline information contained in the
parameter sheet
Cost Life- cycle costs are broken down by (1) all sources of
appropriations funding, (2) all fiscal years over the project's
entire life cycle, and (3) 12 detailed cost elements (program
management, testing and evaluation, training, data management,
physical integration, systems and equipment, implementation,
product support, operations and maintenance, in- service support,
in- service monitoring and assessment, and disposal of replaced
assets).
Total life- cycle costs are broken down by source of
appropriations funding.
Schedule Schedule includes a list of all events related to
satisfying mission need, providing intended operational
capability, and accruing benefits as well as events crucial to
other related NAS programs or systems.
Schedule includes a subset of events most crucial to satisfying
mission need, providing operational services, and accruing
benefits for example, contract award, in- service decision (the
point at which the new system is certified ready for operational
use) and the dates that the first and last sites are commissioned
into operational use.
Benefits Total life- cycle benefits are listed for both the
government and users, with specific measures for evaluating the
annual economic benefits and whether these benefits have been
achieved.
Benefits include total life- cycle economic benefits.
Performance Performance includes all requirements for project
milestones; technical performance; acquisition of technical
systems, equipment, facilities, and services; coordination with
outside organizations; and operations support and management
requirements.
Performance includes the total number of systems, sites, or
services provided. Additionally, it includes other performance
requirements deemed most critical to achieving operational
effectiveness, accruing benefits, or meeting other dependent NAS
needs and those considered to pose the greatest risk for cost or
schedule growth.
Source: FAA.
At the agencywide level, FAA has a two- tiered process for
monitoring projects' estimated versus actual baselines. Under one
process, known as the Integrated Baseline Establishment and
Management (I- BEAM) process, the project officials and officials
responsible for overseeing projects' baselines monitor the
projects and prepare reports. Project officials prepare a monthly
status report that is supposed to analyze estimated versus actual
results for the subset of baseline information that is monitored
by the Joint Resources Council and summarized in the parameter
sheets prepared for every project. Project officials must also
inform the Joint Resources Council of any baseline violation or
breach through a document called a baseline management notice and
receive the Council's approval for any change in the baselines.
Additionally, FAA's acquisition oversight officials prepare a
monthly report that analyzes the
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 25
Chapter 2 FAA's AMS Is Designed to Provide a Disciplined,
Structured Process for Selecting and Controlling Investments
status reports, baseline management notices, and other documents
to identify baseline variances and report this information to the
Joint Resources Council. Under a second process, the Acquisition
Executive who heads the Council and other senior managers conduct
acquisition reviews that are designed to periodically examine the
status of the information in each project's acquisition program
baseline, along with assessing a project's progress, risks, and
other issues.
In addition to these two primary control processes, FAA also has a
variety of other reporting systems for monitoring projects'
status, including executive- level metrics that analyze project
performance, project- level reviews, and numerous contractor
reports on costs, schedule, and technical performance. However,
these various reporting systems do not analyze a project's status
in meeting all of the detailed baseline requirements contained in
the acquisition program baselines and parameter sheets.
Conclusions FAA has taken a positive step in establishing an
investment management approach through AMS. Under AMS, FAA has
developed structured processes
for selecting and controlling its investments. AMS provides a
defined set of policies, procedures, and reporting requirements
that are designed to facilitate FAA's efforts to analyze mission
needs, identify alternative solutions to meet those needs, assess
the solutions' affordability, and establish and control a
project's performance once a solution is selected.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 26
Chapter 3 Lack of Oversight of the Operations Portion of Projects
Prevents FAA From Managing Investments as a Complete Portfolio
FAA's oversight of its investments is confined to new projects and
those under development, limiting its ability to fully assess and
make trade- offs between new and existing systems and preventing
the agency from managing all of its investments as a complete
portfolio. FAA's oversight includes the agency's processes for
scoring and ranking investments prior to selection as well as its
processes for establishing and monitoring financial baselines
after a project has been selected. But this oversight is limited
to projects that receive funding from the facilities and equipment
budget account that is, new projects and those under development
and excludes projects funded by the operations account. Moreover,
the link between FAA's investment management process and its
budget process is limited to the facilities and equipment budget
account, which has detailed information on investments that does
not exist for the operations budget account. FAA has two ongoing
initiatives that may improve its information on operations costs.
Leading organizations determine priorities and make decisions
about which projects will be funded based on analyses of the
relative costs, benefits, and risks of all the projects in their
investment portfolios, including projects that are proposed, under
development, and in operation. Such a portfolio approach allows
the organizations to evaluate the relative merits of spending
funds to develop new systems, enhance current systems, or continue
operating and maintaining existing systems.
FAA's Selection Process Does Not Address Operations- Funded
Investment Projects
FAA's process for scoring and ranking projects prior to selection
is applied to proposed projects and those under development that
will receive facilities and equipment funding, not to existing
systems that are funded from the operations budget account. In
contrast, leading organizations include all types of information
technology projects in their selection process to create a
complete strategic investment portfolio. By analyzing the entire
portfolio, the senior managers of any organization can examine the
costs of maintaining existing systems versus investing in new
ones; comparatively rank projects based on expected costs,
benefits, and risks; and reach decisions based on projects'
overall contribution to the most pressing organizational needs.
FAA has not included operations- funded projects in its scoring
and ranking process because, as discussed below, the agency lacks
reliable data on actual operations costs for existing systems and
projections of operations costs for new projects.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 27
Chapter 3 Lack of Oversight of the Operations Portion of Projects
Prevents FAA From Managing Investments as a Complete Portfolio
FAA Does Not Have a Complete and Sound Operations Cost Baseline
for Its Investments
While FAA's policy requires baseline cost estimates for the full
life cycles of all projects under AMS, in practice, FAA has not
yet developed a sound estimate of the operations costs for each of
its projects. As of February 1999, the agency had developed
operations cost projections for only 26 of the 70 projects
identified by FAA as requiring an acquisition program baseline or
parameter sheet under AMS. For existing systems already in the
operations phase when AMS was implemented in April 1996, FAA also
lacked information on life- cycle cost projections for operating
the systems in the future or actual operations costs incurred for
each of these systems. Moreover, FAA officials throughout the
organization indicated that the estimates of operations costs that
have been developed for projects are not reliable. Given the lack
of data on operations costs, FAA's financial plan for its
modernization effort reports only costs funded by the facilities
and equipment budget account, omitting the operations costs
associated with its investments.
FAA has long recognized the need to project the operations costs
of its projects over a multiyear period. In March 1993, FAA issued
guidance under its predecessor acquisition system that required an
annual operations plan to support long- range resource allocation
planning. This plan which was never prepared was supposed to
summarize the operations and support funding requirements of
modernization projects and other acquisitions funded by the
operations appropriation. The operations plan was supposed to
contain a financial baseline that identified detailed operations
requirements over a 5- year period and general requirements for an
additional 10 years.
FAA officials told us that the agency lacks the information needed
to reliably estimate operations costs over a project's life cycle
or to track actual operations costs against estimates because it
does not have a cost accounting system. In January 1997, we
reported that FAA lacked reliable cost estimating processes and
cost accounting practices needed to effectively manage its
investments in information technology. 16 We concluded that, as a
result, the Congress does not have reliable cost information to
use in making decisions about FAA's billion- dollar modernization
investments.
FAA has two initiatives that it believes will improve data on
operations costs. In August 1998, FAA formed an operations
baseline team to address agencywide concerns about the quality of
its estimates of the operations
16 See Air Traffic Control: Improved Cost Information Needed to
Make Billion- Dollar Modernization Investment Decisions (GAO/AIMD-
97-20, Jan. 22, 1997).
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 28
Chapter 3 Lack of Oversight of the Operations Portion of Projects
Prevents FAA From Managing Investments as a Complete Portfolio
costs for modernization projects and the lack of integration
between the facilities and equipment and operations budget
accounts. This team is evaluating FAA's current processes for
estimating and reporting on operations costs, assessing the
validity of operations cost data on a sample of projects, and
exploring ways to improve the estimating process. The team, which
expects to report its findings in May 1999, plans to develop a 10-
year operations cost baseline for selected projects and to
recommend revisions to the budget formulation process that will
allow FAA to budget for the operations costs of both new and
existing systems and to better address the interrelationships
between the facilities and equipment and operations budget
accounts.
FAA is also developing a cost accounting system that it believes
will provide more reliable information on actual operations costs;
however, FAA has missed its initial milestones for completing the
new system's design and generating improved operations cost data.
According to officials responsible for the new cost accounting
system, the agency had planned to begin accumulating data for
domestic and oceanic air traffic services by October 1998. FAA
officials indicated that they underestimated the complexity of
developing the system and that they now expect to accumulate data
for air traffic services by April 2000 and to fully implement the
system by April 2001.
FAA's Operations Budget Justifies Only a Small Portion of Its
Spending for Investments
While FAA's budget provides detailed analyses of the actual and
projected costs for each of the projects funded by the facilities
and equipment budget account, it provides very little project-
level detail in its justification for the operations budget
account. In its fiscal year 1999 President's budget request, for
example, FAA justified only 5 percent, or $295 million, of its
total $5.6 billion operations budget account. This $295 million
request which represented the incremental increase over the prior
year's appropriation contained information on all of the
activities to be funded in a given year, including pay increases
for the current staff, costs of hiring new staff, training,
accident investigations, and other activities.
Among the details provided for the fiscal year 1999 operations
budget account, FAA provided project- or system- level
justifications for about 1 percent, or $62 million, of its total
$5.6 billion operations request. This information, known as the
NAS Handoff portion of the operations account, contains 1- year
estimates for new equipment, systems, and facilities that were
initially acquired with facilities and equipment funding but will
now be funded under the operations budget account. The NAS
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 29
Chapter 3 Lack of Oversight of the Operations Portion of Projects
Prevents FAA From Managing Investments as a Complete Portfolio
Handoff contains details on a variety of projected operations
costs, including controller overtime, logistics, systems
maintenance, leased communications, and flight inspections and
procedures.
One factor contributing to the different levels of detail for the
two budget accounts is the lack of reliable data on operations
costs for individual projects or systems. Also, FAA officials told
us that they only justify incremental increases over the prior
year's operations budget account because that is traditionally all
that the Congress requires of FAA in preparing its budget
justifications.
Conclusions AMS' lack of oversight of the operations portion of
FAA's investments impedes the agency's ability to rigorously
assess and manage all of its
modernization projects as a complete strategic, investment
portfolio and to make sound decisions about continuing, modifying,
or canceling projects. Excluding operations projects from its
selection process prevents FAA from considering the relative
merits of existing systems when deciding which projects to fund
each year.
Recommendation We recommend that the Secretary of Transportation
direct the Administrator of FAA to establish a complete portfolio
of
investments including existing systems funded by the operations
budget account as well as projects funded by the facilities and
equipment account and to require the Joint Resources Council to
periodically review the baseline status and merits of each of
these investments throughout their entire life cycles. As part of
this portfolio, cost baselines for operating and maintaining all
projects should be developed, and this information should be
included in the agency's financial plan for its investments and in
its annual budget request to the Congress.
Agency Comments FAA agreed with our recommendation.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 30
Chapter 4 Weaknesses in the Selection, Control, and Evaluation
Phases Limit FAA's Effectiveness in Managing Its Investments
AMS has weaknesses in all three investment management phases
selection, control, and evaluation that limit FAA's ability to
effectively manage its modernization investments. First, the
information used to select projects is not validated to ensure
quality control, and critical cost information used to support
selection decisions is of questionable reliability. Second, FAA
has not fully implemented an effective process for controlling the
baselines for the cost, schedule, benefits, performance, and risks
of its investments. Third, FAA lacks a post- implementation
evaluation process for assessing projects' outcomes and feeding
lessons learned back into the selection and control phases for
future projects. Finally, FAA has not fully implemented a
standardized management information system for capturing and
maintaining consistent, reliable, and easily accessible data on
investments.
Weaknesses in Some Supporting Data Limit the Effectiveness of the
Selection Process
AMS' processes for selecting investments contain the key elements
that leading organizations follow to ensure the selection of
projects that enhance mission performance and that are cost-
effective; however, weaknesses in some of the data used to support
the selection processes limit AMS' effectiveness. AMS' mission and
investment analysis processes provide FAA's senior management with
a basis for screening proposed projects; evaluating their relative
costs, benefits, and risks; and selecting projects for funding
based on their relative merits. But the cost information used to
make selection decisions is of questionable reliability, and there
is little evidence that the data or underlying analyses used in
the selection process are validated to ensure accuracy,
completeness, and appropriateness. As a result, FAA's managers
cannot be assured that they have all of the information needed to
make sound selection decisions.
Cost Data Are of Questionable Reliability, Though Improvements Are
Under Way
Consistently producing reliable cost estimates for projects
requires defining institutional processes for deriving estimates
and measuring actual performance against these estimates. However,
the cost data used in FAA's selection process are of questionable
reliability. The five projects we reviewed prepared their cost
estimates using techniques and data that we criticized in our
January 1997 report. 17 We found that FAA's modernization
program's cost estimating processes do not satisfy recognized
standards and that the agency does not have a cost accounting
system capable of reliably accumulating full cost information for
projects. FAA's cost estimating techniques do not satisfy
recognized standards that call for organizing and retaining cost
information on projects in a
17 GAO/AIMD-97-20 (Jan. 22, 1997).
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 31
Chapter 4 Weaknesses in the Selection, Control, and Evaluation
Phases Limit FAA's Effectiveness in Managing Its Investments
historical database and using cost models that are calibrated and
validated on the basis of actual experience. FAA's processes allow
each project to approach cost estimating in whatever manner its
estimators choose. The result is inconsistency in the rigor and
discipline with which cost estimates are derived, which in turn
means estimates vary in their degrees of reliability. For example,
of the six projects reviewed in our 1997 report, two were too
poorly documented to permit any comparative analysis, and none of
the remaining four satisfied all the recognized standards.
Compounding the estimating weaknesses is FAA's practice of
presenting cost estimates as precise point estimates, thus failing
to disclose the estimates' inherent uncertainty and risks.
Moreover, the effectiveness of FAA's cost estimating processes
also relies heavily on the quality of projects' actual cost
information, but FAA does not have a cost accounting system for
capturing and reporting the full costs of its projects.
Consequently, FAA cannot reliably use information about actual
cost experiences to improve its future cost estimating efforts. We
recommended that FAA institutionalize defined cost estimating
processes that include, among other items, a historical database
and structured approaches and tools.
In response to our 1997 report, FAA is developing a cost
estimating process for its projects that is intended to satisfy
recognized estimating standards; drafting guidance on reporting
projects' cost estimates as ranges rather than precise point
estimates and, in fact, reporting ranges on some systems; and
developing a cost accounting system. Additionally, FAA has
developed a document, known as a standard work breakdown
structure, 18 which provides a good first step toward the
development of a historical database on costs. FAA officials also
indicated that they are completing a cost estimating handbook that
contains a detailed discussion of cost estimating practices. When
completed, this handbook should contribute to improving FAA's
approach to estimating projects' costs. However, it does not
require a disciplined process for estimating costs throughout the
agency, and the draft handbook acknowledges that FAA still needs
to develop sophisticated tools and a historical database to
advance its cost estimating processes. FAA has not established
firm deadlines for completing the handbook or the other tasks
related to cost estimating. As for the cost accounting system, as
we discuss earlier, FAA officials underestimated the complexity of
developing the system and found that their implementation
milestones were unrealistic. The agency plans to fully implement
the cost accounting system by April 2001.
18 The work breakdown structure, which contains a detailed list of
activities to be accomplished in carrying out projects, is used to
develop life- cycle cost estimates for projects.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 32
Chapter 4 Weaknesses in the Selection, Control, and Evaluation
Phases Limit FAA's Effectiveness in Managing Its Investments
Selection Data Are Not Validated
Leading organizations validate the information and analyses
submitted in a new project proposal, which helps to ensure that
all the information is up- to- date, cost numbers are accurate,
benefits are quantified to the extent possible, alternatives are
identified, underlying assumptions are reasonable, and sensitivity
analyses are conducted. Validation is also important for ensuring
that a project's risks are identified, that the impact of risks on
the project's outcomes is quantitatively or qualitatively
projected, and that risk mitigation strategies are explained.
Explicit verification and validation steps sensitize
decisionmakers to important factors that have a bearing on
projects' actual outcomes, enhance accountability, and decrease
the likelihood that project proposals will contain analyses that
are based on inaccurate or incomplete data or faulty assumptions.
FAA has not completely defined the requirements for the validation
process under AMS, and it is not fully carrying out validation
activities. First, AMS does not require the validation of all data
used in the selection process. Under AMS guidance, the FAA
organization charged with carrying out the investment analysis
phase is responsible for validating only the cost and schedule
data, not the performance, benefits, or risk analyses used as part
of the selection process. Second, AMS guidance does not specify
what steps should be taken in validating selection data, nor does
it require documentation of the results and resolution of the
validation process. Finally, our review of the two projects for
which investment decisions had been made under AMS indicated that
FAA is not fully carrying out validation activities. For one
project, FAA did not provide any documentary evidence of the
validation efforts that were performed. For the other project, FAA
provided evidence of a validation review of the cost- benefit
analysis, but most of the results of that review were not
incorporated into the final version of the analysis used to
support the investment decision.
FAA Has Not Fully Implemented Its Process for Establishing and
Tracking Key Project Baselines
To control its projects at the agencywide level, FAA relies on
periodic reviews of each project's acquisition program baseline,
which, as noted earlier, is a document that establishes a
project's cost, schedule, benefits, and performance boundaries and
is intended to be used to monitor a project's status in achieving
those baselines. This document is incomplete, however, because its
schedule baseline does not include any milestones for project
reviews during the operations phase of a project and because it
does not address a project's risks. Moreover, FAA has not
completed about half of the baselines for the projects or project
segments that require baselines. Additionally, the agency- level
processes for tracking actual
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 33
Chapter 4 Weaknesses in the Selection, Control, and Evaluation
Phases Limit FAA's Effectiveness in Managing Its Investments
baseline performance against estimates frequently provided
incomplete information on projects' costs, schedules, benefits,
and performance. Finally, the investment control group of senior
managers, the Joint Resources Council, is not actively involved in
monitoring all projects after the investment decisions are made.
As a result, FAA's senior managers lack key information needed to
make sound decisions about the future of each project.
Leading organizations maintain a cycle of continual control and
monitoring after a project has been selected. Senior executives
review a project at specific milestones as the project moves
through its life cycle and as the dollars spent on the project
increase. At these milestones, the executives compare the expected
costs, risks, and benefits of earlier phases with the actual costs
incurred, risks encountered, and benefits realized to date. During
the control phase, senior executives determine whether projects
should be modified, continued, accelerated, delayed, or
terminated. The Clinger- Cohen Act also stresses the importance of
consistently monitoring the progress of federal investments in
information technology in meeting cost, schedule, and performance
objectives.
FAA's Requirements for Baseline Estimates Include Most, but Not
All, Key Project Parameters
Our review indicated that FAA's acquisition program baseline is
designed to capture sufficient information on most of a project's
key baseline elements, except for two limitations in the areas of
schedule and risk. One limitation is that the schedule baseline
does not address the operations phase of the project. At leading
organizations, even after a project has been implemented, senior
managers regularly review how well the acquired system meets
organizational needs, including whether it needs unexpected
modifications or premature replacement to meet emerging needs.
These reviews are used to make decisions pertaining to the
retirement or replacement of systems. In addition, because
operations activities such as hardware upgrades, system software
changes, ongoing training, and maintenance costs can consume a
significant level of resources, a plan for the review of each
project should be developed and periodically reevaluated. Given
that AMS is designed to track projects during their entire life
cycles, we expected to see one or more milestones in the schedule
baseline for a project review during the operations phase of FAA's
information technology investments, which can last as long as 15
years. Such milestones would allow the Joint Resources Council to
review projects periodically during the operations phase to
determine whether expected performance requirements and benefits
are actually being
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 34
Chapter 4 Weaknesses in the Selection, Control, and Evaluation
Phases Limit FAA's Effectiveness in Managing Its Investments
achieved in a cost- effective manner and to evaluate the need for
upgrading or enhancing the technology.
The second limitation is that the acquisition program baseline
does not include a project's expected risks. The Office of
Management and Budget's guidance requires that federal agencies
estimate a project's risks and develop a strategy for mitigating
those risks. 19 While FAA performs a risk analysis as a part of
the investment analysis phase, the agency does not systematically
monitor each project's risks during the control phase. Given that
the acquisition program baseline is FAA's primary mechanism for
controlling projects, we expected it to include a baseline
assessment of risks that could be used to monitor mitigation and
resolution of actual risks that occur during a project's life
cycle. Although all five projects we reviewed had discussions of
risk issues during their project reviews, none of them presented
any systematic assessment of estimated versus actual risks because
risk is not a required element of the acquisition program
baseline.
Requiring an assessment of a project's risks as an element of the
acquisition program baseline and systematically monitoring those
risks would allow FAA to identify red flag issues that may have an
impact on a project's cost, schedule, and performance. For
example, two of the projects we reviewed the Standard Terminal
Automation Replacement System and the NAS Infrastructure
Management System have experienced difficulties with acquiring or
developing software. FAA promoted both of these as projects that
would use commercial off- the- shelf software that would require
very little software development. For both projects, FAA
underestimated the lines of software code that needed to be
developed or modified, and as a result, the costs for the projects
have increased and the schedules have been delayed. FAA has
historically had difficulty acquiring software the most costly and
complex component of information technology systems and the agency
has initiated some efforts to improve its software acquisition
processes. 20 Improvements in these processes, coupled with an
identification of the risks associated with software acquisition
for new projects, would help FAA better manage such risks.
19 See Office of Management and Budget Circular No. A- 11, Part 3,
Planning, Budgeting, and Acquisition of Capital Assets, Executive
Office of the President (June 1997), and Capital Programming
Guide, Version 1.0, Supplement to Office of Management and Budget
Circular A- 11 (July 1997).
20 See Air Traffic Control: Immature Software Acquisition
Processes Increase FAA System Acquisition Risks (GAO/AIMD-97-47,
Mar. 21, 1997).
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 35
Chapter 4 Weaknesses in the Selection, Control, and Evaluation
Phases Limit FAA's Effectiveness in Managing Its Investments
FAA Has Approved Half of Its Required Baselines, and Some Baseline
Documentation Is Incomplete
FAA has identified 70 modernization projects or project segments
that require baseline documentation. 21 These 70 projects and
segments account for $1.266 billion, or 55 percent, of the $2.319
billion appropriation FAA requested for fiscal year 2000 for all
modernization activities funded by the facilities and equipment
budget account. Most of the modernization activities that do not
have baselines involve improving or sustaining ATC facilities and
other buildings; providing technical support services; or funding
personnel, compensation, benefits, and travel costs.
Of the 70 projects and segments, 10 involve projects that are
currently in the investment analysis phase of AMS and, hence, do
not yet require an approved baseline. Thus, 60 projects and
segments that account for $1.205 billion in fiscal year 2000
estimated costs, currently require a baseline. Of the 60, half
have approved baseline documentation (either acquisition program
baselines or parameter sheets). 22 These 30 projects and segments
account for $619.4 million in estimated costs for fiscal year
2000. Table 4.1 shows the status of FAA's efforts to develop AMS
baseline documentation on the 60 ongoing projects and segments.
21 Some of the 70 baselines cover entire projects, while others
cover segments of projects, meaning that some projects have more
than one baseline. For example, 4 of the 70 baselines are related
to the Automated Surface Observing System, 2 are related to the
Oceanic Automation Program, 2 are related to the NAS
Infrastructure Management System, and 2 are related to the Air
Traffic Management (Infrastructure) Program.
22 As noted in chapter 2, parameter sheets summarize a subset of
baseline information that is designated for control by the Joint
Resources Council and that is considered critical to assessing a
project's ability to satisfy mission need, to achieve needed
operational capability and benefits, and to meet the schedule
requirements of interdependent programs.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 36
Chapter 4 Weaknesses in the Selection, Control, and Evaluation
Phases Limit FAA's Effectiveness in Managing Its Investments
Table 4.1: Status of FAA's Efforts to Develop AMS Baseline
Documentation on 60 Ongoing Projects and Segments for Which
Investment Decisions Have Been Made
Dollars in millions
FAA's efforts to develop baseline documentation
Number of projects and
segments Percentage of
total projects and segments
Fiscal year 2000 estimated
cost of projects and
segments Percentage of
total fiscal year 2000 estimated
costs
Acquisition program baselines and parameter sheets approved by the
Joint Resources Council 30 50 $619.4 51
Parameter sheets undergoing review for validation of operations
costs and other funding issues 4 7 292.1 24
Acquisition program baselines and parameter sheets being planned
26 43 293.9 24
Total - all acquisition program baselines and parameter sheets 60
100 $1,205.4 99 a
a Percentages do not add to 100 because of rounding. Source: GAO's
analysis of FAA's data.
Our review of five projects found that some of the baseline
documentation (both acquisition program baselines and parameter
sheets) is incomplete. Of the five projects, two had acquisition
program baselines (plus parameter sheets), and three had parameter
sheets only. Only the performance element was fully documented on
all of the projects, while the cost, benefits, and schedule
elements were missing some of the information required by AMS. For
the cost baseline, for example, two of the five projects did not
estimate operations costs, and one of the two projects with
acquisition program baselines did not provide information on the
detailed cost elements required by AMS guidance. For the benefits
baseline, one of the two projects with acquisition program
baselines did not identify
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 37
Chapter 4 Weaknesses in the Selection, Control, and Evaluation
Phases Limit FAA's Effectiveness in Managing Its Investments
the measurements that would be used to determine whether a benefit
had been achieved. For the schedule baseline, none of the projects
estimated the in- service decision, a milestone required by AMS
that specifies when the newly acquired system or equipment is
certified ready for operational use.
FAA Is Not Completely Monitoring Projects' Actual Performance
Against Their Baselines
Our review and FAA's internal evaluation of acquisition reform 23
found that the I- BEAM and acquisition review processes frequently
provide incomplete reporting on projects' estimated versus actual
performance in the areas of cost, schedule, benefits, and
performance. We reviewed the monthly status reports on the five
projects and found information was missing on all of the baseline
elements. For example, none of the monthly status reports on the
five projects analyzed operations costs; assessed estimated versus
actual benefits, even though benefits were projected for fiscal
years 1997 or 1998 on four of the five projects; or contained
information on the performance requirements outlined in the
acquisition program baseline or parameter sheet to confirm whether
the original baseline requirements still applied. Our results are
consistent with those of FAA's internal evaluation, which found
that the cost, schedule, and performance data in the monthly
status reports were generally inconsistent with the estimates in
the acquisition program baselines and parameter sheets. With
regard to FAA's tracking of baseline information through the
acquisition review process, our review of the five projects showed
that information was missing for all of the baseline parameters,
as shown in table 4.2.
23 Evaluation of Acquisition Reform The First Two Years: April
1996- March 1998, FAA Program Evaluation Branch, Office of Systems
Architecture and Investment Analysis (May 29, 1998).
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 38
Chapter 4 Weaknesses in the Selection, Control, and Evaluation
Phases Limit FAA's Effectiveness in Managing Its Investments
Table 4.2: Analysis of the Missing Baseline Information Reported
in Acquisition Reviews of Five Projects
Baseline element Missing information
Cost Four projects did not analyze operations costs. Three
projects did not compare the original (or revised) baseline cost
estimates with actual results achieved to date.
Of the two projects with acquisition program baselines, neither
provided information on the detailed cost elements required by AMS
guidance.
Schedule Three projects did not compare the original (or revised)
baseline milestones with actual results achieved to date.
One project did not address two of the Joint Resources Council-
controlled milestones: contract award and in- service decision.
One project only showed milestones for 3 years of the 9- year
acquisition cycle identified in the parameter sheet.
Benefits The acquisition review did not address the baseline
benefits in the acquisition program baselines or parameter sheets
for any of the five projects, even though, for four of them,
benefits were projected for fiscal years 1997 or 1998.
Performance Four projects did not address all of the performance
parameters contained in the acquisition program baseline or
parameter sheet, and one of those projects had no information on
performance at all.
Source: GAO's analysis of FAA's data.
FAA officials managing the five projects we reviewed cited several
factors to explain the lack of complete baseline data on their
projects. First, project officials told us that frequent budget
reductions imposed either by the Congress or by FAA make it very
difficult to establish a stable baseline and to monitor a
project's performance against that baseline. Second, AMS guidance
on the acquisition program baseline requirements has evolved over
time, and project officials told us that detailed guidance was not
available to projects that established their baselines during the
first year of AMS' implementation. Finally, the two mechanisms
used by FAA to monitor projects' status the monthly status report
and the acquisition reviews were developed prior to AMS and thus
do not always address the specific baseline elements that are
supposed to be contained in the acquisition program baselines,
report the current status of all baseline estimates, or report the
actual deviations from those estimates.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 39
Chapter 4 Weaknesses in the Selection, Control, and Evaluation
Phases Limit FAA's Effectiveness in Managing Its Investments
We recognize that variances in funding levels exist and, in some
cases, have had an impact on FAA's ability to manage its projects.
However, we and others have found that FAA's problems with
baseline parameters have resulted from factors other than funding.
Although AMS' detailed guidance and reporting requirements are
evolving, establishing complete acquisition program baseline
documentation that has been approved by the Joint Resources
Council helps to ensure that all projects are being held
accountable to the cost, schedule, benefits, and performance
baselines established when the Council made the initial investment
decision. Furthermore, the absence of complete, up- to- date data
on estimated versus actual results means that FAA has little
assurance that its estimates of projects' costs, schedule,
benefits, performance, or risks are sound and accurate or that
projects will be managed so that they meet the agency's
expectations. This, in turn, restricts the ability of FAA's senior
management to make sound decisions about continuing the agency's
investments.
FAA's long- standing problems with implementing projects that meet
their cost, schedule, and performance objectives illustrate the
need for the agency to better manage its baselines. As noted
earlier, over the past 17 years, FAA's modernization projects have
experienced substantial cost overruns, lengthy delays, and
significant performance shortfalls, problems that have persisted
since the implementation of AMS in April 1996. Two of the projects
in our review provide examples of continuing cost and schedule
problems. In the case of the Standard Terminal Automation
Replacement System, although FAA has not officially changed the
baseline approved in February 1996, the baseline is in jeopardy of
being breached because of unions' concerns about human- factor and
design issues, 24 the refinement of the project's requirements,
and the interjection of a new project phase. 25 FAA estimates that
these issues have the potential to increase the project's costs
from $294 million to $410 million above the approved baseline. FAA
also estimates that the project's initial completion date could be
delayed by almost 2- 1/ 2 years. Similarly, in the case of the NAS
Infrastructure Management System, the agency has not officially
changed the baseline that was approved in March 1997. However, the
project's leader expects significant baseline breaches to occur,
including a 58- percent increase in the costs from $100.8 million
to $159.5 million, and a 123- percent increase in the schedule
from 48 to 107 months for the project's total duration.
24 Concerns were raised by two unions, the National Air Traffic
Controllers Association and the Professional Airways Systems
Specialists. 25 GAO/RCED-99-25, Dec. 3, 1998.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 40
Chapter 4 Weaknesses in the Selection, Control, and Evaluation
Phases Limit FAA's Effectiveness in Managing Its Investments
Joint Resources Council Has Limited Involvement in Reviewing
Projects After the Investment Decisions
Under AMS, the Joint Resources Council conducts reviews and makes
decisions on each project funded from the facilities and equipment
account, from the determination of mission needs to the point at
which an investment decision is made. After the investment
decision, the Joint Resources Council generally only becomes
directly involved in monitoring a project when there is a
potential or actual breach of the established baseline. As a
result, the Joint Resources Council is not proactively involved in
making decisions about the future of most projects when they are
being developed, deployed, operated, and maintained; when
decisions are being made about technology upgrades or other
enhancements; and when assessments are being made about whether
the system or product is achieving the intended benefits and
meeting the expected performance requirements.
The Acquisition Executive, who chairs the Joint Resources Council,
told us that he highlights projects whose baseline status is
problematic according to data in the monthly status report and
that he provides additional oversight for those projects. However,
as we have stated, the data contained in the monthly status
reports are incomplete and generally inconsistent with the
estimated baseline elements in the acquisition program baselines
and parameter sheets.
FAA's AMS Lacks a Post- Implementation Review Process for
Evaluating Investments
FAA does not have a defined, documented process for conducting
post- implementation reviews of projects to assess their
performance and to improve the selection and control of its other
investments. FAA performs some elements of a post- implementation
review in its life- cycle management review process, including
such tasks as the independent operational test and evaluation of
some projects prior to their deployment, operational performance
monitoring, customer satisfaction surveys, and periodic reviews
throughout an investment's life cycle. However, this process is
not standardized and is not required for all projects. As a
result, there is no evidence that changes, especially to the
selection and control phases, are being implemented based on
lessons learned. Although FAA has not yet designed or implemented
a post- implementation review process for individual projects, its
fiscal year 1999 performance plan for the Research and
Acquisitions operating division includes a new requirement for a
post- deployment assessment of NAS modernization systems.
The evaluation phase closes the loop on the investment management
process by comparing actual results against baseline estimates to
assess performance and identify areas where future decision-
making can be
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 41
Chapter 4 Weaknesses in the Selection, Control, and Evaluation
Phases Limit FAA's Effectiveness in Managing Its Investments
improved. Lessons that are learned during the evaluation phase can
be used to improve future decisions about selecting and
controlling projects. Central to this process is the post-
implementation review, with its evaluation of the historical
record of a project and its comparison of actual versus expected
costs and benefits. Recognizing the importance of this phase of
investment management, the Clinger- Cohen Act requires federal
agencies to evaluate the performance of information technology
projects and to use that information to decide whether to
continue, modify, or terminate projects.
At leading organizations, this review generally occurs about 3 to
12 months after a project has reached its end point (i. e., the
point at which the project has been fully implemented or canceled)
and is generally conducted by a group other than the project team
to ensure that the review is independent and objective. In
conducting post- implementation reviews, an organization can
survey customers to determine users' satisfaction with the
completed product and how well the project supports business
processes; assess whether the investment has had its intended
impacts on mission goals, cost savings, compliance with the
system's architecture, and other issues involving information
accuracy, timeliness, adequacy, and appropriateness; and evaluate
current and future technical issues associated with the
investment.
FAA's Efforts to Implement an Agencywide Management Information
System Do Not Include Key Selection Data
Until recently, FAA lacked a centralized, standardized management
information system or historical database for capturing and
maintaining project information. While FAA had a number of stand-
alone databases within different groups, none provided a complete
picture of estimates, the assumptions that made up the estimates,
revisions, and actual performance on projects. As a result, agency
officials had no assurance that the project data from these stand-
alone systems were complete, accurate, and up- to- date.
In March 1999, FAA officials began implementing a centralized
repository and management information system under AMS called the
Simplified Program Information Reporting and Evaluation (SPIRE) to
provide access to projects' baselines and other information. This
system consolidates project information related to FAA's processes
for managing key baselines, including data from the acquisition
program baselines and parameter sheets, monthly status reports,
baseline management notices, meeting minutes from the acquisition
reviews, Joint Resources Council decisions, and other baseline
information. The system does not, however, contain
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 42
Chapter 4 Weaknesses in the Selection, Control, and Evaluation
Phases Limit FAA's Effectiveness in Managing Its Investments
key information from the selection process, such as mission need
statements, cost- benefit analyses, risk assessments, or other
required reports. FAA plans to implement SPIRE in three phases.
Phase I, which commenced in March 1999, provides the capability to
store and display the status and variance reports that have been
input by project leaders. Subsequent phases, which do not yet have
implementation dates, will focus on automatically generating
various reports on projects' baseline status and variances.
Informed management decisions can be made only if accurate,
reliable, and up- to- date information from all phases of the
investment management process is included in the decision- making
process. To do this requires that agencies have a uniform
mechanism that is, a management information system with uniform
data standards and entry procedures for collecting, automating,
and processing data on projects' expected versus actual outcomes.
Data in this system should include the initial cost, schedule,
benefits, performance, and risk estimates that were developed
during the selection process. Various analyses that were conducted
to initially justify the project, along with revised estimates,
reasons for revisions, and actual performance measured against the
estimates, should also be included. These data need to be
continually updated as each project's implementation continues and
as expenditures increase. The data also need to be easily
accessible to both the project team and senior managers.
A management information system, if kept accurate and up- to-
date, can make data verification and validation easier by allowing
an organization to track costs, risks, and other factors over
time. It is also essential from the standpoint of establishing an
organizational memory throughout the selection, control, and
evaluation phases of the investment management process. As such,
it can be used to help assess whether projects are still aligned
with mission needs and organizational objectives, determine
whether projects are meeting planned performance goals, and
identify possible revisions to the overall investment management
process based on previous experiences and lessons learned.
FAA's efforts to consolidate project baseline information in SPIRE
are a positive step toward improving the agency's mechanisms for
tracking projects and helping to ensure the consistency of
information and reporting on all the projects in its investment
portfolio. However, SPIRE provides an incomplete record of project
information since it does not include key information from the
selection process.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 43
Chapter 4 Weaknesses in the Selection, Control, and Evaluation
Phases Limit FAA's Effectiveness in Managing Its Investments
Conclusions AMS has weaknesses in its selection, control, and
evaluation phases that impede FAA's ability to manage its
investments effectively and to make
sound decisions about continuing, modifying, or canceling
projects. First, using data that have not been validated and
unreliable cost information reduces the likelihood that FAA's
managers can make informed decisions about the relative merits of
competing investments. Second, requiring FAA's senior managers to
stay actively involved in the process of controlling project
baselines and providing them with complete information on all
projects is critical to monitoring how well projects are achieving
their intended results. Third, establishing post- implementation
reviews is essential for evaluating projects' performance and
identifying areas for which future decision- making could be
improved during the selection and control phases. Finally,
establishing a standardized management information system that
includes complete information from all three investment management
phases selection, control, and evaluation will help facilitate
FAA's efforts to track projects' costs, risks, and other factors
over time, providing senior managers with uniform, accurate,
reliable, and easily accessible data on all the projects in the
agency's portfolio. Taking steps to correct these weaknesses
increases the likelihood that FAA's projects will meet established
cost and schedule objectives and contribute to measurable
improvements in the agency's mission performance.
Recommendations We recommend that the Secretary of Transportation
direct the Administrator of FAA to take the following actions:
Improve the selection process by (1) establishing clearly defined
procedures for validating projects' cost, schedule, benefit,
performance, and risk information and (2) requiring documentation
of the results of the validation procedures applied to each
project. Strengthen control over investments by (1) revising the
acquisition
program baseline requirements to include project risks and to add
milestones for project reviews during the operations phase and (2)
ensuring that project officials fully track and document estimated
versus actual results on all the elements (i. e., costs, schedule,
benefits, performance, and risks) contained in the baseline
documentation. Initiate post- implementation evaluations for
projects within 3 to 12 months
of deployment or cancellation to compare the completed projects'
costs, schedule, performance, and mission improvement outcomes
with the original estimates.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 44
Chapter 4 Weaknesses in the Selection, Control, and Evaluation
Phases Limit FAA's Effectiveness in Managing Its Investments
Incorporate key information from the selection process (e. g.,
mission need statements, cost- benefit analyses, and risk
assessments) into FAA's management information system for
investments.
Agency Comments FAA agreed with our recommendations.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 45
Appendix I Background and Status of Five Projects
This appendix provides detailed information on the purpose, scope,
and status of the five information technology projects we
reviewed. The Federal Aviation Administration (FAA) considers each
of these projects key to replacing the aging National Airspace
System's (NAS) infrastructure or to improving its capacity and
effectiveness. Each project is estimated to require the
expenditure of more than $100 million from the agency's facilities
and equipment budget account before it becomes operational.
NAS Infrastructure Management System
Background The NAS Infrastructure Management System will provide
the next generation of tools, services, and operational
philosophies that govern the management, operation, and
maintenance of the NAS infrastructure. Currently, the air traffic
control system relies on a nationwide infrastructure of facilities
and equipment to provide communications, navigation, surveillance,
and automation system capabilities. To offset the continual loss
of experienced technical staff devoted to the maintenance of the
NAS systems, FAA has dispersed its maintenance workforce into work
centers that, in some cases, are colocated with high- impact
facilities such as terminal radar approach control facilities and
en route centers. The agency is also using a remote maintenance
monitoring system to recognize and address problems in a more
timely manner. However, this monitoring system simply collects
performance data from individual systems and logs historical
maintenance actions. The NAS Infrastructure Management System will
build on the investment made in this monitoring system and will
focus on workforce and event management two separate but related
components of system management. Through this project, FAA will be
able to track and monitor the actual costs of providing NAS
services.
The mission need statement for this project was submitted in
December 1995 and revalidated in February 1996, and a cost-
benefit analysis was completed in December 1995. FAA approved the
investment decision and acquisition program baseline for Phase I
of the project in March 1997.
According to project officials, the NAS Infrastructure Management
System will be implemented in three phases. Only Phase I, which
covers fiscal years 1998 through 2002, is fully defined. In this
phase, FAA plans to create the initial infrastructure needed to
manage, operate, and maintain overall
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 46
Appendix I Background and Status of Five Projects
NAS operations. This infrastructure will consist of a National
Operations Control Center in Herndon, Va., and three regional
operational control centers in Hampton, Ga.; Olathe, Kans.; and
San Diego, Calif. Below these three regional centers will be up to
50 service operations centers modeled after the facilities that
are currently colocated with high- impact facilities. Under Phase
II, FAA plans to provide centralized asset management, develop
customer and user interaction tools, and analyze technical and
cost trends. Under Phase III, the agency plans to provide
intelligent fault correlation, information sharing, and additional
functionality commensurate with NAS technological improvements.
Status According to FAA, the National Operations Control Center
became operational in January 1999. The three regional centers are
scheduled to begin operations in October 2000. Phase I of the NAS
Infrastructure Management System will support FAA's Free Flight
Phase I in fiscal years 2000- 2002. 26 Phases II and III are
currently undergoing an investment analysis to determine if
further investment is warranted.
Although FAA has not officially changed the project's baseline
that was approved in 1997, the project's leader estimated
significant breaches in December 1998, as follows: (1) the cost
baseline is expected to increase by 58 percent, from $100.8
million to $159.5 million, and (2) the schedule is expected to
increase by 123 percent, from 48 to 107 months' total duration for
Phase I.
Oceanic Automation Program
Background The Oceanic Automation Program is designed to provide a
platform for improved air traffic control over the oceans. It
evolved as a series of projects each advancing on the technology
of its predecessor from the Oceanic Display and Planning System
into the Oceanic Automation System, and now, into the Advanced
Oceanic Automation System. In the late 1980s, the Oceanic Display
and Planning System improved oceanic traffic control by providing
flight data processing and a situational display of estimated
aircraft positions. This system also alerted controllers when
26 Free flight is a new system of air traffic management that will
provide controllers and pilots with new technologies and
procedures that will allow them to increase the safety, capacity,
and efficiency of air traffic operations throughout the NAS.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 47
Appendix I Background and Status of Five Projects
any flight plan or any route change requested by a pilot (referred
to as conflict probe capability) violated appropriate separation
standards. In the early 1990s, FAA introduced the Oceanic
Automation System, which improved data display and communications.
This system is now being upgraded to the Advanced Oceanic
Automation System, which is designed to provide such features as a
new flight data processor, automatic dependent surveillance
position reporting, an advanced conflict probe, and data link. FAA
awarded a contract to the Raytheon Systems Corporation in
September 1995 for the Advanced Oceanic Automation System. The
contract is composed of flexible segments, which will allow for
incremental functional development and delivery of benefits.
Oceanic air traffic control systems are installed at the en route
centers at Oakland and New York and in Anchorage, Alaska.
The mission need statement was approved in May 1992. In October
1992, the acquisition plan was approved to consolidate and
integrate the primary oceanic improvement projects into a single
Oceanic Automation Program. A revised mission need statement and
acquisition plan was approved in January 1994.
Status Since FAA awarded the Advanced Oceanic Automation System's
contract in September 1995, the scope of the project has been
gradually cut back from an original plan of five segments (that
is, five incremental deliveries of capabilities) to only a portion
of the first segment. In July 1996, 10 months after the contract's
award, FAA canceled segments 3, 4, and 5 of the project because
the agency recognized that the cost of executing these segments
was beyond the funding that had been allocated for this project.
As a result, FAA abandoned many controller productivity tools
needed to increase the system's capacity. Then, in December 1996,
funding concerns forced FAA to revise the Segment 2 of the
project, which was designed to replace infrastructure hardware and
software that supports controller equipment. Eventually, in
September 1997, FAA canceled the entire Segment 2 because the
agency needed to use the project's funds to correct Year 2000
problems in existing oceanic automation software and because it
needed to transfer funds to the Host replacement program. 27
Meanwhile, FAA's contractor was reporting performance problems
with the Segment 1 of the project, which adds data link and
automatic dependent surveillance in the oceanic environment. To
avoid a potential $45 million
27 The Host replacement project replaces en route center and
oceanic automation hardware that has reached the end of its
commercial support life and may have problems with Year 2000 date
requirements.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 48
Appendix I Background and Status of Five Projects
cost overrun for this segment, FAA reduced its scope in September
1998 by eliminating the capability for automatic dependent
surveillance. According to project officials, the remaining
elements of Segment 1 (the air- to- ground data link, the ground-
to- ground data link, and controller tools) have successfully
completed the operational test and evaluation and are expected to
be delivered on schedule. The last site implementation is
estimated for October 1999.
Operational and Supportability Implementation System
Background The Operational and Supportability Implementation
System project (1) replaces the Flight Service Automation System's
hardware and software with a leased commercial, off- the- shelf-
based service; (2) provides an improved graphic weather display
capability; and (3) incorporates direct user access functionality
that is currently being obtained through two direct user access
terminal contracts. The integration of these three capabilities
and functions into a single system will enable flight service
specialists to more efficiently provide weather and flight-
planning information for pilots.
The mission need statement was approved in October 1993 and
revalidated in December 1996. The acquisition program baseline was
approved in April 1997, and in August 1997, FAA awarded a contract
to Harris Corporation for the project. The contract requires
Harris to provide up to 61 operational systems and 3 support
systems.
Status Since FAA awarded the contract, the project's schedule has
slipped because the development effort has been larger than
planned. FAA's January 1998 review of the Harris system's
architecture for the project revealed that the contractor's
commercial, off- the- shelf solution was not as mature as FAA had
envisioned when the contract was awarded and that many of the
contractor's commercial products did not fully satisfy FAA's
requirements. In May 1998, the agency decided to replace
workstation consoles in response to human- factor concerns raised
by the unions that represent its
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 49
Appendix I Background and Status of Five Projects
controllers and the technicians. This caused the project's costs
to increase by $15.8 million. FAA also delayed first- site
implementation from July 1998 to January 1999 a 6- month slip. The
protracted development effort is not expected to delay the
completion of the project, with the last site to receive the
system still scheduled to be operational in August 2001.
Standard Terminal Automation Replacement System
Background The Standard Terminal Automation Replacement System is
designed to replace FAA's automated radar terminal system, which
comprises 15- to 25- year- old air traffic controller workstations
and the supporting computer systems that allow controllers at
terminal radar approach control facilities to separate and
sequence aircraft. According to FAA, the old system is prone to
failures and requires extensive maintenance. The old system also
has capacity constraints that restrict the agency from making
required safety and efficiency enhancements. Besides remedying
those problems, the new equipment is also expected to allow the
system to increase the level of air traffic control automation and
to improve surveillance, communications, and weather display. This
system replaced a segment of another project (the Advanced
Automation System) that was terminated because of serious cost and
schedule problems.
The mission need statement for the Standard Terminal Automation
Replacement System was submitted in July 1993 and revised in June
1995. The acquisition plan was approved in March 1996, and in
September 1996, FAA signed a contract with Raytheon Corporation to
acquire this system. In producing it, Raytheon originally intended
to rely exclusively on commercially available hardware and, to a
large extent, on commercially available software.
The initial strategy for replacing and enhancing the system is
divided into two stages. Stage 1 is expected to provide the same
functions as the current automated radar terminal systems. Stage 2
is expected to implement new functions to help controllers move
aircraft more safely and efficiently. In 1997, FAA created another
stage, known as early display configuration, because of concerns
about operational problems at Ronald Reagan National Airport in
Washington, D. C. This new stage will be
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 50
Appendix I Background and Status of Five Projects
implemented prior to Stages 1 and 2. The new stage replaces the
current controller displays and monitoring equipment but uses the
existing computer system and software. It also provides an
emergency backup system.
Status Although FAA has not officially changed the project's
baseline that was approved in 1996, the baseline is in jeopardy of
being breached because of the unions' concerns about human- factor
and design issues, the refinement of the requirements, and the
interjection of a new project phase. FAA estimates that these
issues have the potential to increase the project's costs anywhere
from $294 million to $410 million over the approved baseline. FAA
also estimates that the project's initial completion date could be
delayed by almost 2- 1/ 2 years. In addition, the project has
experienced other challenges mainly involving software testing.
While project officials stated that they have been able to absorb
the cost increases associated with this issue within the existing
baseline, additional problems could cause further cost increases
and schedule delays. The last site implementation is estimated for
February 2005.
Voice Switching and Control System
Background The Voice Switching and Control System replaces
existing communication systems at en route centers with an
expandable, highly reliable system for both ground- to- ground and
air- to- ground communication. This system will also provide
communication capability for new en route center controller
workstations that are being installed. FAA is also installing the
Voice Switching and Communications System Training and Backup
Switch an emergency backup communications system at all en route
centers.
This system was designed to provide the communication capabilities
for the new Initial Sector Suite System workstations under the
Advanced Automation System program. By the time the contract was
awarded in December 1991 to the Harris Corporation, FAA had spent
5 years developing prototypes and had incurred cost growth of
about $1 billion. 28 The contract required Harris to deliver 23
systems 21 for en route
28 According to project officials, the primary reason for this
growth was the inability of commercially available products to
effectively and accurately manage air traffic control
communications functions.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 51
Appendix I Background and Status of Five Projects
centers and 2 support systems. FAA's plans called for the Voice
Switching and Communications System to be installed with both the
current equipment and with the new controller workstations. During
the initial development, the cost of the project increased by
$53.1 million to approximately $1.45 billion primarily because of
FAA's decision in 1994 to cancel the Initial Sector Suite System
component of the Advanced Automation System and replace it with
the Display System Replacement project. The restructuring resulted
in the need for additional equipment and testing and in the
retention of contractor and project personnel longer than planned
to field the communications equipment with new controller
workstation equipment. FAA has also added new functionality
requirements to the project.
The original concept for this project was defined in 1980. In
1984, operational requirements were finalized, and in 1985, the
project was approved for development. A revised draft mission need
statement was completed in January 1994.
Status Harris developed and installed the system in the existing
en route controller work stations in February 1997 5 months ahead
of the
schedule established at the time the contract was awarded. Harris
is currently reinstalling the controller interface equipment into
the en route Display System Replacement controller workstations.
Harris has completed the software development for the primary
system to be fielded with the new Display System Replacement
controller workstations. According to the project manager, the
project has not encountered any technical problems and is not
expected to incur any major delays. The last site implementation
is estimated for May 2000.
FAA is in the process of installing the emergency backup system at
en route centers, and by November 1999, FAA expects to have
completed installation at all en route centers, the FAA Technical
Center, and the FAA Academy.
GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page 52
Appendix II Major Contributors to This Report
Resources, Community, and Economic Development Division,
Washington, D. C.
John H. Anderson, Jr. Rita A. Grieco Tina M. Kinney Belva M.
Martin Thomas F. Noone
Accounting and Information Management Division, Washington, D. C.
David L. McClure Colleen M. Phillips Robert C. Reining
(348085) GAO/ RCED/ AIMD- 99- 88 FAA's Investment Management Page
53
Ordering Information The first copy of each GAO report and
testimony is free. Additional copies are $2 each. Orders should be
sent to the following address, accompanied by a check or money
order made out to the Superintendent of Documents, when necessary.
VISA and MasterCard credit cards are accepted, also. Orders for
100 or more copies to be mailed to a single address are discounted
25 percent.
Orders by mail: U. S. General Accounting Office P. O. Box 37050
Washington, DC 20013
or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW)
U. S. General Accounting Office Washington, DC
Orders may also be placed by calling (202) 512- 6000 or by using
fax number (202) 512- 6061, or TDD (202) 512- 2537.
Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512- 6000 using a
touchtone phone. A recorded menu will provide information on how
to obtain these lists.
For information on how to access GAO reports on the INTERNET, send
an e- mail message with "info" in the body to:
info@ www. gao. gov or visit GAO's World Wide Web Home Page at:
http:// www. gao. gov
PRINTED ON RECYCLED PAPER
United States General Accounting Office Washington, D. C. 20548-
0001
Official Business Penalty for Private Use $300
Address Correction Requested Bulk Rate
Postage & Fees Paid GAO Permit No. G100
*** End of document. ***