National Airspace System: Better Cost Data Could Improve FAA's	 
Management of the Standard Terminal Automation Replacement System
(31-JAN-03, GAO-03-343).					 
                                                                 
To enhance the capacity and safety of the national airspace	 
system, the Federal Aviation Administration (FAA), within the	 
Department of Transportation, is acquiring 74 Standard Terminal  
Automation Replacement Systems (STARS). STARS will replace some  
outdated air traffic control equipment. Since 1996, when FAA	 
initiated this major computer hardware and software acquisition, 
the scope and estimated costs of STARS have changed many times.  
FAA now estimates that STARS's remaining costs will total about  
$2.54 billion. GAO was asked to assess the reliability of FAA's  
life-cycle cost estimate for STARS, determine the impact of	 
STARS's estimated costs on future FAA budgets, and identify any  
alternatives to STARS that FAA is considering. GAO based its	 
analysis on published FAA cost data and the guidance FAA uses for
managing major acquisitions.					 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-03-343 					        
    ACCNO:   A06040						        
  TITLE:     National Airspace System: Better Cost Data Could Improve 
FAA's Management of the Standard Terminal Automation Replacement 
System								 
     DATE:   01/31/2003 
  SUBJECT:   Cost analysis					 
	     Future budget projections				 
	     Life cycle costs					 
	     Program evaluation 				 
	     Strategic planning 				 
	     FAA Standard Terminal Automation			 
	     Replacement System 				 
                                                                 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-03-343

                                       A

Report to the Honorable Ellen O. Tauscher, House of Representatives

January 2003 NATIONAL AIRSPACE SYSTEM Better Cost Data Could Improve FAA*s
Management of the Standard Terminal Automation Replacement System

GAO- 03- 343

Letter 1 Results in Brief 2 Background 4 Reliability of FAA*s Life- Cycle
Cost Estimate for STARS Is

Uncertain 8 Impact of STARS*s Estimated Costs on FAA*s Budgets Is Expected
to

Decline through Fiscal Year 2007 13 After Deploying STARS at 74
Facilities, FAA May Modify Its

Approach for the Remaining Facilities 16 Conclusions 16 Recommendations
for Executive Action 17 Agency Comments 17 Scope and Methodology 18

Appendix

Appendix I: GAO Contacts and Staff Acknowledgments 20 GAO Contacts 20
Staff Acknowledgments 20

Table Table 1: Proposed Funding for FAA*s Total Facilities and Equipment
Budget and for STARS*s Development and Deployment 15

Figures Figure 1: Standard Terminal Automation Replacement System (STARS)
5

Figure 2: Life- Cycle Cost Estimate for STARS, Fiscal Years 2004- 2030 9
Figure 3: Allocation of Facilities and Equipment Funding for Fiscal

Year 2004 in FAA*s Capital Investment Plan 14

Abbreviations

ARTS Automated Radar Terminal System DCMA Defense Contract Management
Agency DOD Department of Defense FAA Federal Aviation Administration STARS
Standard Terminal Automation Replacement System

This is a work of the U. S. Government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. It may contain
copyrighted graphics, images or other materials. Permission from the
copyright holder may be necessary should you wish to reproduce copyrighted
materials separately from GAO*s product.

Letter

January 31, 2003 The Honorable Ellen O. Tauscher House of Representatives
Dear Ms. Tauscher: On November 17, 2002, the Federal Aviation
Administration (FAA) began using the Standard Terminal Automation
Replacement System (STARS) to control air traffic at the Philadelphia air
route traffic control center, the first *busy* FAA facility to use this
version of the new system. 1 FAA*s current plan is to procure 74 STARS
systems, including 70 for terminal

facilities and 4 for support facilities. 2 STARS will replace outdated
computer equipment that is used to control air traffic within 5 to 50
nautical miles of an airport. 3 With STARS, air traffic controllers at
these

facilities will receive new hardware and software that produce color
displays of aircraft position and flight information. In the future, FAA
will be able to upgrade the software to provide air traffic control tools,
such as those that will allow better spacing of aircraft as they descend
into airports. STARS is complex, costly, and software- intensive. Since
1996, when FAA initiated STARS, the number of systems scheduled to be
procured has ranged from as many as 188 to as few as 74, 4 and the
program*s cost and schedule have also varied considerably. Over the years,
we have reported on these changes, most recently in September 2002. 5 As
agreed with your office, this report addresses the following questions:

1 A previous version of STARS has been in use at smaller facilities since
1999. Philadelphia is the 14th- largest terminal facility in terms of
handling operations under instrument flight rules. FAA*s data show that
Philadelphia handled 686,000 operations between January and December 2000.

2 To support STARS operations at terminal facilities, FAA currently has
four systems dedicated to maintenance. 3 The Department of Defense (DOD)
is also procuring STARS for 153 of its facilities. 4 FAA currently plans
to deploy STARS to a total of about 170 terminal and support facilities.

5 U. S. General Accounting Office, National Airspace System: Status of
FAA*s Standard Terminal Automation Replacement System, GAO- 02- 1071
(Washington, D. C.: Sept. 17, 2002).

 How reliable is FAA*s estimate of the life- cycle costs to develop and
deploy, operate and maintain, and upgrade STARS?

 What impact will STARS*s estimated cost have on future FAA budgets given
competing demands for funds to enhance aviation safety, security, and
capacity?

 What alternatives to STARS is FAA considering? In addition, you asked
whether STARS falls under the termination provisions of Public Law 104-
264, as an acquisition that is more than 50 percent over its cost goal or
behind schedule. Our analysis of the law and

its legislative history indicates that STARS is not subject to these
termination provisions. According to our analysis, the termination
provisions apply to acquisitions initiated after October 9, 1996, the date
of the law*s enactment. Because FAA approved the initial acquisition plan
for STARS in March 1996 and signed the contract with Raytheon Corporation
in September 1996, STARS is not subject to those provisions. This report
covers cost and performance issues related to FAA*s

procurement of STARS for 74 terminal and support facilities. 6 To conduct
our work, we reviewed FAA*s 5- year Capital Investment Plan, which
proposes funding for programs to modernize the national airspace system.
We also analyzed data from cost performance reports that the STARS
contractor developed for FAA. However, we did not independently verify
these cost and performance data.

Results in Brief The reliability of FAA*s life- cycle cost estimate for
STARS is uncertain. According to FAA, the costs to develop and deploy, 7
operate and maintain, and upgrade STARS will amount to about $2.54 billion
for 74 systems for

fiscal years 2004 through 2030. More specifically, the costs of completing
STARS*s development and deployment will amount to about $153 million for
fiscal years 2004 through 2008, the costs of operating and maintaining the
systems at those facilities over their useful lives will add another $1.46

6 The report does not address DOD*s efforts to deploy the equipment. 7
Throughout this report, we use the term *deploy* to denote efforts by FAA
to put STARS software, hardware, and other supporting equipment into a
facility to test it and eventually use it to control traffic.

billion for fiscal years 2004 through 2030, 8 and the costs of upgrading
STARS technology will amount to about $930 million over the same period.
FAA*s development cost estimate is based largely on the contractor*s
recent proposals and projections, which incorporate the costs of new work
specified in major modifications to the STARS contract. FAA has not yet
independently analyzed these proposals and projections as its guidance
directs and therefore does not know whether the development cost estimate
of $153 million is reliable. Furthermore, FAA has not worked with the
contractor to incorporate the requirements and costs of the new work

into cost performance reports, which it receives from the contractor.
These reports are intended to provide FAA with accurate, current data for
monitoring and overseeing the contractor*s progress and for estimating the
program*s remaining costs, but FAA is not using the reports because they
do not reflect the current status of the contract. FAA is now analyzing
the contractor*s cost data and working with the contractor to align the
cost performance reports with the current status of the contract. FAA
expects to complete these tasks in the spring of 2003 and then should be
able to use the cost performance reports as intended. The reliability of
FAA*s estimates of the life- cycle costs to operate, maintain, and upgrade
STARS technology is unknown, primarily because FAA has limited experience
with STARS

equipment and the estimates extend nearly 30 years into the future.
However, despite these uncertainties, FAA has expressed its cost estimates
as point values, rather than as ranges. As a result, the estimates may
imply

more certainty than is appropriate. Moreover, in this instance, the use of
point values limits disclosure of the program*s investment risks. We are
making recommendations to strengthen FAA*s management of STARS, and FAA
officials indicated that the recommendations were in line with the
agency*s ongoing and planned efforts. These recommendations, we believe,
will also help FAA better manage the planned modernization of terminal

facilities that are no longer included in the STARS program. According to
FAA*s latest budget planning documents, the impact of STARS*s estimated
costs on FAA*s budgets will decline for fiscal years 2004 through 2007;
for later years, however, the impact of these estimated costs is unknown.
Since FAA has nearly finished developing STARS, the program*s development
costs are expected to decrease over the next 4 fiscal years, while its
operation and maintenance costs are expected to grow with increased
deployment. According to FAA*s documents, the

8 FAA noted that the $1.46 billion would be predominantly for labor costs
that the agency would incur for STARS or a similar automation system.

funding proposed to develop and deploy STARS represents 4 percent of FAA*s
proposed $3.06 billion facilities and equipment budget account for fiscal
year 2004 and smaller percentages for fiscal years 2005 through 2007. This
account funds the development, procurement, and installation of

equipment to help increase the capacity and improve the safety,
efficiency, and security of the national airspace system. Given these
small and declining percentages for STARS, the program*s estimated costs
should, over this period, have a small and declining impact on other
aviation safety, security, and capacity efforts funded from the facilities
and equipment

account. FAA*s budget planning documents also show that as FAA deploys
STARS at more facilities, more funding will come from the agency*s
operations account, which supports training and compensation for the

controllers and technicians who operate and maintain STARS. However, the
impact of STARS*s estimated costs on FAA*s budgets for later years is
currently unknown because it is too soon in the budget cycle for FAA to
have developed detailed budgets beyond fiscal year 2007. In addition, as
we previously noted, FAA has limited operational experience for projecting
STARS*s budgetary impact, and long- range cost estimates are inherently
uncertain.

FAA is committed to deploying STARS at the 74 terminal and support
facilities included in the STARS program, but for the nearly 100 other
facilities that remain to be modernized, the agency could deploy STARS,
another contractor*s hardware and software, or a combination of STARS and
the other contractor*s technologies that are currently being used in FAA
terminal facilities. FAA knows that each contractor*s technology works
independently, and FAA is assessing the feasibility of operating
Raytheon*s STARS display system with the processing system from Lockheed
Martin Corporation*s Common Automated Radar Terminal System, which was
recently installed at some terminal facilities. Combining the two
technologies could be cost- effective because it would allow FAA to use
both the customized display system that accounted for a substantial
portion of STARS*s development costs and the recently acquired processing
system. FAA plans to announce the results of this assessment in April
2003.

Background STARS will replace controller workstations with new color
displays, processors, and computer software at FAA and Department of
Defense

terminal air traffic control facilities. (See fig. 1.) FAA*s goal for
STARS is to provide an open, expandable terminal automation platform that
can accommodate future air traffic growth and allow for the introduction
of new hardware- and software- based tools to promote safety, maximize

operational efficiency, and improve controllers* productivity. FAA
believes that STARS will facilitate efforts to optimally configure the
terminal airspace around the country, exchange digital information between
pilots and controllers, and introduce new position and surveillance
capabilities for pilots.

Figure 1: Standard Terminal Automation Replacement System (STARS)

Source: FAA.

FAA has given high priority to STARS. Both the past and the current FAA
Administrator have emphasized the program*s importance to enhancing the
capacity of the national airspace system. In addition, STARS is 1 of 19
programs on FAA*s list of top programs. Recently, FAA gave priority
consideration to STARS so that it could meet its commitment to the
Congress to deploy STARS at the Philadelphia terminal, the first busy
facility, on November 17, 2002. Furthermore, according to FAA officials,
the

agency remains committed to funding STARS until completion, even if that
means postponing the funding for other programs. Agency officials have
indicated, for example, that if the fiscal year 2004 budget falls below
the level shown in FAA*s planning documents, funds from other programs

would likely be cut to fully fund STARS. For each acquisition program that
FAA undertakes, it officially estimates the program*s life- cycle costs
(from development and deployment through operations and maintenance),
schedule, benefits, and performance in a formal budget document called the
acquisition program baseline. FAA uses

this document* which its acquisition decision- making body, the Joint
Resources Council, must approve* to decide whether to fund the program
and, if it is funded, to monitor its progress. FAA also uses the approved
acquisition program baseline to develop a 5- year budget- planning
document, called the Capital Investment Plan. Program managers rely on

the acquisition program baseline to oversee the program*s progress and to
ensure that no action is taken that would breach the approved baseline. To
support their oversight, program managers typically require contractors to
deliver cost performance reports, each of which includes a performance
measurement baseline for assessing the contractor*s progress in meeting
the contract*s cost, schedule, and technical performance goals. For STARS,
FAA has one contract with Raytheon, which accounts for 82 percent of the
funding approved in the acquisition program baseline. Most of the
remaining funding is used for contracts that support FAA*s internal
program management.

For guidance in managing its major acquisition programs, FAA relies
largely on two documents* its own Acquisition System Toolset, a *one- stop
acquisition information system* on FAA*s Web site that contains the
agency*s official acquisition policy and guidance, and the Defense
Contract Management Agency*s (DCMA) guide on implementing earned value
management. 9 The Acquisition System Toolset includes policy, guidance,

instructions, examples, best practices, lessons learned, references, and
other related information tailored to each type of procurement contract.
The earned value management guide provides information on how to use cost
and performance measurement to manage acquisition programs.

9 Defense Contract Management Agency, Earned Value Management
Implementation Guide (October 1997).

FAA funds STARS primarily through two of its budget accounts: (1)
facilities and equipment and (2) operations. The facilities and equipment
account covers the costs to develop, procure, and place the new equipment
in operation. 10 After the equipment has been fully operating for at least
a year, the funding source shifts to the operations account, which covers
the costs to support and maintain the equipment over its life cycle.
Operations costs are, in large part, personnel costs* for the controllers
who operate

and the technicians who maintain the equipment. Planned product
improvements and technology upgrades are primarily funded from the
facilities and equipment account.

Since 1996, when FAA initiated STARS, it has spent approximately $1. 2
billion, or about 86 percent of the funding budgeted for the program, and
it has twice approved major changes to the program's cost and schedule
estimates. First, in October 1999, FAA modified its acquisition approach
(from off- the- shelf software only to a combination of customized and
offthe-

shelf software) and increased to 188 the number of facilities scheduled to
receive STARS. We reported on these changes in September 2002. 11 At that
time, FAA also concluded that it did not have adequate funding to deploy
STARS to all 188 facilities with the remainder of the STARS funding.
Instead, FAA decided to deploy STARS to 74 terminal and support
facilities. The selected facilities had frequent equipment failures, were
new, or had the digital radar needed to operate STARS. FAA has since
reduced the total number of facilities to about 170. The agency is
currently studying options for modernizing display systems at the nearly
100 remaining facilities and is identifying the additional costs of
upgrading the other facilities with STARS or an alternative system. The
STARS program office stated that these additional costs, together with the
funds already committed to STARS, would more than likely exceed the $1.4
billion originally planned. The

program office is in the process of developing options and estimates for
these additional sites.

10 This account funds security activities related mainly to the security
of FAA facilities and equipment. Funding for some airport security comes
from another FAA account* Grants- inAid for Airports.

11 GAO- 02- 1071.

Reliability of FAA*s FAA has estimated the life- cycle cost for STARS,
including the costs to

Life- Cycle Cost develop and deploy systems to 74 terminal and support
facilities and to operate, maintain, and upgrade the systems. However, the
reliability of

Estimate for STARS Is these cost estimates is uncertain. If FAA*s
estimates are not reliable, both

Uncertain the agency and the Congress will be limited in their ability to
project and

compare the costs and benefits of completing STARS and modernizing other
facilities as well as in their ability to budget realistically for other
capital investments.

FAA Has Estimated STARS*s According to FAA*s estimates, the agency will
need about $2.54 billion for

Remaining Life- Cycle Costs STARS over the remaining life of the program.
As indicated in figure 2, FAA

will need about (1) $153 million for fiscal years 2004 through 2008 to
complete the development and deployment of 74 systems; (2) $1.46 billion
for fiscal years 2004 through 2030* or about $54 million a year, on
average* to operate and maintain the systems; 12 and (3) $930 million to
upgrade STARS technology over the same period. FAA*s operating cost

estimate assumes that STARS*s operating costs will grow with deployment
through about fiscal year 2007 and will subsequently keep pace with
inflation, at least through fiscal year 2009. FAA expects the costs of
technology upgrades to more than triple between fiscal years 2007 and 2008
because of a technology update cycle, which will peak when the hardware

and software are due for replacement. This projection assumes that STARS
software will be upgraded every 3 years.

12 As previously noted, FAA officials said the $1.46 billion would be
labor costs that the agency would incur for STARS or a similar automation
system.

Figure 2: Life- Cycle Cost Estimate for STARS, Fiscal Years 2004- 2030
Dollars in millions

Fiscal year Cost element Appropriation 2004 2005

2006 2007 2008 2009 2010 * 2030 Total

Development Facilities and $39.5 $25.5 $12.5 $2.0 $1.1 $0.0 $0.0 $80.6

equipment Deployment Facilities and 54.6 18.0 0.0 0.0 0.0 0.0 0.0 72.6
equipment Subtotal $94.1 $43.5 $12.5 $2.0 $1.1 $0.0 $0.0 $153.2

Technology Facilities and

$19.9 $13.6 $12.6 $16.5 $58.8 $65.7 $742.5 $929.5 upgrades equipment
Operations and

Operations 22.3 41.9 47.3 51.1 53.9 55.7 1,187.5 1,459.7 maintenance Total
$136.3 $99.0 $72.4 $69.6 $113.8 $121.4 $1,930.0 $2,542.4

Source: FAA. Note: These data are from FAA*s March 2002 acquisition
program baseline for STARS and have been adjusted to reflect the effects
of inflation. Numbers may not add because of rounding.

FAA does not yet know to what extent its estimate of STARS*s remaining
development costs is reliable. The estimate is derived from the
contractor*s proposals and projections, which reflect the costs of major
modifications

to the contract made since May 2000. However, FAA has not yet
independently analyzed and validated the proposals and projections and
therefore cannot assess the reliability of the development cost estimate.

FAA obtains monthly cost performance reports from the contractor, which
the agency should be able to use both to oversee the contractor*s
performance and to estimate the program*s remaining development costs.

However, FAA does not use these reports because they are not current. More
specifically, their central component, the performance management
baseline* which establishes performance, cost, and schedule milestones for
the contract* has not been updated since May 2000 and therefore does not
incorporate the effects of major contract modifications approved since
that date. For example, the September 2002 cost performance report does
not reflect FAA*s March 2002 reduction in the scope of STARS from 188 to
74 systems, and the report does not include the costs of new work that FAA

authorized between May 2000 and September 2002. Consequently, the report
indicates that STARS is on schedule and within 1 percent of budget, even
though, compared with the program envisioned in May 2000, FAA is now under
contract to modernize fewer than half as many facilities at more than
twice the cost per facility. Cost Performance Reports

The cost performance reports for STARS do not meet the criteria for such
Do Not Meet FAA*s Criteria

reports established in the guidance for managing major acquisitions that
FAA has adopted from DCMA. 13 According to this guidance, cost performance
reports are valid only when

 a reliable performance measurement baseline is established and
maintained (i. e., regularly updated and validated),

 changes to the baseline are carefully controlled (i. e., negotiated and
approved before being authorized),

 an integrated baseline review 14 takes place within 6 months of a
contract*s award or significant modification, and

 contract oversight occurs regularly. FAA and the contractor have
established a performance measurement baseline for the STARS contract, but
they have not satisfied the remaining criteria. The baseline has not been
updated since May 2000, and FAA has not validated it. For example, FAA has
not asked an independent organization, such as the Defense Contract Audit
Agency, to verify the costs in the contractor*s cost performance reports
by tracing the costs back to the contractor*s accounting system. In
addition, FAA has not controlled changes to the baseline because, between
May 2000 and September 2002, it

approved up to $179 million in authorized, unpriced work* that is,
additional work that FAA agreed to let the contractor perform without
first negotiating or independently verifying the costs. Furthermore, the
additional tasks and costs have not been incorporated in the baseline,
even

13 Earned Value Management Guide (October 1997). 14 The purpose of this
review, conducted jointly by the agency and the contractor, is to
understand and assess the adequacy, accuracy, and risks of a performance
measurement baseline with respect to the contract*s work scope, schedule,
technical requirements, and resource availability.

though the DCMA guidance calls for processing contract modifications
expeditiously and incorporating them in a timely manner to maintain the
baseline*s integrity. Although the DCMA guidance does not define *a timely
manner,* experts generally agree that this term means no longer than 3
months. FAA has not maintained and controlled the baseline because,
according to program officials, it has been *schedule driven** committed
to deploying STARS at the Philadelphia terminal by November 17, 2002. FAA
is currently analyzing the contractor*s cost data and working with the
contractor to incorporate modifications in the performance measurement
baseline.

Although the DCMA guidance calls for performing an integrated baseline
review within 6 months of awarding or significantly modifying a contract,
FAA has not performed such a review of STARS since August 2000, even
though it has subsequently made two major modifications to the contract.
An integrated baseline review is important to ensure that the contract*s
cost data are aligned with the current status of the program after a major

contract modification. According to the Manager of Terminal Automation,
who oversees STARS, FAA had planned to initiate an integrated baseline
review of the STARS contract in November 2002 and expects to begin this
effort as soon as funding is available.

To provide regular contract oversight as the DCMA guidance requires, FAA
and Raytheon meet monthly to, among other things, discuss the cost
performance reports. However, discussions of these reports do not, in our
view, constitute regular contract oversight because, without a current,
valid performance measurement baseline, FAA cannot compare what the
contractor has done with what the contractor agreed to do in the contract.
With a current, valid baseline, the reports would indicate when cost or
schedule thresholds had been exceeded, and FAA could then require the
contractor to explain the reasons for the variances and to identify and
take appropriate corrective actions. But because the baseline has not been
maintained and is not aligned with the current status of the program, the
reports are not useful for evaluating the contractor*s performance or for
projecting the contract*s remaining costs. The Manager of Terminal
Automation agreed that the current cost performance reports are not useful
for these purposes and said that the agency therefore uses the monthly
meetings with the contractor to discuss other program control and
financial issues.

FAA*s Cost Estimates Do FAA has limited information for determining the
reliability of the costs it

Not Reflect Levels of Risk has estimated to operate and maintain STARS and
to upgrade its

technology. The agency first projected the costs of operating and
maintaining STARS in March 2002 and said that if these costs were not
funded, essential STARS maintenance would not be completed. However, FAA
has just begun to operate STARS and has limited experience for projecting
future operating and maintenance costs. In addition, FAA did not perform a
risk assessment, as its Acquisition System Toolset guidance specifies, to
identify the minimum, most likely, and maximum expected costs for the
entire program. Although FAA performed such an assessment for one
facility, it did not extend this effort* again because of time pressures,
according to FAA officials. Because FAA has just begun to operate STARS,
it also has limited experience for projecting the costs of technology
upgrades over the life of the program. Additionally, both the operating
and the technology upgrade cost estimates are uncertain because they
extend many years into the future.

Despite the uncertainty inherent in estimates* especially long- range
ones* FAA has expressed its cost estimates for STARS as point values. As
we reported in 1997, 15 point values imply certainty and therefore are not
suitable for expressing estimates. Instead, ranges or numbers with
confidence levels would be more appropriate. For instance, a cost estimate
of $1 million could be presented either as a range of $900, 000 to $1. 1
million or as $1 million with a confidence interval of 90 percent,
indicating that there is a 10 percent chance that the cost will exceed the
estimate. 16 Because FAA did not perform a risk assessment for the entire
STARS

program, it did not develop ranges or confidence levels that it could use
to express its cost estimates for STARS. And because FAA used point values
instead, it limited its disclosure of the program*s investment risks.

The reliability of FAA*s development and operating cost estimates is
important not only for managing STARS effectively but also for planning
appropriately for other terminal modernization efforts and other FAA
capital investments. Without reliable estimates for comparing STARS*s

15 U. S. General Accounting Office, Air Traffic Control: Improved Cost
Information Needed to Make Billion Dollar Modernization Investment
Decisions, GAO/ AIMD- 97- 20 (Washington, D. C.: Jan. 22, 1997). 16 The 90
percent confidence level is based on an analysis of the estimate*s
uncertainty made by using a Monte Carlo model.

costs and benefits and for anticipating future expenditures, FAA and the
Congress cannot make informed decisions about how best to modernize the
nearly 100 facilities that are not currently scheduled to receive STARS.
Furthermore, because FAA has given high priority to completing STARS and
has said that it will fund the program even if it has to postpone the
funding for other programs, FAA*s estimates for STARS will influence the
agency*s plans for funding other capital investments.

Impact of STARS*s According to our analysis of budgetary data for fiscal
years 2004 through Estimated Costs on

2007 that we obtained from FAA, the estimated costs of STARS should have a
declining impact on FAA*s budgets during these years because the FAA*s
Budgets Is

program*s development phase is nearly over and its operations will still
be Expected to Decline

limited. For later years, the impact of these estimated costs on FAA*s
through Fiscal Year

budgets is unknown because it is too soon in the budget cycle for FAA to
have developed detailed budgets beyond fiscal year 2007. 2007

Because FAA has nearly completed the development of STARS and has begun to
install the equipment, the agency is budgeting less from the facilities
and equipment account for STARS. Therefore, the proposed funding for STARS
does not have a significant impact on the funding for other efforts to
improve the safety, security, and capacity of the national airspace
system. For fiscal year 2004, the $94.1 million planned to develop and
deploy STARS represents 3 percent of the total funding for facilities and
equipment proposed in FAA*s Capital Investment Plan. When the $19.9
million planned for technology upgrades is included in the projections for
fiscal year 2004, the estimated cost for STARS is about 4 percent of FAA*s
planned total facilities and equipment budget (see fig. 3).

Figure 3: Allocation of Facilities and Equipment Funding for Fiscal Year
2004 in FAA*s Capital Investment Plan FY04 Investments at $3.062 Billion
Level

Efficiency

3% Security

4%

STARS Safety

9% 29%

Capacity

11%

Mission support

16% 13% 15%

Pay and benefits Reliability Proposed funding for other safety, security,
and capacity efforts. Proposed funding for STARS.

Sources: GAO and FAA. Notes: GAO analysis of data from FAA. Beginning with
the fiscal year 2003 budget, FAA showed its facilities and equipment
budget in broad performance (mission) outcome areas. Individual programs
support these broad performance areas. For example, STARS supports several
mission areas, such as safety, security, and capacity. To illustrate the
impact of STARS on the other broad mission areas, this figure consolidates
the allocation

for STARS in one percentage and compares this percentage with the
allocations to the broad mission areas.

According to FAA*s Capital Investment Plan, the proposed funding for FAA*s
total facilities and equipment budget will increase slightly during fiscal
years 2005 through 2007, while the proposed funding for STARS*s
development and deployment will continue to decline (see table 1).

Therefore, the impact of these estimated STARS costs on the facilities and
equipment budget for safety, security, and capacity would be even less
during these fiscal years. The estimated costs of technology upgrades, if

included in the facilities and equipment cost estimates for STARS for
these 3 fiscal years, would likewise have a similarly small impact on the
facilities and equipment budget for these fiscal years.

Table 1: Proposed Funding for FAA*s Total Facilities and Equipment Budget
and for STARS*s Development and Deployment

Dollars in millions

Proposed funding Fiscal year 2005 Fiscal year 2006 Fiscal year 2007

Total facilities and equipment budget $3, 129 $3, 202 $3, 277

STARS development and deployment 43.5 12.5 2. 0

STARS technology upgrades 13.6 12.6 16.5 Source: FAA. Note: These data are
from the Capital Investment Plan that FAA submitted with its fiscal year
2003 budget request and FAA*s March 2002 acquisition program baseline for
STARS.

As FAA begins to operate STARS equipment at more terminal facilities, the
primary funding source for the program will shift from the facilities and
equipment budget account to the operations budget account. Although FAA
has projected that it will increasingly need funding from (1) the
operations account to cover the costs of operating and maintaining STARS
and (2) the

facilities and equipment account to upgrade STARS technology, it has not
yet developed detailed estimates of the program*s operations and
maintenance and technology upgrade costs beyond fiscal year 2007. FAA
officials noted that FAA, like most federal agencies, develops its budget
in 5- year plans and has not yet begun to develop detailed budgets beyond
fiscal year 2007. Consequently, FAA does not currently know what impact

the estimated costs of operating, maintaining, and upgrading STARS will
have on the agency*s future budgets. Furthermore, as we previously noted,
FAA has limited operational experience with STARS to use in estimating and
budgeting for the costs of the program*s operations, maintenance, and
technology upgrades. Finally, the uncertainty of long- range estimates
makes it difficult to determine the impact of STARS*s estimated costs on
FAA*s budgets for outlying fiscal years.

After Deploying STARS FAA is committed to deploying STARS at the 74
terminal and support

at 74 Facilities, FAA facilities included in the STARS program, but for
the nearly 100 other

facilities that remain to be modernized, the agency has at least three May
Modify Its

options: It could deploy Raytheon*s STARS, it could procure Lockheed
Approach for the Martin Corporation*s Common Automated Radar Terminal
System

Remaining Facilities (Common ARTS) technology, or it could combine
Raytheon*s STARS display system with Lockheed Martin*s Common ARTS
processing system. Between 1997 and 1999, FAA deployed Common ARTS to 131
small to

medium- sized facilities and to 5 larger facilities. From its experience
with both systems, FAA knows that each works independently. However, by
combining the two technologies, FAA could both (1) take advantage of the

customized software developed to resolve complex computer- human
(controller and technician) interface issues that accounted for a
significant portion of STARS*s development costs and (2) continue to use
portions of the Common ARTS equipment that it recently deployed.

In October 2002, FAA asked Raytheon to consider the feasibility of merging
the STARS display system with the Common ARTS processing system. According
to the Manager of Terminal Automation, Raytheon finished negotiating a
subcontract with Lockheed Martin in January 2003, and FAA

anticipates that the two contractors will now begin analyzing the
feasibility of a merger of components of the two systems. After FAA
receives the results of this analysis, which it expects in April 2003, it
plans to determine the cost and schedule for modernizing the remaining
terminal facilities.

Conclusions FAA lacks accurate, valid, current data on the STARS program*s
costs and progress. Without such data, FAA is limited in its ability to
effectively

oversee the contractor*s performance and reliably estimate future costs.
FAA cannot use the contractor*s cost performance reports for these
purposes until the contract*s performance measurement baseline has been
revised to incorporate the results of contract modifications, and FAA has
verified that the revised baseline is aligned with the current status of
the contract. Furthermore, without performing a risk assessment to
identify the program*s minimum, most likely, and maximum expected costs,
FAA cannot reliably determine the level of uncertainty inherent in its
cost estimates. Finally, by using point values, rather than ranges or
other appropriate measures, to express its cost estimates, FAA is not
revealing the extent of their uncertainty. Its current estimates,
expressed as point values, are misleading because they convey undue
certainty and limit

disclosure of the program*s investment risks. As such, the estimates*
usefulness to program managers and congressional overseers is limited.

The earned value management guidance that FAA has adopted from DCMA and
the Acquisition System Toolset that it compiled for itself establish clear
procurement management policies and procedures that are applicable to
STARS and to subsequent terminal modernization programs. In light of FAA*s
incomplete or inconsistent application of this guidance to the STARS
program thus far and the resulting cost overruns and schedule delays, we
believe it is essential that the agency revisit the guidance, not only
when it updates its baseline and performs an integrated baseline review
this spring,

but also throughout the remainder of STARS and throughout subsequent
terminal modernization programs.

Recommendations for To improve FAA*s management of STARS and of subsequent
terminal

Executive Action modernization programs and to provide the Congress with
more reliable

information for overseeing these programs, we recommend that the Secretary
of Transportation direct the FAA Administrator to follow the agency*s
guidance for managing major acquisition systems by

 establishing, maintaining, and controlling an accurate, valid, and
current performance measurement baseline, which would include negotiating
all authorized, unpriced work within 3 months;

 conducting an integrated baseline review of any major contract
modifications within 6 months; and

 preparing a rigorous life- cycle cost estimate, including a risk
assessment, in accordance with the Acquisition System Toolset*s guidance
and identifying the level of uncertainty inherent in the estimate.

Agency Comments We requested comments on a draft of this report from the
Secretary of Transportation or his designee. On January 23, 2003, FAA
officials,

including the Manager of Terminal Automation, provided us with the
following oral comments on the draft. FAA agreed with the general tone and
intent of the draft report as well as with our conclusions and
recommendations. FAA noted that, in focusing its time and energy on
meeting the technical performance and schedule requirements of STARS, it

spent less time and energy on its business management of the program.
However, the manager noted, the agency is now taking steps to reduce the
program*s cost risks, such as defining and negotiating with the prime
contractor all of the contract work planned for fiscal years 2003 through
2005. FAA also plans to conduct an integrated baseline review when it
receives its appropriation for fiscal year 2003. According to FAA, our
recommendations will further strengthen FAA*s commitment to improve the
business management of the STARS program. FAA provided additional
clarifying and technical information, which we incorporated as

appropriate. Scope and

To conduct our work, we reviewed FAA*s 5- year Capital Investment Plan,
Methodology

which proposes funding for programs to modernize the national airspace
system. We also analyzed data from cost performance reports that the STARS
contractor developed for FAA. We interviewed FAA officials responsible for
air traffic control modernization planning and budgeting. We also reviewed
data from the FAA STARS program, contracting officials, and the Department
of Transportation*s Office of the Inspector General. We did not
independently verify the cost and performance data we received

from FAA. We performed our work from October 2002 through January 2003 in
accordance with generally accepted government auditing standards.

As arranged with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 5 days after
the date of this letter. At that time, we will send copies to interested
Members of Congress, the Secretary of Transportation, and the FAA
Administrator. We will also make copies available to others on request. In
addition, the

report will be available at no charge on the GAO Web site at http:// www.
gao. gov.

Should you or your staff have questions on matters discussed in this
report, please contact Gerald L. Dillingham at (202) 512- 2834 or Keith A.
Rhodes at (202) 512- 6412. We can also be reached by E- mail at
dillinghamg@ gao. gov and rhodesk@ gao. gov, respectively. GAO contacts
and key contributors to

this report are listed in appendix I. Sincerely yours,

Gerald L. Dillingham, Ph. D. Director, Physical Infrastructure Issues

Keith A. Rhodes Chief Technologist, Applied Research and

Methods

Appendi Appendi xes x I

GAO Contacts and Staff Acknowledgments GAO Contacts Belva Martin (202)
512- 2834 Madhav Panwar (202) 512- 6228 Staff

In addition to those individuals named above, Yvette Banks, Geraldine
Acknowledgments

Beard, Jennifer Echard, Elizabeth Eisenstadt, Elizabeth Marchak, and Karen
Richey made key contributions to this report.

(540041)

a

GAO United States General Accounting Office

The reliability of FAA*s life- cycle cost estimate for STARS is uncertain.
This estimate includes estimates of the program*s development, operation,
and technology upgrade costs, shown for the next 6 fiscal years in the
figure below. The development cost estimate is based on the contractor*s
projections, which FAA has not yet independently analyzed as its guidance
directs. Furthermore, baseline data in cost performance reports that FAA

obtains from the contractor are not accurate because the data do not
reflect the current status of the contract. As a result, FAA is limited in
its ability to manage the contract effectively. FAA plans to address these
problems. In addition, the program*s operation and technology upgrade cost
estimates are based on limited experience with STARS and extend many years
into the future. However, the estimates do not reflect these
uncertainties.

Estimated STARS Funding, Fiscal Years 2004- 2009 0 10

20 30

40 50

60 70

80 90

100 2004 Cost element (Dollars in millions)

Sources: GAO and FAA.

Fiscal year

Development/ Deployment Operations/ Maintenance Technology upgrades

2005 2006 2007 2008 2009

For fiscal years 2004 through 2007, the years for which FAA provided
budget information, STARS*s estimated costs should have a declining impact
on FAA*s budgets because the program*s development is nearly over and its
operations are still limited. For fiscal years 2008 through 2030, the
impact of STARS*s estimated costs on FAA*s budgets is unknown because it
is still too soon in the budget cycle for FAA to have developed detailed
budgets for these years.

After deploying STARS at the 74 terminal and support facilities included
in the program, FAA could use STARS, another contractor*s technology, or a
combination of the two technologies for the nearly 100 remaining
facilities. FAA is exploring the feasibility of combining the technologies
and expects to announce its plans in April 2003.

NATIONAL AIRSPACE SYSTEM

Better Cost Data Could Improve FAA*s Management of the Standard Terminal
Automation Replacement System

www. gao. gov/ cgi- bin/ getrpt? GAO- 03- 343. To view the full report,
including the scope and methodology, click on the link above. For more
information, contact Gerald L. Dillingham at (202) 512- 5555 or Keith A.
Rhodes at (202) 512- 6412. Highlights of GAO- 03- 343, a report to the

Honorable Ellen O. Tauscher, House of Representatives

January 2003

To enhance the capacity and safety of the national airspace system, the
Federal Aviation Administration (FAA), within the Department of
Transportation, is acquiring 74 Standard Terminal Automation

Replacement Systems (STARS). STARS will replace some outdated air traffic
control equipment. Since

1996, when FAA initiated this major computer hardware and software
acquisition, the scope and estimated costs of STARS have changed many
times. FAA now

estimates that STARS*s remaining costs will total about $2. 54 billion.
GAO was asked to assess the reliability of FAA*s life- cycle cost estimate
for STARS, determine the impact of STARS*s estimated costs on future FAA
budgets, and identify any alternatives to STARS that FAA is considering.
GAO based its

analysis on published FAA cost data and the guidance FAA uses for managing
major acquisitions. To improve FAA*s management of STARS and of subsequent
terminal modernization programs and to provide the Congress with more
reliable information for oversight, GAO recommends that the

Secretary of Transportation direct the FAA Administrator to maintain
accurate, current baseline data; review baseline data within 6 months of
any major modification

to ensure that the data reflect the current status of the contract; and
prepare a rigorous life- cycle cost estimate that identifies the level of
uncertainty.

Page i GAO- 03- 343 National Airspace System

Contents

Contents

Page ii GAO- 03- 343 National Airspace System

Page 1 GAO- 03- 343 National Airspace System United States General
Accounting Office

Washington, D. C. 20548 Page 1 GAO- 03- 343 National Airspace System

A

Page 2 GAO- 03- 343 National Airspace System

Page 3 GAO- 03- 343 National Airspace System

Page 4 GAO- 03- 343 National Airspace System

Page 5 GAO- 03- 343 National Airspace System

Page 6 GAO- 03- 343 National Airspace System

Page 7 GAO- 03- 343 National Airspace System

Page 8 GAO- 03- 343 National Airspace System

Page 9 GAO- 03- 343 National Airspace System

Page 10 GAO- 03- 343 National Airspace System

Page 11 GAO- 03- 343 National Airspace System

Page 12 GAO- 03- 343 National Airspace System

Page 13 GAO- 03- 343 National Airspace System

Page 14 GAO- 03- 343 National Airspace System

Page 15 GAO- 03- 343 National Airspace System

Page 16 GAO- 03- 343 National Airspace System

Page 17 GAO- 03- 343 National Airspace System

Page 18 GAO- 03- 343 National Airspace System

Page 19 GAO- 03- 343 National Airspace System

Page 20 GAO- 03- 343 National Airspace System

Appendix I

The General Accounting Office, the audit, evaluation and investigative arm
of Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability of
the federal government for the American people. GAO examines the use of
public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO*s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO*s Web site (www. gao. gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as *Today*s Reports,* on its
Web site daily. The list contains links to the full- text document files.
To have GAO e- mail

this list to you every afternoon, go to www. gao. gov and select
*Subscribe to daily E- mail alert for newly released products* under the
GAO Reports heading.

The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to: U. S. General Accounting Office 441 G Street NW, Room LM
Washington, D. C. 20548 To order by Phone: Voice: (202) 512- 6000

TDD: (202) 512- 2537 Fax: (202) 512- 6061

Contact: Web site: www. gao. gov/ fraudnet/ fraudnet. htm E- mail:
fraudnet@ gao. gov Automated answering system: (800) 424- 5454 or (202)
512- 7470 Jeff Nelligan, managing director, NelliganJ@ gao. gov (202) 512-
4800

U. S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.
C. 20548 GAO*s Mission Obtaining Copies of

GAO Reports and Testimony

Order by Mail or Phone To Report Fraud, Waste, and Abuse in Federal
Programs Public Affairs

United States General Accounting Office Washington, D. C. 20548- 0001
Official Business Penalty for Private Use $300 Address Service Requested

Presorted Standard Postage & Fees Paid

GAO Permit No. GI00
*** End of document. ***