Air Traffic Control: FAA Reports Progress in System Acquisitions,
but Changes in Performance Measurement Could Improve Usefulness  
of Information (18-DEC-07, GAO-08-42).				 
                                                                 
Acquiring new systems on budget and on schedule is critically	 
important in transitioning to the Next Generation Air		 
Transportation System (NextGen). However, air traffic control	 
modernization has been on GAO's high-risk list since 1995, in	 
part due to acquisitions exceeding budget and schedule targets.  
The Federal Aviation Administration's (FAA) Air Traffic 	 
Organization (ATO) has responsibility for managing air traffic	 
control acquisitions. GAO was asked to examine (1) ATO's goals,  
performance measures, and reporting for systems acquisitions; (2)
the validity of ATO's performance measures; and (3) the 	 
implications of using ATO's performance measures to assess	 
progress in transitioning to NextGen. To address these issues,	 
GAO compared ATO's measures with attributes of successful	 
performance measures, interviewed agency officials, and sought	 
perspectives of aviation experts.				 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-08-42						        
    ACCNO:   A79028						        
  TITLE:     Air Traffic Control: FAA Reports Progress in System      
Acquisitions, but Changes in Performance Measurement Could	 
Improve Usefulness of Information				 
     DATE:   12/18/2007 
  SUBJECT:   Air traffic control systems			 
	     Aviation						 
	     Budget administration				 
	     Budget controllability				 
	     Budgeting						 
	     Commercial aviation				 
	     Cost control					 
	     Evaluation criteria				 
	     Performance measures				 
	     Program evaluation 				 
	     Reporting requirements				 
	     Risk assessment					 
	     Risk management					 
	     Schedule slippages 				 
	     Strategic planning 				 
	     Civil aviation					 
	     Program goals or objectives			 
	     Program implementation				 
	     GAO High Risk Series				 
	     Next Generation Air Transportation 		 
	     System						 
                                                                 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-08-42

   

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Requesters: 

United States Government Accountability Office: 

GAO: 

December 2007: 

Air Traffic Control: 

FAA Reports Progress in System Acquisitions, but Changes in Performance 
Measurement Could Improve Usefulness of Information: 

GAO-08-42: 

GAO Highlights: 

Highlights of GAO-08-42, a report to congressional requesters. 

Why GAO Did This Study: 

Acquiring new systems on budget and on schedule is critically important 
in transitioning to the Next Generation Air Transportation System 
(NextGen). However, air traffic control modernization has been on GAOï¿½s 
high-risk list since 1995, in part due to acquisitions exceeding budget 
and schedule targets. The Federal Aviation Administrationï¿½s (FAA) Air 
Traffic Organization (ATO) has responsibility for managing air traffic 
control acquisitions. GAO was asked to examine (1) ATOï¿½s goals, 
performance measures, and reporting for systems acquisitions; (2) the 
validity of ATOï¿½s performance measures; and (3) the implications of 
using ATOï¿½s performance measures to assess progress in transitioning to 
NextGen. To address these issues, GAO compared ATOï¿½s measures with 
attributes of successful performance measures, interviewed agency 
officials, and sought perspectives of aviation experts. 

What GAO Found: 

To be consistent with federal guidance and with targets set in the 
Department of Transportationï¿½s strategic plan, ATO established annual 
acquisition goals and performance measures that call for a high 
percentage of its major acquisitions to be within 10 percent of budget 
and on schedule. ATO identifies major acquisitions and reports 
performance against its goals using its most recently approved budget 
and schedule estimates. To measure on-budget performance, ATO 
calculates budget increases over an 8-month periodï¿½between January and 
August of each year. To measure on-schedule performance, ATO selects a 
minimum of two annual milestones from its major acquisitions and 
calculates the percentage of milestones that are on schedule. 

Because ATOï¿½s acquisition performance measures lack objectivity, 
reliability, coverage of core activities, and clarity, and focus only 
on the preceding year, they may not provide a valid assessment of 
performance over time. On the positive side, the measures are aligned 
with FAAï¿½s strategic objectives, are measurable, have no overlap, and 
address governmentwide priorities. However, the performance measures 
lack objectivity because ATO has no objective criteria for designating 
which programs are ï¿½majorï¿½ and should be selected for performance 
reporting. This makes it possible for subjective considerations to 
dominate the outcome and leaves the performance measures vulnerable to 
bias in the selection of programs for reporting. The lack of objective 
criteria for designating major programs also impairs the reliability of 
the measures (the ability of the measures to produce the same results 
each time they are applied under similar conditions) and undermines 
assurance that ATO managers include all core program activities in 
performance reporting each year. The performance measures also lack 
clarity in that they do not indicate that ATO measures the performance 
of many acquisitions against the most recently approved budget and 
schedule estimates rather than the original estimates. 

ATOï¿½s acquisition performance measurement and reporting could mask 
budget increases and schedule delays that could have a negative effect 
on the transition to NextGen. Although ATO reported performance that 
exceeded its goals for fiscal years 2004 through 2006 and showed nearly 
steady improvement, when measured against original baselines, 
acquisition performance improved but was lower than reported. Going 
forward, the absence of original budget and schedule information on 
ATOï¿½s acquisitions could give the impression that the transition to 
NextGen is progressing more smoothly than might actually be the case. 
It will be important for ATO and Congress to recognize budget increases 
and schedule delays so that the capacity, efficiency, and safety 
benefits of NextGen can be realized in a cost-efficient and timely 
fashion. 

What GAO Recommends: 

GAO recommends that FAA establish objective criteria for selecting 
acquisitions for performance reporting, and improve performance 
reporting by discussing original budgets and schedules and the 
implications of acquisition performance on NextGen. FAA officials said 
they generally concur with our recommendations and offered technical 
corrections which we incorporated, as appropriate. 

To view the full product, including the scope and methodology, click on 
[hyperlink, http://www.GAO-08-42]. For more information, contact Gerald 
L. Dillingham, (202) 512-2834, [email protected]. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

ATO Established Annual Performance Measures to Be Consistent with 
Federal Guidance and Selects Major Programs for Measuring and 
Reporting: 

ATO's Acquisition Performance Measures' Lack of Certain Successful 
Attributes and 1-Year Focus Impairs Validity of Performance Reporting: 

ATO's Performance Measurement Could Mask Budget Increases and Delays in 
the Transition to NextGen: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Baseline History for Programs Selected for Acquisition 
Performance Measurement: 

Appendix III: Performance Measures from the FAA's Fiscal Year 2006 
Performance and Accountability Report: 

Appendix IV: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Comparison of Key Attributes of Successful Performance 
Measures with ATO's Acquisition Performance Measures: 

Table 2: Annual Performance Targets, Performance of ATO's Selected 
Major Acquisitions Measured against Annual Targets and Original 
Baselines, Fiscal Years 2003-2012: 

Table 3: Budget and Schedule Baseline History for Baselined Programs 
Selected for Acquisition Performance Measurement for 1 or More Years, 
Fiscal Years 2003 through 2006: 

Figures: 

Figure 1: Breakout of ATO's Funded CIP Programs for Fiscal Year 2006: 

Figure 2: Types of Acquisition Programs Selected for Performance 
Reporting, Fiscal Year 2006: 

Figure 3: The Transition to NextGen: 

Abbreviations: 

ASR-11: Airport Surveillance Radar - Model 11: 

ATC: air traffic control: 

ATO: Air Traffic Organization: 

CIP: Capital Investment Plan: 

FAA: Federal Aviation Administration: 

ITWS: Integrated Terminal Weather System: 

JRC: Joint Resources Council: 

OMB: Office of Management and Budget: 

NAS: National Airspace System: 

NextGen: Next Generation Air Transportation System: 

STARS: Standard Terminal Automation Replacement System: 

WAAS: Wide Area Augmentation System: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

December 18, 2007: 

Congressional Requesters: 

Between fiscal year 2007 and fiscal year 2011, the Federal Aviation 
Administration (FAA) plans to acquire more than $14 billion worth of 
new systems to continue operating the nation's current air traffic 
control (ATC) system while simultaneously transitioning to the Next 
Generation Air Transportation System (NextGen). The transition to 
NextGen involves acquiring numerous systems to support precision 
satellite navigation; digital, networked communications; integrated 
weather information; layered, adaptive security; and more. A cost- 
effective and timely transition to NextGen depends in large part on 
FAA's ability to keep these acquisitions within budget and on schedule. 
Historically, however, FAA has had chronic difficulties meeting budget, 
schedule, and performance targets for acquisitions aimed at modernizing 
the National Airspace System (NAS). In our previous reports on the ATC 
modernization program,[Footnote 1] we have pointed out the potentially 
serious consequences of missing budget and schedule targets, including 
significant budget increases to maintain existing systems and delays in 
increasing the efficiency and safety of the NAS. 

In 2003, as part of its efforts to implement more businesslike 
operations, FAA established annual acquisition performance goals that 
called for a high percentage of critical acquisition programs to be 
within 10 percent of budget and on schedule.[Footnote 2] In 2004, as 
part of its effort to improve its ATC modernization program, FAA 
established the Air Traffic Organization (ATO) as the FAA office 
responsible for operating the ATC system, including managing the 
agency's ATC acquisitions.[Footnote 3] For fiscal years 2004 through 
2006, ATO reported exceeding its annual goals for having its major 
acquisitions on budget and on schedule. For fiscal year 2006, ATO 
reported that 100 percent of its major system acquisition investments 
were within 10 percent of their annual budgets and over 97 percent were 
on schedule. 

Because of the criticality of on-budget and on-time acquisitions to the 
efficient transition to NextGen, you asked us to determine the status 
of ATO's performance in acquiring ATC systems. To meet this objective, 
we addressed the following research questions: (1) How did ATO 
establish goals and performance measures for acquiring ATC systems and 
how are they reported? (2) How do ATO's acquisition performance 
measures compare with key attributes of successful performance 
measures? (3) What are the implications of using ATO's existing 
performance measures to assess progress in the transition to NextGen? 

To determine how ATO established goals and performance measures for 
acquiring ATC systems, we obtained and analyzed key acquisition 
documents and interviewed FAA and ATO officials. To determine how goals 
and performance are reported, we reviewed agency performance and 
accountability reports and discussed internal tracking and reporting 
methods with ATO officials. While ATO's 2006 portfolio of acquisitions 
contained 120 programs, we focused our analysis only on the programs 
determined by ATO to be major acquisitions, and thus included in ATO's 
performance reporting. Between 2003 and 2006, the number of major 
acquisitions selected for performance reporting varied from 29 to 42. 
To evaluate how ATO's acquisition performance measures compare with key 
attributes of successful performance measures, we compared ATO's 
performance measures with eight key attributes of successful 
performance measures that we identified in past GAO work.[Footnote 4] 
We also obtained the perspectives of five aviation experts on the 
reasonableness of ATO's acquisition performance measures. We selected 
these experts from government and industry to obtain a balanced 
perspective. To evaluate the implications of using ATO's existing 
performance measures to assess progress in the transition to NextGen, 
we obtained and analyzed acquisition budget and schedule data for 
fiscal years 2003 through 2006, and drew upon our past work that 
determined systemic causes of budget increases and schedule delays. 
Through discussions with ATO officials, we determined that these data 
were sufficiently reliable for the purposes of our report. See appendix 
I for a more detailed explanation of our scope and methods. We 
conducted our work from January 2007 through December 2007 in 
accordance with generally accepted government auditing standards. 

Results in Brief: 

ATO established its annual goals and performance measures for acquiring 
ATC systems to be consistent with federal guidance and with targets set 
in the Department of Transportation's strategic plan. For fiscal year 
2003, ATO's acquisition targets were to have 80 percent of acquisition 
programs within 10 percent of budget and on schedule. These targets 
gradually increase until fiscal year 2008, when they reach and remain 
at 90 percent. ATO reports its performance to Congress, the President, 
and the American people in FAA's annual Performance and Accountability 
Report. Each year, ATO selects major acquisition programs and measures 
their performance using the acquisitions' most recently approved budget 
and schedule estimates. From fiscal years 2003 through 2006, the number 
of acquisitions selected for performance reporting varied from 29 to 
42. To measure budget performance, ATO compares a selected 
acquisition's total budget (i.e., the budget to complete the 
acquisition), as estimated in August, with the total budget estimated 
in January of that year. ATO considers any program with budget growth 
of more than 10 percent during that 8-month period as not on budget for 
the year. To measure schedule performance, at the start of each fiscal 
year ATO managers select a minimum of two schedule milestones expected 
to occur that year for each major acquisition selected for performance 
reporting. Milestones could be dates to complete activities such as 
final installation or establishing a procurement plan. At the end of 
the fiscal year, ATO divides the number of selected milestones that 
were met by the total number of selected milestones. Using these 
calculations, ATO reports each year the percentage of major programs 
within the annual budget and schedule targets in its Performance and 
Accountability Report. 

Because ATO's acquisition performance measures meet only four of the 
eight key attributes of successful performance measures that we have 
identified, and focus only on the preceding year, they may not provide 
a valid assessment of performance over time.[Footnote 5] On the 
positive side, ATO's acquisition performance measures are aligned with 
FAA's strategic objectives, have measurable targets, have no overlap, 
and address governmentwide priorities. For example, the measures are 
linked with FAA's mission and agencywide goals in the strategic plan, 
known as the Flight Plan, which is available on FAA's Web site. 
Additionally, because the measures are numerical, they provide a 
measurable target--another key attribute. However, ATO's lack of 
objective criteria for selecting the programs and milestones impairs 
attributes of objectivity, reliability, and coverage of core program 
activities. Although ATO uses some general guidelines for selecting 
acquisitions for performance reporting, ATO has no objective criteria 
for designating an acquisition as major, and executive judgment was the 
primary basis for selecting acquisitions for performance reporting. For 
fiscal years 2003 through 2006, ATO designated about a quarter of its 
acquisitions as major. While these acquisitions accounted for about 80 
percent of the value of ATO's acquisitions portfolio, the absence of 
objective criteria increases the potential for bias, and also impairs 
the key attribute of reliability. Reliable performance measures produce 
the same result under similar conditions. Also, the lack of objective 
criteria for selecting programs does not ensure that core program 
activities are included in the performance measure each year. Four of 
our five experts noted that ATO needs to improve its criteria for 
selecting programs for performance reporting. ATO's performance 
measures also lack clarity in that they do not indicate that 
performance of baselined acquisitions is measured against the most 
recently approved budget and schedule estimates. Of the 31 baselined 
programs that ATO has selected for acquisition performance reporting 
for one or more years from fiscal years 2003 through 2006, 18 have been 
rebaselined (i.e., have had budget or schedule estimates revised) and 
some of these have been rebaselined more than once. (See app. II.) ATO 
officials stressed that they measure and report annual performance, and 
stated they do not measure performance against original baselines 
because the agency has already determined that the original baselines 
cannot be met. We agree that rebaselining may be appropriate in such 
cases and that measuring performance against the current baseline has 
some value. However, annual measurements for acquisitions that have 
been rebaselined and span several years do not provide a complete 
picture of acquisition performance over time. Four of our five experts 
commented that the extent of rebaselining should be identified in some 
form in ATO's performance reports. Additionally, ATO's on-schedule 
acquisition performance measure does not indicate that the reported 
performance is based on selected annual milestones, which could give 
the misleading impression that an acquisition as a whole is on 
schedule. Taken as a whole, the lack of objectivity, reliability, 
coverage of core program activities, and clarity, as well as ATO's 1- 
year focus, may not provide a valid reflection of acquisition 
performance. 

ATO's acquisition performance measurement and reporting could mask 
budget increases and schedule delays that could negatively affect the 
transition to NextGen. ATO reported that for fiscal years 2004 through 
2006, it substantially exceeded its on-budget and on-schedule 
performance goals and showed nearly steady improvement. However, when 
performance is measured against original baselines instead of annual 
budgets or milestones, acquisition performance was lower than reported, 
but still showed a general trend of improvement for fiscal years 2003 
through 2006. In fact, even when measured against original baselines, 
ATO would have met its goals for budget in fiscal years 2004 through 
2006. However, ATO would have met its schedule goals only in 2005. 
ATO's annual performance measurement and reporting could overstate the 
success of the transition to NextGen. For example, ATO's performance 
reporting does not indicate that rebaselined acquisitions, some of 
which are key to NextGen, have exceeded original budget and/or schedule 
baselines. Consequently, budget increases and delays that could impact 
the capacity, efficiency, and safety benefits of NextGen may not be 
apparent to Congress or aviation stakeholders. Including original 
budget and schedule baselines in ATO's performance reporting could 
improve the reports' usefulness by helping Congress and other 
stakeholders identify trends and take corrective action to ensure that 
the capacity, efficiency, and safety benefits of NextGen are achieved 
in a cost-effective and timely manner. 

We are recommending that the Secretary of Transportation direct the FAA 
Administrator to establish objective criteria for selecting programs 
for annual performance reporting and improve the clarity of ATO's 
annual reporting of acquisition performance. We are also recommending 
that ATO regularly report to Congress and the public on its overall, 
long-term performance in acquiring ATC systems by providing original 
budget and schedule baselines for each rebaselined program and the 
reasons for the rebaselining, as well as reporting the potential 
effects that budget or schedule slippages could have on the transition 
to NextGen. 

We provided a draft of this report to the Department of Transportation 
for comment. Senior officials from ATO's Office of Finance provided 
oral comments. FAA generally concurred with our recommendations and 
noted that they already are considering some changes to their 
performance measurement and reporting process for systems acquisitions 
and would consider additional changes based on our recommendations. 
Officials also provided technical comments that were incorporated 
throughout this report, as appropriate. 

Background: 

Over the years, FAA has taken a number of steps to better manage its 
ATC modernization program. In 1995, based on the premise that FAA would 
be better able to manage ATC modernization if it were not constrained 
by federal acquisition laws, FAA requested and Congress enacted 
legislation that exempted the agency from most federal procurement laws 
and regulations and directed FAA to develop and implement a new 
acquisition management system that would address the unique needs of 
the agency.[Footnote 6] In 1996, FAA implemented its Acquisition 
Management System, which provides high-level acquisition policy and 
guidance for selecting and controlling ATC system acquisitions through 
all phases of the acquisition life cycle. In February 2004, FAA created 
the performance-based ATO to control and improve FAA's investments and 
operations. ATO incorporated FAA's former Research and Acquisitions and 
Air Traffic Services organizations--essentially those that develop and 
acquire systems and those that operate them--into a single 
organization. 

ATO catalogs its acquisition programs in its Capital Investment Plan 
(CIP). The fiscal year 2006 CIP contained 120 funded acquisition 
programs and their anticipated total budgets. The 120 acquisition 
programs include 37 that have had acquisition program baselines 
approved by FAA's Joint Resources Council (JRC).[Footnote 7] These 
baselined programs include communications, navigation, and surveillance 
systems that are key to air traffic control operations. Acquisition 
program baselines show, among other things, executive agreement on an 
acquisition's estimated budget and schedule. The JRC also approves 
rebaselining, through which the agency documents and approves changes 
to a program's budget and schedule. Of the 37 baselined programs, 29 
also have an exhibit 300, a document prepared for the Office of 
Management and Budget (OMB) that provides investment justification and 
management plans for major ATC acquisitions. OMB requires agencies to 
submit exhibit 300s for major investments, such as those that require 
special management attention due to the investment's importance to the 
agency, have significant program or policy implications, or that the 
agency defines as major. Figure 1 provides a breakout of the programs 
in the fiscal year 2006 CIP. 

Figure 1: Breakout of ATO's Funded CIP Programs for Fiscal Year 2006: 

[See PDF for image] 

This figure is an illustration of the breakout of ATO's funded CIP 
Programs for fiscal year 2006. The following data is depicted: 

120 Acquisition Programs: 
* 83 not baselined; 
* 37 baselined; 
- 29 have exhibit 300; 
- 8 baselined without exhibit 300. 

Source: GAO analysis of FAA (data). 

[End of figure] 

The 83 programs that are not baselined include facilities and 
infrastructure programs; a variety of mission support programs such as 
training, information technology services, contractual services, and 
ancillary systems; and systems that are commercially available and 
ready for ATO to use without modification. 

FAA's annual goals and measures for performance reporting, including 
the two for performance in managing acquisitions, are described in the 
agency's annual strategic plan, known as the FAA Flight Plan.[Footnote 
8] The Flight Plan, which FAA began publishing in 2004, sets forth 
goals, objectives, strategies, initiatives, and specific performance 
targets for the agency. The Government Performance and Results Act of 
1993 requires each agency to prepare and submit to the President and 
Congress annual reports on program performance for the previous fiscal 
year. As an administration within the Department of Transportation, FAA 
is not required to prepare a separate report, but has elected to do so 
following the statutory framework and guidance for federal agencies. 
Each year, FAA reports its level of success in meeting its two 
acquisition performance targets, as well as its other performance 
targets, in its Performance and Accountability Report. The full suite 
of FAA's performance measures for fiscal year 2006 is listed in 
appendix III. 

In our past work, we identified nine key attributes of successful 
performance measures used to evaluate agencies' performance goals and 
measures.[Footnote 9] We determined that eight of these were applicable 
to our study of ATO's on-budget and on-schedule acquisition goals and 
performance measures.[Footnote 10] See appendix I for more information 
on our methods. The eight key attributes to which we compared ATO's 
acquisition performance measures are as follows: 

1. Linkage. Measure is aligned with division-and agencywide goals and 
mission and clearly communicated throughout the organization. 

2. Measurable target. Measure has a numerical goal. 

3. Limited overlap. Measure provides new information beyond that 
provided by other measures. 

4. Governmentwide priorities. Each measure covers a priority such as 
quality, timeliness, and cost of service. 

5. Objectivity. Measure is reasonably free from significant bias or 
manipulation. 

6. Reliability. Measure produces the same result under similar 
conditions. 

7. Core program activities. Measure covers the activities that an 
entity is expected to perform to support the intent of the program. 

8. Clarity. Measure is clearly stated and the name and definition are 
consistent with the methodology used to calculate it. 

ATO Established Annual Performance Measures to Be Consistent with 
Federal Guidance and Selects Major Programs for Measuring and 
Reporting: 

ATO developed its performance targets to be consistent with targets set 
in the Department of Transportation's strategic plan, OMB guidance, 
[Footnote 11] and the Federal Acquisition Streamlining Act of 1994, 
which call for other federal agencies to establish cost and schedule 
goals for acquisitions and to achieve at least 90 percent of those 
goals.[Footnote 12] In FAA's latest Flight Plan, covering fiscal years 
2008 through 2012, the performance targets for acquisitions are stated 
as follows: 

* In fiscal year 2008, 90 percent of major system acquisition 
investments are within 10 percent of annual budget and maintain through 
fiscal year 2012; 

* In fiscal year 2008, 90 percent of major system acquisition 
investments are on schedule and maintain through fiscal year 2012. 

FAA began using the basic structure of its current annual acquisition 
performance measures in 2003 when FAA sought to achieve 80 percent of 
designated milestones and maintain 80 percent of critical program costs 
within 10 percent of the total budget as published in the CIP. In 2005, 
ATO split the measure into separate targets for budget and schedule. 
ATO's acquisitions targets gradually increased to 90 percent within 10 
percent of budget and 90 percent on schedule by fiscal year 2008. 

Although ATO measures performance of all acquisitions, it reports 
performance only on major acquisitions in its annual Performance and 
Accountability Report. At the beginning of each fiscal year, ATO 
managers identify major acquisitions for performance reporting. 
According to ATO officials, their selections are based on a number of 
program characteristics; key among these is an acquisition's 
criticality to the NAS. ATO officials told us that judging a program as 
critical could be based on a number of factors, such as the program 
having an OMB exhibit 300 or a baseline document. In addition, 
officials stated that the selected acquisitions were meant to represent 
a cross section of ATC system acquisitions within ATO. From fiscal 
years 2003 through 2006, the number of acquisition programs selected 
for annual performance reporting varied between 29 and 42, or roughly a 
quarter of ATO's total acquisitions portfolio each year. 

ATO measures budget and schedule performance against selected 
acquisitions' most recently approved estimates. To measure on-budget 
performance, ATO compares the total amount budgeted for each selected 
acquisition (i.e., the estimated budget to complete an acquisition) 
reported in the January CIP with the corresponding amount reported in 
the August CIP.[Footnote 13] ATO considers any program with budget 
growth of more than 10 percent in this 8-month time frame as not on 
budget. At the end of the fiscal year, ATO divides the number of 
selected acquisitions that are considered on-budget by the total number 
of selected acquisitions and then reports the result as the percentage 
of major acquisitions within 10 percent of annual budget. To measure 
schedule performance, at the start of each fiscal year ATO managers 
select a minimum of two schedule milestones from each major acquisition 
selected for performance reporting that year. Milestones could be dates 
to complete activities such as a final installation or establishing a 
procurement plan. At the end of the fiscal year, ATO divides the number 
of selected milestones that were met by the total number of selected 
milestones. ATO reports the result as the percentage of major programs 
that are on schedule. 

ATO's Acquisition Performance Measures' Lack of Certain Successful 
Attributes and 1-Year Focus Impairs Validity of Performance Reporting: 

ATO's acquisition performance measures meet four of eight key 
attributes for successful performance measures that we have identified, 
but the lack of objective criteria for selecting programs for 
performance measurement reduces objectivity, reliability, and assurance 
that core programs are included; the clarity of the performance 
measures could also be improved. Taken together, the lack of successful 
attributes, combined with the 1-year focus of the performance measures, 
may not provide a valid measure of ATO's acquisition performance. 

ATO's Acquisition Performance Measures Meet Four of Eight Key 
Attributes of Successful Performance Measures: 

We determined that eight key attributes of successful performance 
measures were applicable to our study of ATO's two acquisition 
performance measures. Our analysis determined that ATO met half of 
these key attributes, as detailed in table 1. 

Table 1: Comparison of Key Attributes of Successful Performance 
Measures with ATO's Acquisition Performance Measures: 

Key attributes of successful performance measures: Linkage. Measure is 
aligned with division-and agencywide goals and mission and clearly 
communicated throughout the organization; 
ATO's measure has attribute: [Check]; 
Attributes of ATO's acquisition performance measures: ATO's acquisition 
performance measures are aligned with the annual goals set forth in the 
agency's strategic plan, known as the Flight Plan, which is available 
on FAA's Web site. 

Key attributes of successful performance measures: Measurable target. 
Measure has a numerical goal; 
ATO's measure has attribute: [Check]; 
Attributes of ATO's acquisition performance measures: ATO's acquisition 
performance measures have numerical budget and schedule targets. 

Key attributes of successful performance measures: Limited overlap. 
Measure provides new information beyond that provided by other 
measures; 
ATO's measure has attribute: [Check]; 
Attributes of ATO's acquisition performance measures: ATO's acquisition 
performance measures encompass 2 of 30 metrics listed in the 
Performance and Accountability Report. These two measures are 
specifically for the purpose of measuring annual performance in 
managing major ATC acquisitions. 

Key attributes of successful performance measures: Governmentwide 
priorities. Each measure covers a priority such as quality, timeliness, 
and cost of service; 
ATO's measure has attribute: [Check]; 
Attributes of ATO's acquisition performance measures: ATO's acquisition 
performance measures are consistent with governmentwide priorities that 
set numerical cost and schedule performance standards for acquisitions, 
specifically 49 U.S.C. ï¿½ 40121(a) and (b), and the Federal Acquisition 
Streamlining Act of 1994 (Public Law 103-355). 

Key attributes of successful performance measures: Objectivity. Measure 
is reasonably free from significant bias or manipulation; 
ATO's measure has attribute: [Empty]; 
Attributes of ATO's acquisition performance measures: ATO measures 
acquisition performance for "major" programs but has no objective 
definition for major programs. 

Key attributes of successful performance measures: Reliability. Measure 
produces the same result under similar conditions; 
ATO's measure has attribute: [Empty]; 
Attributes of ATO's acquisition performance measures: ATO uses the same 
methodology to measure the performance of its selected acquisition 
programs each year, but the lack of objective criteria for selecting 
programs for performance reporting reduces the likelihood of producing 
the same results under similar conditions. 

Key attributes of successful performance measures: Core program 
activities. Measure covers the activities that an entity is expected to 
perform to support the intent of the program; 
ATO's measure has attribute: [Empty]; 
Attributes of ATO's acquisition performance measures: ATO managers 
select for performance reporting the acquisition programs they consider 
most critical to the NAS, but the lack of objective criteria for 
selecting programs for performance reporting reduces the chances that 
core program activities are included each year. 

Key attributes of successful performance measures: Clarity. Measure is 
clearly stated and the name and definition are consistent with the 
methodology used to calculate it; 
ATO's measure has attribute: [Empty]; 
Attributes of ATO's acquisition performance measures: ATO does not 
clearly state that its basis of measurement for baselined programs is 
the most current estimate of budget and schedule, which may have 
changed significantly since the start of an acquisition. ATO also does 
not make clear that on-schedule performance means only that a program 
has met selected annual milestones. 

Source: GAO. 

[End of table] 

ATO's Performance Measures Lack Objectivity, Reliability, and Assurance 
That Core Programs Are Included: 

Although ATO provides some general guidance for selecting acquisitions 
for performance reporting, executive judgment was the primary basis for 
ATO's selections. ATO described the scope of its performance measure 
for managing acquisitions in its Portfolio of Goals for fiscal year 
2006 as follows: "FAA's Air Traffic Organization (ATO) Service Units 
select specific programs that are determined to provide a capital asset 
to the NAS. For FY06, 31 acquisition programs will be tracked and 
monitored.[Footnote 14] Most of the programs selected are considered 
'major' and must submit an exhibit 300. Those that do not provide 
exhibit 300s are included because they contribute an asset to the NAS 
with a useful life of more than two years. The designation of 'critical 
acquisition programs' in the title of this performance target expresses 
the critical value of the program to the NAS." 

As this description illustrates, ATO's guidance in designating programs 
as major allows for a significant amount of professional judgment and 
does not clearly define which programs should be included or excluded 
for performance reporting. Figure 2 illustrates the types of 
acquisitions that ATO selected for performance reporting in fiscal year 
2006. 

Figure 2: Types of Acquisition Programs Selected for Performance 
Reporting, Fiscal Year 2006: 

[See PDF for image] 

This figure is an illustration of the total universe of All acquisition 
programs, with a subset of Baselined programs contained within that 
universe. An additional subset, Exhibit 300 programs is a subset within 
the Baselined programs subset. The following data is depicted: 

Total, All acquisition programs: 120; 
Total, Baselined programs: 37; 
Total, Exhibit 300 programs: 29. 

Total, "major programs" selected for performance reporting: 29; 
* 14 exhibited baselined programs; 
* 5 other baseined programs; 
* 10 other acquisition programs. 

Source: GAO analysis of ATO data. 

[End of figure] 

ATO's guidance lacks objectivity in that it does not indicate 
specifically what is to be observed and in which population or 
conditions. Contrary to ATO's statement in the Portfolio of Goals that 
most of the programs selected are considered major and must have an OMB 
exhibit 300 prepared, only 14 of the 29 programs selected that year 
actually had an OMB exhibit 300. Additionally, about half of the 
programs with an OMB exhibit 300 were not selected for performance 
reporting. For example, the System Approach for Safety Oversight 
program and the Aviation Safety Knowledge Management Environment 
program each have an OMB exhibit 300, but, according to ATO officials, 
these programs are considered "non-NAS" and are therefore not selected 
for performance reporting. Likewise, the Facilities Security Risk 
Management program has an approved baseline but is not selected. ATO 
officials told us that facilities and mission support programs such as 
this are generally not selected for acquisition performance reporting; 
however, ATO has no written guidance on this policy, and has included 
mission support programs in its performance reporting in the past. 

Between 2003 and 2006, the number of major acquisitions selected for 
performance reporting varied from 29 to 42, and represented about a 
quarter of all acquisitions each year. ATO officials told us that the 
variation in the number of programs selected from year to year occurred 
because of decisions to report performance on specific acquisitions, or 
changes in acquisitions' status from year to year, such as the 
introduction of new acquisitions, the completion or cancellation of 
acquisitions, or lapses in funding for a fiscal year. Although ATO 
reports performance on about 25 percent of the acquisitions portfolio, 
ATO officials pointed out that the selected acquisitions have 
represented between 76 and 84 percent of the value of that portfolio. 
Nevertheless, because ATO has no objective criteria for designating 
major programs, its performance measure is vulnerable to bias and the 
possibility that important or troubled programs could be excluded from 
performance reporting. 

Objective performance measures should not allow subjective 
considerations or judgments to dominate the outcome of the measurement. 
Objectivity is important in selecting acquisitions for performance 
reporting. OMB guidance allows an agency to define major investment in 
its capital planning and investment control process.[Footnote 15] For 
example, the Department of Defense defines a major acquisition program 
as one requiring an estimated total expenditure for research, 
development, testing, and evaluation of more than $365 million, or an 
estimated total procurement expenditure of more than $2.2 billion in 
fiscal year 2000.[Footnote 16] However, the department also allows for 
professional judgment as the Secretary of Defense can designate as 
major any programs below the dollar thresholds, but determined to be 
important. 

In addition to selecting major programs for performance reporting, ATO 
managers select two or more schedule milestones from each selected 
program, and use these to measure schedule performance. ATO's guidance 
allows managers wide latitude to select milestones for performance 
measurement and provides no guidance regarding the significance of the 
milestones that managers should select. Because this provides managers 
the opportunity to exclude milestones that they do not expect to meet 
during the coming fiscal year, it further weakens the objectivity of 
the measure. Four of our five experts commented that ATO's lack of 
criteria for selecting milestones or the ability to pick and choose 
milestones for performance measurement was a shortcoming of ATO's 
performance measurement. 

The lack of objective criteria for designating major programs also 
impairs the key attribute of reliability and the assurance that the 
measures include core program activities. Performance measures possess 
the key attribute of reliability when they produce the same results 
each time they are applied under the same conditions. We have reported 
that judgmental evaluations can impair reliability and introduce 
inconsistencies, which can affect the outcome of performance 
measurement. With executive judgment serving as the primary determinant 
of which programs and milestones are selected and measured, different 
managers could select different programs each year, resulting in 
different performance results. Likewise, the lack of objective criteria 
does not ensure that ATO managers include all core program activities 
in performance measurement each year. We found that ATO eliminated some 
acquisitions from performance reporting for 1 or more years and then 
resumed reporting these acquisitions in a subsequent year, although ATO 
provided reasonable explanations for these occurrences. Nevertheless, 
reliability and the inclusion of core program activities would be 
better assured if executive judgment was grounded in written and 
objective criteria. Four of the five experts who advised us during our 
review agreed that ATO needs to improve its criteria for determining 
which programs are major and consequently are included in performance 
reporting. 

ATO's Performance Measures Do Not Clearly Indicate the Use of 
Rebaselined Budget Estimates or Annual Schedule Milestones: 

ATO's acquisition performance measures also lack a fourth key attribute 
of successful performance measures: clarity. A performance measure that 
is not clearly stated (i.e., contains extraneous or omits key data 
elements) or that has a name or definition that is inconsistent with 
the way it is calculated can confuse users and could cause managers or 
other stakeholders to think that performance was better or worse than 
it actually was. 

ATO's On-Budget Performance Measure Does Not Clearly Indicate the Use 
of Revised Budget Estimates: 

ATO's on-budget acquisition performance measure lacks clarity in 
reporting because ATO does not indicate that the acquisition 
performance of baselined programs is measured using the most recently 
approved budget estimates, as reflected in the January CIP.[Footnote 
17] While ATO does present Congress with valuable information about a 
program's most recent budget performance by comparing the August budget 
estimate against the January budget estimate, this provides only one 
perspective on performance because rebaselining resets the measurement 
of budget or schedule variances to zero. Of the 31 baselined programs 
that ATO selected for acquisition performance reporting in 1 or more 
years from fiscal year 2003 through fiscal year 2006, the agency has 
rebaselined 18, and has rebaselined some of these more than once. (See 
app. II.) For example, the Standard Terminal Automation Replacement 
System (STARS) was originally budgeted at $940.2 million, but has been 
rebaselined twice and is now budgeted at almost $2.8 billion. However, 
because ATO measures budget performance for an 8-month timeframe 
against the most recently approved budget estimate, STARS was 
considered on budget for fiscal years 2003 through 2006. Other 
acquisitions had exceeded original budgets by between $9 million and 
$159 million by March 2007, the date of the most recently rebaselined 
acquisition. This information is not disclosed in ATO's performance 
reporting.[Footnote 18] (See app. II.) 

ATO officials emphasized that they measure and report on annual 
performance and stated they do not measure performance against original 
baselines because the agency has already determined that the original 
baselines cannot be met. One of the experts who advised us pointed out 
that it serves no purpose to continually call a program over budget and 
behind schedule if it has been successfully managed for a significant 
period of time. We agree that when original baselines cannot be met, 
rebaselining can be appropriate. We also agree that measuring annual 
progress against the current program baseline has some value, but using 
annual measurement as the sole basis for acquisitions that have been 
rebaselined does not provide a complete picture of performance over 
time. Four of our five experts commented that disclosure of 
rebaselining was important in some form, and suggestions ranged from 
disclosing rebaselining in a footnote to clearly reporting all 
rebaselining. The absence of this information on rebaselining in ATO's 
performance reporting could cause managers and other stakeholders, 
including Congress, to think that performance was better than it 
actually was. 

ATO's On-Schedule Performance Measure Does Not Clearly Indicate That It 
Reflects Only the Achievement of Selected Milestones within a Given 
Fiscal Year: 

ATO reports meeting its schedule performance goal when at least a 
specified percentage "of major system acquisition investments are on 
schedule...." This same wording is how the goal is reported in FAA's 
2006 Performance and Accountability Report, where the agency noted on- 
schedule performance of 97.44 percent. However, ATO is actually basing 
its schedule performance measurement on two or more schedule milestones 
within a selected program. Thus, the wording of the target and the 
performance reporting gives the misleading impression that the entire 
acquisition is on schedule when the reported performance is based only 
on selected milestones. For example, ATO reported that the $286 million 
Integrated Terminal Weather System (ITWS) acquisition was on schedule 
in fiscal year 2006 because it hit its selected milestones for that 
year. However, the ITWS acquisition (which began in 1997) has 
encountered funding reductions, requirements growth and unplanned work, 
and greater-than-expected software complexity. ITWS is now scheduled 
for completion in October 2009. ATO's annual reporting based on 
milestones simply notes ITWS as on-schedule and does not make clear 
that the program was originally scheduled for completion 6 years 
earlier, in July 2003. 

Because ATO's performance measures lack several attributes of 
successful performance measures, and are focused on 1-year snapshots of 
performance, they may not provide a valid assessment of acquisition 
performance over time. A valid measure provides an accurate 
representation of what is being measured. Many of ATO's acquisitions 
span several years and, as the next section shows, measuring 
performance against original baselines provides a different perspective 
on acquisition performance than that reported by ATO over the past 4 
years. 

ATO's Performance Measurement Could Mask Budget Increases and Delays in 
the Transition to NextGen: 

When measured against original baselines, ATO shows improvement in its 
managing of acquisitions, but its performance is lower than indicated 
in FAA's annual Performance and Accountability Report. The lack of 
original baseline information in ATO's performance reporting could 
provide Congress and the American people with the impression that the 
transition to NextGen is progressing more smoothly than might actually 
be the case. 

Against Original Baselines, ATO's Acquisition Performance Has Improved, 
but Was Lower Than Reported: 

Comparing the current status of ATO's major ATC system acquisitions 
(i.e., those that ATO selected each year for performance reporting) 
with the budgets and schedules in these acquisitions' original 
baselines yields lower performance results than those reported to 
Congress and the American people. According to ATO's performance 
reports, the organization showed nearly steady improvement in fiscal 
years 2003 through 2006 and substantially exceeded its targets for 
those years, twice hitting 100 percent. (See table 2.) However, when 
performance is measured against original baselines instead of annual 
budgets or milestones, acquisition performance was lower than reported, 
but still showed a general trend of improvement for fiscal years 2003 
through 2006. In fact, even when measured against original baselines, 
ATO would have met its goals for budget in fiscal years 2004 through 
2006. However, ATO did not perform as well in meeting schedules when 
measured against original baselines. ATO would have met its schedule 
goal only in 2005. 

Table 2: Annual Performance Targets, Performance of ATO's Selected 
Major Acquisitions Measured against Annual Targets and Original 
Baselines, Fiscal Years 2003-2012: 

Fiscal year: 2003; 
Performance targets: 80%; 
Major acquisitions on budget: Reported performance measured using 
annual budgets: 88%; 
Major acquisitions on budget: Performance measured using original 
budget baselines: 76%; 
Major acquisitions on schedule: Reported performance measured using 
annual milestones: 77%; 
Major acquisitions on schedule: Performance measured using original 
schedule baselines: 70%. 

Fiscal year: 2004; 
Performance targets: 80%; 
Major acquisitions on budget: Reported performance measured using 
annual budgets: 100%; 
Major acquisitions on budget: Performance measured using original 
budget baselines: 83%; 
Major acquisitions on schedule: Reported performance measured using 
annual milestones: 91.5%; 
Major acquisitions on schedule: Performance measured using original 
schedule baselines: 79%. 

Fiscal year: 2005; 
Performance targets: 80%; 
Major acquisitions on budget: Reported performance measured using 
annual budgets: 97%; 
Major acquisitions on budget: Performance measured using original 
budget baselines: 91%; 
Major acquisitions on schedule: Reported performance measured using 
annual milestones: 92%; 
Major acquisitions on schedule: Performance measured using original 
schedule baselines: 83%. 

Fiscal year: 2006; 
Performance targets: 85%; 
Major acquisitions on budget: Reported performance measured using 
annual budgets: 100%; 
Major acquisitions on budget: Performance measured using original 
budget baselines: 90%; 
Major acquisitions on schedule: Reported performance measured using 
annual milestones: 97.44%; 
Major acquisitions on schedule: Performance measured using original 
schedule baselines: 76%. 

Fiscal year: 2007; 
Performance targets: 87.5%; 
Major acquisitions on budget: Reported performance measured using 
annual budgets: TBD; 
Major acquisitions on budget: Performance measured using original 
budget baselines: TBD; 
Major acquisitions on schedule: Reported performance measured using 
annual milestones: TBD; 
Major acquisitions on schedule: Performance measured using original 
schedule baselines: TBD. 

Fiscal year: 2008-2012; 
Performance targets: 90%; 
Major acquisitions on budget: Reported performance measured using 
annual budgets: TBD; 
Major acquisitions on budget: Performance measured using original 
budget baselines: TBD; 
Major acquisitions on schedule: Reported performance measured using 
annual milestones: TBD; 
Major acquisitions on schedule: Performance measured using original 
schedule baselines: TBD. 

Source: GAO analysis of ATO data. 

[End of table] 

ATO officials told us they use a number of methods to measure the 
performance of all of the acquisitions contained in the CIP. They have 
implemented earned value management on all new major acquisitions as a 
way to prevent, detect, report, and correct problems in acquiring major 
systems and to ensure that major programs are within budget and 
schedule targets.[Footnote 19] ATO officials also noted that they have 
monthly meetings on the status of acquisitions and that management is 
constantly apprised of program performance. In addition, officials 
stated that significant changes in acquisition status, such as 
rebaselining, must go through high-level agency and OMB review, and FAA 
reports to Congress any program that exceeds its baseline costs by more 
than 50 percent.[Footnote 20] While ATO has reported acquisitions' 
variances against original baselines to Congress on an ad hoc basis in 
response to questions for the record, the only systematic reporting of 
ATO's acquisitions performance to the Congress and the American people 
is FAA's annual Performance and Accountability Report. 

ATO's establishment of annual goals and subsequent annual reporting of 
performance are appropriate actions aimed at improving ATO's 
performance in managing ATC system acquisitions. Annual goals are used 
throughout government to keep programs on track. We have noted that 
such goals illustrate a commitment to achieving immediate, concrete, 
and measurable results in the near term. However, it also is important 
to provide decision makers and stakeholders with an overall 
understanding of program performance in a more holistic sense. We first 
noted the shortcomings of annual performance goals for acquisitions in 
2005, after ATO reported that it met its acquisition performance goals 
for the first time. We cautioned that, while meeting a 1-year goal was 
a positive step, annual performance targets should continue to be 
viewed in the broader context of acquisitions' original and revised 
baselines.[Footnote 21] The 18 rebaselined acquisitions on which ATO 
reported performance from fiscal year 2003 through 2006 have 
collectively exceeded their original budget estimates by approximately 
$4.4 billion, or 66 percent; however, over 95 percent of this increase 
occurred in the Standard Terminal Automation Replacement System and in 
WAAS. The 18 rebaselined acquisitions have experienced schedule 
slippages between 1 and 10 years. 

ATO's Performance Measurement Could Overstate the Success of the 
Transition to NextGen: 

Because some of FAA's current acquisitions form the basic building 
blocks for NextGen, delays and budget increases in these acquisitions 
could have significant implications for the transition to NextGen. 
Figure 3 illustrates the planned transition from current systems to 
future systems and the anticipated benefits. For example, STARS, 
discussed previously, is listed as a current program leading to the 
transition to NextGen in figure 3. Our research disclosed that the near 
tripling of the acquisition's budget resulted from insufficient 
involvement of stakeholders and requirements growth--two systemic 
factors that we found led to acquisitions missing their budget and 
schedule targets.[Footnote 22] However, the budget increases that STARS 
experienced are not discussed in Performance and Accountability Reports 
or in any other regular reporting to Congress. 

Figure 3: The Transition to NextGen: 

[See PDF for image] 

This figure illustrates the planned transition from current systems to 
future systems and the anticipated benefits. The following data is 
depicted: 

Current Programs: 
ERAMSTARS/CARTS; 
ADS-B; 
TMA; 
TFM-M; 
SWIM; 
Data Comm; 
Precision Navigation; 
Network Enabled Weather. 

Key Near-Term Investments: 
ERAM Enhancements: Automated Problem Resolution; 
Concept Demonstrations: Trajectory Based Ops/High Density; 
Infrastructure-Trajectory Based Ops: Time Based Metering; 
TFM-M Enhancements: Time-Based Metering; 
RNP/RNAV Expansion: Precise Navigation; 
Data Communications: Flight Intent Downlink; 
ADS-B: Aircraft Separation; 
SWIM/Net-Enabled Weather: Net-Centric Information Sharing. 

FAA Solution Sets: 
Initiate trajectory based operations; 
Increase arrivals/departures at high density airports; 
Increase flexibility in the terminal environment; 
Improve collaborative ATM; 
Reduce weather impact; 
Increase safety, security, and environmental performance; 
Transform facilities. 

Source: Joint Planning and Development Office. 

Note: Abbreviations used above are spelled out as follows: 

ERAM: En Route Automation Modernization: 

STARS: Standard Terminal Automation Replacement System: 

CARTS: Common Automated Radar Terminal System: 

ADS-B: Automatic Dependent Surveillance - Broadcast: 

TMA: Traffic Management Advisor: 

SWIM: System Wide Information Management: 

Data Comm: data communications: 

Ops: operations: 

TFM-M: Traffic Flow Management - Modernization: 

RNP: Required Navigation Performance: 

RNAV: Area Navigation: 

ATM: air traffic management: 

[End of figure] 

Another example is the Airport Surveillance Radar - Model 11 (ASR-11), 
which is an integrated digital system intended to replace aging analog 
radars. NextGen's plans call for the ASR-11 to provide aircraft and 
weather surveillance in terminal areas of small and medium-sized 
airports, which also may serve as a part of the back-up surveillance 
system in case the primary satellite-based ATC system fails. However, 
the ASR-11 has encountered a 59-percent increase in budget per deployed 
system and its completion date has been delayed from 2005 to 2009, in 
part due to requirements growth. As with STARS, the budget increases 
and schedule delays experienced in the ASR-11 acquisition are not 
discussed in ATO's Performance and Accountability Reports or in any 
routine report to Congress. 

The absence of original budget and schedule estimates in ATO's 
performance reporting could give the impression to Congress and the 
American people that ATO's acquisitions and the transition to NextGen 
are progressing more smoothly than is actually the case. Including 
original budget and schedule baselines in ATO's performance reporting 
could improve the reports' usefulness by helping Congress and other 
stakeholders identify trends and take corrective action to ensure that 
the capacity, efficiency, and safety benefits of NextGen are achieved 
in a cost-effective and timely manner. 

Conclusions: 

Although ATO's acquisition performance measures meet some of the key 
attributes of successful performance measures, the attributes that the 
measures lack are significant and, considered together, raise serious 
questions about the measures' validity. While a 1-year focus may be 
appropriate for some performance measures, it may not provide a valid 
assessment of performance over time for major ATC acquisitions that 
span a number of years. Moreover, ATO's use of subjective criteria to 
pick a subset of acquisitions and milestones for performance 
measurement and its lack of disclosure regarding rebaselining may not 
provide Congress, aviation stakeholders, and the public with a complete 
picture of ATO's ability to deliver major ATC acquisitions on budget 
and on time. Such reporting could also make budget increases and 
schedule delays more difficult to identify. These issues are critical 
as ATO begins acquiring new systems with a goal of completing the 
transition to NextGen by 2025. Recognizing impending budget increases 
and schedule delays and taking corrective action will be necessary to 
keep the overall NextGen effort on track. The more quickly ATO can 
transition to NextGen, the more quickly the nation will realize the 
increased efficiencies and safety benefits of new systems and 
technologies, and avoid the costs and inefficiencies of maintaining 
existing systems. By presenting the most accurate and complete 
assessment possible when reporting its performance in acquiring ATC 
systems, FAA will better facilitate congressional understanding and 
oversight of FAA's progress in implementing NextGen. 

Recommendations for Executive Action: 

Because of the importance of ensuring that key administration and 
congressional decision makers and stakeholders have complete 
information on the budget and schedule performance of FAA's critical 
ATC acquisition programs--both for the most recent fiscal year and 
since their inception--we are recommending that the Secretary of 
Transportation direct the FAA Administrator to take the following four 
actions: 

1. Improve the objectivity, reliability, and inclusion of core programs 
in ATO's acquisition performance measures by establishing written, 
objective criteria and guidance for managers to use in determining 
which programs are major--and thus selected for performance reporting-
-and in selecting schedule milestones. 

2. Improve the clarity of ATO's annual acquisition performance 
measurement process by disclosing in its Performance and Accountability 
Reports that the measurement for on-budget performance covers 8 months 
and is measured against the most recently approved budget baselines. 
Similarly, improve the wording of the target and reporting for on- 
schedule acquisitions to disclose that this measures 1 year of 
performance against selected program milestones. 

3. Identify or establish a vehicle for regularly reporting to Congress 
and the public on ATO's overall, long-term performance in acquiring ATC 
systems by providing original budget and schedule baselines for each 
rebaselined program and the reasons for the rebaselining. If this 
information is not added to FAA's annual Performance and Accountability 
Report, then the Performance and Accountability Report should reference 
where this information can be found. 

4. Improve the usefulness of ATO's acquisition performance reporting by 
including information (in the Performance and Accountability Report or 
elsewhere) on the potential effects that any budget or schedule 
slippages could have on the overall transition to NextGen. This also 
could include information concerning any mitigation plans ATO has 
developed to lessen the effects of program slippages on the 
implementation of NextGen systems. 

Agency Comments: 

We provided a draft of this report to the Department of Transportation 
for comment. Senior officials from ATO's Office of Finance provided 
oral comments. ATO officials generally concurred with our 
recommendations and noted that they already are considering some 
changes to their performance measurement and reporting process for 
system acquisitions. In view of the standards discussed in the report, 
ATO officials agreed to review current selection criteria of programs 
included for annual reporting to address concerns over objectivity. ATO 
agreed to clarify wording in the Flight Plan and future Performance and 
Accountability Reports to ensure that readers understand that the 
report reflects agency performance for the prior fiscal year only. In 
our draft report, we recommended that ATO endeavor to report on the 
overall, long-term status of its acquisitions in its annual Performance 
and Accountability Report. ATO officials felt strongly that the 
Performance and Accountability Report is meant to reflect performance 
for a single fiscal year and would not be the proper vehicle for 
reporting on long-term performance. In response, we modified our 
recommendation to be less prescriptive about where this information 
appears, as long as it is publicly reported. ATO officials said they 
would consider other reporting methods to provide Congress with longer- 
term status information about the organization's performance in 
acquiring ATC systems. ATO officials also provided technical comments 
that were incorporated throughout this report, as appropriate. 

We are sending copies of this report to the appropriate congressional 
committees, the Secretary of Transportation, and the FAA Administrator. 
We will make copies available to others upon request. In addition, the 
report will be available at no charge on the GAO Web site at 
[hyperlink, http://www.gao.gov]. 

If you or your staffs have any questions about this report, please 
contact me at (202) 512-2834 or [email protected]. Contact points for 
our Offices of Congressional Relations and Public Affairs may be found 
on the last page of this report. GAO staff who made key contributions 
to this report are listed in appendix IV. 

Signed by: 

Gerald L. Dillingham, Ph.D.: 
Director, Physical Infrastructure Issues: 

[End of section] 

List of Requesters: 

The Honorable Bart Gordon: 
Chairman: 
The Honorable Ralph Hall: 
Ranking Member: 
Committee on Science and Technology: 
House of Representatives: 

The Honorable John D. Rockefeller IV: 
Chairman: 
The Honorable Trent Lott: 
Ranking Member: 
Subcommittee on Aviation Operations, Safety, and Security: 
Committee on Commerce, Science, and Transportation: 
United States Senate: 

The Honorable Jerry Costello: 
Chairman: 
The Honorable Thomas Petri: 
Ranking Member: 
Subcommittee on Aviation: 
Committee on Transportation and Infrastructure: 
House of Representatives: 

The Honorable John Mica: 
House of Representatives: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

We examined (1) how the Air Traffic Organization (ATO) establishes 
goals and performance measures for acquiring air traffic control (ATC) 
systems and how they are reported; (2) how ATO's acquisition 
performance measures compare with key attributes of successful 
performance measures; and (3) the implications of using ATO's existing 
performance measures to assess progress in the transition to the Next 
Generation Air Transportation System (NextGen). 

To determine how ATO established acquisition goals and performance 
measures for acquiring ATC systems, we reviewed Federal Aviation 
Administration's (FAA) Flight Plans and Acquisition Management System 
policy and obtained historical budget and schedule data from ATO's 
finance office on the acquisitions for which ATO reported performance 
from fiscal years 2003 through 2006. To ensure that these data were 
consistent with documents obtained in previous GAO work, we noted 
potential discrepancies and obtained clarifying documents from ATO's 
finance office. We also discussed ATO's acquisition goals and 
performance measurement process with ATO officials. To determine how 
performance is reported, we reviewed agency performance and 
accountability reports and discussed ATO's criteria for selecting 
acquisitions for performance reporting with ATO officials. 

To determine how ATO's acquisition performance measures compare with 
key attributes of successful performance measures, we used eight key 
attributes of successful performance measures that were previously 
identified by GAO[Footnote 23] as criteria for comparison against ATO's 
acquisition performance measures. The eight key attributes are: 

1. Linkage. Measure is aligned with division-and agencywide goals and 
mission and clearly communicated throughout the organization. 

2. Measurable target. Measure has a numerical goal. 

3. Limited overlap. Measure provides new information beyond that 
provided by other measures. 

4. Governmentwide priorities. Each measure covers a priority such as 
quality, timeliness, and cost of service. 

5. Objectivity. Measure is reasonably free from significant bias or 
manipulation. 

6. Reliability. Measure produces the same result under similar 
conditions. 

7. Core program activities. Measure covers the activities that an 
entity is expected to perform to support the intent of the program. 

8. Clarity. Measure is clearly stated and the name and definition are 
consistent with the methodology used to calculate it. 

There was a ninth key attribute identified by GAO that we determined 
was not applicable to our study of ATO's acquisition performance 
measures. This ninth attribute is balance, which exists when a suite of 
measures ensures that an organization's various priorities are covered. 
Although ATO has other performance measures that it applies to its 
acquisitions, in this report we focused only on the two that FAA uses 
to report its performance--the percentages of acquisitions on budget 
and acquisitions on schedule. Because we did not examine ATO's full 
suite of performance measures for acquisitions, we did not consider the 
key attribute of balance. 

We compared attributes of ATO's acquisition performance measures 
against each of the eight key attributes to determine whether and how 
ATO's process met each attribute. We reviewed past GAO reports on FAA's 
management of the ATC modernization program,[Footnote 24] FAA's 
management of major acquisition programs,[Footnote 25] and the 
Department of Defense's acquisition management and reporting.[Footnote 
26] We identified acquisition programs whose targets and milestones 
were revised or rebaselined to determine validity and consistency in 
program performance reporting. We also interviewed ATO officials. 
Additionally, we obtained the perspectives of five aviation experts on 
the reasonableness of ATO's acquisition performance measures. To ensure 
that we collectively received a balanced and unbiased perspective, we 
selected experts with varying government and industry experience. We 
asked each expert to address the same set of questions relating to the 
reasonableness of ATO's acquisition performance measurement process. 

To determine the implications of using ATO's existing performance 
measures to assess progress in the transition to NextGen, we analyzed 
the trends for budget and schedule outcomes between the original 
baselines and current budget and schedule baselines for the 
acquisitions that ATO selected for performance reporting and monitoring 
between fiscal years 2003 and 2006. We also drew upon past work in 
which we undertook detailed reviews of the status of ATC acquisition 
programs, and obtained updated information as necessary from FAA by 
reviewing documents and interviewing agency officials. Through 
discussions with ATO officials, we determined that these data were 
sufficiently reliable for the purposes of our report. We did not 
conduct an individual or in-depth review of the effectiveness of the 
specific programs selected for performance reporting. We also did not 
identify a comprehensive list of programs that were excluded from 
acquisition performance reporting. This was beyond the scope and intent 
of this study. 

We conducted our work from January 2007 through December 2007 in 
accordance with generally accepted government auditing standards. 

[End of section] 

Appendix II: Baseline History for Programs Selected for Acquisition 
Performance Measurement: 

Table 3: Budget and Schedule Baseline History for Baselined Programs 
Selected for Acquisition Performance Measurement for 1 or More Years, 
Fiscal Years 2003 through 2006: 

Program[A]: STARS; 
Original start date: Feb-96; 
Original completion date: Oct-05; 
Original budget[B]: $940.2; 
New APB[C] date: Oct-99; 
Revised completion date: Sept-08; 
Revised budget: $1,402.6; 
New APB date: May-04; 
Revised completion date: Dec-07; 
Revised budget: $2,769.5. 

Program[A]: NEXCOM; 
Original start date: Sept-98; 
Original completion date: Sept-08; 
Original budget[B]: $407.6; 
New APB[C] date: May-00; 
Revised completion date: Sept-10; 
Revised budget: $318.4; 
New APB date: Dec-05; 
Revised completion date: Sept-13; 
Revised budget: $324.7. 

Program[A]: OASIS; 
Original start date: Dec-96; 
Original completion date: Aug-01; 
Original budget[B]: $174.7; 
New APB[C] date: Mar-00; 
Revised completion date: May-05; 
Revised budget: $249.5; 
New APB date: Feb-05; 
Revised completion date: July-04; 
Revised budget: $169.0. 

Program[A]: ITWS; 
Original start date: June-97; 
Original completion date: July-03; 
Original budget[B]: $276.1; 
New APB[C] date: Aug-01; 
Revised completion date: Oct-03; 
Revised budget: $282.3; 
New APB date: June-04; 
Revised completion date: Apr-09; 
Revised budget: $286.1. 

Program[A]: WAAS; 
Original start date: Jan-98; 
Original completion date: Aug-99; 
Original budget[B]: $1,000.6; 
New APB[C] date: Dec-99; 
Revised completion date: Dec-06; 
Revised budget: $2,978.0; 
New APB date: May-04; 
Revised completion date: Dec-08; 
Revised budget: $3,339.7. 

Program[A]: FTI; 
Original start date: July-99; 
Original completion date: Dec-08; 
Original budget[B]: $205.7; 
New APB[C] date: Dec-04; 
Revised completion date: Dec-07; 
Revised budget: $310.2; 
New APB date: Aug-06; 
Revised completion date: Dec-08; 
Revised budget: $318.8. 

Program[A]: ASWON; 
Original start date: Oct-99; 
Original completion date: Apr-02; 
Original budget[B]: $350.9; 
New APB[C] date: Aug-01; 
Revised completion date: Sept-09; 
Revised budget: $403.8; 
New APB date: June-06; 
Revised completion date: Sept-12; 
Revised budget: $384.3. 

Program[A]: NIMS II; 
Original start date: May-00; 
Original completion date: Sept-05; 
Original budget[B]: $172.9; 
New APB[C] date: Mar-06; 
Revised completion date: Sept-06; 
Revised budget: $90.2; 
New APB date: Mar-07; 
Revised completion date: Nov-07; 
Revised budget: $90.2. 

Program[A]: WARP; 
Original start date: Dec-96; 
Original completion date: Feb-00; 
Original budget[B]: $126.4; 
New APB[C] date: Oct-99; 
Revised completion date: Feb-01; 
Revised budget: $143.6; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: RCE; 
Original start date: Oct-98; 
Original completion date: Dec-01; 
Original budget[B]: $260.4; 
New APB[C] date: N/A; 
Revised completion date: Sept-03; 
Revised budget: $260.4; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: ATCBI; 
Original start date: Aug-97; 
Original completion date: Sept-04; 
Original budget[B]: $282.9; 
New APB[C] date: Jan-02; 
Revised completion date: Jan-06; 
Revised budget: $282.9; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: ASR-11; 
Original start date: Nov-97; 
Original completion date: Sept-05; 
Original budget[B]: $743.3; 
New APB[C] date: Sept-05; 
Revised completion date: Sept-09; 
Revised budget: $696.5; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: LAAS; 
Original start date: Jan-98; 
Original completion date: Dec-06; 
Original budget[B]: $536.1; 
New APB[C] date: Dec-99; 
Revised completion date: Oct-11; 
Revised budget: $696.0; [Empty]; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: HOCSR; 
Original start date: Mar-98; 
Original completion date: Sept-08; 
Original budget[B]: $424.1; 
New APB[C] date: May-03; 
Revised completion date: June-04; 
Revised budget: $368.5; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: AMASS; 
Original start date: Oct-98; 
Original completion date: Aug-00; 
Original budget[B]: $74.1; 
New APB[C] date: Mar- 00; 
Revised completion date: Sept-02; 
Revised budget: $151.7; [Empty]; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: LLWAS; 
Original start date: Oct-98; 
Original completion date: Oct-01; 
Original budget[B]: $43.5; 
New APB[C] date: May- 01; 
Revised completion date: June-04; 
Revised budget: $52.6; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: ASDE-X; 
Original start date: Sept-01; 
Original completion date: Jan-07; 
Original budget[B]: $505.2[D]; 
New APB[C] date: Sept-05; 
Revised completion date: May-11; 
Revised budget: $550.1; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: UHF Replace; 
Original start date: Nov-02; 
Original completion date: Sept-10; 
Original budget[B]: $85.1; 
New APB[C] date: Dec-05; 
Revised completion date: Sept-13; 
Revised budget: $85.1; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: CPDLC; 
Original start date: Mar-99; 
Original completion date: Dec-05; 
Original budget[B]: $166.7; 
New APB[C] date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]; [Empty]; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: BUEC; 
Original start date: Mar-00; 
Original completion date: Apr-04; 
Original budget[B]: $54.1; [Empty]; 
New APB[C] date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]; [Empty]; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: ATOP; 
Original start date: May-01; 
Original completion date: Mar-06; 
Original budget[B]: $548.2; 
New APB[C] date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]; [Empty]; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: PRM; 
Original start date: Dec-01; 
Original completion date: Dec-05; 
Original budget[B]: $145.8; 
New APB[C] date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]; [Empty]; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: ECG; 
Original start date: Mar-02; 
Original completion date: Dec-05; 
Original budget[B]: $315.1; 
New APB[C] date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]; [Empty]; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: URET; 
Original start date: June-02; 
Original completion date: Sept-06; 
Original budget[B]: $285.3; 
New APB[C] date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]; [Empty]; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: TMA; 
Original start date: June-02; 
Original completion date: Sept-07; 
Original budget[B]: $135.5; 
New APB[C] date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]; [Empty]; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: ERAM; 
Original start date: June-03; 
Original completion date: Dec-10; 
Original budget[B]: $2,154.6; 
New APB[C] date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]; [Empty]; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: En Route System Mod; 
Original start date: Aug-03; 
Original completion date: May-09; 
Original budget[B]: $201.9; 
New APB[C] date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]; [Empty]; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: TFM-I; 
Original start date: Aug-05; 
Original completion date: Apr-10; 
Original budget[B]: $398.1; 
New APB[C] date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]; [Empty]; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: VRRP Next Generation; 
Original start date: Mar-06; 
Original completion date: May-13; 
Original budget[B]: $48.1; 
New APB[C] date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]; [Empty]; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: WSP Tech Refresh; 
Original start date: Mar-06; 
Original completion date: Feb-09; 
Original budget[B]: $6.1; 
New APB[C] date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]; [Empty]; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Program[A]: VSCS Tech Refresh Phase 2; 
Original start date: Aug-06; 
Original completion date: June-12; 
Original budget[B]: $83.6; 
New APB[C] date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]; [Empty]; 
New APB date: [Empty]; 
Revised completion date: [Empty]; 
Revised budget: [Empty]. 

Source: GAO based on ATO data. 

[A] The full name of the acquisition programs listed above are as 
follows: 

STARS: Standard Terminal Automation Replacement System: 

NEXCOM: Next Generation Air-to-Ground Communication System: 

OASIS: Operational and Supportability Implementation System: 

ITWS: Integrated Terminal Weather System: 

WAAS: Wide Area Augmentation System: 

FTI: FAA Telecommunications Infrastructure: 

ASWON: Aviation Surface Weather Observation Network: 

NIMS II: National Airspace System Infrastructure Management System- 
Phase 2: 

WARP: Weather and Radar Processor: 

RCE: Radio Control Equipment: 

ATCBI: Air Traffic Control Beacon Interrogator Replacement: 

ASR-11: Airport Surveillance Radar - Model 11: 

LAAS: Local Area Augmentation System: 

HOCSR: HOST/Oceanic Computer System Replacement: 

AMASS: Airport Movement Area Safety System: 

LLWAS: Low Level Wind-shear Alert System: 

ASDE-X: Airport Surface Detection Equipment - Model X: 

UHF Replace: Ultra High Frequency Replacement: 

CPDLC: Controller-Pilot Data Link Communications: 

BUEC: Back-Up Emergency Communications: 

ATOP: Advanced Technologies and Oceanic Procedures: 

PRM: Precision Runway Monitor: 

ECG: En Route Communication Gateway: 

URET: User Request Evaluation Tool: 

TMA: Traffic Management Advisor: 

ERAM: En Route Automation Modernization: 

En Route System Mod: En Route Control Center System Modernization: 

TFM-I: Traffic Flow Management-Infrastructure: 

VRRP Next Generation: Voice Recorder Replacement Program Next 
Generation: 

WSP Tech Refresh: Weather Systems Processor Tech Refresh: 

VSCS Tech Refresh Phase 2: Voice Switching and Control System Tech 
Refresh Phase 2: 

[B] All figures are expressed in millions of nominal dollars. 

[C] APB: acquisition program baseline: 

[D] Includes $80.9 million for the ASDE-3X baseline approved in June 
2002, which added ASDE-X capabilities to seven ASDE-3 sites. The ASDE- 
X and ASDE-3X acquisitions were combined in the September 2005 
rebaselining. 

[End of table] 

[End of section] 

Appendix III: Performance Measures from the FAA's Fiscal Year 2006 
Performance and Accountability Report: 

1. Commercial air carrier fatal accident rate: 

2. General aviation fatal accidents: 

3. General aviation Alaska accidents: 

4. Runway incursions (rate): 

5. Commercial space launch accidents: 

6. Operational errors (rate): 

7. Safety risk management (number of changes): 

8. Average daily airport capacity (35 Operational Evolution Plan 
[Footnote 27] (OEP) airports): 

9. Average daily airport capacity (eight metropolitan areas): 

10. Annual service volume: 

11. Adjusted operational availability (35 OEP airports): 

12. National airspace system on-time arrivals: 

13. Noise exposure: 

14. Aviation fuel efficiency: 

15. Aviation safety leadership: 

16. Bilateral safety agreements: 

17. External funding: 

18. Global positioning system-based technologies: 

19. Employee attitude survey (cumulative percent increase): 

20. Cost control (number of activities per organization): 

21. Critical acquisitions on budget: 

22. Critical acquisitions on schedule: 

23. Information security: 

24. Customer satisfaction (American Customer Satisfaction Index): 

25. Cost-reimbursable contracts: 

26. Mission-critical positions: 

27. Reducing workplace injuries: 

28. Clean audit with no material weaknesses: 

29. Grievance processing time: 

30. Air traffic controller hiring plan (within 5 percent of plan): 

[End of section] 

Appendix IV: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Gerald Dillingham, (202) 512-2834 or [email protected]: 

Staff Acknowledgments: 

In addition to the contact named above, key contributors to this report 
were Faye Morrison (Assistant Director), David Best, Elizabeth Curda, 
Elizabeth Eisenstadt, David Hooper, Edmond Menoche, Sara Ann 
Moessbauer, Colleen Phillips, and Taylor Reeves. 

[End of section] 

Footnotes: 

[1] GAO, National Airspace System: FAA Has Made Progress but Continues 
to Face Challenges in Acquiring Major Air Traffic Control Systems, GAO-
05-331 (Washington, D.C.: June 10, 2005); GAO, Air Traffic Control: 
FAA's Acquisition Management Has Improved, but Policies and Oversight 
Need Strengthening to Help Ensure Results, GAO-05-23 (Washington, D.C.: 
Nov. 12, 2004). 

[2] ATO uses the words "critical" and "major" interchangeably in 
measuring its acquisition performance and has used both terms in its 
goals for system acquisitions at different points in time. In this 
report, we use major unless referring to or quoting ATO documents which 
used the term critical. 

[3] In December of 2000, President Clinton issued an executive order 
and Congress passed supporting legislation, which together gave FAA the 
authority to create the performance-based ATO to control and improve 
FAA's management of ATC modernization. 

[4] GAO, Tax Administration: IRS Needs to Further Refine Its Tax Filing 
Season Performance Measures, GAO-03-143 (Washington, D.C.: Nov. 22, 
2002). 

[5] A valid measure provides an accurate representation of what is 
being measured. 

[6] 49 U.S.C. ï¿½40110. 

[7] A baseline is a formal, management-approved document that details, 
among other things, an acquisition's estimated budget and schedule. The 
JRC is an executive body consisting of associate and assistant 
administrators, acquisition executives, the chief financial officer, 
the chief information officer, and legal counsel. The JRC makes 
corporate-level decisions, including those that determine whether an 
acquisition meets a mission need and should proceed. 

[8] FAA, 2007-2011 FAA Flight Plan: Charting the Path for the Next 
Generation (Washington, D.C.) 

[9] GAO-03-143. 

[10] The ninth key attribute is balance, which exists when a suite of 
measures ensures that an organization's various priorities are covered. 
ATO has other performance measures that it applies to its acquisitions, 
but which are not reported in its annual Performance and Accountability 
Report. In this report, we focus on the two acquisition performance 
measures that ATO includes in its reporting--the percentage of major 
acquisitions on budget and the percentage of major acquisitions on 
schedule. Because we are not examining ATO's full suite of performance 
measures for acquisitions, we did not consider the key attribute of 
balance. 

[11] OMB Circular A-11, Part 7, Section 300. 

[12] Under 49 U.S.C. ï¿½ 40110(d)(2)(C), the Federal Acquisition 
Streamlining Act of 1994 (Public Law 103-355), except for section 315, 
does not apply to the acquisition management system developed by FAA. 
However, FAA is required to terminate or consider terminating ATC 
modernization acquisition programs that exceed established baselines by 
specific percentages. 49 U.S.C. ï¿½ 40121. 

[13] ATO uses the January and August CIPs to coincide with the timing 
of key events in the federal budget cycle. The January CIP will 
normally reflect the current year's appropriation, while the August CIP 
contains budget targets for future years provided by OMB. 

[14] In fiscal year 2006, ATO reported on the performance of only 29 
programs. 

[15] OMB, Circular No. A-11, Part 7: Planning, Budgeting, Acquisition, 
and Management of Capital Assets (Washington, D.C.: July 2003). 

[16] This amount is reported in constant dollars. Using constant 
dollars removes the effects of inflation and shows all dollars at the 
value they would have in a selected base year. 

[17] According to ATO officials, the January CIP reflects each 
program's most current estimated budget at completion, as stated in the 
program's most recent baseline document. 

[18] In addition, the Wide Area Augmentation System (WAAS) exceeded its 
original budget estimate by $2.3 billion primarily because the original 
estimate included only acquisition costs, and not total life cycle 
costs. 

[19] Earned value management compares the actual work performed at 
certain stages of a job to its actual costs--rather than comparing 
budgeted and actual costs, the traditional management approach to 
assessing progress. By measuring the value of the work that has been 
completed at certain stages in a job, earned value management can alert 
program managers, contractors, and administrators to potential cost 
growth and schedule delays before they occur and to problems that need 
correcting before they worsen. We have begun work examining FAA's 
implementation of earned value management. 

[20] 49 U.S.C. ï¿½40121 states that an agency must report to Congress 
when any program exceeds costs by more than 50 percent and is not 
terminated. 

[21] GAO, National Airspace System: Transformation Will Require 
Cultural Change, Balanced Funding Priorities, and Use of All Available 
Management Tools, GAO-06-154 (Washington, D.C.: Oct. 14, 2005). 

[22] GAO-05-331. 

[23] GAO, Tax Administration: IRS Needs To Further Refine its Tax 
Filing Season Performance Measures, GAO-03-143 (Washington, D.C.: Nov. 
22, 2002). 

[24] GAO, National Airspace System: Transformation Will Require 
Cultural Change, Balanced Funding Priorities, and Use of All Available 
Management Tools, GAO-06-154 (Washington, D.C.: Oct. 14, 2005). 

[25] GAO, National Airspace System: FAA Has Made Progress but Continues 
to Face Challenges in Acquiring Major Air Traffic Control Systems, GAO-
05-331 (Washington, D.C.: June 10, 2005). 

[26] GAO, Defense Acquisitions: Information for Congress on Performance 
of Major Programs Can Be More Complete, Timely, and Accessible, GAO-05-
182 (Washington, D.C.: Mar. 28, 2005). 

[27] The OEP is now called the Operational Evolution Partnership. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "Subscribe to Updates." 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office: 
441 G Street NW, Room LM: 
Washington, D.C. 20548: 

To order by Phone: 
Voice: (202) 512-6000: 
TDD: (202) 512-2537: 
Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: [email protected]: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Gloria Jarmon, Managing Director, [email protected]: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, [email protected]: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: 

*** End of document. ***