Air Traffic Control: FAA's Acquisition Management Has Improved,  
but Policies and Oversight Need Strengthening to Help Ensure	 
Results (12-NOV-04, GAO-05-23). 				 
                                                                 
The Federal Aviation Administration's (FAA) multibillion-dollar  
effort to modernize the nation's air traffic control (ATC) system
has resulted in cost, schedule, and performance shortfalls for	 
over two decades and has been on GAO's list of high-risk federal 
programs since 1995. According to FAA, performance shortfalls	 
were due, in part, to restrictions imposed by federal acquisition
and personnel regulations. In response, Congress granted FAA	 
exemptions in 1995 and directed it to develop a new acquisition  
management system. In this report, GAO compared FAA's AMS with	 
(1) the FAR and (2) commercial best practices for major 	 
acquisitions, and (3) examined FAA's implementation of AMS and	 
its progress in resolving problems with major acquisitions.	 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-05-23						        
    ACCNO:   A13534						        
  TITLE:     Air Traffic Control: FAA's Acquisition Management Has    
Improved, but Policies and Oversight Need Strengthening to Help  
Ensure Results							 
     DATE:   11/12/2004 
  SUBJECT:   Air traffic control systems			 
	     Best practices					 
	     Comparative analysis				 
	     Comparative benchmarking products			 
	     Federal procurement				 
	     Performance measures				 
	     Procurement policy 				 
	     Procurement practices				 
	     FAA Acquisition Management System			 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-05-23

     

     * Report to the Chairman and Ranking Minority Member, Committee on
       Governm\ent Reform, House of Representatives
          * November 2004
     * AIR TRAFFIC CONTROL
          * FAA's Acquisition Management Has Improved, but Policies and
            Oversight Ne\ed Strengthening to Help Ensure Results
     * Contents
          * Results in Brief
          * Background
          * AMS Is Broader and Less Prescriptive Than the FAR
               * AMS Addresses Both Procurement and Project Management,
                 Whereas the FAR F\ocuses Primarily and in Far Greater Detail
                 on Procurement
               * AMS Provides Broad Guidance While the FAR Establishes
                 Detailed Requireme\nts, but Managers Have Flexibility under
                 Both
          * AMS Provides Some Discipline but Does Not Ensure a
            Knowledge-Based Appro\ach to Acquisition
               * Best Practices for Managing Acquisitions Call for a
                 Knowledge-Based Appr\oach, Including Criteria for Knowledge
                 Needed and Oversight at the Corpo\rate Executive Level
               * AMS Has Some Good Features but Does Not Ensure That High
                 Levels of Knowl\edge Are Attained Before Major Commitments
                 Are Made
          * As Implemented, AMS Has Not Resolved Long-standing Acquisition
            Problems,\ but FAA Is Beginning to Focus More on Results
               * Our Reviews of Seven Major Systems Show That Problems with
                 Requirements \and Software Management Persist under AMS
               * Inadequate Development or Definition of Requirements Led to
                 Requirements\ Growth or Unplanned Work for Five Acquisitions
                    * FAA Underestimated Software Complexity for Three
                      Systems
                    * ATC Systems Have Required Multiple Rebaselining
                      Decisions to Address Del\ays and Cost Growth
               * Internal and External Reviews Have Found That FAA Has Made
                 Some Progress\ but Continues to Experience Problems in
                 Acquiring Major ATC System unde\r AMS
               * FAA's ATO Is Taking Steps to Improve Major ATC Acquisitions
                    * Improvements to Requirements Development
                    * Improvements to Managing Software and System
                      Acquisition and Development\
                    * Improvements to Estimating Costs
                    * Other Improvement Efforts
          * Conclusions
          * Recommendations for Executive Action
          * Agency Comments
     * FAA Has Begun Analyzing Spending Trends to Take a More Strategic
       Approac\h to Procurement
     * Objectives, Scope, and Methodology
     * Comparison of the Scope and Flexibility of FAA's Acquisition
       Management \System and the Federal Acquisition Regulation Process
          * Background
          * AMS Defines an Investment/Life-Cycle Project Management System
          * Only a Portion of AMS Deals Directly with the Procurement Process
          * AMS States a Nonregulatory FAA Policy
          * AMS Chapter 3 Parallels a Subset of the FAR
          * AMS Includes a Less Rigorous Competition Requirement Than Does
            the FAR
          * The FAR Gives Procurement Professionals Tighter Control over
            Procurement\ Decisions
          * Although FAA Project Managers View AMS as More Efficient and
            Flexible Th\an the FAR, Some Procurement Officials We Interviewed
            Do Not Agree
     * How FAA's Acquisition Policy Adapted Key Recommendations Made by GAO
       and\ DOT (1996-2003)
          * FAA Refined AMS in Response to Recommendations
     * Status of the Seven ATC Modernization Acquisitions That GAO Reviewed
     * Key Contributors
          * GAO Contacts
          * Staff Acknowledgments
     * app4new.pdf
          * How FAA's Acquisition Policy Adapted Key Recommendations Made by
            GAO and\ DOT (1996-2003)
               * FAA Refined AMS in Response to Recommendations

Report to the Chairman and Ranking Minority Member, Committee on
Government Reform, House of Representatives

November 2004

AIR TRAFFIC CONTROL

FAA's Acquisition Management Has Improved, but Policies and Oversight Need
Strengthening to Help Ensure Results

Contents

Tables

Figures

Abbreviations

November 12, 2004erLetter

The Honorable Tom Davis Chairman The Honorable Henry A. Waxman Ranking
Minority Member Committee on Government Reform House of Representatives

In late 1981, the Federal Aviation Administration (FAA) began a
modernization program to replace and upgrade the National Airspace
System's (NAS) equipment and facilities to meet the expected increase in
traffic volume, enhance the margin of safety, and increase the efficiency
of the air traffic control (ATC) system-the principal component of the
NAS. Historically, the modernization program has experienced cost
overruns, schedule delays, and performance shortfalls of large proportions
and has been on our list of high-risk programs since 1995. To date, FAA
has spent $41 billion and expects to spend an additional $7.6 billion
through fiscal year 2007 to, among other things, finalize key
modernization projects designed to replace radar, navigation,
communications, and information-processing systems.1

According to FAA, the performance shortfalls in its modernization program
were due, in part, to restrictions imposed by federal acquisition and
personnel requirements. In response, Congress passed legislation in 1995
that granted FAA unique acquisition and personnel exemptions, or
flexibilities, and directed FAA to develop a new acquisition management
policy. FAA issued its new acquisition management policy, called the
Acquisition Management System (AMS), in 1996 and began using the new
system instead of the Federal Acquisition Regulation (FAR). To further
address long-standing weaknesses in the ATC modernization program, the
President and Congress in 2000 directed FAA to reorganize and establish a
new organization. FAA has just begun to do so.

Now that FAA has had several years to implement the earlier procurement
flexibilities, as well as some time to reorganize, some results of its
acquisition reform should be discernable. Moreover, FAA's experiences in
exercising its acquisition flexibilities could provide valuable
information to Congress in overseeing the use of these flexibilities.

You asked us to review the steps that FAA has taken to reform its
acquisition of major ATC systems and the impact of the reforms on FAA's
acquisition outcomes. Specifically, you asked us to (1) compare the scope
and flexibility of AMS and the FAR, (2) compare AMS with commercial best
practices for major acquisitions, and (3) examine FAA's implementation of
AMS and progress in addressing long-standing problems with major
acquisitions. In addition, you asked us to review FAA's general
procurement of goods and services; we cover this topic in appendix I.

To address the first objective, we compared the topics addressed by, and
the implementation options afforded to contracting and procurement
officials under AMS and the FAR. To address the second objective, we used
a model of best practices that we derived from our body of work on how
leading private firms manage costly and complex product developments and
how the Department of Defense (DOD) manages major weapon systems
acquisitions.2 We used this model to assess the extent to which FAA's
acquisition management policy mirrors the acquisition policies of
high-performing organizations in the public and private sectors. This
model consists of four phases: (1) concept and technology development; (2)
product development, which includes both integration and demonstration
activities; (3) production; and (4) operations and support. In between
these four phases are three key knowledge decision points at which
commercial firms and the government must have sufficient knowledge to make
large investment decisions. To address the third objective, we selected
the seven ATC systems with the largest budgets to explore the results of
FAA's implementation of its acquisition management policy and procedures
and to determine how FAA has addressed issues found to have contributed to
cost, schedule, or performance problems. In selecting these seven systems,
we ensured that some were initiated before and some after April 1996, when
FAA implemented AMS. While the results of these analyses are not
generalizable to all of FAA's major ATC acquisitions, they indicate the
extent to which the agency has made progress in addressing long-standing
problems we have identified. To further assess both the implementation and
the impact of FAA's acquisition reforms, we reviewed our work on FAA's
major ATC acquisition efforts since 1996 as well as the work of the
Department of Transportation's Office of Inspector General (DOTIG), FAA,
and others. We also reviewed the actions that FAA has taken to refine AMS
in response to internal and external reviews. Finally, to review FAA's
procurement of goods and services across the agency, we used a commercial
best-practices model for taking a more strategic approach to procurement,
along with interviews with key agency officials, to determine whether FAA
has begun to analyze spending trends to identify opportunities to leverage
its buying power. We conducted our work from December 2003 through
November 2004 in accordance with generally accepted government auditing
standards. See appendix II for additional information on our objectives,
scope, and methodology.

Results in Brief

AMS consists of broad guidance for acquisition life-cycle management-from
defining the requirements for a system through fielding (deploying) and
decommissioning it (removing it from service). This broad guidance
contrasts with the rather more detailed and prescriptive
contract-formation and contract-administration requirements contained in
the FAR. AMS is broader in scope because it addresses, among other areas
of life-cycle management, both contract and program management, providing
both policies and procedures for contracting and a toolset of recommended
practices for managing individual acquisition projects over their life
cycles. By contrast, the FAR focuses in far greater detail on contracting
policies and procedures. FAA managers believe they have greater
flexibility in interpreting and applying AMS than they would have under
the FAR, in part because, in areas addressed by both, AMS is less
directive than the FAR. For example, although AMS states a "preference"
for competition, FAA personnel may use single-source contracting when
necessary to fulfill FAA's mission. By contrast, other federal agency
contracting officials operating under the FAR are generally required to
seek "full and open competition"-a more rigorous standard. These other
agency officials can generally use sole-source or limited-competition
contracting only after higher-level agency procurement officials have
approved a written justification. In addition, FAA contracting personnel
operate as part of acquisition teams that are responsible to program
managers; under the FAR, contracting decisions are made by contracting
personnel who are responsible only to contracting officials. Nonetheless,
the FAR also affords flexibility because it encourages innovation and
addresses a wide selection of contracting methods; therefore, procurement
officials can choose the approach that they consider most appropriate to
their procurement. According to some current and former FAA procurement
officials with experience in using both the FAR and AMS, the FAR may
appear inflexible and cumbersome to inexperienced managers, but those who
are familiar with it can navigate it effectively.

AMS provides some discipline through its various phases, activities, and
decision points for acquiring major ATC systems; however, it does not
ensure the use of a knowledge-based approach found in the best practices
for managing commercial product developments and DOD acquisitions3 that we
have identified in numerous past reports. Commercial best practices call
for specific knowledge to be captured and used by corporate-level
decision-makers to determine whether a product has reached a level of
development (product maturity) sufficient to demonstrate its readiness to
move forward in the acquisition process. The capture of such knowledge and
its use by executives helps to avoid cost overruns, schedule slips, and
performance shortfalls that can occur if decision-makers commit to a
system design before acquiring critical technology, design, or
manufacturing knowledge. AMS has some good features, which indicate a
process that has some elements of discipline. For example, like the best
practices model, AMS identifies critical junctures that it terms "decision
points," the first three of which call for the preparation of detailed
technical and programmatic information that FAA's corporate
executive-level body, the Joint Resources Council,4 can use to assess
whether or not FAA should initiate an acquisition program. However, AMS
departs from recognized best practices primarily by (1) not requiring the
attainment of specific knowledge satisfying explicit written criteria for
decision-makers to use at each key decision point and (2) not requiring
corporate executive-level oversight at all key decisions. For example, AMS
allows the Joint Resources Council to delegate two key decisions-the
decision to begin production and the decision to place a system in
service. FAA maintains that this approach gives program managers
flexibility, expedites decision-making, and allows those executives with
the most knowledge about a major acquisition to make key decisions about
its continued development. FAA's reliance on delegation assumes that
managers will inform their superiors if they are unable to meet the
performance schedules and system requirements approved by the Joint
Resources Council. However, best practices call for more than this,
including the use of measurable criteria at key points in the acquisition
process to ensure that specific knowledge has been captured and the
independent review of this knowledge by corporate executive-level
decision-makers before the acquisition moves forward in its development.
These criteria and reviews are particularly important for acquisitions
that require a large funding commitment, such as those that include the
production of multiple costly units (e.g., radars and controller
workstations). In addition, oversight at the corporate-executive or
agencywide level is needed to ensure consideration of an acquisition's
likely impact on other agency projects or operations. These departures
from best practices put FAA's major ATC acquisitions at risk of cost,
schedule, and performance shortfalls. We are making recommendations to the
Secretary of Transportation to align AMS more closely with commercial best
practices.

According to our review of seven major ATC systems and analysis of FAA's
performance in acquiring major systems, AMS has not resolved management
problems that FAA experienced before it implemented AMS, but the agency is
beginning to focus more on the expected results of its major acquisitions.
(See table 5.) Specifically, our review found that AMS did not call for
requirements that were specific enough to minimize the development of
further requirements (requirements growth) or unplanned work in five of
these systems. This lack of specificity resulted in the inadequate
development or definition of requirements, requirements growth, unplanned
work, or a reduction in performance for five of these systems. In
addition, for three of these systems, FAA underestimated the difficulty of
modifying available software to fulfill its mission needs. Consequently,
FAA encountered unexpected software development needs, higher costs, and
schedule delays. Because AMS guidance was not sufficient to account for
the risks associated with modifying available software, the two systems we
reviewed that were initiated after AMS's implementation-though currently
meeting cost and schedule milestones-are nevertheless showing symptoms of
FAA's past problems with developing requirements and managing software. It
is too soon to tell if these two systems will remain within their cost,
schedule, and performance parameters. In addition, our work on FAA's major
acquisitions, along with that of the DOTIG and others, has shown that many
of the problems FAA experienced in acquiring major systems before 1996
persist under AMS and that effective acquisition management, rather than
the use of a specific contracting process (e.g., the FAR or AMS) is the
key to successful acquisitions. To its credit, FAA is beginning to focus
more on results, largely through its new Air Traffic Organization, which
has been charged with taking a more performance-based approach to managing
the agency's major acquisitions. This approach includes implementing a
training framework for FAA's acquisition workforce. While FAA has taken
some steps to develop an evaluation program with criteria for measuring
the extent to which this framework is achieving organizational goals by
improving the knowledge base of FAA's acquisition workforce, at the time
of our audit FAA had no plans to conduct a comprehensive evaluation. We
are making recommendations to the Secretary of Transportation to improve
FAA's development of requirements and management of complex software, and
to comprehensively evaluate FAA's implementation of the training framework
to ensure that it is having the intended effect of improving the knowledge
base of FAA's acquisition workforce. In commenting on a draft of this
report, FAA said that it generally agreed with the report's contents and
said that our recommendations would be helpful to them as they continue to
refine AMS.

Background

Maintaining that federal procurement requirements contributed to some of
its cost, schedule, and performance problems in the 1980s and early 1990s,
FAA sought a statutory exemption from the federal acquisition system,5
including the FAR, and those parts of title 5 of the United States Code,
parts II and III, that govern federal civilian personnel management.
According to FAA, exemptions from these requirements would enable it to
streamline its acquisition processes, be more responsive to the airline
industry's needs, and increase the efficiency of ATC operations while
maintaining safety. Congress enacted legislation in November 1995 that
exempted FAA from key federal procurement statutes and the FAR, and
directed FAA to develop a new acquisition management system. In response
to these legislative initiatives, FAA implemented a new, streamlined
acquisition process-the Acquisition Management System (AMS)- on April 1,
1996.

We developed a knowledge-based model of commercial best practices based on
our findings about how leading private firms manage costly and complex
acquisitions effectively-that is, within cost, schedule, and performance
targets. The use of this knowledge-based model has been found to reduce
the risks associated with developing products and increase the likelihood
of successful outcomes. The model divides the product development cycle
into four phases and related activities. Table 1 presents these phases and
activities and explains what takes place during each.

Table 1: Structure of Best Practices Model for Major Product Developments

Phase/Activity            What occurs during this phase or activity        
1.Concept and technology  Leading companies work to understand their       
development               mission needs and confirm that the technologies  
                             to be used are mature; that is, the technologies 
                             needed to meet essential product requirements    
                             have been demonstrated to work in their intended 
                             environment.                                     
2. Product development    
o Integration             Components and subsystems are integrated into    
                             the product to stabilize the overall system      
                             design and show that the design can meet the     
                             product requirements.                            
o Demonstration           Tests show that the product will work as         
                             required and can be manufactured within targets. 
3. Production             Operational test articles are built.             
4. Operations and support Our best practices model does not explicitly     
                             cover operations and support activities;         
                             however, this phase focuses on maintenance of    
                             the system through its retirement.               

Source: GAO.

AMS provides guidance for selecting and overseeing investments over their
life cycle. Like our best practices model, it is divided into phases and
activities, although the divisions sometimes occur at different points.
Table 2 summarizes AMS's phases and activities.

Table 2: Structure of AMS

Phase/Activity          What occurs during this phase or activity          
Needs and solution      
identification          
o Mission analysis      FAA identifies a capability shortfall and          
                           determines that it needs an investment to better   
                           carry out its mission. Recently, FAA began         
                           analyzing its mission needs within the context of  
                           its overall goals for the National Airspace        
                           System.                                            
o Investment analysis   FAA, using an investment analysis team, evaluates  
                           alternatives, selects practical and affordable     
                           solutions, and develops a baseline of cost,        
                           schedule, and performance requirements. This       
                           document is called the acquisition program         
                           baseline.                                          
Solution implementation 
o System integration    Both hardware and software components and          
                           subsystems are integrated into a product. Also,    
                           intra- and intersystem compatibility are tested    
                           and analyzed.                                      
o System demonstration  Tests show that the product can work as required   
                           and be manufactured within targets.                
o System production     All activities are carried out to produce needed   
                           quantities. Each end item is tested before it      
                           leaves the factory to verify that it conforms to   
                           specifications and is free from manufacturing      
                           defects.                                           
In-service management   All required activities are carried out, including 
                           directly operating, providing maintenance          
                           functions (both scheduled and unscheduled), and    
                           furnishing technical and logistics support for the 
                           maintenance of FAA systems, subsystems, services,  
                           or equipment.                                      

Source: FAA.

To implement the new, performance-based organization for managing ATC
modernization and operations, as the President and Congress directed in
2000, FAA appointed a chief operating officer in August 2003 and formally
established the Air Traffic Organization (ATO) in February 2004. ATO,
under the direction of a six-member executive council, is now responsible
for further implementing acquisition reforms for major ATC systems.

AMS Is Broader and Less Prescriptive Than the FAR

AMS establishes an acquisition life-cycle management system that
encompasses both contracting and program management, whereas the FAR is
primarily a contracting system that focuses on contract formation and
contract administration. As a result, AMS is broader in scope than the
FAR. See figure 1. In addition, AMS takes the form of guidance. This
guidance is expressed in documentation of FAA policy, handbooks,
templates, flowcharts, forms, and standard contract language. It is not
regulatory. By contrast, the FAR is a set of published regulations-a legal
foundation that has the force and effect of law for the federal agencies
that

are required to follow it.6 Furthermore, the FAR is more detailed and
prescriptive in establishing contracting requirements and can require more
administrative involvement. This fundamental difference between AMS and
the FAR may suggest to some that AMS is more flexible. FAA personnel can
choose how to apply AMS's provisions to a major acquisition. Nonetheless,
procurement officials under the FAR also have flexibility because the FAR
encourages innovation consistent with its direction (and other applicable
legal requirements), provides a wide selection of contracting solutions,
and permits contracting officials to choose the methods that they consider
most suitable for a given situation.

Figure 1: Scope of AMS and the FAR

Note: AMS provides policy for the four phases of life-cycle management, as
well as 14 functional areas, (e.g., test and evaluation, human factors,
procurement, real estate, security, and systems engineering).

aThe NAS in-service decision is a key program milestone that authorizes
the deployment of a system into the National Airspace System after
thoroughly testing the system to verify its operational readiness.

AMS Addresses Both Procurement and Project Management, Whereas the FAR
Focuses Primarily and in Far Greater Detail on Procurement

AMS comprises six policy sections and five appendixes.7 The procurement
policy section of AMS covers a range of topics, including contract funding
and administration, contracting with small and disadvantaged businesses,
and compliance with labor laws. According to this section, competition is
FAA's preferred method of contracting, but single-source contracting is
permitted when appropriate to fulfill the agency's mission. This policy
section also describes the procurement of commercially available or
nondevelopmental items.

Other sections of AMS cover project management tools that the FAR does not
address, such as investment analysis, configuration management,8 and
integrated logistics support.9 AMS also addresses areas that fall outside
project management and procurement, including real property management-an
area that becomes important when FAA must lease or purchase real property
so that it can install ATC systems such as radars or antennas on property
that it does not currently own. FAA's policy directs FAA staff to "conduct
this business in a fair and equitable manner following best practices."

Although the FAR includes requirements that address procurement planning10
and major systems acquisition,11 it does so only in the context of
government procurement policy and procedure. Agencies subject to the FAR
find the broader program planning requirements, which appear in AMS but
not in the FAR, in documents such as the Office of Management and Budget's
Circular A-109 and in their own planning guidance. For

example, DOD has issued a series of directives and instructions on this
subject.12

The contracting procedures set forth in section 3 of AMS do not prescribe
detailed contracting procedures for various categories of procurements, as
do those detailed under the FAR. Instead, AMS provides two basic
contracting models for obtaining products and services through FAA's
contracting process. The first model is called "Complex and Noncommercial
Source Selection" and is used for complex, large-dollar, developmental,
noncommercial items and services. This is the model that typically would
be used for investments approved by the Joint Resources Council. The
second model is called the "Commercial and Simplified Purchase Method" and
is typically used for commercial items that are less complex and less
costly. Procurements of such products or services may be routine in nature
and are generally purchased on a fixed-price basis. Generally, source
selection under AMS follows a screening process, with the awardee being
selected on a "best value" basis from among those who remain in
consideration when the selection is made.

AMS Provides Broad Guidance While the FAR Establishes Detailed
Requirements, but Managers Have Flexibility under Both

AMS sets out a nonregulatory FAA policy that is binding on FAA personnel
as FAA employees. AMS also sets out other guidelines that FAA states
should be followed unless there is a rational basis for doing otherwise.
AMS is subject to such internal controls and enforcement as the
Administrator decides and to general overarching legal requirements, such
as the Government Performance and Results Act of 1993 (GPRA).13  FAA has
also deemed certain acquisition laws applicable to its procurements
(sometimes with modifications), such as the Service Contract Act.14 There
is also a legal requirement, created by the 1995 legislation exempting FAA
from the FAR, that small and socially or economically disadvantaged firms
be given all reasonable opportunities to receive contract awards. FAA has
adopted a dispute resolution process with some legal underpinnings.15
Otherwise, as the preface to AMS states, "nothing in this document creates
or conveys any substantive [legal] rights." In short, although FAA is
subject to the general legal requirement that government decisions cannot
be arbitrary or capricious, AMS does not establish regulatory requirements
for the conduct of procurements and does not create or convey substantive
legal rights.

In contrast to AMS, the FAR is a set of published regulatory requirements.
It has the force and effect of law, and agencies that are subject to it
are bound to follow it. The FAR's requirements provide for a range of
procurement strategies and approaches. In addition to negotiated
procurement methods, it allows two-step sealed-bid and two-phase
design-build methods,16 among others. It includes streamlined procedures
for soliciting and evaluating offers to furnish commercial items, as well
as permits the use of simplified acquisition procedures in a broad range
of procurements. Furthermore, the FAR supports a diverse selection of
available contract types, product-testing tools, and other tools that an
agency's contracting personnel may select when conducting an acquisition
to meet the agency's needs.

Although contracting personnel in agencies subject to the FAR are required
to comply with it, they enjoy broad discretion in their management of
procurements. For example, the FAR allows wide latitude in drafting
requirements statements, from performance-based statements of work to
design specifications as necessary. It allows broad discretion in framing
solicitations and in conducting procurements, including scoring proposals,
determining how negotiations will be conducted, eliminating firms whose
proposals are not competitive, and selecting the awardees whose proposals
afford the government the best value when evaluated against the selection
criteria established in the solicitations.

Because AMS consists of broad guidance while the FAR comprises detailed
and prescriptive regulatory requirements, FAA managers view AMS as giving
them more flexibility than they would have under the FAR, particularly in
two areas-competition and oversight. Whereas the FAR generally requires
full and open competition, AMS calls for providing "reasonable access to"
competition to firms interested in obtaining contracts-a less rigorous
standard than full and open competition. AMS further states that the
"preferred" method of selecting sources is to compete requirements among
two or more sources. By contrast, full and open competition requires that
all responsible sources be permitted to compete.17 Under AMS, there is no
policy that firms that want to participate actually get a chance to do so.
Rather, FAA told us that its system is beneficial because the agency can
use screening requests to preselect competing firms, eliminating those
firms that FAA believes are not likely to receive an award. The following
example illustrates the differences between AMS and the FAR in their
respective requirements on exceptions to competition. FAA may contract
with a single source when this approach is determined to be in the best
interest of FAA.18 The FAR, however, allows exceptions to full and open
competition only for certain specified conditions (such as unusual and
compelling urgency or the availability of only one source). The FAR
describes in detail the circumstances of these conditions and the
requirements for using them as justification for not providing for full
and open competition. The FAR also requires the contracting officer to
prepare a justification document that must generally be approved by
higher-level agency procurement officials (up to the agency's senior
procurement executive) depending on the estimated dollar value of the
procurement. The content of this justification is prescribed by the FAR.
When not providing for full and open competition, the contracting officer
is required under the FAR to solicit offers from as many potential sources
as is practicable under the circumstances. The FAR prohibits contracting
if the justification for less than full and open competition results from
a lack of advanced planning. For a more detailed comparison of AMS and the
FAR, see appendix III.

Although some of the FAA personnel we interviewed see AMS as more
efficient and flexible than the FAR, other current and former FAA
procurement officials we interviewed who have experience using both the
FAR and AMS did not agree that AMS is more flexible than the FAR.
According to these officials, the FAR may appear inflexible and cumbersome
to persons who lack experience with it, but those who are familiar with it
are able to navigate its complexities effectively. The FAR requires full
and open competition, but as experienced procurement personnel know, the
system does not break down when emergencies necessitate quick and decisive
action. For example, we recently reported that agencies generally complied
with applicable FAR requirements in awarding new contracts for work in
Iraq using other than full and open competition.19  In some circumstances,
the government's legitimate need for prompt action was sufficient to
justify selecting a contractor on an expedited basis from among the firms
that appeared able to meet the government's emergency need. In other
cases, the agencies reasonably determined that only one source could meet
their requirements.

AMS Provides Some Discipline but Does Not Ensure a Knowledge-Based
Approach to Acquisition

AMS provides some discipline through its various phases, activities, and
decision points for acquiring major ATC systems; however, it does not
ensure the use of a knowledge-based approach found in the best practices
for managing commercial product developments and DOD acquisitions that we
have identified in numerous past reports.20  Commercial best practices
call for specific knowledge to be captured and used by corporate-level
decision-makers to determine whether a product has reached a level of
development (product maturity) sufficient to demonstrate its readiness to
move forward in the acquisition process. The capture of such knowledge and
its use by executives helps to avoid cost overruns, schedule slips, and
performance shortfalls that can occur if decision-makers commit to a
system design before acquiring critical technology, design, or
manufacturing knowledge. The absence of these key best practices under AMS
puts FAA's major ATC acquisitions at risk of cost overruns, schedule
slips, and performance shortfalls.

Best Practices for Managing Acquisitions Call for a Knowledge-Based
Approach, Including Criteria for Knowledge Needed and Oversight at the
Corporate Executive Level

Commercial best practices call for managing acquisitions using a
knowledge-based approach, including (1) using established criteria to
attain specific knowledge at three critical junctures in the acquisition
cycle, which we call knowledge points, and (2) requiring oversight at the
corporate executive level for each of these knowledge points. For example,
at each knowledge point, successful product developers apply specific
indicators, or criteria, to determine whether they have attained the
knowledge they need to move to the next phase or activity in the
acquisition process. Such developers also conduct corporate
executive-level reviews to ensure that they obtain the insights and
perspectives of stakeholders throughout their organization. If the
knowledge attained does not meet the criteria for advancement or if the
executive reviewers determine that further development is inconsistent
with their priorities, the acquisition does not move forward. Table 3
summarizes the knowledge points, criteria, oversight reviews, and timing
of oversight reviews included in our model of best practices for major
acquisitions.

Table 3: Knowledge-Based Approach Called for in Our Best Practices Model

Knowledge    Criteria                      Oversight review Timing of      
point                                                       oversight      
                                                               review         
1. Resources o Match customers' needs with Executive-level  Knowledge      
and needs    available                     review required  point 1 should 
matched      resources-technology, design, to initiate the  precede the    
                time, and funding.            program.         commitment to  
                                                               begin product  
                o Demonstrate that                             development.   
                technologies needed to meet                    
                essential product                              
                requirements can work in                       
                intended environment.                          
                                                               
                o Complete a preliminary                       
                product design using systems                   
                engineering to balance                         
                customers' desires and                         
                available resources.                           
2. Product   o Complete 90 percent of      Executive-level  Knowledge      
design       design drawings by critical   review required  point 2 should 
stable       design review.                to move to       precede the    
                                              demonstration.   commitment to  
                o Obtain stakeholders'                         build          
                concurrence that drawings are                  prototypes to  
                complete and producible.                       demonstrate    
                                                               the design.    
                o Review subsystem and system                  
                designs.                                       
                                                               
                o Demonstrate with prototype                   
                that design meets users'                       
                requirements.                                  
                                                               
                o Identify critical                            
                manufacturing processes.                       
3.           o Demonstrate manufacturing   Executive-level  Knowledge      
Production   processes.                    review required  point 3 should 
processes                                  to move to       precede the    
mature       o Build and test production   production.      commitment to  
                prototypes.                                    begin          
                                                               production.    
                o Test                                         
                production-representative                      
                prototypes to achieve                          
                reliability goals.                             
                                                               
                o Test                                         
                production-representative                      
                prototypes to demonstrate                      
                product performance in                         
                operational environment.                       
                                                               
                o Collect statistical process                  
                control data.                                  

Source: GAO.

Experience with commercial best practices has shown that to the extent
that the level of knowledge called for at each knowledge point is not
attained, organizations take on risks in the form of unknowns that will
persist into the later stages of development, where they will take more
time and money to resolve if they become problems. Such problems lead to
cost increases and schedule delays.

AMS Has Some Good Features but Does Not Ensure That High Levels of
Knowledge Are Attained Before Major Commitments Are Made

AMS has some good features, including phases and key decision points
indicative of an acquisition process that has some elements of discipline;
however, AMS does not ensure that high levels of knowledge are attained
and that corporate executive-level reviews occur before major commitments
of agency resources are made. For example, like the best practices model,
AMS identifies critical junctures, which it terms "decision points." Three
of these decision points occur during the initial acquisition phase
(mission need, initial investment, and the final investment decision). A
fourth decision point occurs before production, and a fifth decision point
occurs before the start of the final acquisition phase (in-service
management). AMS also calls for detailed technical and programmatic
information that decision-makers can use at the first three decision
points to assess whether or not FAA should initiate an acquisition
program. This information includes a final requirements document, a final
acquisition program baseline, a final investment analysis report, an
acquisition strategy paper, and an integrated program plan. Finally, AMS,
like our best practices model, calls for senior executives to review the
information and determine whether the acquisition is ready to move
forward. The FAA executives who make the decisions at these points include
associate and assistant administrators, acquisition executives, the chief
financial officer, the chief information officer, and legal counsel; they
form the Joint Resources Council (JRC), FAA's senior decision-making body
for major ATC acquisitions. Table 4 summarizes this information.

Table 4: AMS's Decision Points, Information Sources, and Oversight Reviews

Decision point by            Information sources and oversight reviews     
phase/activity               
Phase: Needs and solution    
identification               
o Activity: Mission analysis 
Decision point: Mission need Information sources: Input from users in the  
decision                     field and mission need statement.             
                                                                              
                                Oversight review: JRC review called for to    
                                move from mission analysis to investment      
                                analysis.                                     
o Activity: Investment       
analysis                     
Decision Point: Initial      Information sources: Initial investment       
investment decision          analysis report, initial life-cycle program   
                                baseline for the most viable alternative,     
                                updated initial requirements document and     
                                action plan for final investment analysis.    
                                                                              
                                Oversight review: JRC review called for to    
                                select a preferred solution.                  
Decision point: Final        Information sources: Final requirements       
investment decision          document, final acquisition program baseline, 
                                final investment analysis report, acquisition 
                                strategy paper, integrated program plan.      
                                                                              
                                Oversight review: JRC review called for to    
                                move from investment analysis to solution     
                                implementation.                               
Phase: Solution              
implementation               
o Activity: System           
integration                  
o Activity: System           
demonstration                
Decision point: Production   Information sources: Determined by JRC.       
decision                                                                   
                                Oversight review: JRC may retain or delegate  
                                decision making authority.                    
o Activity: System           
production                   
Decision point by            Information sources and oversight reviews     
phase/activity               
Decision point: In-service   Information Sources: Determined by JRC.       
decision                                                                   
                                Oversight Review: JRC review called for to    
                                move from solution implementation to          
                                in-service management; however, the JRC may   
                                retain or delegate decision making authority. 
Phase: In-service management 

Source: GAO analysis of FAA data.

Note: In this report, we place FAA's "mission analysis" and "investment
analyses" activities in the "Needs and Solution Identification" phase to
facilitate comparison with the "concept and technology development" phase
in our best practices model. Similarly, we place "system integration" and
"system demonstration" in the solution implementation phase for
comparative purposes.

AMS departs from the best practices model in two key ways-it does not call
for high levels of knowledge to be attained at three critical junctures
(knowledge points), and does not call for corporate executive-level
oversight at one of five junctures. Specifically, AMS does not establish
explicit, written criteria for (1) the information needed to determine
technology maturity at solution implementation, (2) releasable drawings at
critical design review and production process controls at production. Our
best practices model calls for attaining specific knowledge and setting
out criteria for what information should be available to help
organizations minimize risks in the form of unknowns. Risks associated
with such unknowns can persist into the later stages of development, where
they can take more time and money to resolve if they become problems,
potentially leading to cost increases and schedule delays.

In addition, AMS does not provide for corporate executive-level oversight
reviews at two of the three key junctures where our best practices model
calls for such reviews. Although AMS calls for three Joint Resources
Council reviews during the initial acquisition phase-while our model calls
for a single corporate executive-level review-AMS allows the council to
delegate its oversight responsibility later in the acquisition process to
the program managers within the service organization responsible for an
acquisition. By contrast, our model calls for two corporate
executive-level reviews later in the acquisition process.

According to FAA, its approach gives program managers flexibility,
expedites decision-making, and allows the executives with the most
knowledge about a major acquisition to make key decisions about its
continued development. FAA's reliance on this approach assumes that the
program managers will inform higher-level managers if they are unable to
meet the performance schedules and systems requirements approved by the
Joint Resources Council. However, although program managers may have the
most knowledge about their particular acquisition, they may not have the
agencywide perspective of the Joint Resources Council members. Having an
agencywide perspective, including a broad understanding of an
acquisition's potential impact on other agency projects and operations, is
especially critical when an acquisition includes the production of
multiple units and requires a substantial commitment of agency resources,
as do FAA's primarily multimillion-dollar acquisitions, such as controller
workstations and radars.

Because decisions about moving a major acquisition forward require both a
program manager's specific knowledge of the acquisition itself and a
senior executive's understanding of the acquisition's potential impact on
other agency projects and operations, our best practices model calls for
both measurable criteria at key points in the acquisition process to
ensure that specific knowledge has been captured and corporate
executive-level reviews to ensure that senior decision-makers have the
opportunity to independently consider this knowledge. Without higher-level
reviews such as our best practices model recommends and the Joint
Resources Council could provide later as well as early in the acquisition
process, FAA cannot ensure that it has fully considered the impact of
advancing an acquisition on other agency projects and operations. This
opportunity for full consideration is a central advantage of managing
acquisitions as a portfolio, as we concluded in our August 2004 report on
FAA's information technology investment management process.21

Figure 2 contrasts FAA's process for reviewing an acquisition's progress
under AMS with the process that we found leads to successful commercial
acquisitions.

Figure 2: Review Process under Our Best Practices Model and under AMS

aTo facilitate the comparison of AMS with out best practices model in this
report, we have done the following: (1) placed FAA's "Mission Analysis"
and "Investment Analyses" activities in the "Needs and Solution
Identification" phase to make it comparable with the "concept and
technology development" phase in our best practices model; (2) depicted
only the final investment decision point, recognizing that the investment
analysis phase includes an initial investment decision; and (3) placed
"system integration" and "system demonstration" in the solution
implementation phase.

bAMS does not explicitly call for a design review decision point, which
would fall between system integration and system demonstration.

cThe in-service decision is a key program milestone. It authorizes the
deployment of a system into the National Airspace System. At times, the
JRC delegates its decision authority for the production and in-service
decisions to service organizations.

To its credit, FAA continues to improve its AMS process. For example, the
agency is currently modifying its mission needs activity to make the
selection of major ATC acquisitions more consistent with the overall goals
of modernizing the National Airspace System. In addition, the Air Traffic
Organization has established an executive council to review major
acquisitions before they are sent to the Joint Resources Council. This
review is designed to screen acquisitions to determine which ones are
important enough to warrant higher-level review by the Council. Finally,
FAA is currently revising AMS to bring it in line with the Office of
Management and Budget's guidance. Specifically, the agency is
incorporating OMB Exhibit 300, which provides the investment
justifications and management plans required for major ATC acquisitions.

As Implemented, AMS Has Not Resolved Long-standing Acquisition Problems,
but FAA Is Beginning to Focus More on Results

According to our review of seven major ATC systems and analysis of FAA's
performance in acquiring major systems, AMS has not resolved the
long-standing problems that FAA experienced before implementing AMS, but
the agency is beginning to focus more on the expected results of its major
acquisitions. (See table 5.) Specifically, our review found that AMS
guidance did not call for requirements that were specific enough to
minimize requirements growth or unplanned work for five of these systems.
This lack of specificity resulted in the inadequate development or
definition of requirements, growth in requirements, unplanned work, or a
reduction in performance for five of these systems. In addition, for three
of these systems, FAA underestimated the difficulty of modifying available
software to fulfill its mission needs. Because AMS guidance was not
sufficient to account for the risks associated with modifying available
software, FAA encountered unexpected software development needs, higher
costs, and schedule delays. The two systems we reviewed that were
initiated after AMS was implemented are currently meeting cost and
schedule milestones; however, both systems are showing symptoms of FAA's
past problems with developing requirements and managing software, and it
is too soon to tell if these programs will remain within their cost,
schedule, and performance parameters. In addition, our work on FAA's major
acquisitions, along with that of the DOTIG and others has shown that the
problems FAA experienced before 1996 in acquiring major systems persist
under AMS and that effective acquisition management, rather than the use
of a specific contracting process (e.g., the FAR or AMS) is key to
successful acquisitions. To its credit, FAA is beginning to focus more on
results, largely through its new Air Traffic Organization, which has been
charged with taking a more performance-based approach to managing the
agency's acquisitions.

Table 5: Description and Status of Seven Selected ATC Acquisitions

Dollars in                                             
millions                                               
Project and      Original  Current  Original  Current  Acquisition issues  
description          cost     cost  schedule schedule  and status          
STARS--new         $940.0 $1,460.0      1998     2003  STARS is a joint    
controller and                                         FAA and DoD         
maintenance                                            program. STARS      
workstations to                                        delays and cost     
replace the                                            increases resulted  
legacy system at                                       from poor           
terminal air                                           requirements        
traffic control                                        definition and      
facilitiesa                                            schedule estimates. 
                                                          STARS is fully      
                                                          operational at 25   
                                                          FAA terminal radar  
                                                          facilities and 17   
                                                          DoD facilities.     
                                                          Only 50 of the      
                                                          planned 172 systems 
                                                          are being deployed. 
                                                          STARS had           
                                                          difficulties in     
                                                          achieving many      
                                                          human factor        
                                                          requirements for    
                                                          improving system    
                                                          efficiency and      
                                                          safety.             
ASR-11--digital      $743   $891.7      1997     2013  ASR-11 was approved 
radar for                                              for its in-service  
terminal                                               decision in         
environments                                           September 2003 and  
                                                          is being deployed   
                                                          at 108 sites. These 
                                                          systems are being   
                                                          deployed at a       
                                                          slower pace than    
                                                          originally planned  
                                                          because of budget   
                                                          cuts and deferrals. 
ITWS--computer     $276.1   $288.3 September     2002  Currently, six ITWS 
processors and                          2001           systems are         
displays to                                            operational. In May 
automate weather                                       2004, the ATO       
data near the                                          Executive Council   
airport                                                rebaselined the     
                                                          program to include  
                                                          a new               
                                                          weather-forecasting 
                                                          capability into the 
                                                          production          
                                                          baseline. FAA       
                                                          proposes to defer   
                                                          12 of the 34        
                                                          systems it planned  
                                                          to procure.         
LAAS--a            $530.1   $696.1      2002 Deferred  LAAS has been       
precision                                    at least  adversely affected  
approach and                                    until  by poor             
landing system                                   2009  requirements        
that augments                                          development, a lack 
the Global                                             of understanding of 
Positioning                                            its technical       
System                                                 complexity,         
                                                          incomplete software 
                                                          development, and an 
                                                          unrealistic         
                                                          development         
                                                          schedule.           
                                                          Unresolved radio    
                                                          interference        
                                                          precludes the safe  
                                                          operation of LAAS.  
                                                          As a result, FAA    
                                                          has delayed         
                                                          national deployment 
                                                          to continue further 
                                                          research on this    
                                                          issue.              
NEXCOM--digital    $318.4   $318.4   October     2004  NEXCOM program      
radios to                               2002           delays were due to  
improve air                                            misunderstanding of 
traffic                                                a program           
communications                                         requirement and     
                                                          testing procedures. 
                                                          NEXCOM was recently 
                                                          approved for its    
                                                          in-service decision 
                                                          in July 2004.       
ATOP--new          $548.2   $548.2 June 2004     2004  ATOP achieved its   
workstations and                                       acquisition program 
processing                                             baseline            
capability to                                          objectives;         
control ocean                                          however, this       
air traffic                                            baseline does not   
                                                          reflect program     
                                                          delays and cost     
                                                          increases resulting 
                                                          from poor           
                                                          requirements        
                                                          development,        
                                                          unrealistic         
                                                          schedule estimates, 
                                                          and inadequate      
                                                          evaluation of       
                                                          software            
                                                          complexity.         
Project and      Original  Current  Original  Current  Acquisition issues  
description          cost     cost  schedule schedule  and status          
ERAM--upgrades   $3,649.0 $3,649.0  December December  To date, ERAM has   
the existing en                         2009     2009  not breached any    
route system                                           cost and schedule   
with improved                                          parameters.         
hardware and                                           However, it remains 
software                                               a high-risk program 
                                                          because of the      
                                                          large amount of     
                                                          software that must  
                                                          be developed. The   
                                                          ERAM contractor is  
                                                          experiencing        
                                                          software            
                                                          engineering         
                                                          difficulties as a   
                                                          result of           
                                                          lower-than-expected 
                                                          productivity and    
                                                          software code       
                                                          growth.             

Source: GAO analysis of FAA data.

aTerminal air traffic control facilities, known as Terminal Radar Approach
Control (TRACON) facilities, direct aircraft in the airspace that extends
from the point where the tower's control ends to about 50 nautical miles
from the airport. A TRACON can be located at or outside an airport.

Our Reviews of Seven Major Systems Show That Problems with Requirements
and Software Management Persist under AMS

Our reviews of seven of FAA's costliest ATC system acquisitions found that
the problems FAA experienced with requirements and software management and
their related impact on cost, schedule, and performance goals persist
today under AMS.22  Figure 3 identifies these seven acquisitions and their
milestones, which are expressed in terms of AMS decisions even when the
acquisitions were initiated before AMS was implemented. (See app. V for a
description and the status of each of these projects.) Specifically, for 6
of these 7 major ATC acquisitions, FAA did not consistently (1) clearly
define system requirements at the investment decision point or (2)
adequately assess software complexity. Moreover, as FAA has acknowledged,
it has never managed its major acquisitions by focusing on how each would
improve the efficiency of ATC operations while maintaining or improving
safety. Although FAA has made progress in improving its acquisition of
major ATC systems-by, for example, improving the maturity of its processes
for acquiring software, using a "build a little, test a little" approach
to acquisitions as it did for Free Flight Phase 1,23  and restructuring
its organization to minimize stovepipes-long-standing problems persist in
these areas. In addition, the two systems we reviewed that were initiated
after AMS's implementation are currently operating within cost and
schedule goals; however, they are showing symptoms of past problems with
developing requirements and managing software complexity. Moreover, our
work for more than two decades-before and after AMS's implementation-has
cited these types of weaknesses as central reasons for the agency's long
history of cost, schedule, and performance shortfalls. This work has also
found that the effectiveness of an agency's acquisition management has had
a greater impact on the success of its major acquisitions than the
contracting process used (e.g., the FAR or AMS).

Figure 3: Key Milestones for Selected ATC Acquisitions Initiated before
and after AMS

Inadequate Development or Definition of Requirements Led to Requirements
Growth or Unplanned Work for Five Acquisitions

For five of the seven acquisitions we reviewed, AMS guidance did not call
for requirements that were specific enough to minimize requirements growth
or unplanned work. For four of these five acquisitions-STARS, LAAS,
NEXCOM, and ATOP-incomplete and poorly defined requirements in the final
requirements documents, used at the investment decision point to assess an
acquisition's readiness to enter the development phase, led to
requirements growth, unplanned development work, or a reduction in system
performance.24  For the fifth acquisition-ASR-11-FAA misjudged the extent
to which the high-level requirements that were used to support the
commercial-off-the-shelf/nondevelopmental item (COTS/NDI) procurement by
the Department of Defense could result in a product capable of meeting
FAA's mission or user needs. As a result, unplanned software changes were
required.

o FAA's cost estimate for STARS has grown from its original estimate of
$0.94 billion in 1996 to $1.46 billion in 2004 and will deploy only 50 of
the 172 STARS initially planned. Much of the cost growth has been due to
FAA requirements creep. As a result, the STARS program has experienced
delays of more than five years from its original plan, in part due to
added requirements to the commercial-off-the-shelf Initial System
Configuration (ISC). However, the STARS ISC was satisfactory for use by
the Department of Defense as deployed.

o A final requirements document was approved, and the development of LAAS
was scheduled to begin in 1999. However, poorly established requirements
resulted in the addition of 113 new requirements to the initial
specification, entailing significant software and hardware changes.
Furthermore, LAAS may not achieve its promised capabilities because FAA
has been unable to develop technologies necessary to warn pilots of a
disruption in the LAAS signal. Until this technology is developed, LAAS
cannot be operated safely. As a result, FAA recently cut the fiscal year
2005 funding for LAAS, and the program will revert to a research and
development effort.

o FAA developed a final requirements document for the NEXCOM system, but
the requirements lacked the specificity needed to assess the development
risk. According to a NEXCOM contractor program official, this led to
miscommunication about the program requirement relating to signal
interference. This official stated that they misunderstood this
requirement and had not planned on the additional development work for the
NDI solution to meet such program objectives and delayed the program 21
months. Another program requirement involved the NEXCOM radios meeting or
exceeding the operational coverage area of the existing voice system. The
existing radios had power output levels of 50 watts but the NEXCOM
contractor could only achieve 34 watts of power to meet the coverage
requirement. A program official stated that the contractor and FAA had not
agreed on the testing procedures to assess the power levels. This posed an
"unacceptable consequence" and, as a result, FAA performed additional
testing or flight checks of the reduced radio performance (50 watts versus
34 watts) and determined that the performance reduction should not affect
NEXCOM's mission or its coverage requirement.

o FAA did not follow the AMS guidelines that call for completing a final
requirements document before proceeding to the development phase for ATOP.
The Joint Resources Council approved a delay in developing the final
requirements until after contract award. This decision resulted in
schedule delays and additional unplanned software development. The ATOP
program office asserted that the requirements remained very stable and
that the program is within cost and schedule objectives established by the
Council. However, FAA's internal documents revealed that the requirements
were not adequately defined. For example, the ATOP Investment Analysis
Study reported to the Joint Resources Council prior to contract award that
the lack of more detailed ATOP requirements at this stage of acquisition
added risk and was of concern to the investment analysis team. Under AMS,
this team is responsible for, among other things, conducting risk analyses
for the various acquisitions. Furthermore, an ATOP Assessment Team
conducted a study in March 2003 and determined that at the ATOP contract
award, "requirements were written at a high level and not mutually
understood by FAA and the contractor." However, FAA management allowed the
ATOP program to proceed to solution implementation without the final
requirements document and, according to the contractor, this resulted in
schedule delays and growth in the amount of software needing development.

o The high-level requirements for ASR-11, jointly generated by FAA and the
Department of Defense, to support a COTS/NDI acquisition, resulted in a
product that did not initially meet the FAA mission or user needs. The
software changes that were required to meet FAA's target detection needs,
as well as significant hardware design changes, parts obsolescence, and
production issues, added approximately two years to system qualification
and acceptance.

FAA Underestimated Software Complexity for Three Systems

For three of the seven major ATC acquisitions we reviewed-ITWS, LAAS, and
ATOP-FAA's AMS guidance was not sufficient to address the risks associated
with modifying available software25 to fulfill FAA's mission needs. In all
three cases, FAA officials underestimated the difficulty of modifying
available software. Our work has shown that underestimates are likely to
result in unexpected software development, higher costs, and schedule
delays.

o ITWS experienced delays from the beginning because of the complexity of
its software development. Although the program appeared to be progressing
according to its baseline, immediately after the critical design review in
September 1998, the contractor revealed that it had exceeded the target
cost by $4 million. In addition, the contractor claimed that the program
did not recognize that the computer processor originally planned for the
program was becoming outdated, that the manufacturer planned to
discontinue its production because the market was demanding a processor
with greater processing and storage capability, and that as a result, the
original computer processor would not be available to the program.
Consequently, ITWS experienced cost increases, schedule delays, and
performance shortfalls. According to the contractor and the original
acquisition plan, all systems were scheduled for delivery by December
2001, but that date has now stretched to after 2009.

o LAAS's technology maturity was not adequately assessed, and further
development was needed. Specifically, the potential for radio interference
through the atmosphere was not understood and could limit LAAS's
operations. FAA has now placed all LAAS activities in research and
development. FAA did not adequately assess LAAS's software development. At
the time of the contract award, the contractor and FAA estimated that 80
percent of the software that LAAS required had been developed. FAA later
determined that only 20 percent had been developed. FAA and the contractor
attribute this discrepancy to a lack of communication on the steps
necessary to satisfy the program's requirements. FAA agrees that it should
have conducted a software audit and a software capabilities assessment,
but pressures to keep LAAS on schedule resulted in an inadequate
assessment.

o The ATOP contractor underestimated by about half the extent to which
legacy nondevelopmental item software, which is the core of the ATOP
system, met the program's 1,036 requirements. As a result, a significant
amount of unanticipated new software code development and other
modifications were required.26

ATC Systems Have Required Multiple Rebaselining Decisions to Address
Delays and Cost Growth

As figure 3 illustrates, FAA initiated at least one rebaselining decision
for three of the five acquisitions that were begun before AMS was
implemented and were later transitioned to AMS. These rebaselining
decisions responded to delays and cost growth-problems that arise when
requirements are not stable, a program's design is not fixed, or software
code growth is not controlled. For example, FAA rebaselined STARS two
times-first in 1999 and again in 2002. Similarly, 2 years after the
investment decision for ITWS, FAA rebaselined the program twice, in 1997
and again in 2001. Given the frequency of these past rebaselining
decisions for major ATC systems and the number of years that elapsed
before or between the rebaselining decisions (3 to 4 years), it is too
soon to tell whether the two systems that were initiated under AMS-ATOP
and ERAM-will require similar rebaselinings and ultimately meet their
cost, schedule, and performance goals. Although both programs are
currently operating within their cost and schedule goals and have not yet
been rebaselined, FAA has had problems with managing its major
acquisitions in the past and is currently having difficulties developing
requirements and managing software complexity. Furthermore, as we reported
in May 2004, FAA's budget increased from $9 billion in 1998 to $14 billion
in 2004 but will be constrained for the foreseeable future. In such a
constrained budget environment, cost growth and schedule problems can have
serious negative consequences for ongoing modernization efforts-postponed
benefits, costly interim systems, delays in funding other systems, or
reductions in the number of units purchased.

Internal and External Reviews Have Found That FAA Has Made Some Progress
but Continues to Experience Problems in Acquiring Major ATC System under
AMS

Reviews of FAA's acquisition process, conducted by FAA, GAO, the DOTIG,
and others have shown that FAA has improved its management of major ATC
acquisitions in recent years but continues to experience cost overruns,
schedule slips, and performance shortfalls under AMS. Table 7 summarizes
the results of 22 internal and external reviews of FAA's major ATC
acquisitions. According to these reviews, issued from 1997 through 2004,
the same problems have persisted over many years, despite various
initiatives to address them, and FAA needs to strengthen its management
controls. For example, a key FAA review of eight major ATC acquisitions,
published in 1999, 3 years after AMS was implemented, found that these
acquisitions, though on track to meet their performance goals, were not
meeting their cost and schedule baselines. FAA attributed these cost and
schedule issues to new or poorly understood requirements, underestimates
of the acquisitions' technical complexity, and funding shortfalls.

In addition, our reviews of major FAA acquisitions-initiated before and
after AMS was implemented-have found for more than two decades that FAA's
failure to meet schedule, cost, and performance baselines for major ATC
acquisitions has been due to shortfalls in planning, weak management
controls, and a lack of systematic processes for acquiring new systems,
including inadequate requirements management, cost-accounting data, and
estimates of technical difficulty. As we reported in August 2004, judged
against the criteria of GAO's framework for information technology (IT)
investment management, which measures the maturity of an organization's
investment management processes, FAA has established about 80 percent of
the basic selection and control practices that it needs to manage its
mission-critical investments for the National Airspace System.27 For
example, FAA's business units actively monitor projects throughout their

life cycles.28  However, the agency's senior IT investment board does not
regularly review investments that are in the "in-service management," or
operational phase, and this creates a weakness in FAA's ability to oversee
more than $1 billion of its IT investments. In addition, the agency has
not yet established the practices that would enable it to effectively
manage its annual IT budget of about $2.5 billion, and agency executives
lack assurance that they are selecting and managing the mix of investments
that best meets the agency's needs and priorities. DOT has responded to
our recommendations to FAA to strengthen its IT investment management
capability.

Moreover, other reviews, such as those by Booz-Allen & Hamilton and MITRE,
have identified other shortfalls, which reflect a lack of proper
management controls and planning. For example, in 1997, Booz-Allen &
Hamilton found, among other things, that FAA had not clearly defined
organizational roles and responsibilities within the various phases of AMS
and that greater guidance and training under AMS were warranted. In 1999,
Booz-Allen & Hamilton reported that FAA had not demonstrated improvement
in adhering to planned costs and schedules under AMS and that the agency
needed to better manage its development of requirements and address
persistent funding shortfalls. Moreover, in 2001, a MITRE report on
selected major acquisitions found inadequate management controls and
deficiencies in both contractors' performance and in FAA's measurement of
acquisition performance. See table 7 for a chronological listing of the
reviews.

Table 6: Internal and External Reviews of FAA's Use of AMS for Acquiring
Major ATC Systems

Review                     Selected findings       Contributing factors    
GAO, Air Traffic Control:  FAA's cost-estimation   FAA's cost-accounting   
Improved Cost Information  practices do not        practices do not        
Needed to Make             satisfy recognized      provide for the proper  
Billion-Dollar             estimating requisites,  accumulation of actual  
Modernization Investment   increasing the          project costs.          
Decisions,  GAO/AIMD-97-20 likelihood of poor      
, (Washington, D.C.: Jan.  acquisition selection   
22, 1997).                 decisions.              
GAO, Air Traffic Control:  Incompatibilities exist FAA lacks a complete    
Complete and Enforced      between current and     systems architecture or 
Architecture Needed for    planned ATC             overall "blueprint" to  
FAA Systems Modernization, acquisitions, resulting guide and constrain the 
GAO/AIMD-97-30 ,           in high costs and       development and         
(Washington, D.C.: Feb. 3, reduced performance.    maintenance of ATC      
1997).                                             acquisitions.           
GAO, Air Traffic Control:  Planned acquisitions    Weaknesses in some key  
Immature Software          frequently are not      process areas, such as  
Acquisition Processes      delivered on time and   planning, requirements  
Increase FAA System        within budget.          development, and        
Acquisition Risks,                                 management, limit FAA's 
GAO/AIMD-97-47 ,                                   ability to consistently 
(Washington, D.C.: Mar.                            acquire                 
21, 1997).                                         software-intensive ATC  
                                                      systems on time and     
                                                      within budget.          
FAA, Evaluation of FAA     AMS addresses 15 of the Inadequate management   
Acquisition Reform-The     17 problems facing      has not enabled FAA to  
First Year: April 1996 -   acquisitions.           meet its goals of       
March 1997, (Washington,                           reducing acquisition    
D.C.: May 1997).                                   deployment time by 50   
                                                      percent and cost by 20  
                                                      percent.                
FAA, Evaluation of FAA     Further improvements    Procedural weaknesses   
Acquisition Reform-The     are necessary if        limit FAA's ability to  
First Two Years: April     acquisition reform is   achieve cost and        
1996 - March 1998,  Report going to allow FAA to   schedule goals.         
#1998-02, (Washington,     meet its cost and       
D.C.: May 29, 1998).       schedule goals.         
GAO, Air Traffic Control:  From the inception of   Weaknesses persist in   
Observations on FAA's Air  its modernization       key areas, such as how  
Traffic Control            efforts, FAA has not    FAA monitors the status 
Modernization Program,     consistently followed a of its acquisitions     
GAO/T-RCED/AIMD-99-137 ,   disciplined management  throughout their life   
(Washington, D.C.: Mar.    approach for new        cycles.                 
25, 1999).                 acquisitions.           
GAO, Air Traffic Control:  AMS contained           FAA lacked adequate     
FAA's Modernization        weaknesses in the       cost data for making    
Investment Management      selection of            selection decisions;    
Approach Could Be          acquisitions and in the adequate management     
Strengthened,              review of acquisitions' controls, and a         
GAO/RCED/AIMD-99-88 ,      performance during the  defined, documented     
(Washington, D.C.: Apr.    postimplementation      process for conducting  
30, 1999).                 phase.                  reviews during the      
                                                      in-service management   
                                                      phase.                  
FAA, Evaluation of FAA     FAA's cost and schedule Requirements changed or 
Acquisition Reform-The     plans were not on       were misunderstood;     
First Three Years: April   track, but performance  technical difficulties  
1996 - March 1999,  Report plans were met.         were underestimated;    
#1999-04,                                          and funding fell short. 
                                                      
(Washington, D.C.: May 28,                         
1999).                                             
Booz-Allen & Hamilton,     FAA has yet to          AMS is not being        
Independent Assessment of  implement a seamless    consistently            
the Federal Aviation       life-cycle approach to  implemented across all  
Administration's           acquisitions            life-cycle phases.      
Acquisition Management     management.             
System, (McLean, VA: July                          
6, 1999).                                          
GAO, National Airspace     FAA experienced delays  FAA lacks a             
System: Persistent         and cost increases in   comprehensive plan with 
Problems in FAA's New      developing its global   checkpoints for         
Navigation System          positioning navigation  reviewing the           
Highlight Need for         system; as a result, it contractor's approach   
Periodic Reevaluation,     is unclear whether the  to meeting the system's 
GAO/RCED/AIMD-00-130 ,     benefits of the system  performance             
(Washington, D.C.: June    will outweigh the cost. requirements.           
12, 2000).                                         
Review                     Selected findings       Contributing factors    
GAO, National Airspace     FAA experienced cost    FAA underestimated the  
System: Problems Plaguing  and schedule problems   complexity of           
the Wide Area Augmentation in developing this      developing the          
System and FAA's Actions   navigational system     acquisition.            
to Address Them,           because of unplanned    
GAO/T-RCED-00-229 ,        software development    
(Washington, D.C.: June    needs and a requirement 
29, 2000).                 to warn pilots of any   
                              system failure that     
                              would provide           
                              misleading information. 
GAO, National Airspace     Three acquisitions that FAA needs better data   
System: Free Flight Tools  are components of FAA's collection and analysis 
Show Promise, but          planned new approach    processes to ensure     
Implementation Challenges  for air traffic         that benefits are       
Remain, GAO-01-932 ,       management have         realized.               
(Washington, D.C.: Aug.    uncertain potential     
31, 2001).                 benefits and may not be 
                              worth FAA's investment. 
GAO, National Airspace     The reliability of the  The development cost    
System: Better Cost Data   life-cycle cost         estimate is based on    
Could Improve FAA's        estimate for STARS is   the contractor's        
Management of the Standard uncertain because cost  projections, which FAA  
Terminal Automation        data obtained from the  has not yet             
Replacement System,        contractor do not       independently analyzed, 
GAO-03-343 , (Washington,  reflect the current     as called for under     
D.C.: Jan. 31, 2003).      status of the contract. AMS.                    
GAO, National Airspace     FAA was unable to hire  Uncertainties about the 
System: Current Efforts    a chief operating       position's              
and Proposed Changes to    officer to head the     responsibilities,       
Improve Performance of     ATO.                    reporting               
FAA's Air Traffic Control                          relationships, and      
System, GAO-03-542 ,                               performance measurement 
(Washington, D.C.: May 30,                         criteria hampered the   
2003).                                             hiring.                 
DOT/OIG, Status of FAA's   Cost growth, schedule   Cost and schedule       
Major Acquisitions,        delays, and performance baselines are not       
AV-2003-045, (Washington,  problems continue with  reliable, and decisions 
D.C.: June 26, 2003).      FAA's major             are being made with     
                              acquisitions.           unclear data.           
GAO, Air Traffic Control:  Systemic management     FAA lacked the          
FAA's Modernization        issues, including       information technology  
Efforts-Past, Present, and inadequate management   and financial           
Future, GAO-04-227T ,      controls and human      management systems that 
(Washington, D.C.: Oct.    capital issues, have    would have helped it    
30, 2003).                 contributed to major    reliably determine the  
                              ATC acquisitions'       acquisitions' technical 
                              persistent cost         requirements and        
                              overruns, schedule      estimate and control    
                              delays, and performance their costs and         
                              shortfalls.             schedules; and the      
                                                      agency's organizational 
                                                      culture discouraged     
                                                      collaboration among     
                                                      technical experts and   
                                                      users.                  
GAO, Information           Although weaknesses     Remaining weaknesses    
Technology: FAA Has Many   remain, FAA has         include inadequate      
Investment Management      established about 80    management controls and 
Capabilities in Place, but percent of the basic    the lack of a defined,  
More Oversight of          practices needed to     documented process for  
Operational Systems Is     manage its              conducting reviews      
Needed, GAO-04-822 ,       mission-critical        during the in-service   
(Washington, D.C.: Aug.    acquisitions so that it management phase.       
20, 2004).                 can be assured that it  
                              is selecting and        
                              managing the mix of     
                              investments that best   
                              meets its needs and     
                              priorities.             
GAO, Air Traffic Control:  FAA made progress in    Process improvement     
System Management          improving its system    efforts have not been   
Capabilities Improved, but management              institutionalized.      
More Can Be Done to        capabilities, but can   
Institutionalize           do more to              
Improvements, GAO-04-901 , institutionalize        
(Washington, D.C.: Aug.    process improvement     
20, 2004).                 initiatives.            

Source: GAO analysis.

FAA's ATO Is Taking Steps to Improve Major ATC Acquisitions

FAA's recent reorganization, which brought ATC acquisitions and operations
together in the ATO,29 is expected to help the agency address many of the
concerns we have identified for more than two decades, including those
identified in this report. For example, the ATO is continuing to develop
and refine specific guidance for critical areas, such as requirements
management, software development, and cost estimation. In addition, as the
overseer of both ATC acquisitions and operations, the ATO is in a position
to facilitate more effective management of major ATC acquisitions than has
occurred in the past. The ATO is attempting, for example, to link
acquisition decisions directly with expected improvements in operational
efficiency without compromising safety. This is important, given that FAA
has spent about $2.5 billion on ATC modernization per year since 1996
while operating costs have continued to rise-from $4.6 billion to $7.5
billion over the past decade.  FAA had not completed its reorganization or
implemented all of its initiatives at the time of our audit.

Improvements to Requirements Development

With the establishment of the ATO, FAA consolidated requirements
development from two organizations (the organization sponsoring an
acquisition and the former agencywide acquisition organization) into a
single new organization-the Air Traffic System Requirements Service.30 In
addition, the ATO developed guidance to better manage requirements during
the middle phase of AMS (solution implementation). According to FAA
officials, some more complex development efforts may need to develop
systems requirements and a more detailed requirements document than AMS
currently calls for in the final requirements document. More important, in
January 2003, FAA issued guidance on requirements management, Roles in
Requirements Management During Solution Implementation Phase, which
provides for integrated requirements teams that maintain responsibility
for requirements management throughout an acquisition's life cycle.
According to this guidance, when the final requirements document is
accepted by the Joint Resources Council at the investment decision point,
a requirements baseline is established and any proposed changes to the
requirements must be assessed for their impact on the program and shown to
be operationally suitable, affordable, executable, and justifiable. An FAA
official on an integrated requirements team stated that any changes that
may affect an acquisition's cost and the schedule require approval by the
Executive Committee. The FAA official also stated that this guidance has
already helped to stabilize NEXCOM's requirements during the solution
implementation phase. Other FAA officials representing the Joint Resources
Council acknowledged that the guidance should ensure greater control over
program requirements growth, but said that not all program offices have
consistently applied it.

Improvements to Managing Software and System Acquisition and Development

To better manage software programs for ATC modernization acquisitions, FAA
established a centralized process improvement office that reports to the
Chief Information Officer (CIO).31 This office developed an FAA integrated
capability maturity model (i-CMM), a software development and management
model that is similar to a model developed by Carnegie Mellon University
called the Capability Maturity Model Integration (CMMI(R)), which is used
to appraise the maturity of an organization's processes for acquiring
software. However, FAA's i-CMM goes beyond Carnegie Mellon's model to
reflect international standards. The CMMI(R) appraisal methodology calls
for assessing process areas-such as project planning, requirements
management, and quality assurance-by determining whether key practices are
implemented and overarching goals are satisfied. Both the i-CMM model and
CMMI(R) appraisal methodologies provide a logical framework for measuring
and improving key processes needed for achieving quality software and
systems.

However, as we reported in August 2004,32 FAA projects are not required to
use the capability maturity model for process improvement, and individual
projects that use the i-CMM model are allowed to choose which process
areas they seek to improve and to determine when they are ready for an
appraisal of their progress. To date, fewer than half of FAA's major ATC
projects have used this model. The recurring weaknesses we identified in
our project-specific evaluations are due in part to the flexibility these
projects were given in deciding whether and how to adopt this process
improvement initiative. Furthermore, after combining its ATC organizations
into a single performance-based organization (the ATO), FAA is
reconsidering prior policies, and it is not yet clear whether process
improvement will remain a priority. Without a strong senior-level
commitment to process improvement and a consistent, institutionalized
approach to implementing and evaluating it, FAA cannot ensure that key
projects will continue to improve systems acquisition and development
capabilities. As a result, FAA will continue to risk the project
management problems-including cost overruns, schedule delays, and
performance shortfalls-that have plagued past acquisitions. To address
these shortcomings, we recommended that the Secretary of Transportation
address specific weaknesses and institutionalize FAA's process improvement
initiatives by establishing a policy and plans for implementing and
overseeing process improvement initiatives.

Improvements to Estimating Costs

FAA has taken steps to improve its cost estimation for major ATC projects
by issuing guidance on how to develop and use pricing under AMS. For
example, AMS policy calls for audit trails to record and explain the
values that are used as inputs to cost models. In addition, it calls for
agency officials, when reporting to executive oversight agencies and
Congress, to disclose the level of uncertainty and imprecision that are
inherent in cost estimates for major ATC systems. According to AMS policy,
estimators record the procedures, ground rules and assumptions, data,
environment, and events that underlie their development or update of a
cost estimate. This information supports the credibility of the cost
estimate, aids in the analysis of changes in program costs, enables
reviewers to assess the cost estimate effectively, and contributes to the
population of FAA databases that can be used for estimating the cost of
future programs. Finally, despite a delay of many years, FAA officials
told us that they are in the final stages of completing the agency's
cost-accounting system and plan to have it in place across the agency by
the end of this calendar year, which will bring FAA into compliance with
the Federal Managers' Financial Integrity Act of 1982. This measure will
help reduce the likelihood of cost overruns or improper payments for
unallowable costs and provide decision-makers with critical information.
As we have reported in the past,33 a cost-accounting system is critical to
managing major ATC acquisitions, because without it, FAA lacks the
information it needs to reliably estimate operating costs over an
acquisition's life cycle.

Other Improvement Efforts

In May 2004, the FAA Administrator testified to Congress that, to date, in
attempting to improve the efficiency of ATC operations while maintaining
safety, FAA had not managed its major ATC acquisitions to be aware of
their cost implications for its operations. The Administrator said,
however, that the agency was taking its first steps to fundamentally
change how it makes acquisition decisions by adopting a more
results-oriented approach. Under this approach, the agency plans to link
its decisions to fund major acquisitions directly with their expected
contribution to improving operational efficiency and controlling
escalating operating costs. Whereas, in the past, FAA measured results in
terms of its progress in completing and deploying a major ATC system, it
was now going to focus on how a given system improved operational
efficiency. Such an approach holds promise for helping FAA more
effectively manage its portfolio of major ATC acquisitions by providing a
sound basis for choosing among competing priorities. However, because FAA
has only recently begun to incorporate this type of analysis of
acquisitions' costs and operational efficiency into its decision-making
and management processes, it is still too early to assess the results.

In addition, to its credit, FAA has created a training framework for its
acquisition workforce, which we found mirrors human capital best practices
that we have identified. In January 2003, we reported on FAA's efforts to
define and train its workforce to meet the requirements of the
Clinger-Cohen Act of 1996.34 This act required FAA and other civilian
agencies to establish education, training, and experience requirements for
their acquisition workforce. Our work on public and private best practices
has identified six elements of training as critical to acquisition. These
elements include (1) prioritizing the acquisition initiatives most
important to the agency, (2) securing top-level commitment and resources,
(3) identifying those who need training on specific initiatives, (4)
tailoring training to meet the needs of the workforce, (5) tracking
training to ensure it reaches the right people, and (6) measuring the
effectiveness of training.  These six elements are crucial for
successfully implementing acquisition initiatives and reforms. Agencies
that do not focus their attention on these critical elements risk having
an acquisition workforce that is ill equipped to implement new processes. 
The probability of success is higher if training is well planned rather
than left to chance. In 2003, we found that FAA's model for training its
acquisition workforce largely mirrored public and private-

successfully introducing and implementing effective acquisition best
practices. FAA's acquisition workforce plays a critical role in addressing
long-standing weaknesses that we and others have identified with FAA's
acquisition of major ATC systems. Given the importance of training for
acquisition workforces, it will be important for the ATO to put mechanisms
in place to comprehensively evaluate the effectiveness of the training it
provides to improve the knowledge base of FAA's acquisition workforce.

To improve its investment management decision-making and oversight of
major ATC acquisitions, the ATO also initiated the following procedures:

o Integrate AMS and the Office of Management and Budget's Capital Planning
and Investment Control Process36  to develop a process for analyzing,
tracking, and evaluating the risks and results of all major capital
investments made by FAA.

o Conduct Executive Council reviews of project breaches of 5 percent in
cost, schedule, and performance to better manage cost growth;

o Issue monthly variance reports to upper management to keep them apprised
of cost and schedule trends.

o Monitor progress in meeting the goals identified in FAA's Flight Plan,
the agency's blueprint for action through 2008. The Executive Council
tracks this progress monthly and reports to the Administrator, using a
color-coded system to keep her apprised of how well FAA is meeting its
goals. Green denotes that a goal will be met, yellow denotes that some of
the activities leading to a main goal may be in jeopardy but the overall
goal can be achieved, and red denotes serious concerns about reaching a
goal without major intervention. A formal progress report is issued
quarterly and made publicly available on the agency's Web site; and

o Increase the use of cost monitoring or earned value management systems37
to improve oversight of programs.

Despite FAA's current and planned efforts to improve its acquisition of
major ATC systems under the ATO, given the newness of these efforts and
the agency's poor track record in this area for more than two decades, it
is critical for FAA to (1) modify AMS to more fully reflect the best
practices followed by high-performing acquisition organizations, (2)
follow through on planned improvement initiatives, and (3) adopt a
continuous improvement approach to acquiring new ATC systems.

Conclusions

In the early 1990s, FAA contended that it needed relief from the FAR to
remedy long-standing problems with cost, schedule, and performance
shortfalls in its major ATC acquisitions; however, our work for more than
two decades in this area has found that acquiring major ATC systems
successfully depends more on managing an acquisition process well than on
using a specific contracting process (e.g., the FAR or AMS). While our
recent work has shown some improvement in FAA's management of major ATC
system acquisitions, some key problems that existed before 1996 persist
under AMS-including difficulty with clearly defining system requirements
at the investment milestone and adequately assessing complex software
requirements. These problems continue to make these acquisitions
vulnerable to cost, schedule, and performance shortfalls. Without further
measures to improve the development and management of requirements and to
better estimate the complexity of the software development needed for
major ATC systems, such shortfalls are likely to persist.

Although AMS provides some discipline for acquiring major ATC systems
through its various phases, activities, and decision points, it does not
require that (1) specific knowledge be attained using explicit written
criteria and (2) corporate executive-level oversight be provided to
determine-independently from the program offices-whether a system has
reached a level of development (product maturity) sufficient to move
forward in the acquisition process. Commercial best practices call for
such knowledge-based decision-making at the corporate executive-level to
help ensure that acquisitions are not moved into the development phase
prematurely, to obtain greater predictability in ATC system program costs
and schedules, to improve the quality of the ATC systems that are
deployed, and to deliver new capability to the National Airspace System
faster. A knowledge-based approach is also important because it provides
assurance that agency decision-makers have critical information about an
acquisition's ability to meet a mission need and FAA's readiness to move
forward in the acquisition process before making large commitments of
agency resources. Absent such an approach, FAA lacks assurance that it has
obtained the critical technological, design, or manufacturing knowledge
that best practices call for to avoid cost overruns, schedule slips, and
performance shortfalls. As a result, FAA is not doing all that it can to
systematically address persistent shortcomings in its management of major
ATC acquisitions. Moreover, although FAA has established a framework for
training its acquisition workforce under the ATO, it has not yet developed
comprehensive performance criteria to evaluate how effectively it has
implemented this framework. As a result, the agency lacks assurance that
its use of this framework is having the intended effect of improving the
knowledge base of this workforce.

Recommendations for Executive Action

We are making five recommendations to the Secretary of Transportation. To
reduce the risk of persistent cost and schedule shortfalls in major ATC
system acquisition programs, to improve the quality of the ATC systems
that are deployed, and to deliver new capability to the National Airspace
System faster, we recommend that the Secretary of Transportation advise
the FAA Administrator to take the four following actions:

o Modify AMS to specify that requirements be more clearly defined for
major ATC systems, including providing more detailed guidance on setting
clear, objective, and measurable requirements that reflect customers'
needs, before making large investments of agency resources.

o Establish a strategy for identifying and measuring all additional
development needed for complex software (e.g., commercial-off-the-shelf or
nondevelopmental items) used for major ATC systems.

o Develop explicit written criteria for the key decision points called for
under best practices, including the capture of specific design and
manufacturing knowledge.

o Require corporate executive-level decisions at these key decision points
(before an acquisition moves from integration to demonstration and, again,
before it moves to production).

In addition, to assure FAA that the training framework it has adopted for
the ATO's acquisition workforce is improving the knowledge base of this
workforce as intended, we recommend that the Secretary advise the
Administrator to develop performance criteria to comprehensively evaluate
the framework's effectiveness.

Agency Comments

We provided copies of a draft of this report to DOT for review and comment
and met with Department and FAA officials, including the ATO's Vice
President for Acquisition and Business Services, to obtain their comments.
FAA officials told us that they have made great strides in improving their
acquisition of major ATC systems under AMS; however, they recognize that
there is room for improvement and are firmly committed to implementing
best practices for acquisitions. These officials generally agreed with the
report's findings and conclusions and said that our recommendations would
be useful to them as they continue to refine their acquisition management
system, including training their acquisition workforce. The agency
provided us with oral comments, primarily technical clarifications, which
we have incorporated as appropriate.

As agreed with your office, unless you publicly announce the contents of
this report earlier, we plan no further distribution until 30 days from
the report date. At that time, we will send copies of this report to
interested congressional committees, the Secretary of Transportation, and
the Administrator, FAA. We will also make copies available to others upon
request. In addition, the report will be available at no charge on the GAO
Web site at http://www.gao.gov.

concerning this report. Key contributors to this report are listed in
appendix VI.

JayEtta Z. Hecker Director, Physical Infrastructure Team

FAA Has Begun Analyzing Spending Trends to Take a More Strategic Approach
to ProcurementAppendix I

Our review of the Federal Aviation Administration's (FAA) general
procurement of goods and services focused on the Air Traffic Organization
(ATO) and its predecessor offices. According to FAA officials, the ATO has
recently begun to consider ways to better leverage its buying power by
taking a more strategic approach to procurement. While FAA uses the
Acquisition Management System (AMS) for all FAA acquisitions, including
the procurement of such goods and services as office supplies, computers,
telephone services, and engineering and technical support services, these
procurement activities take place in a decentralized environment of
independent, transaction-oriented buying processes. Each FAA unit
determines its need for goods and services and procures them as necessary,
leaving headquarters with limited oversight of the agency's total
procurement spending. For example, in 2003, FAA units carried out over
346,000 procurement actions for goods and services and purchase
cardholders1 made an additional 335,000 transactions. This fragmented
environment does not permit the agency to leverage its buying power
through lower-cost, consolidated contracts, at the local, regional, or
national level and to rationalize the number of suppliers best suited to
meet the agency's needs. At the same time, as part of a strategic
procurement effort, FAA can use spend analysis to monitor trends in small
and disadvantaged business participation so that it can balance the goals
of lower-cost contract consolidation and promoting small business
contracting opportunities.

Spend analysis, a tool used in a strategic approach to procurement,
provides knowledge about how much is being spent for what goods and
services, who the buyers are, who the suppliers are, and where the
opportunities are to leverage buying power. Our past work2 shows that
private companies are using spend analysis as a foundation for employing a
strategic approach to procurement. The analysis identifies where numerous
suppliers are providing similar goods and services-often at varying
prices-and where purchasing costs can be reduced and performance improved
by better leveraging buying power and reducing the number of suppliers to
meet the company's needs. Our research on commercial best practices has
found that spend analysis is an important driver of strategic planning and
execution. As part of an overall strategic procurement effort, companies
use spend analysis to (1) define the magnitude and the characteristics of
their spending, (2) understand their internal clients and supply chain,
(3) create lower-cost consolidated contracts, and (4) monitor spending
with small and disadvantaged businesses to achieve socioeconomic
procurement goals.

We previously reported3  that six agencies, including DOT, did not take
advantage of opportunities to obtain more favorable prices on purchase
card buys with frequently used vendors-vendors where an agency spends more
than $1 million annually. In these six agencies, which accounted for over
85 percent of federal government purchase card spending, frequently used
vendors accounted for purchases totaling nearly $3 billion in 2002. We
recommended several actions-including conducting spend analysis using
available data and gathering additional information where feasible-that
could ultimately help these agencies achieve $300 million annually in
potential savings.

In fiscal year 2003, FAA procured nearly $4 billion in goods and services
and spent an additional $132 million using purchase cards. According to
senior FAA officials, the agency has just begun to implement a strategic
approach to general procurements. Other federal agencies are beginning to
use strategic tools such as spend analysis to improve their spending for
goods and services, and some have initiatives under way to obtain more
favorable prices on purchase card buys. According to a senior FAA
acquisition official, FAA has to balance the need of its units to
independently make purchases that pertain solely to unit requirements with
the agency's need to aggregate purchases of goods and services that are
used by more than one unit. FAA has hired a consultant to help begin the
use of spend analysis. This effort could reduce the agencywide costs for
mobile wireless services by 40 percent-an effort expected to save the
agency $8 to $10 million annually. FAA intends to expand its use of spend
analysis to target other procurement category savings opportunities,
including information technologies, training, facilities, and professional
services, as its accounting systems improve.

FAA has taken some preliminary steps to set up a spend analysis program;
however, progress has been challenging for FAA because of deficiencies in
its accounting systems. For example, because the agency's accounting
system did not identify all of the mobile wireless services for which it
was being billed, the contractor implementing the spend analysis had to
obtain this information from the wireless providers. FAA will need to
expedite its efforts in this area to fully realize potential savings. Our
prior research has shown that setting up a spend analysis program can be
challenging. Companies have had problems accumulating sufficient data from
internal financial systems that do not capture information on all of what
a company buys or is using in different, unconnected parts of the company.
Despite these challenges, companies that have developed formal,
centralized spend analysis programs have been able to track their costs
and identify areas for strategic sourcing and savings opportunities.

In our recent report on spend analysis,4 we found that DOT, at the time of
our review, had not yet begun to collect the data needed for a strategic
approach to procurement; however, the department is engaged in ongoing
efforts to improve procurements, and its top leadership is committed to
using spend analysis to change the way goods and services are purchased.
One obstacle to using spend analysis that the department cited during our
review was a lack of comprehensive and reliable spending data. However,
since we completed our review, the department reports stepping up efforts
to use currently available data and evaluate business intelligence
software to overcome those obstacles. In commenting on our report,
Transportation's senior procurement executive told us that the department
is expanding its spend analysis efforts. For example, his office recently
reviewed purchase card spending data to identify volume discount
opportunities and is now using the results to negotiate new discount
agreements with several office product vendors. In addition, he told us
that to facilitate future agencywide purchase card spend analyses, DOT
awarded a task order in June 2004 to one bank card company that will
provide purchase-card audit software and enhanced data-mining
capabilities. He also indicated that the department's leadership supports
fiscal year 2005 funding to enhance spend analysis capabilities and that
software options for the new agencywide spend analysis system are now
being evaluated as part of an ongoing financial and procurement review.

Objectives, Scope, and MethodologyAppendix II

To compare FAA's Acquisition Management System (AMS) with the Federal
Acquisition Regulation (FAR), we reviewed AMS and changes in it over time.
We also compared FAA's acquisition authority under the FAR and under AMS.
In addition, we identified relevant recommendations from reports that we,
the Department of Transportation's Inspector General (DOTIG), and others
have issued to determine which recommendations have been implemented,
rejected, or left open, and to evaluate how those recommendations have
modified FAA's acquisition policies and practices. We also collected and
summarized published reports and analyzed available life-cycle management
data on the current status of major and nonmajor acquisitions being
carried out under AMS.

To determine the ways in which FAA's acquisition policies compare with our
best practices model, we used information from several of our products
that examine how commercial best practices can improve outcomes for
acquisition programs. This model consists of four phases: (1) concept and
technology development; (2) product development, which includes both
integration and demonstration activities; (3) production; and (4)
operations and support. In between these four phases are three key
knowledge points at which commercial firms must have sufficient knowledge
to make large investment decisions. We also reviewed and analyzed AMS,
accessible at http://fast.faa.gov . Furthermore, to clarify the content of
FAA's acquisition process, we met with various FAA vice-presidents and
officials from FAA's Acquisition Planning and Policy Division. Next, we
compared and contrasted FAA's acquisition policies with the best practices
for commercial acquisitions identified in our past reports. Our analysis
focused on whether FAA's policies contained the measurable criteria and
management controls necessary to achieve FAA's intent of minimizing cost,
schedule, and performance risks. We also interviewed current and former
FAA procurement officials that have experience using both the FAR and AMS.

To determine if FAA has effectively implemented its new acquisition
authority and improved its acquisition outcomes, we reviewed seven of
FAA's most expensive major ATC acquisitions, including the Airport
Surveillance Radar 11 (ASR-11), Standard Terminal Automation Replacement
System (STARS), Integrated Terminal Weather System (ITWS),  Local Area
Augmentation System (LAAS), Next Generation Air/Ground Communications
System (NEXCOM), Advanced Technologies and Oceanic Procedures (ATOP), and
En Route Automation Modernization (ERAM). See table 7 for specific program
costs.

Table 7: Program Costs for the Seven Systems We Reviewed

Dollars in millions            
Program                                   Total program cost as of 9/30/04 
STARS                                                             $1,460.0 
ASR-11                                                               891.7 
ITWS                                                                 288.3 
LAAS                                                                 696.1 
NEXCOM                                                               318.4 
ATOP                                                                 548.2 
ERAM                                                               2,154.6 
Total                                                             $6,357.3 

Source: GAO analysis.

Note: These amounts are for facilities and equipment only (not operations
and maintenance).

We also selected these seven acquisitions because we considered them to
fall into two basic categories-pre-AMS and post-AMS. Five of the
acquisitions were initiated before AMS was implemented in April 1996 and
were transitioned into AMS at various times before their completion. The
two remaining acquisitions-ATOP and ERAM-were initiated and have remained
completely under AMS. We then reviewed program documents and reports and
interviewed program and agency officials responsible for developing these
acquisitions, as well as other acquisitions experts in the private sector.
For some acquisitions, we discussed programmatic issues with
representatives of the primary contractor for the specific acquisition to
obtain information on the practices and procedures used for the
acquisition. In addition, we interviewed some current and former FAA
procurement officials with experience using both the FAR and AMS to obtain
their views on the use of each contracting process and how the two
compare. Furthermore, to see how FAA has progressed in addressing problems
with its acquisitions, we reviewed our work on acquisitions over the last
20 years, as well as reports by the DOTIG, FAA, Booz-Allen & Hamilton, and
MITRE. Because the data in this report on cost, schedule and performance
are used as background information or to otherwise provide a description
of acquisitions, we did not assess their reliability.

The effect of the current budget process on FAA's ability to successfully
modernize the National Airspace System, including acquiring major ATC
systems is not within the scope of this review.

Comparison of the Scope and Flexibility of FAA's Acquisition Management
System and the Federal Acquisition Regulation ProcessAppendix III

Background

FAA's business processes, including its acquisition of major systems,
differ significantly from the business processes followed by most other
federal agencies. FAA relies on its Acquisition Management System (AMS),
which establishes FAA internal acquisition policy. AMS resulted from the
adoption of language in the Department of Transportation and Related
Agencies Appropriations Act,1 which directed the FAA Administrator to
develop and implement an acquisition management system for FAA. The
adoption of this language (section 348) followed FAA's assertions that the
requirement that it conduct procurements in accordance with the Federal
Acquisition Regulation (FAR) was at least a contributing factor in its
repeated failure to complete air traffic control (ATC) and other
modernization programs on schedule. The Administrator was directed to put
in place a system that would address the "unique needs of the agency" that
FAA contended prevented its acquisitions from being timely and
cost-effective.

Section 348 distinguished FAA from other federal agencies by removing FAA
from the federal acquisition system. Under section 348, FAA was no longer
subject to title III of the Federal Property and Administrative Services
Act of 1949,2 which among other things requires that the government
procure supplies and services competitively. It removed FAA as an agency
subject to the Office of Federal Procurement Policy Act3 and eliminated
the requirement that FAA comply with the FAR. While mandating that FAA
conduct its acquisitions so that "all reasonable opportunities to be
awarded contracts shall be provided to small business concerns and small
business concerns owned and controlled by socially and economically
disadvantaged individuals," section 348 eliminated the requirement that
FAA comply with the Small Business Act.4 Furthermore, it made the
procurement protest system of the U.S. Government Accountability Office
inapplicable to FAA, although disappointed offerors

can still file protests with FAA's Office of Dispute Resolution for
Acquisition.5

AMS Defines an Investment/Life-Cycle Project Management System

Much of AMS guidance concerns project, financial, and property life-cycle
management issues. In fact, FAA's policy describes AMS as applying to all
investment programs regardless of cost or the appropriation funding them.
It recognizes that a single investment program may span multiple
procurements and projects. It applies, according to its terms, to the
activities associated with needs analysis, determination of requirements,
analysis of investment alternatives, establishment of investment programs,
allocation and expenditure of resources, procurement and deployment of
needed products and services, in-service management of fielded capability,
and eventual disposal of obsolete products.

AMS focuses on the following key program milestones:

o Mission Analysis-encompasses those key corporate and service-level
processes that define, coordinate, and integrate the work of service
organizations,6 thereby providing strategic direction to keep FAA
responsive to the service needs of its customers. Mission analysis is used
to update a mission need statement, which in turn may identify capability
shortfalls or technological opportunities, that is, unmet needs. Unmet
needs are presented to the Joint Resources Council (JRC) for a mission
need decision. To be approved, the unmet need should be supported by the
updated mission need statement and the initial requirements document,
including a concept of use, and the initial investment plan.

o Investment Analysis-builds on the results of the mission need decision
by developing detailed plans and final requirements for each proposed
investment program and by defining an acquisition program baseline that
establishes cost, schedule, performance, benefit, and risk-management
boundaries for the program. AMS calls for planning the entire solution-an
effort that may use market survey data but is based in large measure on
FAA's assumptions and data. The service organization produces a final
implementation and life-cycle support strategy. A detailed program plan
and an acquisition program baseline are also produced. The results are
presented to the JRC for a "final investment decision."7

o Solution Implementation-encompasses acquiring, accepting, deploying,
installing and preparing for the operational use of an approved
investment. Approval of the investment carries with it authorization for
the service organization to conduct all acquisitions needed to execute the
investment decision, subject to any constraints established in the final
investment decision.

o In-Service Decision-is an FAA system qualification milestone, which is
achieved when an otherwise operational investment is satisfactorily tested
to demonstrate its operational effectiveness and suitability before it is
placed in service in the National Airspace System. The JRC designates the
decision maker.

o In-Service Management-covers activities throughout a system's life
cycle, starting at the time that an investment becomes operational.
In-service product improvements may eliminate latent defects, fix systemic
problems, and enhance the utility of the investment. These changes may be
made within the approved acquisition program baseline without
corporate-level approval. In-service management also includes planning,
programming, and developing supporting budget input; monitoring and
assessing performance, cost of ownership, and support trends; and planning
for service-life investment decisions.

o Service Life Extension-seeks a new investment decision by the JRC when a
current capability is unable to satisfy demand or when another solution
may be more effective. The JRC can decide to revalidate the mission need
satisfied by the solution by upgrading or refurbishing fielded capability
or by replacing that capability with another equivalent or new superior
solution. The JRC may also decide that the capability should be retired.

Only a Portion of AMS Deals Directly with the Procurement Process

Although the FAR includes requirements addressing procurement planning and
major system acquisition, AMS as just outlined differs significantly from
the FAR in its focus and scope. The FAR addresses planning8 and major
system acquisition9 in the context of government procurement policy and
procedure. Agencies other than FAA find the broader program planning and
management issues addressed in AMS outside of the FAR, in documents such
as the Office of Management and Budget's (OMB) Circular A-109, in their
own planning guidance, such as the Department of Defense's (DOD) 5000
series,10  and in established knowledge-based best practices. As indicated
earlier, much of AMS focuses on just such issues. Only AMS section 3
addresses procurement policy and procedure.11

AMS States a Nonregulatory FAA Policy

A further significant foundational difference between AMS and the FAR is
that AMS sets out a nonregulatory FAA policy, whereas the FAR was adopted
and is maintained as a set of published governmentwide regulatory
requirements, which form a legal basis for federal agencies' contract
decision-making. AMS is binding on FAA personnel as FAA employees and
establishes other guidelines that FAA states should be followed unless
there is a rational basis for doing otherwise. AMS is subject to such
internal controls as the Administrator chooses to enforce and general
overarching legal requirements, such as the Government Performance and
Results Act of 1993 (GPRA).12 There is a legal requirement, created by
section 348, that small and socially or economically disadvantaged firms
be given all reasonable opportunities to receive contract awards. FAA in
its Office of Dispute Resolution for Acquisition has adopted a dispute
resolution process with some legal underpinnings.13 Otherwise, as the
preface to AMS states, "nothing in this document creates or conveys any
substantive [legal] rights." In short, FAA has assumed no legal obligation
to follow AMS other than to ensure that its actions are not arbitrary and
capricious or contrary to law. By contrast, the FAR has the force and
effect of law, and agencies that are subject to the FAR are bound to
follow it.

AMS Chapter 3 Parallels a Subset of the FAR

When FAA personnel apply the procurement methodology in AMS chapter 3,
they are applying guidance that closely parallels some of the procedures
set out in the FAR. The AMS Chapter 3 acquisition process parallels a
subset of the varied selection of procurement methods available under the
FAR, requiring that all competitive FAA contracts be negotiated with the
awardee being selected on a "best value" basis. The FAR also provides a
much more detailed set of information and guidance than does AMS. A
comparison of high-level differences and similarities between AMS and the
FAR is presented in table 8.

Table 8: Comparison of AMS and the FAR

                        AMS                         FAR                       
Best value source    Yes, following screening.   Yes, although other       
selection                                        methods are also          
                                                    available for use when    
                                                    appropriate.              
Public announcement  Public announcement through Yes, for proposed         
of requirement       Internet or other means     contract actions expected 
                        when value of contract is   to exceed $25,000.        
                        anticipated to exceed       
                        $100,000.                   
Competition          FAA's policy is to provide  Full and open             
                        reasonable access to        competition-all           
                        competition for firms       responsible sources are   
                        interested in obtaining     permitted to compete.     
                        contracts. In selecting     
                        sources, the preferred      
                        method of procurement is to 
                        compete requirements among  
                        two or more sources.        
Sole-source          Yes, when deemed to be in   Yes, full and open        
procurement          FAA's "best interest" as    competition need not be   
                        determined by the service   obtained under certain    
                        organization on the basis   specified conditions      
                        of "adequate objective      based upon a written      
                        supporting data."           justification from the    
                                                    contracting officer that  
                                                    is approved at an         
                                                    appropriate level of      
                                                    authority.                
Prequalification     Yes, qualification          Yes, for products or      
                        information screens for     manufacturers when        
                        those vendors that meet     justified in writing and  
                        FAA's stated minimum        conducted in a manner     
                        capabilities or             that meets requirements   
                        requirements for providing  justifying the use of     
                        a given product or service. qualifications            
                                                    requirements.             
Basic methodology in FAA issues one or more      Agency issues a           
negotiated           "screening requests," which solicitation, usually a   
procurement          may include requests for    request for proposals.    
                        binding offers from         
                        competing firms.            
Methodology for      FAA encourages one-on-one   Clarification and         
negotiation          communications throughout   discussions are           
                        the process provided that   permitted; one offeror    
                        no offeror is given an      cannot be favored over    
                        "unfair advantage."         another.                  
                        AMS                         FAR                       
Evaluation and award Selection is based on       Selection is based on     
selection            evaluation in accord with   evaluation in accord with 
                        criteria identified in the  criteria identified in    
                        screening request. The      the request for           
                        selection decision is a     proposals. The selection  
                        judgmental decision made by decision is a judgmental  
                        the source selection        decision made by the      
                        official.                   source selection          
                                                    official.                 
Use of simplified    Commercial and simplified   Generally required for    
acquisition methods  purchases are used for      purchases up to $100,000, 
                        commercial items or for     for noncommercial items,  
                        products or services that   or on a test basis, up to 
                        have been sold at           $5,000,000 for commercial 
                        established catalog or      items competition is to   
                        market prices and are       be obtained to the        
                        generally purchased on a    maximum practicable       
                        fixed-price basis.          extent.                   
Use of credit card   Permitted.                  Permitted.                
purchases                                        
Procurement          AMS does not include the    Provides a broad          
methodology          level of detail found in    selection of procurement  
                        the FAR. It does not        methods and techniques    
                        prescribe many of the       suitable for use in most  
                        procurement methods and     circumstances.            
                        techniques permitted under  
                        the FAR, but encourages use 
                        of "any method of           
                        procurement deemed          
                        appropriate."               
Responsibility       Awards to responsible       Awards to responsible     
                        offerors only.              offerors only.            

Source: GAO analysis.

AMS Includes a Less Rigorous Competition Requirement Than Does the FAR

As table 8 indicates, AMS incorporates a less rigorous competition
standard than the FAR imposes on the rest of the government. AMS states
that it is FAA's policy to provide reasonable access to competition for
firms interested in obtaining contracts. According to AMS, in selecting
sources, the preferred method of procurement is to compete requirements
among two or more sources. However, there is no requirement to ensure that
firms that want to participate actually get a chance to do so. Instead FAA
may limit competition for further consideration in its screening process
to firms with known capabilities or past performance.

The FAR Gives Procurement Professionals Tighter Control over Procurement
Decisions

AMS states that authority is delegated to appropriate levels. Once the
final investment decision is made, and subject only to any constraints
imposed by that decision, the service-level organization is responsible
for conducting required acquisitions. Contracting personnel as well as
other specialists are then assigned to teams that are responsible to a
program manager within the service-level organization. FAA states that
this approach increases the pace of doing business. By comparison, the FAR
gives contracting professionals clear control over contracting decisions
by requiring that procurement decisions be made by procurement
professionals-typically contracting officers or their superiors.

Although FAA Project Managers View AMS as More Efficient and Flexible Than
the FAR, Some Procurement Officials We Interviewed Do Not Agree

As part of our work, we interviewed project management personnel within
FAA as well as current and former FAA procurement officials that have
experience using both the FAR and AMS. Generally, FAA personnel see AMS as
more efficient and flexible than the FAR, although 9 years after AMS's
adoption, many FAA officials have only limited knowledge of and experience
with the FAR. The FAA project managers we interviewed see AMS as more
efficient and flexible than the FAR,14 but some procurement officials with
experience in applying both AMS and the FAR did not agree with the view
that the FAR was unduly rigid. According to these officials, the FAR may
appear inflexible and cumbersome to persons who are inexperienced with it,
but those who are familiar with it are able to navigate its complexities
effectively. For example, even though the FAR generally requires full and
open competition-a process that can take time to give all interested firms
an opportunity to participate-contracting officers may be able to expedite
the procurement process by using authorized streamlined procedures or, if
circumstances warrant, by justifying sole-source or limited competition.

Status of the Seven ATC Modernization Acquisitions That GAO
ReviewedAppendix V

Key ContributorsAppendix VI

GAO Contacts

JayEtta Z. Hecker, (202) 512-2834 Beverly L. Norwood, (202) 512-2834

Staff Acknowledgments

In addition to the individuals named above, Tamera Dorland, Elizabeth
Eisenstadt, Brandon Haller, Bert Japikse, Carolyn Kirby, Steve Martinez,
Richard Scott, Adam Vodraska, and Dale Yuge made key contributions to this
report.

(540072)

How FAA's Acquisition Policy Adapted Key Recommendations Made by GAO and
DOT (1996-2003)Appendix IV

FAA Refined AMS in Response to Recommendations

Since FAA developed and implemented AMS in 1996, GAO and the DOTIG have
made recommendations to improve FAA's acquisition processes.  FAA has
adopted many of these recommendations and incorporated them into AMS (see
table 9). These implemented recommendations address four main themes:

o Developing a strategy for culture change that relies on successfully
integrating the various elements of acquisition, including specific
responsibilities and performance measures for all stakeholders, and
providing the incentives needed to promote the desired changes.

o Establishing an effective management structure for developing,
maintaining, and enforcing the ATC systems architecture to provide an
overall plan for the National Airspace System (NAS). This management
structure should assign the responsibility and accountability to develop,
maintain, and enforce a complete and unified ATC system by ensuring that
every project conforms to the overall plan.

o Improving cost and schedule tracking to provide data for estimating the
costs and schedules of programs. To estimate the costs and time needed for
projects, a historical database that includes cost and schedule estimates,
revisions, reasons for revisions, actual cost and schedule information,
and relevant contextual information is needed.

o Improving the management of modernization projects, including the use of
project reviews, milestones, and baselines, and cost-accounting
information to ensure that programs can be adjusted as needed.

The reports identified in table 10 provide recommendations to address
problems we and the DOTIG have identified under these four themes.

Table 9:

                                        

      Key recommendation         Evidence of policy    Rationale for change   
                                       change         
Aviation Acquisition: A     FAA issued an          Over the past 15 years, 
Comprehensive Strategy Is   organizational culture FAA's ATC modernization 
Needed for Cultural         framework in 1997 and  projects have           
Change at FAA               is working to          experienced substantial 
                               implement it.          cost overruns, lengthy  
August 22, 1996,                                   schedule delays, and    
                                                      significant performance 
(GAO/RCED-96-159)                                  shortfalls. We found    
                                                      that FAA's              
FAA should develop a                               organizational culture  
comprehensive strategy                             has been an underlying  
for cultural change. This                          cause of the agency's   
strategy should include                            acquisition problems.   
specific responsibilities                          Its acquisitions were   
and performance measures                           impaired because        
for all stakeholders                               employees acted in ways 
throughout FAA and                                 that did not reflect a  
provide the incentives                             strong commitment to    
needed to promote the                              mission focus,          
desired behaviors and to                           accountability,         
achieve agencywide                                 coordination, and       
cultural change.                                   adaptability.           
Air Traffic Control:        Chapter 19 of FAA's    We found that FAA's ATC 
Improved Cost Information   Pricing Handbook       modernization program's 
Needed to Make Billion      embodies SEI's         cost estimating         
Dollar Modernization        philosophy, which      processes do not        
Investment Decisions        maintains that         satisfy recognized      
                               developing credible    estimating requisites,  
January 22, 1997,           software estimates is  and its cost-accounting 
                               a function of how      practices do not        
(GAO/AIMD-97-20)            thorough and           provide for proper      
                               disciplined an         accumulation of actual  
Because the success of      organization's         costs. The result is an 
FAA's investment analysis   estimating processes   absence of reliable     
and decision-making         are. SEI's six         project cost and        
process depends in large    institutional process  financial information   
measure on the              requisites are         that the Congress has   
reliability of ATC          designed to ensure     legislatively specified 
project cost information,   that organizations     and that leading        
FAA should                  consistently produce   public-sector and       
institutionalize defined    reliable cost          private-sector          
processes for estimating    estimates for          organizations point to  
ATC projects' costs. At a   software-intensive     as essential to making  
minimum, these processes    systems. These         fully informed          
should include the          requisites are as      investment decisions    
following six               follows:               among competing ATC     
institutional process                              projects. Not having    
requisites, developed for   o a corporate memory,  this information,       
organizations that are      or historical          increases the           
building or acquiring       database(s), for       likelihood of poor ATC  
software-intensive          cataloging cost        investment decisions,   
systems by Carnegie         estimates, revisions,  not only when a project 
Mellon University's         reasons for revisions, is initiated but also   
Software Engineering        actual cost and        throughout its life     
Institute (SEI), an         schedule information,  cycle. It also means    
institution recognized      and other descriptive  that Congress does not  
for its expertise in        information, such as   have reliable cost      
software processes. Each    any constraints or     information to use in   
of these requisites is      trends that affect the making funding          
described in more detail    project;               decisions about FAA.    
in this report:                                    Such a situation is     
                               o structured processes unacceptable when       
o a corporate memory, or    for estimating         making small            
historical database(s),     software size and the  investments, but is     
which includes cost and     amount and complexity  especially egregious    
schedule estimates,         of existing software   when making             
revisions, reasons for      that can be reused;    multimillion or         
revisions, actual cost                             billion-dollar          
and schedule information,   o cost models          investments in          
and relevant descriptive    calibrated/tuned to    mission-critical ATC    
information;                reflect demonstrated   systems.                
                               accomplishments on     
o structured approaches     similar past projects; 
for estimating software                            
size and the amount and     o audit trails that    
complexity of existing      record and explain the 
software that can be        values used as cost    
reused;                     model inputs;          
                                                      
o cost models               o processes for        
calibrated/tuned to         dealing with           
reflect demonstrated        externally imposed     
accomplishments on past     cost or schedule       
projects;                   constraints to ensure  
                               the integrity of the   
o audit trails that         estimating process;    
record and explain all                             
values used as cost model   o data collection and  
inputs;                     feedback processes     
                               that foster capturing  
o processes for dealing     and correctly          
with externally imposed     interpreting data from 
cost or schedule            work performed.        
constraints in order to                            
ensure the integrity of                            
the estimating process;                            
                                                      
o data collection and                              
feedback processes that                            
foster capturing and                               
correctly interpreting                             
data from work performed.                          
FAA should immediately      Chapter 19 of FAA's    
begin disclosing the        Pricing Handbook       
inherent uncertainty and    incorporates our       
range of imprecision in     recommendation and     
all ATC projects'           refers explicitly to   
official cost estimates     GAO/AIMD-97-20 and the 
presented to executive      work of other experts. 
oversight agencies or       The handbook suggests  
Congress.                   where to incorporate   
                               audit trails,          
                               constraint processes,  
                               and the inherent       
                               uncertainty and range  
                               of imprecision in all  
                               ATC cost estimates.    
                               The handbook advocates 
                               that staff qualify     
                               early project          
                               estimates by           
                               disclosing the level   
                               of uncertainty         
                               associated with them   
                               and refining the       
                               estimates as the       
                               project is completed   
                               and the uncertainty    
                               eliminated.            
FAA should acquire or       The Department of      
develop and implement a     Transportation is in   
managerial                  the process of meeting 
cost-accounting             key objectives of the  
capability that will        Federal Managers'      
satisfy the requirements    Financial Integrity    
of Statement of Federal     Act (FMFIA) of 1982. A 
Financial Accounting        key material weakness  
Standards no. 4 (SFFAS 4)   was FAA's oversight of 
Managerial Cost             cost reimbursable      
Accounting Concepts and     contracts. FAA made    
Standards for the Federal   significant progress   
Government. This system     in the closeout of     
capability should provide   past cost reimbursable 
the cost-accounting and     contracts. To resolve  
financial management        this material          
information needed by FAA   weakness, FAA needs to 
management and those who    complete the close out 
make investment             of old contracts and   
decisions. Such             increase the use of    
information should          cost incurred audits.  
include full life-cycle     Additionally, FAA      
costs, which include the    needs to ensure that   
costs of resources          appropriate audits are 
consumed by a project       obtained for all       
that directly or            active contracts.      
indirectly contribute to    These steps will help  
the output and the costs    reduce the likelihood  
of identifiable             of cost overruns or    
supporting services         improper payments for  
provided by other           unallowable costs.     
organizations within the                           
reporting entity.                                  
FAA should report its                              
lack of a cost-accounting                          
capability for its ATC                             
modernization as a                                 
material internal control                          
weakness in the                                    
Department's fiscal year                           
1996 Federal Managers'                             
Financial Integrity Act                            
(FMFIA) report and in                              
subsequent annual FMFIA                            
reports until the problem                          
is corrected.                                      
FAA should report to the                           
Secretary of                                       
Transportation and FAA's                           
authorizing and                                    
appropriation committees                           
on its progress in                                 
implementing these                                 
recommendations as part                            
of its fiscal year 1999                            
budget submission.                                 
Air Traffic Control:        AMS states the         FAA lacks a complete    
Complete and Enforced       National Air Space     system architecture, or 
Architecture Needed for     (NAS) Configuration    overall blueprint, to   
FAA Systems Modernization   Control Board shall    guide and constrain the 
                               approve changes to NAS development and         
February 3,1997,            technical              maintenance of the many 
                               documentation, and     interrelated systems    
(GAO/AIMD-97-30)            shall ensure the       that make up its ATC    
                               traceability of        infrastructure. To its  
FAA should ensure that a    requirements from the  credit, FAA is          
complete ATC systems        NAS level to the       developing one of the   
architecture is developed   system and subsystem   two principal           
and enforced                level. This            components of a         
expeditiously before        responsibility begins  complete systems        
deciding on the             with the approval of   architecture, namely,   
architectural               the technical          the "logical"           
characteristics of a        architecture by the    description of FAA's    
replacement for the Host    Joint Resources        current and future      
Computer System. FAA        Council at the         concept of ATC          
should also take the        investment decision    operations as well as   
following steps to          and continues          descriptions of the ATC 
establish an effective      throughout the life of business functions to   
management structure for    the program.           be performed, the       
developing, maintaining,                           associated systems to   
and enforcing the           AMS states that the    be used, and the        
complete ATC systems        Joint Resources        information flows among 
architecture:               Council approves FAA   systems. However, FAA   
                               budget submissions for is not developing, nor  
o Assign the                Research, Engineering  does it have plans to   
responsibility and          and Development (RE&D) develop, the second     
accountability needed to    and Facilities and     essential component-the 
develop, maintain, and      Equipment (F&E)        ATC-wide "technical"    
enforce a complete ATC      appropriations,        descriptions that       
systems architecture to a   participates in the    define all required     
single FAA organizational   development of FAA     information technology  
entity.                     budget submissions for (IT) and                
                               the operations         telecommunications      
o Provide this single       appropriation, and     standards and critical  
entity with the             approves the NAS       ATC systems' technical  
resources, expertise, and   architecture baseline. characteristics.        
budgetary and/or                                                           
organizational authority    AMS states that a      We also found that an   
needed to fulfill its       configuration control  architecture is the     
architectural               board with an approved centerpiece of sound    
responsibilities.           charter and operating  systems development and 
                               procedures shall be    maintenance;            
o Direct this single        the official FAA-wide                          
entity to ensure that       forum used to          FAA is developing a     
every ATC project           establish              logical architecture    
conforms to the             configuration          component for ATC       
architecture unless         management baselines   modernization and       
careful, thorough, and      and to approve or      evolution; FAA lacks a  
documented analysis         disapprove subsequent  technical architectural 
supports an exception.      changes to those       component to guide and  
Given the importance and    baselines.             constrain ATC           
the magnitude of the IT                            modernization and       
initiative at FAA, a                               evolution; without a    
management structure                               technical ATC           
similar to the                                     architecture, costly    
department-level chief                             system                  
information officer (CIO)                          incompatibilities have  
structure prescribed in                            resulted and            
the Clinger-Cohen Act                                                      
should be established for                          will continue; and FAA  
FAA.                                               lacks an effective      
                                                      management structure    
                                                      for developing and      
                                                      enforcing an ATC        
                                                      systems architecture.   
Air Traffic Control:        FAA states that the    To accommodate          
Immature Software           CIO:                   forecasted growth in    
Acquisition Processes                              air traffic and replace 
Increase FAA System         o serves as the        aging equipment, FAA    
Acquisition Risks           principal adviser to   embarked on an          
                               the Administrator,     ambitious ATC           
March 21, 1997,             Deputy Administrator,  modernization program   
                               and FAA offices on     in 1981. FAA estimated  
(GAO/AIMD-97-47)            information management that it would spend     
                               and technology across  about $20 billion to    
Given the importance and    the agency. As the     replace and modernize   
the magnitude of IT at      agency's senior        software-intensive ATC  
FAA, this report            management official,   systems between 1982    
reiterates our earlier      serves as the          and 2003. Our work over 
recommendation calling      spokesperson on IT     the years has           
for the establishment at    matters before         chronicled many FAA     
FAA of a CIO management     Congress, other        failures in meeting ATC 
structure similar to the    agencies, and the      projects' cost,         
department-level CIO        public;                schedule, and           
structure prescribed in                            performance goals,      
the Clinger-Cohen Act of    o leads and directs    largely because of      
1996.                       agencywide strategic   software-related        
                               planning for IT;       problems. As a result   
To improve its ability to                          of these failures as    
acquire software for its    o oversees IT          well as the tremendous  
ATC modernization, FAA      investments to ensure  cost, complexity, and   
should                      optimization across    mission criticality of  
                               all agency groups and  FAA's ATC modernization 
o assign responsibility     the full range of cost program, we designated  
for software acquisition    trade-offs;            the program as a        
process improvement to                             high-risk IT initiative 
the agency's CIO;           o creates and          in our 1995 and 1997    
                               maintains an IT        report series on        
o provide the CIO with      strategy to guide      high-risk programs.     
the authority needed to     research, development,                         
implement and enforce ATC   maintenance, and       Software quality is     
modernization software      sharing of information governed largely by the 
acquisition process         systems, applications, quality of the          
improvement;                data, and other        processes involved in   
                               resources across the   developing or           
o require the CIO to        lines of business and  acquiring, and          
develop and implement a     throughout the agency; maintaining it. SEI has 
formal plan for ATC                                developed models and    
modernization software      o leads the            methods that define and 
acquisition process         establishment of       determine               
improvement that is based   world-class software   organizations' software 
on the software             and information        process maturity.       
capability evaluation       systems engineering    Together, they provide  
results contained in this   methodologies          a logical framework for 
report and specifies        including Capability   baselining an           
measurable goals and time   Maturity Models, and   organization's current  
frames, prioritizes         applies them to agency process capabilities    
initiatives, estimates      systems, operations,   (i.e., strengths and    
resource requirements,      and processes to       weaknesses) and         
and assigns roles and       provide continuous     providing a structured  
responsibilities;           improvement of IT      plan for incremental    
                               performance; and       rocess improvement.     
o allocate adequate                                                        
resources to ensure that    o leads and directs    We found that           
planned initiatives are     agencywide efforts on                          
implemented and enforced;                          o FAA's ATC             
and                         information systems    modernization software  
                               security, ensuring     acquisitions processes  
o require that, before      that standards and     are immature and        
being approved, every       policies are in place                          
                               to provide security    o FAA's approach for    
ATC modernization           for the critical       improving AT            
acquisition project have    information                                    
software acquisition        architecture of the    modernization software  
processes that satisfy at   agency.                acquisition processes   
least Software                                     is not effective.       
Acquisition Capability                             
Maturity Model (SA-CMM)                            
level 2.                                           
Air Traffic Control:        FAA's AMS states that  Over the past 17 years, 
FAA's Modernization         five decisions are     FAA's modernization     
Investment Management       always made at the     projects have           
Approach Could Be           corporate level by the experienced substantial 
Strengthened,               Joint Resources        cost overruns, lengthy  
                               Council: the mission   delays, and significant 
April 30, 1999,             need decision, the     performance shortfalls. 
                               investment decision,   Because of FAA's        
(GAO/RCED/AIMD-99-88)       the decision to        contention that some of 
                               approve a change to an its modernization       
FAA should implement a      acquisition program    problems were caused by 
comprehensive investment    baseline, approval of  federal acquisition     
management approach         the RE&D and F&E       regulations, the        
through AMS that includes   budget submissions,    Congress enacted        
the following actions:      and approval of the    legislation in November 
                               NAS Architecture       1995 that exempted the  
o Establish a complete      baseline. The          agency from most        
portfolio of                selection of a         federal procurement     
investments-including       solution to satisfy a  laws and regulations    
existing systems funded     mission need, the      and directed FAA to     
by the operations budget    investment of          develop a new           
account as well as          resources into a fully acquisition management  
projects funded by the      funded program, and    system. In response,    
facilities and equipment    the possible need to   FAA implemented AMS on  
account-and require the     cancel other programs  April 1, 1996. AMS      
Joint Resources Council     to accommodate a new   provides high-level     
to periodically review      program make the       acquisition policy and  
the baseline status and     investment decision    guidance for selecting  
merits of each of these     the most important in  and controlling         
investments throughout      the life-cycle         investments throughout  
their entire life cycle.    management process.    all phases of the       
As part of this                                    acquisition life        
portfolio, cost baselines                          cycle.                  
for operating and                                                          
maintaining all projects                           GAO found that:         
should be developed, and                                                   
this information should                            o FAA's AMS is designed 
be included in the                                 to provide a            
agency's financial plan                            discipline, structured  
for its investments and                            process for selecting   
in its annual budget                               and controlling         
request to Congress.                               investments;            
                                                                              
                                                      o Lack of oversight of  
                                                      the operations portion  
                                                      of projects prevents    
                                                      FAA from managing       
                                                      investments as a        
                                                      complete portfolio;     
                                                                              
                                                      o Weaknesses in         
                                                      selection, control, and 
                                                      evaluation phases limit 
                                                      FAA's effectiveness in  
                                                      managing its portfolio. 
o Improve the selection   o FAA's AMS states that  
process by (1)              the investment         
establishing clearly        analysis team develops 
defined procedures for      an initial acquisition 
validating each project's   program baseline       
cost, schedule, benefit,    (i.e., performance,    
performance, and risk       cost, schedule,        
information and (2)         benefits, and risk)    
requiring documentation     for each alternative   
of the results of the       solution offering      
validation procedures       superior value and     
applied to each project.    benefit to FAA and its 
                               customers. Service     
                               organization members   
                               of the investment      
                               analysis team lead the 
                               development of cost    
                               and schedule baselines 
                               using FAA's work       
                               breakdown structure    
                               and other applicable   
                               standards.             
o Strengthen control over o AMS states that the    
investments by (1)          acquisition program    
revising the acquisition    baseline should        
program baseline            include cost,          
requirements to include     schedule, performance, 
project risks and to add    benefits, and risk     
milestones for project      information. It also   
reviews during the          should include all     
operations phase and (2)    events that are key to 
ensuring that project       satisfying mission     
officials fully track and   need, providing        
document estimated versus   intended operational   
actual results for all      capability, and        
the elements (i.e., cost,   accruing benefits, as  
schedule, benefit,          well as events crucial 
performance, and risk)      to interrelated        
contained in the baseline   programs or NAS        
documentation.              systems. Once an       
                               estimate has been      
                               completed and a        
                               project started, FAA   
                               establishes reporting  
                               and performance        
                               measures to compare    
                               estimated and actual   
                               costs, schedules, and  
                               performance.           
o Initiate post           o FAA published a        
implementation              methodology for        
evaluations for projects    conducting such        
within 3 to 12 months of    evaluations entitled   
deployment or               An Approach for        
cancellation to compare     Developing a Standard  
the completed projects'     Method for Conducting  
cost, schedule,             Post-Implementation    
performance, and mission    Reviews, Report        
improvement outcomes with   #2001-13, June 6,      
the original estimates.     2001.                  
                                                      
o Incorporate key                                  
information from the                               
selection process (e.g.,                           
mission need statements,                           
cost-benefit analyses,                             
and risk assessments)                              
into FAA's management                              
information system for                             
investments.                                       
Major Management            FAA has appointed an   DOT's management of its 
Challenges and Program      independent            major acquisitions and  
Risks, Department of        board-consisting of    assets needs            
Transportation              external experts in    improvement in several  
                               satellite navigation,  areas. FAA and the U.S. 
January 2001,               safety certification,  Coast Guard are         
                               and radio              undertaking costly,     
(GAO-01-253)                spectrum-that reports  long-term programs to   
                               directly to the FAA    modernize and replace   
FAA should develop a        Administrator. The     aging equipment. Over   
comprehensive plan that     board is tasked with   the past 19 years,      
would include established   reviewing the          FAA's                   
checkpoints at which the    soundness of the       multibillion-dollar ATC 
agency would determine,     panel's                modernization program   
among other things,         recommendations and    has experienced cost    
whether users' needs have   with revalidating the  overruns, delays, and   
changed and whether other   future path for WAAS.  performance shortfalls  
technologies have matured   However, given the     of large proportions.   
and could better meet       past problems in       FAA is making progress  
users' needs and the        developing this system in addressing some of   
agency's requirements for   and the long-term      our recommendations,    
satellite navigation. FAA   effort that is still   but its reform efforts  
should also have an         required, we believe   are not complete, and   
external organization       that continued         major projects continue 
evaluate its progress at    oversight by an        to face cost, schedule, 
established checkpoints     independent group of   and performance         
and include the results     experts is warranted.  problems. Because of    
of this evaluation in its   It is not clear        its size, complexity,   
request for future          whether the current    cost, and               
funding of the navigation   independent board will problem-plagued past,   
system.                     fulfill this role. We  we designated FAA's IT  
                               will continue to       program as a high-risk  
                               evaluate FAA's         IT initiative in 1995.  
                               progress on this and   
                               other system           
                               acquisition efforts.   
Status of FAA's Major       FAA officials          FAA has made progress   
Acquisitions                generally agreed with  with a number of        
                               the analysis and       acquisitions, including 
DOT/OIG, AV-2003-045,       recommendations in     Free Flight Phase 1 and 
                               this report. FAA is    new information         
June 26, 2003               implementing this      exchange systems that   
                               recommendation. It     link FAA and airline    
Update the cost,            updated the baseline   operations centers.     
schedule, and performance   of STARS in April 2004 However, other          
baselines for many of       and updated the        modernization programs  
FAA's major acquisition,    baselines of ITWS and  have experienced cost,  
including STARS, ITWS,      WAAS in May 2004. The  schedule, and           
LAAS, and WAAS at a         LAAS program was       performance problems.   
minimum. Develop-and        deferred because of    Problems with           
use-performance goals for   budget cuts.           acquisition efforts     
assessing progress with                            have serious            
its major                                          consequences because    
acquisitions. This should                          they result in costly   
involve holding staff and                          interim systems, reduce 
contractors accountable                            the number of units     
for keeping projects                               procured, postpone      
within cost and schedule,                          benefits, or "crowd     
as appropriate.                                    out" other              
                                                      modernization projects. 
Status Report on FAA's      FAA officials          The OEP is an important 
Operational Evolution       generally agreed with  effort because it will  
Plan                        the analysis and       shape FAA and industry  
                               recommendations in     investments over the    
DOT/OIG, AV-2003-048,       this report. FAA is    next decade. However,   
                               currently updating the much has changed since  
July 23, 2003               OEP, which includes    the OEP was             
                               design changes to the  introduced.  The demand 
Develop realistic cost      National Airspace to,  for air travel has      
estimates, and link the     for example, enhance   declined, major network 
Operational Evolution       capacity.              carriers are in         
Plan (OEP) with FAA's                              financial distress, and 
budget in order to set                             Aviation Trust Fund     
priorities for what can                            revenues have declined  
be accomplished in the                             sharply. The Inspector  
short term.                                        General found that      
                                                      fundamental assumptions 
Determine-in concert with                          about the OEP, such as  
the aviation                                       the cost, schedule, and 
community-how to move                              benefits of key efforts 
forward (and at what                               as well as the ability  
pace) with systems that                            of airspace users to    
require airspace users to                          pay for and equip with  
purchase and install new                           new technologies in the 
technologies.                                      near term, are no       
                                                      longer valid and need   
Determine and maximize                             to be revised.          
the benefits associated                            
with airspace design                               
changes, new procedures,                           
and capabilities                                   
currently onboard                                  
aircraft to enhance                                
system capacity.                                   

Key Recommendations Made to Improve FAA's Acquisition Processes

Source: GAO analysis.

sector best practices and that the agency had highly developed processes
for four of these six elements. See figure 4.

Figure 4: Our Analysis of FAA's Progress as of 2003 in Implementing Key
Elements of Training for Its Acquisition Workforce

Since 2003, FAA has taken some steps to measure the effectiveness of its
training. For example, the agency collects and reviews participants'
assessments of the knowledge they have gained, the extent that learning
objectives were achieved and the applicability and usefulness of the
training. In addition, members of FAA's Intellectual Capital Investment
Plan Council35 have attempted to make qualitative judgments about the
impact of the training on the effectiveness or efficiency of their
organizations. However, FAA is still developing an evaluation program with
metrics to measure the extent to which organizational goals are achieved
when individual training objectives are met. Industry and government
experts believe training and human capital investments are prerequisites
for
*** End of document. ***