Air Traffic Control: FAA's Acquisition Management Has Improved,
but Policies and Oversight Need Strengthening to Help Ensure
Results (12-NOV-04, GAO-05-23).
The Federal Aviation Administration's (FAA) multibillion-dollar
effort to modernize the nation's air traffic control (ATC) system
has resulted in cost, schedule, and performance shortfalls for
over two decades and has been on GAO's list of high-risk federal
programs since 1995. According to FAA, performance shortfalls
were due, in part, to restrictions imposed by federal acquisition
and personnel regulations. In response, Congress granted FAA
exemptions in 1995 and directed it to develop a new acquisition
management system. In this report, GAO compared FAA's AMS with
(1) the FAR and (2) commercial best practices for major
acquisitions, and (3) examined FAA's implementation of AMS and
its progress in resolving problems with major acquisitions.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-05-23
ACCNO: A13534
TITLE: Air Traffic Control: FAA's Acquisition Management Has
Improved, but Policies and Oversight Need Strengthening to Help
Ensure Results
DATE: 11/12/2004
SUBJECT: Air traffic control systems
Best practices
Comparative analysis
Comparative benchmarking products
Federal procurement
Performance measures
Procurement policy
Procurement practices
FAA Acquisition Management System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-05-23
* Report to the Chairman and Ranking Minority Member, Committee on
Governm\ent Reform, House of Representatives
* November 2004
* AIR TRAFFIC CONTROL
* FAA's Acquisition Management Has Improved, but Policies and
Oversight Ne\ed Strengthening to Help Ensure Results
* Contents
* Results in Brief
* Background
* AMS Is Broader and Less Prescriptive Than the FAR
* AMS Addresses Both Procurement and Project Management,
Whereas the FAR F\ocuses Primarily and in Far Greater Detail
on Procurement
* AMS Provides Broad Guidance While the FAR Establishes
Detailed Requireme\nts, but Managers Have Flexibility under
Both
* AMS Provides Some Discipline but Does Not Ensure a
Knowledge-Based Appro\ach to Acquisition
* Best Practices for Managing Acquisitions Call for a
Knowledge-Based Appr\oach, Including Criteria for Knowledge
Needed and Oversight at the Corpo\rate Executive Level
* AMS Has Some Good Features but Does Not Ensure That High
Levels of Knowl\edge Are Attained Before Major Commitments
Are Made
* As Implemented, AMS Has Not Resolved Long-standing Acquisition
Problems,\ but FAA Is Beginning to Focus More on Results
* Our Reviews of Seven Major Systems Show That Problems with
Requirements \and Software Management Persist under AMS
* Inadequate Development or Definition of Requirements Led to
Requirements\ Growth or Unplanned Work for Five Acquisitions
* FAA Underestimated Software Complexity for Three
Systems
* ATC Systems Have Required Multiple Rebaselining
Decisions to Address Del\ays and Cost Growth
* Internal and External Reviews Have Found That FAA Has Made
Some Progress\ but Continues to Experience Problems in
Acquiring Major ATC System unde\r AMS
* FAA's ATO Is Taking Steps to Improve Major ATC Acquisitions
* Improvements to Requirements Development
* Improvements to Managing Software and System
Acquisition and Development\
* Improvements to Estimating Costs
* Other Improvement Efforts
* Conclusions
* Recommendations for Executive Action
* Agency Comments
* FAA Has Begun Analyzing Spending Trends to Take a More Strategic
Approac\h to Procurement
* Objectives, Scope, and Methodology
* Comparison of the Scope and Flexibility of FAA's Acquisition
Management \System and the Federal Acquisition Regulation Process
* Background
* AMS Defines an Investment/Life-Cycle Project Management System
* Only a Portion of AMS Deals Directly with the Procurement Process
* AMS States a Nonregulatory FAA Policy
* AMS Chapter 3 Parallels a Subset of the FAR
* AMS Includes a Less Rigorous Competition Requirement Than Does
the FAR
* The FAR Gives Procurement Professionals Tighter Control over
Procurement\ Decisions
* Although FAA Project Managers View AMS as More Efficient and
Flexible Th\an the FAR, Some Procurement Officials We Interviewed
Do Not Agree
* How FAA's Acquisition Policy Adapted Key Recommendations Made by GAO
and\ DOT (1996-2003)
* FAA Refined AMS in Response to Recommendations
* Status of the Seven ATC Modernization Acquisitions That GAO Reviewed
* Key Contributors
* GAO Contacts
* Staff Acknowledgments
* app4new.pdf
* How FAA's Acquisition Policy Adapted Key Recommendations Made by
GAO and\ DOT (1996-2003)
* FAA Refined AMS in Response to Recommendations
Report to the Chairman and Ranking Minority Member, Committee on
Government Reform, House of Representatives
November 2004
AIR TRAFFIC CONTROL
FAA's Acquisition Management Has Improved, but Policies and Oversight Need
Strengthening to Help Ensure Results
Contents
Tables
Figures
Abbreviations
November 12, 2004erLetter
The Honorable Tom Davis Chairman The Honorable Henry A. Waxman Ranking
Minority Member Committee on Government Reform House of Representatives
In late 1981, the Federal Aviation Administration (FAA) began a
modernization program to replace and upgrade the National Airspace
System's (NAS) equipment and facilities to meet the expected increase in
traffic volume, enhance the margin of safety, and increase the efficiency
of the air traffic control (ATC) system-the principal component of the
NAS. Historically, the modernization program has experienced cost
overruns, schedule delays, and performance shortfalls of large proportions
and has been on our list of high-risk programs since 1995. To date, FAA
has spent $41 billion and expects to spend an additional $7.6 billion
through fiscal year 2007 to, among other things, finalize key
modernization projects designed to replace radar, navigation,
communications, and information-processing systems.1
According to FAA, the performance shortfalls in its modernization program
were due, in part, to restrictions imposed by federal acquisition and
personnel requirements. In response, Congress passed legislation in 1995
that granted FAA unique acquisition and personnel exemptions, or
flexibilities, and directed FAA to develop a new acquisition management
policy. FAA issued its new acquisition management policy, called the
Acquisition Management System (AMS), in 1996 and began using the new
system instead of the Federal Acquisition Regulation (FAR). To further
address long-standing weaknesses in the ATC modernization program, the
President and Congress in 2000 directed FAA to reorganize and establish a
new organization. FAA has just begun to do so.
Now that FAA has had several years to implement the earlier procurement
flexibilities, as well as some time to reorganize, some results of its
acquisition reform should be discernable. Moreover, FAA's experiences in
exercising its acquisition flexibilities could provide valuable
information to Congress in overseeing the use of these flexibilities.
You asked us to review the steps that FAA has taken to reform its
acquisition of major ATC systems and the impact of the reforms on FAA's
acquisition outcomes. Specifically, you asked us to (1) compare the scope
and flexibility of AMS and the FAR, (2) compare AMS with commercial best
practices for major acquisitions, and (3) examine FAA's implementation of
AMS and progress in addressing long-standing problems with major
acquisitions. In addition, you asked us to review FAA's general
procurement of goods and services; we cover this topic in appendix I.
To address the first objective, we compared the topics addressed by, and
the implementation options afforded to contracting and procurement
officials under AMS and the FAR. To address the second objective, we used
a model of best practices that we derived from our body of work on how
leading private firms manage costly and complex product developments and
how the Department of Defense (DOD) manages major weapon systems
acquisitions.2 We used this model to assess the extent to which FAA's
acquisition management policy mirrors the acquisition policies of
high-performing organizations in the public and private sectors. This
model consists of four phases: (1) concept and technology development; (2)
product development, which includes both integration and demonstration
activities; (3) production; and (4) operations and support. In between
these four phases are three key knowledge decision points at which
commercial firms and the government must have sufficient knowledge to make
large investment decisions. To address the third objective, we selected
the seven ATC systems with the largest budgets to explore the results of
FAA's implementation of its acquisition management policy and procedures
and to determine how FAA has addressed issues found to have contributed to
cost, schedule, or performance problems. In selecting these seven systems,
we ensured that some were initiated before and some after April 1996, when
FAA implemented AMS. While the results of these analyses are not
generalizable to all of FAA's major ATC acquisitions, they indicate the
extent to which the agency has made progress in addressing long-standing
problems we have identified. To further assess both the implementation and
the impact of FAA's acquisition reforms, we reviewed our work on FAA's
major ATC acquisition efforts since 1996 as well as the work of the
Department of Transportation's Office of Inspector General (DOTIG), FAA,
and others. We also reviewed the actions that FAA has taken to refine AMS
in response to internal and external reviews. Finally, to review FAA's
procurement of goods and services across the agency, we used a commercial
best-practices model for taking a more strategic approach to procurement,
along with interviews with key agency officials, to determine whether FAA
has begun to analyze spending trends to identify opportunities to leverage
its buying power. We conducted our work from December 2003 through
November 2004 in accordance with generally accepted government auditing
standards. See appendix II for additional information on our objectives,
scope, and methodology.
Results in Brief
AMS consists of broad guidance for acquisition life-cycle management-from
defining the requirements for a system through fielding (deploying) and
decommissioning it (removing it from service). This broad guidance
contrasts with the rather more detailed and prescriptive
contract-formation and contract-administration requirements contained in
the FAR. AMS is broader in scope because it addresses, among other areas
of life-cycle management, both contract and program management, providing
both policies and procedures for contracting and a toolset of recommended
practices for managing individual acquisition projects over their life
cycles. By contrast, the FAR focuses in far greater detail on contracting
policies and procedures. FAA managers believe they have greater
flexibility in interpreting and applying AMS than they would have under
the FAR, in part because, in areas addressed by both, AMS is less
directive than the FAR. For example, although AMS states a "preference"
for competition, FAA personnel may use single-source contracting when
necessary to fulfill FAA's mission. By contrast, other federal agency
contracting officials operating under the FAR are generally required to
seek "full and open competition"-a more rigorous standard. These other
agency officials can generally use sole-source or limited-competition
contracting only after higher-level agency procurement officials have
approved a written justification. In addition, FAA contracting personnel
operate as part of acquisition teams that are responsible to program
managers; under the FAR, contracting decisions are made by contracting
personnel who are responsible only to contracting officials. Nonetheless,
the FAR also affords flexibility because it encourages innovation and
addresses a wide selection of contracting methods; therefore, procurement
officials can choose the approach that they consider most appropriate to
their procurement. According to some current and former FAA procurement
officials with experience in using both the FAR and AMS, the FAR may
appear inflexible and cumbersome to inexperienced managers, but those who
are familiar with it can navigate it effectively.
AMS provides some discipline through its various phases, activities, and
decision points for acquiring major ATC systems; however, it does not
ensure the use of a knowledge-based approach found in the best practices
for managing commercial product developments and DOD acquisitions3 that we
have identified in numerous past reports. Commercial best practices call
for specific knowledge to be captured and used by corporate-level
decision-makers to determine whether a product has reached a level of
development (product maturity) sufficient to demonstrate its readiness to
move forward in the acquisition process. The capture of such knowledge and
its use by executives helps to avoid cost overruns, schedule slips, and
performance shortfalls that can occur if decision-makers commit to a
system design before acquiring critical technology, design, or
manufacturing knowledge. AMS has some good features, which indicate a
process that has some elements of discipline. For example, like the best
practices model, AMS identifies critical junctures that it terms "decision
points," the first three of which call for the preparation of detailed
technical and programmatic information that FAA's corporate
executive-level body, the Joint Resources Council,4 can use to assess
whether or not FAA should initiate an acquisition program. However, AMS
departs from recognized best practices primarily by (1) not requiring the
attainment of specific knowledge satisfying explicit written criteria for
decision-makers to use at each key decision point and (2) not requiring
corporate executive-level oversight at all key decisions. For example, AMS
allows the Joint Resources Council to delegate two key decisions-the
decision to begin production and the decision to place a system in
service. FAA maintains that this approach gives program managers
flexibility, expedites decision-making, and allows those executives with
the most knowledge about a major acquisition to make key decisions about
its continued development. FAA's reliance on delegation assumes that
managers will inform their superiors if they are unable to meet the
performance schedules and system requirements approved by the Joint
Resources Council. However, best practices call for more than this,
including the use of measurable criteria at key points in the acquisition
process to ensure that specific knowledge has been captured and the
independent review of this knowledge by corporate executive-level
decision-makers before the acquisition moves forward in its development.
These criteria and reviews are particularly important for acquisitions
that require a large funding commitment, such as those that include the
production of multiple costly units (e.g., radars and controller
workstations). In addition, oversight at the corporate-executive or
agencywide level is needed to ensure consideration of an acquisition's
likely impact on other agency projects or operations. These departures
from best practices put FAA's major ATC acquisitions at risk of cost,
schedule, and performance shortfalls. We are making recommendations to the
Secretary of Transportation to align AMS more closely with commercial best
practices.
According to our review of seven major ATC systems and analysis of FAA's
performance in acquiring major systems, AMS has not resolved management
problems that FAA experienced before it implemented AMS, but the agency is
beginning to focus more on the expected results of its major acquisitions.
(See table 5.) Specifically, our review found that AMS did not call for
requirements that were specific enough to minimize the development of
further requirements (requirements growth) or unplanned work in five of
these systems. This lack of specificity resulted in the inadequate
development or definition of requirements, requirements growth, unplanned
work, or a reduction in performance for five of these systems. In
addition, for three of these systems, FAA underestimated the difficulty of
modifying available software to fulfill its mission needs. Consequently,
FAA encountered unexpected software development needs, higher costs, and
schedule delays. Because AMS guidance was not sufficient to account for
the risks associated with modifying available software, the two systems we
reviewed that were initiated after AMS's implementation-though currently
meeting cost and schedule milestones-are nevertheless showing symptoms of
FAA's past problems with developing requirements and managing software. It
is too soon to tell if these two systems will remain within their cost,
schedule, and performance parameters. In addition, our work on FAA's major
acquisitions, along with that of the DOTIG and others, has shown that many
of the problems FAA experienced in acquiring major systems before 1996
persist under AMS and that effective acquisition management, rather than
the use of a specific contracting process (e.g., the FAR or AMS) is the
key to successful acquisitions. To its credit, FAA is beginning to focus
more on results, largely through its new Air Traffic Organization, which
has been charged with taking a more performance-based approach to managing
the agency's major acquisitions. This approach includes implementing a
training framework for FAA's acquisition workforce. While FAA has taken
some steps to develop an evaluation program with criteria for measuring
the extent to which this framework is achieving organizational goals by
improving the knowledge base of FAA's acquisition workforce, at the time
of our audit FAA had no plans to conduct a comprehensive evaluation. We
are making recommendations to the Secretary of Transportation to improve
FAA's development of requirements and management of complex software, and
to comprehensively evaluate FAA's implementation of the training framework
to ensure that it is having the intended effect of improving the knowledge
base of FAA's acquisition workforce. In commenting on a draft of this
report, FAA said that it generally agreed with the report's contents and
said that our recommendations would be helpful to them as they continue to
refine AMS.
Background
Maintaining that federal procurement requirements contributed to some of
its cost, schedule, and performance problems in the 1980s and early 1990s,
FAA sought a statutory exemption from the federal acquisition system,5
including the FAR, and those parts of title 5 of the United States Code,
parts II and III, that govern federal civilian personnel management.
According to FAA, exemptions from these requirements would enable it to
streamline its acquisition processes, be more responsive to the airline
industry's needs, and increase the efficiency of ATC operations while
maintaining safety. Congress enacted legislation in November 1995 that
exempted FAA from key federal procurement statutes and the FAR, and
directed FAA to develop a new acquisition management system. In response
to these legislative initiatives, FAA implemented a new, streamlined
acquisition process-the Acquisition Management System (AMS)- on April 1,
1996.
We developed a knowledge-based model of commercial best practices based on
our findings about how leading private firms manage costly and complex
acquisitions effectively-that is, within cost, schedule, and performance
targets. The use of this knowledge-based model has been found to reduce
the risks associated with developing products and increase the likelihood
of successful outcomes. The model divides the product development cycle
into four phases and related activities. Table 1 presents these phases and
activities and explains what takes place during each.
Table 1: Structure of Best Practices Model for Major Product Developments
Phase/Activity What occurs during this phase or activity
1.Concept and technology Leading companies work to understand their
development mission needs and confirm that the technologies
to be used are mature; that is, the technologies
needed to meet essential product requirements
have been demonstrated to work in their intended
environment.
2. Product development
o Integration Components and subsystems are integrated into
the product to stabilize the overall system
design and show that the design can meet the
product requirements.
o Demonstration Tests show that the product will work as
required and can be manufactured within targets.
3. Production Operational test articles are built.
4. Operations and support Our best practices model does not explicitly
cover operations and support activities;
however, this phase focuses on maintenance of
the system through its retirement.
Source: GAO.
AMS provides guidance for selecting and overseeing investments over their
life cycle. Like our best practices model, it is divided into phases and
activities, although the divisions sometimes occur at different points.
Table 2 summarizes AMS's phases and activities.
Table 2: Structure of AMS
Phase/Activity What occurs during this phase or activity
Needs and solution
identification
o Mission analysis FAA identifies a capability shortfall and
determines that it needs an investment to better
carry out its mission. Recently, FAA began
analyzing its mission needs within the context of
its overall goals for the National Airspace
System.
o Investment analysis FAA, using an investment analysis team, evaluates
alternatives, selects practical and affordable
solutions, and develops a baseline of cost,
schedule, and performance requirements. This
document is called the acquisition program
baseline.
Solution implementation
o System integration Both hardware and software components and
subsystems are integrated into a product. Also,
intra- and intersystem compatibility are tested
and analyzed.
o System demonstration Tests show that the product can work as required
and be manufactured within targets.
o System production All activities are carried out to produce needed
quantities. Each end item is tested before it
leaves the factory to verify that it conforms to
specifications and is free from manufacturing
defects.
In-service management All required activities are carried out, including
directly operating, providing maintenance
functions (both scheduled and unscheduled), and
furnishing technical and logistics support for the
maintenance of FAA systems, subsystems, services,
or equipment.
Source: FAA.
To implement the new, performance-based organization for managing ATC
modernization and operations, as the President and Congress directed in
2000, FAA appointed a chief operating officer in August 2003 and formally
established the Air Traffic Organization (ATO) in February 2004. ATO,
under the direction of a six-member executive council, is now responsible
for further implementing acquisition reforms for major ATC systems.
AMS Is Broader and Less Prescriptive Than the FAR
AMS establishes an acquisition life-cycle management system that
encompasses both contracting and program management, whereas the FAR is
primarily a contracting system that focuses on contract formation and
contract administration. As a result, AMS is broader in scope than the
FAR. See figure 1. In addition, AMS takes the form of guidance. This
guidance is expressed in documentation of FAA policy, handbooks,
templates, flowcharts, forms, and standard contract language. It is not
regulatory. By contrast, the FAR is a set of published regulations-a legal
foundation that has the force and effect of law for the federal agencies
that
are required to follow it.6 Furthermore, the FAR is more detailed and
prescriptive in establishing contracting requirements and can require more
administrative involvement. This fundamental difference between AMS and
the FAR may suggest to some that AMS is more flexible. FAA personnel can
choose how to apply AMS's provisions to a major acquisition. Nonetheless,
procurement officials under the FAR also have flexibility because the FAR
encourages innovation consistent with its direction (and other applicable
legal requirements), provides a wide selection of contracting solutions,
and permits contracting officials to choose the methods that they consider
most suitable for a given situation.
Figure 1: Scope of AMS and the FAR
Note: AMS provides policy for the four phases of life-cycle management, as
well as 14 functional areas, (e.g., test and evaluation, human factors,
procurement, real estate, security, and systems engineering).
aThe NAS in-service decision is a key program milestone that authorizes
the deployment of a system into the National Airspace System after
thoroughly testing the system to verify its operational readiness.
AMS Addresses Both Procurement and Project Management, Whereas the FAR
Focuses Primarily and in Far Greater Detail on Procurement
AMS comprises six policy sections and five appendixes.7 The procurement
policy section of AMS covers a range of topics, including contract funding
and administration, contracting with small and disadvantaged businesses,
and compliance with labor laws. According to this section, competition is
FAA's preferred method of contracting, but single-source contracting is
permitted when appropriate to fulfill the agency's mission. This policy
section also describes the procurement of commercially available or
nondevelopmental items.
Other sections of AMS cover project management tools that the FAR does not
address, such as investment analysis, configuration management,8 and
integrated logistics support.9 AMS also addresses areas that fall outside
project management and procurement, including real property management-an
area that becomes important when FAA must lease or purchase real property
so that it can install ATC systems such as radars or antennas on property
that it does not currently own. FAA's policy directs FAA staff to "conduct
this business in a fair and equitable manner following best practices."
Although the FAR includes requirements that address procurement planning10
and major systems acquisition,11 it does so only in the context of
government procurement policy and procedure. Agencies subject to the FAR
find the broader program planning requirements, which appear in AMS but
not in the FAR, in documents such as the Office of Management and Budget's
Circular A-109 and in their own planning guidance. For
example, DOD has issued a series of directives and instructions on this
subject.12
The contracting procedures set forth in section 3 of AMS do not prescribe
detailed contracting procedures for various categories of procurements, as
do those detailed under the FAR. Instead, AMS provides two basic
contracting models for obtaining products and services through FAA's
contracting process. The first model is called "Complex and Noncommercial
Source Selection" and is used for complex, large-dollar, developmental,
noncommercial items and services. This is the model that typically would
be used for investments approved by the Joint Resources Council. The
second model is called the "Commercial and Simplified Purchase Method" and
is typically used for commercial items that are less complex and less
costly. Procurements of such products or services may be routine in nature
and are generally purchased on a fixed-price basis. Generally, source
selection under AMS follows a screening process, with the awardee being
selected on a "best value" basis from among those who remain in
consideration when the selection is made.
AMS Provides Broad Guidance While the FAR Establishes Detailed
Requirements, but Managers Have Flexibility under Both
AMS sets out a nonregulatory FAA policy that is binding on FAA personnel
as FAA employees. AMS also sets out other guidelines that FAA states
should be followed unless there is a rational basis for doing otherwise.
AMS is subject to such internal controls and enforcement as the
Administrator decides and to general overarching legal requirements, such
as the Government Performance and Results Act of 1993 (GPRA).13 FAA has
also deemed certain acquisition laws applicable to its procurements
(sometimes with modifications), such as the Service Contract Act.14 There
is also a legal requirement, created by the 1995 legislation exempting FAA
from the FAR, that small and socially or economically disadvantaged firms
be given all reasonable opportunities to receive contract awards. FAA has
adopted a dispute resolution process with some legal underpinnings.15
Otherwise, as the preface to AMS states, "nothing in this document creates
or conveys any substantive [legal] rights." In short, although FAA is
subject to the general legal requirement that government decisions cannot
be arbitrary or capricious, AMS does not establish regulatory requirements
for the conduct of procurements and does not create or convey substantive
legal rights.
In contrast to AMS, the FAR is a set of published regulatory requirements.
It has the force and effect of law, and agencies that are subject to it
are bound to follow it. The FAR's requirements provide for a range of
procurement strategies and approaches. In addition to negotiated
procurement methods, it allows two-step sealed-bid and two-phase
design-build methods,16 among others. It includes streamlined procedures
for soliciting and evaluating offers to furnish commercial items, as well
as permits the use of simplified acquisition procedures in a broad range
of procurements. Furthermore, the FAR supports a diverse selection of
available contract types, product-testing tools, and other tools that an
agency's contracting personnel may select when conducting an acquisition
to meet the agency's needs.
Although contracting personnel in agencies subject to the FAR are required
to comply with it, they enjoy broad discretion in their management of
procurements. For example, the FAR allows wide latitude in drafting
requirements statements, from performance-based statements of work to
design specifications as necessary. It allows broad discretion in framing
solicitations and in conducting procurements, including scoring proposals,
determining how negotiations will be conducted, eliminating firms whose
proposals are not competitive, and selecting the awardees whose proposals
afford the government the best value when evaluated against the selection
criteria established in the solicitations.
Because AMS consists of broad guidance while the FAR comprises detailed
and prescriptive regulatory requirements, FAA managers view AMS as giving
them more flexibility than they would have under the FAR, particularly in
two areas-competition and oversight. Whereas the FAR generally requires
full and open competition, AMS calls for providing "reasonable access to"
competition to firms interested in obtaining contracts-a less rigorous
standard than full and open competition. AMS further states that the
"preferred" method of selecting sources is to compete requirements among
two or more sources. By contrast, full and open competition requires that
all responsible sources be permitted to compete.17 Under AMS, there is no
policy that firms that want to participate actually get a chance to do so.
Rather, FAA told us that its system is beneficial because the agency can
use screening requests to preselect competing firms, eliminating those
firms that FAA believes are not likely to receive an award. The following
example illustrates the differences between AMS and the FAR in their
respective requirements on exceptions to competition. FAA may contract
with a single source when this approach is determined to be in the best
interest of FAA.18 The FAR, however, allows exceptions to full and open
competition only for certain specified conditions (such as unusual and
compelling urgency or the availability of only one source). The FAR
describes in detail the circumstances of these conditions and the
requirements for using them as justification for not providing for full
and open competition. The FAR also requires the contracting officer to
prepare a justification document that must generally be approved by
higher-level agency procurement officials (up to the agency's senior
procurement executive) depending on the estimated dollar value of the
procurement. The content of this justification is prescribed by the FAR.
When not providing for full and open competition, the contracting officer
is required under the FAR to solicit offers from as many potential sources
as is practicable under the circumstances. The FAR prohibits contracting
if the justification for less than full and open competition results from
a lack of advanced planning. For a more detailed comparison of AMS and the
FAR, see appendix III.
Although some of the FAA personnel we interviewed see AMS as more
efficient and flexible than the FAR, other current and former FAA
procurement officials we interviewed who have experience using both the
FAR and AMS did not agree that AMS is more flexible than the FAR.
According to these officials, the FAR may appear inflexible and cumbersome
to persons who lack experience with it, but those who are familiar with it
are able to navigate its complexities effectively. The FAR requires full
and open competition, but as experienced procurement personnel know, the
system does not break down when emergencies necessitate quick and decisive
action. For example, we recently reported that agencies generally complied
with applicable FAR requirements in awarding new contracts for work in
Iraq using other than full and open competition.19 In some circumstances,
the government's legitimate need for prompt action was sufficient to
justify selecting a contractor on an expedited basis from among the firms
that appeared able to meet the government's emergency need. In other
cases, the agencies reasonably determined that only one source could meet
their requirements.
AMS Provides Some Discipline but Does Not Ensure a Knowledge-Based
Approach to Acquisition
AMS provides some discipline through its various phases, activities, and
decision points for acquiring major ATC systems; however, it does not
ensure the use of a knowledge-based approach found in the best practices
for managing commercial product developments and DOD acquisitions that we
have identified in numerous past reports.20 Commercial best practices
call for specific knowledge to be captured and used by corporate-level
decision-makers to determine whether a product has reached a level of
development (product maturity) sufficient to demonstrate its readiness to
move forward in the acquisition process. The capture of such knowledge and
its use by executives helps to avoid cost overruns, schedule slips, and
performance shortfalls that can occur if decision-makers commit to a
system design before acquiring critical technology, design, or
manufacturing knowledge. The absence of these key best practices under AMS
puts FAA's major ATC acquisitions at risk of cost overruns, schedule
slips, and performance shortfalls.
Best Practices for Managing Acquisitions Call for a Knowledge-Based
Approach, Including Criteria for Knowledge Needed and Oversight at the
Corporate Executive Level
Commercial best practices call for managing acquisitions using a
knowledge-based approach, including (1) using established criteria to
attain specific knowledge at three critical junctures in the acquisition
cycle, which we call knowledge points, and (2) requiring oversight at the
corporate executive level for each of these knowledge points. For example,
at each knowledge point, successful product developers apply specific
indicators, or criteria, to determine whether they have attained the
knowledge they need to move to the next phase or activity in the
acquisition process. Such developers also conduct corporate
executive-level reviews to ensure that they obtain the insights and
perspectives of stakeholders throughout their organization. If the
knowledge attained does not meet the criteria for advancement or if the
executive reviewers determine that further development is inconsistent
with their priorities, the acquisition does not move forward. Table 3
summarizes the knowledge points, criteria, oversight reviews, and timing
of oversight reviews included in our model of best practices for major
acquisitions.
Table 3: Knowledge-Based Approach Called for in Our Best Practices Model
Knowledge Criteria Oversight review Timing of
point oversight
review
1. Resources o Match customers' needs with Executive-level Knowledge
and needs available review required point 1 should
matched resources-technology, design, to initiate the precede the
time, and funding. program. commitment to
begin product
o Demonstrate that development.
technologies needed to meet
essential product
requirements can work in
intended environment.
o Complete a preliminary
product design using systems
engineering to balance
customers' desires and
available resources.
2. Product o Complete 90 percent of Executive-level Knowledge
design design drawings by critical review required point 2 should
stable design review. to move to precede the
demonstration. commitment to
o Obtain stakeholders' build
concurrence that drawings are prototypes to
complete and producible. demonstrate
the design.
o Review subsystem and system
designs.
o Demonstrate with prototype
that design meets users'
requirements.
o Identify critical
manufacturing processes.
3. o Demonstrate manufacturing Executive-level Knowledge
Production processes. review required point 3 should
processes to move to precede the
mature o Build and test production production. commitment to
prototypes. begin
production.
o Test
production-representative
prototypes to achieve
reliability goals.
o Test
production-representative
prototypes to demonstrate
product performance in
operational environment.
o Collect statistical process
control data.
Source: GAO.
Experience with commercial best practices has shown that to the extent
that the level of knowledge called for at each knowledge point is not
attained, organizations take on risks in the form of unknowns that will
persist into the later stages of development, where they will take more
time and money to resolve if they become problems. Such problems lead to
cost increases and schedule delays.
AMS Has Some Good Features but Does Not Ensure That High Levels of
Knowledge Are Attained Before Major Commitments Are Made
AMS has some good features, including phases and key decision points
indicative of an acquisition process that has some elements of discipline;
however, AMS does not ensure that high levels of knowledge are attained
and that corporate executive-level reviews occur before major commitments
of agency resources are made. For example, like the best practices model,
AMS identifies critical junctures, which it terms "decision points." Three
of these decision points occur during the initial acquisition phase
(mission need, initial investment, and the final investment decision). A
fourth decision point occurs before production, and a fifth decision point
occurs before the start of the final acquisition phase (in-service
management). AMS also calls for detailed technical and programmatic
information that decision-makers can use at the first three decision
points to assess whether or not FAA should initiate an acquisition
program. This information includes a final requirements document, a final
acquisition program baseline, a final investment analysis report, an
acquisition strategy paper, and an integrated program plan. Finally, AMS,
like our best practices model, calls for senior executives to review the
information and determine whether the acquisition is ready to move
forward. The FAA executives who make the decisions at these points include
associate and assistant administrators, acquisition executives, the chief
financial officer, the chief information officer, and legal counsel; they
form the Joint Resources Council (JRC), FAA's senior decision-making body
for major ATC acquisitions. Table 4 summarizes this information.
Table 4: AMS's Decision Points, Information Sources, and Oversight Reviews
Decision point by Information sources and oversight reviews
phase/activity
Phase: Needs and solution
identification
o Activity: Mission analysis
Decision point: Mission need Information sources: Input from users in the
decision field and mission need statement.
Oversight review: JRC review called for to
move from mission analysis to investment
analysis.
o Activity: Investment
analysis
Decision Point: Initial Information sources: Initial investment
investment decision analysis report, initial life-cycle program
baseline for the most viable alternative,
updated initial requirements document and
action plan for final investment analysis.
Oversight review: JRC review called for to
select a preferred solution.
Decision point: Final Information sources: Final requirements
investment decision document, final acquisition program baseline,
final investment analysis report, acquisition
strategy paper, integrated program plan.
Oversight review: JRC review called for to
move from investment analysis to solution
implementation.
Phase: Solution
implementation
o Activity: System
integration
o Activity: System
demonstration
Decision point: Production Information sources: Determined by JRC.
decision
Oversight review: JRC may retain or delegate
decision making authority.
o Activity: System
production
Decision point by Information sources and oversight reviews
phase/activity
Decision point: In-service Information Sources: Determined by JRC.
decision
Oversight Review: JRC review called for to
move from solution implementation to
in-service management; however, the JRC may
retain or delegate decision making authority.
Phase: In-service management
Source: GAO analysis of FAA data.
Note: In this report, we place FAA's "mission analysis" and "investment
analyses" activities in the "Needs and Solution Identification" phase to
facilitate comparison with the "concept and technology development" phase
in our best practices model. Similarly, we place "system integration" and
"system demonstration" in the solution implementation phase for
comparative purposes.
AMS departs from the best practices model in two key ways-it does not call
for high levels of knowledge to be attained at three critical junctures
(knowledge points), and does not call for corporate executive-level
oversight at one of five junctures. Specifically, AMS does not establish
explicit, written criteria for (1) the information needed to determine
technology maturity at solution implementation, (2) releasable drawings at
critical design review and production process controls at production. Our
best practices model calls for attaining specific knowledge and setting
out criteria for what information should be available to help
organizations minimize risks in the form of unknowns. Risks associated
with such unknowns can persist into the later stages of development, where
they can take more time and money to resolve if they become problems,
potentially leading to cost increases and schedule delays.
In addition, AMS does not provide for corporate executive-level oversight
reviews at two of the three key junctures where our best practices model
calls for such reviews. Although AMS calls for three Joint Resources
Council reviews during the initial acquisition phase-while our model calls
for a single corporate executive-level review-AMS allows the council to
delegate its oversight responsibility later in the acquisition process to
the program managers within the service organization responsible for an
acquisition. By contrast, our model calls for two corporate
executive-level reviews later in the acquisition process.
According to FAA, its approach gives program managers flexibility,
expedites decision-making, and allows the executives with the most
knowledge about a major acquisition to make key decisions about its
continued development. FAA's reliance on this approach assumes that the
program managers will inform higher-level managers if they are unable to
meet the performance schedules and systems requirements approved by the
Joint Resources Council. However, although program managers may have the
most knowledge about their particular acquisition, they may not have the
agencywide perspective of the Joint Resources Council members. Having an
agencywide perspective, including a broad understanding of an
acquisition's potential impact on other agency projects and operations, is
especially critical when an acquisition includes the production of
multiple units and requires a substantial commitment of agency resources,
as do FAA's primarily multimillion-dollar acquisitions, such as controller
workstations and radars.
Because decisions about moving a major acquisition forward require both a
program manager's specific knowledge of the acquisition itself and a
senior executive's understanding of the acquisition's potential impact on
other agency projects and operations, our best practices model calls for
both measurable criteria at key points in the acquisition process to
ensure that specific knowledge has been captured and corporate
executive-level reviews to ensure that senior decision-makers have the
opportunity to independently consider this knowledge. Without higher-level
reviews such as our best practices model recommends and the Joint
Resources Council could provide later as well as early in the acquisition
process, FAA cannot ensure that it has fully considered the impact of
advancing an acquisition on other agency projects and operations. This
opportunity for full consideration is a central advantage of managing
acquisitions as a portfolio, as we concluded in our August 2004 report on
FAA's information technology investment management process.21
Figure 2 contrasts FAA's process for reviewing an acquisition's progress
under AMS with the process that we found leads to successful commercial
acquisitions.
Figure 2: Review Process under Our Best Practices Model and under AMS
aTo facilitate the comparison of AMS with out best practices model in this
report, we have done the following: (1) placed FAA's "Mission Analysis"
and "Investment Analyses" activities in the "Needs and Solution
Identification" phase to make it comparable with the "concept and
technology development" phase in our best practices model; (2) depicted
only the final investment decision point, recognizing that the investment
analysis phase includes an initial investment decision; and (3) placed
"system integration" and "system demonstration" in the solution
implementation phase.
bAMS does not explicitly call for a design review decision point, which
would fall between system integration and system demonstration.
cThe in-service decision is a key program milestone. It authorizes the
deployment of a system into the National Airspace System. At times, the
JRC delegates its decision authority for the production and in-service
decisions to service organizations.
To its credit, FAA continues to improve its AMS process. For example, the
agency is currently modifying its mission needs activity to make the
selection of major ATC acquisitions more consistent with the overall goals
of modernizing the National Airspace System. In addition, the Air Traffic
Organization has established an executive council to review major
acquisitions before they are sent to the Joint Resources Council. This
review is designed to screen acquisitions to determine which ones are
important enough to warrant higher-level review by the Council. Finally,
FAA is currently revising AMS to bring it in line with the Office of
Management and Budget's guidance. Specifically, the agency is
incorporating OMB Exhibit 300, which provides the investment
justifications and management plans required for major ATC acquisitions.
As Implemented, AMS Has Not Resolved Long-standing Acquisition Problems,
but FAA Is Beginning to Focus More on Results
According to our review of seven major ATC systems and analysis of FAA's
performance in acquiring major systems, AMS has not resolved the
long-standing problems that FAA experienced before implementing AMS, but
the agency is beginning to focus more on the expected results of its major
acquisitions. (See table 5.) Specifically, our review found that AMS
guidance did not call for requirements that were specific enough to
minimize requirements growth or unplanned work for five of these systems.
This lack of specificity resulted in the inadequate development or
definition of requirements, growth in requirements, unplanned work, or a
reduction in performance for five of these systems. In addition, for three
of these systems, FAA underestimated the difficulty of modifying available
software to fulfill its mission needs. Because AMS guidance was not
sufficient to account for the risks associated with modifying available
software, FAA encountered unexpected software development needs, higher
costs, and schedule delays. The two systems we reviewed that were
initiated after AMS was implemented are currently meeting cost and
schedule milestones; however, both systems are showing symptoms of FAA's
past problems with developing requirements and managing software, and it
is too soon to tell if these programs will remain within their cost,
schedule, and performance parameters. In addition, our work on FAA's major
acquisitions, along with that of the DOTIG and others has shown that the
problems FAA experienced before 1996 in acquiring major systems persist
under AMS and that effective acquisition management, rather than the use
of a specific contracting process (e.g., the FAR or AMS) is key to
successful acquisitions. To its credit, FAA is beginning to focus more on
results, largely through its new Air Traffic Organization, which has been
charged with taking a more performance-based approach to managing the
agency's acquisitions.
Table 5: Description and Status of Seven Selected ATC Acquisitions
Dollars in
millions
Project and Original Current Original Current Acquisition issues
description cost cost schedule schedule and status
STARS--new $940.0 $1,460.0 1998 2003 STARS is a joint
controller and FAA and DoD
maintenance program. STARS
workstations to delays and cost
replace the increases resulted
legacy system at from poor
terminal air requirements
traffic control definition and
facilitiesa schedule estimates.
STARS is fully
operational at 25
FAA terminal radar
facilities and 17
DoD facilities.
Only 50 of the
planned 172 systems
are being deployed.
STARS had
difficulties in
achieving many
human factor
requirements for
improving system
efficiency and
safety.
ASR-11--digital $743 $891.7 1997 2013 ASR-11 was approved
radar for for its in-service
terminal decision in
environments September 2003 and
is being deployed
at 108 sites. These
systems are being
deployed at a
slower pace than
originally planned
because of budget
cuts and deferrals.
ITWS--computer $276.1 $288.3 September 2002 Currently, six ITWS
processors and 2001 systems are
displays to operational. In May
automate weather 2004, the ATO
data near the Executive Council
airport rebaselined the
program to include
a new
weather-forecasting
capability into the
production
baseline. FAA
proposes to defer
12 of the 34
systems it planned
to procure.
LAAS--a $530.1 $696.1 2002 Deferred LAAS has been
precision at least adversely affected
approach and until by poor
landing system 2009 requirements
that augments development, a lack
the Global of understanding of
Positioning its technical
System complexity,
incomplete software
development, and an
unrealistic
development
schedule.
Unresolved radio
interference
precludes the safe
operation of LAAS.
As a result, FAA
has delayed
national deployment
to continue further
research on this
issue.
NEXCOM--digital $318.4 $318.4 October 2004 NEXCOM program
radios to 2002 delays were due to
improve air misunderstanding of
traffic a program
communications requirement and
testing procedures.
NEXCOM was recently
approved for its
in-service decision
in July 2004.
ATOP--new $548.2 $548.2 June 2004 2004 ATOP achieved its
workstations and acquisition program
processing baseline
capability to objectives;
control ocean however, this
air traffic baseline does not
reflect program
delays and cost
increases resulting
from poor
requirements
development,
unrealistic
schedule estimates,
and inadequate
evaluation of
software
complexity.
Project and Original Current Original Current Acquisition issues
description cost cost schedule schedule and status
ERAM--upgrades $3,649.0 $3,649.0 December December To date, ERAM has
the existing en 2009 2009 not breached any
route system cost and schedule
with improved parameters.
hardware and However, it remains
software a high-risk program
because of the
large amount of
software that must
be developed. The
ERAM contractor is
experiencing
software
engineering
difficulties as a
result of
lower-than-expected
productivity and
software code
growth.
Source: GAO analysis of FAA data.
aTerminal air traffic control facilities, known as Terminal Radar Approach
Control (TRACON) facilities, direct aircraft in the airspace that extends
from the point where the tower's control ends to about 50 nautical miles
from the airport. A TRACON can be located at or outside an airport.
Our Reviews of Seven Major Systems Show That Problems with Requirements
and Software Management Persist under AMS
Our reviews of seven of FAA's costliest ATC system acquisitions found that
the problems FAA experienced with requirements and software management and
their related impact on cost, schedule, and performance goals persist
today under AMS.22 Figure 3 identifies these seven acquisitions and their
milestones, which are expressed in terms of AMS decisions even when the
acquisitions were initiated before AMS was implemented. (See app. V for a
description and the status of each of these projects.) Specifically, for 6
of these 7 major ATC acquisitions, FAA did not consistently (1) clearly
define system requirements at the investment decision point or (2)
adequately assess software complexity. Moreover, as FAA has acknowledged,
it has never managed its major acquisitions by focusing on how each would
improve the efficiency of ATC operations while maintaining or improving
safety. Although FAA has made progress in improving its acquisition of
major ATC systems-by, for example, improving the maturity of its processes
for acquiring software, using a "build a little, test a little" approach
to acquisitions as it did for Free Flight Phase 1,23 and restructuring
its organization to minimize stovepipes-long-standing problems persist in
these areas. In addition, the two systems we reviewed that were initiated
after AMS's implementation are currently operating within cost and
schedule goals; however, they are showing symptoms of past problems with
developing requirements and managing software complexity. Moreover, our
work for more than two decades-before and after AMS's implementation-has
cited these types of weaknesses as central reasons for the agency's long
history of cost, schedule, and performance shortfalls. This work has also
found that the effectiveness of an agency's acquisition management has had
a greater impact on the success of its major acquisitions than the
contracting process used (e.g., the FAR or AMS).
Figure 3: Key Milestones for Selected ATC Acquisitions Initiated before
and after AMS
Inadequate Development or Definition of Requirements Led to Requirements
Growth or Unplanned Work for Five Acquisitions
For five of the seven acquisitions we reviewed, AMS guidance did not call
for requirements that were specific enough to minimize requirements growth
or unplanned work. For four of these five acquisitions-STARS, LAAS,
NEXCOM, and ATOP-incomplete and poorly defined requirements in the final
requirements documents, used at the investment decision point to assess an
acquisition's readiness to enter the development phase, led to
requirements growth, unplanned development work, or a reduction in system
performance.24 For the fifth acquisition-ASR-11-FAA misjudged the extent
to which the high-level requirements that were used to support the
commercial-off-the-shelf/nondevelopmental item (COTS/NDI) procurement by
the Department of Defense could result in a product capable of meeting
FAA's mission or user needs. As a result, unplanned software changes were
required.
o FAA's cost estimate for STARS has grown from its original estimate of
$0.94 billion in 1996 to $1.46 billion in 2004 and will deploy only 50 of
the 172 STARS initially planned. Much of the cost growth has been due to
FAA requirements creep. As a result, the STARS program has experienced
delays of more than five years from its original plan, in part due to
added requirements to the commercial-off-the-shelf Initial System
Configuration (ISC). However, the STARS ISC was satisfactory for use by
the Department of Defense as deployed.
o A final requirements document was approved, and the development of LAAS
was scheduled to begin in 1999. However, poorly established requirements
resulted in the addition of 113 new requirements to the initial
specification, entailing significant software and hardware changes.
Furthermore, LAAS may not achieve its promised capabilities because FAA
has been unable to develop technologies necessary to warn pilots of a
disruption in the LAAS signal. Until this technology is developed, LAAS
cannot be operated safely. As a result, FAA recently cut the fiscal year
2005 funding for LAAS, and the program will revert to a research and
development effort.
o FAA developed a final requirements document for the NEXCOM system, but
the requirements lacked the specificity needed to assess the development
risk. According to a NEXCOM contractor program official, this led to
miscommunication about the program requirement relating to signal
interference. This official stated that they misunderstood this
requirement and had not planned on the additional development work for the
NDI solution to meet such program objectives and delayed the program 21
months. Another program requirement involved the NEXCOM radios meeting or
exceeding the operational coverage area of the existing voice system. The
existing radios had power output levels of 50 watts but the NEXCOM
contractor could only achieve 34 watts of power to meet the coverage
requirement. A program official stated that the contractor and FAA had not
agreed on the testing procedures to assess the power levels. This posed an
"unacceptable consequence" and, as a result, FAA performed additional
testing or flight checks of the reduced radio performance (50 watts versus
34 watts) and determined that the performance reduction should not affect
NEXCOM's mission or its coverage requirement.
o FAA did not follow the AMS guidelines that call for completing a final
requirements document before proceeding to the development phase for ATOP.
The Joint Resources Council approved a delay in developing the final
requirements until after contract award. This decision resulted in
schedule delays and additional unplanned software development. The ATOP
program office asserted that the requirements remained very stable and
that the program is within cost and schedule objectives established by the
Council. However, FAA's internal documents revealed that the requirements
were not adequately defined. For example, the ATOP Investment Analysis
Study reported to the Joint Resources Council prior to contract award that
the lack of more detailed ATOP requirements at this stage of acquisition
added risk and was of concern to the investment analysis team. Under AMS,
this team is responsible for, among other things, conducting risk analyses
for the various acquisitions. Furthermore, an ATOP Assessment Team
conducted a study in March 2003 and determined that at the ATOP contract
award, "requirements were written at a high level and not mutually
understood by FAA and the contractor." However, FAA management allowed the
ATOP program to proceed to solution implementation without the final
requirements document and, according to the contractor, this resulted in
schedule delays and growth in the amount of software needing development.
o The high-level requirements for ASR-11, jointly generated by FAA and the
Department of Defense, to support a COTS/NDI acquisition, resulted in a
product that did not initially meet the FAA mission or user needs. The
software changes that were required to meet FAA's target detection needs,
as well as significant hardware design changes, parts obsolescence, and
production issues, added approximately two years to system qualification
and acceptance.
FAA Underestimated Software Complexity for Three Systems
For three of the seven major ATC acquisitions we reviewed-ITWS, LAAS, and
ATOP-FAA's AMS guidance was not sufficient to address the risks associated
with modifying available software25 to fulfill FAA's mission needs. In all
three cases, FAA officials underestimated the difficulty of modifying
available software. Our work has shown that underestimates are likely to
result in unexpected software development, higher costs, and schedule
delays.
o ITWS experienced delays from the beginning because of the complexity of
its software development. Although the program appeared to be progressing
according to its baseline, immediately after the critical design review in
September 1998, the contractor revealed that it had exceeded the target
cost by $4 million. In addition, the contractor claimed that the program
did not recognize that the computer processor originally planned for the
program was becoming outdated, that the manufacturer planned to
discontinue its production because the market was demanding a processor
with greater processing and storage capability, and that as a result, the
original computer processor would not be available to the program.
Consequently, ITWS experienced cost increases, schedule delays, and
performance shortfalls. According to the contractor and the original
acquisition plan, all systems were scheduled for delivery by December
2001, but that date has now stretched to after 2009.
o LAAS's technology maturity was not adequately assessed, and further
development was needed. Specifically, the potential for radio interference
through the atmosphere was not understood and could limit LAAS's
operations. FAA has now placed all LAAS activities in research and
development. FAA did not adequately assess LAAS's software development. At
the time of the contract award, the contractor and FAA estimated that 80
percent of the software that LAAS required had been developed. FAA later
determined that only 20 percent had been developed. FAA and the contractor
attribute this discrepancy to a lack of communication on the steps
necessary to satisfy the program's requirements. FAA agrees that it should
have conducted a software audit and a software capabilities assessment,
but pressures to keep LAAS on schedule resulted in an inadequate
assessment.
o The ATOP contractor underestimated by about half the extent to which
legacy nondevelopmental item software, which is the core of the ATOP
system, met the program's 1,036 requirements. As a result, a significant
amount of unanticipated new software code development and other
modifications were required.26
ATC Systems Have Required Multiple Rebaselining Decisions to Address
Delays and Cost Growth
As figure 3 illustrates, FAA initiated at least one rebaselining decision
for three of the five acquisitions that were begun before AMS was
implemented and were later transitioned to AMS. These rebaselining
decisions responded to delays and cost growth-problems that arise when
requirements are not stable, a program's design is not fixed, or software
code growth is not controlled. For example, FAA rebaselined STARS two
times-first in 1999 and again in 2002. Similarly, 2 years after the
investment decision for ITWS, FAA rebaselined the program twice, in 1997
and again in 2001. Given the frequency of these past rebaselining
decisions for major ATC systems and the number of years that elapsed
before or between the rebaselining decisions (3 to 4 years), it is too
soon to tell whether the two systems that were initiated under AMS-ATOP
and ERAM-will require similar rebaselinings and ultimately meet their
cost, schedule, and performance goals. Although both programs are
currently operating within their cost and schedule goals and have not yet
been rebaselined, FAA has had problems with managing its major
acquisitions in the past and is currently having difficulties developing
requirements and managing software complexity. Furthermore, as we reported
in May 2004, FAA's budget increased from $9 billion in 1998 to $14 billion
in 2004 but will be constrained for the foreseeable future. In such a
constrained budget environment, cost growth and schedule problems can have
serious negative consequences for ongoing modernization efforts-postponed
benefits, costly interim systems, delays in funding other systems, or
reductions in the number of units purchased.
Internal and External Reviews Have Found That FAA Has Made Some Progress
but Continues to Experience Problems in Acquiring Major ATC System under
AMS
Reviews of FAA's acquisition process, conducted by FAA, GAO, the DOTIG,
and others have shown that FAA has improved its management of major ATC
acquisitions in recent years but continues to experience cost overruns,
schedule slips, and performance shortfalls under AMS. Table 7 summarizes
the results of 22 internal and external reviews of FAA's major ATC
acquisitions. According to these reviews, issued from 1997 through 2004,
the same problems have persisted over many years, despite various
initiatives to address them, and FAA needs to strengthen its management
controls. For example, a key FAA review of eight major ATC acquisitions,
published in 1999, 3 years after AMS was implemented, found that these
acquisitions, though on track to meet their performance goals, were not
meeting their cost and schedule baselines. FAA attributed these cost and
schedule issues to new or poorly understood requirements, underestimates
of the acquisitions' technical complexity, and funding shortfalls.
In addition, our reviews of major FAA acquisitions-initiated before and
after AMS was implemented-have found for more than two decades that FAA's
failure to meet schedule, cost, and performance baselines for major ATC
acquisitions has been due to shortfalls in planning, weak management
controls, and a lack of systematic processes for acquiring new systems,
including inadequate requirements management, cost-accounting data, and
estimates of technical difficulty. As we reported in August 2004, judged
against the criteria of GAO's framework for information technology (IT)
investment management, which measures the maturity of an organization's
investment management processes, FAA has established about 80 percent of
the basic selection and control practices that it needs to manage its
mission-critical investments for the National Airspace System.27 For
example, FAA's business units actively monitor projects throughout their
life cycles.28 However, the agency's senior IT investment board does not
regularly review investments that are in the "in-service management," or
operational phase, and this creates a weakness in FAA's ability to oversee
more than $1 billion of its IT investments. In addition, the agency has
not yet established the practices that would enable it to effectively
manage its annual IT budget of about $2.5 billion, and agency executives
lack assurance that they are selecting and managing the mix of investments
that best meets the agency's needs and priorities. DOT has responded to
our recommendations to FAA to strengthen its IT investment management
capability.
Moreover, other reviews, such as those by Booz-Allen & Hamilton and MITRE,
have identified other shortfalls, which reflect a lack of proper
management controls and planning. For example, in 1997, Booz-Allen &
Hamilton found, among other things, that FAA had not clearly defined
organizational roles and responsibilities within the various phases of AMS
and that greater guidance and training under AMS were warranted. In 1999,
Booz-Allen & Hamilton reported that FAA had not demonstrated improvement
in adhering to planned costs and schedules under AMS and that the agency
needed to better manage its development of requirements and address
persistent funding shortfalls. Moreover, in 2001, a MITRE report on
selected major acquisitions found inadequate management controls and
deficiencies in both contractors' performance and in FAA's measurement of
acquisition performance. See table 7 for a chronological listing of the
reviews.
Table 6: Internal and External Reviews of FAA's Use of AMS for Acquiring
Major ATC Systems
Review Selected findings Contributing factors
GAO, Air Traffic Control: FAA's cost-estimation FAA's cost-accounting
Improved Cost Information practices do not practices do not
Needed to Make satisfy recognized provide for the proper
Billion-Dollar estimating requisites, accumulation of actual
Modernization Investment increasing the project costs.
Decisions, GAO/AIMD-97-20 likelihood of poor
, (Washington, D.C.: Jan. acquisition selection
22, 1997). decisions.
GAO, Air Traffic Control: Incompatibilities exist FAA lacks a complete
Complete and Enforced between current and systems architecture or
Architecture Needed for planned ATC overall "blueprint" to
FAA Systems Modernization, acquisitions, resulting guide and constrain the
GAO/AIMD-97-30 , in high costs and development and
(Washington, D.C.: Feb. 3, reduced performance. maintenance of ATC
1997). acquisitions.
GAO, Air Traffic Control: Planned acquisitions Weaknesses in some key
Immature Software frequently are not process areas, such as
Acquisition Processes delivered on time and planning, requirements
Increase FAA System within budget. development, and
Acquisition Risks, management, limit FAA's
GAO/AIMD-97-47 , ability to consistently
(Washington, D.C.: Mar. acquire
21, 1997). software-intensive ATC
systems on time and
within budget.
FAA, Evaluation of FAA AMS addresses 15 of the Inadequate management
Acquisition Reform-The 17 problems facing has not enabled FAA to
First Year: April 1996 - acquisitions. meet its goals of
March 1997, (Washington, reducing acquisition
D.C.: May 1997). deployment time by 50
percent and cost by 20
percent.
FAA, Evaluation of FAA Further improvements Procedural weaknesses
Acquisition Reform-The are necessary if limit FAA's ability to
First Two Years: April acquisition reform is achieve cost and
1996 - March 1998, Report going to allow FAA to schedule goals.
#1998-02, (Washington, meet its cost and
D.C.: May 29, 1998). schedule goals.
GAO, Air Traffic Control: From the inception of Weaknesses persist in
Observations on FAA's Air its modernization key areas, such as how
Traffic Control efforts, FAA has not FAA monitors the status
Modernization Program, consistently followed a of its acquisitions
GAO/T-RCED/AIMD-99-137 , disciplined management throughout their life
(Washington, D.C.: Mar. approach for new cycles.
25, 1999). acquisitions.
GAO, Air Traffic Control: AMS contained FAA lacked adequate
FAA's Modernization weaknesses in the cost data for making
Investment Management selection of selection decisions;
Approach Could Be acquisitions and in the adequate management
Strengthened, review of acquisitions' controls, and a
GAO/RCED/AIMD-99-88 , performance during the defined, documented
(Washington, D.C.: Apr. postimplementation process for conducting
30, 1999). phase. reviews during the
in-service management
phase.
FAA, Evaluation of FAA FAA's cost and schedule Requirements changed or
Acquisition Reform-The plans were not on were misunderstood;
First Three Years: April track, but performance technical difficulties
1996 - March 1999, Report plans were met. were underestimated;
#1999-04, and funding fell short.
(Washington, D.C.: May 28,
1999).
Booz-Allen & Hamilton, FAA has yet to AMS is not being
Independent Assessment of implement a seamless consistently
the Federal Aviation life-cycle approach to implemented across all
Administration's acquisitions life-cycle phases.
Acquisition Management management.
System, (McLean, VA: July
6, 1999).
GAO, National Airspace FAA experienced delays FAA lacks a
System: Persistent and cost increases in comprehensive plan with
Problems in FAA's New developing its global checkpoints for
Navigation System positioning navigation reviewing the
Highlight Need for system; as a result, it contractor's approach
Periodic Reevaluation, is unclear whether the to meeting the system's
GAO/RCED/AIMD-00-130 , benefits of the system performance
(Washington, D.C.: June will outweigh the cost. requirements.
12, 2000).
Review Selected findings Contributing factors
GAO, National Airspace FAA experienced cost FAA underestimated the
System: Problems Plaguing and schedule problems complexity of
the Wide Area Augmentation in developing this developing the
System and FAA's Actions navigational system acquisition.
to Address Them, because of unplanned
GAO/T-RCED-00-229 , software development
(Washington, D.C.: June needs and a requirement
29, 2000). to warn pilots of any
system failure that
would provide
misleading information.
GAO, National Airspace Three acquisitions that FAA needs better data
System: Free Flight Tools are components of FAA's collection and analysis
Show Promise, but planned new approach processes to ensure
Implementation Challenges for air traffic that benefits are
Remain, GAO-01-932 , management have realized.
(Washington, D.C.: Aug. uncertain potential
31, 2001). benefits and may not be
worth FAA's investment.
GAO, National Airspace The reliability of the The development cost
System: Better Cost Data life-cycle cost estimate is based on
Could Improve FAA's estimate for STARS is the contractor's
Management of the Standard uncertain because cost projections, which FAA
Terminal Automation data obtained from the has not yet
Replacement System, contractor do not independently analyzed,
GAO-03-343 , (Washington, reflect the current as called for under
D.C.: Jan. 31, 2003). status of the contract. AMS.
GAO, National Airspace FAA was unable to hire Uncertainties about the
System: Current Efforts a chief operating position's
and Proposed Changes to officer to head the responsibilities,
Improve Performance of ATO. reporting
FAA's Air Traffic Control relationships, and
System, GAO-03-542 , performance measurement
(Washington, D.C.: May 30, criteria hampered the
2003). hiring.
DOT/OIG, Status of FAA's Cost growth, schedule Cost and schedule
Major Acquisitions, delays, and performance baselines are not
AV-2003-045, (Washington, problems continue with reliable, and decisions
D.C.: June 26, 2003). FAA's major are being made with
acquisitions. unclear data.
GAO, Air Traffic Control: Systemic management FAA lacked the
FAA's Modernization issues, including information technology
Efforts-Past, Present, and inadequate management and financial
Future, GAO-04-227T , controls and human management systems that
(Washington, D.C.: Oct. capital issues, have would have helped it
30, 2003). contributed to major reliably determine the
ATC acquisitions' acquisitions' technical
persistent cost requirements and
overruns, schedule estimate and control
delays, and performance their costs and
shortfalls. schedules; and the
agency's organizational
culture discouraged
collaboration among
technical experts and
users.
GAO, Information Although weaknesses Remaining weaknesses
Technology: FAA Has Many remain, FAA has include inadequate
Investment Management established about 80 management controls and
Capabilities in Place, but percent of the basic the lack of a defined,
More Oversight of practices needed to documented process for
Operational Systems Is manage its conducting reviews
Needed, GAO-04-822 , mission-critical during the in-service
(Washington, D.C.: Aug. acquisitions so that it management phase.
20, 2004). can be assured that it
is selecting and
managing the mix of
investments that best
meets its needs and
priorities.
GAO, Air Traffic Control: FAA made progress in Process improvement
System Management improving its system efforts have not been
Capabilities Improved, but management institutionalized.
More Can Be Done to capabilities, but can
Institutionalize do more to
Improvements, GAO-04-901 , institutionalize
(Washington, D.C.: Aug. process improvement
20, 2004). initiatives.
Source: GAO analysis.
FAA's ATO Is Taking Steps to Improve Major ATC Acquisitions
FAA's recent reorganization, which brought ATC acquisitions and operations
together in the ATO,29 is expected to help the agency address many of the
concerns we have identified for more than two decades, including those
identified in this report. For example, the ATO is continuing to develop
and refine specific guidance for critical areas, such as requirements
management, software development, and cost estimation. In addition, as the
overseer of both ATC acquisitions and operations, the ATO is in a position
to facilitate more effective management of major ATC acquisitions than has
occurred in the past. The ATO is attempting, for example, to link
acquisition decisions directly with expected improvements in operational
efficiency without compromising safety. This is important, given that FAA
has spent about $2.5 billion on ATC modernization per year since 1996
while operating costs have continued to rise-from $4.6 billion to $7.5
billion over the past decade. FAA had not completed its reorganization or
implemented all of its initiatives at the time of our audit.
Improvements to Requirements Development
With the establishment of the ATO, FAA consolidated requirements
development from two organizations (the organization sponsoring an
acquisition and the former agencywide acquisition organization) into a
single new organization-the Air Traffic System Requirements Service.30 In
addition, the ATO developed guidance to better manage requirements during
the middle phase of AMS (solution implementation). According to FAA
officials, some more complex development efforts may need to develop
systems requirements and a more detailed requirements document than AMS
currently calls for in the final requirements document. More important, in
January 2003, FAA issued guidance on requirements management, Roles in
Requirements Management During Solution Implementation Phase, which
provides for integrated requirements teams that maintain responsibility
for requirements management throughout an acquisition's life cycle.
According to this guidance, when the final requirements document is
accepted by the Joint Resources Council at the investment decision point,
a requirements baseline is established and any proposed changes to the
requirements must be assessed for their impact on the program and shown to
be operationally suitable, affordable, executable, and justifiable. An FAA
official on an integrated requirements team stated that any changes that
may affect an acquisition's cost and the schedule require approval by the
Executive Committee. The FAA official also stated that this guidance has
already helped to stabilize NEXCOM's requirements during the solution
implementation phase. Other FAA officials representing the Joint Resources
Council acknowledged that the guidance should ensure greater control over
program requirements growth, but said that not all program offices have
consistently applied it.
Improvements to Managing Software and System Acquisition and Development
To better manage software programs for ATC modernization acquisitions, FAA
established a centralized process improvement office that reports to the
Chief Information Officer (CIO).31 This office developed an FAA integrated
capability maturity model (i-CMM), a software development and management
model that is similar to a model developed by Carnegie Mellon University
called the Capability Maturity Model Integration (CMMI(R)), which is used
to appraise the maturity of an organization's processes for acquiring
software. However, FAA's i-CMM goes beyond Carnegie Mellon's model to
reflect international standards. The CMMI(R) appraisal methodology calls
for assessing process areas-such as project planning, requirements
management, and quality assurance-by determining whether key practices are
implemented and overarching goals are satisfied. Both the i-CMM model and
CMMI(R) appraisal methodologies provide a logical framework for measuring
and improving key processes needed for achieving quality software and
systems.
However, as we reported in August 2004,32 FAA projects are not required to
use the capability maturity model for process improvement, and individual
projects that use the i-CMM model are allowed to choose which process
areas they seek to improve and to determine when they are ready for an
appraisal of their progress. To date, fewer than half of FAA's major ATC
projects have used this model. The recurring weaknesses we identified in
our project-specific evaluations are due in part to the flexibility these
projects were given in deciding whether and how to adopt this process
improvement initiative. Furthermore, after combining its ATC organizations
into a single performance-based organization (the ATO), FAA is
reconsidering prior policies, and it is not yet clear whether process
improvement will remain a priority. Without a strong senior-level
commitment to process improvement and a consistent, institutionalized
approach to implementing and evaluating it, FAA cannot ensure that key
projects will continue to improve systems acquisition and development
capabilities. As a result, FAA will continue to risk the project
management problems-including cost overruns, schedule delays, and
performance shortfalls-that have plagued past acquisitions. To address
these shortcomings, we recommended that the Secretary of Transportation
address specific weaknesses and institutionalize FAA's process improvement
initiatives by establishing a policy and plans for implementing and
overseeing process improvement initiatives.
Improvements to Estimating Costs
FAA has taken steps to improve its cost estimation for major ATC projects
by issuing guidance on how to develop and use pricing under AMS. For
example, AMS policy calls for audit trails to record and explain the
values that are used as inputs to cost models. In addition, it calls for
agency officials, when reporting to executive oversight agencies and
Congress, to disclose the level of uncertainty and imprecision that are
inherent in cost estimates for major ATC systems. According to AMS policy,
estimators record the procedures, ground rules and assumptions, data,
environment, and events that underlie their development or update of a
cost estimate. This information supports the credibility of the cost
estimate, aids in the analysis of changes in program costs, enables
reviewers to assess the cost estimate effectively, and contributes to the
population of FAA databases that can be used for estimating the cost of
future programs. Finally, despite a delay of many years, FAA officials
told us that they are in the final stages of completing the agency's
cost-accounting system and plan to have it in place across the agency by
the end of this calendar year, which will bring FAA into compliance with
the Federal Managers' Financial Integrity Act of 1982. This measure will
help reduce the likelihood of cost overruns or improper payments for
unallowable costs and provide decision-makers with critical information.
As we have reported in the past,33 a cost-accounting system is critical to
managing major ATC acquisitions, because without it, FAA lacks the
information it needs to reliably estimate operating costs over an
acquisition's life cycle.
Other Improvement Efforts
In May 2004, the FAA Administrator testified to Congress that, to date, in
attempting to improve the efficiency of ATC operations while maintaining
safety, FAA had not managed its major ATC acquisitions to be aware of
their cost implications for its operations. The Administrator said,
however, that the agency was taking its first steps to fundamentally
change how it makes acquisition decisions by adopting a more
results-oriented approach. Under this approach, the agency plans to link
its decisions to fund major acquisitions directly with their expected
contribution to improving operational efficiency and controlling
escalating operating costs. Whereas, in the past, FAA measured results in
terms of its progress in completing and deploying a major ATC system, it
was now going to focus on how a given system improved operational
efficiency. Such an approach holds promise for helping FAA more
effectively manage its portfolio of major ATC acquisitions by providing a
sound basis for choosing among competing priorities. However, because FAA
has only recently begun to incorporate this type of analysis of
acquisitions' costs and operational efficiency into its decision-making
and management processes, it is still too early to assess the results.
In addition, to its credit, FAA has created a training framework for its
acquisition workforce, which we found mirrors human capital best practices
that we have identified. In January 2003, we reported on FAA's efforts to
define and train its workforce to meet the requirements of the
Clinger-Cohen Act of 1996.34 This act required FAA and other civilian
agencies to establish education, training, and experience requirements for
their acquisition workforce. Our work on public and private best practices
has identified six elements of training as critical to acquisition. These
elements include (1) prioritizing the acquisition initiatives most
important to the agency, (2) securing top-level commitment and resources,
(3) identifying those who need training on specific initiatives, (4)
tailoring training to meet the needs of the workforce, (5) tracking
training to ensure it reaches the right people, and (6) measuring the
effectiveness of training. These six elements are crucial for
successfully implementing acquisition initiatives and reforms. Agencies
that do not focus their attention on these critical elements risk having
an acquisition workforce that is ill equipped to implement new processes.
The probability of success is higher if training is well planned rather
than left to chance. In 2003, we found that FAA's model for training its
acquisition workforce largely mirrored public and private-
successfully introducing and implementing effective acquisition best
practices. FAA's acquisition workforce plays a critical role in addressing
long-standing weaknesses that we and others have identified with FAA's
acquisition of major ATC systems. Given the importance of training for
acquisition workforces, it will be important for the ATO to put mechanisms
in place to comprehensively evaluate the effectiveness of the training it
provides to improve the knowledge base of FAA's acquisition workforce.
To improve its investment management decision-making and oversight of
major ATC acquisitions, the ATO also initiated the following procedures:
o Integrate AMS and the Office of Management and Budget's Capital Planning
and Investment Control Process36 to develop a process for analyzing,
tracking, and evaluating the risks and results of all major capital
investments made by FAA.
o Conduct Executive Council reviews of project breaches of 5 percent in
cost, schedule, and performance to better manage cost growth;
o Issue monthly variance reports to upper management to keep them apprised
of cost and schedule trends.
o Monitor progress in meeting the goals identified in FAA's Flight Plan,
the agency's blueprint for action through 2008. The Executive Council
tracks this progress monthly and reports to the Administrator, using a
color-coded system to keep her apprised of how well FAA is meeting its
goals. Green denotes that a goal will be met, yellow denotes that some of
the activities leading to a main goal may be in jeopardy but the overall
goal can be achieved, and red denotes serious concerns about reaching a
goal without major intervention. A formal progress report is issued
quarterly and made publicly available on the agency's Web site; and
o Increase the use of cost monitoring or earned value management systems37
to improve oversight of programs.
Despite FAA's current and planned efforts to improve its acquisition of
major ATC systems under the ATO, given the newness of these efforts and
the agency's poor track record in this area for more than two decades, it
is critical for FAA to (1) modify AMS to more fully reflect the best
practices followed by high-performing acquisition organizations, (2)
follow through on planned improvement initiatives, and (3) adopt a
continuous improvement approach to acquiring new ATC systems.
Conclusions
In the early 1990s, FAA contended that it needed relief from the FAR to
remedy long-standing problems with cost, schedule, and performance
shortfalls in its major ATC acquisitions; however, our work for more than
two decades in this area has found that acquiring major ATC systems
successfully depends more on managing an acquisition process well than on
using a specific contracting process (e.g., the FAR or AMS). While our
recent work has shown some improvement in FAA's management of major ATC
system acquisitions, some key problems that existed before 1996 persist
under AMS-including difficulty with clearly defining system requirements
at the investment milestone and adequately assessing complex software
requirements. These problems continue to make these acquisitions
vulnerable to cost, schedule, and performance shortfalls. Without further
measures to improve the development and management of requirements and to
better estimate the complexity of the software development needed for
major ATC systems, such shortfalls are likely to persist.
Although AMS provides some discipline for acquiring major ATC systems
through its various phases, activities, and decision points, it does not
require that (1) specific knowledge be attained using explicit written
criteria and (2) corporate executive-level oversight be provided to
determine-independently from the program offices-whether a system has
reached a level of development (product maturity) sufficient to move
forward in the acquisition process. Commercial best practices call for
such knowledge-based decision-making at the corporate executive-level to
help ensure that acquisitions are not moved into the development phase
prematurely, to obtain greater predictability in ATC system program costs
and schedules, to improve the quality of the ATC systems that are
deployed, and to deliver new capability to the National Airspace System
faster. A knowledge-based approach is also important because it provides
assurance that agency decision-makers have critical information about an
acquisition's ability to meet a mission need and FAA's readiness to move
forward in the acquisition process before making large commitments of
agency resources. Absent such an approach, FAA lacks assurance that it has
obtained the critical technological, design, or manufacturing knowledge
that best practices call for to avoid cost overruns, schedule slips, and
performance shortfalls. As a result, FAA is not doing all that it can to
systematically address persistent shortcomings in its management of major
ATC acquisitions. Moreover, although FAA has established a framework for
training its acquisition workforce under the ATO, it has not yet developed
comprehensive performance criteria to evaluate how effectively it has
implemented this framework. As a result, the agency lacks assurance that
its use of this framework is having the intended effect of improving the
knowledge base of this workforce.
Recommendations for Executive Action
We are making five recommendations to the Secretary of Transportation. To
reduce the risk of persistent cost and schedule shortfalls in major ATC
system acquisition programs, to improve the quality of the ATC systems
that are deployed, and to deliver new capability to the National Airspace
System faster, we recommend that the Secretary of Transportation advise
the FAA Administrator to take the four following actions:
o Modify AMS to specify that requirements be more clearly defined for
major ATC systems, including providing more detailed guidance on setting
clear, objective, and measurable requirements that reflect customers'
needs, before making large investments of agency resources.
o Establish a strategy for identifying and measuring all additional
development needed for complex software (e.g., commercial-off-the-shelf or
nondevelopmental items) used for major ATC systems.
o Develop explicit written criteria for the key decision points called for
under best practices, including the capture of specific design and
manufacturing knowledge.
o Require corporate executive-level decisions at these key decision points
(before an acquisition moves from integration to demonstration and, again,
before it moves to production).
In addition, to assure FAA that the training framework it has adopted for
the ATO's acquisition workforce is improving the knowledge base of this
workforce as intended, we recommend that the Secretary advise the
Administrator to develop performance criteria to comprehensively evaluate
the framework's effectiveness.
Agency Comments
We provided copies of a draft of this report to DOT for review and comment
and met with Department and FAA officials, including the ATO's Vice
President for Acquisition and Business Services, to obtain their comments.
FAA officials told us that they have made great strides in improving their
acquisition of major ATC systems under AMS; however, they recognize that
there is room for improvement and are firmly committed to implementing
best practices for acquisitions. These officials generally agreed with the
report's findings and conclusions and said that our recommendations would
be useful to them as they continue to refine their acquisition management
system, including training their acquisition workforce. The agency
provided us with oral comments, primarily technical clarifications, which
we have incorporated as appropriate.
As agreed with your office, unless you publicly announce the contents of
this report earlier, we plan no further distribution until 30 days from
the report date. At that time, we will send copies of this report to
interested congressional committees, the Secretary of Transportation, and
the Administrator, FAA. We will also make copies available to others upon
request. In addition, the report will be available at no charge on the GAO
Web site at http://www.gao.gov.
concerning this report. Key contributors to this report are listed in
appendix VI.
JayEtta Z. Hecker Director, Physical Infrastructure Team
FAA Has Begun Analyzing Spending Trends to Take a More Strategic Approach
to ProcurementAppendix I
Our review of the Federal Aviation Administration's (FAA) general
procurement of goods and services focused on the Air Traffic Organization
(ATO) and its predecessor offices. According to FAA officials, the ATO has
recently begun to consider ways to better leverage its buying power by
taking a more strategic approach to procurement. While FAA uses the
Acquisition Management System (AMS) for all FAA acquisitions, including
the procurement of such goods and services as office supplies, computers,
telephone services, and engineering and technical support services, these
procurement activities take place in a decentralized environment of
independent, transaction-oriented buying processes. Each FAA unit
determines its need for goods and services and procures them as necessary,
leaving headquarters with limited oversight of the agency's total
procurement spending. For example, in 2003, FAA units carried out over
346,000 procurement actions for goods and services and purchase
cardholders1 made an additional 335,000 transactions. This fragmented
environment does not permit the agency to leverage its buying power
through lower-cost, consolidated contracts, at the local, regional, or
national level and to rationalize the number of suppliers best suited to
meet the agency's needs. At the same time, as part of a strategic
procurement effort, FAA can use spend analysis to monitor trends in small
and disadvantaged business participation so that it can balance the goals
of lower-cost contract consolidation and promoting small business
contracting opportunities.
Spend analysis, a tool used in a strategic approach to procurement,
provides knowledge about how much is being spent for what goods and
services, who the buyers are, who the suppliers are, and where the
opportunities are to leverage buying power. Our past work2 shows that
private companies are using spend analysis as a foundation for employing a
strategic approach to procurement. The analysis identifies where numerous
suppliers are providing similar goods and services-often at varying
prices-and where purchasing costs can be reduced and performance improved
by better leveraging buying power and reducing the number of suppliers to
meet the company's needs. Our research on commercial best practices has
found that spend analysis is an important driver of strategic planning and
execution. As part of an overall strategic procurement effort, companies
use spend analysis to (1) define the magnitude and the characteristics of
their spending, (2) understand their internal clients and supply chain,
(3) create lower-cost consolidated contracts, and (4) monitor spending
with small and disadvantaged businesses to achieve socioeconomic
procurement goals.
We previously reported3 that six agencies, including DOT, did not take
advantage of opportunities to obtain more favorable prices on purchase
card buys with frequently used vendors-vendors where an agency spends more
than $1 million annually. In these six agencies, which accounted for over
85 percent of federal government purchase card spending, frequently used
vendors accounted for purchases totaling nearly $3 billion in 2002. We
recommended several actions-including conducting spend analysis using
available data and gathering additional information where feasible-that
could ultimately help these agencies achieve $300 million annually in
potential savings.
In fiscal year 2003, FAA procured nearly $4 billion in goods and services
and spent an additional $132 million using purchase cards. According to
senior FAA officials, the agency has just begun to implement a strategic
approach to general procurements. Other federal agencies are beginning to
use strategic tools such as spend analysis to improve their spending for
goods and services, and some have initiatives under way to obtain more
favorable prices on purchase card buys. According to a senior FAA
acquisition official, FAA has to balance the need of its units to
independently make purchases that pertain solely to unit requirements with
the agency's need to aggregate purchases of goods and services that are
used by more than one unit. FAA has hired a consultant to help begin the
use of spend analysis. This effort could reduce the agencywide costs for
mobile wireless services by 40 percent-an effort expected to save the
agency $8 to $10 million annually. FAA intends to expand its use of spend
analysis to target other procurement category savings opportunities,
including information technologies, training, facilities, and professional
services, as its accounting systems improve.
FAA has taken some preliminary steps to set up a spend analysis program;
however, progress has been challenging for FAA because of deficiencies in
its accounting systems. For example, because the agency's accounting
system did not identify all of the mobile wireless services for which it
was being billed, the contractor implementing the spend analysis had to
obtain this information from the wireless providers. FAA will need to
expedite its efforts in this area to fully realize potential savings. Our
prior research has shown that setting up a spend analysis program can be
challenging. Companies have had problems accumulating sufficient data from
internal financial systems that do not capture information on all of what
a company buys or is using in different, unconnected parts of the company.
Despite these challenges, companies that have developed formal,
centralized spend analysis programs have been able to track their costs
and identify areas for strategic sourcing and savings opportunities.
In our recent report on spend analysis,4 we found that DOT, at the time of
our review, had not yet begun to collect the data needed for a strategic
approach to procurement; however, the department is engaged in ongoing
efforts to improve procurements, and its top leadership is committed to
using spend analysis to change the way goods and services are purchased.
One obstacle to using spend analysis that the department cited during our
review was a lack of comprehensive and reliable spending data. However,
since we completed our review, the department reports stepping up efforts
to use currently available data and evaluate business intelligence
software to overcome those obstacles. In commenting on our report,
Transportation's senior procurement executive told us that the department
is expanding its spend analysis efforts. For example, his office recently
reviewed purchase card spending data to identify volume discount
opportunities and is now using the results to negotiate new discount
agreements with several office product vendors. In addition, he told us
that to facilitate future agencywide purchase card spend analyses, DOT
awarded a task order in June 2004 to one bank card company that will
provide purchase-card audit software and enhanced data-mining
capabilities. He also indicated that the department's leadership supports
fiscal year 2005 funding to enhance spend analysis capabilities and that
software options for the new agencywide spend analysis system are now
being evaluated as part of an ongoing financial and procurement review.
Objectives, Scope, and MethodologyAppendix II
To compare FAA's Acquisition Management System (AMS) with the Federal
Acquisition Regulation (FAR), we reviewed AMS and changes in it over time.
We also compared FAA's acquisition authority under the FAR and under AMS.
In addition, we identified relevant recommendations from reports that we,
the Department of Transportation's Inspector General (DOTIG), and others
have issued to determine which recommendations have been implemented,
rejected, or left open, and to evaluate how those recommendations have
modified FAA's acquisition policies and practices. We also collected and
summarized published reports and analyzed available life-cycle management
data on the current status of major and nonmajor acquisitions being
carried out under AMS.
To determine the ways in which FAA's acquisition policies compare with our
best practices model, we used information from several of our products
that examine how commercial best practices can improve outcomes for
acquisition programs. This model consists of four phases: (1) concept and
technology development; (2) product development, which includes both
integration and demonstration activities; (3) production; and (4)
operations and support. In between these four phases are three key
knowledge points at which commercial firms must have sufficient knowledge
to make large investment decisions. We also reviewed and analyzed AMS,
accessible at http://fast.faa.gov . Furthermore, to clarify the content of
FAA's acquisition process, we met with various FAA vice-presidents and
officials from FAA's Acquisition Planning and Policy Division. Next, we
compared and contrasted FAA's acquisition policies with the best practices
for commercial acquisitions identified in our past reports. Our analysis
focused on whether FAA's policies contained the measurable criteria and
management controls necessary to achieve FAA's intent of minimizing cost,
schedule, and performance risks. We also interviewed current and former
FAA procurement officials that have experience using both the FAR and AMS.
To determine if FAA has effectively implemented its new acquisition
authority and improved its acquisition outcomes, we reviewed seven of
FAA's most expensive major ATC acquisitions, including the Airport
Surveillance Radar 11 (ASR-11), Standard Terminal Automation Replacement
System (STARS), Integrated Terminal Weather System (ITWS), Local Area
Augmentation System (LAAS), Next Generation Air/Ground Communications
System (NEXCOM), Advanced Technologies and Oceanic Procedures (ATOP), and
En Route Automation Modernization (ERAM). See table 7 for specific program
costs.
Table 7: Program Costs for the Seven Systems We Reviewed
Dollars in millions
Program Total program cost as of 9/30/04
STARS $1,460.0
ASR-11 891.7
ITWS 288.3
LAAS 696.1
NEXCOM 318.4
ATOP 548.2
ERAM 2,154.6
Total $6,357.3
Source: GAO analysis.
Note: These amounts are for facilities and equipment only (not operations
and maintenance).
We also selected these seven acquisitions because we considered them to
fall into two basic categories-pre-AMS and post-AMS. Five of the
acquisitions were initiated before AMS was implemented in April 1996 and
were transitioned into AMS at various times before their completion. The
two remaining acquisitions-ATOP and ERAM-were initiated and have remained
completely under AMS. We then reviewed program documents and reports and
interviewed program and agency officials responsible for developing these
acquisitions, as well as other acquisitions experts in the private sector.
For some acquisitions, we discussed programmatic issues with
representatives of the primary contractor for the specific acquisition to
obtain information on the practices and procedures used for the
acquisition. In addition, we interviewed some current and former FAA
procurement officials with experience using both the FAR and AMS to obtain
their views on the use of each contracting process and how the two
compare. Furthermore, to see how FAA has progressed in addressing problems
with its acquisitions, we reviewed our work on acquisitions over the last
20 years, as well as reports by the DOTIG, FAA, Booz-Allen & Hamilton, and
MITRE. Because the data in this report on cost, schedule and performance
are used as background information or to otherwise provide a description
of acquisitions, we did not assess their reliability.
The effect of the current budget process on FAA's ability to successfully
modernize the National Airspace System, including acquiring major ATC
systems is not within the scope of this review.
Comparison of the Scope and Flexibility of FAA's Acquisition Management
System and the Federal Acquisition Regulation ProcessAppendix III
Background
FAA's business processes, including its acquisition of major systems,
differ significantly from the business processes followed by most other
federal agencies. FAA relies on its Acquisition Management System (AMS),
which establishes FAA internal acquisition policy. AMS resulted from the
adoption of language in the Department of Transportation and Related
Agencies Appropriations Act,1 which directed the FAA Administrator to
develop and implement an acquisition management system for FAA. The
adoption of this language (section 348) followed FAA's assertions that the
requirement that it conduct procurements in accordance with the Federal
Acquisition Regulation (FAR) was at least a contributing factor in its
repeated failure to complete air traffic control (ATC) and other
modernization programs on schedule. The Administrator was directed to put
in place a system that would address the "unique needs of the agency" that
FAA contended prevented its acquisitions from being timely and
cost-effective.
Section 348 distinguished FAA from other federal agencies by removing FAA
from the federal acquisition system. Under section 348, FAA was no longer
subject to title III of the Federal Property and Administrative Services
Act of 1949,2 which among other things requires that the government
procure supplies and services competitively. It removed FAA as an agency
subject to the Office of Federal Procurement Policy Act3 and eliminated
the requirement that FAA comply with the FAR. While mandating that FAA
conduct its acquisitions so that "all reasonable opportunities to be
awarded contracts shall be provided to small business concerns and small
business concerns owned and controlled by socially and economically
disadvantaged individuals," section 348 eliminated the requirement that
FAA comply with the Small Business Act.4 Furthermore, it made the
procurement protest system of the U.S. Government Accountability Office
inapplicable to FAA, although disappointed offerors
can still file protests with FAA's Office of Dispute Resolution for
Acquisition.5
AMS Defines an Investment/Life-Cycle Project Management System
Much of AMS guidance concerns project, financial, and property life-cycle
management issues. In fact, FAA's policy describes AMS as applying to all
investment programs regardless of cost or the appropriation funding them.
It recognizes that a single investment program may span multiple
procurements and projects. It applies, according to its terms, to the
activities associated with needs analysis, determination of requirements,
analysis of investment alternatives, establishment of investment programs,
allocation and expenditure of resources, procurement and deployment of
needed products and services, in-service management of fielded capability,
and eventual disposal of obsolete products.
AMS focuses on the following key program milestones:
o Mission Analysis-encompasses those key corporate and service-level
processes that define, coordinate, and integrate the work of service
organizations,6 thereby providing strategic direction to keep FAA
responsive to the service needs of its customers. Mission analysis is used
to update a mission need statement, which in turn may identify capability
shortfalls or technological opportunities, that is, unmet needs. Unmet
needs are presented to the Joint Resources Council (JRC) for a mission
need decision. To be approved, the unmet need should be supported by the
updated mission need statement and the initial requirements document,
including a concept of use, and the initial investment plan.
o Investment Analysis-builds on the results of the mission need decision
by developing detailed plans and final requirements for each proposed
investment program and by defining an acquisition program baseline that
establishes cost, schedule, performance, benefit, and risk-management
boundaries for the program. AMS calls for planning the entire solution-an
effort that may use market survey data but is based in large measure on
FAA's assumptions and data. The service organization produces a final
implementation and life-cycle support strategy. A detailed program plan
and an acquisition program baseline are also produced. The results are
presented to the JRC for a "final investment decision."7
o Solution Implementation-encompasses acquiring, accepting, deploying,
installing and preparing for the operational use of an approved
investment. Approval of the investment carries with it authorization for
the service organization to conduct all acquisitions needed to execute the
investment decision, subject to any constraints established in the final
investment decision.
o In-Service Decision-is an FAA system qualification milestone, which is
achieved when an otherwise operational investment is satisfactorily tested
to demonstrate its operational effectiveness and suitability before it is
placed in service in the National Airspace System. The JRC designates the
decision maker.
o In-Service Management-covers activities throughout a system's life
cycle, starting at the time that an investment becomes operational.
In-service product improvements may eliminate latent defects, fix systemic
problems, and enhance the utility of the investment. These changes may be
made within the approved acquisition program baseline without
corporate-level approval. In-service management also includes planning,
programming, and developing supporting budget input; monitoring and
assessing performance, cost of ownership, and support trends; and planning
for service-life investment decisions.
o Service Life Extension-seeks a new investment decision by the JRC when a
current capability is unable to satisfy demand or when another solution
may be more effective. The JRC can decide to revalidate the mission need
satisfied by the solution by upgrading or refurbishing fielded capability
or by replacing that capability with another equivalent or new superior
solution. The JRC may also decide that the capability should be retired.
Only a Portion of AMS Deals Directly with the Procurement Process
Although the FAR includes requirements addressing procurement planning and
major system acquisition, AMS as just outlined differs significantly from
the FAR in its focus and scope. The FAR addresses planning8 and major
system acquisition9 in the context of government procurement policy and
procedure. Agencies other than FAA find the broader program planning and
management issues addressed in AMS outside of the FAR, in documents such
as the Office of Management and Budget's (OMB) Circular A-109, in their
own planning guidance, such as the Department of Defense's (DOD) 5000
series,10 and in established knowledge-based best practices. As indicated
earlier, much of AMS focuses on just such issues. Only AMS section 3
addresses procurement policy and procedure.11
AMS States a Nonregulatory FAA Policy
A further significant foundational difference between AMS and the FAR is
that AMS sets out a nonregulatory FAA policy, whereas the FAR was adopted
and is maintained as a set of published governmentwide regulatory
requirements, which form a legal basis for federal agencies' contract
decision-making. AMS is binding on FAA personnel as FAA employees and
establishes other guidelines that FAA states should be followed unless
there is a rational basis for doing otherwise. AMS is subject to such
internal controls as the Administrator chooses to enforce and general
overarching legal requirements, such as the Government Performance and
Results Act of 1993 (GPRA).12 There is a legal requirement, created by
section 348, that small and socially or economically disadvantaged firms
be given all reasonable opportunities to receive contract awards. FAA in
its Office of Dispute Resolution for Acquisition has adopted a dispute
resolution process with some legal underpinnings.13 Otherwise, as the
preface to AMS states, "nothing in this document creates or conveys any
substantive [legal] rights." In short, FAA has assumed no legal obligation
to follow AMS other than to ensure that its actions are not arbitrary and
capricious or contrary to law. By contrast, the FAR has the force and
effect of law, and agencies that are subject to the FAR are bound to
follow it.
AMS Chapter 3 Parallels a Subset of the FAR
When FAA personnel apply the procurement methodology in AMS chapter 3,
they are applying guidance that closely parallels some of the procedures
set out in the FAR. The AMS Chapter 3 acquisition process parallels a
subset of the varied selection of procurement methods available under the
FAR, requiring that all competitive FAA contracts be negotiated with the
awardee being selected on a "best value" basis. The FAR also provides a
much more detailed set of information and guidance than does AMS. A
comparison of high-level differences and similarities between AMS and the
FAR is presented in table 8.
Table 8: Comparison of AMS and the FAR
AMS FAR
Best value source Yes, following screening. Yes, although other
selection methods are also
available for use when
appropriate.
Public announcement Public announcement through Yes, for proposed
of requirement Internet or other means contract actions expected
when value of contract is to exceed $25,000.
anticipated to exceed
$100,000.
Competition FAA's policy is to provide Full and open
reasonable access to competition-all
competition for firms responsible sources are
interested in obtaining permitted to compete.
contracts. In selecting
sources, the preferred
method of procurement is to
compete requirements among
two or more sources.
Sole-source Yes, when deemed to be in Yes, full and open
procurement FAA's "best interest" as competition need not be
determined by the service obtained under certain
organization on the basis specified conditions
of "adequate objective based upon a written
supporting data." justification from the
contracting officer that
is approved at an
appropriate level of
authority.
Prequalification Yes, qualification Yes, for products or
information screens for manufacturers when
those vendors that meet justified in writing and
FAA's stated minimum conducted in a manner
capabilities or that meets requirements
requirements for providing justifying the use of
a given product or service. qualifications
requirements.
Basic methodology in FAA issues one or more Agency issues a
negotiated "screening requests," which solicitation, usually a
procurement may include requests for request for proposals.
binding offers from
competing firms.
Methodology for FAA encourages one-on-one Clarification and
negotiation communications throughout discussions are
the process provided that permitted; one offeror
no offeror is given an cannot be favored over
"unfair advantage." another.
AMS FAR
Evaluation and award Selection is based on Selection is based on
selection evaluation in accord with evaluation in accord with
criteria identified in the criteria identified in
screening request. The the request for
selection decision is a proposals. The selection
judgmental decision made by decision is a judgmental
the source selection decision made by the
official. source selection
official.
Use of simplified Commercial and simplified Generally required for
acquisition methods purchases are used for purchases up to $100,000,
commercial items or for for noncommercial items,
products or services that or on a test basis, up to
have been sold at $5,000,000 for commercial
established catalog or items competition is to
market prices and are be obtained to the
generally purchased on a maximum practicable
fixed-price basis. extent.
Use of credit card Permitted. Permitted.
purchases
Procurement AMS does not include the Provides a broad
methodology level of detail found in selection of procurement
the FAR. It does not methods and techniques
prescribe many of the suitable for use in most
procurement methods and circumstances.
techniques permitted under
the FAR, but encourages use
of "any method of
procurement deemed
appropriate."
Responsibility Awards to responsible Awards to responsible
offerors only. offerors only.
Source: GAO analysis.
AMS Includes a Less Rigorous Competition Requirement Than Does the FAR
As table 8 indicates, AMS incorporates a less rigorous competition
standard than the FAR imposes on the rest of the government. AMS states
that it is FAA's policy to provide reasonable access to competition for
firms interested in obtaining contracts. According to AMS, in selecting
sources, the preferred method of procurement is to compete requirements
among two or more sources. However, there is no requirement to ensure that
firms that want to participate actually get a chance to do so. Instead FAA
may limit competition for further consideration in its screening process
to firms with known capabilities or past performance.
The FAR Gives Procurement Professionals Tighter Control over Procurement
Decisions
AMS states that authority is delegated to appropriate levels. Once the
final investment decision is made, and subject only to any constraints
imposed by that decision, the service-level organization is responsible
for conducting required acquisitions. Contracting personnel as well as
other specialists are then assigned to teams that are responsible to a
program manager within the service-level organization. FAA states that
this approach increases the pace of doing business. By comparison, the FAR
gives contracting professionals clear control over contracting decisions
by requiring that procurement decisions be made by procurement
professionals-typically contracting officers or their superiors.
Although FAA Project Managers View AMS as More Efficient and Flexible Than
the FAR, Some Procurement Officials We Interviewed Do Not Agree
As part of our work, we interviewed project management personnel within
FAA as well as current and former FAA procurement officials that have
experience using both the FAR and AMS. Generally, FAA personnel see AMS as
more efficient and flexible than the FAR, although 9 years after AMS's
adoption, many FAA officials have only limited knowledge of and experience
with the FAR. The FAA project managers we interviewed see AMS as more
efficient and flexible than the FAR,14 but some procurement officials with
experience in applying both AMS and the FAR did not agree with the view
that the FAR was unduly rigid. According to these officials, the FAR may
appear inflexible and cumbersome to persons who are inexperienced with it,
but those who are familiar with it are able to navigate its complexities
effectively. For example, even though the FAR generally requires full and
open competition-a process that can take time to give all interested firms
an opportunity to participate-contracting officers may be able to expedite
the procurement process by using authorized streamlined procedures or, if
circumstances warrant, by justifying sole-source or limited competition.
Status of the Seven ATC Modernization Acquisitions That GAO
ReviewedAppendix V
Key ContributorsAppendix VI
GAO Contacts
JayEtta Z. Hecker, (202) 512-2834 Beverly L. Norwood, (202) 512-2834
Staff Acknowledgments
In addition to the individuals named above, Tamera Dorland, Elizabeth
Eisenstadt, Brandon Haller, Bert Japikse, Carolyn Kirby, Steve Martinez,
Richard Scott, Adam Vodraska, and Dale Yuge made key contributions to this
report.
(540072)
How FAA's Acquisition Policy Adapted Key Recommendations Made by GAO and
DOT (1996-2003)Appendix IV
FAA Refined AMS in Response to Recommendations
Since FAA developed and implemented AMS in 1996, GAO and the DOTIG have
made recommendations to improve FAA's acquisition processes. FAA has
adopted many of these recommendations and incorporated them into AMS (see
table 9). These implemented recommendations address four main themes:
o Developing a strategy for culture change that relies on successfully
integrating the various elements of acquisition, including specific
responsibilities and performance measures for all stakeholders, and
providing the incentives needed to promote the desired changes.
o Establishing an effective management structure for developing,
maintaining, and enforcing the ATC systems architecture to provide an
overall plan for the National Airspace System (NAS). This management
structure should assign the responsibility and accountability to develop,
maintain, and enforce a complete and unified ATC system by ensuring that
every project conforms to the overall plan.
o Improving cost and schedule tracking to provide data for estimating the
costs and schedules of programs. To estimate the costs and time needed for
projects, a historical database that includes cost and schedule estimates,
revisions, reasons for revisions, actual cost and schedule information,
and relevant contextual information is needed.
o Improving the management of modernization projects, including the use of
project reviews, milestones, and baselines, and cost-accounting
information to ensure that programs can be adjusted as needed.
The reports identified in table 10 provide recommendations to address
problems we and the DOTIG have identified under these four themes.
Table 9:
Key recommendation Evidence of policy Rationale for change
change
Aviation Acquisition: A FAA issued an Over the past 15 years,
Comprehensive Strategy Is organizational culture FAA's ATC modernization
Needed for Cultural framework in 1997 and projects have
Change at FAA is working to experienced substantial
implement it. cost overruns, lengthy
August 22, 1996, schedule delays, and
significant performance
(GAO/RCED-96-159) shortfalls. We found
that FAA's
FAA should develop a organizational culture
comprehensive strategy has been an underlying
for cultural change. This cause of the agency's
strategy should include acquisition problems.
specific responsibilities Its acquisitions were
and performance measures impaired because
for all stakeholders employees acted in ways
throughout FAA and that did not reflect a
provide the incentives strong commitment to
needed to promote the mission focus,
desired behaviors and to accountability,
achieve agencywide coordination, and
cultural change. adaptability.
Air Traffic Control: Chapter 19 of FAA's We found that FAA's ATC
Improved Cost Information Pricing Handbook modernization program's
Needed to Make Billion embodies SEI's cost estimating
Dollar Modernization philosophy, which processes do not
Investment Decisions maintains that satisfy recognized
developing credible estimating requisites,
January 22, 1997, software estimates is and its cost-accounting
a function of how practices do not
(GAO/AIMD-97-20) thorough and provide for proper
disciplined an accumulation of actual
Because the success of organization's costs. The result is an
FAA's investment analysis estimating processes absence of reliable
and decision-making are. SEI's six project cost and
process depends in large institutional process financial information
measure on the requisites are that the Congress has
reliability of ATC designed to ensure legislatively specified
project cost information, that organizations and that leading
FAA should consistently produce public-sector and
institutionalize defined reliable cost private-sector
processes for estimating estimates for organizations point to
ATC projects' costs. At a software-intensive as essential to making
minimum, these processes systems. These fully informed
should include the requisites are as investment decisions
following six follows: among competing ATC
institutional process projects. Not having
requisites, developed for o a corporate memory, this information,
organizations that are or historical increases the
building or acquiring database(s), for likelihood of poor ATC
software-intensive cataloging cost investment decisions,
systems by Carnegie estimates, revisions, not only when a project
Mellon University's reasons for revisions, is initiated but also
Software Engineering actual cost and throughout its life
Institute (SEI), an schedule information, cycle. It also means
institution recognized and other descriptive that Congress does not
for its expertise in information, such as have reliable cost
software processes. Each any constraints or information to use in
of these requisites is trends that affect the making funding
described in more detail project; decisions about FAA.
in this report: Such a situation is
o structured processes unacceptable when
o a corporate memory, or for estimating making small
historical database(s), software size and the investments, but is
which includes cost and amount and complexity especially egregious
schedule estimates, of existing software when making
revisions, reasons for that can be reused; multimillion or
revisions, actual cost billion-dollar
and schedule information, o cost models investments in
and relevant descriptive calibrated/tuned to mission-critical ATC
information; reflect demonstrated systems.
accomplishments on
o structured approaches similar past projects;
for estimating software
size and the amount and o audit trails that
complexity of existing record and explain the
software that can be values used as cost
reused; model inputs;
o cost models o processes for
calibrated/tuned to dealing with
reflect demonstrated externally imposed
accomplishments on past cost or schedule
projects; constraints to ensure
the integrity of the
o audit trails that estimating process;
record and explain all
values used as cost model o data collection and
inputs; feedback processes
that foster capturing
o processes for dealing and correctly
with externally imposed interpreting data from
cost or schedule work performed.
constraints in order to
ensure the integrity of
the estimating process;
o data collection and
feedback processes that
foster capturing and
correctly interpreting
data from work performed.
FAA should immediately Chapter 19 of FAA's
begin disclosing the Pricing Handbook
inherent uncertainty and incorporates our
range of imprecision in recommendation and
all ATC projects' refers explicitly to
official cost estimates GAO/AIMD-97-20 and the
presented to executive work of other experts.
oversight agencies or The handbook suggests
Congress. where to incorporate
audit trails,
constraint processes,
and the inherent
uncertainty and range
of imprecision in all
ATC cost estimates.
The handbook advocates
that staff qualify
early project
estimates by
disclosing the level
of uncertainty
associated with them
and refining the
estimates as the
project is completed
and the uncertainty
eliminated.
FAA should acquire or The Department of
develop and implement a Transportation is in
managerial the process of meeting
cost-accounting key objectives of the
capability that will Federal Managers'
satisfy the requirements Financial Integrity
of Statement of Federal Act (FMFIA) of 1982. A
Financial Accounting key material weakness
Standards no. 4 (SFFAS 4) was FAA's oversight of
Managerial Cost cost reimbursable
Accounting Concepts and contracts. FAA made
Standards for the Federal significant progress
Government. This system in the closeout of
capability should provide past cost reimbursable
the cost-accounting and contracts. To resolve
financial management this material
information needed by FAA weakness, FAA needs to
management and those who complete the close out
make investment of old contracts and
decisions. Such increase the use of
information should cost incurred audits.
include full life-cycle Additionally, FAA
costs, which include the needs to ensure that
costs of resources appropriate audits are
consumed by a project obtained for all
that directly or active contracts.
indirectly contribute to These steps will help
the output and the costs reduce the likelihood
of identifiable of cost overruns or
supporting services improper payments for
provided by other unallowable costs.
organizations within the
reporting entity.
FAA should report its
lack of a cost-accounting
capability for its ATC
modernization as a
material internal control
weakness in the
Department's fiscal year
1996 Federal Managers'
Financial Integrity Act
(FMFIA) report and in
subsequent annual FMFIA
reports until the problem
is corrected.
FAA should report to the
Secretary of
Transportation and FAA's
authorizing and
appropriation committees
on its progress in
implementing these
recommendations as part
of its fiscal year 1999
budget submission.
Air Traffic Control: AMS states the FAA lacks a complete
Complete and Enforced National Air Space system architecture, or
Architecture Needed for (NAS) Configuration overall blueprint, to
FAA Systems Modernization Control Board shall guide and constrain the
approve changes to NAS development and
February 3,1997, technical maintenance of the many
documentation, and interrelated systems
(GAO/AIMD-97-30) shall ensure the that make up its ATC
traceability of infrastructure. To its
FAA should ensure that a requirements from the credit, FAA is
complete ATC systems NAS level to the developing one of the
architecture is developed system and subsystem two principal
and enforced level. This components of a
expeditiously before responsibility begins complete systems
deciding on the with the approval of architecture, namely,
architectural the technical the "logical"
characteristics of a architecture by the description of FAA's
replacement for the Host Joint Resources current and future
Computer System. FAA Council at the concept of ATC
should also take the investment decision operations as well as
following steps to and continues descriptions of the ATC
establish an effective throughout the life of business functions to
management structure for the program. be performed, the
developing, maintaining, associated systems to
and enforcing the AMS states that the be used, and the
complete ATC systems Joint Resources information flows among
architecture: Council approves FAA systems. However, FAA
budget submissions for is not developing, nor
o Assign the Research, Engineering does it have plans to
responsibility and and Development (RE&D) develop, the second
accountability needed to and Facilities and essential component-the
develop, maintain, and Equipment (F&E) ATC-wide "technical"
enforce a complete ATC appropriations, descriptions that
systems architecture to a participates in the define all required
single FAA organizational development of FAA information technology
entity. budget submissions for (IT) and
the operations telecommunications
o Provide this single appropriation, and standards and critical
entity with the approves the NAS ATC systems' technical
resources, expertise, and architecture baseline. characteristics.
budgetary and/or
organizational authority AMS states that a We also found that an
needed to fulfill its configuration control architecture is the
architectural board with an approved centerpiece of sound
responsibilities. charter and operating systems development and
procedures shall be maintenance;
o Direct this single the official FAA-wide
entity to ensure that forum used to FAA is developing a
every ATC project establish logical architecture
conforms to the configuration component for ATC
architecture unless management baselines modernization and
careful, thorough, and and to approve or evolution; FAA lacks a
documented analysis disapprove subsequent technical architectural
supports an exception. changes to those component to guide and
Given the importance and baselines. constrain ATC
the magnitude of the IT modernization and
initiative at FAA, a evolution; without a
management structure technical ATC
similar to the architecture, costly
department-level chief system
information officer (CIO) incompatibilities have
structure prescribed in resulted and
the Clinger-Cohen Act
should be established for will continue; and FAA
FAA. lacks an effective
management structure
for developing and
enforcing an ATC
systems architecture.
Air Traffic Control: FAA states that the To accommodate
Immature Software CIO: forecasted growth in
Acquisition Processes air traffic and replace
Increase FAA System o serves as the aging equipment, FAA
Acquisition Risks principal adviser to embarked on an
the Administrator, ambitious ATC
March 21, 1997, Deputy Administrator, modernization program
and FAA offices on in 1981. FAA estimated
(GAO/AIMD-97-47) information management that it would spend
and technology across about $20 billion to
Given the importance and the agency. As the replace and modernize
the magnitude of IT at agency's senior software-intensive ATC
FAA, this report management official, systems between 1982
reiterates our earlier serves as the and 2003. Our work over
recommendation calling spokesperson on IT the years has
for the establishment at matters before chronicled many FAA
FAA of a CIO management Congress, other failures in meeting ATC
structure similar to the agencies, and the projects' cost,
department-level CIO public; schedule, and
structure prescribed in performance goals,
the Clinger-Cohen Act of o leads and directs largely because of
1996. agencywide strategic software-related
planning for IT; problems. As a result
To improve its ability to of these failures as
acquire software for its o oversees IT well as the tremendous
ATC modernization, FAA investments to ensure cost, complexity, and
should optimization across mission criticality of
all agency groups and FAA's ATC modernization
o assign responsibility the full range of cost program, we designated
for software acquisition trade-offs; the program as a
process improvement to high-risk IT initiative
the agency's CIO; o creates and in our 1995 and 1997
maintains an IT report series on
o provide the CIO with strategy to guide high-risk programs.
the authority needed to research, development,
implement and enforce ATC maintenance, and Software quality is
modernization software sharing of information governed largely by the
acquisition process systems, applications, quality of the
improvement; data, and other processes involved in
resources across the developing or
o require the CIO to lines of business and acquiring, and
develop and implement a throughout the agency; maintaining it. SEI has
formal plan for ATC developed models and
modernization software o leads the methods that define and
acquisition process establishment of determine
improvement that is based world-class software organizations' software
on the software and information process maturity.
capability evaluation systems engineering Together, they provide
results contained in this methodologies a logical framework for
report and specifies including Capability baselining an
measurable goals and time Maturity Models, and organization's current
frames, prioritizes applies them to agency process capabilities
initiatives, estimates systems, operations, (i.e., strengths and
resource requirements, and processes to weaknesses) and
and assigns roles and provide continuous providing a structured
responsibilities; improvement of IT plan for incremental
performance; and rocess improvement.
o allocate adequate
resources to ensure that o leads and directs We found that
planned initiatives are agencywide efforts on
implemented and enforced; o FAA's ATC
and information systems modernization software
security, ensuring acquisitions processes
o require that, before that standards and are immature and
being approved, every policies are in place
to provide security o FAA's approach for
ATC modernization for the critical improving AT
acquisition project have information
software acquisition architecture of the modernization software
processes that satisfy at agency. acquisition processes
least Software is not effective.
Acquisition Capability
Maturity Model (SA-CMM)
level 2.
Air Traffic Control: FAA's AMS states that Over the past 17 years,
FAA's Modernization five decisions are FAA's modernization
Investment Management always made at the projects have
Approach Could Be corporate level by the experienced substantial
Strengthened, Joint Resources cost overruns, lengthy
Council: the mission delays, and significant
April 30, 1999, need decision, the performance shortfalls.
investment decision, Because of FAA's
(GAO/RCED/AIMD-99-88) the decision to contention that some of
approve a change to an its modernization
FAA should implement a acquisition program problems were caused by
comprehensive investment baseline, approval of federal acquisition
management approach the RE&D and F&E regulations, the
through AMS that includes budget submissions, Congress enacted
the following actions: and approval of the legislation in November
NAS Architecture 1995 that exempted the
o Establish a complete baseline. The agency from most
portfolio of selection of a federal procurement
investments-including solution to satisfy a laws and regulations
existing systems funded mission need, the and directed FAA to
by the operations budget investment of develop a new
account as well as resources into a fully acquisition management
projects funded by the funded program, and system. In response,
facilities and equipment the possible need to FAA implemented AMS on
account-and require the cancel other programs April 1, 1996. AMS
Joint Resources Council to accommodate a new provides high-level
to periodically review program make the acquisition policy and
the baseline status and investment decision guidance for selecting
merits of each of these the most important in and controlling
investments throughout the life-cycle investments throughout
their entire life cycle. management process. all phases of the
As part of this acquisition life
portfolio, cost baselines cycle.
for operating and
maintaining all projects GAO found that:
should be developed, and
this information should o FAA's AMS is designed
be included in the to provide a
agency's financial plan discipline, structured
for its investments and process for selecting
in its annual budget and controlling
request to Congress. investments;
o Lack of oversight of
the operations portion
of projects prevents
FAA from managing
investments as a
complete portfolio;
o Weaknesses in
selection, control, and
evaluation phases limit
FAA's effectiveness in
managing its portfolio.
o Improve the selection o FAA's AMS states that
process by (1) the investment
establishing clearly analysis team develops
defined procedures for an initial acquisition
validating each project's program baseline
cost, schedule, benefit, (i.e., performance,
performance, and risk cost, schedule,
information and (2) benefits, and risk)
requiring documentation for each alternative
of the results of the solution offering
validation procedures superior value and
applied to each project. benefit to FAA and its
customers. Service
organization members
of the investment
analysis team lead the
development of cost
and schedule baselines
using FAA's work
breakdown structure
and other applicable
standards.
o Strengthen control over o AMS states that the
investments by (1) acquisition program
revising the acquisition baseline should
program baseline include cost,
requirements to include schedule, performance,
project risks and to add benefits, and risk
milestones for project information. It also
reviews during the should include all
operations phase and (2) events that are key to
ensuring that project satisfying mission
officials fully track and need, providing
document estimated versus intended operational
actual results for all capability, and
the elements (i.e., cost, accruing benefits, as
schedule, benefit, well as events crucial
performance, and risk) to interrelated
contained in the baseline programs or NAS
documentation. systems. Once an
estimate has been
completed and a
project started, FAA
establishes reporting
and performance
measures to compare
estimated and actual
costs, schedules, and
performance.
o Initiate post o FAA published a
implementation methodology for
evaluations for projects conducting such
within 3 to 12 months of evaluations entitled
deployment or An Approach for
cancellation to compare Developing a Standard
the completed projects' Method for Conducting
cost, schedule, Post-Implementation
performance, and mission Reviews, Report
improvement outcomes with #2001-13, June 6,
the original estimates. 2001.
o Incorporate key
information from the
selection process (e.g.,
mission need statements,
cost-benefit analyses,
and risk assessments)
into FAA's management
information system for
investments.
Major Management FAA has appointed an DOT's management of its
Challenges and Program independent major acquisitions and
Risks, Department of board-consisting of assets needs
Transportation external experts in improvement in several
satellite navigation, areas. FAA and the U.S.
January 2001, safety certification, Coast Guard are
and radio undertaking costly,
(GAO-01-253) spectrum-that reports long-term programs to
directly to the FAA modernize and replace
FAA should develop a Administrator. The aging equipment. Over
comprehensive plan that board is tasked with the past 19 years,
would include established reviewing the FAA's
checkpoints at which the soundness of the multibillion-dollar ATC
agency would determine, panel's modernization program
among other things, recommendations and has experienced cost
whether users' needs have with revalidating the overruns, delays, and
changed and whether other future path for WAAS. performance shortfalls
technologies have matured However, given the of large proportions.
and could better meet past problems in FAA is making progress
users' needs and the developing this system in addressing some of
agency's requirements for and the long-term our recommendations,
satellite navigation. FAA effort that is still but its reform efforts
should also have an required, we believe are not complete, and
external organization that continued major projects continue
evaluate its progress at oversight by an to face cost, schedule,
established checkpoints independent group of and performance
and include the results experts is warranted. problems. Because of
of this evaluation in its It is not clear its size, complexity,
request for future whether the current cost, and
funding of the navigation independent board will problem-plagued past,
system. fulfill this role. We we designated FAA's IT
will continue to program as a high-risk
evaluate FAA's IT initiative in 1995.
progress on this and
other system
acquisition efforts.
Status of FAA's Major FAA officials FAA has made progress
Acquisitions generally agreed with with a number of
the analysis and acquisitions, including
DOT/OIG, AV-2003-045, recommendations in Free Flight Phase 1 and
this report. FAA is new information
June 26, 2003 implementing this exchange systems that
recommendation. It link FAA and airline
Update the cost, updated the baseline operations centers.
schedule, and performance of STARS in April 2004 However, other
baselines for many of and updated the modernization programs
FAA's major acquisition, baselines of ITWS and have experienced cost,
including STARS, ITWS, WAAS in May 2004. The schedule, and
LAAS, and WAAS at a LAAS program was performance problems.
minimum. Develop-and deferred because of Problems with
use-performance goals for budget cuts. acquisition efforts
assessing progress with have serious
its major consequences because
acquisitions. This should they result in costly
involve holding staff and interim systems, reduce
contractors accountable the number of units
for keeping projects procured, postpone
within cost and schedule, benefits, or "crowd
as appropriate. out" other
modernization projects.
Status Report on FAA's FAA officials The OEP is an important
Operational Evolution generally agreed with effort because it will
Plan the analysis and shape FAA and industry
recommendations in investments over the
DOT/OIG, AV-2003-048, this report. FAA is next decade. However,
currently updating the much has changed since
July 23, 2003 OEP, which includes the OEP was
design changes to the introduced. The demand
Develop realistic cost National Airspace to, for air travel has
estimates, and link the for example, enhance declined, major network
Operational Evolution capacity. carriers are in
Plan (OEP) with FAA's financial distress, and
budget in order to set Aviation Trust Fund
priorities for what can revenues have declined
be accomplished in the sharply. The Inspector
short term. General found that
fundamental assumptions
Determine-in concert with about the OEP, such as
the aviation the cost, schedule, and
community-how to move benefits of key efforts
forward (and at what as well as the ability
pace) with systems that of airspace users to
require airspace users to pay for and equip with
purchase and install new new technologies in the
technologies. near term, are no
longer valid and need
Determine and maximize to be revised.
the benefits associated
with airspace design
changes, new procedures,
and capabilities
currently onboard
aircraft to enhance
system capacity.
Key Recommendations Made to Improve FAA's Acquisition Processes
Source: GAO analysis.
sector best practices and that the agency had highly developed processes
for four of these six elements. See figure 4.
Figure 4: Our Analysis of FAA's Progress as of 2003 in Implementing Key
Elements of Training for Its Acquisition Workforce
Since 2003, FAA has taken some steps to measure the effectiveness of its
training. For example, the agency collects and reviews participants'
assessments of the knowledge they have gained, the extent that learning
objectives were achieved and the applicability and usefulness of the
training. In addition, members of FAA's Intellectual Capital Investment
Plan Council35 have attempted to make qualitative judgments about the
impact of the training on the effectiveness or efficiency of their
organizations. However, FAA is still developing an evaluation program with
metrics to measure the extent to which organizational goals are achieved
when individual training objectives are met. Industry and government
experts believe training and human capital investments are prerequisites
for
*** End of document. ***