Air Traffic Control: Immature Software Acquisition Processes Increase FAA
System Acquisition Risks (Chapter Report, 03/21/97, GAO/AIMD-97-47).
Pursuant to a congressional request, GAO reviewed the Federal Aviation
Administration's (FAA) air traffic control (ATC) modernization software
acquisition efforts, focusing on the: (1) maturity of FAA's ATC
modernization software acquisition processes; and (2) steps/actions FAA
has underway or planned to improve these processes, including any
obstacles that may impede FAA's progress.
GAO noted that: (1) because of the number and severity of FAA ATC
modernization software acquisition process weaknesses, FAA did not fully
satisfy any of the seven key process areas (KPA) necessary to achieve
the "repeatable" level of process maturity; (2) as a result, its
processes for acquiring software, the most costly and complex component
of ATC systems, are ad hoc, sometime chaotic, and not repeatable across
projects; (3) in addition, serious process weaknesses prevented FAA from
satisfying the one KPA specified under the Software Engineering
Institute's "defined" maturity level; (4) while FAA showed process
strengths, primarily in the solicitation and evaluation KPAs, GAO found
extensive weaknesses in these and the remaining six KPAs; (5) some of
these weaknesses were systemic, recurring in each of the KPAs; (6) for
example, no software project teams measured or reported to management on
the status of activities performed, and management never verified that
critical activities were being done; (7) these types of problems are
some of the reasons for FAA's frequent failures to deliver promised ATC
system capabilities on time and within budget; (8) FAA has stated its
commitment to increasing ATC modernization process maturity; (9)
however, despite 4 years of activity in this area, FAA lacks an
effective management approach for improving software acquisition
processes; (10) currently, the Software Engineering Process Group (SEPG)
is responsible for process improvement, but the SEPG has neither
organizational nor budgetary authority over the product teams that
acquire software, and, therefore, cannot effectively implement or
enforce process change; (11) instead, it can only recommend and
encourage change; (12) additionally, FAA does not have an effective plan
to correctly target and prioritize improvements and measure improvement
progress; (13) in the absence of this plan, it has initiated a "hodge
podge" of software acquisition improvement efforts without any
analytical justification; and (14) as a result, FAA's process
improvement activities have yet to produce more repeatable, better
defined, more disciplined software acquisition processes.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: AIMD-97-47
TITLE: Air Traffic Control: Immature Software Acquisition
Processes Increase FAA System Acquisition Risks
DATE: 03/21/97
SUBJECT: ADP procurement
Computer software
Air traffic control systems
Strategic information systems planning
Computer software verification and validation
Requirements definition
IDENTIFIER: Software Capability Maturity Model
FAA Automated Radar Terminal System
FAA Display System Replacement
FAA Voice Switching and Control System
FAA National Airspace System
FAA Air Traffic Control Modernization Program
FAA Advanced Automation System
FAA Integrated Terminal Weather System
FAA Air Route Surveillance Radar-4
FAA Initial Sector Suite System
FAA Aviation Weather Observing System
NAVSTAR Global Positioning System
FAA Weather and Radar Processor
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Chairman, Subcommittee on Transportation, Committee on
Appropriations, House of Representatives
March 1997
AIR TRAFFIC CONTROL - IMMATURE
SOFTWARE ACQUISITION PROCESSES
INCREASE FAA SYSTEM ACQUISITION
RISKS
GAO/AIMD-97-47
Air Traffic Control
(511487)
Abbreviations
=============================================================== ABBREV
AAR - Office of Aviation Research
AAS - Advanced Automation System
ACT - William J. Hughes Technical Center
AIMD - Accounting and Information Management Division
AIT - Office of Information Technology
AND - Office of Communication, Navigation, and Surveillance Systems
ARA - Associate Administrator for Research and Acquisitions
ARINC - Aeronautical Radio Incorporated
ARTS - Automated Radar Terminal System
ASD - Office of System Architecture and Investment Analysis
ASU - Office of Acquisitions
ATC - air traffic control
ATCSCC - Air Traffic Control System Command Center
ATS - Associate Administrator for Air Traffic Services
AUA - Office of Air Traffic Systems Development
CIO - Chief Information Officer
CMM - Capability Maturity Model
COTS - commercial, off-the-shelf
DSR - Display System Replacement
FAA - Federal Aviation Administration
GAO - General Accounting Office
IPT - Integrated Product Team
ISSS - Initial Sector Suite System
KPA - key process area
NAS - National Airspace System
NDI - non-development item
NIMS - NAS Infrastructure Management System
SA-CMM - Software Acquisition Capability Maturity Model
SE-CMM - Systems Engineering Capability Maturity Model
SCE - Software Capability Evaluation
SEEC - Software Engineering Executive Committee
SEI - Software Engineering Institute
SEPG - Software Engineering Process Group
SW-CMM - Capability Maturity Model for Software
TRACON - Terminal Radar Approach Control
VSCS - Voice Switching and Control System
WARP - Weather and Radar Processor
Letter
=============================================================== LETTER
B-271531
March 21, 1997
The Honorable Frank R. Wolf
Chairman, Subcommittee on
Transportation
Committee on Appropriations
House of Representatives
Dear Mr. Chairman:
This report responds to your request that we review the Federal
Aviation Administration's (FAA) air traffic control modernization
software acquisition maturity and improvement activities. FAA plans
to spend billions of dollars replacing its existing air traffic
control systems, but has a history of performing poorly when
acquiring these software-intensive systems. We found that FAA's
software acquisition processes are immature, and are making
recommendations to the Secretary of Transportation for strengthening
them.
We are sending copies of this report to the Secretary of
Transportation, the Director of the Office of Management and Budget,
the Administrator of the Federal Aviation Administration, and other
congressional committees. We will also make copies available to
other interested parties upon request. If you have questions or wish
to discuss the issues in this report, please contact me at (202)
512-6412. Major contributors to this report are listed in appendix
II.
Sincerely yours,
Dr. Rona B. Stillman
Chief Scientist for Computers
and Telecommunications
EXECUTIVE SUMMARY
============================================================ Chapter 0
PURPOSE
---------------------------------------------------------- Chapter 0:1
The Federal Aviation Administration (FAA) is modernizing the air
traffic control (ATC) systems upon which it will rely to ensure safe,
orderly, and efficient air travel well into the 21st century. Since
software is the most expensive and complex component of these
systems, FAA must use defined and disciplined processes when it
acquires software.
Recognizing software's growing importance and prevalence in ATC
systems, the Chairman, Subcommittee on Transportation and Related
Agencies, House Committee on Appropriations, asked GAO to determine
(1) the maturity of FAA's ATC modernization software acquisition
processes, and (2) the steps/actions FAA has underway or planned to
improve these processes, including any obstacles that may impede
FAA's progress.
BACKGROUND
---------------------------------------------------------- Chapter 0:2
To accommodate forecasted growth in air traffic and replace aging
equipment, FAA embarked on an ambitious ATC modernization program in
1981. FAA estimates that it will spend about $20 billion to replace
and modernize software-intensive ATC systems between 1982 and 2003.
Our work over the years has chronicled many FAA failures in meeting
ATC projects' cost, schedule, and performance goals, largely because
of software-related problems. As a result of these failures as well
as the tremendous cost, complexity, and mission criticality of FAA's
ATC modernization program, we designated the program as a high-risk
information technology initiative in our 1995 and 1997 report series
on high-risk programs.\1
Software quality is governed largely by the quality of the processes
involved in developing or acquiring, and maintaining it. Carnegie
Mellon University's Software Engineering Institute (SEI), recognized
for its expertise in software processes, has developed models and
methods that define and determine organizations' software process
maturity. Together, they provide a logical framework for baselining
an organization's current process capabilities (i.e., strengths and
weaknesses) and providing a structured plan for incremental process
improvement.
Using SEI's software acquisition capability maturity model
(SA-CMM),\2 SEI's software capability evaluation method, and SA-CMM
authors as consultants, GAO staff trained at SEI evaluated FAA's ATC
modernization software acquisition maturity in the seven key process
areas (KPA) necessary to attain a "repeatable" level of process
capability, and one KPA associated with the "defined" level of
process maturity.\3 Repeatability ensures that an organization has
the necessary process discipline in place to repeat earlier successes
on projects in similar domains. Repeatable processes are at the
second level on SEI's five-level scale of process maturity.
Organizations that do not satisfy the requirements for the
"repeatable" level are by default judged to be at the "initial" level
of maturity, meaning that their processes are ad hoc, sometimes even
chaotic, with few of the processes defined and success dependent
mainly on the heroic efforts of individuals. The one KPA associated
with the third level of process maturity, which is called the
"defined" level, is acquisition risk management. It was included
because many software experts consider it to be a very important
process area.
As part of its evaluation, GAO examined five ongoing ATC
modernization projects selected by FAA.\4 These were the Automated
Radar Terminal System, Display System Replacement, National Airspace
System Infrastructure Management System, Voice Switching and Control
System, and the Weather and Radar Processor. (See chapter 1 of this
report for more detailed information on GAO's evaluation scope and
methodology.)
--------------------
\1 High-Risk Series: An Overview (GAO/HR-95-1, Feb. 1995);
High-Risk Series: Information Management and Technology
(GAO/HR-97-9, Feb. 1997).
\2 We used a draft version of the model for our evaluation (version
00.03, dated April 1996). The first published version of the model
was released on October 1996, after we performed our evaluation.
According to the model's authors, the published version differed only
editorially from the draft we used.
\3 The seven KPAs relating to the repeatable level are software
acquisition planning, solicitation, requirements development and
management, project office management, contract tracking and
oversight, evaluation, and transition and support.
\4 GAO asked FAA to choose five projects that are: (1) major efforts
with large software acquisition components, (2) managed by different
FAA product teams, (3) at different life cycle stages, and (4) among
FAA's best managed.
RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3
Because of the number and severity of FAA ATC modernization software
acquisition process weaknesses, FAA did not fully satisfy any of the
seven KPAs necessary to achieve the "repeatable" level of process
maturity. As a result, its processes for acquiring software, the
most costly and complex component of ATC systems, are ad hoc,
sometimes chaotic, and not repeatable across projects. In addition,
serious process weaknesses prevented FAA from satisfying the one KPA
specified under SEI's "defined" maturity level. While FAA showed
process strengths, primarily in the solicitation and evaluation
(i.e., testing) KPAs,\5 GAO found extensive weaknesses in these and
the remaining six KPAs (i.e., software acquisition planning,
requirements development and management, project office management,
contract tracking and oversight, transition and support, and
acquisition risk management).\6 Some of these weaknesses were
systemic, recurring in each of the KPAs. For example, no software
project teams measured or reported to management on the status of
activities performed, and management never verified that critical
activities were being done. These types of problems are some of the
reasons for FAA's frequent failures to deliver promised ATC system
capabilities on time and within budget.
FAA has stated its commitment to increasing ATC modernization process
maturity. However, despite 4 years of activity in this area, FAA
lacks an effective management approach for improving software
acquisition processes. Currently, the Software Engineering Process
Group (SEPG) is responsible for process improvement; but the SEPG has
neither organizational nor budgetary authority over the product teams
that acquire software, and, therefore, cannot effectively implement
or enforce process change. Instead, it can only recommend and
encourage change. Additionally, FAA does not have an effective plan
to correctly target and prioritize improvements and measure
improvement progress. In the absence of this plan, it has initiated
a "hodge podge" of software acquisition improvement efforts without
any analytical justification. As a result, FAA's process improvement
activities have yet to produce more repeatable, better defined, more
disciplined software acquisition processes.
--------------------
\5 According to the SA-CMM, solicitation is the process of ensuring
that award is made to the contractor most capable of satisfying the
specified requirements, and evaluation is the process of determining
that acquired software products and services satisfy contract
requirements prior to acceptance.
\6 According to the SA-CMM, software acquisition planning is the
process for ensuring that reasonable planning for all elements of the
software acquisition occur; requirements development and management
is the process for establishing an unambiguous and agreed upon set of
software requirements; project office management is the process for
effective and efficient management of project office activities;
contract tracking and oversight is the process of ensuring that
contractor activities, products, and services satisfy contract
requirements; transition and support is the process of transferring
acquired software products to the eventual support organization; and
acquisition risk management is the process of identifying software
risks early and adjusting the acquisition strategy to mitigate those
risks.
PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:4
ATC MODERNIZATION SOFTWARE
ACQUISITION PROCESSES ARE
IMMATURE
-------------------------------------------------------- Chapter 0:4.1
To attain a given SEI-defined maturity level, an organization must
satisfy the key practices for the KPAs associated with that level.
FAA's ATC modernization organization had too many weaknesses to
satisfy any of the "repeatable" KPAs (i.e., software acquisition
planning, solicitation, requirements development and management,
project office management, contract tracking and oversight,
evaluation, and transition and support), nor does it satisfy the
acquisition risk management KPA from the "defined" or third maturity
level.
For FAA to satisfy any of these eight KPAs, it must eliminate the key
practice weaknesses identified in this report.\7 Each practice that
is performed effectively constitutes a strength, and each practice
not performed or performed poorly constitutes a weakness. While
FAA's ATC modernization has some strengths, it has more weaknesses.
Table 1 tallies these strengths on the five projects that GAO
evaluated. In summary, of the total number of KPA practices rated,
38 percent constituted strengths, 50 percent were weaknesses, and 12
percent were observations. An observation indicates that the
evidence was inconclusive and did not clearly support a determination
of either strength or weakness.
Table 1
Collective Number of KPA Strengths,
Weaknesses, and Observations on the Five
Projects
Number of
Number of Number of observatio
Key Process Area strengths weaknesses ns
---------------------------------- ---------- ---------- ----------
Software acquisition planning 16 37 7
Solicitation 36 28 14
Requirements development and 17 35 6
management
Project office management 26 35 6
Contract tracking and oversight 26 32 6
Evaluation 43 21 8
Transition and support 27 32 8
Acquisition risk management 16 46 7
======================================================================
Totals 207 266 62
----------------------------------------------------------------------
Additionally, GAO found that while the five projects varied as to
practice strengths, weaknesses, and observations under three of the
five "common features" or practice groupings (commitment to perform,
ability to perform, and activities performed), the projects were
consistently weak in all practices under the remaining two groupings
(measurement and analysis and verifying implementation). As a
result, software project teams and FAA management consistently lack
reliable information on project team performance.
--------------------
\7 SEI groups each of its KPA practices into one of five "common
features" or practice categories. These are "commitment to perform,
ability to perform, activities performed, measurement and analysis,
and verifying implementation."
FAA'S APPROACH FOR IMPROVING
ATC MODERNIZATION SOFTWARE
ACQUISITION PROCESSES IS NOT
EFFECTIVE
-------------------------------------------------------- Chapter 0:4.2
To be effective, the FAA organization responsible for software
acquisition process improvement must have (1) organizational and/or
budgetary authority over the ATC modernization units acquiring the
software; and (2) an effective plan to guide improvement efforts and
measure progress on each. The FAA organizational entity currently
responsible for ATC modernization software acquisition process
improvement, the SEPG, has neither. As a result, little progress has
been made over the last 4 years in instituting definition and
discipline into ATC modernization software acquisition processes.
The SEPG is a multilevel committee structure chaired by a member of
FAA's Chief Information Officer's (CIO) staff. The SEPG is directed
by the Software Engineering Executive Committee, which is chaired by
the head of the ATC modernization program. The SEPG has no authority
to implement and enforce process change. Consequently, it can only
attempt to encourage and persuade software acquirers to establish and
follow defined and disciplined software acquisition processes.
The SEPG and its predecessors have advocated and initiated a
collection of efforts intended to strengthen ATC modernization
software-related processes, including software acquisition processes.
However, there is no analytical basis for the focus, content, timing,
and interrelationships of these efforts. Specifically, the efforts
(1) are not based on any assessment of current software acquisition
process strengths and weaknesses; and (2) are not detailed in a
formal plan that specifies measurable goals, objectives, milestones,
and needed resources, prioritizes efforts, fixes responsibility and
accountability, and defines metrics for measuring progress. Instead,
these efforts were undertaken with no sound analytical basis and,
rather than being part of a comprehensive plan, are discussed in
general terms without detail and specificity in briefing documents,
minutes of meetings, and working group recommendations. While the
SEPG is now taking steps to establish the analytical basis needed to
formulate a comprehensive software process improvement plan, that
plan does not yet exist, and no schedule has been established for
completing it.
RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5
Given the importance and the magnitude of information technology at
FAA, GAO reiterates its earlier recommendation that a CIO management
structure similar to the department-level CIOs as prescribed in the
Clinger-Cohen Act of 1996 be established for FAA.\8
To improve FAA's software acquisition capability for its ATC
modernization, GAO recommends that the Secretary of Transportation
direct the FAA Administrator to:
-- assign responsibility for software acquisition process
improvement to FAA's CIO;
-- provide FAA's CIO with the authority needed to implement and
enforce ATC modernization software acquisition process
improvement;
-- require the CIO to develop and implement a formal plan for ATC
modernization software acquisition process improvement that is
based on the software capability evaluation results contained in
this report and specifies measurable goals and time frames,
prioritizes initiatives, estimates resource requirements, and
assigns roles and responsibilities;
-- allocate adequate resources to ensure that planned initiatives
are implemented and enforced; and
-- require that, before being approved, every ATC modernization
acquisition project have software acquisition processes that
satisfy at least SA-CMM level 2 requirements.
--------------------
\8 Air Traffic Control: Complete and Enforced Architecture Needed
for FAA Systems Modernization (GAO/AIMD-97-30, February 3, 1997).
AGENCY COMMENTS AND GAO'S
EVALUATION
---------------------------------------------------------- Chapter 0:6
In its written comments on a draft of this report, the Department of
Transportation recognized the importance of mature software
acquisition processes, agreed that FAA's processes are insufficiently
mature, acknowledged that FAA process improvement activities have yet
to produce greater software acquisition process discipline, and
reaffirmed FAA's commitment to improving its software acquisition
capabilities using the SA-CMM. However, the Department did not state
what, if any, specific action it would take on GAO's recommendations.
Additionally, it took the positions that (1) the SA-CMM by itself is
inadequate to evaluate ATC system acquisition capabilities, is too
new to use as an authoritative source of guidance, and "may" have
been misapplied by GAO, (2) the report does not sufficiently
recognize FAA's process improvement organization and progress nor the
difficulties and time required to affect process improvement change,
(3) the SEPG, which is FAA's designated agent for software
acquisition process change, is organized as the Department
"understands" other SEPGs to be organized, and (4) the report "leads
the reader to erroneously conclude that the five programs reviewed
are in trouble" relative to attainment of cost and schedule goals.
None of these positions are valid. First, the SA-CMM, like the
SW-CMM (another SEI software-specific capability maturity model),
focuses on software because it is widely recognized as the most
difficult and costly component of modern computer systems; the one
most frequently associated with late deliveries, cost overruns, and
performance shortfalls; and the one in greatest need of special
management attention. Further, while the SA-CMM is relatively new,
the processes it requires are well established, experience-based
tenets of effective software acquisition that are widely supported
throughout industry and government. Moreover, GAO applied the SA-CMM
at FAA properly and with extraordinary diligence: Every member of
the evaluation team was trained at SEI; the team leader was certified
by SEI as a lead evaluator; and three SEI professionals, including
two authors of the SA-CMM, participated in the evaluation and
concurred with every practice determination (e.g., strength,
weakness).
Second, FAA's many software acquisition process improvement
activities were undertaken without assessing current software
acquisition process strengths and weaknesses, and were not part of a
comprehensive plan for process improvement. Therefore, FAA had no
analytical basis for deciding what improvement activities to
initiate, or what priorities to assign them. Further, although FAA
began drafting a plan during the course of GAO's evaluation, it has
no schedule for completing it. In describing FAA's progress to date
in improving its processes, the report delineates a wide array of FAA
process improvement activities, but distinguishes these activities
from actual progress. In fact, after 4 years of activity, FAA could
not point to a single case in which it had instituted a more
disciplined software acquisition process. Since SEI published
statistics show that the median time to improve from software
development CMM level 1 to level 2 is 26 months, and from level 2 to
level 3 is 17 months, it is entirely reasonable to expect FAA to be
able to demonstrate some improvement in its processes after 4 years.
Third, the issue is not whether FAA's SEPG is organized as the
Department "understands" other SEPGs to be organized, but whether the
SEPG, or any FAA organizational entity responsible for implementing
and enforcing software acquisition process change, has the authority
needed to accomplish the task. Currently, no organizational entity
in FAA has the requisite authority.
Last, the report addresses the maturity of FAA's software acquisition
processes and concludes that these processes are ad hoc and
undisciplined, reducing the probability that software-intense ATC
modernization projects will consistently perform as intended and be
delivered on schedule and within budget. The report does not address
the overall status of the projects covered by GAO's review, and,
therefore, provides no basis for drawing conclusions about the
projects' overall cost or schedule performance.
The Department's comments and GAO's evaluation of them are presented
in greater detail in chapter 11 of this report.
INTRODUCTION
============================================================ Chapter 1
The Federal Aviation Administration's (FAA) primary mission is to
ensure safe, orderly, and efficient air travel in the national
airspace. FAA's ability to fulfill this mission depends on the
adequacy and reliability of the nation's air traffic control (ATC)
system, a vast network of computer hardware, software, and
communications equipment.\1 The ATC system, however, is being
strained by aging equipment, much of which is 1960's technology, and
growing air traffic. This growth should continue as the number of
passengers traveling on U.S. airlines is expected to increase by 38
percent between 1995 and 2003, from about 580 million to nearly 800
million.
To accommodate the forecasted growth in air traffic and to relieve
the problems of the aging ATC system, FAA embarked on an ambitious
ATC modernization program in 1981. FAA estimates that it will spend
about $20 billion to replace and modernize software-intensive ATC
systems between 1982 and 2003. Our work over the years has
chronicled many FAA failures in meeting ATC projects' cost, schedule,
and performance goals.\2 As a result of these failures as well as the
tremendous cost, complexity, and mission criticality of FAA's ATC
modernization program, we designated the program as a high-risk
information technology initiative in our 1995 and 1997 report series
on high-risk programs.\3
--------------------
\1 The ATC system is a major component of the National Airspace
System (NAS).
\2 Air Traffic Control: Status of FAA's Modernization Program
(GAO/RCED-95-175FS, May 26, 1995); Air Traffic Control: Status of
FAA's Modernization Program (GAO/RCED-94-167FS, Apr. 15, 1994); Air
Traffic Control: Status of FAA's Modernization Program
(GAO/RCED-93-121FS, Apr. 16, 1993).
\3 High-Risk Series: An Overview (GAO/HR-95-1, Feb. 1995);
High-Risk Series: Information Management and Technology
(GAO/HR-97-9, Feb. 1997).
OVERVIEW OF ATC
---------------------------------------------------------- Chapter 1:1
Automated information processing and display, communication,
navigation, surveillance, and weather resources permit air traffic
controllers to view key information, such as aircraft location,
aircraft flight plans, and prevailing weather conditions, and to
communicate with pilots. These resources reside at, or are
associated with, several ATC facilities--air traffic control towers,
terminal radar approach control (TRACON) facilities, air route
traffic control centers (en route centers), flight service stations,
and the Air Traffic Control System Command Center (ATCSCC). These
facilities' ATC functions are described below.
-- Airport towers control aircraft on the ground and before landing
and after take-off when they are within about 5 nautical miles
of the airport, and up to 3,000 feet above the airport. Air
traffic controllers rely on a combination of technology and
visual surveillance to direct aircraft departures and
approaches, maintain safe distances between aircraft, and
communicate weather-related information, clearances, and other
instructions to pilots and other personnel.
-- Approximately 180 TRACONs sequence and separate aircraft as they
approach and leave busy airports, beginning about 5 nautical
miles and ending about 50 nautical miles from the airport, and
generally up to 10,000 feet above the airport, where en route
centers' control begins.
-- Twenty en route centers control planes over the continental
United States in transit and during approaches to some airports.
Each en route center handles a different region of airspace,
passing control from one to another as respective borders are
reached until the aircraft reaches TRACON airspace. En route
center controlled airspace usually extends above 18,000 feet for
commercial aircraft. En route centers also handle lower
altitudes when dealing directly with a tower, or when agreed
upon with a TRACON.
-- Two en route centers--Oakland and New York--also control
aircraft over the ocean. Controlling aircraft over oceans is
radically different from controlling aircraft over land because
radar surveillance only extends 175 to 225 miles offshore.
Beyond the radars' sight, controllers must rely on periodic
radio communications through a third party--Aeronautical Radio
Incorporated (ARINC), a private organization funded by the
airlines and FAA to operate radio stations--to determine
aircraft locations.
-- About 90 flight service stations provide pre-flight and
in-flight services, such as flight plan filing and weather
report updates, primarily for general aviation aircraft.
See figure 1.1 for a visual summary of air traffic control over the
continental United States and oceans.
Figure 1.1: Summary of ATC
Over the Continental United
States and Oceans
(See figure in printed
edition.)
THE ATC MODERNIZATION PROGRAM
IS COMPLEX, COSTLY, AND
HISTORICALLY PROBLEMATIC
---------------------------------------------------------- Chapter 1:2
FAA faced two problems in continuing to fulfill its mission to ensure
safe, orderly, and efficient air travel in the national airspace.
First, the ATC system of the late 1970s was a blend of several
generations of automated and manual equipment, much of it
labor-intensive and obsolete. Second, air traffic was projected to
increase dramatically as a result of airline deregulations of the
late 1970s. FAA recognized that it could increase ATC operating
efficiency by increasing automation. It also anticipated that
meeting the demand safely and efficiently would require improved and
expanded services, additional facilities and equipment, improved work
force productivity, and the orderly replacement of aging equipment.
Accordingly, in December 1981, FAA initiated its plan to modernize,
automate, and consolidate its enormous ATC system infrastructure by
the year 2000. In doing so, it chose to acquire new ATC systems by
contracting for systems development services from vendors rather than
building new ATC systems in-house.
This ambitious modernization program includes the acquisition of new
surveillance, data processing, navigation, and communication
equipment in addition to new facilities and support equipment.
Totaling over 200 separate projects, the modernization is estimated
to cost over $34 billion through the year 2003. Software-intensive
ATC systems make up a large portion of this total, accounting for 169
projects costing $20.7 billion. The Congress will have provided FAA
with approximately $14.7 billion of the $20.7 billion through fiscal
year 1997. Many of these projects, for example the Display System
Replacement and the Voice Switching and Control System, each involve
the acquisition of over a million lines of code. Moreover, because
the software must operate in a real-time environment in which human
life is at stake, it must be fault tolerant, meaning that it must be
able to monitor its own execution and recover from failures without
losing any data.
Over the past 15 years, FAA's modernization projects have experienced
substantial cost overruns, lengthy schedule delays, and significant
performance shortfalls. To illustrate, the long-time centerpiece of
this modernization program--the Advanced Automation System (AAS)--was
restructured in 1994 after estimated costs tripled from $2.5 billion
to $7.6 billion and delays in putting significantly
less-than-promised system capabilities into operation were expected
to run 8 years or more over original estimates. Similarly, increases
in costs for three other ATC projects\4 have ranged from 51 to 511
percent, and schedule delays have averaged almost 4 years. For
example, the per-unit cost estimate for the Voice Switching and
Control System increased 511 percent, and the first site
implementation was delayed 6 years from the original estimate.
AAS and other ATC projects have also experienced shortfalls in
performance. For example, the critical Initial Sector Suite System
component of AAS, which was intended to replace controllers'
workstations at en route centers, faced so many technical problems
that its functionality was severely scaled back. In addition,
difficulties in developing the Air Route Surveillance Radar-4
software and integrating it with other ATC systems delayed its
implementation for years.
--------------------
\4 The three projects and their respective percentage increase in
unit costs are the Voice Switching and Control System (511 percent),
the Integrated Terminal Weather System (129 percent), and the
Aviation Weather Observing System (51 percent).
ATC MODERNIZATION NOW
PROCEEDING UNDER A NEW
ACQUISITION MANAGEMENT SYSTEM
---------------------------------------------------------- Chapter 1:3
Because of FAA's contention that many of its modernization problems
were rooted in the Federal Acquisition System, the Congress enacted
legislation in October 1995 that exempted FAA from most federal
procurement and personnel laws and regulations and directed FAA to
develop and implement a new acquisition system that would address the
unique needs of the agency.\5 At a minimum, the system was to provide
for more timely and cost-effective acquisitions. On April 1, 1996,
in response to the Act, the FAA Administrator began implementation of
a new acquisition management system.
The new system is intended to reduce the time and cost to field new
products and services by introducing a new investment management
system that spans the investments' entire life cycles, a new
procurement system that provides flexibility in selecting and
managing contractors, and organizational learning and cultural reform
that supports the new investment management and procurement systems.
This high-level policy promulgated by the new acquisition management
system is intended to be supplemented by guidelines in three areas:
software/systems acquisition, facilities acquisition, and services
acquisition. These guidelines will be available to FAA staff via the
Internet and were scheduled to be online by October 1, 1996. As of
February 1, 1997, these guidelines were still in draft form and not
available to FAA staff.
--------------------
\5 Department of Transportation and Related Agencies Appropriations
Act 1996, P.L. No. 104-50, sec. 348, 109 Stat. 436, 460 (1995).
FAA ORGANIZATIONS RESPONSIBLE
FOR ATC SYSTEMS ACQUISITION AND
MAINTENANCE
---------------------------------------------------------- Chapter 1:4
Two major FAA organizations play key roles in the modernization of
ATC systems--the Office of the Associate Administrator for Research
and Acquisitions (ARA) and the Office of the Associate Administrator
for Air Traffic Services (ATS). Briefly, ARA is responsible for
acquiring ATC systems, while ATS is responsible for operating and
maintaining ATC systems. Cross-functional integrated product teams
(IPT) residing in ARA are responsible for specific ATC system
acquisition projects.
ARA manages ATC modernization research and development and
acquisition activities. According to the Associate Administrator for
ARA, only about one-half of the total ATC systems development budget
is spent by ARA, while the other one-half is spent by ATS
implementing system enhancements. Within ARA, two groups are
responsible for acquiring systems, while other groups handle
cross-cutting management functions, such as budget formulation and
program evaluation. These two groups are the Office of Air Traffic
Systems Development (AUA) and the Office of Communication,
Navigation, and Surveillance Systems (AND).
Five IPTs reside in AUA and are organized by ATC business areas
(i.e., en route, terminal, weather and flight service, air traffic
management, oceanic), and five IPTs reside in AND and are organized
by ATC functional areas (i.e., infrastructure, communications,
surveillance, Global Positioning System/navigation,
aircraft/avionics). IPTs are responsible for research, development,
and acquisition as well as for ensuring that new equipment is
delivered, installed, and working properly. For example, the en
route IPT comprises product teams for the Display Channel Complex
Rehost, the Display System Replacement, the Voice Switching and
Control System, and several other en route systems. Each IPT
includes systems and specialty engineers, logistics personnel,
testing personnel, contract personnel, and lawyers as well as
representatives from the organizations responsible for operating and
maintaining the ATC equipment. The organization chart below shows
the structure of the ARA organization.
Figure 1.2: ARA Organization
Chart
(See figure in printed
edition.)
OBJECTIVES, SCOPE, AND
METHODOLOGY
---------------------------------------------------------- Chapter 1:5
The Chairman, Subcommittee on Transportation and Related Agencies,
House Committee on Appropriations, requested that we review FAA's
ability to acquire software for ATC systems. Our objectives were to
determine (1) the maturity of FAA's ATC modernization software
acquisition processes; and (2) the steps/actions FAA has underway or
planned to improve these processes, and any obstacles that may impede
FAA's improvement actions.
To determine FAA's software acquisition process maturity, we applied
the Software Engineering Institute's Software Acquisition Capability
Maturity Model (SA-CMM)\6 and its Software Capability Evaluation
(SCE) method. SEI's expertise in software process assessment is
accepted throughout the industry. Our evaluators were all
SEI-trained software specialists. In addition, we employed SEI
consultants, two of whom are authors of the model, as advisors to
ensure proper application of the model.
The SA-CMM ranks organizational maturity according to five levels
(see figure 1.3). Maturity levels 2 through 5 require the verifiable
existence and use of certain software acquisition processes, known as
key process areas (KPA). According to the SEI, an agency that has
these acquisition processes in place is in a much better position to
successfully acquire software than an organization that does not have
these processes in place. We evaluated FAA's software acquisition
processes against all level 2 KPAs and one level 3 KPA (see table
1.1). We included one level 3 KPA--acquisition risk
management--because it is considered by software experts to be a very
important process area.
Figure 1.3: SA-CMM Levels and
Descriptions
(See figure in printed
edition.)
Table 1.1
SA-CMM KPAs Used to Assess FAA Software
Acquisition Maturity
SA-CMM Level 2 Key
process areas Description
------------------ --------------------------------------------------
Software Ensuring that reasonable planning for the software
acquisition acquisition is conducted and that all elements of
planning the project are included.
Solicitation Ensuring that award is made to the contractor most
capable of satisfying the specified requirements.
Requirements Establishing a common and unambiguous definition
development and of software acquisition requirements understood by
management the acquisition team, system user, and the
contractor.
Project office Managing the activities of the project office and
management supporting contractor(s) to ensure a timely,
efficient, and effective software acquisition.
Contract tracking Ensuring that the software activities under
and oversight contract are being performed in accordance with
contract requirements, and that products and
services will satisfy contract requirements.
Evaluation Determining that the acquired software products
and services satisfy contract requirements prior
to acceptance and transition to support.
Transition and Providing for the transition of the software
support products being acquired to their eventual support
organization.
CMM Level 3 Description
Key process area
Acquisition risk Identifying risks as early as possible, adjusting
management acquisition strategy to mitigate those risks, and
developing and implementing a risk management
process as an integral part of the acquisition
process.
----------------------------------------------------------------------
As established by the model, each KPA contains five common attributes
that indicate whether the implementation and institutionalization of
a KPA can be effective, repeatable, and lasting. The five common
features are:
-- Commitment to perform. Commitment to perform describes the
actions that the organization must take to establish the process
and ensure that it can endure. Commitment to perform typically
involves establishing organizational policies and sponsorship.
-- Ability to perform. Ability to perform describes the
preconditions that must exist in the project or organization to
implement the software acquisition process competently. Ability
to perform typically involves resources, organizational
structures, and training.
-- Activities performed. Activities performed describes the roles
and procedures necessary to implement a KPA. Activities
performed typically involve establishing plans and procedures,
performing the work, tracking it, and taking appropriate
management actions.
-- Measurement and analysis. Measurement and analysis describes
activities performed to measure the process and analyze the
measurements. Measurement and analysis typically includes
defining the measurements to be taken and the analyses to be
conducted to determine the status and effectiveness of the
activities performed.
-- Verifying implementation. Verifying implementation describes
the steps to ensure that the activities are performed in
compliance with the process that has been established.
Verification typically encompasses reviews by management.
In accordance with SEI's SCE method, for each KPA in level 2 and the
one KPA in level 3 (risk management), we evaluated institutional FAA
policies and practices and compared project-specific guidance and
practices against the five common attributes. This project-specific
comparison can result in one of four possible outcomes: (1) project
strength--an effective implementation of the key practice; (2)
project weakness--ineffective implementation of a key practice or
failure to implement a key practice; (3) project observation--key
practice evaluated but evidence inconclusive and cannot be
characterized as either strength or weakness; and (4) not rated--key
practice not currently relevant to project, therefore not evaluated.
We performed the project-specific evaluations on five ongoing ATC
modernization projects, each of which is described below. We asked
FAA to choose these projects using the following criteria: (1) the
projects are major efforts with large software acquisition
components; (2) the projects are managed by different integrated
product teams, (3) the projects are in different stages of their life
cycles, and (4) the projects are among FAA's best-managed
acquisitions. The projects that FAA selected for our evaluation are:
-- Automated Radar Terminal System (ARTS) IIIE: ARTS gathers
information from surveillance sensors, processes it, and sends
it to air traffic controllers in terminal radar approach control
facilities and control towers at airports. A series of
improvements to ARTS have provided increased processor capacity
and the ability to support a greater number of controller
displays. The ARTS IIIE improvements provide for more
controller positions and surveillance sensor inputs at selected
large facilities. ARTS IIIE is operational at New York,
Chicago, and Dallas/Fort Worth with additional systems planned
for Southern California and Denver. FAA estimates that the
enhancement will cost $383.8 million to develop and deploy.
-- Display System Replacement (DSR): DSR is intended to replace
air traffic controllers' display-related systems in each of the
en route centers. DSR consists of controller work stations
connected via a local area network to three interfacing systems
(Host Computer System, Enhanced Direct Access Radar Channel, and
Weather and Radar Processor). FAA plans to deploy DSR to all 20
en route centers in the continental United States, as well as
ATC facilities in Anchorage and potentially in Honolulu. FAA is
now conducting system acceptance testing. FAA estimates that
DSR will cost $1,055.3 million to develop and deploy.
-- National Airspace System (NAS) Infrastructure Management System
(NIMS): NIMS is intended to provide the system infrastructure,
including data architecture and network communications, to
permit remote ATC system operational monitoring and maintenance.
This program will provide a three-tiered architecture consisting
of a national control center, 4 to 10 operational control
centers, and over 300 local work centers. NIMS is in the
pre-solicitation phase, and FAA estimates that it will cost
about $500 million to develop and deploy.
-- Voice Switching and Control System (VSCS): VSCS is intended to
provide air-to-ground and ground-to-ground communications
between en route centers and aircraft. The VSCS is to replace
the aging ground-to-ground switching equipment and the
air-to-ground circuits with a single integrated,
computer-controlled, digital voice switching system. The
development of VSCS is completed and all systems are
operational. FAA estimates that VSCS will cost about $1.5
billion to develop and deploy.
-- Weather and Radar Processor (WARP): WARP is a next generation
weather and radar processing and display system that is intended
to permit consolidation of weather data from several sources
into a single, integrated display for controllers. Currently,
the weather information provided to controllers in the en route
centers comes from long-range aircraft surveillance radars,
which are not well-suited for this purpose. Next generation
weather radars are to replace the surveillance radars as the
source of weather information. WARP is to collect, process, and
disseminate this and other weather data to controllers, traffic
management specialists, pilots, and meteorologists. WARP is
currently under development, and FAA estimates that it will cost
$124.6 million to develop and deploy.
To address our second objective (what steps/actions FAA has underway
or planned to improve its software acquisition processes and what
obstacles, if any, may impede FAA's progress), we interviewed FAA's
Chief Scientist for Software Engineering and his staff to determine:
(1) process improvements that are planned and underway; (2) the
rationale for each initiative; (3) the relative priority of each; (4)
progress made on each initiative; and (5) obstacles, if any, impeding
progress. We also analyzed process improvement plans, meeting
minutes, and related documentation to further address these areas.
Finally, we interviewed representatives from the ATC modernization
product teams and the SEPG to obtain their perspectives in assessing
process improvement support, activities, progress, and obstacles.
The Department of Transportation provided written comments on a draft
of this report. These comments are presented and evaluated in
chapter 11, and are reprinted in appendix I. We performed our work
at FAA headquarters offices in Washington, D.C. between March 1996
and February 1997, in accordance with generally accepted government
auditing standards.
--------------------
\6 We used a draft version of the model for our evaluation (version
00.03, dated April 1996). The first published version of the model
was released in October 1996, after we performed our evaluation.
According to the model's authors, the published version differed only
editorially from the draft we used.
SOFTWARE ACQUISITION PLANNING
============================================================ Chapter 2
The purpose of software acquisition planning is to ensure that
reasonable planning for the software acquisition is conducted and
that all aspects of the total software acquisition effort are
included in these plans. According to the SA-CMM, a repeatable
software acquisition planning process, among other things, includes
(1) addressing software life cycle support in acquisition plans, (2)
preparing life cycle software cost estimates, (3) having a written
software acquisition policy, (4) measuring and reporting on the
status of software acquisition planning activities, and (5) having
guidance on software training and experience requirements for project
personnel.
FAA'S SOFTWARE ACQUISITION
PLANNING PROCESS IS NOT
EFFECTIVE
---------------------------------------------------------- Chapter 2:1
All five projects have some ability and/or activity strengths in this
KPA. For example, every project addresses software life cycle
support in planning documents and software life cycle cost estimates
were prepared for four of the projects. However, we found many more
process weaknesses than strengths. For example, FAA has a systems
acquisition policy, but the policy does not specifically address or
provide guidance on software acquisition. Therefore, FAA management
has not formally recognized the importance and uniqueness of software
acquisition issues in the system acquisition process, and has not
formally committed to managing software acquisition in a disciplined
manner. Also, the product teams do not measure or report on the
status of software acquisition planning activities. As a result,
management is not always aware of problems in project performance,
and cannot always take corrective action expeditiously.
Additionally, none of the five projects has specific guidance on
software training or experience requirements for project
participation. As a result, software training is ad hoc, and
decisions about project personnel assignments are subjective.
Figure 2.1 provides a comprehensive listing of the five projects'
strengths, weaknesses, and observations for the software acquisition
planning KPA. The specific findings supporting the practice ratings
cited in figure 2.1 are in tables 2.1 through 2.5.
Figure 2.1: Software
Acquisition Planning
(See figure in printed
edition.)
Table 2.1
Software Acquisition Planning Findings
for ARTSIIIE
Automated Radar Terminal System IIIE
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition The system acquisition Weakness
organization has a policy does not
written policy for adequately address
planning the software software, e.g., it does
acquisition. not address items that
should be included in
software planning such
as contract tracking and
oversight, requirements
development, evaluation,
and risk management.
Ability 1 Personnel are assigned No personnel are Weakness
the responsibility for assigned the
performing software responsibility for
acquisition planning. software acquisition
planning.
Ability 2 The acquisition The acquisition Observation
organization has organization has no
experienced software guidance regarding
acquisition management training or experience
personnel. requirements for project
participation.
Ability 3 Software acquisition There are no guidelines Weakness
management personnel are that define domain
experienced in the knowledge or experience.
domain of the project.
Activity 1 Software acquisition No one on the product Weakness
planning personnel are team is specifically
involved in system assigned responsibility
acquisition planning. for software
acquisition.
Activity 2 The project's software There is no documented Weakness
acquisition planning is software acquisition
documented and the plan.
planning documentation
is maintained over the
life of the project.
Activity 3 The software acquisition There is no software Weakness
strategy for the project acquisition strategy.
is developed.
Activity 4 Software acquisition The product team ensures Strength
planning includes that life cycle support
provisions for ensuring is included in planning
that the life cycle documentation.
support of the system is
included in planning
documentation.
Activity 5 A life cycle cost The life cycle cost Weakness
estimate for the estimate was prepared
software activity is but not independently
prepared and verified.
independently verified.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
software acquisition the status of activities
planning activities. for any of the key
process areas.
Verification The activities for While the Integrated Weakness
1 software acquisition Product Team leader
planning are reviewed by reviews the status of
acquisition organization the contract and the
management on a periodic contractor's cost and
basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 software acquisition leader reviews the
planning are reviewed by status of the contract
the project manager on and the contractor's
both a periodic and cost and schedule, he
event-driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 2.2
Software Acquisition Planning Findings
for DSR
Display System Replacement
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition The system acquisition Weakness
organization has a policy does not
written policy for adequately address
planning the software software, e.g., it does
acquisition. not address items that
should be included in
software planning such
as contract tracking and
oversight, requirements
development, evaluation,
and risk management.
Ability 1 Personnel are assigned Personnel are assigned Strength
the responsibility for the responsibility for
performing software performing software
acquisition planning. acquisition planning.
Ability 2 The acquisition The acquisition Observation
organization has organization has no
experienced software guidance regarding
acquisition management training or experience
personnel. requirements for project
participation.
Ability 3 Software acquisition There are no guidelines Weakness
management personnel are that define domain
experienced in the knowledge or experience.
domain of the project.
Activity 1 Software acquisition Software acquisition Strength
planning personnel are personnel are involved
involved in system in system acquisition
acquisition planning. planning.
Activity 2 The project's software There is no software Weakness
acquisition planning is acquisition planning
documented and the documentation.
planning documentation
is maintained over the
life of the project.
Activity 3 The software acquisition Officials stated that Weakness
strategy for the project the software acquisition
is developed. strategy is developed,
however, the documents
provided did not address
such things as
objectives,
technologies, and
schedule.
Activity 4 Software acquisition Software acquisition Strength
planning includes planning includes life
provisions for ensuring cycle support planning.
that the life cycle
support of the system is
included in planning
documentation.
Activity 5 A life cycle cost A life cycle cost Strength
estimate for the estimate is prepared and
software activity is independently verified.
prepared and
independently verified.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
software acquisition the status of activities
planning activities. for any of the key
process areas.
Verification The activities for While the Integrated Weakness
1 software acquisition Product Team leader
planning are reviewed by reviews the status of
acquisition organization the contract and the
management on a periodic contractor's cost and
basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 software acquisition leader reviews the
planning are reviewed by status of the contract
the project manager on and the contractor's
both a periodic and cost and schedule, he
event-driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 2.3
Software Acquisition Planning Findings
for NIMS
NAS Infrastructure Management System
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition The system acquisition Weakness
organization has a policy does not
written policy for adequately address
planning the software software, e.g., it does
acquisition. not address items that
should be included in
software planning such
as contract tracking and
oversight, requirements
development, evaluation,
and risk management.
Ability 1 Personnel are assigned The team members are Strength
the responsibility for assigned the
performing software responsibility for
acquisition planning. software acquisition
planning.
Ability 2 The acquisition The acquisition Observation
organization has organization has no
experienced software guidance regarding
acquisition management training or experience
personnel. requirements for project
participation.
Ability 3 Software acquisition There are no guidelines Weakness
management personnel are that define domain
experienced in the knowledge or experience.
domain of the project.
Activity 1 Software acquisition The team members for Strength
planning personnel are software acquisition are
involved in system assigned collective
acquisition planning. responsibility and are
actively involved in
system acquisition
planning.
Activity 2 The project's software At this early stage in Observation
acquisition planning is the program, the
documented and the software acquisition
planning documentation planning documentation
is maintained over the is being written but is
life of the project. not complete.
Activity 3 The software acquisition Officials stated that Strength
strategy for the project the software acquisition
is developed. strategy will be
contained within the
acquisition plan.
Activity 4 Software acquisition Software acquisition Strength
planning includes planning includes
provisions for ensuring provisions for ensuring
that the life cycle that life cycle support
support of the system is is included in planning
included in planning documentation.
documentation.
Activity 5 A life cycle cost A life cycle cost Strength
estimate for the estimate for software
software activity is has been prepared and
prepared and independently verified.
independently verified.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the to determine the status
software acquisition of activities for any of
planning activities. the key process areas.
Verification The activities for While the Integrated Weakness
1 software acquisition Product Team leader
planning are reviewed by reviews the status of
acquisition organization the contract and the
management on a periodic contractor's cost and
basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 software acquisition leader reviews the
planning are reviewed by status of the contract
the project manager on and the contractor's
both a periodic and cost and schedule, he
event-driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 2.4
Software Acquisition Planning Findings
for VSCS
Voice Switching and Control System
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition The system acquisition Weakness
organization has a policy does not
written policy for adequately address
planning the software software, e.g., it does
acquisition. not address items that
should be included in
software planning such
as contract tracking and
oversight, requirements
development, evaluation,
and risk management.
Ability 1 Personnel are assigned Personnel are assigned Strength
the responsibility for the responsibility for
performing software performing software
acquisition planning. acquisition planning.
Ability 2 The acquisition The acquisition Observation
organization has organization has no
experienced software guidance regarding
acquisition management training or experience
personnel. requirements for project
participation.
Ability 3 Software acquisition There are no guidelines Weakness
management personnel are that define domain
experienced in the knowledge or experience.
domain of the project.
Activity 1 Software acquisition Software acquisition Strength
planning personnel are personnel are involved
involved in system in system acquisition
acquisition planning. planning.
Activity 2 The project's software The project's software Weakness
acquisition planning is acquisition planning is
documented and the not documented.
planning documentation
is maintained over the
life of the project.
Activity 3 The software acquisition No software acquisition Weakness
strategy for the project strategy exists. The
is developed. system acquisition
strategy does not
address software.
Activity 4 Software acquisition The life cycle support Strength
planning includes of the system is
provisions for ensuring included in the
that the life cycle acquisition planning
support of the system is documentation.
included in planning
documentation.
Activity 5 A life cycle cost The life cycle cost Strength
estimate for the estimate has been
software activity is prepared and
prepared and independently verified.
independently verified.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the or used to determine the
software acquisition status of activities for
planning activities. any of the key process
areas.
Verification The activities for While the Integrated Weakness
1 software acquisition Product Team leader
planning are reviewed by reviews the status of
acquisition organization the contract and the
management on a periodic contractor's cost and
basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 software acquisition leader reviews the
planning are reviewed by status of the contract
the project manager on and the contractor's
both a periodic and cost and schedule, he
event-driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 2.5
Software Acquisition Planning Findings
for WARP
Weather and Radar Processor
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition The system acquisition Weakness
organization has a policy does not
written policy for adequately address
planning the software software, e.g., it does
acquisition. not address items that
should be included in
software planning such
as contract tracking and
oversight, requirements
development, evaluation,
and risk management.
Ability 1 Personnel are assigned Although the product Observation
the responsibility for team stated that they
performing software are assigned collective
acquisition planning. responsibility for
systems acquisition,
they could not provide
documentation to show a
specific assignment for
software acquisition.
Ability 2 The acquisition The acquisition Observation
organization has organization has no
experienced software guidance regarding
acquisition management training or experience
personnel. requirements for project
participation.
Ability 3 Software acquisition There are no guidelines Weakness
management personnel are that define domain
experienced in the knowledge or experience.
domain of the project.
Activity 1 Software acquisition Although the product Weakness
planning personnel are team is responsible for
involved in system systems acquisition,
acquisition planning. there is no one
specifically assigned
for software nor does
any document expressly
state that software is
part of systems
acquisition.
Activity 2 The project's software There is no software Weakness
acquisition planning is acquisition plan.
documented and the
planning documentation
is maintained over the
life of the project.
Activity 3 The software acquisition There is no software Weakness
strategy for the project acquisition strategy for
is developed. the project. The system
acquisition strategy
covers only software
enhancements.
Activity 4 Software acquisition The acquisition plan Strength
planning includes includes provisions for
provisions for ensuring ensuring that life cycle
that the life cycle support is included.
support of the system is
included in planning
documentation.
Activity 5 A life cycle cost The life cycle cost Strength
estimate for the estimate is prepared and
software activity is independently verified.
prepared and
independently verified.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
software acquisition the status of activities
planning activities. for any of the key
process areas.
Verification The activities for While the Integrated Weakness
1 software acquisition Product Team leader
planning are reviewed by reviews the status of
acquisition organization the contract and the
management on a periodic contractor's cost and
basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 software acquisition leader reviews the
planning are reviewed by status of the contract
the project manager on and the contractor's
both a periodic and cost and schedule, he
event-driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
CONCLUSIONS
---------------------------------------------------------- Chapter 2:2
Effective planning is the cornerstone of successful software
acquisition. While FAA showed some strengths in this KPA, its many
weaknesses render the software acquisition planning capability ad hoc
and chaotic. Therefore, it is unlikely that projects are effectively
measuring and monitoring software acquisition progress and taking
corrective actions expeditiously.
SOLICITATION
============================================================ Chapter 3
The purpose of solicitation is to prepare a request for proposal that
delineates a project's software-related requirements, and select a
contractor that can most cost-effectively satisfy these requirements,
while complying with relevant solicitation laws and regulations.
According to the SA-CMM, specific requirements for a repeatable
solicitation process include, among other things, (1) having and
following a solicitation plan, (2) assigning responsibility and
ensuring sufficient resources for coordinating and conducting
solicitation activities, (3) preparing and reviewing cost and
schedule estimates for the software products and services being
acquired, and (4) periodically measuring solicitation work completed
and effort and funds expended, comparing these measures to plans, and
reporting the results to management.
PRODUCT TEAMS PERFORMING MANY
SOLICITATION PRACTICES
---------------------------------------------------------- Chapter 3:1
All five projects have strengths in many of the practices required by
this KPA. For example, most projects have written solicitation
plans, assign responsibility for coordinating and conducting the
solicitation activities, and prepare and review contract-related
software cost and schedule estimates.
However, the projects are weak in several areas. For example, even
though most projects had a written solicitation plan, not all
projects followed their plans. Also, none of the projects adequately
identified the resources needed to conduct solicitation activities.
While FAA personnel stated that they had adequate solicitation
resources, they provided no evidence of either a mechanism for
identifying required resources or for ensuring that required
resources are provided. These weaknesses increase the risk of FAA
not adequately evaluating the offerors' proposals, and making a
suboptimal selection. Additionally, none of the five measured or
reported on the status of product team solicitation activities. As a
result, management cannot identify solicitation problems early and
resolve them expeditiously.
Figure 3.1 provides a comprehensive listing of the five projects'
strengths, weaknesses, and observations for the solicitation KPA.
The specific findings supporting the practice ratings cited in figure
3.1 are in tables 3.1 through 3.5.
Figure 3.1: Solicitation
Summary
(See figure in printed
edition.)
(See figure in printed
edition.)
Table 3.1
Solicitation Findings for ARTSIIIE
Automated Radar Terminal System IIIE
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition FAA Order 1810.1F is the Strength
organization has a written policy.
written policy for the
conduct of the
solicitation.
Commitment 2 Responsibility for the Officials gave Weakness
software portion of the conflicting answers as
solicitation is to who is responsible
designated. for the software portion
of the solicitation, and
could not provide a
document that formally
designates
responsibility.
Commitment 3 A selection official has The Administrator was Strength
been designated to be the selection official
responsible for the for the sole-source
selection process and contract.
the decision.
Ability 1 A group that is A group (matrix team) is Strength
responsible for responsible for
coordinating and coordinating and
conducting the conducting the
solicitation activities solicitation activities.
exists.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for the identifying resources
solicitation activities. required and for
ensuring that the needed
resources are provided
to the project.
Ability 3 Individuals performing The acquisition Observation
the solicitation organization has no
activities have guidance regarding
experience or receive training or experience
training. requirements for project
participation.
Activity 1 The project team The team documents its Strength
documents its plans for plans for solicitation
solicitation activities. activities.
Activity 2 The project's While officials stated Observation
solicitation activities that solicitation
are performed in activities are performed
accordance with its in accordance with its
plans. plans, they could not
provide documentation to
support this.
Activity 3 The project team The team does not Weakness
documents its plans for document its plans for
proposal evaluation proposal evaluation
activities. activities.
Activity 4 The project team's No evaluation plan Weakness
proposal evaluation exists, therefore, the
activities are performed team could not perform
in accordance with its in accordance with its
plans. plan.
Activity 5 A cost estimate and A cost estimate and Strength
schedule for the schedule were prepared.
software activity being
sought are prepared.
Activity 6 The software cost The cost estimate and Strength
estimate and schedule schedule were
are independently independently reviewed.
reviewed for
comprehensiveness and
realism.
Activity 7 The groups supporting No orientation briefing Weakness
the solicitation (e.g., occurred.
end user, systems
engineering, support
organization, and
application domain
experts) receive
orientation on the
solicitation's
objectives and
procedures.
Activity 8 The project team and the Officials stated that Observation
offeror review the meetings were held with
project's software the contractor to ensure
requirements during mutual understanding,
negotiations to ensure however, they could not
mutual understanding. provide documents to
support this.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
solicitation activities. the status of activities
for any of the key
process areas.
Verification The activities for While the Integrated Weakness
1 solicitation are Product Team leader
reviewed by the reviews the status of
designated selection the contract and the
official or acquisition contractor's cost and
organization management schedule, he does not
on a periodic basis. review the status of the
activities that are
required to be performed
for any of the various
key process areas.
Verification The activities for While the product team Weakness
2 solicitation are leader reviews the
reviewed by the project status of the contract
manager on both a and the contractor's
periodic and event- cost and schedule, he
driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 3.2
Solicitation Findings for DSR
Display System Replacement
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition There is an FAA policy Strength
organization has a that addresses
written policy for the solicitation conduct.
conduct of the
solicitation.
Commitment 2 Responsibility for the Officials stated that Observation
software portion of the responsibility for the
solicitation is software portion of the
designated. solicitation has been
assigned, however, they
could not provide
documents to support
this.
Commitment 3 A selection official has Not applicable. DSR was Not rated
been designated to be a change order from an
responsible for the existing larger contract
selection process and that went through the
the decision. acquisition phase in
1984. Current team
members joined the team
after the change order
was negotiated and,
therefore, could not
address this key
practice.
Ability 1 A group that is A group responsible for Strength
responsible for the solicitation exists.
coordinating and
conducting the
solicitation activities
exists.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for the identifying resources
solicitation activities. required and for
ensuring that the needed
resources are provided
to the project.
Ability 3 Individuals performing The acquisition Observation
the solicitation organization has no
activities have guidance regarding
experience or receive training or experience
training. requirements for project
participation.
Activity 1 The project team Not applicable. DSR was Not rated
documents its plans for a change order from an
solicitation activities. existing larger contract
that went through the
acquisition phase in
1984. Current team
members joined the team
after the change order
was negotiated and,
therefore, could not
address this key
practice.
Activity 2 The project's Not applicable. DSR was Not rated
solicitation activities a change order from an
are performed in existing larger contract
accordance with its that went through the
plans. acquisition phase in
1984. Current team
members joined the team
after the change order
was negotiated and,
therefore, could not
address this key
practice.
Activity 3 The project team The product team uses a Strength
documents its plans for change order evaluation
proposal evaluation plan to document plans
activities. for proposal evaluation
activities.
Activity 4 The project team's Not applicable. DSR was Not rated
proposal evaluation a change order from an
activities are performed existing larger contract
in accordance with its that went through the
plans. acquisition phase in
1984. Current team
members joined the team
after the change order
was negotiated and,
therefore, could not
address this key
practice.
Activity 5 A cost estimate and A cost estimate and Strength
schedule for the schedule were generated.
software activity being
sought are prepared.
Activity 6 The software cost Officials could not Observation
estimate and schedule produce documentation
are independently that supported their
reviewed for statement that the
comprehensiveness and software cost estimate
realism. and schedule were
independently reviewed.
Activity 7 The groups supporting Not applicable. DSR was Not rated
the solicitation (e.g., a change order from an
end user, systems existing larger contract
engineering, support that went through the
organization, and acquisition phase in
application domain 1984. Current team
experts) receive members joined the team
orientation on the after the change order
solicitation's was negotiated and,
objectives and therefore, could not
procedures. address this key
practice.
Activity 8 The project team and the A series of scheduled Strength
offeror review the meetings were held to
project's software ensure mutual
requirements during understanding of
negotiations to ensure requirements.
mutual understanding.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
solicitation activities. the status of activities
for any of the key
process areas.
Verification The activities for While the Integrated Weakness
1 solicitation are Product Team leader
reviewed by the reviews the status of
designated selection the contract and the
official or acquisition contractor's cost and
organization management schedule, he does not
on a periodic basis. review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 solicitation are leader reviews the
reviewed by the project status of the contract
manager on both a and the contractor's
periodic and event- cost and schedule, he
driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 3.3
Solicitation Findings for NIMS
NAS Infrastructure Management System
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition The Acquisition Strength
organization has a Management System is the
written policy for the written policy.
conduct of the
solicitation.
Commitment 2 Responsibility for the Responsibility for the Strength
software portion of the software portion of the
solicitation is solicitation has been
designated. designated to software
experts on the team.
Commitment 3 A selection official has A selection official has Strength
been designated to be been designated to be
responsible for the responsible for the
selection process and selection process and
the decision. decision.
Ability 1 A group that is The contracting officer, Strength
responsible for support staff, and the
coordinating and parent ASU organization
conducting the are responsible for
solicitation activities coordinating and
exists. conducting the
solicitation activities.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for the identifying resources
solicitation activities. required and for
ensuring that the needed
resources are provided
to the project.
Ability 3 Individuals performing The acquisition Observation
the solicitation organization has no
activities have guidance regarding
experience or receive training or experience
training. requirements for project
participation.
Activity 1 The project team The product team Strength
documents its plans for documents its plans for
solicitation activities. solicitation activities.
Activity 2 The project's Officials stated that Strength
solicitation activities solicitation activities
are performed in will be performed in
accordance with its accordance with its
plans. plans. NIMS is in the
presolicitation phase.
Activity 3 The project team The project team has Strength
documents its plans for documented its plans for
proposal evaluation proposal evaluation
activities. activities.
Activity 4 The project team's In accordance with the Strength
proposal evaluation plan, prequalification
activities are performed was completed and
in accordance with its vendors were down-
plans. selected from it.
Activity 5 A cost estimate and The Acquisition Program Strength
schedule for the Baseline includes a cost
software activity being estimate and schedule
sought are prepared. for the software
acquisition.
Activity 6 The software cost An independent Strength
estimate and schedule assessment was done.
are independently
reviewed for
comprehensiveness and
realism.
Activity 7 The groups supporting Solicitation activities Strength
the solicitation (e.g., orientation was
end user, systems conducted for NIMS
engineering, support personnel.
organization, and
application domain
experts) receive
orientation on the
solicitation's
objectives and
procedures.
Activity 8 The project team and the The NIMS project has not Not rated
offeror review the yet reached this stage,
project's software therefore, this activity
requirements during was not rated.
negotiations to ensure
mutual understanding.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the to determine the status
solicitation activities. of activities for any of
the key process areas.
Verification The activities for While the Integrated Weakness
1 solicitation are Product Team leader
reviewed by the reviews the status of
designated selection the contract and the
official or acquisition contractor's cost and
organization management schedule, he does not
on a periodic basis. review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 solicitation are leader reviews the
reviewed by the project status of the contract
manager on both a and the contractor's
periodic and event- cost and schedule, he
driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 3.4
Solicitation Findings for VSCS
Voice Switching and Control System
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition FAA Order 1810.1F is the Strength
organization has a written policy.
written policy for the
conduct of the
solicitation.
Commitment 2 Responsibility for the Officials gave Weakness
software portion of the conflicting answers as
solicitation is to who is responsible
designated. for software
acquisition, and could
not provide
documentation that
formally designates
responsibility.
Commitment 3 A selection official has The Administrator was Strength
been designated to be the selection official.
responsible for the
selection process and
the decision.
Ability 1 A group that is Officials stated that a Observation
responsible for group was responsible
coordinating and for coordinating and
conducting the conducting solicitation
solicitation activities activities; however,
exists. they could not provide
documentation to support
this claim.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for the identifying the
solicitation activities. resources required and
for ensuring that the
needed resources are
provided to the project.
Ability 3 Individuals performing The acquisition Observation
the solicitation organization has no
activities have guidance regarding
experience or receive training or experience
training. requirements for project
participation.
Activity 1 The project team Officials said that Observation
documents its plans for solicitation activities
solicitation activities. were documented in the
Acquisition Plan and
Source Evaluation Plan;
however, they could not
provide these documents.
Activity 2 The project's Solicitation activities Weakness
solicitation activities were not performed in
are performed in accordance with plans.
accordance with its
plans.
Activity 3 The project team Officials stated that Observation
documents its plans for proposal evaluation
proposal evaluation activities were
activities. documented; however,
they could not provide
the documentation.
Activity 4 The project team's Officials could not Weakness
proposal evaluation describe how or if
activities are performed proposal evaluation
in accordance with its activities were
plans. performed in accordance
with plans;
additionally, they could
not provide documents to
support this activity.
Activity 5 A cost estimate and A cost estimate and Strength
schedule for the schedule for the
software activity being software activity were
sought are prepared. developed.
Activity 6 The software cost The software cost Strength
estimate and schedule estimate and schedule
are independently were independently
reviewed for reviewed for
comprehensiveness and comprehensiveness and
realism. realism.
Activity 7 The groups supporting Officials stated that Weakness
the solicitation (e.g., orientation was held,
end user, systems however, the
engineering, support documentation provided
organization, and did not indicate that
application domain the product team
experts) receive received orientation on
orientation on the the solicitation
solicitation's objectives and
objectives and procedures.
procedures.
Activity 8 The project team and the Officials stated that Observation
offeror review the the product team and the
project's software offeror reviewed project
requirements during software requirements
negotiations to ensure during pre-award
mutual understanding. negotiations, however,
they could not provide
documentation to support
this.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the or used to determine the
solicitation activities. status of activities for
any of the key process
areas.
Verification The activities for While the Integrated Weakness
1 solicitation are Product Team leader
reviewed by the reviews the status of
designated selection the contract and the
official or acquisition contractor's cost and
organization management schedule, he does not
on a periodic basis. review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 solicitation are leader reviews the
reviewed by the project status of the contract
manager on both a and the contractor's
periodic and event- cost and schedule, he
driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 3.5
Solicitation Findings for WARP
Weather and Radar Processor
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition There is a written Strength
organization has a policy for solicitation.
written policy for the
conduct of the
solicitation.
Commitment 2 Responsibility for the Responsibility for the Strength
software portion of the solicitation has been
solicitation is designated.
designated.
Commitment 3 A selection official has Because there was only Not rated
been designated to be one offeror, this
responsible for the commitment was not
selection process and rated.
the decision.
Ability 1 A group that is A group is responsible Strength
responsible for for coordinating and
coordinating and conducting the
conducting the solicitation activities.
solicitation activities
exists.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for the identifying and for
solicitation activities. ensuring that the needed
resources are provided
to the project.
Ability 3 Individuals performing The acquisition Observation
the solicitation organization has no
activities have guidance regarding
experience or receive training or experience
training. requirements for project
participation.
Activity 1 The project team The project team Strength
documents its plans for documents its plans for
solicitation activities. solicitation activities.
Activity 2 The project's The project's Strength
solicitation activities solicitation activities
are performed in were performed in
accordance with its accordance with its
plans. plans.
Activity 3 The project team The team documented its Strength
documents its plans for plans for proposal
proposal evaluation evaluation activities.
activities.
Activity 4 The project team's Proposal evaluation Strength
proposal evaluation activities were
activities are performed performed in accordance
in accordance with its with the plan.
plans.
Activity 5 A cost estimate and An independent Strength
schedule for the government cost estimate
software activity being was prepared which
sought are prepared. included major milestone
data.
Activity 6 The software cost The software cost Strength
estimate and schedule estimate and schedule
are independently were independently
reviewed for reviewed.
comprehensiveness and
realism.
Activity 7 The groups supporting Officials stated that Observation
the solicitation (e.g., team members received
end user, systems orientation at the
engineering, support beginning of the
organization, and solicitation, however,
application domain they could not provide
experts) receive documentation to support
orientation on the this.
solicitation's
objectives and
procedures.
Activity 8 The project team and the Numerous negotiation Strength
offeror review the sessions were held.
project's software
requirements during
negotiations to ensure
mutual understanding.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
solicitation activities. the status of activities
for any of the key
process areas.
Verification The activities for While the Integrated Weakness
1 solicitation are Product Team leader
reviewed by the reviews the status of
designated selection the contract and the
official or acquisition contractor's cost and
organization management schedule, he does not
on a periodic basis. review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 solicitation are leader reviews the
reviewed by the project status of the contract
manager on both a and the contractor's
periodic and event- cost and schedule, he
driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
CONCLUSIONS
---------------------------------------------------------- Chapter 3:2
While FAA has many strengths in this KPA, systemic weaknesses in
areas including measurement and analysis and management verification
of practices, along with other project-specific weaknesses, render
this KPA non-repeatable and dependent upon the capabilities and
commitment of individual employees.
REQUIREMENTS DEVELOPMENT AND
MANAGEMENT
============================================================ Chapter 4
The purpose of requirements development and management is to
establish and maintain a common and unambiguous definition of
software requirements among the acquisition team, the system users,
and the software development contractor. This KPA involves two
subprocesses: (1) developing a baseline set of software-related
contractual requirements, and (2) managing these requirements and
changes to these requirements for the duration of the acquisition.
The SA-CMM specifies a number of requirements development and
management practices necessary to achieve a repeatable maturity
level. These include (1) having a written organizational policy for
establishing and managing requirements allocated to software; (2)
documenting plans for the development and management of requirements;
(3) having documented processes for requirements development,
including elicitation, analysis, and verification; (4) measuring and
reporting on the status of requirements development and management
activities to management; (5) appraising the impact on software of
system-level requirements changes; and (6) having a mechanism to
ensure that contractor-delivered work products meet specified
requirements.
REQUIREMENTS DEVELOPMENT AND
MANAGEMENT PROCESS IS NOT
EFFECTIVE
---------------------------------------------------------- Chapter 4:1
In the past, we have attributed ATC modernization problems, in part,
to FAA's failure to effectively manage requirements. For example, we
reported in 1994 that FAA did not adequately specify or effectively
control changes to the requirements of its Initial Sector Suite
System (ISSS) component of the Advanced Automation System.\1
Our evaluation of FAA's capability relative to this KPA's
requirements reiterates our earlier reported concerns in this area
and pinpoints specific weaknesses. For example, while FAA has a
policy on requirements development and management, this policy does
not address establishing and managing software requirements.
Further, product teams do not always document their requirements
development and management plans, and while two had a defined process
for controlling changes to existing, baselined requirements, they did
not have a documented process for developing new software
requirements, including requirements planning, elicitation, analysis,
or verification. Additionally, management does not oversee or verify
requirements development and management activities, which means that
management has no assurance that specified requirements are correct
and complete, and does not know when management action is warranted.
We also found some practice strengths. For example, most projects
(1) are assessing the impact on software requirements of system-level
requirements changes and (2) have a mechanism to ensure that
contractor-delivered work products and services satisfied specified
software requirements.
Figure 4.1 provides a comprehensive listing of the five projects'
strengths, weaknesses, and observations for the requirements
development and management KPA. The specific findings supporting the
practice ratings in figure 4.1 are in tables 4.1 through 4.5.
Figure 4.1: Requirements
Development and Management
Summary
(See figure in printed
edition.)
Table 4.1
Requirements Development and Management
Findings for ARTSIIIE
Automated Radar Terminal System IIIE
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition Officials stated that Weakness
organization has a FAA Order 1810.1F is the
written policy for written policy for
establishing and requirements development
managing the system and management, however,
requirements allocated it does not address
to software. software requirements.
Ability 1 A group that is A group responsible for Strength
responsible for requirements development
performing requirements and management exists.
development and
management activities
exists.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for identifying and for
requirements development ensuring that the needed
and management resources are provided
activities. to the project.
Ability 3 Individuals performing The acquisition Observation
requirements development organization has no
and management guidance regarding
activities have training or experience
experience or receive requirements for project
training. participation.
Activity 1 The project team The project team does Weakness
documents its plans for not document its plans
requirements development for requirements
and management development, planning,
activities. elicitation, analysis,
and verification.
Activity 2 The project's There is no requirements Weakness
requirements development development and
and management management plan,
activities are performed therefore, the
in accordance with its activities are not
plans. performed in accordance
with it.
Activity 3 The project team A requirements baseline, Strength
baselines the software which is under change
requirements and places control, was established
them under change prior to the release of
control early in the the solicitation.
project, but not later
than release of the
solicitation.
Activity 4 The project team The project team's Strength
appraises system appraisals of the impact
requirements change of system requirements
requests for their changes on software are
impact on software. documented.
Activity 5 The project team The project team's Strength
appraises software appraisal of software
requirements changes for requirements changes'
their impact on impact on performance,
performance, schedule, schedule, cost, and
cost, system capacities, system capacities is
supportability, and documented, but not the
architecture. impact on system
supportability or
architecture.
Activity 6 The project team There is a mechanism for Strength
maintains a requirements traceability of software
mechanism for requirements
traceability during the implementation.
software effort to
ensure requirements have
been included in the
implemented work
products and services.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
requirements development the status of activities
and management for any of the key
activities. process areas.
Verification The activities for While the Integrated Weakness
1 requirements development Product Team leader
and management are reviews the status of
reviewed by acquisition the contract and the
organization management contractor's cost and
(and the contractor) on schedule, he does not
a periodic basis. review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 requirements development leader reviews the
and management are status of the contract
reviewed by the project and the contractor's
manager on both a cost and schedule, he
periodic and event- does not review the
driven basis. status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 4.2
Requirements Development and Management
Findings for DSR
Display System Replacement
--------------------------------------------------------------------------------
Key Practice Finding Rating
--------- ---------------------------- ----------------------------- --------
Commitmen The acquisition organization Officials stated that FAA Weakness
t 1 has a written policy for Order 1810.1F is the written
establishing and managing policy for requirements
the system requirements development and management,
allocated to software. however, it does not address
software requirements.
Ability 1 A group that is responsible The group responsible for Strength
for performing requirements requirements development and
development and management management is the product
activities exists. team.
Ability 2 Adequate resources are Adequate resources are Strength
provided for requirements provided for requirements
development and management development and management
activities. activities.
Ability 3 Individuals performing The acquisition organization Observat
requirements development and has no guidance regarding ion
management activities have training or experience
experience or receive requirements for project
training. participation.
Activity The project team documents While processes exist for Weakness
1 its plans for requirements milestone review, these
development and management documents do not cover the
activities. activities to be performed
such as user involvement,
elicitation, and requirements
development.
Activity The project's requirements There is no requirements Weakness
2 development and management development and management
activities are performed in plan, therefore, the
accordance with its plans. activities are not performed
in accordance with it.
Activity The project team baselines Software requirements are Weakness
3 the software requirements baselined as part of the
and places them under change contract process, but not
control early in the explicitly prior to
project, but not later than solicitation.
release of the solicitation.
Activity The project team appraises The product team appraises Strength
4 system requirements change system requirements changes
requests for their impact on for their impact on software.
software.
Activity The project team appraises Software requirements were Weakness
5 software requirements not appraised for their
changes for their impact on impact on performance,
performance, schedule, cost, schedule, cost, system
system capacities, capacities, supportability,
supportability, and and architecture.
architecture.
Activity The project team maintains a The product team maintains a Strength
6 requirements mechanism for mechanism for requirements
traceability during the traceability.
software effort to ensure
requirements have been
included in the implemented
work products and services.
Measureme Measurements are made and No internal measurements are Weakness
nt 1 used to determine the status taken and used to determine
of the requirements the status of activities for
development and management any of the key process areas.
activities.
Verificat The activities for While the Integrated Product Weakness
ion 1 requirements development and Team leader reviews the
management are reviewed by status of the contract and
acquisition organization the contractor's cost and
management (and the schedule, he does not review
contractor) on a periodic the status of the activities
basis. that are required to be
performed for any of the key
process areas.
Verificat The activities for While the product team leader Weakness
ion 2 requirements development and reviews the status of the
management are reviewed by contract and the contractor's
the project manager on both cost and schedule, he does
a periodic and event-driven not review the status of the
basis. activities that are required
to be performed for any of
the key process areas.
--------------------------------------------------------------------------------
Table 4.3
Requirements Development and Management
Findings for NIMS
NAS Infrastructure Management System
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------ --------------------------- --------------------------- --------
Commitment 1 The acquisition Officials cited the Weakness
organization has a written Acquisition Management
policy for establishing and System and FAA Order
managing the system 1810.1F, but these do not
requirements allocated to address software
software. requirements.
Ability 1 A group that is responsible The team members are Strength
for performing requirements assigned collective
development and management responsibility for the
activities exists. requirements process.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for requirements identifying resources
development and management required and for ensuring
activities. that the needed resources
are provided to the
project.
Ability 3 Individuals performing The acquisition Observat
requirements development organization has no ion
and management activities guidance regarding training
have experience or receive or experience requirements
training. for project participation.
Activity 1 The project team documents The project has not reached Not
its plans for requirements the point where these rated
development and management activities are performed.
activities.
Activity 2 The project's requirements The project has not reached Not
development and management the point where these rated
activities are performed in activities are performed.
accordance with its plans.
Activity 3 The project team baselines It is too early in the Not
the software requirements project life to assess: the rated
and places them under software requirements have
change control early in the not been developed.
project, but not later than
release of the
solicitation.
Activity 4 The project team appraises It is too early in the Not
system requirements change project life to assess: no rated
requests for their impact software has been developed
on software. or specified.
Activity 5 The project team appraises It is too early in the Not
software requirements project life to assess: no rated
changes for their impact on software requirements have
performance, schedule, been developed or
cost, system capacities, specified.
supportability, and
architecture.
Activity 6 The project team maintains No traceability matrix of Not
a requirements mechanism software requirements has rated
for traceability during the been developed at this
software effort to ensure point in the project.
requirements have been
included in the implemented
work products and services.
Measurement Measurements are made and No internal process Weakness
1 used to determine the measurements are taken to
status of the requirements determine the status of
development and management activities for any of the
activities. key process areas.
Verification The activities for While the Integrated Weakness
1 requirements development Product Team leader reviews
and management are reviewed the status of the contract
by acquisition organization and the contractor's cost
management (and the and schedule, he does not
contractor) on a periodic review the status of the
basis. activities that are
required to be performed
for any of the key process
areas.
Verification The activities for While the product team Weakness
2 requirements development leader reviews the status
and management are reviewed of the contract and the
by the project manager on contractor's cost and
both a periodic and event- schedule, he does not
driven basis. review the status of the
activities that are
required to be performed
for any of the key process
areas.
--------------------------------------------------------------------------------
Table 4.4
Requirements Development and Management
Findings for VSCS
Voice Switching and Control System
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition Officials stated that Weakness
organization has a FAA Order 1810.1F is the
written policy for written policy for
establishing and requirements development
managing the system and management, however,
requirements allocated it does not address
to software. software requirements.
Ability 1 A group that is The product team is Strength
responsible for responsible for
performing requirements requirements development
development and and management planning.
management activities
exists.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for identifying the
requirements development resources required and
and management for ensuring that the
activities. needed resources are
provided to the project.
Ability 3 Individuals performing The acquisition Observation
requirements development organization has no
and management guidance regarding
activities have training or experience
experience or receive requirements for project
training. participation.
Activity 1 The project team There are no documented Weakness
documents its plans for plans for requirements
requirements development development and
and management management activities.
activities.
Activity 2 The project's There is no requirements Weakness
requirements development development and
and management management plan,
activities are performed therefore, the
in accordance with its activities cannot be
plans. performed in accordance
with it.
Activity 3 The project team Officials stated that Observation
baselines the software requirements are
requirements and places baselined at contract
them under change award, but no
control early in the documentation was
project, but not later provided to support this
than release of the statement.
solicitation.
Activity 4 The project team The project team Strength
appraises system appraises system
requirements change requirements change
requests for their requests for their
impact on software. impact on software.
Activity 5 The project team The project team Weakness
appraises software appraises software
requirements changes for requirements changes for
their impact on their impact on
performance, schedule, performance, schedule,
cost, system capacities, cost, and system
supportability, and capacities, but not on
architecture. system supportability
and architecture.
Activity 6 The project team A mechanism for Strength
maintains a requirements traceability during the
mechanism for software effort is
traceability during the maintained.
software effort to
ensure requirements have
been included in the
implemented work
products and services.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the or used to determine the
requirements development status of activities for
and management any of the key process
activities. areas.
Verification The activities for While the Integrated Weakness
1 requirements development Product Team leader
and management are reviews the status of
reviewed by acquisition the contract and the
organization management contractor's cost and
(and the contractor) on schedule, he does not
a periodic basis. review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 requirements development leader reviews the
and management are status of the contract
reviewed by the project and the contractor's
manager on both a cost and schedule, he
periodic and event- does not review the
driven basis. status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 4.5
Requirements Development and Management
Findings for WARP
Weather and Radar Processor
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition Officials stated that Weakness
organization has a FAA Order 1810.1F is the
written policy for written policy for
establishing and requirements development
managing the system and management, however,
requirements allocated it does not address
to software. software requirements.
Ability 1 A group that is The team is responsible Strength
responsible for for requirements
performing requirements development and
development and measurement activities.
management activities
exists.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for identifying and for
requirements development ensuring that the needed
and management resources are provided
activities. to the project.
Ability 3 Individuals performing The acquisition Observation
requirements development organization has no
and management guidance regarding
activities have training or experience
experience or receive requirements for project
training. participation.
Activity 1 The project team While a process for Weakness
documents its plans for managing requirements
requirements development changes exists, there is
and management no documented process
activities. for requirements
development and
management.
Activity 2 The project's There is no plan, thus, Weakness
requirements development it cannot be followed.
and management
activities are performed
in accordance with its
plans.
Activity 3 The project team The product team Strength
baselines the software baselined the software
requirements and places requirements and placed
them under change them under change
control early in the control before the
project, but not later release of the
than release of the solicitation.
solicitation.
Activity 4 The project team The product team Strength
appraises system appraises system
requirements change requirements change
requests for their requests for their
impact on software. impact on software.
Activity 5 The project team WARP has not had a Not rated
appraises software software requirement
requirements changes for change yet; therefore,
their impact on this activity was not
performance, schedule, rated.
cost, system capacities,
supportability, and
architecture.
Activity 6 The project team There is a traceability Strength
maintains a requirements matrix for tracking
mechanism for software requirements
traceability during the implementation in the
software effort to system specification.
ensure requirements have
been included in the
implemented work
products and services.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
requirements development the status of activities
and management for any of the key
activities. process areas.
Verification The activities for While the Integrated Weakness
1 requirements development Product Team leader
and management are reviews the status of
reviewed by acquisition the contract and the
organization management contractor's cost and
(and the contractor) on schedule, he does not
a periodic basis. review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 requirements development leader reviews the
and management are status of the contract
reviewed by the project and the contractor's
manager on both a cost and schedule, he
periodic and event- does not review the
driven basis. status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
--------------------
\1 Advanced Automation System: Implications of Problems and Recent
Changes (GAO/T-RCED-94-188, Apr. 13, 1994).
CONCLUSIONS
---------------------------------------------------------- Chapter 4:2
Requirements management has been a pervasive and longstanding problem
with FAA's ATC modernization, and the results of our evaluation point
to many software-specific weaknesses in this area. Because of these
weaknesses, it is likely that requirements management problems will
continue to jeopardize projects' cost, schedule, and performance
goals.
PROJECT OFFICE MANAGEMENT
============================================================ Chapter 5
The purpose of project office management is to manage the activities
of the project office and supporting contractors to ensure a timely,
efficient, and effective software acquisition. According to the
SA-CMM, effective project office management requires, among other
things, that project teams (1) be organized to accomplish the
project's objective, (2) have a written policy for the management of
the software project, (3) document its plans for the activities of
the project team, (4) have the authority to alter either the
project's performance, cost, or schedule baseline while maintaining
the other two, and (5) periodically brief management on the status of
project office management activities.
FAA'S PROJECT OFFICE MANAGEMENT
PROCESS AREA IS NOT EFFECTIVE
---------------------------------------------------------- Chapter 5:1
ATC modernization teams are organized to accomplish project
objectives, with each team including representatives from key
functional areas (e.g., software engineering, contracting, test and
evaluation, operations and maintenance). However, serious weaknesses
in other KPA requirements undermine FAA's project office management
capability. For example, most teams lack a written policy for
software project management, do not document its plans for software
acquisition management activities, and could not identify which team
member(s) is responsible for different team activities (e.g.,
software, support, requirements, testing, and/or reviews). As a
result, lines of accountability and decision-making are blurred,
increasing the chances of delays and mistakes. Additionally, the
product lead cannot adjust either software performance, cost, or
schedule baseline while holding the other two constant. This
inflexibility limits the teams' ability to effectively and
efficiently respond to such events as valid requirements changes and
funding changes. Also, project teams do not periodically brief
management on the status of project office activities, which means
that management may not be able take corrective action when
warranted.
Figure 5.1 provides a comprehensive listing of the five projects'
strengths, weaknesses, and observations for the project office
management KPA. The specific findings supporting the practice
ratings cited in figure 5.1 are in tables 5.1 through 5.5.
Figure 5.1: Project Office
Management Summary
(See figure in printed
edition.)
(See figure in printed
edition.)
Table 5.1
Project Office Management Findings for
ARTSIIIE
Automated Radar Terminal System IIIE
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition FAA Order 1810.1F was Weakness
organization has a cited as the written
written policy for policy, but it does not
execution of the contain policy for
software project. software project
execution.
Commitment 2 Performance, cost, and Performance, cost, and Strength
schedule baselines are schedule baselines are
supported. supported.
Ability 1 A project team that is The product team is Strength
responsible for responsible for
performing the project's performing software
software acquisition acquisition management
management activities activities.
exists.
Ability 2 Adequate resources for No mechanism exists for Weakness
the project team and identifying resources
matrix support required and for
organization(s) are ensuring that the needed
provided for the resources are provided
duration of the software to the project.
acquisition project.
Ability 3 The project manager is The acquisition baseline Weakness
permitted to alter process does not allow
either the performance, the project manager to
cost, or schedule alter the baseline.
software acquisition
baseline while
maintaining the other
two constant.
Ability 4 The project team and The acquisition Observation
matrix support organization has no
individual(s) have guidance regarding
experience or receive training or experience
training in project requirements for project
office software participation.
acquisition management
activities.
Activity 1 The project team The project plans do not Weakness
documents its plans for address software
software acquisition acquisition project
management activities. management.
Activity 2 The project team is The product team is Strength
organized to accomplish organized to accomplish
the project's the project's
objectives. objectives.
Activity 3 The software acquisition The activities of the Strength
activities of the product team are
project team are directed and controlled
directed to accomplish to accomplish the
the project's project's objectives.
objectives.
Activity 4 The software acquisition The software activities Strength
activities of the of the product team are
project team are controlled.
controlled.
Activity 5 Measurements are used to Measurements are used to Strength
track project status, track project status,
execution, and funding execution, and funding
expenditures. expenditures.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
project office the status of activities
management activities. for any of the key
process areas.
Verification The activities for While the Integrated Weakness
1 project office Product Team leader
management are reviewed reviews the status of
by acquisition the contract and the
organization management contractor's cost and
on a periodic basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 project office leader reviews the
management are reviewed status of the contract
by the project manager and the contractor's
on both a periodic and cost and schedule, he
event-driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 5.2
Project Office Management Findings for
DSR
Display System Replacement
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------ --------------------------- --------------------------- --------
Commitment 1 The acquisition FAA Order 1810.1F was cited Weakness
organization has a written as the written policy, but
policy for execution of the it does not contain policy
software project. for software project
execution.
Commitment 2 Performance, cost, and Performance, cost, and Strength
schedule baselines are schedule baselines are
supported. supported.
Ability 1 A project team that is The product team is Strength
responsible for performing responsible for managing
the project's software the software acquisition
acquisition management management activities.
activities exists.
Ability 2 Adequate resources for the No mechanism exists for Weakness
project team and matrix identifying resources
support organization(s) are required and for ensuring
provided for the duration that the needed resources
of the software acquisition are provided to the
project. project.
Ability 3 The project manager is The product team leader Weakness
permitted to alter either cannot alter the
the performance, cost, or performance, cost, or
schedule software schedule software
acquisition baseline while acquisition baseline.
maintaining the other two
constant.
Ability 4 The project team and matrix The acquisition Observat
support individual(s) have organization has no ion
experience or receive guidance regarding training
training in project office or experience requirements
software acquisition for project participation.
management activities.
Activity 1 The project team documents Plans for software Weakness
its plans for software acquisition management
acquisition management activities are not
activities. documented.
Activity 2 The project team is The product team is Strength
organized to accomplish the organized to achieve the
project's objectives. project's objectives.
Activity 3 The software acquisition The software acquisition Strength
activities of the project activities of the product
team are directed to team are directed to
accomplish the project's accomplish the project's
objectives. objectives.
Activity 4 The software acquisition The software acquisition Strength
activities of the project activities of the product
team are controlled. team are controlled by the
Integrated Product Team
leader.
Activity 5 Measurements are used to The product team is using Strength
track project status, measurements to track
execution, and funding product status, execution,
expenditures. and funding expenditures.
Measurement Measurements are made and No internal process Weakness
1 used to determine the measurements are taken and
status of the project used to determine the
office management status of activities for
activities. any of the key process
areas.
Verification The activities for project While the Integrated Weakness
1 office management are Product Team leader reviews
reviewed by acquisition the status of the contract
organization management on and the contractor's cost
a periodic basis. and schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key process
areas.
Verification The activities for project While the product team Weakness
2 office management are leader reviews the status
reviewed by the project of the contract and the
manager on both a periodic contractor's cost and
and event-driven basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key process
areas.
--------------------------------------------------------------------------------
Table 5.3
Project Office Management Findings for
NIMS
NAS Infrastructure Management System
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition The policy for project Strength
organization has a office management is
written policy for contained in the
execution of the Acquisition Management
software project. System.
Commitment 2 Performance, cost, and Performance, cost, and Strength
schedule baselines are schedule baselines were
supported. developed and are being
reviewed through the
Acquisition Program
Baseline process.
Ability 1 A project team that is The product team is Weakness
responsible for assigned the
performing the project's responsibility for
software acquisition acquisition management
management activities activities. However, no
exists. one is assigned
responsibility for
software acquisition
management activities.
Ability 2 Adequate resources for No mechanism exists for Weakness
the project team and identifying resources
matrix support required and for
organization(s) are ensuring that the needed
provided for the resources are provided
duration of the software to the project.
acquisition project.
Ability 3 The project manager is Officials said they Observation
permitted to alter could change the cost,
either the performance, performance, or schedule
cost, or schedule baseline, but could not
software acquisition provide documentation to
baseline while support this.
maintaining the other
two constant.
Ability 4 The project team and The acquisition Observation
matrix support organization has no
individual(s) have guidance regarding
experience or receive training or experience
training in project requirements for project
office software participation.
acquisition management
activities.
Activity 1 The project team The Integrated Program Weakness
documents its plans for Plan and the Product
software acquisition Team Plan provide plans
management activities. for acquisition
management, but these do
not address software
acquisition management
activities.
Activity 2 The project team is The product team is Strength
organized to accomplish being organized to
the project's accomplish the project's
objectives. objectives.
Activity 3 The software acquisition Too early in project Not rated
activities of the life to assess: no
project team are software acquisition
directed to accomplish activities performed.
the project's
objectives.
Activity 4 The software acquisition Too early in project Not rated
activities of the life to assess: no
project team are software acquisition
controlled. activities performed.
Activity 5 Measurements are used to The project has not Not rated
track project status, reached a stage where
execution, and funding this activity applies.
expenditures.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the to determine the status
project office of activities for any of
management activities. the key process areas.
Verification The activities for While the Integrated Weakness
1 project office Product Team leader
management are reviewed reviews the status of
by acquisition the contract and the
organization management contractor's cost and
on a periodic basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 project office leader reviews the
management are reviewed status of the contract
by the project manager and the contractor's
on both a periodic and cost and schedule, he
event-driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 5.4
Project Office Management Findings for
VSCS
Voice Switching and Control System
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------ --------------------------- --------------------------- --------
Commitment 1 The acquisition FAA Order 1810.1F was cited Weakness
organization has a written as the written policy, but
policy for execution of the it does not contain policy
software project. for software project
execution.
Commitment 2 Performance, cost, and Performance, cost, and Weakness
schedule baselines are schedule baselines are not
supported. supported.
Ability 1 A project team that is The product team is Strength
responsible for performing responsible for performing
the project's software software acquisition
acquisition management management activities.
activities exists.
Ability 2 Adequate resources for the No mechanism exists for Weakness
project team and matrix identifying the resources
support organization(s) are required and for ensuring
provided for the duration that the needed resources
of the software acquisition are provided to the
project. project.
Ability 3 The project manager is The product team leader Weakness
permitted to alter either does not have the
the performance, cost, or flexibility to alter cost,
schedule software performance, or schedule
acquisition baseline while while maintaining the other
maintaining the other two two.
constant.
Ability 4 The project team and matrix The acquisition Observat
support individual(s) have organization has no ion
experience or receive guidance regarding training
training in project office or experience requirements
software acquisition for project participation.
management activities.
Activity 1 The project team documents The Product Team Plan and Strength
its plans for software the Contract Management
acquisition management Plan document acquisition
activities. activities.
Activity 2 The project team is The product team is Strength
organized to accomplish the organized to accomplish the
project's objectives. project's objectives.
Activity 3 The software acquisition While officials stated that Weakness
activities of the project software activities of the
team are directed to team are directed to
accomplish the project's accomplish the project's
objectives. objectives, documents
provided do not specify the
activities that the team
members must accomplish.
Activity 4 The software acquisition The software acquisition Strength
activities of the project activities are controlled.
team are controlled.
Activity 5 Measurements are used to Measurements are used to Strength
track project status, track project status,
execution, and funding execution, and funding
expenditures. expenditures.
Measurement Measurements are made and No internal process Weakness
1 used to determine the measurements are taken or
status of the project used to determine the
office management status of activities for
activities. any of the key process
areas.
Verification The activities for project While the Integrated Weakness
1 office management are Product Team leader reviews
reviewed by acquisition the status of the contract
organization management on and the contractor's cost
a periodic basis. and schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key process
areas.
Verification The activities for project While the product team Weakness
2 office management are leader reviews the status
reviewed by the project of the contract and the
manager on both a periodic contractor's cost and
and event-driven basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key process
areas.
--------------------------------------------------------------------------------
Table 5.5
Project Office Management Findings for
WARP
Weather and Radar Processor
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------ --------------------------- --------------------------- --------
Commitment 1 The acquisition FAA Order 1810.1F was cited Weakness
organization has a written as the written policy, but
policy for execution of the it does not contain policy
software project. for software project
execution.
Commitment 2 Performance, cost, and Performance, cost, and Strength
schedule baselines are schedule baselines are
supported. generated and supported.
Ability 1 A project team that is The product team is Strength
responsible for performing responsible for performing
the project's software software acquisition
acquisition management management activities.
activities exists.
Ability 2 Adequate resources for the No mechanism exists for Weakness
project team and matrix identifying and for
support organization(s) are ensuring that the needed
provided for the duration resources are provided to
of the software acquisition the project.
project.
Ability 3 The project manager is The acquisition baseline Weakness
permitted to alter either process does not allow the
the performance, cost, or project manager to alter
schedule software the baseline.
acquisition baseline while
maintaining the other two
constant.
Ability 4 The project team and matrix The acquisition Observat
support individual(s) have organization has no ion
experience or receive guidance regarding training
training in project office or experience requirements
software acquisition for project participation.
management activities.
Activity 1 The project team documents The product team documents Strength
its plans for software its plans for software
acquisition management acquisition management
activities. activities.
Activity 2 The project team is The product team is Strength
organized to accomplish the organized to accomplish the
project's objectives. project's objectives.
Activity 3 The software acquisition The software acquisition Strength
activities of the project activities of the project
team are directed to team are directed to
accomplish the project's accomplish the project's
objectives. objectives.
Activity 4 The software acquisition No individual is Weakness
activities of the project controlling the software
team are controlled. acquisition activities of
the product team.
Activity 5 Measurements are used to Measurements are used to Strength
track project status, track project status,
execution, and funding execution, and funding
expenditures. status.
Measurement Measurements are made and No internal process Weakness
1 used to determine the measurements are taken and
status of the project used to determine the
office management status of activities for
activities. any of the key process
areas.
Verification The activities for project While the Integrated Weakness
1 office management are Product Team leader reviews
reviewed by acquisition the status of the contract
organization management on and the contractor's cost
a periodic basis. and schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key process
areas.
Verification The activities for project While the product team Weakness
2 office management are leader reviews the status
reviewed by the project of the contract and the
manager on both a periodic contractor's cost and
and event-driven basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key process
areas.
--------------------------------------------------------------------------------
CONCLUSIONS
---------------------------------------------------------- Chapter 5:2
Numerous ad hoc project office management practices, including a
pervasive lack of measurement, analysis, and verification of project
status and progress, are limiting FAA's ability to meet ATC
modernization project commitments. More discipline and definition in
this KPA is needed before ATC modernization teams can consistently
repeat project successes.
CONTRACT TRACKING AND OVERSIGHT
============================================================ Chapter 6
The purpose of contract tracking and oversight is to ensure that (1)
the software development contractor performs according to the terms
of the contract; (2) needed contract changes are identified,
negotiated, and incorporated into the contract; and (3) contractor
performance issues are identified early, when they are easier to
address. According to the SA-CMM, a repeatable contract tracking and
oversight process, among other things, includes (1) having a written
organizational policy for contract tracking and oversight, (2) having
a documented plan for contract tracking and oversight, (3) conducting
tracking and oversight activities in accordance with the plan, and
(4) ensuring that individuals performing contract tracking and
oversight are suitably experienced or trained.
FAA LACKS AN EFFECTIVE CONTRACT
TRACKING AND OVERSIGHT PROCESS
---------------------------------------------------------- Chapter 6:1
Our past work on ATC modernization projects has raised concerns about
contract tracking and oversight. For example, in 1994 we reported
that FAA did not provide adequate oversight of its contractor during
the initial development of the ISSS.\1 As a result, development
problems and lack of progress were not always recognized in a timely
manner. The results of this software capability evaluation indicate
that these problems persist and pinpoint the underlying contract
tracking and oversight weaknesses. For example, FAA does not have a
written organizational policy for contract tracking and oversight,
and most teams have no documented plan for contract tracking and
oversight activities. Furthermore, the team that has a plan does not
always follow the plan, and none of the teams ensure that persons
responsible for managing software contracts have suitable experience
or training. As a result, the product teams cannot formulate an
independent assessment of contract progress and are forced to rely on
data provided by the contractor. Since contractor reports do not
always identify problems expeditiously, FAA is not always positioned
to correct them promptly.
Figure 6.1 provides a comprehensive listing of the five projects'
strengths, weaknesses, and observations for the contractor tracking
and oversight KPA. The specific findings supporting the practice
ratings cited in figure 6.1 are in tables 6.1 through 6.5.
Figure 6.1: Contract Tracking
and Oversight Summary
(See figure in printed
edition.)
(See figure in printed
edition.)
Table 6.1
Contract Tracking and Oversight Findings
for ARTSIIIE
Automated Radar Terminal System IIIE
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition FAA Order 1810.1F was Weakness
organization has a cited as the written
written policy for the policy, but it does not
contract tracking and provide the policy for
oversight of the contract tracking and
contracted software oversight.
effort.
Commitment 2 Responsibility for the Responsibility for Strength
contract tracking and contract tracking and
oversight activities is oversight is designated.
designated.
Commitment 3 The project team is Contract specialists are Strength
supported by contracting assigned to the product
specialists in the team.
execution of the
contract.
Ability 1 A group that is Officials gave Weakness
responsible for managing conflicting statements
contract tracking and as to who has the
oversight activities responsibility for
exists. contract tracking and
oversight activities,
and could not provide
documentation that
formally delegates
responsibility.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for contract identifying resources
tracking and oversight required and for
activities. ensuring that the needed
resources are provided
to the project.
Ability 3 Individuals performing The acquisition Observation
contract tracking and organization has no
oversight activities guidance regarding
have experience or training or experience
receive training. requirements for project
participation.
Activity 1 The project team Contract tracking and Weakness
documents its plans for oversight plans do not
contract tracking and exist.
oversight activities.
Activity 2 The project's contract No plan exists, Weakness
tracking and oversight therefore, activities
activities are performed cannot be performed in
in accordance with its accordance with it.
plans.
Activity 3 The project team reviews While officials stated Observation
required contractor that the product team
software planning reviews contractor
documents which, when software planning
satisfactory, are made documents, they could
part of the contractor's not provide
baseline. documentation to support
this.
Activity 4 The project team, with Periodic reviews are Strength
end user input, conducts held.
periodic reviews and
interchanges with the
contractor.
Activity 5 The project team tracks The product team tracks Strength
the contractor's the development of the
development of the software engineering
software engineering environment.
environment required to
support the software.
Activity 6 Any problems or issues Issues found during Strength
found by the project meetings and reviews are
team during contract documented in minutes
tracking and oversight and tracked.
are recorded in the
appropriate corrective
action system and
tracked to closure.
Activity 7 The project team While product team Weakness
maintains the integrity members (including the
of the contract contracting officer)
throughout the contract stated that the
performance period. contracting officer is
responsible for
maintaining the
integrity of the
contract, they could not
provide any documents
that support this.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
contract tracking and the status of activities
oversight activities. for any of the key
process areas.
Verification The activities for While the Integrated Weakness
1 contract tracking and Product Team leader
oversight are reviewed reviews the status of
by acquisition the contract and the
organization management contractor's cost and
on a periodic basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 contract tracking and leader reviews the
oversight are reviewed status of the contract
by the project manager and the contractor's
on both a periodic and cost and schedule, he
event-driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 6.2
Contract Tracking and Oversight Findings
for DSR
Display System Replacement
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition FAA Order 1810.1F was Weakness
organization has a cited as the written
written policy for the policy, but it does not
contract tracking and provide the policy for
oversight of the contract tracking and
contracted software oversight.
effort.
Commitment 2 Responsibility for the Officials gave Weakness
contract tracking and conflicting answers on
oversight activities is who is responsible for
designated. contract tracking and
oversight, and they
could not provide
documentation that
formally delegates
responsibility.
Commitment 3 The project team is The product team is Strength
supported by contracting supported by a
specialists in the contracting specialist
execution of the in execution of the
contract. contract.
Ability 1 A group that is The product team is Strength
responsible for managing responsible for managing
contract tracking and contract tracking and
oversight activities oversight activities.
exists.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for contract identifying resources
tracking and oversight required and for
activities. ensuring that the needed
resources are provided
to the project.
Ability 3 Individuals performing The acquisition Observation
contract tracking and organization has no
oversight activities guidance regarding
have experience or training or experience
receive training. requirements for project
participation.
Activity 1 The project team There are no written Weakness
documents its plans for plans for contract
contract tracking and tracking and oversight
oversight activities. activities.
Activity 2 The project's contract No plan exists, Weakness
tracking and oversight therefore, the product
activities are performed team cannot perform in
in accordance with its accordance with it.
plans.
Activity 3 The project team reviews Reviews are used to Strength
required contractor approve contract
software planning planning documents,
documents which, when which, when
satisfactory, are made satisfactory, are made
part of the contractor's part of the contractor's
baseline. baseline.
Activity 4 The project team, with While it was stated that Observation
end user input, conducts continuous interactions
periodic reviews and with the contractor are
interchanges with the held, officials could
contractor. provide no documents to
support this.
Activity 5 The project team tracks The product team tracks Strength
the contractor's the contractor's
development of the development of the
software engineering software engineering
environment required to environment required to
support the software. support the software.
Activity 6 Any problems or issues The product team records Strength
found by the project problems and issues
team during contract found during contract
tracking and oversight tracking and oversight
are recorded in the and tracks them to
appropriate corrective closure.
action system and
tracked to closure.
Activity 7 The project team The product team Strength
maintains the integrity maintains the integrity
of the contract of the contract
throughout the contract throughout the contract
performance period. performance period.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
contract tracking and the status of activities
oversight activities. for any of the key
process areas.
Verification The activities for While the Integrated Weakness
1 contract tracking and Product Team leader
oversight are reviewed reviews the status of
by acquisition the contract and the
organization management contractor's cost and
on a periodic basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 contract tracking and leader reviews the
oversight are reviewed status of the contract
by the project manager and the contractor's
on both a periodic and cost and schedule, he
event-driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 6.3
Contract Tracking and Oversight Findings
for NIMS
NAS Infrastructure Management System
--------------------------------------------------------------------------------
Key Practice Findings Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition Since NIMS is not under Not rated
organization has a contract yet, it was not
written policy for the evaluated against this
contract tracking and KPA.
oversight of the
contracted software
effort.
Commitment 2 Responsibility for the Too early to assess: the Not rated
contract tracking and project has not reached
oversight activities is a stage where this
designated. applies.
Commitment 3 The project team is Too early to assess: the Not rated
supported by contracting project has not reached
specialists in the a stage where this
execution of the applies.
contract.
Ability 1 A group that is Too early to assess: the Not rated
responsible for managing project has not reached
contract tracking and a stage where this
oversight activities applies.
exists.
Ability 2 Adequate resources are Too early to assess: the Not rated
provided for contract project has not reached
tracking and oversight a stage where this
activities. applies.
Ability 3 Individuals performing Too early to assess: the Not rated
contract tracking and project has not reached
oversight activities a stage where this
have experience or applies.
receive training.
Activity 1 The project team Too early to assess: the Not rated
documents its plans for project has not reached
contract tracking and a stage where this
oversight activities. applies.
Activity 2 The project's contract Too early to assess: the Not rated
tracking and oversight project has not reached
activities are performed a stage where this
in accordance with its applies.
plans.
Activity 3 The project team reviews Too early to assess: the Not rated
required contractor project has not reached
software planning a stage where this
documents which, when applies.
satisfactory, are made
part of the contractor's
baseline.
Activity 4 The project team, with Too early to assess: the Not rated
end user input, conducts project has not reached
periodic reviews and a stage where this
interchanges with the applies.
contractor.
Activity 5 The project team tracks Too early to assess: the Not rated
the contractor's project has not reached
development of the a stage where this
software engineering applies.
environment required to
support the software.
Activity 6 Any problems or issues Too early to assess: the Not rated
found by the project project has not reached
team during contract a stage where this
tracking and oversight applies.
are recorded in the
appropriate corrective
action system and
tracked to closure.
Activity 7 The project team Too early to assess: the Not rated
maintains the integrity project has not reached
of the contract a stage where this
throughout the contract applies.
performance period.
Measurement 1 Measurements are made Too early to assess: the Not rated
and used to determine project has not reached
the status of the a stage where this
contract tracking and applies.
oversight activities.
Verification The activities for Too early to assess: the Not rated
1 contract tracking and project has not reached
oversight are reviewed a stage where this
by acquisition applies.
organization management
on a periodic basis.
Verification The activities for Too early to assess: the Not rated
2 contract tracking and project has not reached
oversight are reviewed a stage where this
by the project manager applies.
on both a periodic and
event-driven basis.
--------------------------------------------------------------------------------
Table 6.4
Contract Tracking and Oversight Findings
for VSCS
Voice Switching and Control System
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition There is no policy on Weakness
organization has a contract tracking and
written policy for the oversight.
contract tracking and
oversight of the
contracted software
effort.
Commitment 2 Responsibility for the Responsibility for Strength
contract tracking and contract tracking and
oversight activities is oversight is designated.
designated.
Commitment 3 The project team is The product team is Strength
supported by contracting supported by a
specialists in the contracting specialist.
execution of the
contract.
Ability 1 A group that is A group exists that is Strength
responsible for managing responsible for managing
contract tracking and contract tracking and
oversight activities oversight.
exists.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for contract identifying the
tracking and oversight resources required and
activities. for ensuring that the
needed resources are
provided to the project.
Ability 3 Individuals performing The acquisition Observation
contract tracking and organization has no
oversight activities guidance regarding
have experience or training or experience
receive training. requirements for project
participation.
Activity 1 The project team The product team has Strength
documents its plans for documented its plans.
contract tracking and
oversight activities.
Activity 2 The project's contract The product team could Weakness
tracking and oversight not provide evidence
activities are performed that shows its
in accordance with its contracting tracking and
plans. oversight activities are
performed in accordance
with its plans.
Activity 3 The project team reviews While reviews are Weakness
required contractor conducted, officials
software planning could not provide
documents which, when documentation that shows
satisfactory, are made the results are made
part of the contractor's part of the contractor's
baseline. baseline.
Activity 4 The project team, with The product team Strength
end user input, conducts conducts periodic
periodic reviews and reviews and interchanges
interchanges with the with the contractor.
contractor.
Activity 5 The project team tracks The contractor is Strength
the contractor's required to provide a
development of the list of common tools and
software engineering support equipment, which
environment required to the product team uses to
support the software. track the software
engineering environment
development.
Activity 6 Any problems or issues The contractor has an Strength
found by the project extensive software
team during contract development environment
tracking and oversight and problems or issues
are recorded in the are tracked to closure.
appropriate corrective
action system and
tracked to closure.
Activity 7 The project team There is no evidence Weakness
maintains the integrity that either the
of the contract contracting officer or
throughout the contract the product team are
performance period. following the process
and maintaining the
integrity of the
contract.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the or used to determine the
contract tracking and status of activities for
oversight activities. any of the key process
areas.
Verification The activities for While the Integrated Weakness
1 contract tracking and Product Team leader
oversight are reviewed reviews the status of
by acquisition the contract and the
organization management contractor's cost and
on a periodic basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 contract tracking and leader reviews the
oversight are reviewed status of the contract
by the project manager and the contractor's
on both a periodic and cost and schedule, he
event-driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 6.5
Contract Tracking and Oversight Findings
for WARP
Weather and Radar Processor
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition There is no written Weakness
organization has a policy for contract
written policy for the tracking and oversight
contract tracking and activities.
oversight of the
contracted software
effort.
Commitment 2 Responsibility for the The product team is Strength
contract tracking and responsible for contract
oversight activities is tracking and oversight
designated. activities.
Commitment 3 The project team is The product team is Strength
supported by contracting supported by contracting
specialists in the specialists.
execution of the
contract.
Ability 1 A group that is The product team is Strength
responsible for managing collectively responsible
contract tracking and for contract tracking
oversight activities and oversight
exists. activities.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for contract identifying and for
tracking and oversight ensuring that the needed
activities. resources are provided
to the project.
Ability 3 Individuals performing The acquisition Observation
contract tracking and organization has no
oversight activities guidance regarding
have experience or training or experience
receive training. requirements for project
participation.
Activity 1 The project team Plans for contract Weakness
documents its plans for tracking and oversight
contract tracking and activities are not
oversight activities. documented.
Activity 2 The project's contract Since there is no Weakness
tracking and oversight contract tracking and
activities are performed oversight plan, there is
in accordance with its no way to assess whether
plans. activities are being
performed in accordance
with the plan.
Activity 3 The project team reviews The product team reviews Strength
required contractor required contractor
software planning software planning
documents which, when documents which, when
satisfactory, are made satisfactory, are made
part of the contractor's part of the contractor's
baseline. baseline.
Activity 4 The project team, with The product team Strength
end user input, conducts conducts periodic
periodic reviews and reviews and interchanges
interchanges with the with the contractor.
contractor.
Activity 5 The project team tracks As a deliverable, the Strength
the contractor's status of the software
development of the support environment
software engineering development is reviewed
environment required to and tracked.
support the software.
Activity 6 Any problems or issues The contracting Strength
found by the project officer's technical
team during contract representative is
tracking and oversight responsible for managing
are recorded in the and tracking action
appropriate corrective items, and these are
action system and recorded in an
tracked to closure. appropriate correction
system.
Activity 7 The project team The contracting officer Strength
maintains the integrity maintains the integrity
of the contract of the contract and is
throughout the contract responsible for doing so
performance period. throughout the contract
performance period.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
contract tracking and the status of activities
oversight activities. for any of the key
process areas.
Verification The activities for While the Integrated Weakness
1 contract tracking and Product Team leader
oversight are reviewed reviews the status of
by acquisition the contract and the
organization management contractor's cost and
on a periodic basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 contract tracking and leader reviews the
oversight are reviewed status of the contract
by the project manager and the contractor's
on both a periodic and cost and schedule, he
event-driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
--------------------
\1 Advanced Automation System: Implications of Problems and Recent
Changes (GAO/T-RCED-94-188, Apr. 13, 1994).
CONCLUSIONS
---------------------------------------------------------- Chapter 6:2
To effectively and efficiently acquire software, FAA must have a
well-defined and enforced process that provides for proactive
tracking and oversight of its software development contractors.
FAA's current process for ATC modernization contractor tracking and
oversight is ad hoc and reactive, thereby increasing the chances of
its ATC software acquisitions being late, costing more than expected,
and not performing as intended.
EVALUATION
============================================================ Chapter 7
The purpose of evaluation, or testing, is to determine that the
acquired software products and services satisfy contract requirements
prior to acceptance. According to the SA-CMM, a repeatable
evaluation process includes (1) documenting evaluation plans and
conducting evaluation activities in accordance with the plan, (2)
developing and managing evaluation requirements in conjunction with
developing software technical requirements, (3) incorporating
evaluation requirements into the solicitation and the resulting
contract, (4) tracking contractor performance of evaluation
activities for compliance with the contract, (5) ensuring that
adequate resources are provided for evaluation activities, and (6)
measuring and reporting on the status of evaluation activities to
management.
FAA IS STRONG IN MOST BUT NOT
ALL EVALUATION KPA PRACTICES
---------------------------------------------------------- Chapter 7:1
All of the projects were strong in many evaluation practice areas.
For example, all rated projects have documented test and evaluation
plans and conduct test and evaluation activities in accordance with
the plans. In addition, most teams develop evaluation requirements
for contractor-conducted software tests concurrent with developing
software technical requirements, and all teams incorporate evaluation
requirements into the solicitation and resulting contract. Also,
most teams track contractor performance of test activities for
compliance with the contract.
Despite these many strengths, several weaknesses prevented FAA from
meeting this KPA. For example, only one of the teams ensures that
adequate resources are provided for evaluation activities.
Additionally, none of the teams measure and report on the status of
all evaluation activities to management. As a result, management
does not have a complete and accurate picture of evaluation status
and progress, which could impair decision-making.
Figure 7.1 provides a comprehensive listing of the five projects'
strengths, weaknesses, and observations for the evaluation KPA. The
specific findings supporting the practice ratings cited in figure 7.1
are in tables 7.1 through 7.5.
Figure 7.1: Evaluation Summary
(See figure in printed
edition.)
(See figure in printed
edition.)
Table 7.1
Evaluation Findings for ARTSIIIE
Automated Radar Terminal System IIIE
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition FAA Order 1810.4B is the Strength
organization has a written policy.
written policy for
managing the evaluation
of the acquired software
products and services.
Commitment 2 Responsibility for Evaluation Strength
evaluation activities is responsibility is
clearly defined. clearly defined.
Ability 1 A group that is The product team is Strength
responsible for responsible for
planning, managing, and evaluation activities.
performing evaluation
activities for the
project exists.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for evaluation identifying resources
activities. required and for
ensuring that the needed
resources are provided
to the project.
Ability 3 Individuals performing The acquisition Observation
evaluation activities organization has no
have experience or guidance regarding
receive training. training or experience
requirements for project
participation.
Ability 4 Members of the project The product team did not Weakness
team and groups receive orientation on
supporting the software the evaluation approach.
acquisition receive
orientation on the
objectives of the
evaluation approach.
Activity 1 The project team The Test and Evaluation Strength
documents its plans for Master Plan delineates
evaluation activities. all evaluation
activities.
Activity 2 The project's evaluation Evaluation activities Strength
activities are performed are performed in
in accordance with its accordance with plans.
plans.
Activity 3 Evaluation requirements Evaluation requirements Strength
are developed and are developed and
managed in conjunction managed in conjunction
with development of the with development of the
system or software system or software
technical requirements. technical requirements.
Activity 4 The evaluation Evaluation requirements Strength
requirements are are part of the
incorporated into the contract.
solicitation and
resulting contract.
Activity 5 The project team tracks The evaluation Strength
contractor's performance activities performed by
in terms of evaluation the contractor are
requirements for tracked.
compliance with the
contract.
Activity 6 Planned evaluations are Planned evaluations are Strength
performed on the performed to ensure that
acquired software technical and contract
products and services requirements are met
prior to acceptance for prior to acceptance.
operational use.
Activity 7 Results of the Technical and contract Strength
evaluations are analyzed requirements are met
and compared to the prior to acceptance.
contract's requirements
to establish a basis for
acceptance.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
evaluation activities. the status of activities
for any of the key
process areas.
Verification The activities for While the Integrated Weakness
1 evaluation are reviewed Product Team leader
with acquisition reviews the status of
organization management the contract and the
on a periodic basis. contractor's cost and
schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 evaluation are reviewed leader reviews the
with the project manager status of the contract
on both a periodic and and the contractor's
event-driven basis. cost and schedule, he
does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 7.2
Evaluation Findings for DSR
Display System Replacement
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition FAA Order 1810.4B is the Strength
organization has a written policy.
written policy for
managing the evaluation
of the acquired software
products and services.
Commitment 2 Responsibility for The product team leader Strength
evaluation activities is is responsible for
clearly defined. evaluation activities.
Ability 1 A group that is The test and maintenance Strength
responsible for leader, along with the
planning, managing, and product team, are
performing evaluation responsible for
activities for the planning, managing, and
project exists. performing evaluation
activities.
Ability 2 Adequate resources are Adequate resources are Strength
provided for evaluation provided for evaluation
activities. activities.
Ability 3 Individuals performing The acquisition Observation
evaluation activities organization has no
have experience or guidance regarding
receive training. training or experience
requirements for project
participation.
Ability 4 Members of the project Members of the project Strength
team and groups team received
supporting the software orientation on the
acquisition receive evaluation approach.
orientation on the
objectives of the
evaluation approach.
Activity 1 The project team The product team Strength
documents its plans for documents its plans for
evaluation activities. evaluation activities.
Activity 2 The project's evaluation The project's evaluation Strength
activities are performed activities are performed
in accordance with its in accordance with its
plans. plans.
Activity 3 Evaluation requirements Evaluation requirements Strength
are developed and are developed in
managed in conjunction conjunction with the
with development of the system requirements.
system or software
technical requirements.
Activity 4 The evaluation The evaluation Strength
requirements are requirements are
incorporated into the incorporated into the
solicitation and contract.
resulting contract.
Activity 5 The project team tracks The product team tracks Strength
contractor's performance the contractor's
in terms of evaluation performance, in terms of
requirements for evaluation requirements,
compliance with the using the A-level and B-
contract. level specifications in
the contract as
criteria.
Activity 6 Planned evaluations are Planned evaluations are Strength
performed on the performed on the
acquired software acquired software
products and services products and services
prior to acceptance for prior to acceptance for
operational use. operational use.
Activity 7 Results of the Results of evaluations Strength
evaluations are analyzed are analyzed and
and compared to the compared to the
contract's requirements contract's requirements
to establish a basis for to establish a basis for
acceptance. acceptance.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
evaluation activities. the status of activities
for any of the key
process areas.
Verification The activities for While the Integrated Weakness
1 evaluation are reviewed Product Team leader
with acquisition reviews the status of
organization management the contract and the
on a periodic basis. contractor's cost and
schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 evaluation are reviewed leader reviews the
with the project manager status of the contract
on both a periodic and and the contractor's
event-driven basis. cost and schedule, he
does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 7.3
Evaluation Findings for NIMS
NAS Infrastructure Management System
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition The Acquisition Strength
organization has a Management System is the
written policy for written policy.
managing the evaluation
of the acquired software
products and services.
Commitment 2 Responsibility for Officials stated that Observation
evaluation activities is the test leader is
clearly defined. responsible for
evaluation activities;
however, they could not
provide documentation to
support this.
Ability 1 A group that is The product team has Strength
responsible for responsibility for
planning, managing, and planning, managing, and
performing evaluation performing evaluation
activities for the activities.
project exists.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for evaluation identifying resources
activities. required and for
ensuring that the needed
resources are provided
to the project.
Ability 3 Individuals performing The acquisition Observation
evaluation activities organization has no
have experience or guidance regarding
receive training. training or experience
requirements for project
participation.
Ability 4 Members of the project Orientation on the Weakness
team and groups objectives of the
supporting the software evaluation approach was
acquisition receive not received.
orientation on the
objectives of the
evaluation approach.
Activity 1 The project team The project team Strength
documents its plans for documents its plans for
evaluation activities. evaluation activities.
Activity 2 The project's evaluation NIMS has not yet reached Not rated
activities are performed the stage where
in accordance with its evaluation activities
plans. are being performed.
Activity 3 Evaluation requirements Too early to assess: Observation
are developed and system software
managed in conjunction requirements are not
with development of the developed sufficiently
system or software to develop evaluation
technical requirements. requirements.
Activity 4 The evaluation The evaluation Strength
requirements are requirements are
incorporated into the incorporated into the
solicitation and solicitation through the
resulting contract. statement of work, which
will be part of the
contract.
Activity 5 The project team tracks NIMS has not yet reached Not rated
contractor's performance the stage where
in terms of evaluation evaluation activities
requirements for are being performed.
compliance with the
contract.
Activity 6 Planned evaluations are NIMS has not yet reached Not rated
performed on the the stage where
acquired software operational evaluation
products and services activities are being
prior to acceptance for performed.
operational use.
Activity 7 Results of the NIMS has not yet reached Not rated
evaluations are analyzed the stage where
and compared to the evaluation activities
contract's requirements are analyzed and
to establish a basis for compared to the
acceptance. contract.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the to determine the status
evaluation activities. of activities for any of
the key process areas.
Verification The activities for While the Integrated Weakness
1 evaluation are reviewed Product Team leader
with acquisition reviews the status of
organization management the contract and the
on a periodic basis. contractor's cost and
schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 evaluation are reviewed leader reviews the
with the project manager status of the contract
on both a periodic and and the contractor's
event-driven basis. cost and schedule, he
does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 7.4
Evaluation Findings for VSCS
Voice Switching and Control System
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition FAA Order 1810.4B Strength
organization has a provides policy
written policy for guidance.
managing the evaluation
of the acquired software
products and services.
Commitment 2 Responsibility for The Test and Evaluation Strength
evaluation activities is Master Plan defines the
clearly defined. responsibility for
evaluation.
Ability 1 A group that is A group is assigned Strength
responsible for responsibility for
planning, managing, and planning, managing, and
performing evaluation performing evaluation
activities for the activities.
project exists.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for evaluation identifying the
activities. resources required and
for ensuring that the
needed resources are
provided to the project.
Ability 3 Individuals performing The acquisition Observation
evaluation activities organization has no
have experience or guidance regarding
receive training. training or experience
requirements for project
participation.
Ability 4 Members of the project An orientation was Strength
team and groups conducted.
supporting the software
acquisition receive
orientation on the
objectives of the
evaluation approach.
Activity 1 The project team Test and evaluation Strength
documents its plans for activities are
evaluation activities. documented.
Activity 2 The project's evaluation Evaluation activities Strength
activities are performed are performed in
in accordance with its accordance with plans.
plans.
Activity 3 Evaluation requirements Test and evaluation Strength
are developed and requirements are
managed in conjunction developed and managed in
with development of the conjunction with system
system or software requirements.
technical requirements.
Activity 4 The evaluation Evaluation requirements Strength
requirements are are part of the
incorporated into the contract.
solicitation and
resulting contract.
Activity 5 The project team tracks The contractor's Strength
contractor's performance performance in terms of
in terms of evaluation evaluation requirements
requirements for is tracked through
compliance with the weekly meetings.
contract.
Activity 6 Planned evaluations are Evaluation procedures Strength
performed on the are documented in the
acquired software Test and Evaluation
products and services Master Plan and are
prior to acceptance for performed prior to
operational use. acceptance.
Activity 7 Results of the FAA conducts factory Strength
evaluations are analyzed acceptance tests and
and compared to the compares results to the
contract's requirements contract's requirements.
to establish a basis for
acceptance.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the or used to determine the
evaluation activities. status of activities for
any of the key process
areas.
Verification The activities for While the Integrated Weakness
1 evaluation are reviewed Product Team leader
with acquisition reviews the status of
organization management the contract and the
on a periodic basis. contractor's cost and
schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 evaluation are reviewed leader reviews the
with the project manager status of the contract
on both a periodic and and the contractor's
event-driven basis. cost and schedule, he
does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 7.5
Evaluation Findings for WARP
Weather and Radar Processor
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition FAA Order 1810.4B is the Strength
organization has a written policy.
written policy for
managing the evaluation
of the acquired software
products and services.
Commitment 2 Responsibility for Responsibility for Strength
evaluation activities is evaluation activities
clearly defined. has been clearly
defined.
Ability 1 A group that is A group is responsible Strength
responsible for for evaluation
planning, managing, and activities for the
performing evaluation project.
activities for the
project exists.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for evaluation identifying and for
activities. ensuring that the needed
resources are provided
to the project.
Ability 3 Individuals performing The acquisition Observation
evaluation activities organization has no
have experience or guidance regarding
receive training. training or experience
requirements for project
participation.
Ability 4 Members of the project Orientation was not Observation
team and groups needed because the test
supporting the software leader and his team were
acquisition receive familiar with the
orientation on the evaluation approach.
objectives of the
evaluation approach.
Activity 1 The project team The Test and Evaluation Strength
documents its plans for Master Plan documents
evaluation activities. the team's plan for
evaluation activities.
Activity 2 The project's evaluation WARP has not reached the Not rated
activities are performed stage where evaluation
in accordance with its activities are being
plans. performed and can be
compared to a plan.
Activity 3 Evaluation requirements Evaluation requirements Strength
are developed and are developed and
managed in conjunction managed in conjunction
with development of the with development of
system or software system or software
technical requirements. technical requirements.
Activity 4 The evaluation The evaluation Strength
requirements are requirements are
incorporated into the incorporated into the
solicitation and solicitation and
resulting contract. resulting contract.
Activity 5 The project team tracks WARP has not reached the Not rated
contractor's performance stage where the
in terms of evaluation contractor is performing
requirements for evaluation activities
compliance with the that can be tracked for
contract. compliance with
contract.
Activity 6 Planned evaluations are WARP has not yet reached Not rated
performed on the the stage where
acquired software evaluations are being
products and services performed.
prior to acceptance for
operational use.
Activity 7 Results of the WARP has not yet reached Not rated
evaluations are analyzed the stage where
and compared to the evaluations are being
contract's requirements performed.
to establish a basis for
acceptance.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
evaluation activities. the status of activities
for any of the key
process areas.
Verification The activities for While the Integrated Weakness
1 evaluation are reviewed Product Team leader
with acquisition reviews the status of
organization management the contract and the
on a periodic basis. contractor's cost and
schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 evaluation are reviewed leader reviews the
with the project manager status of the contract
on both a periodic and and the contractor's
event-driven basis. cost and schedule, he
does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
CONCLUSIONS
---------------------------------------------------------- Chapter 7:2
FAA performs most but not all evaluation KPA practices. To satisfy
all evaluation practices and thereby have reasonable assurance that
its software acquisition projects will be effectively evaluated and
tested on a repeatable basis, FAA must ensure that its product teams
identify evaluation resource, training, and experience needs, and
that they measure and brief management on the status of all
evaluation activities.
TRANSITION AND SUPPORT
============================================================ Chapter 8
The purpose of transition and support is to provide for the effective
and efficient "hand-off" of the acquired software products to the
support organization responsible for software maintenance. According
to the SA-CMM, repeatable transition and support processes, among
other things, include (1) having a written policy for transitioning
software products to the support organization, (2) designating a
group that is responsible for coordinating transition and support
activities, (3) having a complete inventory of all software and
related items that are to be transitioned, (4) including members of
the support organization in transition and support planning, (5)
requiring the support organization to demonstrate its capability to
modify and support the software, and (6) measuring and reporting to
management on the status of transition and support activities.
TRANSITION AND SUPPORT NOT
BEING PERFORMED EFFECTIVELY
---------------------------------------------------------- Chapter 8:1
All of the projects were strong in several transition and support
practice areas. For example, FAA has a written policy for
transitioning software products to the support organization.
Additionally, all five projects have designated a group responsible
for transition and support. However, various weaknesses in other
practices prevented FAA from satisfying this KPA. In particular,
some of the teams lack a complete inventory of all the software and
related products to be transitioned, thus jeopardizing the efforts of
the support organization to effectively maintain the full software
configuration. Additionally, one team did not include the software
support organization in planning for transition and support, and some
teams do not have plans to require the support organization to
demonstrate its capability to maintain the software after transition.
As a result, support problems, such as the inability to perform
required maintenance, may result. Further, none of the projects are
measuring and reporting to management on the status of transition and
support activities, precluding management from addressing transition
and support problems expeditiously.
Figure 8.1 provides a comprehensive listing of the five projects'
strengths, weaknesses, and observations for the transition and
support KPA. The specific findings supporting the practice ratings
cited in figure 8.1 are in tables 8.1 through 8.5.
Figure 8.1: Transition and
Support Summary
(See figure in printed
edition.)
(See figure in printed
edition.)
Table 8.1
Transition and Support Findings for
ARTSIIIE
Automated Radar Terminal System IIIE
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition FAA Order 1800.58 is the Strength
organization has a written policy.
written policy for the
transitioning of
software products to the
support organization.
Commitment 2 The acquisition The Product Team Plan Weakness
organization ensures identifies the
that the software organization responsible
support organization is for software support,
involved in planning for but transition is not
transition and support. mentioned.
Ability 1 A group that is A group responsible for Strength
responsible for coordinating the
coordinating the transition to support
transition and support activities exists.
activities exists.
Ability 2 Responsibility for Responsibility for Strength
transition and support transition and support
activities is activities is
designated. designated.
Ability 3 Adequate resources are No mechanism exists for Weakness
provided for transition identifying resources
and support activities. required and for
ensuring that the needed
resources are provided
to the project.
Ability 4 The organization The Product Team Plan Strength
responsible for designates
providing support of the responsibility for
software products is providing support of the
identified no later than software products before
release of the release of the
solicitation. solicitation.
Ability 5 The software support The software support Strength
organization has a organization has a
complete inventory of complete inventory of
all software and related all software and related
items that are to be items that are to be
transitioned. transitioned.
Ability 6 Individuals performing The acquisition Observation
transition and support organization has no
activities have guidance regarding
experience or receive training or experience
training. requirements for project
participation.
Ability 7 The members of Orientation was not Weakness
organizations given.
interfacing with the
transition and support
activities receive
orientation on the
salient aspects of the
transition and support
activities.
Activity 1 The project team The product team Strength
documents its plans for documents its plans for
transition and support transition and support
activities. activities.
Activity 2 The project team's The product team's Strength
transition and support activities are being
activities are performed conducted in accordance
in accordance with its with its plans.
plans.
Activity 3 The project team No certification Weakness
transfers responsibility procedures to test the
for the software capability of the
products only after the support organization was
support organization provided.
demonstrates its
capability to modify and
support the software
products.
Activity 4 The project team The product team Strength
oversees the oversees the
configuration control of configuration management
the software products of the software
throughout the throughout transition.
transition.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
transition and support the status of activities
activities. for any of the key
process areas.
Verification The activities for While the Integrated Weakness
1 transition and support Product Team leader
are reviewed by reviews the status of
acquisition and software the contract and the
support organizations' contractor's cost and
management on a periodic schedule, he does not
basis. review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 transition and support leader reviews the
are reviewed by the status of the contract
project manager on both and the contractor's
a periodic and event- cost and schedule, he
driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 8.2
Transition and Support Findings for DSR
Display System Replacement
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition FAA has a written policy Strength
organization has a for transition and
written policy for the support.
transitioning of
software products to the
support organization.
Commitment 2 The acquisition The Product Team Plan Strength
organization ensures identifies the
that the software individual responsible
support organization is for support planning.
involved in planning for
transition and support.
Ability 1 A group that is The product team is Strength
responsible for responsible for
coordinating the transition and support
transition and support activities.
activities exists.
Ability 2 Responsibility for Responsibility is Strength
transition and support assigned to the product
activities is team for transition and
designated. support activities.
Ability 3 Adequate resources are No mechanism exists for Weakness
provided for transition identifying resources
and support activities. required and for
ensuring that the needed
resources are provided
to the project.
Ability 4 The organization Although DSR is in the Weakness
responsible for development stage, the
providing support of the support organization has
software products is not been identified.
identified no later than
release of the
solicitation.
Ability 5 The software support While documents are Weakness
organization has a planned for
complete inventory of configuration audits as
all software and related part of the transition
items that are to be process, a complete
transitioned. inventory does not
exist.
Ability 6 Individuals performing The acquisition Observation
transition and support organization has no
activities have guidance regarding
experience or receive training or experience
training. requirements for project
participation.
Ability 7 The members of While officials stated Observation
organizations that orientation had
interfacing with the occurred, they could not
transition and support provide documents to
activities receive support this.
orientation on the
salient aspects of the
transition and support
activities.
Activity 1 The project team The Program Strength
documents its plans for Implementation Plan
transition and support contains the plans for
activities. transition and support.
Activity 2 The project team's Transition planning is Weakness
transition and support being delayed pending a
activities are performed decision on who will
in accordance with its provide maintenance.
plans.
Activity 3 The project team There is a plan for the Strength
transfers responsibility support organization to
for the software demonstrate its
products only after the capabilities prior to
support organization transition of
demonstrates its responsibilities.
capability to modify and
support the software
products.
Activity 4 The project team The product team Strength
oversees the oversees the contractor,
configuration control of who will maintain
the software products configuration management
throughout the throughout the
transition. transition.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
transition and support the status of activities
activities. for any of the key
process areas.
Verification The activities for While the Integrated Weakness
1 transition and support Product Team leader
are reviewed by reviews the status of
acquisition and software the contract and the
support organizations' contractor's cost and
management on a periodic schedule, he does not
basis. review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 transition and support leader reviews the
are reviewed by the status of the contract
project manager on both and the contractor's
a periodic and event- cost and schedule, he
driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 8.3
Transition and Support Findings for NIMS
NAS Infrastructure Management System
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition The Acquisition Strength
organization has a Management System is the
written policy for the written policy.
transitioning of
software products to the
support organization.
Commitment 2 The acquisition The software support Strength
organization ensures organization is
that the software represented in the
support organization is product team.
involved in planning for
transition and support.
Ability 1 A group that is Officials gave Weakness
responsible for conflicting answers as
coordinating the to who is responsible
transition and support for coordinating the
activities exists. transition and support
activities, and they
could not provide
documentation that
formally designates
responsibility.
Ability 2 Responsibility for Responsibility for Strength
transition and support transition and support
activities is activities is
designated. designated.
Ability 3 Adequate resources are It is too early in the Not rated
provided for transition project's cycle to
and support activities. evaluate this ability.
Ability 4 The organization It is too early in the Not rated
responsible for project's cycle to
providing support of the evaluate this ability.
software products is
identified no later than
release of the
solicitation.
Ability 5 The software support It is too early in the Not rated
organization has a project's cycle to
complete inventory of evaluate this ability.
all software and related
items that are to be
transitioned.
Ability 6 Individuals performing It is too early in the Not rated
transition and support project's cycle to
activities have evaluate this ability.
experience or receive
training.
Ability 7 The members of It is too early in the Not rated
organizations project's cycle to
interfacing with the evaluate this ability.
transition and support
activities receive
orientation on the
salient aspects of the
transition and support
activities.
Activity 1 The project team It is too early in the Not rated
documents its plans for project's cycle to
transition and support evaluate this activity.
activities.
Activity 2 The project team's It is too early in the Not rated
transition and support project's cycle to
activities are performed evaluate this activity.
in accordance with its
plans.
Activity 3 The project team It is too early in the Not rated
transfers responsibility project's cycle to
for the software evaluate this activity.
products only after the
support organization
demonstrates its
capability to modify and
support the software
products.
Activity 4 The project team It is too early in the Not rated
oversees the project's cycle to
configuration control of evaluate this activity.
the software products
throughout the
transition.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the to determine the status
transition and support of activities for any of
activities. the key process areas.
Verification The activities for While the Integrated Weakness
1 transition and support Product Team leader
are reviewed by reviews the status of
acquisition and software the contract and the
support organizations' contractor's cost and
management on a periodic schedule, he does not
basis. review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 transition and support leader reviews the
are reviewed by the status of the contract
project manager on both and the contractor's
a periodic and event- cost and schedule, he
driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 8.4
Transition and Support Findings for VSCS
Voice Switching and Control System
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition A written policy for Observation
organization has a transition exists;
written policy for the however, the officials
transitioning of interviewed did not
software products to the identify it.
support organization.
Commitment 2 The acquisition The acquisition Strength
organization ensures organization ensured
that the software that the software
support organization is support organization was
involved in planning for involved in planning for
transition and support. transition and support.
Ability 1 A group that is The product team is Strength
responsible for responsible for
coordinating the coordinating transition
transition and support and support activities.
activities exists.
Ability 2 Responsibility for Responsibility for Strength
transition and support transition and support
activities is activities is
designated. designated.
Ability 3 Adequate resources are No mechanism exists for Weakness
provided for transition identifying the
and support activities. resources required and
for ensuring that the
needed resources are
provided to the project.
Ability 4 The organization The organization Strength
responsible for responsible was
providing support of the identified before the
software products is release of the
identified no later than solicitation.
release of the
solicitation.
Ability 5 The software support The software support Weakness
organization has a organization does not
complete inventory of have a complete
all software and related inventory of all
items that are to be software and related
transitioned. items that are to be
transitioned.
Ability 6 Individuals performing The acquisition Observation
transition and support organization has no
activities have guidance regarding
experience or receive training or experience
training. requirements for project
participation.
Ability 7 The members of While officials said Observation
organizations that members of
interfacing with the organizations
transition and support interfacing with the
activities receive transition and support
orientation on the activities received
salient aspects of the orientation, they could
transition and support not provide
activities. documentation to support
this.
Activity 1 The project team Only a draft plan Weakness
documents its plans for exists.
transition and support
activities.
Activity 2 The project team's There is no plan for Weakness
transition and support transition and support,
activities are performed therefore, the activity
in accordance with its cannot be performed in
plans. accordance with it.
Activity 3 The project team No demonstrations were Weakness
transfers responsibility held for the support
for the software organization to
products only after the demonstrate its
support organization capability to support
demonstrates its the software products.
capability to modify and
support the software
products.
Activity 4 The project team Since there is no Weakness
oversees the transition and support
configuration control of plan, there is no
the software products evidence that this
throughout the activity is being
transition. performed.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the or used to determine the
transition and support status of activities for
activities. any of the key process
areas.
Verification The activities for While the Integrated Weakness
1 transition and support Product Team leader
are reviewed by reviews the status of
acquisition and software the contract and the
support organizations' contractor's cost and
management on a periodic schedule, he does not
basis. review the status of the
activities that are
required to be performed
for any of the key
processes.
Verification The activities for While the product team Weakness
2 transition and support leader reviews the
are reviewed by the status of the contract
project manager on both and the contractor's
a periodic and event- cost and schedule, he
driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 8.5
Transition and Support Findings for WARP
Weather and Radar Processor
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition There is a written Strength
organization has a policy for transition
written policy for the and support.
transitioning of
software products to the
support organization.
Commitment 2 The acquisition The support organization Strength
organization ensures is part of the product
that the software team and is involved in
support organization is planning for transition.
involved in planning for
transition and support.
Ability 1 A group that is The product team has the Strength
responsible for responsibility for
coordinating the coordinating the
transition and support transition and support
activities exists. activities.
Ability 2 Responsibility for Responsibility for Strength
transition and support transition and support
activities is activities is designated
designated. to the product team.
Ability 3 Adequate resources are No mechanism exists for Weakness
provided for transition identifying and for
and support activities. ensuring that the needed
resources are provided
to the project.
Ability 4 The organization The organization Weakness
responsible for responsible for
providing support of the providing support of the
software products is software products was
identified no later than not chosen before the
release of the release of the
solicitation. solicitation.
Ability 5 The software support It is too early in the Not rated
organization has a project's development
complete inventory of for it to have developed
all software and related an inventory.
items that are to be
transitioned.
Ability 6 Individuals performing The acquisition Observation
transition and support organization has no
activities have guidance regarding
experience or receive training or experience
training. requirements for project
participation.
Ability 7 The members of It is too early in the Not rated
organizations project's development
interfacing with the for it to define
transition and support support.
activities receive
orientation on the
salient aspects of the
transition and support
activities.
Activity 1 The project team The Product Team Plan Strength
documents its plans for documents the initial
transition and support plans for transition and
activities. support activities.
Activity 2 The project team's It is too early in the Not rated
transition and support project's development
activities are performed for it to be evaluated
in accordance with its against this key
plans. practice.
Activity 3 The project team Officials gave Observation
transfers responsibility conflicting answers as
for the software to whether or not the
products only after the demonstration has been
support organization planned.
demonstrates its
capability to modify and
support the software
products.
Activity 4 The project team It is too early in the Not rated
oversees the project's development
configuration control of for it to be evaluated
the software products against this key
throughout the practice.
transition.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
transition and support the status of activities
activities. for any of the key
process areas.
Verification The activities for While the Integrated Weakness
1 transition and support Product Team leader
are reviewed by reviews the status of
acquisition and software the contract and the
support organizations' contractor's cost and
management on a periodic schedule, he does not
basis. review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 transition and support leader reviews the
are reviewed by the status of the contract
project manager on both and the contractor's
a periodic and event- cost and schedule, he
driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
CONCLUSIONS
---------------------------------------------------------- Chapter 8:2
FAA's transition and support process for its ATC modernization
suffers from weaknesses which render it undefined and undisciplined.
In light of FAA's enormous investment in ATC-related software and the
fact that over 65 percent of the life cycle cost of software is
incurred during maintenance, it is essential that these weaknesses be
corrected.
ACQUISITION RISK MANAGEMENT
============================================================ Chapter 9
SEI defines risk as the possibility of suffering a loss. The purpose
of acquisition risk management is to formally identify risks as early
as possible and adjust the acquisition to mitigate those risks.
According to the SA-CMM, an effective risk management process, among
other things, includes (1) having a written policy on acquisition
risk management, (2) developing a software acquisition risk
management plan, (3) conducting software risk management activities
in accordance with the plan (e.g., identifying risks, taking
mitigation actions, and tracking risk mitigation actions to
completion), and (4) measuring and reporting on the status of
acquisition risk management activities to management.
ATC MODERNIZATION SOFTWARE RISK
MANAGEMENT IS INEFFECTIVE
---------------------------------------------------------- Chapter 9:1
FAA is not effectively performing ATC modernization software
acquisition risk management. Although FAA has a written policy on
risk management, it has many weaknesses in this KPA. For example,
most teams have no risk management plan, and the one team that has a
plan failed to follow it. As a result, the teams are not identifying
risks, taking risk mitigation actions, and tracking risk mitigation
actions to completion. Moreover, none of the teams measure and
report to management on the status of acquisition risk management
activities. Consequently, management is not in a position to correct
problems promptly.
Figure 9.1 provides a comprehensive listing of the five projects'
strengths, weaknesses, and observations for the acquisition risk
management KPA. The specific findings supporting the practice
ratings in figure 9.1 are in tables 9.1 through 9.5.
Figure 9.1: Acquisition Risk
Management Summary
(See figure in printed
edition.)
(See figure in printed
edition.)
Table 9.1
Acquisition Risk Management Findings for
ARTSIIIE
Automated Radar Terminal System IIIE
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------ --------------------------- --------------------------- --------
Commitment 1 The acquisition There is a policy for Strength
organization has a written acquisition risk
policy for the management management.
of acquisition risk.
Ability 1 A group that is responsible No group is assigned Weakness
for coordinating acquisition risk management
acquisition risk management tasks.
activities exists.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for acquisition identifying resources
risk management activities. required and for ensuring
that the needed resources
are provided to the
project.
Ability 3 Individuals performing The acquisition Observat
acquisition risk management organization has no ion
activities have experience guidance regarding training
or receive required or experience requirements
training. for project participation.
Activity 1 Risk identification, No risk identification, Weakness
analysis, and mitigation analysis, or mitigation
activities are integrated activities are integrated
into software acquisition into software acquisition
planning. planning.
Activity 2 The Software Risk There is no risk management Weakness
Management Plan is plan.
developed according to the
project's defined software
acquisition process.
Activity 3 The project's acquisition No software risk management Weakness
risk management activities plan exists, therefore, the
are performed in accordance activities are not
with its Software Risk performed in accordance
Management Plan. with it.
Activity 4 The project team Officials stated that the Observat
identifies, analyzes, and product team identifies, ion
takes appropriate software analyzes, and mitigates
risk mitigation actions risks, but could not
during acquisition provide documents to
planning. support this.
Activity 5 Risk identification, Officials could provide no Weakness
analysis, and mitigation evidence of risk
are conducted as an identification, analysis,
integral part of the and mitigation being
project performance conducted as part of
management and contract project and contract
performance management management.
processes.
Activity 6 Software risk mitigation Risks are not tracked to Weakness
actions are tracked to completion.
completion.
Measurement Measurements are made and No internal process Weakness
1 used to determine the measurements are taken and
status of the acquisition used to determine the
risk management activities. status of activities for
any of the key process
areas.
Measurement Resources expended for FAA does not track Weakness
2 acquisition risk management resources.
activities are recorded and
tracked.
Verification The activities for While the Integrated Weakness
1 acquisition risk management Product Team leader reviews
are reviewed by acquisition the status of the contract
organization management on and the contractor's cost
a periodic basis. and schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key process
areas.
Verification The activities for While the product team Weakness
2 acquisition risk management leader reviews the status
are reviewed by the project of the contract and the
manager on both a periodic contractor's cost and
and event-driven basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key process
areas.
--------------------------------------------------------------------------------
Table 9.2
Acquisition Risk Management Findings for
DSR
Display System Replacement
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition There is a written Strength
organization has a policy for acquisition
written policy for the risk management.
management of
acquisition risk.
Ability 1 A group that is The product team is Strength
responsible for responsible for
coordinating acquisition acquisition risk
risk management management.
activities exists.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for acquisition identifying resources
risk management required and for
activities. ensuring that the needed
resources are provided
to the project.
Ability 3 Individuals performing The acquisition Observation
acquisition risk organization has no
management activities guidance regarding
have experience or training or experience
receive required requirements for project
training. participation.
Activity 1 Risk identification, Risk identification, Strength
analysis, and mitigation analysis, and mitigation
activities are are integrated into the
integrated into software software acquisition
acquisition planning. planning.
Activity 2 The Software Risk While there is a risk Weakness
Management Plan is management plan that
developed according to addresses software at a
the project's defined high level, there is no
software acquisition software risk management
process. plan.
Activity 3 The project's Because there is no Weakness
acquisition risk software risk management
management activities plan, activities are not
are performed in performed in accordance
accordance with its with it.
Software Risk Management
Plan.
Activity 4 The project team The risk identification Weakness
identifies, analyzes, activities are not well
and takes appropriate defined. The product
software risk mitigation team used risk
actions during interchangeably with
acquisition planning. currently identified
problems, action items,
issues, and schedule
tracking.
Activity 5 Risk identification, What the product team Strength
analysis, and mitigation considers risks
are conducted as an (problems, action items,
integral part of the issues, etc.) are
project performance identified, analyzed,
management and contract and mitigated.
performance management
processes.
Activity 6 Software risk mitigation Software risk mitigation Weakness
actions are tracked to is not tracked to
completion. completion.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
acquisition risk the status of activities
management activities. for any of the key
process areas.
Measurement 2 Resources expended for FAA does not track Weakness
acquisition risk resources.
management activities
are recorded and
tracked.
Verification The activities for While the Integrated Weakness
1 acquisition risk Product Team leader
management are reviewed reviews the status of
by acquisition the contract and the
organization management contractor's cost and
on a periodic basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 acquisition risk leader reviews the
management are reviewed status of the contract
by the project manager and the contractor's
on both a periodic and cost and schedule, he
event-driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 9.3
Acquisition Risk Management Findings for
NIMS
NAS Infrastructure Management System
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition The Acquisition Weakness
organization has a Management System is the
written policy for the written policy for
management of acquisition risk
acquisition risk. management, but it does
not address software.
Ability 1 A group that is Both the Integrated Strength
responsible for Program Plan and the
coordinating acquisition Product Team Plan assign
risk management the risk management plan
activities exists. activities to the team.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for acquisition identifying resources
risk management required and for
activities. ensuring that the needed
resources are provided
to the project.
Ability 3 Individuals performing The acquisition Observation
acquisition risk organization has no
management activities guidance regarding
have experience or training or experience
receive required requirements for project
training. participation.
Activity 1 Risk identification, Acquisition risk Strength
analysis, and mitigation management is part of
activities are acquisition planning as
integrated into software called for in both the
acquisition planning. Integrated Program Plan
and the Product Team
Plan.
Activity 2 The Software Risk No software risk Weakness
Management Plan is management plan exists.
developed according to The risk management plan
the project's defined does not address
software acquisition software.
process.
Activity 3 The project's There is no software Weakness
acquisition risk risk management plan,
management activities therefore, activities
are performed in cannot be performed in
accordance with its accordance with it.
Software Risk Management
Plan.
Activity 4 The project team The project team does Weakness
identifies, analyzes, not identify, analyze,
and takes appropriate or mitigate software
software risk mitigation risks.
actions during
acquisition planning.
Activity 5 Risk identification, While plans call for Not rated
analysis, and mitigation risk identification,
are conducted as an analysis, and
integral part of the mitigation, the project
project performance has not reached a stage
management and contract where this activity
performance management applies.
processes.
Activity 6 Software risk mitigation A risk tracking system Observation
actions are tracked to is being developed as
completion. called for in the
Integrated Program Plan.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the to determine the status
acquisition risk of activities for any of
management activities. the key process areas.
Measurement 2 Resources expended for FAA does not track Weakness
acquisition risk resources.
management activities
are recorded and
tracked.
Verification The activities for While the Integrated Weakness
1 acquisition risk Product Team leader
management are reviewed reviews the status of
by acquisition the contract and the
organization management contractor's cost and
on a periodic basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 acquisition risk leader reviews the
management are reviewed status of the contract
by the project manager and the contractor's
on both a periodic and cost and schedule, he
event-driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 9.4
Acquisition Risk Management Findings for
VSCS
Voice Switching and Control System
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition FAA Order 1810.1F is the Strength
organization has a policy for acquisition
written policy for the risk management.
management of
acquisition risk.
Ability 1 A group that is The product team leader Strength
responsible for is the acquisition risk
coordinating acquisition manager.
risk management
activities exists.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for acquisition identifying the
risk management resources required and
activities. for ensuring that the
needed resources are
provided to the project.
Ability 3 Individuals performing The acquisition Observation
acquisition risk organization has no
management activities guidance regarding
have experience or training or experience
receive required requirements for project
training. participation.
Activity 1 Risk identification, The risk management plan Weakness
analysis, and mitigation does not call for risk
activities are activities to be
integrated into software integrated into software
acquisition planning. acquisition planning.
Activity 2 The Software Risk There is a software risk Strength
Management Plan is management plan.
developed according to
the project's defined
software acquisition
process.
Activity 3 The project's Although there is a risk Weakness
acquisition risk management plan, there
management activities is no evidence that
are performed in acquisition risk
accordance with its management is performed
Software Risk Management in accordance with the
Plan. plan.
Activity 4 The project team The project team does Weakness
identifies, analyzes, not identify, analyze,
and takes appropriate and take appropriate
software risk mitigation actions during
actions during acquisition planning.
acquisition planning.
Activity 5 Risk identification, The product team does Weakness
analysis, and mitigation not ensure that risks
are conducted as an are identified, or that
integral part of the analyses and mitigations
project performance are conducted as an
management and contract integral part of project
performance management performance management
processes. and contract performance
management.
Activity 6 Software risk mitigation There is no evidence Weakness
actions are tracked to that risk mitigation is
completion. tracked to completion.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the or used to determine the
acquisition risk status of activities for
management activities. any of the key process
areas.
Measurement 2 Resources expended for FAA does not track Weakness
acquisition risk resources.
management activities
are recorded and
tracked.
Verification The activities for While the Integrated Weakness
1 acquisition risk Product Team leader
management are reviewed reviews the status of
by acquisition the contract and the
organization management contractor's cost and
on a periodic basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 acquisition risk leader reviews the
management are reviewed status of the contract
by the project manager and the contractor's
on both a periodic and cost and schedule, he
event-driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
Table 9.5
Acquisition Risk Management Findings for
WARP
Weather and Radar Processor
--------------------------------------------------------------------------------
Key Practice Finding Rating
------------- ------------------------ ------------------------ -------------
Commitment 1 The acquisition FAA Order 1810.1F is the Strength
organization has a written policy.
written policy for the
management of
acquisition risk.
Ability 1 A group that is The product team is Strength
responsible for responsible for
coordinating acquisition acquisition risk
risk management management activities.
activities exists.
Ability 2 Adequate resources are No mechanism exists for Weakness
provided for acquisition identifying and for
risk management ensuring that the needed
activities. resources are provided
to the project.
Ability 3 Individuals performing The acquisition Observation
acquisition risk organization has no
management activities guidance regarding
have experience or training or experience
receive required requirements for project
training. participation.
Activity 1 Risk identification, Risk identification, Strength
analysis, and mitigation analysis, and mitigation
activities are activities are
integrated into software integrated into software
acquisition planning. acquisition planning.
Activity 2 The Software Risk Although there is a risk Weakness
Management Plan is management plan, it
developed according to incorrectly identifies
the project's defined risks as currently
software acquisition identified problems,
process. action items, issues,
etc.
Activity 3 The project's While issues are Weakness
acquisition risk reviewed at team
management activities meetings, they are not
are performed in specifically identified
accordance with its and managed as risks.
Software Risk Management
Plan.
Activity 4 The project team The product team Strength
identifies, analyzes, identifies and analyzes
and takes appropriate risks as defined in the
software risk mitigation risk management plan and
actions during takes appropriate
acquisition planning. actions during
acquisition planning.
Activity 5 Risk identification, The product team Strength
analysis, and mitigation identifies, analyzes,
are conducted as an and mitigates risks (as
integral part of the defined in the risk
project performance management plan) as part
management and contract of project and contract
performance management performance management.
processes.
Activity 6 Software risk mitigation Software risks as Strength
actions are tracked to defined in the risk
completion. management plan are
tracked to completion.
Measurement 1 Measurements are made No internal process Weakness
and used to determine measurements are taken
the status of the and used to determine
acquisition risk the status of activities
management activities. for any of the key
process areas.
Measurement 2 Resources expended for FAA does not track Weakness
acquisition risk resources.
management activities
are recorded and
tracked.
Verification The activities for While the Integrated Weakness
1 acquisition risk Product Team leader
management are reviewed reviews the status of
by acquisition the contract and the
organization management contractor's cost and
on a periodic basis. schedule, he does not
review the status of the
activities that are
required to be performed
for any of the key
process areas.
Verification The activities for While the product team Weakness
2 acquisition risk leader reviews the
management are reviewed status of the contract
by the project manager and the contractor's
on both a periodic and cost and schedule, he
event-driven basis. does not review the
status of the activities
that are required to be
performed for any of the
key process areas.
--------------------------------------------------------------------------------
CONCLUSIONS
---------------------------------------------------------- Chapter 9:2
FAA's software acquisition risk management process has many
weaknesses and is, therefore, undefined and undisciplined. To become
an effective acquirer of software, FAA must adopt a more structured,
rigorous, and disciplined approach to ATC modernization software
acquisition risk management.
FAA LACKS AN EFFECTIVE APPROACH
FOR IMPROVING ATC MODERNIZATION
SOFTWARE ACQUISITION PROCESSES
=========================================================== Chapter 10
In order to be effective, the organization responsible for improving
software acquisition processes must have (1) organizational and/or
budgetary authority over units acquiring software; and (2) a
comprehensive plan to guide software acquisition process improvement
efforts and measure progress on each. At FAA, responsibility for ATC
software acquisition process maturity and improvement has been
assigned to three organizational entities over the last 4 years, and
currently is assigned to FAA's Software Engineering Process Group
(SEPG), a multilevel committee structure chaired by a member of FAA's
Chief Information Officer's (CIO) staff. However, the SEPG, like the
entities previously responsible for software acquisition process
improvement, has neither organizational nor budgetary authority over
the ATC modernization product teams that acquire software. Further,
the SEPG does not have a software acquisition process improvement
plan to guide its maturation efforts.
As a result, management of FAA's software acquisition process
improvement effort is ad hoc and has not instilled software
acquisition process discipline into the ATC modernization program.
While the SEPG is now taking steps to establish the analytical basis
for a defined and comprehensive software process improvement plan,
that plan does not yet exist, no schedule has been established for
completing it, and the SEPG, like the organizational entities before
it that have failed to institute process improvements, has no
authority to implement or to enforce process change.
FAA ORGANIZATION RESPONSIBLE
FOR ATC SOFTWARE ACQUISITION
PROCESS IMPROVEMENT LACKS
AUTHORITY TO AFFECT CHANGE
--------------------------------------------------------- Chapter 10:1
FAA has been attempting to strengthen its software acquisition
processes since 1993. At that time it established the Software
Engineering Specialty Group to, among other things, incrementally
improve FAA's software acquisition processes, first establishing
repeatable processes [SEI maturity level 2], then defined processes
[SEI maturity level 3], and eventually managed processes [SEI
maturity level 4]. This group was to be FAA's focal point for
software process assessment and improvement initiatives through 2002,
and it developed a 10-year strategy and implementation plan for doing
so. It also produced guidance addressing software acquisition
management, software management indicators, and other
software-related processes and practices.
In 1995, FAA established the Office of the Chief Scientist for
Software Engineering under FAA's CIO to lead FAA's software-related
process improvement efforts, including software acquisition.
According to FAA officials, this change was made to strengthen
software process improvement through increased interaction with the
ATC modernization project offices. In May 1995, the Chief Scientist
reaffirmed SEI's CMM as the basis for all FAA process improvement and
set forth three broad "strategies for improving the quality of FAA
software."\1 The three strategies were (1) "improve the software
acquisition, development, certification, and maintenance processes of
the FAA and its suppliers,"\2 (2) "accelerate the adoption of open
systems, COTS, NDI,\3 re-engineering, and reuse by FAA programs
(without jeopardizing system safety or integrity),"\4 and (3)
"promote the use of best software engineering practices throughout
the FAA and its supplier community."\5 The Chief Scientist also began
about a dozen loosely defined process improvement activities.
Examples of these activities include participating in national and
international software engineering activities, interacting with
governmental and professional software engineering organizations,
meeting with FAA suppliers and aviation groups, and assessing
software engineering methods, tools, and best practices.
Another of the Chief Scientist's dozen activities was to form the
SEPG as FAA's "focal point for initiating, planning, motivating, and
facilitating the improvement of 'acquisition life cycle processes'
for software intensive systems."\6 Formed in October 1995, the SEPG
includes representatives from ATC modernization product teams and
their parent organizations as well as other FAA organizations, and it
is chaired by the Chief Scientist for Software Engineering. The SEPG
is directed by the Software Engineering Executive Committee (SEEC),
which is chaired by the Associate Administrator for Research and
Acquisition (i.e., the head of the ATC modernization), and is
composed of senior FAA executives. The SEEC is responsible for
recommending and providing guidance on software engineering issues.
Additionally, some of the ATC modernization product teams' parent
organizations have SEPGs.
None of the organizations responsible for ATC modernization software
acquisition process improvement over the last 4 years, including the
SEPG, have had organizational and/or budgetary authority over the ATC
modernization product teams that acquire ATC software. As a result,
neither the SEPG nor its predecessor organizations have been
positioned to effectively and efficiently implement and enforce
process changes. Instead, they can only attempt to encourage and
persuade product teams to undertake process improvement activities.
To illustrate the ineffectiveness of this management structure, the
Chief Scientist proposed that each product team earmark 5 percent of
its software-related budgets to implement approved process
improvement initiatives. According to the Chief Scientist, however,
the teams refused to spend product team money on the FAA-wide
software improvement activities being proposed. One product team
leader stated that the teams are understaffed and there is not enough
time and resources to support these activities.
--------------------
\1 "Strategies and Tactics for FAA to Improve Software, CIP Steering
Committee Meeting," May 16, 1995, page 8.
\2 See footnote 1.
\3 COTS is commercial, off-the-shelf; NDI is non-development item.
\4 See footnote 1.
\5 See footnote 1.
\6 In a document entitled "SEPG Purpose" Nov. 18, 1996, FAA defines
a software intensive system as "any system that is entirely software
or whose principle (sic) functionality depends on the correct
functioning of software."
FAA PLANNING FOR SOFTWARE
PROCESS IMPROVEMENT HAS NOT
BEEN EFFECTIVE
--------------------------------------------------------- Chapter 10:2
To properly focus and target software acquisition process
improvements, current process strengths and weaknesses (the
capability baseline) must be fully identified and assessed, and an
effective plan for systematically correcting weaknesses must be
developed. At a minimum, this plan should specify measurable goals,
objectives, milestones, and needed resources, should clearly assign
responsibility and accountability for accomplishing well- defined
tasks, and should be documented and approved by FAA leadership.
Despite 4 years of software acquisition process improvement effort,
FAA has not effectively baselined FAA's ATC modernization software
acquisition processes, and has not developed a comprehensive plan for
software acquisition process improvement on the basis of this
baseline. In 1995, the Chief Scientist began an effort to assess
software acquisition processes, but completion of the effort was
delayed 8 months and, because it lacked the requisite depth and
scope, it could not be used to produce an effective baseline to guide
improvement activities. Subsequent plans to perform more detailed
and comprehensive assessments were dropped.
FAA also has no comprehensive plan for software acquisition process
improvement. As a proxy, the Chief Scientist claims that a variety
of documents produced and activities conducted over the last 2 years
collectively form a complete and comprehensive plan. These include
(1) a document containing the "preliminary process improvement
recommendations" of a process improvement working group dated
September 4, 1996, (2) minutes of SEPG and SEEC meetings, (3)
briefings on software process improvement activities, and (4) the
business plan for the ATC modernization organization. However, these
documents and activities, which only briefly address process
improvement initiatives, cite broad strategies, and sometimes mention
general schedules and resource needs, do not constitute an effective
plan. They do not specify well-defined and measurable goals and
milestones, assign responsibility, identify resource needs for each
initiative, or define how and when process improvements will actually
be implemented and enforced. Further, without a capability maturity
baseline, there is no analytical basis for selecting these
initiatives. According to product team officials, the modernization
effort has no software acquisition process improvement plan.
IMPROVEMENT INITIATIVES HAVE
THUS FAR NOT INSTILLED SOFTWARE
PROCESS DISCIPLINE
--------------------------------------------------------- Chapter 10:3
Since 1995, the Chief Scientist has planned or initiated a dozen
activities. Some have never been started, some are behind schedule,
and several have proceeded according to plan. For example, while the
Chief Scientist planned to complete an assessment of ATC
modernization software acquisition capabilities using SEI level 2 and
level 3 requirements by December 1996 and June 1997, respectively,
these efforts were never performed. Other efforts are behind
schedule. For example, software engineering policies, guidance, and
standards that were to be issued by September 1996 are now scheduled
for issuance the third quarter of calendar year 1997; and a software
life cycle tool that was to be developed by October 1996 has been
postponed indefinitely.
Several efforts have met their milestones and begun to build a
foundation for undertaking process improvements. For example, a
software engineering training plan was developed in May 1996, 1 month
ahead of schedule; product teams have been trained to use the SEI
software development CMM and the associated capability evaluation
methodology to evaluate contractors' capabilities to develop
software; one product team used the results of a CMM evaluation as
part of its source selection criteria; and, according to the Chief
Scientist, hundreds of members of various ATC modernization product
teams have been trained in software management techniques, such as
defining software processes, using software management indicators,
estimating software costs, and using standards such as open systems
standards.
Other FAA activities relating to software process improvement include
establishing an SEPG infrastructure, pilot testing selected software
product and process metrics, creating a software quality assurance
capability, reengineering configuration management processes, and
studying software cost estimation tools. In addition, the SEPG and
the Chief Scientist are now taking steps to establish an analytical
basis for software acquisition process improvement. For example, the
SEPG and the Chief Scientist intend to use the results of this GAO
evaluation as the basis for planning future software acquisition
process improvement activities. Also, the SEPG has begun to analyze
the interrelationships among FAA's various process improvement
activities, link the activities to strategic goals and measurable
outcomes, and explore ways to manage these activities in a
coordinated fashion. Further, the Chief Scientist intends to
formalize the results of these steps in a comprehensive plan of
action, although no schedule has been set for doing so.
However, none of these activities or steps, either individually or in
the aggregate, have yet resulted in more repeatable, better defined,
more disciplined ATC modernization software acquisition processes.
In interviews, product team and SEPG officials confirmed that the
software acquisition processes had not yet been improved. Instead,
the activities have begun to lay a foundation for potential process
improvements in the future.
CONCLUSIONS
--------------------------------------------------------- Chapter 10:4
FAA has neither an effective management structure nor plan of action
for improving its software acquisition processes. As a result,
software acquisition processes will remain immature and will not
support effective, efficient, and economical acquisition of
mission-critical software costing billions of dollars. Until
responsibility and accountability for software acquisition process
improvement is assigned to an FAA organizational entity with the
requisite authority to affect process change, and until this
organizational entity pursues a plan for change based on a complete
and objective assessment of current process strengths and weaknesses,
it is unlikely that significant ATC modernization software
acquisition process improvements will be made, and ATC software
acquisition processes will remain immature.
OVERALL CONCLUSIONS AND
RECOMMENDATIONS
=========================================================== Chapter 11
Leading software acquisition organizations rely on defined and
disciplined software acquisition processes to deliver promised
software capabilities on time and within budget, first on a
project-by-project basis, and later, as the organization's processes
become more mature, consistently across the institution. FAA's ATC
modernization software acquisition processes are ad hoc and sometimes
chaotic, and are not repeatable even on a project-by-project basis.
As a result, FAA's success or failure in acquiring ATC software
depends largely on specific individuals, rather than on well-defined
and disciplined software acquisition management practices. This
greatly reduces the probability that software-intense ATC
modernization projects will consistently perform as intended and be
delivered on schedule and within budget. For FAA to mature beyond
this initial level, it must implement basic management controls and
instill self-discipline in its software projects.
FAA recognizes the importance of software process maturity and the
need to improve its software acquisition processes. However, it
lacks an effective management structure for accomplishing this
because the FAA organization responsible for process improvement, the
SEPG, lacks the authority to implement management controls and
instill process discipline within the ATC modernization software
acquiring organizations. Additionally, despite 4 years of FAA
process improvement activity, no well-targeted, comprehensive, and
coordinated plan of action for software acquisition process
improvement exists.
RECOMMENDATIONS
--------------------------------------------------------- Chapter 11:1
Given the importance and the magnitude of information technology at
FAA, we reiterate our recent recommendation that a CIO organizational
structure similar to the department-level CIOs as prescribed in the
Clinger-Cohen Act of 1996 be established for FAA.\1
To improve FAA's software acquisition capability for its ATC
modernization and thereby take the first step in institutionalizing
mature acquisition processes, we recommend that the Secretary of
Transportation direct the FAA Administrator to:
-- assign responsibility for software acquisition process
improvement to FAA's CIO;
-- provide FAA's CIO the authority needed to implement and enforce
ATC modernization software acquisition process improvement;
-- require the CIO to develop and implement a comprehensive plan
for ATC modernization software acquisition process improvement
that is based on the software capability evaluation results
contained in this report and specifies measurable goals and time
frames, prioritizes initiatives, estimates resource
requirements, and assigns roles and responsibilities;
-- allocate adequate resources to ensure that these improvement
efforts are implemented and enforced; and
-- require that, before being approved, every ATC modernization
acquisition project have software acquisition processes that
satisfy at least SA-CMM level 2 requirements.
--------------------
\1 Air Traffic Control: Complete and Enforced Architecture Needed
for FAA Systems Modernization (GAO/AIMD-97-30, Feb. 3, 1997).
AGENCY COMMENTS AND OUR
EVALUATION
--------------------------------------------------------- Chapter 11:2
In its written comments on a draft of this report, the Department of
Transportation recognized the importance of mature software
acquisition processes, agreed that FAA's processes are insufficiently
mature, acknowledged that FAA process improvement activities have yet
to produce greater process discipline, and reaffirmed FAA's
commitment to improving its software acquisition capabilities using
the SA-CMM. However, the Department did not state what, if any,
specific action it would take on our recommendations. Additionally,
the Department expressed several concerns, each of which is presented
below, along with our rebuttal.
First, the Department stated that FAA does not separately procure
software for its ATC systems. Rather, it procures systems that use
software as a major component. Therefore, its policies and
procedures (i.e., processes) are "geared" to system acquisitions, and
evaluating only the software-related aspects of its acquisition
processes "is not an adequate approach."
GAO Rebuttal: All major system modernizations, like the ATC
modernization, involve the acquisition of hardware, software, and
firmware operating interdependently. However, as FAA's own
experience with the Advanced Automation System clearly proves, the
software component is the source of most system risk, and the
component most frequently associated with late deliveries, cost
increases, and performance shortfalls. Moreover, there is widespread
recognition throughout the computer industry that the billions of
dollars being invested in complex, real-time, fault tolerant systems,
like FAA ATC systems, are jeopardized by inadequate management
attention to software in general, and undisciplined, ill-defined
software acquisition and development processes in particular. This
is precisely why SEI developed its software-related CMMs, why the
CMMs have been endorsed and accepted throughout industry and
government, and why the scope of this evaluation focused on software
acquisition processes rather than on broader systems issues.
Further, the FAA system acquisition policies and procedures that the
Department references do not explicitly or adequately address
software issues. For example, they do not address software-specific
acquisition planning for such KPAs as contract tracking and
oversight, requirements development and management, and risk
management. Additionally, they do not provide for measuring and
verifying the performance of software-specific acquisition
activities.
Second, the Department commented that the SA-CMM "is not widely used,
adopted, or validated" and, while it has "significant merit, it is
certainly not to be taken as the same authoritative source for
process improvement guidance as the SW-CMM,\2 which has been in use
worldwide by thousands of organizations for several years."
GAO Rebuttal: This position is clearly inconsistent with the
Department's and FAA's stated commitment to using the SA-CMM as the
basis for efforts to improve FAA's software acquisition capabilities.
More important however is that this position is without substance.
The SA-CMM does not promulgate original or novel concepts of
debatable value. Instead, it presents as requisite processes and
practices those activities that common sense have validated as
essential to effective software acquisition. For example, it
requires disciplined requirements development and management,
solicitation, contract tracking and oversight, and evaluation. Our
evaluations have for years made the same points without rational
dispute. The SA-CMM simply provides a coherent framework and
standard terminology for these concepts. The findings in this
report, which have been corroborated by SEI, are compelling not
because of the age of the model used, but because the criticality of
the processes and practices examined is undeniable.
Third, the Department claimed that "GAO may have misapplied the
model" by (1) giving inadequate consideration to equivalent
alternative practices when determining whether SA-CMM specified
practices were performed (e.g., DSR system acquisition planning being
judged as an insufficient proxy for software acquisition planning
specified in the SA-CMM), (2) not effectively tailoring the SA-CMM to
focus only on project activities that occurred after the cancellation
of the Advanced Automation System, and (3) reporting evaluation
results in a way that "does not create an environment conducive to
process improvement" (i.e., reporting the results for each project,
rather than either aggregating the results or disguising the identity
of the projects).
GAO Rebuttal: We applied the model properly and correctly, and SEI
has attested to this. Every member of our evaluation team was
trained by SEI; the team leader was an SEI designated "lead
evaluator" and has been authorized by SEI to submit results for
inclusion in SEI studies; three senior SEI professionals, two of whom
are authors of the SA-CMM, participated in the evaluation to ensure
that the model was properly used; and SEI concurred with each
practice determination in the report (e.g., strength, weakness).
With respect to each of the Department's subsidiary points regarding
our application of the model:
(1)During the course of extensive interviews with FAA designated
officials, no evidence of reasonable alternative practices was
provided. If such evidence had been provided, we would have
considered it. For example, when FAA provided a system acquisition
plan for DSR as evidence of software acquisition planning, we
reviewed the document and found that it was not a reasonable
alternative practice because it did not adequately address the
software component of the acquisition.
(2)As agreed with SEI, those practices that were deemed inapplicable
were not rated, and those performed years ago were so designated.
Moreover, even if all practices predating the Advance Automation
System's cancellation were ignored, none of our conclusions and
recommendations would change.
(3)The model and evaluation method do not specify any reporting
format. In particular, they do not address whether results should be
reported for each project, or whether the identity of the projects
should be disguised or results reported only in the aggregate. Given
the mission-critical importance and billion dollar cost of these
projects, full disclosure of all relevant facts to the Congress and
the public is both warranted and appropriate.
Fourth, concerning FAA's software acquisition process improvement
efforts, the Department stated that the report does not sufficiently
appreciate the "progress made to date, the difficulties involved in
achieving that progress, and the time that it takes for . . .
changes of
this . . . magnitude." Specifically, the many efforts underway are
not a "hodge podge" of activities, but are "a very healthy sign of
the seriousness and enthusiasm" that FAA assigns to process
improvement and are "organized with respect to specific directorate
objectives." Also, since FAA's process improvement activities have
been underway for less than 2 years, it is too early to expect
results.
GAO Rebuttal: The Department's position that FAA's many process
improvement activities are not a "hodge podge," but rather are part
of an organized and coordinated comprehensive plan of action, is not
supported by the facts. While FAA began drafting a plan during the
course of our evaluation, it had no schedule for finalizing this
plan, and no analytical basis for the software acquisition process
improvement activities underway. Just as its software acquisition
processes lack maturity and discipline, so do its efforts to improve
these processes. Claims that FAA has been engaged in software
improvement efforts for less than 2 years, and thus it is too early
to evaluate results, are also unsupported. In fact, software
acquisition process maturity and improvement efforts began in 1993.
Since SEI published statistics show that the median time to improve
from SW-CMM level 1 to level 2 is 26 months, and from SW-CMM level 2
to level 3 is 17 months, it is entirely reasonable to expect FAA to
be able to demonstrate some improvement in its processes after 4
years.
Fifth, while the report states that the SEPG has neither the
organizational nor the budgetary authority to effectively implement
process change, the Department stated that its "understanding . . .
is that organizations do not normally give their SEPG authority over
product teams." In FAA's case, the SEPG provides advice and counsel
to the Software Engineering Executive Committee, which consists of
senior managers who have authority and responsibility to direct
process improvement actions. The SEPG is the committee's agent for
implementing these improvements.
GAO Rebuttal: The issue is not whether FAA's SEPG is organized as
the Department "understands" other SEPGs to be organized, but whether
the SEPG, or any FAA organizational entity responsible for
implementing and enforcing software process change, has the authority
needed to accomplish this task. Currently, no organizational entity
in FAA has the requisite authority. Accordingly, we have recommended
that a CIO organizational structure similar to the department-level
CIOs prescribed in the Clinger-Cohen Act of 1996 be established for
FAA, and that it be assigned the responsibility and resources needed
to affect and enforce software acquisition process improvement.
Sixth, the Department contends that the report "leads the reader to
erroneously conclude that the five programs reviewed are in trouble"
relative to their cost and schedule goals.
GAO Rebuttal: The report addresses the maturity of FAA's software
acquisition processes and concludes that these processes are ad hoc
and undisciplined, reducing the probability that software-intense ATC
modernization projects will consistently perform as intended and be
delivered on schedule and within budget. The report does not address
the overall status of the projects covered by GAO's review, and,
therefore, provides no basis for drawing conclusions about the
projects' overall cost and schedule performance.
(See figure in printed edition.)Appendix I
--------------------
\2 SW-CMM is SEI's software development capability maturity model.
COMMENTS FROM THE DEPARTMENT OF
TRANSPORTATION
=========================================================== Chapter 11
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II
ACCOUNTING AND INFORMATION
MANAGEMENT DIVISION,
WASHINGTON, D.C.
-------------------------------------------------------- Appendix II:1
Randolph C. Hite, Senior Assistant Director
Keith A. Rhodes, Technical Director
Leonard J. Latham, Technical Assistant Director
Suzanne M. Burns, Senior Information Systems Analyst
Madhav S. Panwar, Senior ADP/Telecommunications Analyst
Ira S. Sachs, Senior Information Systems Analyst
*** End of document. ***