Air Traffic Control: Improved Cost Information Needed to Make Billion
Dollar Modernization Investment Decisions (Chapter Report, 01/22/97,
GAO/AIMD-97-20).

In 1981, the Federal Aviation Administration (FAA) began an air traffic
control modernization program that the agency now expects will cost more
than $34 billion by 2003. The vast majority of these air traffic control
capital investment projects, both in terms of money and number, involve
software-intensive information acquisition, processing, and display
systems. GAO found that the program's cost estimating and accounting
practices are badly flawed, resulting in an absence of reliable cost and
financial information needed to make informed investment decisions. This
report examines the cost estimating and accounting practices that FAA
has used for its air traffic control project. GAO discusses whether (1)
air traffic control cost estimates are based on good estimating
processes and (2) actual air traffic control project costs are being
properly accumulated.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  AIMD-97-20
     TITLE:  Air Traffic Control: Improved Cost Information Needed to 
             Make Billion Dollar Modernization Investment
             Decisions
      DATE:  01/22/97
   SUBJECT:  Federal agency accounting systems
             Computer software
             Cost control
             Air traffic control systems
             Information resources management
             Cost analysis
             Accounting procedures
             Cost accounting
             Life cycle costs
IDENTIFIER:  FAA Voice Switching and Control System
             FAA Integrated Terminal Weather System
             FAA Airport Surface Detection Equipment Radar
             FAA Aviation Weather Observing System
             FAA Automated Terminal Doppler Radar Warning System
             FAA Standard Terminal Automation Replacement System
             FAA Display System Replacement
             FAA Airport Surveillance Radar Program
             FAA Wide Area Augmentation System
             FAA Display Channel Complex Rehost Project
             Software Capability Maturity Model
             DOT Departmental Accounting and Financial Information System
             FAA Air Traffic Control Modernization Program
             FAA Oceanic Display and Planning System
             FAA Acquisition Management System
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Report to the Secretary of Transportation

January 1997

AIR TRAFFIC CONTROL - IMPROVED
COST INFORMATION NEEDED TO MAKE
BILLION DOLLAR MODERNIZATION
INVESTMENT DECISIONS

GAO/AIMD-97-20

Air Traffic Control

(511402)


Abbreviations
=============================================================== ABBREV

  ACEIT - Automated Cost Estimating Integrated Tools
  APO - Office of Aviation Policy and Plans
  ARA - Associate Administrator for Research and Acquisitions
  ARINC - Aeronautical Radio Incorporated
  ASD-400 - Investment Analysis and Operations Research
  ASR-9 - Airport Surveillance Radar-9
  ATC - air traffic control
  ATS - Associate Administrator for Air Traffic Services
  CASE - Computer-Aided Software Engineering
  CBAS - Cost Benefit Analysis System
  CFO - Chief Financial Officer
  CMM - Capability Maturity Model
  COPS - Cost of Performance System
  CPMS - Cost Performance Management System
  DAFIS - Departmental Accounting and Financial Information System
  DCCR - Display Channel Complex Rehost
  DOT - Department of Transportation
  DSR - Display System Replacement
  FAA - Federal Aviation Administration
  FASAB - Federal Accounting Standards Advisory Board
  FMFIA - Federal Managers' Financial Integrity Act of 1982
  FMS - Financial Management System
  F&E - Facilities and Equipment
  GPS - Global Positioning System
  IPDS - Integrated Product Development System
  IPT - Integrated Product Team
  ISSS - Initial Sector Suite System
  JRC - Joint Resources Council
  OIG - Office of the Inspector General
  OMB - Office of Management and Budget
  O&M - Operations and Maintenance
  PCB&T - Personnel Compensation, Benefits, and Travel
  RE&D - Research, Engineering, and Development
  REDMACS - Research, Engineering, and Development Monitoring,
     Analysis, and Control System
  SEI - Software Engineering Institute
  SFFAS 4 - Statement of Federal Financial Accounting Standards no. 
     4
  SLIM - Software Life Cycle Intermediate Model
  STARS - Standard Terminal Automation Replacement System
  TRACON - Terminal Radar Approach Control
  VSCS - Voice Switching and Control System
  WAAS - Wide Area Augmentation System

Letter
=============================================================== LETTER


B-271530

January 22, 1997

The Honorable Federico Peï¿½a
Secretary of Transportation

Dear Mr.  Secretary: 

This report addresses the reliability of the cost information
critical to capital investment decision-making on air traffic control
projects.  Specifically, we evaluated the Federal Aviation
Administration's processes for estimating what projects will cost and
the related accounting for actual project costs. 

This report contains recommendations to you.  The head of a federal
agency is required by 31 U.S.C.  720 to submit a written statement on
actions taken on these recommendations.  You should send your
statement to the Senate Committee on Governmental Affairs and the
House Committee on Government Reform and Oversight within 60 days
after the date of this report.  You must also send the written
statement to the House and Senate Committees on Appropriations with
the agency's first request for appropriations made over 60 days after
the date of this report. 

We are providing copies of this report to the Subcommittees on
Transportation of the House and Senate Committees on Appropriations,
the House and Senate Committees on the Budget, the Director of the
Office of Management and Budget, the Administrator of the Federal
Aviation Administration, and other interested parties.  Copies will
also be made available to others upon request.  Please call me at
(202) 512-6412 if you have any questions concerning the report. 
Other contributors to this report are listed in appendix V. 

Sincerely yours,

Dr.  Rona B.  Stillman
Chief Scientist for Computers
 and Telecommunications


EXECUTIVE SUMMARY
============================================================ Chapter 0


   PURPOSE
---------------------------------------------------------- Chapter 0:1

Wisely managing new investments in capital assets, whether commercial
real estate, factory plant and equipment, or information technology,
requires complete and reliable information about alternative
investment options, such as what the investment is expected to cost
and whether actual costs are in line with expectations.  The Federal
Aviation Administration (FAA) is in the midst of a multibillion
dollar, mission-critical capital investment program aimed at
modernizing its aging air traffic control (ATC) infrastructure.  The
vast majority of these ATC capital investment projects, both in terms
of money and number, involve software-intensive information
acquisition, processing, and display systems.  Past GAO reports have
noted that ATC modernization projects routinely cost considerably
more than envisioned, in part because of poor cost estimates. 

Because of the importance of complete and reliable estimated and
actual cost information to ATC investment management decisions, GAO
examined FAA's ATC project cost estimating and cost accounting
practices.  Our objectives were to determine whether (1) ATC project
cost estimates are based on good estimating processes and (2) actual
ATC project costs are being properly accumulated. 


   BACKGROUND
---------------------------------------------------------- Chapter 0:2

Faced with growing air traffic volume and a deteriorating ATC
infrastructure, in 1981 FAA began an ambitious ATC modernization
program.  This program, which includes investments in both new ATC
facilities and related support as well as new and upgraded
software-intensive computer systems, totals over 200 separate
projects that FAA estimates will cost more than $34 billion between
1982 and 2003.  Of this total, 169 projects, estimated by FAA to cost
about $21 billion, are ATC information systems.  In fact, FAA expects
the cost to develop and deploy ATC information systems between now
and the year 2003 will be $6.9 billion, or about one-third, of the
information systems component of the modernization. 

Over the past 15 years, FAA's ATC modernization projects have
experienced substantial cost overruns, lengthy schedule delays, and
significant performance shortfalls.  To illustrate, the long-time
centerpiece of this modernization program--the Advanced Automation
System--was restructured in 1994 after estimated costs tripled from
$2.5 billion to $7.6 billion and delays in putting significantly
less-than-promised system capabilities into operation were expected
to run 8 years or more.  Because of the size, importance, complexity,
and poor track record of the modernization program, GAO designated it
a high-risk information technology initiative in 1995.\1

In 1994, GAO reported on how leading organizations improved mission
performance through information technology.  Among other matters, GAO
reported that successful organizations manage information system
projects as investments, continually assessing the quality of
projects' estimated costs and carefully monitoring projects' actual
costs against these estimates.\2 This "best practice" relative to
cost estimates has since been embodied in the Clinger-Cohen Act of
1996, which requires the selection of information technology
investments on the basis of competing projects' estimated costs,
benefits, and risks.  In addition, the Chief Financial Officers (CFO)
Act of 1990 requires federal agencies to maintain integrated
accounting and financial management systems that permit the
development and reporting of cost information and the systematic
measurement of performance. 

According to Carnegie Mellon University's Software Engineering
Institute (SEI), deriving credible estimates of software-based
systems' costs is a function of how thorough and disciplined an
organization's estimating processes are.  Accordingly, SEI has
published six institutional process requisites that organizations in
the business of building or acquiring software-intensive systems must
possess if they are to consistently produce reliable cost
estimates.\3 These are

  -- a corporate memory, or historical database(s), for cataloging
     cost estimates, revisions, reasons for revisions, actuals, and
     other descriptive information, such as any constraints or trends
     that affect the project;

  -- structured processes for estimating software size and the amount
     and complexity of existing software that can be reused;

  -- cost models calibrated/tuned to reflect demonstrated
     accomplishments on similar past projects;

  -- audit trails that record and explain the values used as cost
     model inputs;

  -- processes for dealing with externally imposed cost or schedule
     constraints in order to ensure the integrity of the estimating
     process; and

  -- data collection and feedback processes that foster capturing and
     correctly interpreting data from work performed. 

Additionally, SEI, along with other experts in systems and software
engineering and project management, advocates the acknowledgement and
full disclosure of the inherent imprecision of these estimates. 
Accordingly, these experts recommend qualifying early project
estimates by disclosing the level of uncertainty associated with
them, and making the estimate more precise as the project is
completed and the uncertainties are eliminated. 

Although a recent development, the Statement of Federal Financial
Accounting Standards no.  4 (SFFAS 4), Managerial Cost Accounting
Concepts and Standards for the Federal Government,\4 which took
effect October 1, 1996, requires full cost accounting by federal
agencies' reporting entities to support management decision-making. 
These standards build on the CFO Act of 1990, which requires that
agencies develop and maintain integrated accounting and financial
management systems in order to develop and report cost information
and systematically measure performance, among other functions.  In
addition, the Federal Managers' Financial Integrity Act of 1982
(FMFIA) requires that agency systems of internal control comply with
prescribed standards and provide reasonable assurances that among
other things, obligations and costs comply with applicable laws and
that revenues and expenditures applicable to agency operations are
recorded and accounted for properly. 


--------------------
\1 High-Risk Series:  An Overview (GAO/HR-95-1, Feb.  1995). 

\2 Executive Guide:  Improving Mission Performance Through Strategic
Information Management and Technology (GAO/AIMD-94-115, May 1994). 

\3 Checklists and Criteria for Evaluating the Cost and Schedule
Estimating Capabilities of Software Organizations (CMU/SEI-95-SR-005,
Jan.  1995). 

\4 The Federal Accounting Standards Advisory Board (FASAB) recommends
federal accounting standards to the Director of OMB, the Secretary of
the Treasury, and the Comptroller General.  Once approved, the
standards are issued by OMB and GAO. 


   RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3

FAA's ATC modernization program's cost estimating processes do not
satisfy recognized estimating requisites, and its cost accounting
practices do not provide for proper accumulation of actual project
costs.  The result is an absence of reliable project cost and
financial information that the Congress has legislatively specified
and that leading public-sector and private-sector organizations point
to as essential to making fully informed investment decisions among
competing ATC projects.  Without this information, the likelihood of
poor ATC investment decisions is increased, not only when a project
is initiated but also throughout its life cycle.  It also means that
the Congress does not have reliable cost information to use in making
funding decisions about FAA.  Such a situation is unacceptable when
making small investments, but it is especially egregious when making
multimillion or billion dollar investments in mission-critical ATC
systems. 

With respect to cost estimating, FAA fails to meet five of the six
process requisites that SEI's Checklists and Criteria for Evaluating
the Cost and Schedule Estimating Capabilities of Software
Organizations says should be institutionally entrenched and
consistently used for information technology projects.  In the
absence of such institutional policies to guide ATC project cost
estimating, FAA has adopted a cost estimating process that allows
each ATC project to approach cost estimating in whatever manner its
estimators choose.  The result is inconsistency in the rigor and
discipline with which ATC project cost estimates are derived, which
in turn means estimates of varying degrees of reliability.  In fact,
when comparing the approaches that six ATC projects used to derive
their current official life cycle cost estimates to SEI's
project-specific criteria entitled, A Manager's Checklist for
Validating Software Cost and Schedule Estimates,\5 GAO found that two
were too poorly documented to permit any comparative analysis, while
none of the remaining four satisfied all of the criteria SEI
associates with highly credible estimates.\6 Compounding these
estimating process weaknesses is FAA's practice of presenting cost
estimates as precise, point estimates.  By doing so, FAA fails to
disclose the estimates' inherent uncertainty and risks, thus further
limiting the estimates' decision-making value and credibility. 

With respect to cost accounting, FAA is not accumulating all ATC
project costs.  In fact, FAA does not have a cost accounting system
for capturing and reporting the full cost of its ATC projects. 
Instead, FAA decisionmakers use accounting and financial management
systems that omit relevant project costs, such as those associated
with FAA project management.  The result is that FAA cannot reliably
measure the ATC projects' actual cost performance against established
baselines, and cannot reliably use information relating to actual
cost experiences to improve future cost estimating efforts. 


--------------------
\5 CMU/SEI-95-SR-004, Jan.  1995. 

\6 Since the focus of this effort is to assess FAA's cost estimating
processes and not to validate the accuracy or completeness of the
estimates, we did not evaluate the quality of the estimates. 


   PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:4


      WEAK COST ESTIMATING
      PROCESSES AND PRACTICES
      UNDERMINE FAA'S ABILITY TO
      MAKE INFORMED INVESTMENT
      DECISIONS
-------------------------------------------------------- Chapter 0:4.1

The effectiveness of FAA's investment management process is in part a
function of the quality of the cost estimates used in the process. 
Accordingly, GAO assessed FAA's cost estimating processes and
practices at two levels.  First, on an institutional level, FAA's
cost estimating processes for ATC systems partially meet one and do
not meet the remaining five institutional process requisites that
experts say are embedded in leading information technology
development and acquisition organizations.\7 These six requisites,
which are listed in table 1, are the product of SEI's research of
leading government and private-sector estimating practices.  Table 1
also provides a summary assessment of the degree to which FAA
satisfies each requisite.  According to SEI, an organization must
address all six requisites in order to produce reliable estimates. 
The Clinger-Cohen Act emphasizes the need for reliable cost estimates
to allow effective information technology investment decision-making. 



                                Table 1
                
                 Summary of FAA's Satisfaction of SEI's
                        Institutional Requisites

                                    FAA institutional policies and
SEI requisites                      practices
----------------------------------  ----------------------------------
Corporate memory                    No

Sizing and reuse structure          No

Extrapolation using actual          No
performance

Audit trails                        Partial

Integrity in the face of dictated   No
limits

Data collection and feedback on     No
actual performance
----------------------------------------------------------------------
Second, on a project level, GAO reviewed six ATC projects (five of
the largest ATC projects and one small project) as case studies to
determine how these projects' current official life cycle cost
estimates were derived in the absence of any institutional processes
for estimating ATC projects' costs.  To do so, GAO compared the cost
estimating approaches used on each project to SEI criteria developed
expressly for project managers to use in deciding whether or not to
rely on an estimate.\8 GAO found that two of the six projects had
little or no documentation supporting the derivation of their
estimates.  The four remaining projects' estimating approaches were
documented, but none satisfied all the SEI criteria.  Of these four,
one satisfied most of the criteria SEI associates with a credible
cost estimate.  The other three failed to satisfy most of the SEI
criteria.  To FAA's credit, however, estimators on two of the
projects that failed to satisfy most of SEI's criteria are revising
life cycle cost estimates using more rigorous approaches. 

FAA's estimating process weaknesses are exacerbated by its portrayal
of ATC projects' cost estimates as firm, point estimates.  According
to software engineering experts, cost estimates are by definition
imprecise, particularly early in a system's development cycle, and
should be qualified to fully disclose this uncertainty.  By choosing
to present its ATC projects' cost estimates as discrete points, FAA
is improperly implying a level of precision that cannot be supported,
thereby further limiting the value of the estimates to congressional
and agency investment decisionmakers and potentially misleading them. 

During the course of GAO's review, FAA organizations initiated
several efforts to improve ATC cost estimating processes. 
Specifically, FAA's program assessment organization is exploring the
possibility of integrating cost estimating functions across FAA, and
is advocating FAA adoption of a standard tool for recording cost
model data.  Another group is testing a system for tracking a
project's actual operations and maintenance (O&M) costs.  The group's
goal is to eventually use this system's historical data to help
estimate future projects' O&M costs.  However, these separate
initiatives are not coordinated, relatively new, and not endorsed or
institutionalized by FAA. 


--------------------
\7 This evaluation was performed using SEI's Checklists and Criteria
for Evaluating the Cost and Schedule Estimating Capabilities of
Software Organizations. 

\8 This evaluation was performed using SEI's A Manager's Checklist
for Validating Software Cost and Schedule Estimates. 


      SHORTCOMINGS IN PROJECT COST
      ACCOUNTING IMPEDE FAA'S
      ABILITY TO EFFECTIVELY
      MANAGE ATC INVESTMENTS AND
      IMPROVE COST ESTIMATING
-------------------------------------------------------- Chapter 0:4.2

The effectiveness of FAA's investment management process, including
its cost estimating process, also relies heavily on the quality of
actual project cost information.  However, FAA does not adequately
accumulate all project cost information needed by federal
decisionmakers to make fully informed decisions about the billions of
dollars being spent on ATC projects.  Reliable financial information
includes the full cost of a project, including direct and indirect
costs.  Without this information, an organization does not know how
much is being spent on various ATC projects and, therefore, does not
have a basis for managing current costs or reliably estimating future
costs.  Accountability for such costs is required by FMFIA and the
CFO Act. 

Because FAA does not have a cost accounting system for ATC
modernization projects, project and corporate management as well as
FAA oversight agencies and institutions, such as OMB and the
Congress, rely on information produced by an assortment of financial
systems.  None of these financial systems provide for the systematic
accumulation and reporting of all relevant project costs. 

In August 1996, FAA set up an organization and subsequently initiated
an effort to acquire a cost accounting system.  FAA has thus far
specified general functional requirements and plans to procure an
off-the-shelf system that can be tailored to FAA's needs.  FAA plans
to have this cost accounting system in place by October 1, 1997. 


   RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5

Because the success of FAA's investment analysis and decision-making
process depends in large measure on the reliability of ATC project
cost information, GAO recommends that the Secretary of Transportation
direct the FAA Administrator to institutionalize defined processes
for estimating ATC projects' costs.  At a minimum, these processes
should include the following SEI requisites, each of which is
described in more detail in this report: 

  -- a corporate memory (or historical database), which includes cost
     and schedule estimates, revisions, reasons for revisions,
     actuals, and relevant contextual information;

  -- structured approaches for estimating software size and the
     amount and complexity of existing software that can be reused;

  -- cost models calibrated/tuned to reflect demonstrated
     accomplishments on past projects;

  -- audit trails that record and explain all values used as cost
     model inputs;

  -- processes for dealing with externally imposed cost or schedule
     constraints in order to ensure the integrity of the estimating
     process; and

  -- data collection and feedback processes that foster capturing and
     correctly interpreting data from work performed. 

GAO also recommends that the Secretary direct the Administrator to
immediately begin disclosing the inherent uncertainty and range of
imprecision in all ATC projects' official cost estimates presented to
executive oversight agencies or the Congress. 

Additionally, GAO recommends that the Secretary direct the
Administrator to acquire or develop and implement a managerial cost
accounting capability that will satisfy the requirements of SFFAS 4,
Managerial Cost Accounting Concepts and Standards for the Federal
Government.  This system capability should provide the cost
accounting and financial management information needed by FAA
management and those who make investment decisions.  Such information
should include full life cycle costs, which include the costs of
resources consumed by a project that directly or indirectly
contribute to the output and the costs of identifiable supporting
services provided by other organizations within the reporting entity. 

GAO further recommends that the Secretary report FAA's lack of a cost
accounting capability for its ATC modernization as a material
internal control weakness in the Department's fiscal year 1996 FMFIA
report and in subsequent annual FMFIA reports until the problem is
corrected. 

Also, GAO recommends that the Secretary direct the Administrator to
report to the Secretary and FAA's authorizing and appropriation
committees on progress being made on these recommendations as part of
the agency's fiscal year 1999 budget submission. 


   AGENCY COMMENTS AND GAO'S
   EVALUATION
---------------------------------------------------------- Chapter 0:6

GAO received oral comments on a draft of this report from senior
Department of Transportation (DOT) and FAA officials, including
representatives from the Office of the Secretary of Transportation,
the Executive Assistant to the FAA Chief Financial Officer, the FAA
Manager of the Cost Accounting System Division, and the FAA Program
Director for Investment Analysis and Operations Research.  These
officials agreed with most of GAO's findings, conclusions, and
recommendations.  However, they stated that because they plan to have
a cost accounting system in place by October 1, 1997, they do not
believe that our recommendations for implementing a cost accounting
capability and listing the lack of a cost accounting system as a
material internal control weakness in the agency's FMFIA report are
necessary.  While we are encouraged by FAA's recent actions to
procure a cost accounting system, we nevertheless believe that our
recommendations are still warranted to ensure that FAA's efforts are
complete and timely.  Until FAA's cost accounting system--which is
still very early in its acquisition life cycle--is developed,
installed, and operational, the lack of cost accounting information
will continue to pose a risk to sound project management and
investment decision-making.  As such, implementation of this
capability warrants aggressive action and disclosure as a material
internal control weakness in the agency's FMFIA report until the new
system is in place and producing accurate cost information. 


INTRODUCTION
============================================================ Chapter 1

The Federal Aviation Administration's (FAA) primary mission is to
ensure safe, orderly, and efficient air travel throughout the United
States.  FAA's ability to fulfill this mission depends on the
adequacy and reliability of the nation's air traffic control (ATC)
system, a vast network of computer hardware, software, and
communications equipment.  Sustained growth in air traffic and aging
equipment have strained the current ATC system, limiting the
efficiency of ATC operations.  To combat these trends, in 1981 FAA
embarked on an ambitious ATC modernization program.  FAA estimates
that it will spend about $34 billion on the program between 1982 and
2003. 

Our work over the years has chronicled many FAA failures in meeting
ATC projects' cost, schedule, and performance goals.\1 As a result,
we designated FAA's ATC modernization as a high-risk information
technology initiative in our 1995 report series on high-risk
programs. 


--------------------
\1 Air Traffic Control:  Status of FAA's Modernization Program
(GAO/RCED-95-175FS, May 26, 1995); Air Traffic Control:  Status of
>FAA's Modernization Program (GAO/RCED-94-167FS, Apr.  15, 1994); and
Air Traffic Control:  Status of FAA's Modernization Program
(GAO/RCED-93-121FS, Apr.  16, 1993). 


   ATC AT A GLANCE
---------------------------------------------------------- Chapter 1:1

Automated information processing and display, communication,
navigation, surveillance, and weather resources permit air traffic
controllers to view key information, such as aircraft location,
aircraft flight plans, and prevailing weather conditions, and to
communicate with pilots.  These resources reside at, or are
associated with, several ATC facilities--flight service stations, air
traffic control towers, terminal radar approach control (TRACON)
facilities, and air route traffic control centers (en route centers). 
These facilities' ATC functions are described below. 

  -- About 90 flight service stations provide pre-flight and
     in-flight services, such as flight plan filing and weather
     report updates, primarily for general aviation aircraft. 

  -- Airport towers control aircraft on the ground and before landing
     and after take-off when they are within about 5 nautical miles
     of the airport, and up to 3,000 feet above the airport.  Air
     traffic controllers rely on a combination of technology and
     visual surveillance to direct aircraft departures and
     approaches; maintain safe distances between aircraft; and
     communicate weather-related information, clearances, and other
     instructions to pilots and other personnel. 

  -- Approximately 180 TRACONs sequence and separate aircraft as they
     approach and leave busy airports, beginning about 5 nautical
     miles and ending about 50 nautical miles from the airport, and
     generally up to 10,000 feet above the ground, where en route
     centers' control begins. 

  -- Twenty en route centers control planes over the continental
     United States in transit and during approaches to some airports. 
     Each en route center handles a different region of airspace,
     passing control from one to another as respective borders are
     reached until the aircraft reaches TRACON airspace.  En route
     center controlled airspace usually extends above 18,000 feet for
     commercial aircraft.  En route centers also handle lower
     altitudes when dealing directly with a tower, or when agreed
     upon with a TRACON. 

  -- Two en route centers--Oakland and New York--also control
     aircraft over the ocean.  Controlling aircraft over oceans is
     radically different from controlling aircraft over land because
     radar surveillance only extends 175 to 225 miles offshore. 
     Beyond the radars' sight, controllers must rely on periodic
     radio communications through a third party--Aeronautical Radio
     Incorporated (ARINC), a private organization funded by the
     airlines and FAA to operate radio stations--to determine
     aircraft locations. 

See figure 1.1 for a visual summary of the processes for controlling
aircraft over the continental United States and oceans. 

   Figure 1.1:  Summary of ATC
   Over the Continental United
   States and Oceans

   (See figure in printed
   edition.)


   OVERVIEW OF THE ATC
   MODERNIZATION PROGRAM
---------------------------------------------------------- Chapter 1:2

The ATC system of the late 1970s was a blend of several generations
of automated and manual equipment, much of it labor-intensive and
obsolete.  FAA recognized that it could increase ATC operating
efficiency by increasing automation.  Additionally, FAA forecasted
increased future demand for air travel, brought on by airline
deregulation of the late 1970s.  It also anticipated that meeting the
demand safely and efficiently would require improved and expanded
services, additional facilities and equipment, improved workforce
productivity, and the orderly replacement of aging equipment. 
Accordingly, in December 1981, FAA initiated its plan to modernize,
automate, and consolidate the existing ATC system by the year 2000. 

This ambitious modernization program includes the acquisition of new
radars and automated data processing, navigation, and communication
equipment in addition to new facilities and support equipment.  FAA
estimates that the modernization will cost over $34 billion through
the year 2003 and total over 200 separate projects.  ATC information
systems make up a large portion of this total, accounting for 169
projects costing $20.7 billion.  The Congress will have provided FAA
with approximately $14.7 billion of the $20.7 billion through fiscal
year 1997. 

Over the past 15 years, FAA's modernization projects have experienced
substantial cost overruns, lengthy schedule delays, and significant
performance shortfalls.  To illustrate, the long-time centerpiece of
this modernization program--the Advanced Automation System--was
restructured in 1994 after estimated costs tripled from $2.5 billion
to $7.6 billion and delays in putting significantly
less-than-promised system capabilities into operation were expected
to run 8 years or more.  Similarly, increases in per-unit costs for
five other major ATC projects\2 have ranged from 50 to 511 percent,
and schedule delays have averaged almost 4 years. 

Our past work on the ATC modernization raised a number of concerns,
including concerns about the reliability of projects' cost estimates. 
For example, our review of FAA's Oceanic Display and Planning System
found that cost and schedule estimates were questionable because they
were based solely on managers' judgments and were not revised to
reflect changing project demands and conditions.\3 Also, our work on
AAS highlighted that FAA underestimated the complexity of the system
development and relied on its contractor's deficient cost estimating
systems.\4 Most recently, we reported that the organizational culture
intrinsic to FAA has been to deflate realistic estimates of cost and
schedule in order to gain funding approval.\5 In its internal risk
management guidance, FAA acknowledges that it has a history of
unreliable project cost estimates, attributing some of its unenviable
record in ATC projects' cost growth to setting unrealistically low
cost estimates, either because of poor cost estimating processes or
inadequate system descriptions.\6


--------------------
\2 The five projects and their respective percentage change in unit
costs are the Voice Switching and Control System (511 percent), the
Integrated Terminal Weather System (129 percent), the Airport Surface
Detection Equipment (56 percent), the Aviation Weather Observing
System (51 percent), and the Terminal Doppler Weather Radar (50
percent). 

\3 Air Traffic Control:  FAA Needs to Justify Further Investment in
Its Oceanic Display System (GAO/IMTEC-92-80, Sept.  30, 1992). 

\4 Air Traffic Control:  Status of FAA's Modernization Program
(GAO/RCED-94-167FS, Apr.  15, 1994). 

\5 Aviation Acquisition:  A Comprehensive Strategy Is Needed for
Cultural Change at FAA (GAO/RCED-96-159, Aug.  22, 1996). 

\6 Cost growth can occur for various reasons, including changing
requirements, unstable funding streams, poor project and/or contract
management, and poor estimates. 


      ATC MODERNIZATION AND
      MAINTENANCE MANAGEMENT
      STRUCTURE
-------------------------------------------------------- Chapter 1:2.1

Two major FAA organizations play key roles in the modernization and
evolution of ATC systems--the Office of the Associate Administrator
for Research and Acquisitions (ARA) and the Office of the Associate
Administrator for Air Traffic Services (ATS).  The first, ARA,
manages the research, development, and acquisition of modernization
projects.  Within ARA, two groups are responsible for acquiring
systems, while the others handle cross-cutting management functions
(e.g., budget formulation, cost estimation, and program evaluation). 
Also, the William J.  Hughes Technical Center is the ATC system test
and evaluation facility and supports ATC systems' research,
engineering, and development. 

ARA employs an Integrated Product Development System (IPDS) approach. 
A key component of IPDS is the use of Integrated Product Teams (IPT),
which are cross-functional teams aligned with major business and
functional areas (i.e., en route, terminal, weather and flight
services, air traffic management, oceanic, communications,
navigation, surveillance, infrastructure, and information systems). 
IPT members include systems and specialty engineers, logistics
personnel, lawyers, contract specialists, and representatives from
the organization responsible for the system's operations and
maintenance.  IPTs are responsible for systems research, development,
acquisition, and installation.  Product teams within these IPTs are
responsible for individual ATC system acquisitions or projects.  For
example, the en route IPT has product teams for the Display Channel
Complex Rehost, the Display System Replacement, the Voice Switching
and Control System, and several other en route systems. 

The second major organization involved with ATC systems is ATS.  ATS
is responsible for directing, coordinating, controlling, and ensuring
the safe and efficient utilization of the national airspace system. 
Organizations within ATS are responsible for planning, operating, and
maintaining ATC systems.  Responsibility for managing ATC systems is
transferred from the IPT to ATS once the systems have been installed
and are operational.  See figure 1.2 for a visual summary of the ATC
modernization and maintenance management structure. 

   Figure 1.2:  Organizational
   Chart Highlighting ATC
   Modernization and Maintenance
   Management Structure

   (See figure in printed
   edition.)



   (See figure in printed
   edition.)

\a ATS is currently being reorganized

During our review, we assessed six major modernization projects. 
These projects were the

  -- Voice Switching and Control System (VSCS), which provides
     air-to-ground voice communication services and ground-to-ground
     voice communication services between controllers, other ATC
     personnel, and others at the same and different en route centers
     and other ATC facilities;

  -- Standard Terminal Automation Replacement System (STARS), which
     is to replace critical air traffic control computers with new
     traffic computers, displays, and software in TRACON facilities
     and towers;

  -- Display System Replacement (DSR), which is to replace air
     traffic controllers' existing display-related systems in each of
     the en route centers;

  -- Airport Surveillance Radar-9 (ASR-9), which monitors aircraft
     movement and position within a radius of 60 miles of an airport
     terminal;

  -- Wide Area Augmentation System (WAAS) for the Global Positioning
     System (GPS), which is to provide augmentations to the
     Department of Defense's GPS in order to allow improved
     navigation on domestic and oceanic air routes; and

  -- Display Channel Complex Rehost (DCCR), which is an interim
     replacement to the mainframe computer system that processes
     radar and other data into displayable images on controllers'
     screens. 


      ROLES AND RESPONSIBILITIES
      FOR PROJECT COST ESTIMATING
-------------------------------------------------------- Chapter 1:2.2

The FAA organization responsible for estimating costs on the projects
we assessed varied depending on the project's stage in its life
cycle.  According to FAA acquisition rules in place when the latest
life cycle cost estimate for the projects we assessed were developed,
FAA's Investment Analysis and Operations Research organization
(ASD-400), which is one of the groups within ARA that handles
cross-cutting management functions, developed life cycle cost
estimates early in the project life cycle--both when mission needs
were evaluated and again when evaluating the costs, benefits, and
feasibility of project alternatives.  Once the decision was made to
invest in a given alternative, the IPTs assumed responsibility for
updating the cost estimates.  However, IPTs could also choose to
develop their own cost estimates prior to the investment decision
point, rather than have ASD prepare them. 

Of the six projects we reviewed, four used ASD-400 at some point in
the project's life cycle.  The other two did not.  In addition, some
project managers updated the acquisition phase portion of the life
cycle cost estimates periodically between the times that the full
life cycle cost estimates were revised. 

FAA's organizational responsibilities for cost estimating have
recently changed.  In October 1995, the Congress instructed FAA to
develop and implement a new acquisition management system, which
would not be subject to various existing acquisition laws.  On
November 15, 1995, the President signed this bill into law.\7 FAA
began implementing this new acquisition management system in April
1996 with the issuance of broad policies, guiding principles, and
internal procedures.\8 While not yet fully implemented, these specify
that two key decisions be made at the corporate level by the Joint
Resources Council (JRC), a newly formed body comprised of the
associate administrators for operations and acquisitions as well as
officials responsible for acquisitions, financial services, and legal
counsel.  These decisions are (1) whether mission needs warrant entry
into investment analysis and (2) whether to invest in the project at
the conclusion of investment analysis.  FAA identified the latter as
the most important decision in the life cycle acquisition management
process.  Accordingly, FAA plans to establish a "center of
excellence" for investment analysis with experts in cost estimating,
risk assessment, market analysis, and affordability analysis. 

Under this new scenario, investment analysis will be conducted as a
joint enterprise by the FAA organization sponsoring the system (i.e.,
ATS) and the FAA organization responsible for acquiring it (i.e.,
ARA).  The purpose is threefold:  to ensure that (1) users buy into
the solution, (2) acquisition specialists have a voice in the cost,
schedule, and performance baselines they will have to live with, and
(3) the investment analysis staff understands the concerns of the
operations and acquisitions organizations.  Under this approach, the
sponsoring organization, with technical support from the investment
analysis staff, develops and approves its requirements in the form of
a Requirements Document.  The investment analysis staff leads the
effort to identify and analyze candidate solutions through market
surveys, alternatives analysis, and affordability assessments, with
support from the sponsoring organization and the ARA IPT responsible
for acquiring it.  This effort culminates in the Investment Analysis
Report, which is to contain comprehensive quantitative data for each
alternative, such as life cycle cost, cost-benefit ratios, and risk. 
The IPTs use this information to generate cost and schedule baselines
for each alternative in the form of an Acquisition Program Baseline. 
At the investment decision point, the JRC decides on an alternative;
baselines the project's requirements, costs, schedules, performance,
and benefits; and commits the agency to full funding of the program. 
Thereafter, any changes to these baselines must be approved by the
JRC.  In fact, no funding may be committed or obligated that would
exceed the program cost baseline until the increase is approved by
the JRC and included in agency plans and budgets. 

The success of FAA's new investment analysis and decision-making
approach depends on many factors, not the least of which is the
reliability of ATC project cost information discussed in this report. 


--------------------
\7 1996 DOT Appropriations Act (Public Law 104-50), Section 348. 

\8 Federal Aviation Administration Acquisition Management System,
Apr.  1, 1996. 


   SOUND INVESTMENT DECISIONS
   REQUIRE RELIABLE COST
   ESTIMATING AND ACCOUNTING
   PRACTICES AND DATA
---------------------------------------------------------- Chapter 1:3

Reliable cost estimates and monitoring of actual costs are essential
to informed investment decision-making throughout the development and
maintenance of capital items, such as ATC systems.  In the case of
FAA, they are cornerstones to its aforementioned investment analysis
and decision-making processes. 

In 1994, we reported on how leading organizations improved mission
performance through information technology.  Among other things, we
reported that successful organizations manage information system
projects as investments, and continually assess the quality of
projects' estimated costs and carefully monitor projects' actual
costs against these estimates.  Furthering this initiative, OMB's
1995 guidance, Evaluating Information Technology Investments, calls
for selecting information technology project investments on the basis
of cost, benefit, risk, and return; controlling projects by comparing
ongoing actual results being achieved with projected costs, benefits,
and risks; and, finally, evaluating projects after they have been
implemented to determine actual cost, benefits, risks, and returns,
and modifying the selection and control processes based on lessons
learned.  This guidance has since been embodied in (1) the
Clinger-Cohen Act of 1996, which requires the selection of
information technology investments on the basis of competing
projects' estimated costs, benefits, and risks, and (2) the Chief
Financial Officers (CFO) Act of 1990, which requires federal agencies
to maintain integrated accounting and financial management systems
that permit systematic and reliable measurement of projects' cost and
performance.  Additionally, OMB Circular A-11, Part 3, requires
agencies to request full up-front budget authority for all ongoing
and new fixed assets (including information technology) in their
fiscal year 1998 budget submission.  This circular also requires a
fixed asset plan and justification for major acquisitions, including,
among other items, an analysis of full life cycle costs and an
estimate of the risk and uncertainty in meeting project goals. 


      EXPERTS ADVOCATE KEY
      ESTIMATING PROCESS
      REQUISITES
-------------------------------------------------------- Chapter 1:3.1

The Software Engineering Institute's (SEI) Capability Maturity Model
(CMM), the standard used by government and industry to determine the
maturity of an organizations' software development processes, also
highlights the need for good estimates and good estimating
processes.\9 Three of the CMM's key process areas for level 2
(repeatable) process maturity are project planning, project tracking,
and subcontract management.  These process areas must have reliable
estimates for size, effort, schedule, and cost if they are to be
performed successfully.  The CMM further requires that the procedures
that implement these key process areas be documented. 

To improve the state of practice for software cost and schedule
estimating, SEI developed and published (1) criteria for establishing
sound estimating processes and (2) a guide for managers to use in
validating an individual project's estimate.\10 These documents, in
effect, describe "best practices" used in industry and government for
estimating software costs and schedules.  However, SEI found that the
"best practices" are equally applicable to hardware and integrated
systems projects, and therefore allows for substituting the word
"system" for "software" throughout its guides and checklists.  SEI
also noted that while the criteria target the acquisition/development
phase of a project's life cycle, the concepts are also applicable to
other phases of the life cycle. 

According to SEI's Checklists and Criteria for Evaluating the Cost
and Schedule Estimating Capabilities of Software Organizations, in
order to have sound estimating processes, an organization should have
six attributes, or requisites, institutionally embedded in its
policies and procedures.  These include (1) a corporate memory, or
historical database(s), for cataloging cost estimates, revisions,
reasons for revisions, actuals, and other contextual information, (2)
structured processes for estimating software size and the amount and
complexity of existing software that can be reused, (3) cost models
calibrated/tuned to reflect demonstrated accomplishments on similar
past projects, (4) audit trails that record and explain values used
as cost model inputs, (5) processes for dealing with externally
imposed cost or schedule constraints in order to ensure the integrity
of the estimating process, and (6) data collection and feedback
processes that foster capturing and correctly interpreting data from
work performed.  SEI provides detailed checklists for assessing an
organization's satisfaction of each requisite. 

These same six requisites are interwoven through seven questions SEI
poses in A Manager's Checklist for Validating Software Cost and
Schedule Estimates.  The seven questions are (1) Are the objectives
of the estimate clear and correct?  (2) Has the task been
appropriately sized?  (3) Are the estimated cost and schedule
consistent with demonstrated accomplishments on past projects?  (4)
Have the factors that affect the estimate been identified and
explained?  (5) Have steps been taken to ensure the integrity of the
estimating process?  (6) Is the estimate based on reliable evidence
of the organization's past performance?  and (7) Has the situation
remained unchanged since the estimate was prepared?  Once again, SEI
provides detailed checklists for addressing these seven questions. 
These SEI publications are further discussed in chapter 2 and in
appendixes I, II, and III. 


--------------------
\9 Capability Maturity Model for Software, Version 1.1
(CMU/SEI-93-TR-24). 

\10 Checklists and Criteria for Evaluating the Cost and Schedule
Estimating Capabilities of Software Organizations (CMU/SEI-95-SR-005,
Jan.  1995) and A Manager's Checklist for Validating Software Cost
and Schedule Estimates (CMU/SEI-95-SR-004, Jan.  1995). 


      COST ACCOUNTING REQUIREMENTS
-------------------------------------------------------- Chapter 1:3.2

Requirements for agency cost accounting have been evolving for
decades.  In a 1985 report, the Comptroller General presented a
framework for strengthening agencies' financial management
structure.\11 This report called for the integration of accounting
and budgeting systems to better monitor progress against estimates
and to better estimate future program costs.  More specifically, it
states that actual costs must be maintained and monitored in order to
effectively manage programs and control costs.  This approach was
embodied in the CFO Act of 1990, which requires agencies to develop
and maintain integrated accounting and financial management systems
which provide for (1) the development and reporting of cost
information and (2) the systematic measurement of performance. 

Most recently, the Statement of Federal Financial Accounting
Standards no.  4 (SFFAS 4), Managerial Cost Accounting Concepts and
Standards for the Federal Government, effective for fiscal years
beginning after September 30, 1996, was issued.  This standard
requires reporting entities to regularly accumulate and report the
full cost (including all direct, indirect, and supporting costs) of
its activities.  These cost accounting standards are further
discussed in chapter 3. 


--------------------
\11 Managing the Cost of Government:  Building An Effective Financial
Management Structure (GAO/AFMD-85-35 and GAO/AFMD-85-35-A, Feb. 
1985). 


   OBJECTIVES, SCOPE, AND
   METHODOLOGY
---------------------------------------------------------- Chapter 1:4

The objectives of our review were to determine if (1) FAA's project
cost estimates are based on good estimating policies and practices
and (2) the actual costs of ATC modernization projects are being
properly accumulated. 

To determine if FAA's estimates were based on good policies and
practices, we

  -- researched current literature and interviewed project estimating
     experts to identify the key components of good cost estimating
     practices;

  -- obtained and analyzed FAA's policies and practices for
     estimating costs to determine what criteria (directives, orders,
     instructions, and implementing procedures), if any, FAA has in
     place to guide managers in developing projects' cost estimates;

  -- assessed FAA's cost estimating policies, practices, tools, and
     techniques to determine if they incorporate the key components
     of good cost and schedule estimating practices advocated by SEI
     and other experts; and

  -- selected FAA's five largest (based on latest life cycle cost
     estimates) ongoing ATC modernization projects and one project
     that was the subject of another GAO review, and interviewed
     project managers and assessed project documentation on these six
     projects to determine (1) how the current life cycle cost
     baseline was estimated and (2) how this estimating approach
     compared to the SEI project-level questions.\12 To do this, we
     compared each projects' documentation to SEI's detailed
     checklists for each question; determined if the project
     satisfied, partially satisfied, or did not satisfy each
     checklist item and assigned points accordingly (1, .5, or 0
     points, respectively); and then summed the points and presented
     them as a portion of the total points available (e.g., 4/10). 
     Because the focus of this effort is to assess FAA's cost
     estimating processes and not to validate the accuracy or
     completeness of the estimates, we did not evaluate the quality
     of the estimates. 

To determine whether the actual costs of ATC modernization projects
are being properly accumulated, we

  -- obtained and reviewed (1) selected reports and testimonies
     issued by GAO, the Department of Transportation's Office of the
     Inspector General, and the Defense Contract Auditing Agency, (2)
     related policies and procedures issued by the Department of
     Transportation, (3) applicable accounting standards and
     guidance, and (4) applicable OMB directives;

  -- reviewed FAA's policies and procedures governing ATC financial
     management and interviewed program managers and financial
     accounting staff to determine (1) their roles and
     responsibilities for recording and managing ATC cost information
     and (2) the financial processes used to accumulate and record
     ATC costs; and

  -- reviewed available information for the five largest (based on
     life cycle cost estimates) ongoing ATC projects and determined
     if costs are properly accumulated by (1) obtaining available
     financial information on the projects and identifying the cost
     elements included and excluded and (2) assessing reconciliation
     procedures among varying sources of information. 

We requested comments on a draft of this product from the Secretary
of Transportation.  On December 10, 1996, we obtained oral comments
from Transportation and FAA officials, including representatives from
the Office of the Secretary of Transportation, the Executive
Assistant to the FAA Chief Financial Officer, the FAA Manager of the
Cost Accounting System Division, and the FAA Program Director for
Investment Analysis and Operations Research.  Their comments are
presented and addressed in chapters 2 and 3 of this report. 

We performed our work at the Federal Aviation Administration in
Washington, D.C., and the Software Engineering Institute in
Pittsburgh, Pennsylvania, between February and December 1996.  Our
work was performed in accordance with generally accepted government
auditing standards. 


--------------------
\12 A Manager's Checklist for Validating Software Cost and Schedule
Estimates (CMU/SEI-95-SR-004, Jan.  1995). 


ATC MODERNIZATION LACKS THE
INSTITUTIONAL CAPACITY TO PRODUCE
RELIABLE COST ESTIMATES
============================================================ Chapter 2

Reliable estimates of projects' expected costs are essential to
decide among alternative investments.  According to SEI, consistently
producing them requires defined institutional processes for deriving
estimates, archiving them, and measuring actual performance against
these estimates.  FAA's cost estimating processes used on its ATC
modernization projects do not meet SEI criteria.  These weaknesses
are exacerbated by FAA's practice of presenting cost estimates as
precise, point estimates.  By doing so, FAA obscures the estimates'
inherent uncertainty and may mislead decisionmakers. 


   POLICIES AND PRACTICES FOR
   ESTIMATING COSTS DO NOT SATISFY
   "BEST PRACTICES" REQUISITES
---------------------------------------------------------- Chapter 2:1

FAA's institutional processes for estimating ATC projects' costs do
not fully satisfy any of the six SEI requisites.  According to SEI,
all six must be satisfied to produce credible estimates.  The six
requisites are described below, along with our analysis showing that
FAA's institutional policies and practices fail to meet them.  (See
appendixes I and II for more detail.) We shared this analysis with
FAA's cost estimating authority, who agreed that FAA's policies and
practices do not meet SEI's process requirements. 


      REQUISITE 1:  A CORPORATE
      MEMORY
-------------------------------------------------------- Chapter 2:1.1

According to SEI, estimating organizations should have a process for
organizing and retaining project cost and schedule information in a
historical database, and for using it as an integral part of the
estimating process.  This database should contain detailed
information on projects' original estimates, revised estimates,
reasons for revisions, and actual performance against the estimates. 
It should also contain descriptive contextual information that
enables people to understand and correctly interpret the data in the
database. 

FAA has no institutional corporate memory, or historical database, on
ATC projects' cost and schedule estimates and performance.  While FAA
has a number of stand-alone databases within different groups, none
provide a complete picture of estimates, assumptions that make up the
estimates, revisions, and actual performance on projects.  For
example, the Cost Benefit Analysis System (CBAS) is a database that
contains some information on projects' cost estimates and planned
budget levels.  However, it contains no information on how and why
estimates are revised, why budget streams differ from estimates, or
what projects actually cost. 

Further, what limited information is available on actual cost
performance is not an integral part of the project estimating
process, and the information on cost estimates that is retained by
the central estimating organization is not readily available to the
project personnel that are responsible for updating estimates after
the initial investment decision has been made. 


      REQUISITE 2:  STRUCTURED
      PROCESSES FOR ESTIMATING
      PRODUCT SIZE AND THE AMOUNT
      AND COMPLEXITY OF EXISTING
      SOFTWARE THAT CAN BE REUSED
-------------------------------------------------------- Chapter 2:1.2

According to SEI, estimating organizations should follow
well-defined, structured processes for estimating product size and
the amount and complexity of existing software that can be reused. 
It should be clear what is included and what is excluded from size
estimates, and new estimates should be checked by comparing them to
measured sizes of existing software products. 

FAA has no institutional process for estimating product size and the
amount of existing software that can be reused.  Each project manager
decides to estimate software size and reuse as he or she chooses. 
Among ATC projects, these processes range from simple lines-of-code
estimates using an individual's personal knowledge of similar
systems' sizes to sophisticated analysis based on project-unique
variables.  For example, the original software size estimates for the
Standard Terminal Automation Replacement System (STARS) was a rough
approximation based on the number of lines of code in a predecessor
system.  A more recent, though not yet official, estimating effort
established size estimates for each desired STARS software function
and then compared the estimates to known sizes of similar functions
on two other completed projects.  This effort also used a checklist
to ensure that no desired functions were overlooked and accounted for
differences between STARS and its predecessor system.  The latter,
more careful size estimate is over three times the original estimate
for new and modified software. 


      REQUISITE 3:  MECHANISMS FOR
      EXTRAPOLATING FROM
      DEMONSTRATED ACCOMPLISHMENTS
      ON PAST PROJECTS
-------------------------------------------------------- Chapter 2:1.3

According to SEI, estimating organizations should have documented
processes for extrapolating from past experiences.  These processes
should include the use of cost models that are calibrated and
validated on the basis of actual experience.  Further, differences
between cost models' outputs should be analyzed and explained. 

FAA's estimating guidance recommends 10 different cost models as
acceptable estimating tools; however, there is no requirement for
project estimators to use these models or to collect and use past
experience to calibrate these models.  As a result, projects' use and
calibration of the models is inconsistent.  For example, estimators
on three of the six projects we assessed did not use cost models in
their estimates whereas Display System Replacement (DSR) officials
not only used four cost models, but also calibrated them to past
experiences on a predecessor system. 


      REQUISITE 4:  AUDIT TRAILS
-------------------------------------------------------- Chapter 2:1.4

According to SEI, estimating organizations should prepare adequate
audit trails of inputs to the estimates, including parameters used in
cost models and their rationales. 

FAA estimating guidance requires that cost estimates be documented
and reproducible.  However, the degree of documentation and the
extent of any accompanying explanation is left to the discretion of
each project's estimators.  As a result, the detail and quality of
the audit trails on ATC projects is inconsistent.  For example, only
handwritten notes document how the Display Channel Complex Rehost
(DCCR) estimate was derived.  On the other hand, the DSR estimate and
the ongoing STARS estimating effort are supported by volumes of
documentation delineating the assumptions, processes, and model
inputs used. 


      REQUISITE 5:  INTEGRITY IN
      DEALING WITH DICTATED COSTS
      OR SCHEDULES
-------------------------------------------------------- Chapter 2:1.5

According to SEI, organizations should ensure that the effects of
dictated, or externally imposed, costs or schedules are determined
and explicitly presented to management.  Estimators should document
and managers should approve any changes made to model parameters to
accommodate dictated costs or schedules, the rationale for making the
changes, and the effect of the changes on other factors--cost,
schedule, or risk. 

FAA has no institutional process for ensuring integrity in dealing
with dictated costs or schedules.  As a result, each project manager
determines his or her own response to externally imposed constraints. 
Only one of the projects we assessed acknowledged working under an
externally imposed schedule.  STARS project officials are preparing a
cost estimate that shows the cost of meeting the compressed schedule. 
However, without an institutional policy requiring such action, there
is no assurance that all dictated constraints on all ATC projects
will be handled so effectively. 


      REQUISITE 6:  DATA
      COLLECTION AND FEEDBACK
      PROCESSES THAT FOSTER
      CAPTURING AND CORRECTLY
      INTERPRETING DATA FROM WORK
      PERFORMED
-------------------------------------------------------- Chapter 2:1.6

According to SEI, organizations should have a defined process for
gathering information on ongoing and completed projects (including
original estimates, revised estimates, and post-mortem assessments)
and entering this information into the historical database. 

FAA has no institutional process for gathering information on
completed projects and entering it in a historical database. 
Post-mortem reviews are performed rarely, and only on an ad hoc
basis.  Instead, each project manager determines the type and amount
of information retained. 


   INDIVIDUAL ATC PROJECTS' COST
   ESTIMATING APPROACHES ARE
   INCONSISTENT AND SOMETIMES
   UNRELIABLE
---------------------------------------------------------- Chapter 2:2

Because FAA does not have well-defined, institutional processes for
estimating information technology projects' costs, the approaches
used and the reliability of the estimates are inconsistent. 

To assist management in assessing the credibility of a given project
cost estimate, SEI developed seven questions which must be answered. 
The seven questions are (1) Are the objectives of the estimate clear
and correct?  (2) Has the task been appropriately sized?  (3) Are the
estimated cost and schedule consistent with demonstrated
accomplishments on other projects?  (4) Have the factors that affect
the estimate been identified and explained?  (5) Have steps been
taken to ensure the integrity of the estimating process?  (6) Is the
estimate based on reliable evidence of the organization's past
performance?  and (7) Has the situation remained unchanged since the
estimate was prepared?  SEI developed a detailed checklist to assist
in addressing each question.  (See appendix III for more information
on SEI's questions and checklists.)

We applied this checklist to the most recently baselined life cycle
cost estimate on six ATC projects (the five ongoing projects with the
largest life cycle cost estimates plus one ongoing project that was
the subject of another GAO review).  These projects are the Voice
Switching and Control System (VSCS), the Standard Terminal Automation
Replacement System (STARS), the Display System Replacement (DSR), the
Airport Surveillance Radar-9 (ASR-9), the Wide Area Augmentation
System for the Global Positioning System (WAAS), and the Display
Channel Complex Rehost (DCCR). 

Of the six systems, two (ASR-9 and DCCR) could not be assessed
because they lacked documentation describing how the latest official
life cycle cost estimates were derived.  For example, the estimated
cost to develop ASR-9 is $857 million.\1 However, there is no
documentation describing how ASR-9 software size estimates were
determined or what assumptions were used to estimate costs from these
size estimates.  As a result, there is no analytical way to determine
the credibility of the original estimate or estimating approach. 
Moreover, without documentation, FAA cannot systematically use its
ASR-9 experience in other system estimating endeavors. 

Of the remaining four systems, we found that the approaches used on
three (STARS, WAAS, and VSCS) satisfied some but not all of SEI's
reliability questions.  To their credit, officials on two of these
projects are currently revising parts of the life cycle cost estimate
using more rigorous approaches.  For example, the estimating approach
used to establish the current STARS life cycle cost estimate was
performed very early in the project's life cycle, during the mission
needs determination phase.  Project officials estimated the project's
software size on the basis of a predecessor system's size.  However,
the predecessor was so different from STARS that its size was not a
good indicator of the size of STARS' software.  Unlike the
predecessor, STARS is being developed to be easily maintainable,
portable, and expandable.  Additionally, many significant functions,
such as support software for configuration management and simulation,
were not accounted for in the original software estimate.  Project
officials agree that the early estimating process was not strong, and
in fact, that they underestimated the size and complexity of the
software to be developed.  In an ongoing estimating effort, STARS
estimators have been more thorough and rigorous in estimating the
software size and project costs.  They defined existing and missing
STARS functions, comparing the functions to two completed projects'
functions to help determine software size for each function, and
refining this estimate based on advice from experts throughout FAA
and performance demonstrations by competing vendors.  However, even
this improved estimating approach falls short of SEI standards. 
Specifically, STARS estimators did not calibrate the cost estimating
models it used to past FAA performance, choosing instead to use the
models' default values.  Calibrating models to an organization's
historical experience (i.e., determining the productivity value of a
"man-month" or the time and cost to produce a "line of code" based on
the organization's past performance on similar projects, or adding
new cost drivers that are specific to an organization's business,
such as security or privacy) will provide more accurate estimates. 

In addition, project estimators derived official WAAS life cycle cost
estimates based on experiences with the National Satellite Test Bed,
a precursor to the WAAS development.  However, little documentation
exists on how the original WAAS size estimates were derived and what
assumptions and parameters were used to estimate costs.  Project
officials are currently updating estimates using a published sizing
methodology and the Software Life Cycle Intermediate Model (SLIM)
cost model, and stated that they believe this is a much more rigorous
approach than prior estimating efforts.  However, this more
structured approach also falls short of SEI requirements.  For
example, the project software estimator stated that one cost model
parameter (input variable) is a ranking of the sophistication of the
developer's software environment.  The estimator scored the WAAS
contractor as very high on this parameter and showed us where this
information was captured in the SLIM database.  However, the
estimator went on to explain that he chose a high score based on the
contractor's rating as a level 3 organization on SEI's CMM and the
fact that they have and use Computer-Aided Software Engineering
(CASE) tools.  This explanatory information was not recorded in the
SLIM database or in any other documentation, and thus is not
available to anyone trying to understand or validate the estimate or
learn from this estimating experience. 

The current official estimate for the last of the six systems, DSR,
was derived using an approach that partially or fully satisfied six
of the seven SEI reliability questions.  Examples of DSR's adherence
to SEI guidance include (1) DSR's software size estimate was
developed by identifying needed software functions, and then
determining the amount of new code needed and reusable code available
for each of these software functions, (2) DSR estimators calibrated
cost estimating models to past experience on DSR's predecessor
system, the Initial Sector Suite System (ISSS), and (3) estimators
used templates to ensure that key cost factors would not be
overlooked.  However, DSR estimators did not record the rationales
for the parameters they used in their cost models or explain
differences among cost model results.  (See appendix IV for further
information on the results of our assessment.)


--------------------
\1 A project official stated that they have no estimate of what it
will cost to operate and maintain ASR-9 over its life cycle. 


   FAA'S PORTRAYAL OF ATC
   PROJECTS' COST ESTIMATES
   IMPLIES AN UNJUSTIFIED LEVEL OF
   PRECISION
---------------------------------------------------------- Chapter 2:3

Software and systems development experts agree that early project
estimates are by definition imprecise, and that this inherent
imprecision decreases during the project's life cycle as more
information becomes known about the system.  Some have described this
phenomenon as a "cone of uncertainty" that is widest early in the
life cycle and narrows over time as more becomes defined and known
about the project.  (See figure 2.1.) These experts emphasize that
each cost estimate should include an indication of its degree of
uncertainty, possibly as an estimated range or qualified by some
factor of confidence.  For example, a cost estimate of $1 million
could be presented as a range from $750,000 to $1.25 million or as $1
million with a confidence level of 90 percent, indicating that there
is a 10-percent chance that costs will exceed this estimate. 

   Figure 2.1:  Cost Estimating
   Accuracy Over Time

   (See figure in printed
   edition.)

FAA does not reveal its estimates' degree of uncertainty to managers
involved in investment decisions.  Instead, FAA presents its
projects' cost estimates as unqualified point estimates, thereby
suggesting an element of precision that does not exist.  A budget
official stated that FAA presents project cost estimates as such
because "this is the way OMB and Congress want to see it." Further,
he stated that in today's environment of lean budgets, the low-end of
the estimate range is all that FAA can afford and all that is
salable, and therefore, this is what they present. 

By presenting a point estimate instead of a range of estimates or a
realistically qualified estimate, FAA is not fully disclosing all
relevant information about the projects' potential costs and inherent
risk.  The result is uninformed, and thus potentially unwise,
investment decisions. 


   FAA ORGANIZATIONS HAVE
   INITIATED EFFORTS TO IMPROVE
   COST ESTIMATING
---------------------------------------------------------- Chapter 2:4

During the course of our review, FAA organizations initiated several
efforts to improve their processes for estimating and archiving cost
information.  However, these efforts are still relatively new and
have not yet been institutionalized.  We did not evaluate any of
these efforts. 

FAA's Office of Investment Analysis and Operations Research
(ASD-400), which has historically been responsible for early life
cycle cost estimates, has established an informal cost estimating
work group to explore the possibility of integrating cost estimating
functions across FAA.  To date, this group has documented current
cost estimating processes and identified the multitude of
organizations that participate in different components of the
estimating process.  The group has also drafted a workbook to guide
estimators in developing estimates.  However, neither the group nor
the workbook has not yet been officially sanctioned by FAA. 

Additionally, FAA's Office of Information Technology recently
procured an agencywide site license for another cost estimating tool,
the Software Life Cycle Intermediate Model (SLIM).  This tool is one
of several that estimators can use to develop cost estimates. 

FAA's Office of Airway Facilities Requirements initiated a "grass
roots" effort to standardize on a single cost estimating and tracking
tool, called the Automated Cost Estimating Integrated Tools (ACEIT). 
Officials stated that institutionalizing ACEIT would provide FAA a
consistent framework for estimating costs, including a documented
audit trail.  The group supporting ACEIT is currently briefing other
FAA organizations in an attempt to promote adoption of this tool
throughout FAA. 

The same organization is also testing the Cost and Performance
Management System (CPMS) to track operations and maintenance (O&M)
costs by project for the first time.  An airways facilities manager
stated that CPMS will eventually be integrated with ACEIT to allow
estimators to use actual project cost information in estimating new
projects' O&M costs. 


   CONCLUSIONS
---------------------------------------------------------- Chapter 2:5

Multimillion dollar, and even billion-dollar, investment decisions on
air traffic control modernization projects are being made without
reliable information on the projects' estimated and actual costs. 
FAA does not have well-defined, structured estimating processes that
are rigorously followed, and does not disclose the estimates'
inherent uncertainty.  Without better estimates of cost, FAA's new
investment analysis and decision-making processes are unlikely to be
effective. 


   RECOMMENDATIONS
---------------------------------------------------------- Chapter 2:6

We recommend that the Secretary of Transportation direct the FAA
Administrator to institutionalize defined processes for estimating
ATC projects' costs.  At a minimum, these processes should include
the following SEI requisites, each of which are described in more
detail in this report: 

  -- a corporate memory (or historical database), which includes cost
     and schedule estimates, revisions, reasons for revisions,
     actuals, and relevant contextual information;

  -- structured approaches for estimating software size and the
     amount and complexity of existing software that can be reused;

  -- cost models calibrated/tuned to reflect demonstrated
     accomplishments on past projects;

  -- audit trails that record and explain all values used as cost
     model inputs;

  -- processes for dealing with externally imposed cost or schedule
     constraints in order to ensure the integrity of the estimating
     process; and

  -- data collection and feedback processes that foster capturing and
     correctly interpreting data from work performed. 

We also recommend that the Secretary direct the Administrator to
immediately begin disclosing the inherent uncertainty and range of
imprecision in all ATC projects' official cost estimates presented to
executive oversight agencies or the Congress. 

Additionally, we recommend that the Secretary direct the
Administrator to report to the Secretary and FAA's authorizing and
appropriation committees on progress being made on these
recommendations as part of the agency's fiscal year 1999 budget
submission. 


   AGENCY COMMENTS
---------------------------------------------------------- Chapter 2:7

DOT and FAA officials provided oral comments on a draft of this
report.  These officials concurred with the report's findings,
conclusions, and recommendations on cost estimating.  They also
stated that this report will be useful as FAA strives to improve its
cost estimating capabilities. 


FAA IS NOT ADEQUATELY ACCOUNTING
FOR ATC PROJECT COSTS
============================================================ Chapter 3

Agencies are required to maintain adequate systems of accounting and
internal controls to provide managers and other decisionmakers with
reliable financial information to effectively measure performance and
make sound investment decisions.  In the case of the ATC
modernization program, FAA is not satisfying this requirement. 
Specifically, ATC project information does not include all relevant
project costs, including internal personnel compensation, benefits,
and travel (PCB&T) costs, because FAA lacks a cost accounting system
to accumulate and allocate these costs to specific projects.  FAA's
internal accounting policies include a requirement for a cost
accounting system; however, this policy has not been implemented.  As
a result, project managers are unable to measure actual costs and
their ability to make informed decisions is impaired.  Moreover,
complete project information is not available to feed back into, and
thereby improve, future project cost estimates. 


   AGENCIES ARE REQUIRED TO
   MAINTAIN ADEQUATE ACCOUNTING
   SYSTEMS
---------------------------------------------------------- Chapter 3:1

The Federal Managers' Financial Integrity Act of 1982 (FMFIA)
requires that agency systems of internal accounting and
administrative control comply with internal control standards
prescribed by the Comptroller General and provide reasonable
assurance that among other things, obligations and costs comply with
applicable law and revenues and expenditures applicable to agency
operations are recorded and accounted for properly.  FMFIA also
requires that agency heads issue an annual report, transmitted to the
President and the Congress, detailing whether their internal control
systems fully comply with the act's requirements, including the
identification of material systems weaknesses and plans for
corrective actions. 

The Chief Financial Officers (CFO) Act of 1990 requires agencies to
develop and maintain integrated agency accounting and financial
management systems that comply with applicable accounting principles,
standards, and requirements, including

  -- the preparation of complete, reliable, consistent, uniform, and
     timely information that is responsive to agency management's
     financial information needs;

  -- the development and reporting of cost information;

  -- the integration of accounting and budgeting information; and

  -- the systematic measurement of performance. 

Recently, the Statement of Federal Financial Accounting Standards no. 
4 (SFFAS 4), Managerial Cost Accounting Concepts and Standards for
the Federal Government, was issued, effective for fiscal periods
beginning after September 30, 1996.  These standards require a
reporting entity to accumulate and report the full cost of its
activities regularly for management information purposes.  The full
cost of a project is described as the sum of (1) the costs of
resources consumed by the project that directly or indirectly
contribute to the output and (2) the costs of identifiable supporting
services provided by other organizations within the reporting entity
and by other reporting entities.  These standards also require that
the full costs of resources be assigned to outputs through costing
methodologies or cost finding techniques that are most appropriate to
the organization's operating environment and that they be followed
consistently.  While SFFAS 4 has only been effective for a short
time, and therefore was not applicable during the period of our
review, it provides cost accounting criteria which are now required
to be implemented by all agencies. 

Additionally, the 104th Congress passed the Federal Financial
Management Improvement Act of 1996 which, among other provisions,
requires agencies to comply with federal accounting standards. 


   ATC PROJECT COSTS ARE NOT BEING
   PROPERLY ACCUMULATED
---------------------------------------------------------- Chapter 3:2

FAA financial systems supporting the ATC modernization program do not
accumulate all project costs, and thus, managers do not receive all
relevant financial information needed to effectively manage their
projects.  Of the five projects whose financial information we
reviewed, none of the project managers could provide the total of all
costs incurred from the project's inception.  Instead, they provided: 

  -- contract numbers for their respective projects so that cost data
     for each contract could be extracted from FAA's Departmental
     Accounting and Financial Information System (DAFIS) and
     aggregated to provide total contract costs.  However, these
     contract costs could be understated because project officials
     could not verify that they provided us all applicable contract
     numbers. 

  -- incomplete project costs.  These costs were not complete because
     they did not include (1) Personnel Compensation, Benefits, and
     Travel (PCB&T) costs associated with the Facilities and
     Equipment (F&E) appropriations account and (2) all costs paid
     out of the Operations and Maintenance (O&M) appropriations
     account.  PCB&T costs for the F&E appropriation include internal
     FAA costs that are related to project design, contracting, and
     contractor oversight.  O&M costs are costs associated with the
     administration, operation, repair, and maintenance of operating
     FAA facilities and are generally the single largest component of
     an information systems life cycle cost.  In 1995, total PCB&T
     costs were approximately $2 billion for all ATC projects.  PCB&T
     and O&M costs are accounted for separately and are not allocated
     to individual ATC projects.  Because these costs are not
     allocated to specific projects, full life cycle costs of
     projects cannot be determined and may be significantly
     understated. 

An additional limitation is that FAA does not carry over and report
the costs associated with terminated or redirected projects as part
of the successor projects' costs, even though successor projects
reuse parts of the predecessors' components.  As a result, the full
costs of "restructured" ATC projects are understated.  Accounting for
the full costs of projects requires that the costs related to usable
portions of terminated or redirected projects be included in the
costs of the ongoing projects.  In addition, full project cost
accounting information would require that costs of unused parts of
terminated or redirected projects be separately identifiable within
the "corporate memory."

For example, one of the projects we reviewed, the Display System
Replacement (DSR), is a follow-on to an earlier terminated project
known as the Initial Sector Suite System (ISSS).  According to FAA
officials, DSR software and hardware salvaged from ISSS accounts for
about 19 percent of DSR's cost.  However, these costs are not
included in DSR's accumulated and reported costs because, according
to an FAA official, these costs are considered "sunk costs." The term
"sunk costs" is generally used to describe costs that have been
incurred in the past and have no relevance to future decision-making. 
However, we believe these costs should be considered a part of a
project's full cost since they would be instructive in reliably
estimating costs of similar systems.  In addition, information about
the amount of costs associated with the unused portions of terminated
projects should be retained in the "corporate memory" to provide a
full picture of the real cost of development projects. 


   FAA LACKS A COST ACCOUNTING
   SYSTEM FOR ITS ATC
   MODERNIZATION PROGRAM
---------------------------------------------------------- Chapter 3:3

A managerial cost accounting system supports the collection,
measurement, accumulation, analysis, interpretation, and
communication of cost information to allow users to determine the
cost of specific programs and activities and the composition of, and
changes in, these costs.  As mentioned above, the CFO Act requires
agencies to develop and maintain a cost accounting capability that
captures both budgetary and financial accounting data and generates
performance measures.  FAA's internal policies require a cost
accounting system and state that the cost accounting system should be
integrated with the general accounting system.  However, these
policies have not been implemented; thus FAA project managers do not
have the capability to fully account for costs being incurred for the
ATC modernization program. 

Instead of a cost accounting system, several financial management
systems account for specific financial accounting and budgetary data,
but these systems are not integrated and they do not provide the full
cost information necessary for investment decisions.  These systems
include the following. 

  -- Departmental Accounting and Financial Information System
     (DAFIS):  This system is the Department of Transportation's core
     accounting system.  However, this system is not a cost
     accounting system because not all cost information is captured
     by project.  In addition, the Department's Office of the
     Inspector General (OIG) reported that DAFIS data is unreliable
     and inaccurate.\1 For example, the OIG reported that major
     balance discrepancies existed between DAFIS accounts and their
     supporting details.  Further, in April 1996, a cost accounting
     systems consultant reported that DAFIS does not provide all
     levels of management with timely, accurate, relevant, and
     meaningful information. 

  -- Financial Management System (FMS):  This FAA system is used by
     ATC project managers to establish quarterly obligation plans
     under the Facilities and Equipment (F&E) appropriation and to
     track actual obligations against these plans.  However, FMS is
     not a cost accounting system because it (1) contains only
     obligation data and (2) does not contain all relevant cost data,
     such as those for PCB&T. 

  -- Cost of Performance System (COPS):  This system is used by FAA
     organizations responsible for operating and maintaining ATC
     systems to allocate aggregated O&M obligation data to individual
     cost centers (that is, field maintenance organizations).  COPS
     is not a cost accounting system because it does not contain
     information on actual costs.  In addition, COPS' O&M cost data
     is not allocated to and accumulated for individual projects. 

  -- Research, Engineering and Development, Monitoring, Analysis, and
     Control System (REDMACS):  This system is used by FAA
     organizations responsible for ATC research and development
     projects to establish quarterly obligation plans for the
     Research, Engineering, and Development (RE&D) appropriation and
     to track actual obligations against these plans.  REDMACS is not
     a cost accounting system because it does not contain information
     on actual costs. 

Project managers have also developed their own unique systems to
account for F&E and RE&D obligations.  These "cuff" systems range
from spreadsheets to more sophisticated financial management systems,
but do not include O&M costs and generally do not capture actual
costs.  None of the systems listed above, either individually or
combined, constitute a cost accounting system because they do not
provide for accumulation and monitoring of total costs. 

In recognition of the need for accurate and reliable cost
information, FAA's Associate Administrator for Administration
established a new Cost Accounting Systems Division in August 1996 and
engaged a consultant to assist the agency in defining its cost
accounting requirements and in designing and implementing a system to
meet these requirements.  According to the Manager of the Cost
Accounting System Division, this system will use data from several
financial management systems, including DAFIS, and is planned to be
in place by October 1, 1997. 


--------------------
\1 Supplementary Report on Internal Control Systems and Compliance
Related to the Airport and Airway Trust Fund Portion of the Federal
Aviation Administration FY 1993 Financial Statement (Department of
Transportation - Office of Inspector General, AD-FA-5-005, Mar.  29,
1995). 


   CONCLUSIONS
---------------------------------------------------------- Chapter 3:4

Reliable project cost information that is both complete and accurate
is needed to estimate future project costs, make sound investment
decisions, and effectively manage projects.  FAA does not have a cost
accounting system capable of reliably accumulating full project cost
information, and therefore cannot reliably estimate future project
costs, ensure that investment decisions are sound, or manage projects
effectively.  Without better cost information, FAA's new investment
analysis and decision-making processes are unlikely to be effective. 


   RECOMMENDATIONS
---------------------------------------------------------- Chapter 3:5

In light of FAA's weaknesses in accounting for and reporting ATC
project costs, we recommend that the Secretary of Transportation
direct the FAA Administrator to acquire or develop and implement a
managerial cost accounting capability that will satisfy the
requirements of SFFAS 4, Managerial Cost Accounting Concepts and
Standards for the Federal Government.  This system capability should
provide the cost accounting and financial management information
needed by FAA management and those who make investment decisions. 
Such information should include full life cycle costs, which include
the costs of resources consumed by a project that directly or
indirectly contribute to the output and the costs of identifiable
supporting services provided by other organizations within the
reporting entity. 

We further recommend that the Secretary report FAA's lack of a cost
accounting capability for its ATC modernization as a material
internal control weakness in the Department's fiscal year 1996 FMFIA
report and in subsequent annual FMFIA reports until the problem is
corrected. 

Also, we recommend that the Secretary direct the Administrator to
report to the Secretary and FAA's authorizing and appropriation
committees on progress being made on these recommendations as part of
the agency's fiscal year 1999 budget submission. 


   AGENCY COMMENTS AND OUR
   EVALUATION
---------------------------------------------------------- Chapter 3:6

In providing oral comments on a draft of this report, DOT and FAA
officials stated that since they are in the process of acquiring a
cost accounting system and plan to have an "initial operating
capability" by October 1, 1997, they do not agree with our
recommendations and consider them unnecessary.  While we acknowledge
and support FAA's cost accounting organizational and system
initiatives, it is important to note that its cost accounting system
acquisition is still very early in its acquisition life cycle and
much remains to be accomplished before FAA can have the cost
accounting capability we recommend.  In fact, FAA has yet to develop
detailed functional requirements for this system, thereby precluding
our analysis at this time of whether its plans will satisfy our
recommendation.  Additionally, until FAA implements our
recommendation and improves the accuracy of underlying data in feeder
systems like DAFIS, it will continue to lack adequate cost
information needed to effectively manage its ATC system acquisitions. 
Disclosure of such a management control weakness is one of the
objectives of the FMFIA, and therefore we continue to believe that
FAA should report its lack of a cost accounting system as a material
weakness in its FMFIA reports until the problem is corrected. 


SUMMARY COMPARISON OF SEI'S
REQUISITES FOR RELIABLE ESTIMATING
PROCESSES AND FAA'S INSTITUTIONAL
POLICIES AND PRACTICES
=========================================================== Appendix I

Table I.1 identifies SEI's requisites for reliable estimating
processes and contrasts these with the institutional policies and
practices currently in place at FAA.  A more detailed comparison for
each requisite is provided in appendix II. 



                                    Table I.1
                     
                        Comparison of SEI's Requisites for
                      Reliable Estimating Processes and FAA
                     Policies and Practices for ATC Projects

                                                    FAA
                           -----------------------------------------------------
SEI requisites             Institutional policies     Actual practice
-------------------------  -------------------------  --------------------------
A corporate memory         None.                      Some information on
(historical database)                                 project estimates and
containing original and                               budgets are maintained in
revised cost and schedule                             different, unrelated
estimates and actual                                  databases. None of these
costs and schedules on                                databases, however,
completed projects to be                              provide sufficient data to
used as an integral part                              support these estimates
of the estimating process                             and assumptions and to
for new/updated                                       document revisions to
projects.                                             estimates and actual
(See appendix II, table                               performance against
II.1 for more details.)                               estimates.

                                                      Projects vary in the type
                                                      and extent of historical
                                                      information used.

                                                      The limited information
                                                      available on actual
                                                      project cost is not used
                                                      as part of the estimating
                                                      process.

Structured processes for   None.                      Project teams follow
estimating product size                               whatever approach they
and reuse. (See appendix                              choose for estimating
II, table II.2 for more                               product size and reuse. As
details.)                                             a result, size and reuse
                                                      practices vary.

Mechanisms for             FAA policy approves        Project teams use a
extrapolating from         parametric modeling as     variety of models or none
demonstrated               one of five acceptable     at all. The extent to
accomplishments on past    cost estimating            which they use historical
projects--that is, tools   methodologies. FAA         information varies
for forecasting costs and  guidance recommends, but   depending on their
schedules that are         does not require, using    awareness of and access to
calibrated to past         various estimating tools/  such information.
experience. (See appendix  models. FAA does not have
II, table II.3 for more    a policy regarding the
details.)                  calibration/tuning of the
                           models using demonstrated
                           accomplishments.

Audit trails detailing     FAA guidance states that   Audit trails vary among
and explaining values      cost estimates be          projects, ranging from
used for the cost model    documented and             handwritten notes to
parameters. (See appendix  reproducible, but does     detailed volumes.
II, table II.4 for more    not specify what that
details.)                  entails.

Integrity in dealing with  None.                      Project teams determine
dictated costs and                                    how they deal with imposed
schedules. (See appendix                              cost and schedule
II, table II.5 for more                               objectives. The effect of
details.)                                             imposed cost and schedule
                                                      constraints is not always
                                                      explicitly presented.

Data collection and        FAA policies require       Project teams determine
feedback processes that    projects to identify and   the type and amount of
foster capturing and       analyze differences from   information they retain.
correctly interpreting     earlier estimates. FAA     Thus, the type of
data from work performed   does not have a policy     information and its
and entering them into     regarding data collection  location varies among
the historical database.   and feedback.              projects.
(See appendix II, table
II.6 for more details.)
--------------------------------------------------------------------------------

DETAILED COMPARISON OF SEI'S
CHECKLIST FOR EACH REQUISITE AND
FAA PRACTICES FOR ATC PROJECTS
========================================================== Appendix II

The following tables provide a detailed comparison of FAA's practices
to the six SEI requisites' components that were summarized in
appendix I. 



                                    Table II.1
                     
                                A Corporate Memory

Evidence of maturity                     FAA practices
---------------------------------------  ---------------------------------------
The organization has a process for       FAA does not have an institutional
organizing and retaining information on  process for creating, maintaining, and
completed projects (a historical         using a historical database(s) on ATC
database).                               projects.

                                         FAA has several processes for retaining
                                         different subsets of the full set of
                                         information on completed project
                                         estimates and actuals. That is, it has
                                         processes for populating several
                                         databases that organize and retain some
                                         project information for use within
                                         certain groups.\a These individual
                                         processes are not coordinated with or
                                         part of an institutional process for
                                         organizing and retaining all relevant
                                         information on completed projects'
                                         estimates and actuals.

                                         None of these databases (either
                                         individually or combined) provide a
                                         comprehensive cradle-to-grave
                                         understanding of a project's history
                                         from early estimates, through
                                         revisions, and to ultimate completion.

                                         Furthermore, the information catalogued
                                         in the existing databases is limited to
                                         the perceived needs of the group that
                                         manages it. That is, it is not always
                                         known and accessible to, or usable by,
                                         estimators outside each group.
--------------------------------------------------------------------------------
                       (provide the following information:)

         Evidence of maturity                           FAA practices
--------------------------------------        ----------------------------------
                                              FAA does not have an institutional
>The information on completed projects            historical database.\b The
              includes:                             databases noted above
--------------------------------------        ----------------------------------
                                              CBAS        F&E         DAFIS
1. the life cycle model used together         yes         no          no
with the portion covered by the
recorded schedule and costs

2. original size estimate                     yes         no          no

3. changes in size resulting from             yes\c       no          no
changes in requirements

4. the original cost and schedule             y/n\d       no          no
estimate, together with the values and
rationales used for cost model
parameters

5. re-estimates and estimates-to-             yes         yes         no
complete

6. reasons for re-estimates                   no          no          no

7. actual costs and schedules                 no          no          y/n\e

8. actual size of delivered code              no          no          no

9. staffing profile                           no          no          no

10. labor mix                                 no          no          yes\f

11. skill level of the project team,          no          no          no
measured relative to the skill level
of the organization's typical team

12. nonlabor costs                            no          no          no

13. management costs                          no          no          no

14. system integration costs                  no          no          no

15. an estimate at completion                 no          yes         yes

16. extenuating circumstances or              no          yes\g       no
reasons for the differences between
the original and final estimates

17. a work breakdown structure or             no          no          no
alternative description of the tasks
included in the recorded costs

18. a work-flow schematic for the             no          no          no
software process

19. a summary of significant                  no          no          no
deliverables produced by the project
(software, documentation, etc.)

20. a summary of any unusual issues or        no          yes         no
contract factors that affected cost or
schedule

21. if multiple builds or releases            no          no          no
were used, the size, cost, schedule,
and characteristics of each build or
release
--------------------------------------------------------------------------------
Evidence of maturity                     FAA practices
---------------------------------------  ---------------------------------------
The historical database is treated as    Because FAA does not have an
an integral part of the estimating       institutional historical database, it
process, and estimators have active      is not an integral part of the
roles in specifying and sustaining the   estimating process. With the exception
information it contains.                 of the CBAS database, none of the
                                         individual databases are used in the
                                         estimating process. CBAS is used at the
                                         discretion of the ASD-400 estimators if
                                         they are involved in developing a
                                         project's cost estimate.

                                         With the exception of the ASD-400
                                         estimators, estimators are not active
                                         in defining the information in the
                                         respective databases.

The database contains a useful set of    FAA does not have an institutional
completed projects.                      historical database. Each of the
                                         individual databases contains a useful
                                         set of projects.

The elements included in (and excluded   Because FAA does not have an
from) effort, cost, schedule, size, and  institutional historical database, this
reuse measures are clearly identified.   type of information is not recorded
                                         therein. None of the individual
                                         databases provide this level of detail.

                                         The CBAS database provides a reference
                                         to documentation supporting the cost
                                         estimate, which includes information on
                                         cost and schedule elements, but not the
                                         elements included in effort, size, and
                                         reuse measures.

                                         Individual projects may have this
                                         information in their project files.

Schedule milestones (start and finish    Because FAA does not have an
dates) are described in terms of         institutional historical database, this
criteria for initiation or completion.   type of information is not recorded
                                         therein. None of the individual
                                         databases provide this level of
                                         detail.

                                         Individual projects may have this
                                         information in their project files.

Effort and cost data clearly indicate    Because FAA does not have an
which parts of the life cycle and which  institutional historical database, this
activities are covered by the different  type of information is not recorded
categories of hours or costs recorded.   therein. None of the individual
                                         databases provide this level of
                                         detail.

                                         Individual projects may have this
                                         information in their project files.

Records for projects indicate whether    Because FAA does not have an
unpaid overtime was used.                institutional historical database, this
                                         type of information is not recorded
                                         therein. None of the individual
                                         databases provide this level of detail.
                                         Further, none of the projects we
                                         reviewed recorded unpaid overtime.

Unpaid overtime, if used, is             Because FAA does not have an
quantified, so that recorded data        institutional historical database, this
provide a valid basis for estimating     type of information is not recorded
future effort.                           therein. None of the individual
                                         databases provide this level of detail.
                                         Further, none of the projects we
                                         reviewed recorded unpaid overtime.

Cost models are used to provide a        Because FAA does not have an
consistent framework (standard terms     institutional historical database, this
and parameters) for recording            type of information is not recorded
historical data.                         therein. FAA endorses a number of cost
                                         models, thus permitting differences
                                         among projects in the data that are
                                         recorded. Of the individual databases,
                                         only the CBAS database references
                                         models used by estimators. These
                                         models, however, are not used to record
                                         actual performance data on completed
                                         projects.

Historical data have been examined to    Because FAA does not have an
identify inconsistencies, and anomalies  institutional historical database, this
have been corrected or explained. (This  type of information is not recorded
is best done with the same cost models   therein. Thus, these data are not
used for estimating.)                    examined for anomalies. None of the
                                         individual databases provide this level
                                         of detail.

                                         Individual projects may have this
                                         information in their project files and
                                         may correct anomalies.

Workflow schematics are used to          Because FAA does not have an
describe similarities and differences    institutional historical database, this
among projects.                          type of information is not recorded
                                         therein. None of the individual
                                         databases provide this level of detail.

                                         Individual projects may have this
                                         information in their project files.
--------------------------------------------------------------------------------
\a The databases include: 

--the Cost Benefit Analysis System (CBAS), which catalogues project
estimates and provides a reference to the location of supporting
documentation in a repository.  It is only accessible by estimators
in the Investment Analysis and Operations Research Division (ASD-400)
of the Office of Systems Architecture and Investment Analysis;

--the Facilities and Equipment Database, which tracks changes in F&E
estimates over time, but whose supporting documentation is located in
a number of places.  It is used by one individual to answer ad hoc
FAA and congressional management and oversight questions on how
estimates have changed over time; and

--several financial management systems, including REDMACS, DAFIS,
FMS, and COPS, which provide budgetary performance information to
project teams. 

\b Individual projects may contain some of this information in their
project files. 

\c Only if a new cost estimate is generated. 

\d The database repository contains cost and schedule estimates, but
not the values and rationales used for cost model parameters. 

\e DAFIS can provide actual costs, but no schedule information. 

\f This information could be pulled, with effort, from DAFIS. 

\g The database provides a marker to when, or how, the estimate
changed (i.e., at a major acquisition review), not the reason for the
change. 



                                    Table II.2
                     
                       Structured Processes for Estimating
                             Software Size and Reuse

Evidence of maturity       FAA practices
-------------------------  -----------------------------------------------------
The estimating processes   FAA does not have a documented institutional process
for size and reuse are     for estimating size and reuse. Project teams are
documented.                permitted to follow whatever approach they choose for
                           estimating product size and reuse.

The estimating processes   Because FAA does not have a documented institutional
for size and reuse are     process for estimating size and reuse, project teams
followed.                  do not follow such a process. Project teams are
                           permitted to follow whatever approach they choose for
                           estimating product size and reuse.

The descriptions of size   Because FAA does not have a documented institutional
and reuse identify what    process for estimating size and reuse, project
has been included in (and  approaches differ in the extent to which they
excluded from) the size    describe what has been included and excluded from
and reuse measures.        size and reuse measures.

The measures of reuse      Because FAA does not have a documented institutional
distinguish between code   process for estimating size and reuse, project
that will be modified and  approaches differ in how reuse measures are
code that will be          described.
integrated as is into the
system.

Size estimates are         Because FAA does not have a documented institutional
checked by relating them   process for estimating size and reuse, project
to measured sizes of       approaches differ in how size estimates are checked.
other software products
or components.

The size estimating        Because FAA does not have a documented institutional
process is checked         process for estimating size and reuse, project
periodically by comparing  approaches differ in how size estimating approaches
its predictive             are checked.
capabilities with
measured sizes of
completed products.

Because size estimating    FAA does not have any ongoing efforts that focus on
is often the weakest link  improving its size estimating process.
in cost and schedule
estimating, the            Project-specific efforts may exist.
organization has a
continuing effort that
focuses on improving its
size estimating process.
--------------------------------------------------------------------------------


                                    Table II.3
                     
                        Mechanisms for Extrapolating From
                       Demonstrated Accomplishments on Past
                                     Projects

Evidence of maturity       FAA practices
-------------------------  -----------------------------------------------------
The extrapolation process  FAA does not have a documented institutional process
is documented.             requiring, or describing how to go about,
                           extrapolation from past projects. Project teams are
                           permitted to follow whatever approach they choose for
                           their extrapolations, if in fact they choose to
                           extrapolate.

Cost models and other      FAA recommends a number of cost models to assist
tools have been acquired   estimators. However, it does not provide these
or developed to assist     models--estimators are allowed to pick any model they
estimators.                choose, if any, and are individually responsible for
                           obtaining any needed licenses.

The cost models have been  Because FAA does not have a documented institutional
calibrated to relevant     process for extrapolating from past projects, project
historical data.           approaches differ based on if, and how, cost models
                           are calibrated.

Cost model calibrations    Because FAA does not have a documented institutional
are up to date.            process for extrapolating from past projects, project
                           approaches differ based on if, and how, cost models
                           are calibrated.

The cost and schedule      Because FAA does not have a documented institutional
models are used to         process for extrapolating from past projects, project
quantify demonstrated      approaches differ in the extent to which they use
organizational             cost models to quantify demonstrated organizational
performance in ways that   performance.
normalize for differences
among software products
and projects.

The consistency that       Because FAA does not have a documented institutional
estimators achieve when    process for extrapolating from past projects, project
fitting cost models to     approaches differ in the extent to which they track
historical data is         cost model performance.
measured and tracked.

Values used for cost       Because FAA does not have a documented institutional
model parameters are       process for extrapolating from past projects, project
validated by comparisons   approaches differ in the extent to which they
with past projects.        validate cost model parameters.

The methods used to        Because FAA does not have a documented institutional
account for reuse          process for extrapolating from past projects, project
recognize that reuse is    approaches differ in their recognition of software
not free.                  reuse costs.

Extrapolations from past   Because FAA does not have a documented institutional
projects incorporate       process for extrapolating from past projects, project
measured trends in         teams are permitted to determine if they extrapolate
technology improvement,    from past projects, and what this extrapolation
either within the cost     incorporates.
models themselves or as
inputs to them.

Estimators work jointly    If used by the project team, ASD-400 or contract
with project managers and  estimators work jointly with project managers and
experienced technical      technical personnel to identify similarities between
people to identify how     current and prior systems.
the new work compares to
work the organization or
others have done before.

More than one cost model   FAA policies require use of one or more approved cost
or estimating approach is  estimating approaches. However, FAA does not require
used, and differences      that estimators analyze and explain differences in
among results are          estimates. Project teams are permitted to determine
analyzed and explained.    if and how extensively they assess any differences
                           among estimates.

Trends in the              Because FAA does not have a documented institutional
organization's process     process for extrapolating from past project, project
and performance            teams are permitted to determine if and how they will
parameters are tracked to  track and identify the effects of trends in
identify their effects on  organizational parameters on cost model calibrations.
cost model calibrations.
--------------------------------------------------------------------------------


                                    Table II.4
                     
                                  An Audit Trail

Evidence of maturity       FAA practices
-------------------------  -----------------------------------------------------
The organization's         FAA does not have an institutional process
process documentation      identifying who is responsible for preparing an audit
identifies who is          trail for software estimates. Project teams are
responsible for preparing  permitted to determine who documents the audit trail
the audit trail for        and the extent of this documentation.
software estimates.

A list of parameter        FAA does not have an institutional process requiring
values and their           estimators to list all parameter values and their
rationales accompanies     rationales. Project teams are permitted to determine
each estimate.             the extent to which estimators document parameter
                           values and their rationales.

A template or format is    FAA does not have an institutional process for
used to record the values  recording cost model parameters and their rationales.
of cost model parameters   Project teams are permitted to determine how they
and their rationales.      will record the values of cost model parameters and
                           their rationales.

Uncertainties in           FAA does not have an institutional process for
parameter values are       dealing with uncertainties in parameter values.
identified and             Project teams are permitted to determine the extent
quantified.                to which they identify and quantify uncertainties in
                           parameter values.

The lists of parameter     Because FAA does not have an institutional historical
values and their           database, this type of information is not retained
rationales are retained    therein. None of the individual databases provide
in the organization's      this level of detail.
historical database.
--------------------------------------------------------------------------------


                                    Table II.5
                     
                     Integrity in Dealing With Dictated Costs
                                  and Schedules

Evidence of maturity       FAA practices
-------------------------  -----------------------------------------------------
Management reviews and     FAA does not have an institutional policy requiring
agrees to parameter        managers to review and agree to parameter values and
values and rationales      rationales before costs are estimated. Project teams
before costs are           are permitted to determine if and when management
estimated.                 approval is needed in developing cost estimates.
                           FAA's Office of Aviation Policy and Plans (APO) is
                           responsible for reviewing the methodological
                           soundness of cost estimates. However, APO does not
                           evaluate parameter values and their rationales.

Reasons for changing       FAA does not have an institutional process on
parameter values from      changing parameter values. Project teams are
those identified in the    permitted to determine which parameter values they
calibration set are        will use.
documented.

Adjustments to cost model  FAA does not have an institutional process on dealing
parameters to meet         with imposed costs and schedules. Project teams are
desired costs or           permitted to determine how they will deal with cost
schedules are accompanied  or schedule objectives.
by management actions
that make the parameter
values realistic.

The actions that the       Because FAA does not have an institutional process on
organization intends to    dealing with imposed costs and schedules, it does not
take to make its adjusted  require projects to justify how to make adjusted cost
cost model parameters      model parameters valid. Project teams are permitted
valid are spelled out in   to determine how they will deal with cost and
the project plan.          schedule objectives.
--------------------------------------------------------------------------------


                                    Table II.6
                     
                      Data Collection and Feedback Processes
                       That Foster Capturing and Correctly
                      Interpreting Data From Work Performed

Evidence of maturity       FAA practices
-------------------------  -----------------------------------------------------
There is a defined         Because FAA does not have an institutional historical
process for gathering      database, it does not have a process for gathering
information on completed   information on completed projects and entering it
projects and entering it   into the historical database. Of the individual
into the historical        databases, only the financial databases have
database.                  information on completed projects, and this
                           information is strictly budgetary.

                           Individual project teams may have this information in
                           their project files.

Postmortems are held at    FAA does not have an institutional process for
the completion of each     holding postmortem assessments at the completion of
project to ensure that     each project. FAA performs project postmortem reviews
(1) recorded data are      on an ad hoc basis.
valid and (2) events that
affected cost or schedule
get recorded and
described while they are
still fresh in people's
minds.

Estimates used for         Because FAA does not have an institutional historical
original project planning  database, this type of information is not recorded
are saved and entered      therein. Estimates for original project planning are
into the historical        retained by ASD-400, and filed in the CBAS system.
database.

Re-estimates and           Because FAA does not have an institutional historical
estimates for changes to   database, this type of information is not recorded
the product or process     therein. Project teams are required to file updated
are recorded and saved in  cost estimates with ASD-400, for input into the CBAS
the historical database.   database.

Pilots and prototypes of   FAA does not have an institutional process for
new software processes     incorporating full-scale software process
are measured and tracked   improvements into its estimating approaches.
to capture information
that can guide estimates
for full-scale processes.

Organizations that         FAA does not have an institutional process for
acquire software receive   receiving and saving copies of the developer's
and save copies of the     postmortem reports. Project teams are permitted to
developer's postmortem     determine the extent of the documentation they
reports.                   require from contractors and retain it.

There is a structured      FAA policies require monitoring actual program cost,
process for capturing      schedule,and technical achievement. Additionally, all
data on effort and cost    major project teams require contractors to submit
from ongoing and           contract performance reports detailing progress
completed projects.        against cost, schedule, and technical goals. This
                           information is reported at formal acquisition review
                           meetings and retained by the project office.

The capturing of data for  FAA does not have an institutional process for
cost estimating and        integrating its estimating/planning functions with
planning is integrated     its measurement processes used for tracking,
with the measurement       oversight, and process improvement on individual
processes used for         projects.
project tracking and
oversight and process      Individual project teams may incorporate project
improvement.               tracking information in their re-estimates.

Estimates-to-complete are  FAA requires that estimates-to-complete be updated at
updated and reviewed at    major acquisition reviews.
regularly scheduled
intervals (e.g.,
monthly).

Estimates-to-complete are  FAA requires project managers to monitor project
updated and reviewed       performance against the cost, schedule, and technical
whenever there is a major  boundaries in the acquisition program baseline and to
change to requirements,    promptly report any anticipated breaches to the
resources, priorities,     acquisition executive.
commitments, assumptions,
or understanding of the
project.

The processes for          FAA does not have institutional processes for the
capturing, collecting,     automated capture, collection, and dissemination of
and disseminating          measurement results and descriptive data.
measurement results and
descriptive data are
supported by automation,
so that opportunities for
misinformation,
sloppiness, and
indifference are
minimized.
--------------------------------------------------------------------------------

SEI CHECKLIST FOR VALIDATING THE
RELIABILITY OF A PROJECT'S COST
ESTIMATE
========================================================= Appendix III

Table III.1 provides a summary of the checklist items supporting each
SEI question. 



                                   Table III.1
                     
                         SEI Checklist for Validating the
                     Reliability of a Project's Cost Estimate

SEI questions              Checklist items
-------------------------  -----------------------------------------------------
Are the objectives of the  The objectives of the estimate are stated in
estimate clear and         writing.
correct?                   The life cycle to which the estimate applies is
                           clearly defined.
                           The tasks and activities included in (and excluded
                           from) the estimate are clearly identified.
                           The tasks and activities included in the estimate are
                           consistent with the objectives of the estimate.

Has the task been          A structured process has been used to estimate and
appropriately sized?       describe the size of the software product.
                           A structured process has been used to estimate and
                           describe the extent of reuse.
                           The processes for estimating size and reuse are
                           documented.
                           The descriptions of size and reuse identify what is
                           included in (and excluded from) the size and reuse
                           measures used.
                           The measures of reuse distinguish between code that
                           will be modified and code that will be integrated as
                           is into the system.
                           The definitions, measures, and rules used to describe
                           size and reuse are consistent with the requirements
                           (and calibrations) of the models used to estimate
                           cost and schedule.
                           The size estimate was checked by relating it to
                           measured sizes of other software products or
                           components.
                           The size estimating process was checked by testing
                           its predictive capabilities against measured sizes of
                           completed products.

Are the estimated cost     The organization has a structured process for
and schedule consistent    relating estimates to actual costs and schedules of
with demonstrated          completed work.
accomplishments on other   --The process is documented.
projects?                  --The process was followed.
                           The cost and schedule models that were used have been
                           calibrated to relevant historical data. (Models of
                           some sort are needed to provide consistent rules for
                           extrapolating from previous experience.)
                           The cost and schedule models quantify demonstrated
                           organizational performance in ways that normalize for
                           differences among software products and projects. (So
                           that a simple, unnormalized, lines-of-code per staff-
                           month extrapolation is NOT the basis for the
                           estimate.)
                           The consistency achieved when fitting the cost and
                           schedule models to historical data has been measured
                           and reported.
                           The values used for cost and schedule model
                           parameters appear valid when compared to values that
                           fit the models well to past projects.
                           The calibration of cost and schedule models was done
                           with the same versions of the models that were used
                           to prepare the estimate.
                           The methods used to account for reuse recognize that
                           reuse is not free. (The estimate accounts for
                           activities such as interface design, modification,
                           integration, testing, and documentation that are
                           associated with effective reuse.)
                           Extrapolations from past projects account for
                           differences in application technology. (For example,
                           data from projects that implemented traditional
                           mainframe applications require adjustments if used as
                           a basis for estimating client-server implementation.
                           Some cost models provide capabilities for this,
                           others do not.)
                           Extrapolations from past projects account for
                           observed, long-term trends in software technology
                           improvement. (Although some cost models attempt this
                           internally, the best methods are usually based on
                           extrapolating measured trends in calibrated
                           organizational performance.)

                           Extrapolations from past projects account for the
                           effects of introducing new software technology or
                           processes. (Introducing a new technology or process
                           can initially reduce an organizations's
                           productivity.)
                           Work-flow schematics have been used to evaluate how
                           this project is similar to (and how it differs from)
                           projects used to characterize the organization's past
                           performance.

Have the factors that      A written summary of parameter values and their
affect the estimate been   rationales accompanies the estimate.
identified and explained?  Assumptions have been identified and explained.
                           A structured process such as a template or format has
                           been used to ensure that key factors have not been
                           overlooked.
                           Uncertainties in parameter values have been
                           identified and quantified.
                           A risk analysis has been performed, and risks that
                           affect cost or schedule have been identified and
                           documented. (Elements addressed include issues such
                           as probability of occurrence, effects on parameter
                           values, cost impacts, schedule impacts, and
                           interactions with other organizations.)

Have steps been taken to   Management reviewed and agreed to the values for all
ensure the integrity of    descriptive parameters before costs were estimated.
the estimating process?    Adjustments to parameter values to meet a desired
                           cost or schedule have been documented.
                           If a dictated schedule has been imposed, the estimate
                           is accompanied by an estimate of (1) the normal
                           schedule and (2) the additional expenditures required
                           to meet the dictated schedule.
                           Adjustments to parameter values to meet a desired
                           cost or schedule are accompanied by management action
                           that makes the values realistic.
                           More than one cost model or estimating approach has
                           been used, and the differences in results have been
                           analyzed and explained.
                           People from related but different projects or
                           disciplines were involved in preparing the estimate.
                           At least one member of the estimating team is an
                           experienced estimator, trained in the cost models
                           that were used.
                           Estimators independent of the performing organization
                           concur with the reasonableness of the parameter
                           values and estimating methodology.
                           The groups that will be doing the work accept the
                           estimate as an achievable target.
                           Memorandums of agreement have been completed and
                           signed with the other organizations whose
                           contributions affect cost or schedule.

Is the estimate based on   The estimating organization has a method for
reliable evidence of the   organizing and retaining information on completed
organization's past        projects (a historical database).
performance?               The database contains a useful set of completed
                           projects.
                           Elements included in (and excluded from) the effort,
                           cost, schedule, size, and reuse measures in the
                           database are clearly identified. (See, for example,
                           the SEI checklist for defining effort, schedule, and
                           size measures.)
                           Schedule milestones (start and finish dates) are
                           described in terms of criteria for initiation or
                           completion, so that work accomplished between
                           milestones is clearly bounded.
                           Records for completed projects indicate whether
                           unpaid overtime was used.
                           Unpaid overtime, if used, has been quantified so that
                           recorded data provide a valid basis for estimating
                           future effort.
                           Cost models that were used for estimating also have
                           been used to provide consistent frameworks for
                           recording historical data. (This helps ensure that
                           comparable terms and parameters are used across all
                           projects, and that recorded data are suitable for use
                           in the estimating models.)
                           The data in the historical database have been
                           examined to identify inconsistencies, and anomalies
                           have been corrected or explained. (This is best done
                           with the same cost models used for estimating.)
                           The organization has a structured process for
                           capturing effort and cost data from ongoing
                           projects.
                           The producing organization holds postmortems at the
                           completion of its projects to (1) ensure that
                           recorded data are valid and (2) ensure that events
                           that affected costs or schedules get recorded and
                           described while they are still fresh in people's
                           minds.
                           Information on completed projects includes
                           --the life-cycle model used, together with the
                           portion covered by the recorded cost and schedule;
                           --actual (measured) size, cost and schedule;
                           --the actual staffing profile;
                           --an estimate at completion, together with the values
                           for cost model parameters that map the estimate to
                           the actual cost and schedule;
                           --a work breakdown structure or alternative
                           description of the tasks included in the recorded
                           cost;
                           --a work-flow schematic that illustrates the software
                           process used;
                           --nonlabor costs;
                           --management costs;
                           --a summary or list of significant deliverables
                           (software and documentation) produced by the project;
                           and
                           --a summary of any unusual issues that affected cost
                           or schedule.
                           Evolution in the organization's work-flow schematics
                           shows steady improvement in the understanding and
                           measurement of its software processes.

Has the situation          The estimate has not been invalidated by recent
remained unchanged since   events, changing requirements, or management action
the estimate was           (or inaction).
prepared?                  The estimate is being used as the basis for assigning
                           resources, deploying schedules, and making
                           commitments.
                           The estimate is the current baseline for project
                           tracking and oversight.
--------------------------------------------------------------------------------

SIX ATC PROJECTS' COST ESTIMATES'
SATISFACTION OF SEI'S CHECKLIST
========================================================== Appendix IV

Table IV.1 provides a summary comparison of six ATC projects'
estimates against SEI's questions for assessing an estimate's
reliability.  SEI provides detailed checklists for addressing these
seven questions (see appendix III for a summary of the detailed
checklists).  We compared official project life cycle estimates to
the detailed checklists to determine how well the projects satisfied
each question.  The fraction in each block shows the number of
satisfactory (worth 1 point) or partially satisfactory responses
(worth .5 points) to checklist items divided by the total number of
items.  Projects with insufficient documentation to support an
assessment were given a "No Basis" rating, indicating that there was
no basis for an evaluation.  Because SEI checklists focus on key
processes, we did not evaluate project cost estimates to determine if
all applicable costs are included. 

An SEI expert agreed that this method was an acceptable and
conservative approach to scoring projects.  However, he cautioned
that a project with an extremely high score could lack the one
checklist item that is critical to that specific project, thus
rendering the estimate unreliable. 



                                    Table IV.1
                     
                        Six ATC Projects' Cost Estimates'
                         Satisfaction of SEI's Checklist

Projects SEI
Questions             VSCS      DSR       STARS     ASR-9     WAAS      DCCR
--------------------  --------  --------  --------  --------  --------  --------
Are the objectives    4/4       4/4       3/4       No        4/4       No Basis
of the estimate                                     Basis\a
clear and correct?

Has the task been     No Basis  5.5/8     2/8       No Basis  No Basis  No Basis
appropriately sized?

Are the estimated     No Basis  4.5/10    0.5/10    No Basis  2/10      No Basis
cost and schedule
consistent with
demonstrated
accomplishments on
other projects?

Have the factors      2/5       3.5/5     1/5       No Basis  3/5       No Basis
that affect the
estimate been
identified and
explained?

Have steps been       5.5/7     5.5/7     2.5/7     No Basis  3/7       No Basis
taken to ensure the
integrity of the
estimating process?

Is the estimate       1.5/12    2.5/12    0.5/12    No Basis  1/12      No Basis
based on reliable
evidence of the
organization's past
performance?

Has the situation     1/3       3/3       0/3       No Basis  0/3       No Basis
remained unchanged
since the estimate
was prepared?
--------------------------------------------------------------------------------
\a No basis for an evaluation. 


MAJOR CONTRIBUTORS TO THIS REPORT
=========================================================== Appendix V


   ACCOUNTING AND INFORMATION
   MANAGEMENT DIVISION, WASHINGTON
   D.C. 
--------------------------------------------------------- Appendix V:1

Linda M.  Calbom, Director
Randolph C.  Hite, Senior Assistant Director
John C.  Fretwell, Assistant Director
Keith A.  Rhodes Technical Assistant Director
Madhav S.  Panwar, Senior Technical Advisor
Colleen M.  Phillips, Senior Information Systems Analyst
Tomas Ramirez, Jr., Senior Business Process Analyst
Cynthia Jackson, Senior Auditor
Deborah R.  Peay, Staff Auditor

*** End of document. ***