Defense Acquisitions: Further Management and Oversight Changes
Needed for Efforts to Modernize Cheyenne Mountain Attack Warning
Systems (06-JUL-06, GAO-06-666).
The Cheyenne Mountain Operations Center houses numerous complex
computer systems for tracking air, missile, and space events that
could threaten homeland security or undermine military operations
in theater. To ensure this mission can be met, the systems
require ongoing upgrades. The most recent upgrade program--the
Combatant Commanders' Integrated Command and Control System
(CCIC2S)--was initiated in 2000. Given the critical missions
supported by Cheyenne Mountain systems, GAO initiated a review to
(1) determine the status of the CCIC2S program in terms of
meeting its cost, schedule, and performance goals; (2) gauge the
extent to which DOD has followed best practices in managing
program requirements; and (3) assess DOD's control and oversight
mechanisms for CCIC2S.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-06-666
ACCNO: A56400
TITLE: Defense Acquisitions: Further Management and Oversight
Changes Needed for Efforts to Modernize Cheyenne Mountain Attack
Warning Systems
DATE: 07/06/2006
SUBJECT: Command and control systems
Computer systems
Cost overruns
Defense capabilities
Schedule slippages
Program evaluation
Program management
Defense procurement
Systems conversions
Technology modernization programs
Cheyenne Mountain (WY)
Combatant Commanders' Integrated Command
and Control System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-06-666
* Results in Brief
* Background
* Previous Cheyenne Mountain Upgrade Efforts
* CCIC2S Program Has Experienced Cost and Schedule Overruns an
* The CCIC2S Program Is Over Cost and Behind Schedule
* Deferral of Capabilities and Performance Shortfalls Have Imp
* Cost and Schedule Overruns Caused by Not Adhering to Best Pr
* Match Not Made between Program Requirements and Available Re
* The Air Force Has Implemented the CCIC2S Program without Sol
* CCIC2S Program Oversight and Controls Have Been Ineffective
* DOD Did Not Designate CCIC2S as a Major Acquisition
* CCIC2S Milestone Decision Authority Did Not Provide Effectiv
* Program Lacked Sufficient Management Controls over the Contr
* Program Lacked Independent Contractor Performance Assessment
* Needed Management Controls and Redesignation of CCIC2S Progr
* Conclusions
* Recommendations for Executive Action
* Agency Comments and Our Evaluation
* GAO Contact
* Acknowledgments
* GAO's Mission
* Obtaining Copies of GAO Reports and Testimony
* Order by Mail or Phone
* To Report Fraud, Waste, and Abuse in Federal Programs
* Congressional Relations
* Public Affairs
Report to Congressional Committees
United States Government Accountability Office
GAO
July 2006
DEFENSE ACQUISITIONS
Further Management and Oversight Changes Needed for Efforts to Modernize
Cheyenne Mountain Attack Warning Systems
GAO-06-666
Contents
Letter 1
Results in Brief 3
Background 5
CCIC2S Program Has Experienced Cost and Schedule Overruns and Performance
Shortfalls 8
Cost and Schedule Overruns Caused by Not Adhering to Best Practices for
Managing Program Requirements 11
CCIC2S Program Oversight and Controls Have Been Ineffective 16
Conclusions 22
Recommendations for Executive Action 23
Agency Comments and Our Evaluation 24
Appendix I Scope and Methodology 27
Appendix II Comments from the Department of Defense 29
Appendix III Software Development Capability Maturity Model 34
Appendix IV DOD Acquisition Categories, Thresholds, and Oversight 35
Appendix V GAO Contact and Staff Acknowledgments 37
Related GAO Products 38
Tables
Table 1: Cheyenne Mountain Mission Centers, Operations, and Systems 6
Table 2: Comparison of Initial and Current Estimates of CCIC2S Program
Costs from Inception through Fiscal Year 2006 (Then-Year Dollars in
Millions) 8
Table 3: Air Force Cost Estimate for CCIC2S through Fiscal Year 2007
Before Program Initiation and Designation of Acquisition Category Level
(Fiscal Year 1996 Constant Dollars in Millions) 17
Table 4: Software Capability Maturity Model(R) Scale 34
Table 5: DOD Acquisition Categories and Decision Authorities (as of March
15, 1996, When the Air Force Initiated the CCIC2S Program) 36
Figures
Figure 1: Cheyenne Mountain Systems Time Line of Upgrade Efforts 7
Figure 2: Status of Critical Capabilities to Be Delivered 9
Figure 3: CCIC2S Program Rebaselines and Their Impacts on Program Schedule
15
Abbreviations
ACAT acquisition category CMU Cheyenne Mountain Upgrade DCMA Defense
Contract Management Agency DOD Department of Defense EVM earned value
management MAIS Major Automated Information System MDAPS Major Defense
Acquisition Programs NORAD North American Aerospace Defense Command RAIDRS
Rapid Attack Identification Detection and Reporting System RDT&E research,
development, test, and evaluation SPADOC Space Defense Operations Center
TSPR Total System Performance Responsibility USNORTHCOM United States
Northern Command USSTRATCOM United States Strategic Command
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office
Washington, DC 20548
July 6, 2006
Congressional Committees
The Department of Defense's (DOD) Cheyenne Mountain Operations Center,
built in the early 1960s, currently houses numerous complex computer
systems intended to help monitor, process, and interpret air, missile, and
space events that could threaten North America or have operational impacts
on U.S. forces or capabilities. New threats have emerged over time that
have necessitated improved capabilities from Cheyenne Mountain systems.
Examples of these evolving threats include events such as the September
11, 2001, terrorist attacks on the U.S. and attacks on space assets that
could negatively impact military operations as well as the economy.
These systems-which are housed in three major centers: Air Warning,
Missile Correlation, and Space Control-require ongoing upgrades in order
to incorporate new mission capabilities and technologies and reduce the
cost of maintaining older systems. However, our reviews over the past 2
decades have found that DOD's efforts to modernize and integrate Cheyenne
Mountain systems have been fraught with cost increases, schedule delays,
and performance shortfalls due, in large part, to poor program management
and oversight. For example, in September 1994, we reported that upgrade
efforts were 8 years behind schedule and $792 million over budget, due to
development and integration problems stemming from management shortfalls.1
While DOD declared the most recent Cheyenne Mountain upgrades to be
operational in 1998, that same year, it determined that some of the
systems' components were not well integrated, were becoming unsupportable
because they were no longer produced, and would be unresponsive to future
mission needs. Subsequently, DOD initiated a program in 2000 to modernize
and integrate Cheyenne Mountain systems under a program called the
Combatant Commanders' Integrated Command and Control System, or CCIC2S,
and assigned overall responsibility for the program to the Air Force.
1GAO, Attack Warning: Status of Cheyenne Mountain Upgrade Program,
GAO/AIMD-94-175 (Washington, D.C.: Sept. 1, 1994).
We have recently testified that DOD's costly current and planned
acquisitions are running head-on into the nation's unsustainable fiscal
path.2 DOD starts more programs than it can afford and sustain, and in the
past 5 years, it has doubled its planned investments in programs. However,
these programs continue to experience recurring problems with cost
overruns, missed deadlines, and performance shortfalls. Our work has shown
that DOD will continue to experience such problems until it is able to
make oversight and management control improvements, including making sure
programs are executable, locking in requirements before programs are
initiated, and making clear who is responsible for what and holding people
accountable when these responsibilities are not fulfilled. Additionally,
our work has shown that once a program is initiated, inevitable changes to
the requirements baseline need to be controlled in terms of assessing the
cost, schedule, and performance implications of the changes. As part of
these assessments, the risks associated with these factors need to be
identified and mitigated.3
Within this context, we initiated this engagement under the authority of
the Comptroller General of the United States to (1) determine the status
of the CCIC2S program in terms of meeting its cost, schedule, and
performance goals; (2) gauge the extent to which DOD has followed best
practices with regard to managing program requirements, including matching
requirements to available resources; and (3) assess DOD's oversight and
control mechanisms for Cheyenne Mountain systems modernization and
integration efforts under way and planned. We are addressing this report
to you because of your committees' and subcommittees' jurisdictions.
In conducting our work, we spoke with officials from appropriate DOD, Air
Force, and contractor offices. We also reviewed DOD and Air Force
acquisition policies; planning documents; and program requirements, cost,
and schedule data. Additional information on our scope and methodology is
in appendix I. We conducted our work from July 2005 to April 2006 in
accordance with generally accepted government auditing standards.
2GAO, Defense Acquisitions: Actions Needed to Get Better Results on
Weapons Systems Investments, GAO-06-585T (Washington, D.C.: Apr. 5, 2006).
3GAO, Information Technology: DOD's Acquisition Policies and Guidance Need
to Incorporate Additional Best Practices and Controls, GAO-04-722
(Washington, D.C.: July 30, 2004).
Results in Brief
As with previous Cheyenne Mountain upgrade efforts, the CCIC2S program is
over cost, behind schedule, and some capabilities have been deferred
indefinitely. This deferral could pose risks to performing some future
operations. The Air Force initially estimated the CCIC2S program would
complete upgrades of critical air, missile, and space warning capabilities
in fiscal year 2006 at a program cost (including sustainment activities)
of approximately $467 million. However, the Air Force currently expects
the program to cost about $707 million through the same year-about a 51
percent increase-with no estimated completion date and without delivering
most mission critical capabilities. The deferral of capabilities has
significant implications for future missions-especially if program dollars
are needed to maintain legacy systems longer than expected or to modify
these systems. Operations to track man-made space objects could be
particularly affected, given that none of the work on CCIC2S's space
mission critical requirements has been completed and that estimated
completion dates for this work have yet to be determined. Deferring
expected capabilities may also affect programs that rely on CCIC2S for
their implementation, such as a defensive counterspace system that the Air
Force is developing, which is expected to use Cheyenne Mountain systems
information to help thwart attacks against valuable DOD space assets.
The ineffective management of CCIC2S's requirements has contributed to the
program's cost and schedule overruns. In contrast to best practices, which
call for stabilizing requirements and matching them to available
resources, CCIC2S's requirements were not established until 2004, more
than 3 years after the program began. The Air Force-which was responsible
for managing CCIC2S-did not effectively assess the appropriateness of the
program's requirements prior to initiating the program. Since the program
began in 2000, the Air Force has made significant additions, deletions,
and modifications to the initial requirements. At the same time, the Air
Force did not determine the effect of these changes on resources.
Consequently, the Air Force has rebaselined CCIC2S's cost and schedule
goals annually; currently, the program is undergoing its fifth
rebaselining. While rebaselining can provide an important perspective on a
program's current status, it can also obscure how programs perform over
time because a rebaseline shortens the period of reported performance and,
more importantly, resets the measurement of cost and schedule growth to
zero. With each rebaseline of the CCIC2S program, significant amounts of
work have been deferred to address cost increases. As a result, the Air
Force has implemented the program without reliable expectations of what
capabilities are to be delivered, when, and at what cost.
The oversight and control mechanisms of the CCIC2S program have been
inadequate to mitigate many of the problems it has experienced.
Specifically, (1) DOD did not designate the CCIC2S program as a major
automated information system acquisition, which would have required
high-level oversight and comprehensive and independent assessments of the
program; (2) the Assistant Secretary of the Air Force for Acquisition-the
CCIC2S milestone decision authority with overall responsibility for the
program-did not provide effective oversight; and (3) the Air Force's
contract management approach limited the program office's ability to
thoroughly assess the reliability of the contractor's cost and schedule
performance information and the impact of defining, prioritizing, and
adding capabilities. Additionally, the Defense Contract Management Agency
did not independently monitor contractor performance, although it signed a
memorandum of agreement with the program to do so in 2003. According to
DOD officials, actions-such as initiating a formal system-level technical
review process-are being taken to implement better controls and DOD is in
the process of determining whether to categorize the CCIC2S program as a
major automated information system acquisition.
To ensure the program addresses current shortcomings and proceeds on a
successful path, we are recommending that the Secretary of Defense direct
the Secretary of the Air Force to maintain Cheyenne Mountain's essential
operation and maintenance activities and limit future CCIC2S development
activities to those deemed critical to national security until DOD (1)
approves an acquisition approach that designates the program as a major
automated information system acquisition and delineates oversight and
accountability responsibilities, and (2) conducts an affordability
assessment, economic analysis, and independent estimate of life-cycle
costs. Furthermore, the Secretary of Defense should direct the Secretary
of the Air Force to establish and implement effective management controls
for the program by implementing an approach to manage requirements and
resource changes and ensure program cost and performance data are
reliable.
DOD agreed with designating the program as a major automated information
system acquisition, delineating responsibilities, and establishing and
implementing effective management controls for the program. In addition,
DOD agreed with conducting an affordability assessment, economic analysis,
and independent estimate of life-cycle costs on future development
activities but not with the current activities due to potential mission
impacts. We do not disagree with this approach to the extent that
continuing these development activities is critical to national security.
However, continuing CCIC2S development without establishing a match
between requirements and available resources may also hinder DOD's ability
to satisfy national security needs because of the increased risk that the
program fails to achieve its promised capability. DOD comments on a draft
of this report appear in appendix II.
Background
DOD built the Cheyenne Mountain Operations Center in the early 1960s at
Cheyenne Mountain Air Force Station, Colorado. Throughout its history, its
mission has continually evolved to adapt to changing threats-which have
ranged from a perceived Soviet nuclear threat after the 1957 launch of
Sputnik, the first man-made satellite successfully placed in orbit, to
theater ballistic missiles in the Persian Gulf during Operation Desert
Storm in the early 1990s. Currently, Cheyenne Mountain's mission is to
help monitor, process, and interpret air, missile, and space events that
could threaten North America or have operational impacts on U.S. forces or
capabilities, using air, ground, and space-based sensors that link to the
complex's computer systems located more than 2,000 feet under ground. The
President, Secretary of Defense, combatant commanders, and the Prime
Minister of Canada rely on Cheyenne Mountain systems to provide accurate,
timely, and unambiguous information essential for determining courses of
actions warfighters are to carry out. Air Force Space Command is
responsible for operating, funding, and setting requirements for the
systems, and Air Force Materiel Command's Electronic Systems Center is
responsible for maintaining the systems and for acquiring new systems and
capabilities.
Table 1 provides a summary of Cheyenne Mountain's three mission centers,
operations, and systems.
Table 1: Cheyenne Mountain Mission Centers, Operations, and Systems
Mission centers Operations Systems
Air warning
Supports the North Monitors North America's Air and ground-based
American Aerospace airspace to detect aircraft radars inside and
Defense Command (NORAD) or cruise missiles that along the periphery
and United States might violate airspace or of North America
Northern Command represent a threat, which
(USNORTHCOM) in their now includes warning of
missions to monitor and events such as the September
secure North American 11, 2001, terrorist attacks.
airspace
Missile correlation
Supports NORAD, Provides warning of missile A worldwide
USNORTHCOM, and United attacks launched against the communications and
States Strategic Command United States and its ground- and
(USSTRATCOM) in their territories, possessions, space-based sensor
strategic and theater and forces overseas and network
missions to protect U.S. supports USSTRATCOM in its
and Canadian interests mission to defend against
from ballistic missile strategic and theater
attacks missile attacks.a
Space control
Supports USSTRATCOM in Detects, tracks, identifies, A network of
its mission to protect and catalogs all man-made ground-based radars
U.S. space assetsb space objects orbiting the and optical sensors
Earth that are larger than as well as a
about 4 inches in size. space-based sensor
Source: DOD.
aMissile attacks can be characterized as strategic (long-range), such as
intercontinental ballistic missiles launched against the United States as
well as theater (short- and medium-range), such as the Scud missiles used
by Iraq during the Gulf War.
bA key element of this mission is space situational awareness, which
involves identifying man-made objects orbiting the Earth and determining
whom they belong to and their purposes.
Previous Cheyenne Mountain Upgrade Efforts
In 1981, the Air Force began efforts to modernize its Cheyenne Mountain
systems under five separate programs scheduled to be completed in 1987 at
a total estimated cost of $968 million. Subsequently, the Air Force
consolidated the five separate Cheyenne Mountain upgrade programs into a
single Cheyenne Mountain Upgrade (CMU) program. Beginning in the late
1980s, we issued a dozen reports on the program's cost, schedule, and
performance problems. For example, we reported that the Air Force did not
adequately define program requirements which resulted in an
underestimation of the upgrade efforts' cost and schedule goals. In its
attempts to maintain short-term cost and schedule goals, the Air Force
continually deferred requirements and problem resolution to the future, as
well as relaxed performance requirements. (See Related GAO Products at the
end of this report.)
Despite not meeting original performance expectations for the program, in
1998, the Air Force declared the Cheyenne Mountain upgrades to be
operational-almost $1 billion over cost and 11 years late. That same year,
DOD determined that some of the systems were not well integrated, would be
unresponsive to future mission needs, and some components were becoming
unsupportable. As a result, in 2000, the Air Force initiated the CCIC2S
program to modernize and integrate Cheyenne Mountain systems.4 Figure 1
depicts the timeline of Cheyenne Mountain upgrades.
Figure 1: Cheyenne Mountain Systems Time Line of Upgrade Efforts
On September 19, 2000, the Air Force awarded a cost-plus award fee
contract5 to Lockheed Martin Mission Systems6 to begin work on the CCIC2S
program. The initial period of performance was for 6 years, with annual
options for operations and maintenance activities for an additional 9
years.
4The program was initially known as the North American Aerospace Defense
Command/United States Space Command Warfighting Support System. The Air
Force changed the name of the program to CCIC2S in 2002. Also, in that
same year, DOD merged the responsibilities of the United States Space
Command into the United States Strategic Command.
5A cost-plus award fee contract is a cost-reimbursement contract that
provides for a fee consisting of a base amount fixed at inception of the
contract and an award amount, based upon a judgmental evaluation by the
government, sufficient to provide motivation for excellence in contract
performance.
6Lockheed Martin Mission Systems is now Lockheed Martin Integrated Systems
& Solutions.
CCIC2S Program Has Experienced Cost and Schedule Overruns and Performance
Shortfalls
Like its predecessor, the CMU program, the CCIC2S program is over cost,
behind schedule, and some capabilities have been deferred indefinitely.
This deferral could pose risks to performing some future operations. The
Air Force's current program cost estimates through fiscal year 2006
represent about a 51 percent increase over its initial estimate.
Additionally, while the Air Force initially estimated CCIC2S upgrades to
be completed in fiscal year 2006, most critical mission capabilities will
not be delivered by this time, and for some of these capabilities, the Air
Force has yet to estimate a completion date. The deferrals of capabilities
and performance shortfalls could have implications for future missions.
The CCIC2S Program Is Over Cost and Behind Schedule
As shown in table 2, the Air Force initially estimated the CCIC2S program
to complete upgrades in fiscal year 2006 at an estimated cost of about
$467 million.7 However, program cost estimates through the same year have
increased about 51 percent to nearly $707 million. Air Force officials
said they never estimated the life-cycle cost of delivering all CCIC2S
validated requirements.
Table 2: Comparison of Initial and Current Estimates of CCIC2S Program
Costs from Inception through Fiscal Year 2006 (Then-Year Dollars in
Millions)
Time frame of Fiscal year
cost estimate 2000 2001 2002 2003 2004 2005 2006 Total
Initial (July $4.7 $77.5 $78.4 $76.1 $78.6 $75.7 $76.0 $467.0
2000)
Current 4.7a 102.0a 107.0a 94.4a 133.4a 132.7 132.7 706.9
(February 2006)
Source: Air Force data, GAO analysis.
aActual expenditures.
Along with the increases in estimated program costs, schedules have
expanded significantly. The Air Force initially estimated development
activities to be completed in fiscal year 2006, but only a fraction of the
expected capabilities have been delivered (see fig. 2). Of the three
mission areas-air, missile, and space-the Air Force has delivered only
initial air and some initial missile mission critical capabilities. DOD
considers these critical capabilities to be the most essential for the
program-failure to provide these capabilities can be cause for a program
to be reassessed or terminated. The remaining missile warning critical
capabilities are now expected to be delivered in fiscal year 2007-over 3
years later than initially estimated. The Air Force has postponed the
delivery of all critical space mission capabilities to some undetermined
point in the future. According to Air Force officials, the space mission
capability deliveries will not be scheduled until future investment
decisions are made.
7This figure includes the development cost for the system as well as
sustainment activities through this period.
Figure 2: Status of Critical Capabilities to Be Delivered
According to the CCIC2S program office and other DOD officials, one of the
reasons the program's initial cost and schedule estimates differ from
current estimates is that work that was not included in the original
estimates was added to the program. Examples of work added to the program
include replacing an aging missile warning processing and display system
as well as developing a computer operational status monitoring system. We
did not assess the cost and schedule impacts of these added capabilities.
Also, as a result of the September 11, 2001, attacks on the U.S., the Air
Force diverted a portion of CCIC2S funding and personnel to make
improvements to other Cheyenne Mountain systems so that aircraft traffic
within the U.S. could be monitored by the air warning center. However,
according to DOD officials, the impacts of the September 11, 2001,
terrorist attacks on the CCIC2S program were not significant because
monitoring aircraft traffic within the U.S. is not a required upgrade
under the CCIC2S program. Prior to these improvements, the focus of the
air warning system was to monitor aircraft traffic outside the North
American border.
Deferral of Capabilities and Performance Shortfalls Have Implications for Future
Missions
The deferral of capabilities and performance shortfalls that have resulted
from cost and schedule overruns in the CCIC2S program could significantly
impact future missions-especially if program dollars are needed to
maintain legacy systems longer than expected. USSTRATCOM-responsible for
protecting U.S. space assets-could be particularly affected, given that
none of the work on CCIC2S's critical space mission capabilities has been
completed and that estimated completion dates for this work have yet to be
determined.
DOD has been counting on CCIC2S to replace the Space Defense Operations
Center (SPADOC)-Cheyenne Mountain's current space object tracking
system-and provide a system capable of processing larger volumes of data.
Some DOD officials have stated that CCIC2S is to be the foundation for
carrying out future space awareness functions. SPADOC's capabilities are
currently overtaxed. It currently handles about 400,000 space object
observations per day from sensors-about 167 percent more than it was
designed to handle. The Air Force has implemented an inefficient
workaround to address SPADOC's limited processing capabilities.
Specifically, the Space Control Center is using CaveNet, an off-line tool,
to assist with analyzing and processing space object observations.
However, to analyze and process the information in CaveNet, Space Control
Center operators must first download information from SPADOC and then
manually type the processed information back into SPADOC. Air Force
officials stated that manually typing CaveNet information back into SPADOC
has recently become significant enough to request an automated interface.
According to Air Force officials, efforts are on-going to assess the
mission impact of SPADOC performing beyond design specifications. With
other programs to improve data collection sensors that feed data into
CCIC2S-such as the Space Fence and Space Based Space Surveillance
system8-the number of observations and processing requirements will
increase significantly in the future, taxing the current system further.
In a 2005 study, the CCIC2S contractor determined that SPADOC's processing
capacity will not be able to handle the increased number of observations
predicted beginning in 2012. For the near term, according to Air Force
officials, the CCIC2S contractor has identified operations and maintenance
changes for SPADOC to handle increases in the number of observations.
However, DOD has not made a decision on when to pursue upgrades for SPADOC
and cost estimates for upgrading SPADOC have not been finalized. With the
delay of CCIC2S space capabilities and no decision on when SPADOC will be
replaced, USSTRATCOM will have to continue relying on pushing SPADOC to
its limits.
8The Space Fence is to replace the aging Air Force Space Surveillance
System, a ground-based radar, that currently performs detection and
tracking of orbiting space objects. The Space Based Space Surveillance
project is developing a constellation of optical sensing satellites to
find, fix, and track objects in Earth orbit.
Delays in the delivery of expected CCIC2S capabilities also affect
programs that are not directly part of the program but are relying on
CCIC2S for their implementation. For example, the Air Force's Rapid Attack
Identification Detection and Reporting System (RAIDRS)-a defensive
counterspace system-is expected to use information provided by Cheyenne
Mountain systems to aid in the detection, reporting, identification,
location, and classification of attacks against valuable space assets.
However, according to a DOD official, the CCIC2S space capability delays
have presented a serious acquisition dilemma for the RAIDRS program
because it is dependent on the CCIC2S space segment. Funding has not been
made available for the CCIC2S space segment that is to support RAIDRS,
according to Air Force Space Command. Furthermore, other programs will
experience the same acquisition dilemma because Air Force Space Command
has instructed all of its sensor, satellite, and weapon programs to be
able to integrate with CCIC2S.
Cost and Schedule Overruns Caused by Not Adhering to Best Practices for Managing
Program Requirements
The CCIC2S program's failure to match requirements and resources before
its start has contributed to the cost and schedule overruns that have
forced the program to defer delivery of CCIC2S capabilities. For each year
since the program's inception, the Air Force has rebaselined CCIC2S's cost
and schedule goals. The unreliable cost and schedule estimates that
resulted have forced the program to frequently defer work to later years
to control cost growth. The Air Force has yet to determine when and at
what cost development is to be completed.
Match Not Made between Program Requirements and Available Resources
Our body of work on best practices has shown that matching program
requirements to resources-that is, time, funding, technology, and
personnel-is key to successful outcomes.9 We have also reported that the
requirements-setting phase is one of the most important for
software-intensive acquisitions.10 Missing, vague, or changing
requirements tend to be a major cause of poor outcomes in these programs.
For example, according to Air Force officials, the major causes for the
delay in the Air Mission Evolution system11-a CCIC2S system that was
delivered in January 2004, over 2 years behind schedule-included problems
with software development and integration. Additionally, the contractor
stated that changing priorities within the program as well as added
requirements contributed to the delay.12
Without a stable set of requirements, the Air Force could not
appropriately estimate program costs and in hindsight, the resources
needed to satisfy program requirements were severely underestimated. To
better ensure programs are affordable and fully funded before they are
initiated, DOD guidance calls for every acquisition program to stabilize
requirements and determine the program's total costs before program
start.13 Despite best practices and DOD policy, the Air Force initiated
the CCIC2S program without establishing a stable set of program-specific
requirements and did not do so until January 2004-more than 3 years after
program start. In that time, the Air Force made numerous additions,
deletions, and modifications to CCIC2S's requirements. For example, the
Air Force deleted, added back, and then modified a requirement for
multiple security levels. According to an Air Force review of program
requirements, some deleted requirements were duplicative or conflicting.
9GAO, Best Practices: Better Support of Weapon System Program Managers
Needed to Improve Outcomes, GAO-06-110 (Washington, D.C.: Nov. 30, 2005).
10GAO, Defense Acquisitions: Stronger Management Practices Are Needed to
Improve DOD's Software-Intensive Weapon Acquisitions, GAO-04-393
(Washington, D.C.: Mar. 1, 2004).
11The Air Mission Evolution system was to replace the legacy air warning
system, called Granite Sentry, and provide new communications systems and
processing capability. The Air Mission Evolution system has not been
tested for interoperability with other CCIC2S systems; interoperability
was cited as one of the key shortcomings of the Granite Sentry legacy
system.
12Other aspects of requirements management which could contribute to cost
and schedule problems, include, for example, validating the completeness
and correctness of requirements, tracing and verifying contractual
requirements to higher-order program requirements, and delineating
mandatory versus optional requirements in deciding what requirements can
be eliminated or postponed to meet other project goals. We did not assess
these other aspects of requirements management and their impacts on
program execution.
13Memorandum for the Defense Acquisition Community, Subject: Update of the
DOD 5000 Documents, Office of the Secretary of Defense, March 15, 1996.
Currently, this guidance is located in DOD, Department of Defense
Instruction Number 5000.2, Operation of the Defense Acquisition System
(May 12, 2003).
Assessments of available resources were also inadequate. According to DOD
guidance, every acquisition program should rigorously address issues such
as the program's life-cycle costs and whether it is affordable and fully
funded.14 By not conducting these rigorous assessments, DOD and the Air
Force did not have the information needed to commit to the program over
its life cycle. Although the CCIC2S program acquisition plan refers to the
development of a life-cycle cost estimate before the program was
initiated, it did not include the operations and maintenance costs of
sustaining existing systems through the program's entire life cycle.15
Additionally, software development capabilities of the contractor should
also be assessed. The contract for CCIC2S development specifies that the
contractor be rated at the highest level (level 5) using the Capability
Maturity Model(R) for Software developed by Carnegie Mellon University's
Systems Engineering Institute.16 At this most mature level, quantitative
feedback about performance and innovative ideas and technologies
contribute to continuous process improvements. A senior program official
stated that the CCIC2S contractor was rated at a level 5 for software
development. According to DOD documentation, Capability Maturity Model(R)
assessments are typically performed at an organization wide level and may
not necessarily apply to a project team associated with any given
acquisition program. We did not examine the program's software development
efforts and the capabilities of the contractor. For more information on
this model, see appendix III.
14Memorandum for the Defense Acquisition Community, Subject: Update of the
DOD 5000 Documents, Office of the Secretary of Defense, March 15, 1996.
Currently, this guidance is located in DOD, Department of Defense
Instruction Number 5000.2, Operation of the Defense Acquisition System
(May 12, 2003).
15Air Force, Single Acquisition Management Plan for the NORAD/USSPACECOM
Warfighting Support System (N/UWSS) Program (July 28, 2000).
16Capability Maturity Model is registered in the U.S. Patent and Trademark
Office by Carnegie Mellon University.
Furthermore, DOD has not funded the program at requested levels since the
program began due to other priorities. We have previously reported that
program funding instability often occurs because DOD starts too many
programs, creating severe budget constraints. Another factor we have
reported that contributes to funding instability is that DOD's programs
are funded annually, requiring competition among other programs for
limited available funding.17 The CCIC2S program has not been an exception
to these factors. For example, prior to initiating the CCIC2S program in
2000, DOD reduced the program's initial estimated resource requirements by
a total of $75 million. In 2002, DOD decided not to increase requested
program funding. According to DOD, funding was limited to program baseline
levels because the program was a lesser priority. Furthermore, in 2005,
DOD and Congress reduced CCIC2S program funding estimates over the next 6
years by about $135 million, citing higher priority funding needs and poor
program performance.
The Air Force Has Implemented the CCIC2S Program without Solid Expectations of
What Capabilities Are To Be Delivered, When, and at What Cost
According to the acquisition strategy for the CCIC2S program, annual
rebaselines18 were to be conducted on the program to review and approve
program implementation plans, including cost and schedule baselines.
However, because the Air Force failed to make a match between requirements
and resources, including not effectively assessing the impacts of adding
unanticipated work to the program, the result of CCIC2S's annual
rebaselines has been to defer significant amounts of work to the future in
order to address cost increases. As a result, the Air Force has
implemented the program as a "level of effort"-that is, without solid
expectations of what capabilities are to be delivered, when, and at what
cost. Currently, the CCIC2S program is undergoing its fifth rebaseline.
According to program officials, after this fifth rebaseline, CCIC2S will
no longer plan for annual rebaselinings, but would only rebaseline the
program when necessary. Figure 3 depicts the effects of each of the
rebaselines on the program.
17GAO, DOD Acquisition Outcomes: A Case for Change, GAO-06-257T
(Washington, D.C.: Nov. 15, 2005).
18Rebaselining can occur at any time and cover any phase of a defense
acquisition program. All rebaselines must be approved by the milestone
decision authority, which has overall responsibility for the program.
Figure 3: CCIC2S Program Rebaselines and Their Impacts on Program Schedule
The fifth and current rebaseline was prompted by efforts to complete
development of the program's critical missile warning mission
capabilities. To free up resources for the missile warning development
efforts, the Air Force stopped work on many other aspects of the CCIC2S
program in late 2004 and subsequently began rebaselining the program in
early 2005. Air Force officials stated the rebaseline was to be finalized
by August 2005; however, it has yet to be approved as of June 2006.
According to Air Force officials, pending changes in program oversight as
well as recent DOD and congressional funding estimate reductions are
reasons for the delay in approving the rebaseline. As currently
structured, the rebaseline would again significantly defer the amount of
capability scheduled to be delivered. Specifically, missile warning
critical requirements would not be delivered until fiscal year 2007, and
the schedule for the work on the space portion is undetermined.
CCIC2S Program Oversight and Controls Have Been Ineffective
The oversight and control mechanisms of the CCIC2S program have been
inadequate to prevent or mitigate many of the problems it has experienced.
Additionally, the Air Force's contract management approach limited the
program office's ability to thoroughly assess the reliability of the
contractor's cost and schedule performance information and the impact of
defining, prioritizing, and adding capabilities. Furthermore, the Defense
Contract Management Agency (DCMA) did not independently monitor contractor
performance. According to DOD officials, actions are being taken to
implement better controls.
DOD Did Not Designate CCIC2S as a Major Acquisition
Placement into a specific acquisition category (ACAT) determines the
accountability level and analysis requirements of a program. DOD policy
requires major automated information system acquisitions to have
high-level oversight and comprehensive and independent assessments of
life-cycle cost estimates and benefits to ensure they are reasonable and
built on realistic program and schedule assumptions.19 In determining
whether an automated information system acquisition program warrants a
"major" designation, DOD applies dollar value thresholds. If a program
does not meet these thresholds, it can be designated as a major
acquisition based on other factors such as technical complexity. (See
appendix IV for more detail on the designation criteria for acquisition
programs that were in place when the Air Force initiated the CCIC2S
program.)
The CCIC2S program met the dollar threshold criteria for designation as a
major automated information system acquisition program. However, DOD did
not designate the program as a major automated information system
acquisition. According to a senior Air Force Space Command official, the
program was initially considered to be a weapon system acquisition and
therefore below the threshold criteria for designating the program as a
major defense acquisition program. At this lower acquisition category,
oversight for the program was assigned to Air Force headquarters.
Initial cost estimates for CCIC2S indicated that it should have been
designated as a major automated information system acquisition.
Specifically, at program initiation, CCIC2S's program cost estimate of
about $463 million well exceeded DOD's threshold of $360 million (fiscal
year 1996 constant dollars)-DOD's then-threshold for total life-cycle
costs-for designation as a major automated information system program (see
table 3).20
19Independent reviews are typically conducted by offices such as the
Office of the Director, Program Analysis and Evaluation and its Cost
Analysis Improvement Group.
Table 3: Air Force Cost Estimate for CCIC2S through Fiscal Year 2007
Before Program Initiation and Designation of Acquisition Category Level
(Fiscal Year 1996 Constant Dollars in Millions)
FISCAL YEAR
FUNDING TYPE 2000 2001 2002 2003 2004 2005 2006 2007 TOTAL
Research, $0.4 $13.0 $10.4 $10.2 $11.9 $11.5 $10.6 $10.7 $78.7
Development,
Test, and
Evaluation
Procurement 0.9 15.3 13.6 12.4 12.4 12.4 12.4 12.4 91.8
Operations 3.1 42.4 45.9 43.6 42.5 38.8 38.4 38.1 292.8
and
Maintenance
Total 4.4 70.7 69.8 66.2 66.7 62.8 61.4 61.2 463.2
Source: Air Force and modified by GAO to fiscal year 1996 constant
dollars.
Additionally, the CCIC2S program is a technically challenging and complex
acquisition. For example, one of the primary goals of the program is to
integrate 40 stovepiped systems into a single common architecture. The
complexity of the CCIC2S effort is further illustrated by our previous
work which showed that the CMU program had experienced long-standing,
serious integration problems, as well as software development problems
with incorporating new technologies.21
DOD and Office of Management and Budget have issued guidance22 on
conducting analyses that can help decision making on major automated
information system acquisitions. These analyses include
o an affordability assessment to demonstrate that the program's
resources (including projected funding and manpower requirements)
are realistic and achievable in the context of the DOD component's
overall long-range modernization plan to ensure full funding and
funding stability for the program;
o an economic analysis to determine the best program acquisition
alternative by assessing the net life-cycle costs and benefits of
the proposed program; and
o an independent estimate of program life-cycle costs.
20Department of Defense Regulation Number 5000.2-R, Mandatory Procedures
for Major Defense Acquisition Programs (MDAPs) and Major Automated
Information System (MAIS) Acquisition Programs (Mar. 15, 1996).
21GAO, Attack Warning: Lack of System Architecture Contributes to Major
Development Problems, GAO/IMTEC-92-52 (Washington, D.C.: June 11, 1992)
and Attack Warning: Better Management Required to Resolve NORAD
Integration Deficiencies, GAO/IMTEC-89-26 (Washington, D.C.: July 7,
1989).
22DOD, Defense Acquisition Guidebook, Version 1.0 (Oct.17, 2004); DOD,
Department of Defense Instruction Number 5000.2, Operation of the Defense
Acquisition System (May 12, 2003); and Office of Management and Budget,
Circular No. A-11: Planning, Budgeting, Acquisition and Management of
Capital Assets (June 21, 2005) and Circular No. A-94: Guidelines and
Discount Rates for Benefit-Cost Analysis of Federal Programs (Oct. 29,
1992).
Taken together, such reviews can give DOD the knowledge it needs to
prevent or mitigate many of the types of problems the CCIC2S program has
experienced.
We have consistently reported that the decision to invest in any system
should be based on reliable analyses of estimated system costs and
expected benefits over the life of the program.23 Without designation as a
major acquisition, the CCIC2S program lacked the high-level oversight and
independent analyses and review that could have identified key program
weaknesses, including
o funding instability;
o program cost, schedule, and performance problems;
o the mismatch between requirements and available resources; and
o the inability to prioritize and commit to the program within
the context of other acquisition programs and long-range
investment plans.
CCIC2S Milestone Decision Authority Did Not Provide Effective Oversight
DOD acquisition policy states that overall program responsibility rests
with the milestone decision authority. In this role, the milestone
decision authority is responsible for ensuring accountability and
maximizing credibility in program cost, schedule, and performance
reporting by reviewing the program throughout its acquisition life cycle,
including (1) whenever the program reaches a milestone decision point;24
(2) whenever costs, schedule, or performance goals are baselined or must
be changed; and (3) periodically through review of program status reports.
23For example, see GAO, DOD Systems Modernization: Planned Investment in
the Naval Tactical Command Support System Needs to Be Reassessed,
GAO-06-215 (Washington, D.C.: Dec. 5, 2005) and DOD Systems Modernization:
Continued Investment in the Standard Procurement System Has Not Been
Justified, GAO-01-682 (Washington, D.C.: July 31, 2001).
24According to DOD guidance, a milestone is a point at which a
recommendation is made and approval sought regarding starting or
continuing an acquisition program.
While the Assistant Secretary of the Air Force for Acquisition-the
designated milestone decision authority for the CCIC2S program-played an
active oversight role from 2000 to 2001, its program oversight thereafter
was minimal, including during the numerous program rebaselines and
attempts to address cost, schedule, and performance shortfalls. The
Assistant Secretary's oversight appears to have been limited to brief
program status updates provided by the program executive officer as part
of a weekly reporting on all programs under the program executive
officer's purview, and, according to DOD officials, annual and semi-annual
program execution and budget reviews. According to Air Force officials,
the Assistant Secretary delegated its milestone decision authority
responsibilities for the program to the program executive officer level.
However, officials from the Office of the Assistant Secretary of the Air
Force for Acquisition and the Office of the Program Executive Officer were
unable to provide us documentation of a formal delegation of these
responsibilities, or explain why this delegation took place.
Program Lacked Sufficient Management Controls over the Contractor
In addition to matching requirements to resources, our best practices work
has shown that to better ensure programs deliver capabilities within
expected costs and schedules, program offices must have in place certain
controls, including: (1) assessments of the effects of any changes in
requirements on cost, schedule, and performance goals and (2) insights
into contractor activities and progress to ensure commitments are
implemented.25 The contract to develop CCIC2S was awarded under a Total
System Performance Responsibility (TSPR) approach, which transfers certain
government management responsibilities from the program office to the
contractor.
In the mid-1990s, DOD began making sweeping changes to its acquisition
policy and procedures to streamline its acquisition infrastructure. One of
these reforms was the introduction of the TSPR concept, which gave a
contractor total responsibility for the integration of an entire weapon
system and for meeting DOD's requirements. According to DOD officials, the
Air Force no longer uses TSPR for its acquisition programs due to the
shortcomings resulting from using this management concept. We have
previously reported that TSPR contracts often result in the government's
lacking accurate cost, schedule, and performance information needed to
adequately manage programs because it had to rely on unverified contractor
data.26
25GAO, Information Technology: DOD's Acquisition Policies and Guidance
Need to Incorporate Best Practices and Controls, GAO-04-722 (Washington,
D.C.: July 30, 2004).
For CCIC2S, the risks inherent in a TSPR approach were exacerbated by the
program office's lack of sufficient staffing to oversee the definition and
prioritization of needed capabilities. Because the Air Force reduced the
number of personnel in the program office from 200 to between 15 and 30 in
an effort to achieve efficiencies, it lacked the personnel to assess the
cost and schedule impacts of defining and prioritizing requirements on the
program. Without such information, the program office was not in a
position to make trade-off decisions between different types of needed
capabilities. According to the program office, as a result of its
inability to manage the requirements definition and prioritization
process, the contractor ended up communicating directly with the users to
help define and prioritize requirements, sometimes without the knowledge
of the program office.
Program officials also noted that they lacked sufficient staff with the
technical knowledge to determine whether contractor cost, schedule, and
performance data were reliable. For example, according to the program
office and the CCIC2S contractor, the program initially called for a
contractor outside of the CCIC2S program to develop air mission software
and provide it to the CCIC2S contractor for integration into the program.
However, shortly after the Air Force initiated the CCIC2S program, it
terminated the air mission software development contract due to
unsatisfactory performance. Subsequently, the users recommended, and the
program office added, the unfinished work to the CCIC2S program as part of
the re-baseline process in January 2001. However, according to the program
office, it was not able to effectively assess the cost and schedule
impacts of this change on the program.
TSPR does not relieve the government of its oversight responsibilities.
Also, according to DCMA officials, the government should conduct contract
surveillance and gauge contractor performance using specific performance
metrics. However, the CCIC2S program office was not always aware of the
contractor's decisions or performance. For example, program office
officials acknowledged that they were not aware of some re-prioritization
of work done by the contractor to meet the users' needs.
26GAO, Defense Management: DOD Needs to Demonstrate That Performance-Based
Logistics Contracts Are Achieving Expected Benefits, GAO-05-966
(Washington, D.C.: Sept. 9, 2005).
Program Lacked Independent Contractor Performance Assessment
Acquisition programs commonly use DCMA to monitor higher risk contracts.
Typically, after DOD awards a contract, DCMA can monitor contractors'
information systems to ensure that cost, performance, and delivery
schedules are in compliance with the terms and conditions of the contract.
According to DCMA officials, however, their involvement in the CCIC2S
program from inception through 2004 was minimal because DCMA lacked
resources and the CCIC2S program office did not request DCMA assistance
until that time. This is despite the fact that in January 2003, the Air
Force and DCMA signed a memorandum of agreement calling for DCMA
assistance on the contract for the CCIC2S program.
The agreement called for DCMA to analyze the contractor's earned value
management (EVM) systems to verify that monthly cost performance reports
and related documents present a valid picture of contract cost and
schedule progress. This type of analysis could have identified problems
with the EVM system and led to corrective actions sooner. Since 2004, DCMA
has been working with the program office and Air Force systems engineering
officials to improve the quality of metrics to assess cost, technical
performance, and risk, as well as to make improvements to the contractor's
EVM system.
Needed Management Controls and Redesignation of CCIC2S Program Are Being
Considered
DOD is reconsidering CCIC2S's current designation as a non-major automated
information system program. In September 2005, the Assistant Secretary of
Defense for Networks and Information Integration notified the Assistant
Secretary of the Air Force for Acquisition that it intended to
recategorize the CCIC2S program as a major automated information system
acquisition because of increased funding levels, national interest in the
program, and our review of the program. Designating the CCIC2S program as
a major automated information system acquisition would put in place the
high-level oversight and rigorous analyses needed to help ensure (1) the
program has top-level accountability and support for the program, (2) DOD
has the knowledge necessary for making trade-offs in program requirements
so that they match available resources, and (3) DOD has the ability to
prioritize and commit to the program within the context of its other
acquisition programs and long-range investment plans. As of June 2006,
however, the office of the Assistant Secretary of Defense for Networks and
Information Integration has yet to formally make a decision on whether to
designate the program as a major automated information system acquisition.
According to Air Force and DCMA officials, the program office has taken
several steps, which we did not assess, to address some management control
problems, including the following actions:
o Prohibited direct contact between the contractor and
users-which they acknowledge had a direct impact on program cost
and schedule performance-to keep the contractor from
reprioritizing work to meet the immediate needs of the users, in
2004. Additionally, the program office and the contractor
implemented an engineering review board process to evaluate any
proposed change in terms of its validity in relation to validated
requirements as well as its impact to program cost and schedule
estimates.
o Initiated a User Feedback Control Board process to identify
where deliverables are not meeting users' needs and assess
identified solutions in terms of their relationship to validated
requirements as well as program cost and schedule implications,
beginning in the spring of 2005.
o Adopted a planning approach that is to more realistically
account for program risks through a consideration of limiting
factors such as test resources and the complexities surrounding
installing and testing hardware and software in Cheyenne
Mountain's operational environment.
o Took steps to add technical staff to assess the effects of
changes in requirements on cost, schedule, and performance goals
and provide additional insight into contractor activities.
o Implemented a formal system-level technical review process
consisting of milestones, or gates, each of which have entrance
and exit criteria. For example, deficiencies identified under the
formal review must be addressed prior to proceeding into the next
development phase.
Conclusions
Cheyenne Mountain systems-which are to warn the U.S. and Canada of air,
missile, and space attacks-are critical to national security. However,
since the 1980s, DOD's efforts to modernize these systems have been
fraught with cost increases, schedule delays, and performance shortfalls.
Under the current development effort, the Air Force continues to defer the
completion of key critical space requirements, placing the DOD's ability
to perform air and missile defense and space situational awareness
missions at risk. Without matching requirements to available resources,
the CCIC2S program will continue to flounder. Failing to conduct this
match at the onset of this complex effort led DOD to develop unrealistic
cost and schedule estimates as well as to over-promise capability.
Specifically, the Air Force did not ensure that the program had stable
requirements and a sound process for clarifying and controlling them, and
thus the resources to satisfy the requirements were severely
underestimated. The minimal top-level and external oversight and
accountability of the program contributed to the problems. Failure to make
course corrections now without correcting the shortfalls will put the
program at risk of failing to achieve promised capability.
Recommendations for Executive Action
We recommend that the Secretary of Defense direct the Secretary of the Air
Force to maintain Cheyenne Mountain's essential operation and maintenance
activities and to limit future development activities to those deemed
critical to national security needs until the Department takes the
following five actions:
o Designate the program as a major automated information system
acquisition.
o Conduct an affordability assessment to demonstrate that the
program's resource estimates are realistic and achievable in terms
of DOD's overall long-range modernization priorities and
investment plans for Cheyenne Mountain. Such an assessment would
require ensuring the requirements baseline is verified and
validated and making a match between these requirements and
available resources.
o Conduct an economic analysis to assess the life-cycle costs and
benefits of the program.
o Conduct an independent estimate of program life-cycle costs, to
provide a basis for relying on the cost estimates.
o Determine and clearly delineate oversight responsibilities and
accountability for the successful implementation of CCIC2S.
Additionally, the Secretary of Defense should direct the Secretary of the
Air Force to take the following three actions:
o Develop and implement an approach that requires a rigorous
analysis of resource impacts of any change in requirements for
continuously making trade-offs between requirements and resources
to ensure a match is maintained.
o Ensure management controls are in place so that changes to
requirements are decided on the basis of costs, benefits, risks,
and affordability.
o Develop and implement an approach for ensuring program cost and
performance data are reliable.
Agency Comments and Our Evaluation
DOD provided us with written comments on a draft of this report.
DOD agreed that the CCIC2S program be designated as a major automated
information system with the Assistant Secretary of Defense for Networks
and Information Integration as the milestone decision authority with
oversight responsibility and accountability for its successful
implementation. In addition, DOD agreed with conducting an affordability
assessment, economic analysis, and independent estimate of life-cycle
costs on future development activities but not with the current activities
due to potential mission impacts. Specifically, DOD stated these
assessments should be conducted on future delivery blocks of the CCIC2S
program, but not on the current block.27 DOD asserted that its recent $127
million CCIC2S funding reduction across the Future Years Defense Program
(FYDP)28 pared down CCIC2S development activities to those essential to
national security needs.
We do not disagree with this approach to the extent that continuing these
development activities is critical to national security. However,
continuing CCIC2S development without establishing a match between
requirements and available resources may also hinder DOD's ability to
satisfy national security needs because of the increased risk that the
program fails to achieve its promised capability. While DOD raises the
issue of the cost of delaying the program, it does not address the cost of
proceeding as planned. Our work has shown that in numerous programs,
continuing to spend money on development programs without sufficient
knowledge or clear direction results in cost overruns and schedule delays.
Further, given recent congressional concerns over DOD's management of
CCIC2S,29 demonstrating on a timely basis how requirements and available
resources are to be matched while protecting national security is
especially important. Since an assessment of CCIC2S's specific operational
risks to national security was beyond the scope of our review, we
clarified our recommendation to read that the Secretary of Defense direct
the Secretary of the Air Force to limit future development activities to
those deemed critical to national security needs until the Department
takes the actions discussed above.
27Under the draft rebaseline plan, the current development block is to
deliver critical missile warning capabilities in fiscal 2007 as well as
communications and space command and control infrastructure upgrades by
the end of fiscal year 2008.
28The FYDP is a DOD centralized report consisting of thousands of program
elements that provides information on DOD's current and planned out-year
budget requests and is one of DOD's principal tools to manage the spending
for its transformation of strategic capabilities. The FYDP provides
visibility over DOD's projected spending and helps inform DOD and Congress
about resource data relating to identifying priorities and trade-offs.
29House of Representatives, Department of Defense Appropriations Bill,
2007: Report of the Committee on Appropriations, Report No. 109-504 (June
16, 2006) and House of Representatives, National Defense Authorization Act
for Fiscal Year 2007: Report of the Committee on Armed Services, House of
Representatives on H.R. 5122, Report No. 109-452 (May 5, 2006).
DOD agreed to take actions relating to establishing and implementing
management controls and approaches to analyze the resource impacts of
changes in requirements and to ensure reliable program data. DOD stated
that these changes had already been implemented or are being planned.
During our review, program and DCMA officials told us that some
changes-such as prohibiting the contractor from reprioritizing work
without Air Force approval and implementing an engineering review board to
assess requirements changes-had been made. However, they also noted that
other measures to manage requirements and institute better management
controls-such as increasing program office management staffing and
capabilities, modifying the work breakdown structure and earned value
management system, and implementing a new approach and metrics to track
program progress-were under way or planned. Given DOD has not fully
demonstrated the effectiveness of the changes it has made or is making,
and that it is planning to make other changes, we are retaining our
recommendation.
We are sending copies of this report to interested congressional
committees and the Secretaries of Defense and the Air Force. We will also
provide copies to others on request. In addition, this report will be
available at no charge on the GAO Web site at http://www.gao.gov .
If you have any questions about this report or need additional
information, please call me at (202) 512-4841 ( [email protected] ). Contact
points for our Offices of Congressional Relations and Public Affairs may
be found on the last page of this report. GAO staff who made major
contributions to this report are listed in appendix V.
Lisa Shames Acting Director, Acquisition and Sourcing Management
List of Congressional Committees
The Honorable John Warner Chairman The Honorable Carl Levin Ranking
Minority Member Committee on Armed Services United States Senate
The Honorable Ted Stevens Chairman The Honorable Daniel K. Inouye Ranking
Minority Member Subcommittee on Defense Committee on Appropriations United
States Senate
The Honorable Duncan L. Hunter Chairman The Honorable Ike Skelton Ranking
Minority Member Committee on Armed Services House of Representatives
The Honorable C. W. Bill Young Chairman The Honorable John P. Murtha
Ranking Minority Member Subcommittee on Defense Committee on
Appropriations House of Representatives
Appendix I: Scope and Methodology Appendix I: Scope and Methodology
To determine the status of the Combatant Commanders' Integrated Command
and Control System, or CCIC2S, in terms of meeting its cost, schedule, and
performance goals, we reviewed our past work to determine whether the
CCIC2S program has continued to experience the problems previous Cheyenne
Mountain upgrade efforts had experienced. We also reviewed program status
reports and budget data and spoke with Department of Defense (DOD) and
contractor officials to assess program cost, schedule, and performance
progress to date. Additionally, we spoke with CCIC2S users to determine
their level of satisfaction with CCIC2S program performance. Specifically,
we reviewed documentation from and interviewed officials in the Office of
the Assistant Secretary of Defense for Networks and Information
Integration; National Security Space Office; Office of the Secretary of
Defense, Cost Analysis Improvement Group; Office of the Secretary of
Defense, Program Analysis and Evaluation; Office of the Director,
Operational Test and Evaluation; Office of the Joint Chiefs of Staff;
Defense Contract Management Agency's CCIC2S Program Support Team; Office
of the Assistant Secretary of the Air Force for Acquisition; Air Force
Space Command; Air Force Air Combat Command; Office of the Air Force
Program Executive Officer for Command and Control and Combat Support; Air
Force Electronic Systems Center; Air Force Operational Test and Evaluation
Center; United States Strategic Command; United States Northern Command;
North American Aerospace Defense Command; and Lockheed Martin Integrated
Systems & Solutions.
To gauge the extent to which DOD has followed best practices with regard
to managing program requirements, including matching requirements to
available resources, we reviewed DOD, Office of Management and Budget, and
Air Force acquisition guidance, as well as our previous best practices
work, to determine criteria for managing requirements. We also reviewed
program requirements documentation and spoke with DOD officials on how DOD
developed requirements for the CCIC2S program. Furthermore, we reviewed
requirements management documentation and spoke with DOD and contractor
officials to assess how the Air Force managed its requirements, including
the process it used for defining and prioritizing the requirements.
Specifically, we reviewed documentation from and conducted interviews with
officials in the Office of the Secretary of Defense, Cost Analysis
Improvement Group; Office of the Secretary of Defense, Program Analysis
and Evaluation; Office of the Director, Operational Test and Evaluation;
Office of the Joint Chiefs of Staff; Defense Contract Management Agency's
CCIC2S Program Support Team; Office of the Assistant Secretary of the Air
Force for Acquisition; Air Force Space Command; Office of the Air Force
Program Executive Officer for Command and Control and Combat Support; Air
Force Electronic Systems Center; Air Force Operational Test and Evaluation
Center; United States Strategic Command; United States Northern Command;
North American Aerospace Defense Command; and Lockheed Martin Integrated
Systems & Solutions.
To assess DOD's oversight and control mechanisms for Cheyenne Mountain
systems modernization and integration efforts underway and planned, we
reviewed DOD and Air Force acquisition guidance and interviewed officials
in the Office of the Assistant Secretary of Defense for Networks and
Information Integration; Office of the Secretary of Defense, Cost Analysis
Improvement Group; Office of the Secretary of Defense, Program Analysis
and Evaluation; Office of the Director, Operational Test and Evaluation;
Defense Contract Management Agency's CCIC2S Program Support Team; Office
of the Assistant Secretary of the Air Force for Acquisition; Air Force
Space Command; Office of the Air Force Program Executive Officer for
Command and Control and Combat Support; Air Force Electronic Systems
Center; Air Force Operational Test and Evaluation Center; United States
Strategic Command; United States Northern Command; North American
Aerospace Defense Command; and Lockheed Martin Integrated Systems &
Solutions.
We conducted our work from July 2005 through April 2006 in accordance with
generally accepted government auditing standards.
Appendix II: Comments from the Department of Defense Appendix II: Comments
from the Department of Defense
Appendix III: Software Development Capability Maturity Model Appendix III:
Software Development Capability Maturity Model
Carnegie Mellon University's Software Engineering Institute, a federally
funded research and development center, has identified specific processes
and practices that have proven successful in fostering quality software
development. The Software Capability Maturity Model(R)a rates maturity
according to the following five levels of maturity. See table 4 for a
description of these maturity levels.
Table 4: Software Capability Maturity Model(R) Scale
Level of Maturity Description
Level 1 (Initial) The software process is characterized as ad hoc.
Success depends on individual effort.
Level 2 (Repeatable) The basic process is in place to track cost,
schedule, and functionality. Some aspects of the
process can be applied to projects with similar
applications.
Level 3 (Defined) There is a standardized software process for the
organization. All projects use some approved version
of this process to develop and maintain software.
Level 4 (Managed) The organization uses and collects detailed data to
manage and evaluate progress and quality.
Level 5 (Optimizing) Quantitative feedback about performance and
innovative ideas and technologies contribute to
continuous process improvement.
Source: Software Engineering Institute, Carnegie Mellon University.
aCapability Maturity Model is registered in the U.S. Patent and Trademark
Office by Carnegie Mellon University.
Appendix IV: DOD Acquisition Categories, Thresholds, and Oversight
Appendix IV: DOD Acquisition Categories, Thresholds, and Oversight
Placement into a specific acquisition category (ACAT) determines the
accountability level and analysis requirements of a program. As shown in
table 5, for a major automated information system acquisition, the
milestone decision authority is the Assistant Secretary of Defense for
Networks and Information Integration, who is also the DOD Chief
Information Officer. The Assistant Secretary of Defense for Networks and
Information Integration may delegate the authority to the DOD component
acquisition executive (such as the Assistant Secretary of the Air Force
for Acquisition). For non-major automated information system acquisitions,
the milestone decision authority is the component acquisition executive or
his or her designee.
Table 5: DOD Acquisition Categories and Decision Authorities (as of March
15, 1996, When the Air Force Initiated the CCIC2S Program)
Dollar value (fiscal
Acquisition year 1996 constant
category (ACAT) dollars)a Milestone decision authority
ACAT I o Total expenditure The Under Secretary of Defense
for research, for Acquisition and
For major defense development, test, Technologyb designates ACAT I
acquisition and evaluation programs as ACAT ID or IC
programs (not for (RDT&E) of more than
automated $355 million, or for o Milestone Decision
information system procurement of more Authority for ACAT ID:
acquisition than $2.135 billion Under Secretary of Defense
programs) for Acquisition and
Technology
o Milestone Decision
Authority for ACAT IC: Head
of DOD Component (such as
the Secretary of the Air
Force), or, if delegated,
the DOD Component
Acquisition Executive (such
as the Assistant Secretary
of the Air Force for
Acquisition)
ACAT IA o Program costs in The Assistant Secretary of
any single year in Defense for Command, Control,
For major automated excess of $30 Communications, and
information system million Intelligence designates ACAT
acquisition o Total program IA programs as ACAT IAM or
programs costs in excess of ACAT IAC
$120 million
o Total life-cycle o Milestone Decision
costs in excess of Authority for ACAT IAMc
$360 million Assistant Secretary of
Defense for Command,
Control, Communications,
and Intelligence
o Milestone Decision
Authority for ACAT IAC: DOD
Component Chief Information
Officer (such as the Air
Force Chief of Warfighting
Integration and Chief
Information Officer, within
the office of the Secretary
of the Air Force)
ACAT II Does not meet ACAT I Component Acquisition
criteria Executive
For weapon system
acquisition Total expenditure for
programs (not for research, development,
automated test, and evaluation
information system (RDT&E) of more than
acquisition $140 million, or for
programs) procurement of more
than $645 million
ACAT III Does not meet criteria Designee of the DOD Component
for ACAT I, ACAT IA, or Acquisition Executive at the
ACAT II lowest level appropriate.
Source: DOD
aIf a program does not meet a dollar threshold, a program can be
designated as a major acquisition based on other factors such as technical
complexity.
bThe Under Secretary of Defense for Acquisition and Technology is now the
Under Secretary of Defense for Acquisition, Technology, and Logistics.
cCurrently, the Milestone Decision Authority for ACAT IAM programs is the
Assistant Secretary of Defense for Networks and Information
Integration-this Assistant Secretary is also the DOD Chief Information
Officer.
Note: These criteria have since been superseded with updated criteria.
This table is drawn from Department of Defense Regulation Number 5000.2-R,
Mandatory Procedures for Major Defense Acquisition Programs (MDAPS) and
Major Automated Information System (MAIS) Acquisition Programs (Mar. 15,
1996).
Appendix V: A Appendix V: GAO Contact and Staff Acknowledgments
GAO Contact
Lisa Shames (202) 512-4841
Acknowledgments
In addition to the contact named above, Art Gallegos, Peter Grana,
Randolph C. Hite, Arturo Holguin, Rich Horiuchi, Gary Mountjoy, Karl
Seifert, Karen Sloan, and Rona B. Stillman made key contributions to this
report.
Related GA Related GAO Products
Attack Warning: NORAD's Communications System Segment Replacement Program
Should Be Reassessed. GAO/IMTEC-89-1 . Washington, D.C.: November 30,
1988.
Space Defense: Management and Technical Problems Delay Operations Center
Acquisition. GAO/IMTEC-89-18 . Washington, D.C.: April 20, 1989.
Attack Warning: Better Management Required to Resolve NORAD Integration
Deficiencies. GAO/IMTEC-89-26 . Washington, D.C.: July 7, 1989.
Attack Warning: Defense Acquisition Board Should Address NORAD's Computer
Deficiencies. GAO/IMTEC-89-74 . Washington, D.C.: September 13, 1989.
Defense Acquisition: Air Force Prematurely Recommends ADP Acquisitions.
GAO/IMTEC-90-7 . Washington, D.C.: March 29, 1990.
Attack Warning: Costs to Modernize NORAD's Computer System Significantly
Understated. GAO/IMTEC-91-23 . Washington, D.C.: April 10, 1991.
Computer Technology: Air Attack Warning System Cannot Process All Radar
Track Data. GAO/IMTEC-91-15 . Washington, D.C.: May 13, 1991.
Attack Warning: Lack of System Architecture Contributes to Major
Development Problems. GAO/IMTEC-92-52 . Washington, D.C.: June 11, 1992.
Attack Warning: Status of the Survivable Communications Integration
System. GAO/IMTEC-92-61BR . Washington, D.C.: July 9, 1992.
Granite Sentry. GAO/IMTEC-92-84R . Washington, D.C.: September 21, 1992.
Attack Warning: Status of the Cheyenne Mountain Upgrade Program.
GAO/AIMD-94-175 . Washington, D.C.: September 1, 1994.
Defense Acquisitions: Status of Strategic C4 System Modernization and
Plans To Integrate Additional Mission Capabilities. GAO/NSIAD-00-212R .
Washington, D.C.: August 25, 2000.
(120473)
GAO's Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts newly
released reports, testimony, and correspondence on its Web site. To have
GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: [email protected]
Automated answering system: (800) 424-5454 or (202) 512-7470
Congressional Relations
Gloria Jarmon, Managing Director, [email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Washington,
D.C. 20548
Public Affairs
Paul Anderson, Managing Director, [email protected] (202) 512-4800 U.S.
Government Accountability Office, 441 G Street NW, Room 7149 Washington,
D.C. 20548
www.gao.gov/cgi-bin/getrpt? GAO-06-666 .
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Lisa Shames at (202) 512-4841 or
[email protected].
Highlights of GAO-06-666 , a report to congressional committees
July 2006
DEFENSE ACQUISITIONS
Further Management and Oversight Changes Needed for Efforts to Modernize
Cheyenne Mountain Attack Warning Systems
transparent illustrator graphic
The Cheyenne Mountain Operations Center houses numerous complex computer
systems for tracking air, missile, and space events that could threaten
homeland security or undermine military operations in theater. To ensure
this mission can be met, the systems require ongoing upgrades.
The most recent upgrade program -the Combatant Commanders' Integrated
Command and Control System (CCIC2S)-was initiated in 2000. Given the
critical missions supported by Cheyenne Mountain systems, GAO initiated a
review to (1) determine the status of the CCIC2S program in terms of
meeting its cost, schedule, and performance goals; (2) gauge the extent to
which DOD has followed best practices in managing program requirements;
and (3) assess DOD's control and oversight mechanisms for CCIC2S.
What GAO Recommends
GAO is recommending that DOD designate CCIC2S as a major acquisition
program; establish effective management controls; and conduct an
affordability assessment, economic analysis, and independent estimate of
life- cycle costs. DOD agreed to designate CCIC2S as a major acquisition
program and establish management controls. In addition, DOD stated that it
will conduct the affordability assessment and other analyses on future
CCIC2S development activities.
Like its predecessor, the Department of Defense's (DOD) CCIC2S program is
over cost, behind schedule, and some capabilities have been deferred
indefinitely which could pose risks to performing some future operations.
The Air Force, which has overall responsibility for the program, currently
estimates program costs will total about $707 million through fiscal year
2006-about a 51 percent increase over initial estimates. Schedules have
also expanded significantly, and most critical mission capabilities will
not be delivered in fiscal year 2006, as initially planned. The deferral
of capabilities and performance shortfalls has significant implications
for future missions-especially if program dollars are needed to maintain
legacy systems longer than expected. The tracking of space objects could
be particularly affected, given that none of the work on CCIC2S's critical
space mission capabilities has been completed, and estimated completion
dates for this work have yet to be determined.
Comparison of Initial and Current Estimates of CCIC2S Program Costs from
Inception through Fiscal Year 2006 (Then-Year Dollars in Millions)
Unstable program requirements and the failure to match requirements to
available resources have contributed to the program's cost and schedule
overruns. Since the program began in 2000, the Air Force has added,
deleted, and modified requirements without adequately determining the
effect of these changes on resources. To control cost growth, the Air
Force has frequently deferred work to later years and has yet to determine
when and at what cost development is to be completed.
Several key controls needed to mitigate the CCIC2S program's cost and
schedule problems are not in place. First, DOD did not designate the
CCIC2S program as a major automated information system acquisition-a
designation that would have required high-level oversight other than that
provided by the Air Force, which has been ineffective. In addition, the
Air Force's contracting approach has limited the program's ability to
thoroughly assess the reliability of the contractor's cost and schedule
performance information and the impact of defining, prioritizing, and
adding capabilities. Despite this risky approach, the Defense Contract
Management Agency did not independently monitor contractor performance.
According to DOD officials, actions are being taken to implement better
controls and to determine whether the CCIC2S program should be categorized
as a major automated information system acquisition.
*** End of document. ***