Missile Defense: Actions Needed to Improve Information for
Supporting Future Key Decisions for Boost and Ascent Phase
Elements (17-APR-07, GAO-07-430).
The Department of Defense (DOD) has spent about $107 billion
since the mid-1980s to develop a capability to destroy incoming
ballistic missiles. DOD has set key decision points for deciding
whether to further invest in capabilities to destroy missiles
during the initial phases after launch. In March 2006, DOD issued
a report on these capabilities in response to two mandates. To
satisfy a direction from the House Appropriations Committee, GAO
agreed to review the report. To assist Congress in evaluating
DOD's report and preparing for future decisions, GAO studied the
extent to which DOD (1) analyzed technical and operational issues
and (2) presented complete cost information. To do so, GAO
assessed the report's methodology, explanation of assumptions and
their effects on results, and whether DOD followed key principles
for developing life-cycle costs.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-430
ACCNO: A68306
TITLE: Missile Defense: Actions Needed to Improve Information
for Supporting Future Key Decisions for Boost and Ascent Phase
Elements
DATE: 04/17/2007
SUBJECT: Ballistic missiles
Cost analysis
Defense budgets
Defense capabilities
Defense cost control
Defense procurement
Future budget projections
Life cycle costs
Standards
Systems development life cycle
MDA Airborne Laser Program
MDA Kinetic Energy Interceptor System
MDA Terminal High Altitude Area Defense
System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-430
* [1]Results in Brief
* [2]Background
* [3]MDA's March 2006 Report Includes Some Useful Information but
* [4]Report Provided Some Basic Information about Boost and Ascen
* [5]MDA's Analyses Have Limitations but DOD Can Improve Informat
* [6]Data Collection and Analysis for the March 2006 Report
Did N
* [7]Assumptions and Their Effects on Data Presented in the
MDA R
* [8]DOD Can Significantly Improve Life-Cycle Cost Estimates to S
* [9]Cost Estimates Did Not Include All Cost Categories
* [10]Cost Estimates Were Not Assessed or Independently Verified f
* [11]Conclusions
* [12]Recommendations for Executive Action
* [13]Agency Comments and Our Evaluation
* [14]GAO Contacts
* [15]Acknowledgments
* [16]GAO's Mission
* [17]Obtaining Copies of GAO Reports and Testimony
* [18]Order by Mail or Phone
* [19]To Report Fraud, Waste, and Abuse in Federal Programs
* [20]Congressional Relations
* [21]Public Affairs
GAO
April 2007
MISSILE DEFENSE
Actions Needed to Improve Information for Supporting Future Key Decisions
for Boost and Ascent Phase Elements
GAO-07-430
Report to Congressional Committees
United States Government Accountability Office
Contents
Letter 1
Results in Brief 4
Background 7
MDA's March 2006 Report Includes Some Useful Information but Has Several
Limitations 10
DOD Can Significantly Improve Life-Cycle Cost Estimates to Support Future
Investment and Budget Decisions for Boost and Ascent Phase Capabilities 14
Conclusions 18
Recommendations for Executive Action 19
Agency Comments and Our Evaluation 20
Appendix I Scope and Methodology 25
Appendix II Comments from the Department of Defense 28
Appendix III GAO Contacts and Staff Acknowledgments 32
Related GAO Products 33
Table
Table 1: Ballistic Missile Defense Boost and Ascent Phase Elements, Key
Decision Points, and Their Planned Capability Dates 8
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
Abbreviations
ABL Airborne Laser
BMDS Ballistic Missile Defense System
DOD Department of Defense
FYDP Future Years Defense Program
KEI Kinetic Energy Interceptor
MDA Missile Defense Agency
United States Government Accountability Office
Washington, DC 20548
April 17, 2007
Congressional Committees:
The new security environment includes some states, such as North Korea,
that are aggressively pursuing development of long-range missiles and
weapons of mass destruction. In 2002, President Bush directed that the
Department of Defense (DOD) proceed with plans to develop and put in place
an initial set of ballistic missile defense capabilities beginning in
2004. DOD has spent about $107 billion since the mid-1980s to develop a
capability to destroy incoming ballistic missiles in all phases of their
flight, including the initial phases after launch, called the boost and
ascent phases.^1 The purpose of boost and ascent phase elements of a
ballistic missile defense system (BMDS) is to engage incoming missiles
before release of warheads or countermeasures. DOD is developing three
BMDS boost and ascent phase capabilities: Airborne Laser (boost phase
only), Kinetic Energy Interceptor (boost and ascent phase), and the Aegis
Ballistic Missile Defense element that will fire the Standard Missile-3
block 2A^2 (ascent phase) from BMDS-capable Aegis surface combatants.
According to the Missile Defense Agency (MDA), the agency responsible for
developing an integrated ballistic missile defense system, DOD plans to
spend $1.6 billion for the Airborne Laser; $800 million for the Kinetic
Energy Interceptor; and $200 million for the Aegis Standard Missile-3
Block 2A during fiscal years 2006-08. The long-term costs for these
elements will be much greater if DOD continues their development and
fielding. DOD projects the initial boost and ascent phase capabilities
will not be available until about 2014 through 2020. Congress has raised
questions about the affordability of pursuing both the Airborne Laser and
Kinetic Energy Interceptor programs.
Over the next few years, DOD and Congress will be asked to make important
program and investment decisions on boost and ascent phase
capabilities--at a time when the federal government is facing a large and
growing fiscal imbalance.^3 Further, we have identified the competing
demands of investing in transformational capabilities while continuing to
invest in legacy systems as one of the most significant national defense
challenges for the 21st century.^4 MDA has established key decision points
at which it will decide whether to continue to invest in certain elements
of the integrated system. For example, one key decision point is the
booster flight test for the Kinetic Energy Interceptor element, scheduled
for 2008, at which time a decision will be made on the program's future. A
key decision point for the Airborne Laser is the lethal shootdown
demonstration scheduled for 2009.^5 Finally, in 2009, MDA plans a motor
test for the Standard Missile-3 block 2A.
^1The boost phase is the period immediately after launch when the
missile's booster stages are still thrusting. Next is the ascent phase
which ends when the ballistic missile completes deployment of reentry
vehicles and possible decoys.
^2Other versions of the Standard Missile-3 are being developed to
intercept threat missiles in their midcourse phase.
In March 2006, MDA issued a report on boost and ascent phase capabilities
in response to two mandates--one in the House Appropriations Committee
Report on the Defense Appropriations Bill for Fiscal Year 2006 and one in
the National Defense Authorization Act for Fiscal Year 2006. The House
Appropriations Committee directed the Secretary of Defense to conduct a
study on boost and ascent phase intercepts including an assessment of
operational capabilities, quantities of operational assets required for
various deployment periods, basing options, and an assessment of
life-cycle costs.^6 Life-cycle costs are the total cost to the government
for a program over its full life, consisting of research and development,
production, operations, maintenance, and disposal costs and are helpful in
assessing whether a system's cost is affordable. The Department of Defense
Authorization Act for Fiscal Year 2006 required the Secretary of Defense
to conduct an assessment of U.S. missile defense programs that are
designed to provide capability against threat ballistic missiles in the
boost/ascent phase. The purpose of the assessment was to compare and
contrast capabilities, asset requirements, and the costs for making the
boost and ascent phase programs operational.^7 MDA, on behalf of DOD,
prepared one report to satisfy both requirements and sent the report to
all four defense committees on March 30, 2006. According to MDA officials,
that report was prepared specifically to satisfy the congressional
requirements, not for use at any key decision point.
^3GAO, Budget Process: Long-term Focus Is Critical, [22]GAO-04-585T
(Washington, D.C.: Mar. 23, 2004).
^4GAO, 21st Century Challenges: Reexamining the Base of the Federal
Government, [23]GAO-05-325SP (Washington, D.C.: February 2005).
^5Although there are other interim decision points leading up to the
lethal shootdown demonstration, MDA officials stated that the 2009 key
decision point is one where decisions may be made about the future of the
Airborne Laser program.
^6H.R. Rep. No. 109-119 (2005).
^7Pub.L. No. 109-163, S 231 (2005).
To satisfy a direction from the House Appropriations Committee report, we
agreed to review the DOD report including assessing the report's
methodologies, assumptions, completeness, and results.^8 This report is
our assessment of the March 2006 MDA report and how DOD can build on this
information to support future key decision points. Accordingly, we
assessed the extent to which (1) information reported by DOD includes
analysis of technical and operational issues and whether any additional
information is needed to support future decision making and (2) DOD
presented cost information to Congress that is complete and transparent.
To assess the technical and operational information, we analyzed the
extent to which the MDA report clearly explained technical maturity,
modeling assumptions and their effect on results, and involved
stakeholders in analyses of operational issues--steps which contribute to
a sound and complete quality study and are embodied in relevant generally
accepted research standards.^9 To assess the cost information in the March
2006 report, we compared MDA's cost data with key principles compiled from
various DOD and GAO sources that describe how to develop accurate and
reliable life-cycle cost estimates which are to include development,
production, and all direct and indirect operating and support costs. We
discussed the data and results of this comparison with DOD officials and
identified some limitations, which we discuss in this report. We conducted
our work between June 2006 and February 2007 in accordance with generally
accepted government auditing standards. See appendix I for a more complete
description of our scope and methodology.
^8While the House Appropriations Committee report originally directed us
to conduct a joint study with DOD, we agreed with the Defense
Subcommittee, House Appropriations Committee that, in lieu of a joint
study, we would assess the Missile Defense Agency's March 2006 report on
boost and ascent phase capabilities.
^9In a September 2006 report, GAO identified frequently occurring,
generally accepted research standards that are relevant for defense
studies and define a sound and complete study. GAO, Defense
Transportation: Study Limitations Raise Questions about the Adequacy and
Completeness of the Mobility Capabilities Study and Report,
[24]GAO-06-938 (Washington, D.C.: Sept. 20, 2006). Also, see app. I for a
more complete description of our scope and methodology.
This report is one in a series we have issued on ballistic missile defense
(see the list of related GAO products at the end of this report). Some of
these reports have focused on assessments of program goals and progress in
developing each element. For example, our March 2006 report found that MDA
made progress in the development and fielding of some elements but that
time pressures caused MDA to stray from a knowledge-based acquisition
strategy.^10 This is significant because straying from a knowledge-based
approach opens the door to greater cost and performance risks by not
having the right information available for decision makers at the right
time.^11 Another report focused on DOD's planning to operate BMDS. We
reported in May 2006 that DOD has not established criteria that would have
to be met before declaring BMDS operational and that the Future Years
Defense Program (FYDP), a major source of budget information, does not
provide complete and transparent data on ballistic missile defense
operational costs.^12
Results in Brief
MDA's 2006 report to Congress included some useful technical and
operational information on boost and ascent phase capabilities but the
information in the report has several limitations. Specifically, MDA's
report provided some key information on potential boost and ascent phase
capabilities by describing these elements, listing upcoming decision
points, and discussing geographic areas where these elements could
intercept missiles shortly after launch. However, the analysis did not
involve DOD stakeholders such as the services and combatant commands,
which will have a key role in operating the elements, and did not clearly
explain modeling assumptions and their effects on results as identified by
relevant research standards. For example, the report assumed that each
element would perform as desired and that the elements could be deployed
at optimum locations, but it did not explain how each element's
performance would change if developing technologies do not perform as
expected. In addition, the report did not address the challenges in
establishing bases at the locations cited or provide information on the
quantity of each element required for various deployment periods. Two key
reasons for these limitations, which the report acknowledges, are that
some data on the elements are limited at this time, and operational
concepts that discuss operations from forward locations have not been
fully vetted with the services and combatant commands. However, moving
forward, DOD has an opportunity to involve stakeholders in analyzing
operational and technical issues to provide more complete data that senior
DOD and congressional leaders can use to make informed program decisions
following key tests in 2008 and 2009 for the Kinetic Energy Interceptor
and Airborne Laser programs. To provide decision makers with information
to support boost and ascent phase program decisions at future key decision
points, we are recommending that DOD include all DOD stakeholders in
developing and analyzing operational issues regarding what is needed to
support operations at U.S. bases and potential forward locations, provide
specific information on the technical progress of each element, and use
the results of these analyses at each key decision point.
^10GAO, Defense Acquisitions: Missile Defense Agency Fields Initial
Capability but Falls Short of Original Goals, [25]GAO-06-327 (Washington,
D.C.: Mar. 15, 2006).
^11GAO, Missile Defense: Knowledge-Based Practices Are Being Adopted, but
Risks Remain, [26]GAO-03-441 (Washington, D.C.: Apr. 30, 2003).
^12GAO, Defense Management: Actions Needed to Improve Operational Planning
and Visibility of Costs for Ballistic Missile Defense, [27]GAO-06-473
(Washington, D.C.: May 31, 2006).
The Missile Defense Agency provided some cost estimates for developing and
fielding each of the boost and ascent phase capabilities in its March 2006
report to Congress; however, these estimates have several limitations and
will require refinement before they can serve as a basis for DOD and
congressional decision makers to compare the costs of boost and ascent
phase alternatives. We compared the report's cost estimates with various
DOD and GAO sources that describe key principles for developing accurate
and reliable life-cycle cost estimates. Based on our analysis, we found
that the estimates provided in the report did not include all cost
categories, including costs to establish and sustain operations at U.S.
bases and at forward locations. Also, MDA did not calculate costs based on
quantities the warfighter would need to conduct the mission, or conduct a
sensitivity analysis identifying the effects of cost drivers. Although the
report acknowledges that, at this time, there is uncertainty in estimating
life-cycle costs for these elements, it did not fully disclose the
limitations of the cost estimates. DOD can significantly improve the
completeness of and confidence in its cost estimates for boost and ascent
phase capabilities as it prepares for future investment and budget
decisions. For example, although MDA did not have the cost estimates in
its March 2006 report independently verified because doing so would have
required several months, MDA officials agree that independent
verification, which is another key principle of reliable cost estimates,
will be critical to support major decision points. We are recommending
that DOD prepare, periodically update, and independently verify life-cycle
cost estimates for boost and ascent phase elements in accordance with key
principles for developing accurate and reliable life-cycle cost estimates
and that these verified estimates be used for making investment decisions
and be reported with budget requests.
In commenting on a draft of this report, DOD agreed with our
recommendations regarding the need for analysis of technical progress and
operational issues to support key boost and ascent phase element decision
points. DOD also partially concurred that an independent life-cycle cost
estimate may be needed to inform some key decision points but said that
they may not be needed at others. However, DOD did not agree to prepare
and periodically update full life-cycle cost estimates for each boost and
ascent phase element to support key decision points, and report
independently verified life cycle cost estimates with budget requests and
FYDP funding plans. In its comments, DOD stated that it will direct an
independent evaluation of life-cycle costs if circumstances warrant or the
Director of MDA declares an element mature enough to provide a militarily
useful capability. However, if, as DOD's comments suggest, such costs are
not assessed until circumstances warrant or MDA's Director makes such a
declaration, these costs may not be available early enough to help shape
important program and investment decisions and consider trade-offs among
elements. We continue to believe our recommendation has merit because the
development of life-cycle cost estimates that include potential operations
and support costs would improve the information available to decision
makers and increase accountability for key decisions that could involve
billions of dollars at a time when DOD will likely face competing demands
for resources. Finally, DOD did not agree to report independently verified
life-cycle cost estimates along with budget requests and FYDP funding
plans since the development of total life-cycle cost estimates for
operationalized BMDS capabilities require agreement between MDA and the
lead military department on the roles and responsibilities for fielded
missile defense capabilities. However, independent verification allows
decision makers to gauge whether the program is executable. Also, DOD
procedures specify that when cost estimates are presented to the Office of
the Secretary of Defense's Cost Analysis Improvement Group, life-cycle
cost estimates should be compared with the FYDP and differences explained.
Therefore, we continue to believe that our recommendation has merit
because, without an independent cost estimate that can be compared to
budget requests and FYDP funding plans, congressional decision makers may
not have all the necessary information to assess the full extent of future
resource requirements if the boost and ascent phase capabilities go
forward, or assess the completeness of budget requests and FYDP funding
plans.
Background
In 2002, the Secretary of Defense created MDA to develop an integrated
system that would have the ability to intercept incoming missiles in all
phases of their flight.^13 In developing BMDS, MDA is using an incremental
approach to field militarily useful capabilities as they become available.
MDA plans to field capabilities in 2-year blocks. The configuration of a
given block is intended to build on the work completed in previous blocks.
For example, Block 2006 is intended to build on capabilities developed in
Block 2004, and is scheduled to field capabilities during calendar years
2006-07.
The integrated BMDS is comprised of various elements, three of which are
intended to intercept threat missiles in their boost or ascent phase.
Table 1 below describes each of these elements and shows the MDA projected
dates for key decision points, initial capability, and tested operational
capability.
^13The boost phase is the period immediately after launch when the
missile's booster stages are still thrusting and typically lasts 3-5
minutes for intercontinental ballistic missiles. The ascent phase is when
the booster stages have stopped thrusting and dropped away leaving a
warhead and possible decoys. The midcourse phase, lasting for about 20
minutes, begins after the missile has stopped accelerating and the warhead
travels through space on a predictable path. The final or terminal phase
begins when the warhead reenters the atmosphere and lasts approximately a
minute or less.
Table 1: Ballistic Missile Defense Boost and Ascent Phase Elements, Key
Decision Points, and Their Planned Capability Dates
Boost/ascent phase Key decision Initial Tested
element Description points capability capability
Standard Missile-3, Aegis Ballistic 2009--motor Block 2012 Block 2014
block 2A^a Missile Defense, test for the
which employs 21-inch
the Standard diameter
Missile, is a missile
ship-based
system designed
to destroy
medium,
intermediate,
and certain
intercontinental
ballistic
missiles in the
ascent and
midcourse
phases.
Kinetic Energy KEI is being 2008--booster Block 2014 Block to
Interceptor designed to flight test be decided
(KEI)--land-based destroy medium,
intermediate,
and
intercontinental
ballistic
missiles during
the boost,
ascent, and
midcourse phases
of flight. A
land-based unit
may consist of 5
launchers, 10
missiles, and
fire control and
communications
equipment.
Kinetic Energy KEI is being 2007--platform Block to Block to
Interceptor--sea-based designed to selection be decided be decided
destroy medium,
intermediate, 2008--booster
and flight test
intercontinental
ballistic
missiles during
the boost,
ascent, and
midcourse phases
of flight. DOD
is currently
studying
sea-based
platform
alternatives
including
surface
combatants,
submarines, or a
new platform.
Airborne Laser (ABL) ABL is an 2009--lethal Block 2016 Block 2018
air-based shootdown
missile defense demonstration
system designed
to destroy all
classes of
ballistic
missiles during
the boost phase.
ABL also has
onboard sensor
capability
designed to
detect, track,
and transmit
targeting data.
Source: GAO summary of DOD information.
aThe Standard Missile is intended to be used on BMDS upgraded Aegis
cruisers and destroyers. Other versions of the Standard Missile-3 are
being developed to intercept threat missiles in their midcourse phase.
During the past year, Congress requested additional information and
analyses on the boost and ascent phase elements from DOD. Specifically,
House Report 109-119 on the Department of Defense Appropriations Bill for
Fiscal Year 2006 directed the Secretary of Defense to conduct a study to
review the early engagement of ballistic missiles to include boost and
ascent phase intercepts and submit the report to the congressional defense
committees.^14 The report was to include, but not be limited to
o an assessment of the operational capabilities of systems against
ballistic missiles launched from North Korea or a location in the
Middle East against the continental United States, Alaska, or
Hawaii;
o an assessment of the quantity of operational assets required for
deployment periods of 7 days, 30 days, 90 days, and 1 year;
o basing options; and
o an assessment of life-cycle costs^15 to include research and
development efforts, procurement, deployment, operating, and
infrastructure costs.
In addition, the National Defense Authorization Act for Fiscal Year 2006
required the Secretary of Defense to assess missile defense programs
designed to provide capability against threat ballistic missiles in the
boost/ascent phase of flight. The purpose of this assessment was to
compare and contrast
o capabilities of those programs (if operational) to defeat
ballistic missiles from North Korea or a location in the Middle
East against the continental United States, Alaska, or Hawaii; and
o asset requirements and costs for those programs to become
operational with the capabilities referred to above.
MDA, on behalf of DOD, prepared one report to satisfy both of the above
requirements and sent the report to all four defense committees on March
30, 2006. The report included technical, operational, and cost information
for each of the three boost and ascent phase BMDS elements. The remainder
of this report discusses our assessment of the MDA report and how DOD can
build on this information to support future key decision points.
^14H.R. Rep. No. 109-119 (2005).
^15For most major acquisition programs, DOD prepares a life-cycle cost
estimate that is independently verified for major program reviews. A
life-cycle cost includes all costs associated with a weapon system's
research and development, investment, which includes military
construction, operations and support, and disposal.
MDA's March 2006 Report Includes Some Useful Information but Has Several
Limitations
MDA's March 2006 report to Congress included some useful technical and
operational information on boost and ascent phase capabilities. However,
the information in the report has several limitations--such as not
including stakeholders in the analysis or explaining how assumptions
affect results. Moving forward, DOD can enhance its ability to make
informed decisions at future key decision points by including stakeholders
DOD-wide in conducting analyses to provide complete technical and
operational information. Otherwise, senior DOD and congressional decision
makers may be limited in their ability to effectively assess the technical
progress and operational effects of proceeding with one or more boost and
ascent phase element.
Report Provided Some Basic Information about Boost and Ascent Phase Elements
The March 2006 report to Congress contained some useful technical and
operational information for Congress. For example, the report included a
detailed description of the three boost and ascent phase elements, which
could be useful for those unfamiliar with these elements. Additionally,
the report listed upcoming knowledge points where DOD will review the
progress MDA has made toward developing each of the boost and ascent phase
elements. Further, the report discussed geographic areas where boost and
ascent phase elements could intercept missiles shortly after launch based
on desired technical capabilities. Also, MDA used a model^16 to assess the
desired capabilities of each BMDS element for the March 2006 report to
Congress. Further, the modeling environment was used for several past BMDS
analyses and the results were benchmarked against other models. Finally,
MDA performed a sensitivity analysis that compared how the results in the
modeling changed when different assumptions for targets' propellants,
ascent times, hardness levels, and burn times were used. To provide
context, the report explained that the boost and ascent phase elements are
in the early stages of development and that the operational concepts are
not yet mature.
MDA's Analyses Have Limitations but DOD Can Improve Information to Support
Future Decisions
The information in the March 2006 report has several limitations because
the analyses did not involve stakeholders and did not clearly explain
modeling assumptions and their effects on results as identified by
relevant research standards. The relevant research standards and our prior
work have shown that coordination with stakeholders from study design
through reporting, and clearly explained assumptions and their effects on
results, can enable DOD officials to make fully informed program
decisions. As a result, the March 2006 report presents an incomplete
picture of technical capabilities, such as development challenges to be
overcome in order to achieve desired performance, and it does not clearly
explain the effects of operational assumptions, such as basing locations,
asset quantities, and base support requirements. As a step in the right
direction, MDA stated that it plans to develop criteria to assess the
boost/ascent phase elements at major decision points in a process
involving the combatant commands. Although MDA officials told us that they
consult stakeholders in a variety of forums other than the March 2006
report, they did not clearly state whether or how the services or other
DOD stakeholders would be involved in developing criteria for key decision
points or the extent to which their analyses would include information on
technical and operational issues.
^16The model, called WILMA, supports quantitative evaluation of ballistic
missile defense system performance in terms of probabilities of success.
Data Collection and Analysis for the March 2006 Report Did Not Involve Key DOD
Stakeholders
MDA's analyses did not involve soliciting or using information from key
DOD stakeholders such as the services, combatant commands, and joint staff
from study design through reporting. For example, officials from the
Office of the Secretary of Defense for Program Analysis and Evaluation and
the Defense Intelligence Agency stated there were areas where additional
information would have improved the fidelity of the results. First, the
officials stated that there is uncertainty that the boost and ascent phase
elements would achieve their desired capabilities within the timeframe
stated in the report. Second, officials from both organizations stated
that the report could have been enhanced by presenting different views of
the type and capability of threats the United States could face and when
these threats could realistically be expected to be used by adversaries.
Third, officials from the Office of the Secretary of Defense for Program
Analysis and Evaluation said that the MDA report did not distinguish
between countermeasures that could be used in the near term and
countermeasures that may be more difficult to implement. MDA officials
said that they worked with the Office of the Secretary of Defense for
Program Analysis and Evaluation in conducting analyses before they began
work on the March 2006 report. MDA also stated that it discussed the draft
March 2006 report with Office of the Secretary of Defense for Program
Analysis and Evaluation officials and included some of their comments in
the report's final version. However, without communication with
stakeholders from study design through reporting, MDA may not have had all
potential inputs that could have affected how the type, capability, and
likelihood of countermeasures to the boost and ascent phase elements were
presented in its report.
Additionally, MDA did not solicit information from the services, combatant
commands, or Joint Staff regarding operational issues that could have
affected information about basing and the quantities of elements that
could be required to support operations. Although the elements have to be
located in close proximity to their intended targets, and the report
discusses placing the elements at specific forward overseas locations, the
report does not include a basing analysis explaining what would need to be
done to support operations at these locations. Specifically, the report
did not include any discussion of the infrastructure or security/force
protection that will be needed for the BMDS elements. Although the report
mentions some support requirements--such as the Airborne Laser's need for
unique maintenance and support equipment and skilled personnel to maintain
the laser--the report did not fully explain how these support requirements
would be determined, who would provide or fund them, or explain the
operational effect if this support is not provided. For instance, without
an adequate forward operating location, the boost and ascent phase
elements would have to operate from much further away which would
significantly limit the time an element is in close proximity to potential
targets. Developing such information with the services, Joint Staff, and
combatant commands could provide a much more complete explanation of
operational issues and challenges. The services typically perform site
analyses to ascertain what support is needed for a new weapon system at
either a U.S. or overseas location. This comprehensive analysis examines a
range of issues from fire protection to security, to infrastructure, to
roads and airfields. In addition, U.S. Strategic Command and service
officials told us that this type of support must be planned for in advance
when adding a new system to any base, either in the United States or a
forward location.
MDA also did not involve stakeholders in assessing the quantities of each
element for deployment periods of 7 days, 30 days, 90 days, and 1 year.
The report stated that limited data exist at this time for a full
assessment of this issue, and service, Joint Staff, and MDA officials
acknowledged that the quantities of each element used in the report are
MDA assumed quantities. Service, Joint Staff, and U.S. Strategic Command
officials stated that they have not completed analyses to assess
quantities the warfighters may require. We understand that operational
concepts will continue to evolve and could affect required quantities.
However, stakeholders such as the services, Joint Staff, or combatant
commands could have assisted MDA in assessing potential quantities
required for various deployment periods. In addition, MDA did not solicit
information from the services, Joint Staff, or combatant commands to
determine if those organizations were conducting force structure analyses
for the boost and ascent phase elements. We learned that the Navy had done
a preliminary analysis in July 2005 and that the Joint Staff has begun a
capabilities mix study and both include, in part, an analysis of
quantities. Thus, in preparing for future decision points, MDA's analysis
could be strengthened by including stakeholders to leverage other
analyses. For example, MDA could have presented a range of scenarios to
show how the quantities required to intercept adversary missiles could
vary depending upon the number of sites covered and whether continuous,
near-continuous, or sporadic coverage is provided.
Assumptions and Their Effects on Data Presented in the MDA Report Were Not
Clearly Explained
The March 2006 report to Congress did not clearly explain the assumptions
used in the modeling of the BMDS elements' capabilities and did not
explain the effects those assumptions may have had on the results. First,
the model inputs for the technical analysis assumed desired rather than
demonstrated performance, and the report does not fully explain challenges
in maturing technologies or how these performance predictions could change
if the technologies are not developed as desired or assumed. For example,
although the model MDA used is capable of showing different results based
on different performance assumptions, the report did not explain how the
number of successful intercepts may change if less than 100 percent of the
desired technical capabilities are developed as envisioned. Thus the
results represent the best expected outcome. Second, the report does not
explain the current status of technical development or the challenges in
maturing each element's critical technologies as desired or assumed in the
report. DOD best practices define Technology Readiness Levels on a scale
of 1-9, and state which level should be reached to progress past specific
program decision points.^17 However, the March 2006 report does not
explain the current Technology Readiness Level for any of the boost and
ascent phase elements' critical technologies or the extent to which the
technology has to mature to attain the performance assumed in the report.
For example, the report does not explain that some of the technologies for
the Airborne Laser have to improve between 60 percent and 80 percent and
the report does not discuss any of the challenges MDA faces in doing so.
^17Technology Readiness Levels are measured on a scale of 1-9, beginning
with paper studies of a technology's feasibility (level 1) and culminating
with a technology fully integrated into a completed product (level 9).
DOD Can Significantly Improve Life-Cycle Cost Estimates to Support Future
Investment and Budget Decisions for Boost and Ascent Phase Capabilities
The March 2006 report to Congress provides cost estimates for each of the
boost and ascent phase capabilities; however, the cost estimates in the
report have several limitations that raise questions about their
usefulness. We compared the report's cost estimates with various DOD and
GAO sources that describe key principles for developing accurate and
reliable life-cycle cost estimates.^18 Based on our analysis, we found
that MDA did not include all cost categories, calculate costs based on
warfighter quantities, and did not conduct a sensitivity analysis to
assess the effects of cost drivers. Moreover, although MDA's report
acknowledges uncertainty in the cost estimates, the report does not fully
disclose the limitations of the cost estimates. DOD can significantly
improve the completeness of and confidence in cost estimates for boost and
ascent phase capabilities as it prepares for future investment and budget
decisions. For example, although DOD did not have its cost estimate for
its March 2006 report independently verified because doing so would have
taken several months, MDA officials agreed that independently verified
cost estimates will be critical to support major decision points for boost
and ascent phase capabilities.^19 In addition, as these capabilities
mature, MDA officials agreed that showing cost estimates over time and
conducting uncertainty analyses will be needed to support key program and
investment decisions.
Cost Estimates Did Not Include All Cost Categories
The cost estimates provided in the MDA report included some development,
production, and operations/support costs for each boost and ascent phase
element but were not fully developed or verified according to key
principles for developing life-cycle cost estimates. Life-cycle costs are
the total cost to the government for a program over its full life,
including the costs of research and development, investment, operating and
support, and disposal.
^18The criteria for developing accurate and reliable life-cycle cost
estimates were compiled from several sources including the DOD Manual
5000.4-M Cost Analysis Guidance and Procedures, the Defense Acquisition
Guidebook, and an exposure draft of GAO's Cost Assessment Guide. See app.
I for a more complete description of our scope and methodology.
^19As we reported in [28]GAO-05-817 , independent cost estimates are
normally first required for Milestone B, the decision to enter system
development and demonstration. GAO, Defense Acquisitions: Actions needed
to Ensure Adequate Funding for Operation and Sustainment of the Ballistic
Missile Defense System, [29]GAO-05-817 (Washington, D.C.: Sept. 6, 2005).
However, because BMDS elements do not enter DOD's acquisition cycle until
Milestone C, the requirements under Section 2434 of Title 10 may not be
applied to the BMDS until the transition phase that leads to the transfer
to a military service. However, developing cost estimates using the key
principles that we describe for developing accurate and reliable
life-cycle cost estimates would provide useful information to DOD decision
makers and Congress.
Based on our comparison of the life-cycle cost estimates in the report
with key principles for developing life-cycle cost estimates, we found
that the estimates were incomplete in several ways. First, the cost
estimates did not include all cost categories, such as costs to establish
and sustain operations at U.S. bases. Instead, MDA assumed that the
elements would be placed at existing bases with sufficient base support,
infrastructure and security; however, some of these costs such as
infrastructure could be significant. For example, an MDA planning document
cited about $87 million for infrastructure costs to support a ground-based
BMDS element (Terminal High Altitude Area Defense). Army officials
confirmed that training facilities, missile storage buildings, and a motor
pool were built at a U.S. base specifically to support this element and it
is likely that similar infrastructure would be needed to support the
land-based Kinetic Energy Interceptor. Additionally, MDA's cost estimates
did not include costs to establish and sustain operations at forward
overseas locations, even though the report states that the elements will
have to be located in close proximity to their targets, and the
operational concepts for Kinetic Energy Interceptor and Airborne Laser,
although in early development, state that these elements will be operated
from forward locations. Again, these are important factors to
consider--the Airborne Laser operational concept and the MDA report
acknowledge that unique support will be required to support operations at
any forward location for the Airborne Laser such as chemical facilities,
unique ground support equipment, and maintenance. Service, Joint Staff,
and U.S. Strategic Command officials also said that these elements would
have to be located forward and could be used as a strategic deterrent in
peacetime.
Second, the production and operating cost estimates were not based on
warfighter quantities, that is, quantities of each element that the
services and combatant commands may require to provide needed coverage of
potential targets. MDA assumed a certain quantity of each element. For
example, MDA officials told us that they assumed 96 Standard Missile-3
block 2A missiles because, at the time MDA prepared the report, they
planned to buy 96 block 1A missiles developed to intercept short-range
ballistic missiles. However, MDA did not solicit input from the services,
Joint Staff, or combatant commands on whether they had done or begun
analyses to determine element quantities.
Third, MDA did not conduct a sensitivity analysis to identify the effects
of cost drivers. A sensitivity analysis is a way to identify risk by
demonstrating how the cost estimates would change in response to different
values for specific cost drivers. Therefore, a sensitivity analysis should
be performed when developing cost estimates, and the results should be
documented and reported to decision makers. This means, for example, that
MDA could have computed costs with and without significant categories of
costs such as forward bases to identify the effect that adding forward
bases would have on operating costs. The House Armed Services Committee
report on the National Defense Authorization Bill for Fiscal Year 2006
recognized that operational capabilities and costs must be taken into
account when making decisions on future funding support. Finally, the cost
estimates did not estimate costs over time--a process known as time
phasing--which can assist decision makers with budgetary decisions. The
MDA report showed an annual cost estimate but did not state for how many
years the development, production, and operating costs may be incurred.
Although MDA officials stated they did not prepare time-phased cost
estimates in order to prepare the report to Congress in a timely manner,
they agreed that showing cost estimates over time would be important
information to support investment decisions at key decision points.
Cost Estimates Were Not Assessed or Independently Verified for MDA's 2006 Report
but MDA Officials See Need to Do So for Key Decision Points
Key principles for developing life-cycle cost estimates also include two
steps for assessing the confidence of cost estimates. However, MDA did not
take these steps to assess the confidence of the estimates reported in
March 2006. First, the Missile Defense Agency did not conduct a risk
analysis to assess the level of uncertainty for most of the cost estimates
in the MDA report. Risk and uncertainty refer to the fact that, because a
cost estimate is a prediction of the future, it is likely that the
estimated cost will differ from the actual cost. It is useful to perform a
risk analysis to quantify the degree of uncertainty in the estimates. By
using standard computer simulation techniques, an overall level of
uncertainty can be developed for cost estimates. In contrast, MDA
officials told us that they could only provide a judgmental confidence
level for the most of the cost estimates. Second, MDA did not have the
cost estimates in the report verified by an independent organization such
as DOD's Cost Analysis Improvement Group because doing so would have taken
several months. However, MDA officials agreed that independent
verification of cost estimates would be important information to support
investment decisions at key decision points. According to the key
principles that we have identified, all life-cycle cost estimates should
be independently verified to assure accuracy, completeness, and
reliability.^20 MDA has recognized the value in independently developed
cost estimates. In 2003, MDA and the Cost Analysis Improvement Group
developed a memorandum of understanding that said, in part, the Cost
Analysis Improvement Group would develop independent cost estimates for
the approved BMDS and its elements as appropriate during development in
anticipation of transition to production, but MDA officials said that
little work was completed under this agreement, which has expired.
Developing complete cost estimates in which decision makers can have
confidence is important since life-cycle cost estimates usually form the
basis for investment decisions and annual budget requests. Specifically,
life-cycle cost estimates that include all cost categories, show costs
over time, include warfighter quantities, include an assessment of cost
drivers, and are independently verified are important because accurate
life-cycle cost estimates can be used in formulating funding requests
contained in the President's Budget and DOD's future funding plan, the
Future Years Defense Program (FYDP) submitted to Congress. Therefore,
there is a need for DOD to provide transparent budget and cost planning
information to Congress. In May 2006, GAO reported that the FYDP, a major
source of budget and future funding plans, does not provide complete and
transparent data on ballistic missile defense operational costs because
the FYDP's structure does not provide a way to identify and aggregate
these costs. It is important that Congress has confidence in boost and
ascent phase estimates because Congress has indicated that it is concerned
with the affordability of pursuing both the Airborne Laser and Kinetic
Energy Interceptor programs in parallel through 2008.
As we reported in 2003, DOD assumes increased investment risk by not
having information available for decision makers at the right time, and
the level of anticipated spending magnifies this risk.^21 Otherwise,
senior DOD and congressional decision makers may be limited in their
ability to assess the relative cost of the elements if all cost categories
are not included and cost drivers are not identified. Considering
competing demands, this could also limit Congress's ability to consider
investment decisions or evaluate whether continued expenditures are
warranted. MDA officials stated that, in developing the cost estimates for
the March 2006 report, they decided not to follow some of the key
principles for developing life-cycle cost estimates such as time phasing
and independent verification of the cost estimates in order to complete
the report in a timely manner. However, the officials also agreed that
these key principles are important in developing complete, accurate, and
reliable life-cycle cost estimates for supporting investment decisions at
key decision points. Therefore, in the future, when preparing cost
estimates to be used in support of key decision points, MDA could provide
decision makers with more complete, accurate, and reliable cost estimates
by better adhering to key principles for developing life-cycle cost
estimates.
^20As we reported in [30]GAO-05-817 , independent cost estimates are
normally first required for Milestone B, the decision to enter system
development and demonstration. However, because BMDS elements do not enter
DOD's acquisition cycle until Milestone C, the requirements under Section
2434 of Title 10 may not be applied to the BMDS until the transition phase
that leads to the transfer to a military service. However, developing cost
estimates using the key principles that we describe for developing
accurate and reliable life-cycle cost estimates would provide useful
information to DOD decision makers and Congress.
^21 [31]GAO-03-441 .
Conclusions
Our review of MDA's March 2006 report on boost and ascent phase elements
identified a number limitations but helps to illuminate the kind of
information that DOD and congressional decision makers will need following
upcoming tests for boost and ascent phase elements. We recognize that the
March 2006 report was prepared in response to congressional direction
rather than to support program decisions. We also recognize that, at the
time of MDA's report, these elements were early in their development and
information was incomplete and changing. Thus, the focus of our analysis
was to identify additional information that could enhance future program
and investment decisions.
In particular, the House Armed Services Committee has raised questions
about the affordability of pursuing both the Kinetic Energy Interceptor
and the Airborne Laser in parallel through the projected knowledge point
demonstrations,^22 which are now scheduled for 2008 and 2009 respectively.
It is important that these decisions be both well-informed and transparent
because of the long-term funding consequences. DOD and congressional
decision makers' ability to assess which elements can be fully developed,
integrated, and operated relative to the others will be enhanced if they
have the benefit of information based on more rigorous analysis than that
contained in MDA's March 2006 report. Looking forward, as DOD strengthens
its analyses to support future key decisions, DOD and congressional
decision makers will be able to use more complete information to assess
force structure, basing, support, and infrastructure requirements, as well
as technical maturity, budget requests, and FYDP spending plans, in
deciding whether or not to continue developing one, two, or all three
boost and ascent phase elements and in what quantities.
^22H.R. Rep. No. 109-452 (2006).
Recommendations for Executive Action
To provide decision makers with information that enables them to clearly
understand the technical progress and operational implications of each
boost and ascent phase element and make fully informed, fact-based,
program decisions at future key decision points, we recommend that the
Secretary of Defense direct the Under Secretary of Defense for
Acquisition, Technology, and Logistics to take the following actions to
support key decision points for the BMDS boost and ascent phase elements:
o Include all DOD stakeholders (including services, combatant
commands, Joint Staff) in developing and analyzing operational
issues regarding what is needed to support operations at U.S.
bases and potential forward locations, including basing
assessments, force structure and quantity requirements,
infrastructure, security/force protection, maintenance, and
personnel.
o Provide specific information on the technical progress of each
element. Specifically, the analysis should explain current
technical maturity versus desired technical maturity and
capabilities of all major components and subsystems, reasonable
model inputs on element performance, and provide a clear
explanation of assumptions and their effect on results.
o Use the results of these analyses at each key decision point.
To provide decision makers with complete and reliable data on the costs of
each boost/ascent phase BMDS element to enhance investment and budget
decisions, we recommend that the Secretary of Defense take the following
actions:
o Direct the Under Secretary of Defense for Acquisition,
Technology, and Logistics to require MDA to prepare and--to
support key decision points--periodically update a full life-cycle
cost estimate for each boost/ascent phase element, in accordance
with key principles for developing accurate and reliable
life-cycle cost estimates, that includes all operational costs,
including costs to establish and sustain operations at U.S. bases
and forward locations, and that is based on warfighter quantities,
includes sensitivity analyses, and reflects time phasing.
o Direct an independent group, such as the Cost Analysis
Improvement Group, to prepare an independent life-cycle cost
estimate for each capability at each key decision point.
o Direct MDA and services to report independently verified
life-cycle cost estimates along with budget requests and FYDP
funding plans for each boost/ascent phase element.
Agency Comments and Our Evaluation
In written comments on a draft of this report, DOD agreed with our
recommendations regarding the need for analysis of technical progress and
operational issues to support key boost and ascent phase element decision
points. DOD also agreed that an independent life-cycle cost estimate may
be needed to inform some key decision points while they may not be needed
at other decision points. However, DOD did not agree to prepare and
periodically update full life-cycle cost estimates for each boost and
ascent phase element to support key decision points, and report
independently verified life cycle cost estimates with budget requests and
FYDP funding plans. As discussed below, we continue to believe our
recommendations have merit and that DOD should take the additional actions
we have recommended to provide a rigorous analytical basis for future
decisions, enhance the transparency of its analyses, and increase
accountability for key decisions that could involve billions of dollars.
The department's comments are reprinted in their entirety in appendix II.
DOD agreed with our recommendations that all DOD stakeholders be included
in developing and analyzing operational issues, that specific information
on technical progress be provided to explain current versus desired
capabilities, and that the results of both analyses be used at key
decision points. DOD stated in its comments that officials from MDA, the
military departments, the combatant commanders, and other organizations
are collaborating to develop an operational BMDS. Moreover, the annual
BMDS Transition and Transfer Plan is coordinated with the service
secretaries and other stakeholders and serves as a repository for plans,
agreements, responsibilities, authorities, and issues. DOD also stated
that key program decisions are and will continue to be informed by
detailed technical analysis, including assessment of element technical
maturity. However, DOD did not clearly explain how future decision making
will be enhanced or how analyses of operational issues will be conducted
if, as in the case of the Kinetic Energy Interceptor, DOD has not assigned
a service responsibility for operating the element once it is developed.
We continue to believe that DOD and congressional decision makers will
need more complete information on support requirements at upcoming
decision points as well as a clear comparison of current versus desired
technical capabilities in deciding whether or not to continue developing
one, two, or all three boost and ascent phase elements.
Regarding our recommendations to improve cost estimates used to support
key investment decisions, DOD partially concurred that independent
life-cycle cost estimates may be required to inform some key decision
points but stated that other key decision points may not. However, DOD did
not agree that it should routinely prepare and periodically update a full
life-cycle cost estimate for each boost and ascent phase element. DOD said
that it continuously assesses all aspects of its development efforts and
will direct an independent evaluation of life-cycle costs for boost and
ascent phase elements if circumstances warrant or if MDA's Director
declares an element mature enough to provide a militarily useful
capability. However, if, as DOD's comments suggest, such costs are not
assessed until circumstances warrant or MDA's Director declares an element
mature enough to provide a militarily useful capability, these costs may
not be available early enough to help shape important program and
investment decisions and consider trade-offs among elements. Moreover,
DOD's Operating and Support Cost Estimating Guide, published by the Cost
Analysis Improvement Group, states that when the Cost Analysis Improvement
Group assists the Office of the Secretary of Defense components in their
review of program costs, one purpose is to determine whether a new system
will be affordable to operate and support.^23 Therefore, such analysis
must be done early enough to provide cost data that will be considered in
making a decision to field, produce, or transition an element. We continue
to believe our recommendation has merit because the development of
life-cycle cost estimates that include potential operations and support
costs would improve the information available to decision makers and
increase accountability for key decisions that could involve billions of
dollars at a time when DOD will likely face competing demands for
resources.
Finally, DOD did not agree to report independently verified life-cycle
cost estimates along with budget requests and FYDP funding plans for each
boost and ascent phase element. DOD stated that operations and support
segments of the budget are organized by functional area rather than by
weapon system and are dependent on operations and support concepts of the
employing military department. DOD further stated that development of
total life-cycle cost estimates for operational BMDS capabilities requires
agreement between MDA and the lead military department on roles and
responsibilities for fielded BMDS capabilities that transcend the annual
transition planning cycle but serve as a basis for budget submittals. We
recently reported that MDA enjoys flexibility in developing BMDS but this
flexibility comes at the cost of transparency and accountability.^24 One
purpose of cost estimates is to support the budget process by providing
estimates of the funding required to efficiently execute a program. Also,
independent verification of cost estimates allows decision makers to gauge
whether the program is executable. Thus, cost estimating is the basis for
establishing and defending budgets and is at the heart of the
affordability issue. This principle is stated in DOD procedures which
specify that when cost results are presented to the Office of the
Secretary of Defense Cost Analysis Improvement Group, the program
office-developed life-cycle cost estimate should be compared with the FYDP
and differences explained.^25 Therefore, we continue to believe that our
recommendation has merit because, without an independent cost estimate
that can be compared to budget requests and FYDP funding plans,
congressional decision makers may not have all the necessary information
to assess the full extent of future resource requirements if the boost and
ascent phase capabilities go forward, or assess the completeness of the
cost estimates that are in the budget request and FYDP funding plans.
^23DOD, Office of the Secretary of Defense Cost Analysis Improvement
Group, Operating and Support Cost Estimating Guide (May 1992).
We are sending copies of this report to the Secretary of Defense; the
Commander, U.S. Strategic Command; the Director, Missile Defense Agency;
Chairman, the Joint Chiefs of Staff; and the Chiefs of Staff of the Army,
Navy, and Air Force. We will make copies available to others upon request.
In addition, the report will be available at no charge on the GAO Web site
at http://www.gao.gov .
^24GAO, Defense Acquisitions: Missile Defense Acquisition Strategy
Generates Results but Delivers Less at a Higher Cost, [33]GAO-07-387
(Washington, D.C.: Mar. 15, 2007).
^25DOD, Assistant Secretary of Defense (Program Analysis and Evaluation),
Cost Analysis Guidance and Procedures, DOD 5000.4-M (December 1992).
If you or your staff have any questions, please call either Janet St.
Laurent on (202) 512-4402 or Paul Francis on (202) 512-2811. Contact
points for our Offices of Congressional Relations and Public Affairs may
be found on the last page of this report. Staff members who made key
contributions to this report are listed in appendix III.
Janet A. St. Laurent
Director, Defense Capabilities and Management
Paul L. Francis
Director, Acquisition, Sourcing, and Management
List of Committees
The Honorable Carl Levin
Chairman
The Honorable John McCain
Ranking Minority Member
Committee on Armed Services
United States Senate
The Honorable Daniel K. Inouye
Chairman
The Honorable Ted Stevens
Ranking Minority Member
Subcommittee on Defense
Committee on Appropriations
United States Senate
The Honorable Ike Skelton
Chairman
The Honorable Duncan L. Hunter
Ranking Minority Member
Committee on Armed Services
House of Representatives
The Honorable John P. Murtha
Chairman
The Honorable C. W. Bill Young
Ranking Minority Member
Subcommittee on Defense
Committee on Appropriations
House of Representatives
Appendix I: Scope and Methodology
During this review, we focused on assessing the analytical approach the
Missile Defense Agency (MDA) used to develop its March 2006 report to
Congress, as well as the methodology for developing the cost estimates for
each of the three Ballistic Missile Defense System (BMDS) boost and ascent
phase elements.
To assess the extent to which the Department of Defense (DOD) is
developing technical and operational information useful for oversight and
that will support decision making at key points, we compared the
analytical approach DOD used to develop its March 2006 report with
generally accepted research standards that are relevant for defense
studies such as this, that define a sound and complete study, and that
cover all phases of a study--design, execution, and presentation of
results. The following were our sources for these standards:
o GAO, Government Auditing Standards: 2003 Revision,
[34]GAO-03-673G (Washington, D.C.: June 2003);
o GAO, Designing Evaluations, [35]GAO/PEMD-10.1.4 (Washington,
D.C.: March 1991);
o GAO, Dimensions of Quality, [36]GAO/QTM-94-1 (Washington, D.C.:
February 2004);
o RAND Corporation, RAND Standards for High-Quality Research and
Analysis (Santa Monica, Calif.: June 2004);
o Air Force, Office of Aerospace Studies, Analysts Handbook: On
Understanding the Nature of Analysis (January 2000);
o Air Force, Office of Aerospace Studies, Air Force Analysis
Handbook, A Guide for Performing Analysis Studies: For Analysis of
Alternatives or Functional Solution Analysis (July 2004);
o Department of Defense, DOD Modeling and Simulation (M&S)
Verification, Validation, Accreditation (VV&A), Instruction
5000.61 (Washington, D.C.: May 2003);
o Department of Defense, Data Collection, Development, and
Management in Support of Strategic Analysis, Directive 8260.1
(Washington, D.C.: Dec. 2, 2003); and
o Department of Defense, Implementation of Data Collection,
Development, and Management for Strategic Analyses, Instruction
8260.2 (Washington, D.C.: Jan. 21, 2003).
For a more complete description of these standards and how we identified
them, see [37]GAO-06-938 , appendix I.^1 In applying these standards, we
focused on the extent to which stakeholders were involved in study design
and analysis as well as the extent to which assumptions were reasonable
and their effects on results were clearly explained. We assessed MDA
briefings that explained the modeling used for the technical analysis
projecting the elements' capabilities. To assess the basis for the assumed
performance parameters used to model each element's performance, we traced
and verified a nonprobability sample of these parameters to their source
documentation and concluded that they were generally supported. To
evaluate the DOD report's characterization of threats, we reviewed Defense
Intelligence Agency documents and discussed the type and capability of
threats and expected BMDS capabilities with officials from the Office of
the Secretary of Defense for Program Analysis and Evaluation and the
Defense Intelligence Agency. In addition, to gain an understanding of the
extent to which DOD has assessed warfighter quantities for the boost and
ascent phase elements, the development of operational concepts, and
operational implications of employing the boost and ascent phase elements
at forward locations, we evaluated DOD and service guidance on assessing
sites and support for new weapon systems and discussed these issues with
officials from the Joint Staff; U.S. Army Headquarters and Space and
Missile Defense Command; U.S. Strategic Command; the office of the Chief
of Naval Operations Surface Warfare Directorate, Ballistic Missile Defense
Division; Air Combat Command; and the office of the Secretary of the Air
Force for Acquisition, Global Power Directorate. Finally, we discussed the
results of all our analyses with officials in the Joint Staff; U.S.
Strategic Command; the Army's Space and Missile Defense Command; Office of
the Secretary of Defense for Acquisition, Technology, and Logistics;
Missile Defense Agency; the office of the Chief of Naval Operations
Surface Warfare Directorate, Ballistic Missile Defense Division; the
office of the Secretary of the Air Force for Acquisition Global Power
Directorate; and Air Combat Command.
To assess the extent to which DOD presented cost information to Congress
that is complete and transparent, we first assessed how MDA developed its
estimates and then compared the method by which those estimates were
prepared to key principles compiled from various DOD and GAO sources that
describe how to develop accurate and reliable life-cycle cost estimates to
determine their completeness and the extent to which DOD took steps to
assess confidence in the estimates. The following were our sources for
compiling the cost criteria:
^1GAO, Defense Transportation: Study Limitations Raise Questions about the
Adequacy and Completeness of the Mobility Capabilities Study and Report,
[38]GAO-06-938 (Washington, D.C.: Sept. 20, 2006).
o Department of Defense, Assistant Secretary of Defense (Program
Analysis and Evaluation), Cost Analysis Guidance and Procedures,
DOD Manual 5000.4-M (December 1992);
o Department of Defense, Office of the Secretary of Defense Cost
Analysis Improvement Group, Operating and Support Cost Estimating
Guide (May 1992);
o Department of Defense, Defense Acquisition University, Defense
Acquisition Guidebook (online at http://akss.dau.mil/dag );
o Department of Defense, Defense Acquisition University,
Introduction to Cost Analysis (April 2006);
o Air Force, Office of Aerospace Studies, Air Force Analysis
Handbook: A Guide for Performing Analysis Studies for Analysis of
Alternatives or Functional Solution Analysis (July 2004);
o Air Force, Base Support and Expeditionary Site Planning, Air
Force Instruction 10-404 (March 2004); and
o GAO, GAO Cost Assessment Guide (currently under development).
In addition, we met with DOD officials from MDA, U.S. Strategic Command,
the Joint Staff, Army, Navy and Air Force to determine the extent to which
they were involved in developing the cost estimates for the DOD report.
Finally, we corroborated our methodology and results with officials from
the Office of the Under Secretary of Defense, Program, Analysis and
Evaluation (Cost Analysis Improvement Group) and the Office of the Under
Secretary of Defense (Comptroller) and they agreed that our methodology
for examining the report's cost estimates was reasonable and consistent
with key principles for developing accurate and reliable life-cycle cost
estimates. We identified some data limitations with the cost estimates
which we discuss in this report.
We provided a draft of this report to DOD for its review and incorporated
its comments where appropriate. Our review was conducted between June 2006
and February 2007 in accordance with generally accepted government
auditing standards.
Appendix II: Comments from the Department of Defense
Appendix III: GAO Contacts and Staff Acknowledgments
GAO Contacts
Janet A. St. Laurent, (202) 512-4402, [email protected]
Paul L. Francis, (202) 512-2811, [email protected]
Acknowledgments
In addition to the individuals named above, Barbara H. Haynes and
Gwendolyn R. Jaffe, Assistant Directors; Brenda M. Waterfield; Todd Dice;
Jeffrey R. Hubbard; Nabajyoti Barkakati; Hai V. Tran; Ron La Due Lake; and
Susan C. Ditto made key contributions to this report.
Related GAO Products
Defense Transportation: Study Limitations Raise Questions about the
Adequacy and Completeness of the Mobility Capabilities Study and Report.
[42]GAO-06-938 . Washington, D.C.: September 20, 2006.
Defense Management: Actions Needed to Improve Operational Planning and
Visibility of Costs for Ballistic Missile Defense. [43]GAO-06-473 .
Washington, D.C.: May 31, 2006.
Defense Acquisitions: Missile Defense Agency Fields Initial Capability but
Falls Short of Original Goal. [44]GAO-06-327 . Washington, D.C.: March 15,
2006.
Defense Acquisitions: Actions Needed to Ensure Adequate Funding for
Operation and Sustainment of the Ballistic Missile Defense System.
[45]GAO-05-817 . Washington, D.C.: September 6, 2005.
Military Transformation: Actions Needed by DOD to More Clearly Identify
New Triad Spending and Develop a Long-term Investment Approach.
[46]GAO-05-962R . Washington, D.C.: August 4, 2005.
Military Transformation: Actions Needed by DOD to More Clearly Identify
New Triad Spending and Develop a Long-term Investment Approach.
[47]GAO-05-540 . Washington, D.C.: June 30, 2005.
Defense Acquisitions: Status of Ballistic Missile Defense Program in 2004.
[48]GAO-05-243 . Washington, D.C.: March 31, 2005.
Future Years Defense Program: Actions Needed to Improve Transparency of
DOD's Projected Resource Needs. [49]GAO-04-514 . Washington, D.C.: May 7,
2004.
Missile Defense: Actions Are Needed to Enhance Testing and Accountability.
[50]GAO-04-409 . Washington, D.C.: April 23, 2004.
Missile Defense: Actions Being Taken to Address Testing Recommendations,
but Updated Assessment Needed. [51]GAO-04-254 . Washington, D.C.: February
26, 2004.
Missile Defense: Additional Knowledge Needed in Developing System for
Intercepting Long-Range Missiles. [52]GAO-03-600 . Washington, D.C.:
August 21, 2003.
Missile Defense: Alternate Approaches to Space Tracking and Surveillance
System Need to Be Considered. [53]GAO-03-597 . Washington, D.C.: May 23,
2003.
Missile Defense: Knowledge-Based Practices Are Being Adopted, but Risks
Remain. [54]GAO-03-441 . Washington, D.C.: April 30, 2003.
Missile Defense: Knowledge-Based Decision Making Needed to Reduce Risks in
Developing Airborne Laser. [55]GAO-02-631 . Washington, D.C.: July 12,
2002.
Missile Defense: Review of Results and Limitations of an Early National
Missile Defense Flight Test. [56]GAO-02-124 . Washington, D.C.: February
28, 2002.
Missile Defense: Cost Increases Call for Analysis of How Many New Patriot
Missiles to Buy. GAO/NSIAD-00-153. Washington, D.C.: June 29, 2000.
Missile Defense: Schedule for Navy Theater Wide Program Should Be Revised
to Reduce Risk. GAO/NSIAD-00-121. Washington, D.C.: May 31, 2000.
(350861)
GAO's Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site. To
have GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
[email protected] Automated answering system: (800) 424-5454 or (202)
512-7470
Congressional Relations
Gloria Jarmon, Managing Director, [email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Washington,
D.C. 20548
Public Affairs
Paul Anderson, Managing Director, [email protected] (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
www.gao.gov/cgi-bin/getrpt?GAO-07-430 .
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Janet A. St. Laurent at (202) 512-4402 or
[email protected]; or Paul Francis at (202) 512-2811 or [email protected].
Highlights of [64]GAO-07-430 , a report to congressional committees
April2007
MISSILE DEFENSE
Actions Needed to Improve Information for Supporting Future Key Decisions
for Boost and Ascent Phase Elements
The Department of Defense (DOD) has spent about $107 billion since the
mid-1980s to develop a capability to destroy incoming ballistic missiles.
DOD has set key decision points for deciding whether to further invest in
capabilities to destroy missiles during the initial phases after launch.
In March 2006, DOD issued a report on these capabilities in response to
two mandates. To satisfy a direction from the House Appropriations
Committee, GAO agreed to review the report.
To assist Congress in evaluating DOD's report and preparing for future
decisions, GAO studied the extent to which DOD (1) analyzed technical and
operational issues and (2) presented complete cost information. To do so,
GAO assessed the report's methodology, explanation of assumptions and
their effects on results, and whether DOD followed key principles for
developing life-cycle costs.
[65]What GAO Recommends
To support future decisions, DOD should include key stakeholders in
assessing operational issues, report on technical progress, and update and
verify life-cycle cost estimates in accordance with key principles for
developing life-cycle costs. In comments on a draft of this report, DOD
agreed to include stakeholders and assess technical progress but did not
agree to prepare or report life-cycle costs in accordance with key
principles.
The report DOD's Missile Defense Agency (MDA) submitted to Congress in
March 2006 included some useful technical and operational information on
boost and ascent phase capabilities by describing these elements, listing
upcoming decision points, and discussing geographic areas where boost and
ascent elements could intercept missiles shortly after launch. However,
the information in the report has several limitations because the analysis
did not involve key DOD stakeholders such as the services and combatant
commands in preparing the report and did not clearly explain modeling
assumptions and their effects on results as required by relevant research
standards. MDA's report states that, at this time, some data is limited,
and operational concepts that discuss operations from forward locations
have not been fully vetted with the services and combatant commands.
However, the report did not explain how each element's performance may
change if developing technologies do not perform as expected. Also, it did
not address the challenges in establishing bases at the locations cited or
provide information on the quantity of each element required for various
deployment periods. Moving forward, DOD has an opportunity to involve
stakeholders in analyzing operational and technical issues so that senior
DOD and congressional leaders will have more complete information on which
to base upcoming program decisions following key tests in 2008 and 2009
for the Kinetic Energy Interceptor and Airborne Laser boost and ascent
phase programs.
MDA's report provided some cost estimates for developing and fielding
boost and ascent phase capabilities, but these estimates have several
limitations and will require refinement before they can serve as a basis
for DOD and congressional decision makers to compare life-cycle costs for
the elements. MDA's report states that there is uncertainty in estimating
life-cycle costs because the elements are early in development. However,
based on a comparison of the estimates in the report with key principles
for developing life-cycle cost estimates, GAO found that MDA's estimates
did not include all cost categories, including costs to establish and
sustain operations at U.S. bases and at forward overseas operating
locations. Also, MDA's estimates did not calculate costs based on
realistic quantities of each element the combatant commanders or services
would need to conduct the mission. Finally, MDA did not conduct a
sensitivity analysis to assess the effect of key cost drivers on total
costs. MDA officials stated that further analysis of the costs for each
element along with measures to assess their confidence would help to
better inform DOD and congressional decision makers in making investment
decisions following key tests in 2008 and 2009.
References
Visible links
22. http://www.gao.gov/cgi-bin/getrpt?GAO-04-585T
23. http://www.gao.gov/cgi-bin/getrpt?GAO-05-325SP
24. http://www.gao.gov/cgi-bin/getrpt?GAO-06-938
25. http://www.gao.gov/cgi-bin/getrpt?GAO-06-327
26. http://www.gao.gov/cgi-bin/getrpt?GAO-03-441
27. http://www.gao.gov/cgi-bin/getrpt?GAO-06-473
28. http://www.gao.gov/cgi-bin/getrpt?GAO-05-817
29. http://www.gao.gov/cgi-bin/getrpt?GAO-05-817
30. http://www.gao.gov/cgi-bin/getrpt?GAO-05-817
31. http://www.gao.gov/cgi-bin/getrpt?GAO-03-441
33. http://www.gao.gov/cgi-bin/getrpt?GAO-07-387
34. http://www.gao.gov/cgi-bin/getrpt?GAO-03-673G
35. http://www.gao.gov/cgi-bin/getrpt?GAO/PEMD-10.1.4
36. http://www.gao.gov/cgi-bin/getrpt?GAO/GAO/QTM-94-1
37. http://www.gao.gov/cgi-bin/getrpt?GAO-06-938
38. http://www.gao.gov/cgi-bin/getrpt?GAO-06-938
42. http://www.gao.gov/cgi-bin/getrpt?GAO-06-938
43. http://www.gao.gov/cgi-bin/getrpt?GAO-06-473
44. http://www.gao.gov/cgi-bin/getrpt?GAO-06-327
45. http://www.gao.gov/cgi-bin/getrpt?GAO-05-817
46. http://www.gao.gov/cgi-bin/getrpt?GAO-05-962R
47. http://www.gao.gov/cgi-bin/getrpt?GAO-05-540
48. http://www.gao.gov/cgi-bin/getrpt?GAO-05-243
49. http://www.gao.gov/cgi-bin/getrpt?GAO-04-514
50. http://www.gao.gov/cgi-bin/getrpt?GAO-04-409
51. http://www.gao.gov/cgi-bin/getrpt?GAO-04-254
52. http://www.gao.gov/cgi-bin/getrpt?GAO-03-600
53. http://www.gao.gov/cgi-bin/getrpt?GAO-03-597
54. http://www.gao.gov/cgi-bin/getrpt?GAO-03-441
55. http://www.gao.gov/cgi-bin/getrpt?GAO-02-631
56. http://www.gao.gov/cgi-bin/getrpt?GAO-02-124
64. http://www.gao.gov/cgi-bin/getrpt?GAO-07-430
*** End of document. ***