Missile Defense: Actions Being Taken to Address Testing
Recommendations, but Updated Assessment Needed (26-FEB-04,
GAO-04-254).
In August 2000, the Defense Department's (DOD) Director,
Operational Test and Evaluation (DOT&E), made 50 recommendations
on a test program for a system to defeat long-range ballistic
missile threats against the United States. DOD's Missile Defense
Agency (MDA) plans to begin fielding the system by September
2004. GAO examined (1) how MDA addressed DOT&E's recommendations
and (2) what is known about the effectiveness of the system to be
fielded by September 2004. GAO issued a classified report on this
subject in June 2003. This unclassified, updated version reflects
changes in MDA's test schedule.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-04-254
ACCNO: A09333
TITLE: Missile Defense: Actions Being Taken to Address Testing
Recommendations, but Updated Assessment Needed
DATE: 02/26/2004
SUBJECT: Benchmark testing
Defense capabilities
Developmental testing
National defense operations
Strategic planning
Ballistic missiles
National preparedness
Aegis AN/SPY-1 Radar
Cobra Dane Radar
Integrated Flight Test
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-04-254
United States General Accounting Office
GAO
Report to Congressional Requesters
February 2004
MISSILE DEFENSE
Actions Being Taken to Address Testing Recommendations, but Updated Assessment
Needed
GAO-04-254
Highlights of GAO-04-254, a report to Congressional Requesters
In August 2000, the Defense Department `s (DOD) Director, Operational Test
and Evaluation (DOT&E), made 50 recommendations on a test program for a
system to defeat long-range ballistic missile threats against the United
States. DOD's Missile Defense Agency (MDA) plans to begin fielding the
system by September 2004.
GAO examined (1) how MDA addressed DOT&E's recommendations and (2) what is
known about the effectiveness of the system to be fielded by September
2004. GAO issued a classified report on this subject in June 2003. This
unclassified, updated version reflects changes in MDA's test schedule.
GAO recommends DOT&E report on the status of MDA's responses to its
recommendations and advise MDA how to modify the test program to address
long-standing concerns. DOD agreed with these recommendations, while
noting there have been many changes in its test program and its
acquisition strategy and structure since DOT&E's August 2000 report. GAO
revised the report to reflect the latest, approved test program but
believes most DOT&E recommendations remain relevant given the significant
technical challenges and uncertainties facing MDA.
February 2004
MISSILE DEFENSE:
Actions Being Taken to Address Testing Recommendations, but Updated Assessment
Needed
MDA is addressing most of DOT&E's recommendations on flight testing but
will not complete many actions before September 2004. For example, DOT&E
recommended removing flight test range limitations by adding more
intercept regions and launch locations to add greater realism to its
tests. MDA is expanding the test range infrastructure to add five
intercept regions and target and interceptor launches out of new
locations. By September 2004, one of the regions will be tested.
MDA is generally not addressing DOT&E's proposals on ground testing. For
example, although MDA had begun upgrading a ground facility to provide a
realistic testing environment for the interceptor, MDA deferred testing at
the facility to fund other priorities. Finally, MDA is addressing DOT&E's
recommendations on discrimination--the system's ability to find an enemy
warhead among decoys--by funding analysis programs.
Predictions of how well the system will defeat long-range ballistic
missiles are based on limited data. No component of the system to be
fielded by September 2004 has been flight-tested in its deployed
configuration. Significant uncertainties surround the capability to be
fielded by September: MDA will not demonstrate in flight tests a critical
radar called Cobra Dane before that date or conduct a system-level
demonstration, and has yet to test its three-stage boosters as part of a
planned intercept.
GMD Interceptor
www.gao.gov/cgi-bin/getrpt?GAO-04-254.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Bob Levin at (202) 512-4841
or [email protected].
Contents
Letter
Results in Brief
Missile Defense Agency Actions Taken or Planned to Address
DOT&E Recommendations
Characterization of Initial Defensive Capability
Conclusions
Recommendations for Executive Action
Agency Comments and Our Evaluation
1 3
5 14 18 18 19
Appendix I Scope and Methodology
Appendix II Comments from the Department of Defense
Abbreviations
BMDO Ballistic Missile Defense Organization
BMDS Ballistic Missile Defense System
CHOP Countermeasures Hands-On Program
DOD Department of Defense
DOT&E Director, Operational Test and Evaluation
DRR Deployment Readiness Review
EKV Exoatmospheric Kill Vehicle
FY fiscal year
GMD Ground-based Midcourse Defense
GPS Global Positioning System
HWIL Hardware in the Loop
IFICS In-Flight Interceptor Communications System
IFT Integrated Flight Test
IOC Initial Operational Capability
LFT&E Live Fire Test and Evaluation
LIDS Lead System Integrator (LSI) Integrated Distributed Simulation
MDA Missile Defense Agency
NMD National Missile Defense
NSPD National Security Presidential Directive
OPINE Operations in the Nuclear Environment
ORD Operational Requirements Document
UEWR Upgraded Early Warning Radar
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States General Accounting Office Washington, DC 20548
February 26, 2004
The Honorable Henry A. Waxman Ranking Minority Member Committee on
Government Reform House of Representatives
The Honorable John F. Tierney House of Representatives
The Missile Defense Agency (MDA) within the Department of Defense (DOD) is
developing and testing components of the Ground-based Midcourse Defense
(GMD) element, which is intended to defeat long-range ballistic missile
threats in the midcourse phase of flight.1 When deployed, GMD will include
(1) space-and ground-based sensors to provide early warning and tracking
of missile launches; (2) ground- and sea-based radars to identify and
refine the tracks of threatening missiles (called reentry vehicles) and
associated objects; (3) ground-based interceptors, each consisting of a
three-stage booster and kill vehicle, to destroy enemy missiles through
"hit-to-kill" impacts outside the atmosphere; and (4) fire control nodes
for battle management and execution of the GMD mission.
In August 2000, the Director, Operational Test and Evaluation (DOT&E),
submitted a report to the Under Secretary of Defense for Acquisition,
Technology, and Logistics for consideration at DOD's deployment readiness
review for the National Missile Defense system.2 The purpose of the report
was to provide an independent assessment of the system's potential
operational effectiveness and suitability. Based on its assessment of the
system at that time and the adequacy of testing in the context of
1 The midcourse phase of flight refers to that portion of a ballistic
missile's trajectory between the boost phase and reentry phase when the
warheads and decoys travel on ballistic trajectories above the atmosphere.
2 DOT&E is responsible for providing independent oversight of testing of
major DOD acquisition programs to ensure that operational test and
evaluation of major defense programs is adequate for verifying operational
effectiveness and suitability for combat use. The Director is the
principal operational test and evaluation official within DOD and advises
the Secretary of Defense and Under Secretary of Defense for Acquisition,
Technology and Logistics on operational test and evaluation. The Director
also provides advice to responsible officials on developmental testing.
deployment, it stated that test results supported a decision to continue
development but not deployment. In the report, DOT&E stated that the
current test program required augmentation and probably significant
funding increases to demonstrate an operationally effective system for
deployment. The report included a list of 50 detailed recommendations
regarding the test program. In September 2000, the President decided to
defer the deployment decision to the next administration, and MDA (then
named the Ballistic Missile Defense Organization) continued with
development of the system.
In January 2002, the Secretary of Defense refocused the ballistic missile
defense program into a broad-based research and development effort managed
by MDA. The new program follows a "capability-based approach" and aims at
developing layered defenses to intercept missiles in all phases of flight.
According to MDA, such an approach adds flexibility to the acquisition
process by permitting the development and testing of mature technologies
for the quick delivery of some capability. The new approach also allows
DOD to evolve and demonstrate additional improvements in missile defense
systems before committing to procurement and operations. To this end, MDA
is following a "block" approach, which defines, develops, produces or
acquires, and fields operational capability in incremental blocks. Each
block is designed to build capability into the system by introducing new
or improved technology. The first block- Block 2004-in this approach to
offer a defensive capability builds upon MDA's Ballistic Missile Defense
System (BMDS) Test Bed for conducting more realistic testing of ballistic
missile defense elements, of which GMD is the centerpiece.
In December 2002, the President directed DOD to begin fielding components
of the ballistic missile defense system for operational use by 2004. That
is, in addition to focusing resources on the development of the BMDS Test
Bed for developmental testing of missile defense elements, he instructed
MDA to build in an initial defensive capability that would protect the
United States against long-range missile attacks. The Secretary of Defense
stated that "...it would be a very preliminary, modest capability." DOD
decided to begin fielding an initial capability by September 2004.
Because of the significance of recommendations made by DOT&E during DOD's
deployment readiness review and the decision by the President to deploy an
initial defensive capability, you asked us to examine (1) the actions
taken or planned by MDA that address the recommendations made by DOT&E,
and (2) what is known about the effectiveness and limitations
Results in Brief
of the initial defensive capability to defeat long-range ballistic
missiles in the September 2004 time frame. The scope and methodology for
our review is included in appendix I.
We provided you with a classified report on this subject in June 2003.
This is an unclassified version of that report, which has also been
updated as of December 2003 to reflect changes in the GMD test program. In
preparing this unclassified version, we removed details on some of our
findings after an extensive declassification process. These details can be
found in the classified version of the report. The current version has
been reviewed by the DOD and approved for public release.
Of the recommendations listed in the August 2000 DOT&E report-which we
grouped under flight testing, ground testing, target discrimination, and
programmatics categories-MDA is taking actions that, over time, address
most of the flight testing recommendations but, by contrast, substantially
fewer of the remaining recommendations. Specifically,
o MDA is addressing most of the DOT&E's recommendations on flight
testing but will not complete many actions before September 2004. For
example, DOT&E recommended the removal of flight test range limitations by
adding more intercept regions and launch locations to achieve new
intercept geometries, higher closing velocities, and longer ranges flown
by the interceptor. MDA is expanding the test range infrastructure through
the development of the Test Bed to add five intercept regions and target
and interceptor launches out of new locations. By September 2004, MDA will
have tested in one of the five new intercept regions. Other DOT&E
recommendations on enhanced realism and the elimination of artificialities
in flight tests are being acted upon by MDA, but full realization will
take time to complete.
o MDA is generally not addressing DOT&E's recommendations on ground
testing. For example, although MDA had taken steps to proceed with the
design and construction of a hardware-in-the-loop laboratory at the Arnold
Engineering Development Center in Tennessee, comprehensive testing of the
kill vehicle in this facility has been deferred beyond Block 2004 because
of funding constraints. However, lethality testing for GMD engagements on
the ground has been enhanced with investments in the Hypersonic Upgrade
Program facility at Holloman Air Force Base.
o MDA is generally addressing DOT&E's recommendations on
discrimination-the system's ability to identify the true reentry vehicle
from among decoys and associated objects. MDA has substantially increased
the scope of work being done in discrimination, but MDA's flight tests
planned through September 2007 are not designed to address the
challenge posed by the certain countermeasures that DOT&E identified as
simple for an enemy to implement. MDA maintains that the complexity and
challenge of target suites used in integrated flight tests will increase
as the discrimination capability is incrementally proven.
The predicted effectiveness of the initial defensive capability to defeat
long-range ballistic missiles is currently based on a limited set of
flight-test data. For example, none of the components of the initial
defensive capability to be fielded in September 2004-interceptors, fire
control nodes for battle management and execution, upgraded radars, and
forward-deployed Aegis radars on Navy cruisers and destroyers-has been
flight-tested in its deployed configuration. As such, current predictions
of effectiveness are based on analysis and simulations of expected
performance of constituent components. The most significant uncertainties
surrounding the capability to be fielded in September 2004 are (1) MDA
does not plan to demonstrate capabilities of a critical radar for use with
the GMD element, called Cobra Dane, in flight tests before that date; (2)
MDA has yet to test both three-stage boosters as part of an attempted
intercept; and (3) MDA does not plan to conduct a system-level
demonstration of the initial defensive capability in flight testing before
September 2004. Furthermore, a notable limitation of system effectiveness
is the inability of system radars to perform rigorous target
discrimination. The Cobra Dane radar and the upgraded early warning radar
in California can perform rudimentary target discrimination, but the kill
vehicle itself must perform final target selection during the endgame.
The recommendations in this report also appeared in our classified June
2003 report. We have made these recommendations to provide decision makers
DOT&E's assessment of MDA's actions related to concerns raised by DOT&E's
August 2000 report. In commenting on a draft of our June 2003 report, DOD
agreed with our recommendations. The department raised concerns, however,
that the GMD test program as described in this report is no longer current
and the program strategy and structure have changed since DOT&E submitted
its report. While the GMD test program has, indeed, been in a constant
state of flux, thus complicating our analysis, our report presents the
latest, approved test program information provided to us by MDA. Also,
despite alterations to the acquisition strategy and structure of the
ballistic missile defense system, we believe most of the DOT&E
recommendations are still relevant because the technical challenges and
uncertainty with developing, testing, and fielding effective defensive
capabilities, as identified in the August 2000 DOT&E report, remain
significant.
Missile Defense Agency Actions Taken or Planned to Address DOT&E
Recommendations
The August 2000 DOT&E report summarized the progress, up to that date, of
the National Missile Defense program and the adequacy of testing in the
context of a deployment decision.3 At the time, the development program
revolved around a series of ground and flight tests and was to have
culminated in an initial operational capability by the end of fiscal year
2005. Formal test documentation called for a total of 16 integrated flight
tests (system-level intercept attempts) through 2004 with three additional
flight tests during Initial Operational Test and Evaluation during the
2005 fiscal year. DOT&E's principal finding was that ground and flight
tests completed up to that time did not provide results of sufficient
fidelity to support a deployment decision. Indeed, when the deployment
readiness review was held, there had been two failed intercepts out of
three attempts. Furthermore, as stated in the DOT&E report, ground testing
was not adequate to yield credible estimates of GMD system performance.
DOT&E indicated that the current test program required augmentation and
probably significant funding increases to demonstrate an operationally
effective system for deployment. Accordingly, the report included a list
of detailed recommendations for enhancing the test program.
DOT&E made 50 specific, interrelated recommendations, which we organized
into the following four overarching categories: Flight Testing, Ground
Testing, Target Discrimination,4 and Programmatics. Although DOT&E
categorized discrimination-related recommendations under the
flight-testing and ground-testing categories, we created a separate
category because discrimination was of principal concern to DOT&E at the
time. DOD classified the full text of the recommendations. A detailed
assessment indicating whether actions have been initiated by MDA and what
their timing is relative to the September 2004 initial defensive
capability date can be found in our June 2003 classified report on this
subject. A summary of MDA actions to address the DOT&E recommendations is
provided below.
DOT&E Recommendations Integrated flight tests of the GMD element are
demonstrations of system on Flight Testing performance during which an
interceptor is launched to engage and intercept a target reentry vehicle
(mock warhead) above the atmosphere.
3 The National Missile Defense program is now referred to as the
Ground-based Midcourse Defense (GMD) program, terminology we use from this
point on.
4 Target discrimination is the identification of the true reentry vehicle
from among decoys and associated objects.
Flight Test Complexity
Many recommendations (20 of 50) in the DOT&E report pertain to aspects of
integrated flight testing, such as deficiencies in flight-test complexity,
operational realism, and artificialities. DOT&E's concerns with the
composition of target suites in flight tests for testing discrimination
are discussed separately in the discrimination section of this report.
DOT&E reported that increasing the scope of flight testing was essential
to stress the limits of system design and to keep pace with system
development.
MDA is taking actions that address many of the shortcomings in flight
testing DOT&E identified in its August 2000 report. Indeed, the
development of the BMDS Test Bed-the agency's key instrument for enhancing
the existing test infrastructure to provide more realistic testing-should
go far in addressing these DOT&E recommendations over the long term.
Currently, flight tests are limited to target launches out of Vandenberg
Air Force Base, California, and interceptor launches out of Kwajalein
Missile Range in the western Pacific.5 For enhancing the capabilities of
integrated flight testing, the test bed adds an interceptor launch site at
Vandenberg Air Force Base; target launch facilities at Kodiak Launch
Complex, Alaska; a GMD fire control node at Fort Greely, Alaska; an
upgraded early warning radar at Beale Air Force Base, California; upgraded
communication links among test bed components; and test infrastructure to
support five additional intercept regions. The ship-based Aegis AN/SPY-1
radar is also available as a forward-deployed asset for early target
tracking. In addition, the design and construction of a sea-based X-band
radar, which would be positioned on a mobile platform in the Pacific, has
been funded by MDA and is scheduled to be available for test bed
utilization in late 2005. Other components of the BMDS Test Bed such as
the Cobra Dane radar in Shemya, Alaska, and interceptors at Fort Greely
will not actively participate in integrated flight tests at least through
September 2007.
Several August 2000 DOT&E recommendations call for integrated flight
testing with Category B engagements6 and scenarios with multiple
threatening reentry vehicles, both of which are expected to be common
5 Kwajalein Missile Range is now referred to as Reagan Test Site,
terminology we use from this point on.
6 The GMD system is designed to launch interceptors under one of three
"categories" of operation: (A) when a threat reentry vehicle has been
tracked and discriminated by groundbased radars; (B) when ground-based
radars have a track of the threat complex but discrimination is either
incomplete or unavailable; or (C) when space-based sensors provide an
early track of the boosting missile.
during operational missions. In addition, the recommendations call for
integrated flight testing to be performed under increasingly difficult
conditions and to be made more challenging through, for example, testing
under various solar and weather conditions. Our analysis of the GMD test
program as it pertains to flight test complexity, based on the March 2003
Developmental Master Test Plan for the GMD element and related program
documentation, is summarized below.
o Flight Test Complexity-Actions Taken or Planned. The GMD test plan
calls for Category B engagements beginning with Integrated Flight Test 15
(IFT-15), scheduled for the fourth quarter of fiscal year 2004.
Furthermore, it indicates that Category B engagements would be a common
occurrence of flight testing, because the weapon task plan7 would be
generated from Beale or Aegis radar data. According to MDA officials,
however, the decision to conduct future flight tests under Category B
engagements is currently under review; the resolution will depend on the
individual flight test scenario and the maturity of battle management
assets. The GMD Developmental Master Test Plan also shows that an
integrated flight test (designated IFT-22/23) in which two interceptors
are launched against two attacking reentry vehicles (multiple simultaneous
engagements) will be carried out in fiscal year 2007.
o Flight Test Complexity-Actions Not Taken or Planned. Although
previous flight tests have been conducted under limited adverse conditions
(light rain), flight tests to assess the actual effects of severe weather
on system performance are not currently planned. According to the program
office, the verification of system performance in adverse weather will be
achieved through modeling and simulation grounded in technical
measurements and flight test data. Furthermore, a nighttime engagement was
attempted during IFT-10 (December 2002), but the failure of the kill
vehicle to separate from the surrogate booster precluded collection of any
applicable data.
The recommendations on operational realism reflect limitations of the
current test range. Currently, intercept tests are constrained to a single
corridor and intercept region-target launches out of Vandenberg Air Force
Base and interceptor launches out of the Reagan Test Site. As a result,
flight-test engagement conditions are limited to those with low closing
velocities and short interceptor fly-out ranges. DOT&E called for
Operational Realism
7 A weapon task plan consists of pre-launch instructions for generating an
interceptor flyout solution that places it on an intercept path with the
target. Such a plan is required before an interceptor is launched.
an expansion of engagement conditions and suggested adding more intercept
regions and launch locations to achieve new intercept geometries, higher
closing velocities, and longer ranges flown by the interceptor during
flight testing.
o Operational Realism-Actions Taken or Planned. The expansion of the
test range in the Pacific with the development of the BMDS Test Bed will
have a significant impact on achieving operational realism in integrated
flight tests. The Block 2004 Test Bed adds five intercept regions, target
launches out of Kodiak Launch Complex, and interceptor launches out of
Vandenberg Air Force Base. The combination allows for flight tests with
new intercept geometries, additional crossing angles, higher closing
velocities, and longer ranges flown by the interceptor. For example,
IFT-15 (fourth quarter of fiscal year 2004) will be conducted with a
target launch out of Kodiak, and IFT-17 (fourth quarter of fiscal year
2005) will be the first test with an interceptor launched from Vandenberg.
o Operational Realism-Caveats. The principal caveat to the associated
Artificialities
MDA actions addressing operational realism is timing. By September 2004,
one of the five new intercept regions, north of Reagan Test Site, will
have been exercised. The remaining new intercept regions will not be
exercised until after September 2004. For example, the two intercept
regions off the west coast of the United States will be used in IFT-17
(fourth quarter of fiscal year 2005) and IFT-18 (fourth quarter of fiscal
year 2005), respectively. A fourth intercept point will be exercised in
IFT-21 (third quarter of fiscal year 2006). Finally, the fifth intercept
point will be exercised as part of the multiple simultaneous engagement to
be conducted in fiscal year 2007.
The DOT&E recommendations on flight test artificialities-such as the
removal of surrogates (test range assets emulating operational assets)-
also reflect limitations of the current test range. The most artificial
surrogate noted in the August 2000 DOT&E Report was the placement of a
C-band transponder8 on the target reentry vehicle. The transponder was
essential for the execution of flight tests, because in conjunction with
the test range radar (designated FPQ-149), there were no other
non-artificial options available to track the reentry vehicle with
sufficient accuracy for
8 MDA defines a transponder as a "receiver-transmitter that will generate
a reply signal under proper interrogation." The missile defense community
also refers to the transponder as the "C-band beacon."
9 FPQ-14 is a C-band test-range radar located in Oahu, Hawaii.
Elimination of Flight Tests
executing the mission. DOT&E argued that this artificiality be phased out
and, in general, recommended the system utilized in integrated flight
tests be as functional and representative as possible.
o Artificialities-Actions Taken or Planned. Use of the
transponder/FPQ-14 radar combination as a surrogate radar for midcourse
tracking is planned to be phased out. Indeed, IFT-15 (fourth quarter of
fiscal year 2004) would be the first test that does not use this surrogate
for mission execution. Rather, in integrated flight tests IFT-15 and
beyond, midcourse tracking of the target suite would be achieved through
the use of the Beale upgraded early warning radar or, pending ongoing
analysis by GMD, the Aegis SPY-1 radar. The eventual use of the sea-based
X-band radar beginning in late 2005 can also be used for midcourse
tracking. The removal of other surrogates is under way. For example, the
short-range surrogate interceptor booster, which has been used in all
flight tests to date, is scheduled to be replaced with two more
operationally representative boosters beginning with IFT-14 (third quarter
of fiscal year 2004).
o Artificialities-Actions Not Taken or Planned. The MDA is not currently
considering conducting flight tests under unrehearsed and unscripted
conditions.
Overall, the current DOT&E has looked favorably on MDA's actions that
address its recommendations, because the GMD test infrastructure is being
significantly enhanced to allow for more flight test complexity,
operational realism, and artificialities. We noted, however, that since
DOT&E's August 2000 assessment, MDA has reduced the extent of the flight
test program, as follows:
o Integrated Flight Tests-Number of Cancellations. During the initial
planning phases of the revised test program, MDA considered conducting
four intercept attempts per year. But after considerable planning and
contract evaluations, MDA limited the flight test program to no more than
three intercept attempts per year because of overlapping test objectives
and funding constraints. Significantly, the previous GMD test program at
the time of the deployment readiness review called for a total of 19
integrated flight tests to be carried out through fiscal year 2005. The
current test program, however, now has a total of 12 integrated flight
tests through fiscal year 2005-because of the cancellation of IFT-11, 12,
and 16, and the conversion of IFT-13 to booster tests (IFT-13A and 13B).
To date, 8 of the 12 have been completed under largely the same test
conditions that were critically assessed by DOT&E. In short, only two
flight tests under improved test conditions with more representative
hardware are planned
to be conducted before September 2004, the time at which the initial
defensive capability is scheduled to become available. o Operational
Testing-No Longer Required. The previous GMD test program also called for
operational testing-Initial Operational Test and Evaluation-by the
military services. Operational testing is a statutory requirement for
DOT&E to independently determine operational effectiveness and suitability
of a deployed system for use by the warfighter. MDA does not plan to
operationally test the Block 2004 GMD element before it is available for
initial defensive operations. The September 2004 fielding is not connected
with a full-rate production decision that would clearly trigger statutory
operational testing requirements. Nonetheless, the Combined Test Force, a
group of users and developers, plans tests to incorporate both
developmental and operational test requirements in the test program.
DOT&E'S Recommendations on Ground Testing
Hardware-in-the-Loop Testing
The 13 ground testing recommendations formulated by DOT&E in its August
2000 report are focused concerns encompassing four areas: (1) realistic
testing of kill vehicle functions in a Hardware-in-the-Loop (HWIL)
facility,10 (2) ground-based lethality testing, (3) development of the
system-level simulation known as the Lead System Integrator Integration
Distributed Simulation (LIDS), and (4) Operations in a Nuclear Environment
(OPINE) testing of kill vehicle components. In general, DOT&E's
recommendations on ground testing are not being addressed.
A number of the August 2000 DOT&E ground testing recommendations pertain
to the hardware-in-the-loop testing of the kill vehicle built by Raytheon.
For example, a test article is placed in an evacuated chamber to simulate
an exoatmospheric environment, and infrared radiation of a simulated
target scene is projected onto the kill vehicle's sensors. DOT&E
recommended "that an innovative new approach needs to be taken towards
hardware-in-the-loop testing of the kill vehicle, so that potential design
problems or discrimination challenges can be wrung out on the ground in
lieu of expensive flight tests." DOT&E stated that, in order to verify
kill vehicle performance, kill vehicle testing should be executed using
actual unit hardware in a hardware-in-the-loop facility capable of
providing a realistic space environment and threat scene. MDA had taken
steps to proceed with the design and construction of a
hardware-in-the-loop laboratory at the Arnold Engineering Development
Center, Tullahoma, Tennessee. Although an initial test capability had been
10 Hardware-in-the-loop testing is the high fidelity ground testing of a
test article in a realistic yet simulated environment.
Lethality Testing
LIDS
planned for the 2004 time frame, testing at the Arnold Engineering
facility has been deferred beyond Block 2004 based on Test Bed funding
constraints. In response to a draft of this report, MDA stated that future
investments and test events at this facility are subject to MDA internal
management trade-offs among the numerous priorities associated with the
whole missile defense program portfolio.
DOT&E made recommendations in its August 2000 report for improving GMD
lethality testing-testing aimed at assessing a kill vehicle's
effectiveness in destroying a reentry vehicle. Current test plans call for
an approach whereby ground-based experiments are conducted to collect data
to anchor simulations, which in turn are used to assess lethality
performance. Indeed, GMD expects to anchor such simulations from data
derived from improved "sled testing," which uses full-scale targets in the
newly developed Holloman Air Force Base Hypersonic Upgrade Program
facility. However, there are no plans to conduct intercept flight tests of
the interceptor's ability to destroy threat representative targets that
fulfill the Live Fire Test and Evaluation requirements. Rather, hit point
information is collected from actual intercept tests, which, in turn, is
used as input to simulations to determine whether the impact was lethal.
Another area of ground testing recommendations identified in the August
2000 DOT&E report concerned the development and use of system-level
digital simulations. During the time of the deployment readiness review,
the prime contractor's principal tool for assessing system performance
over a broad range of scenarios was the end-to-end digital simulation
known as LIDS. Because the development of the simulation was behind
schedule and unavailable to support analyses of overall system
performance, DOT&E reported that results obtained from it should not be
used in making a deployment decision. DOT&E recommended that LIDS
capability be "evolved to a fully validated, high-fidelity simulation." In
addition, DOT&E recommended that LIDS be made flexible enough to permit
independent use by test agencies. MDA disagrees with the recommendations
pertaining to LIDS. MDA views LIDS as one of many tools to analyze
performance aspects of the GMD element and does not believe that LIDS
needs to be developed to the level expected by DOT&E. According to the
agency, a baseline of models and simulations are available that are
intended to collectively support the entire range of analysis required to
verify the capabilities of the GMD elements. Furthermore, MDA asserts the
evolution of LIDS from Software Build 4 to its current Software Build
6.1.0 has improved the flexibility of the system to allow for sensitivity
analyses by government users. According to MDA,
extensive analysis using LIDS has been conducted at the Joint National
Integration Center at Shriever Air Force Base, Colorado.
OPINE Testing Finally, the remaining ground testing recommendations
identified in the August 2000 DOT&E report focus on OPINE testing, which
refers to the operation of individual GMD components in environments
induced by nuclear explosions. Details can be found in the classified
version of this report.
DOT&E Recommendations Target discrimination is a critical function of a
missile defense engagement
on Target Discrimination that requires the successful execution of a
sequence of functions, including target detection, target tracking,
estimations of physical characteristics of tracked objects, and data
fusion. DOT&E had two overarching concerns with the operational testing of
the discrimination function:
o Capability against diverse threats. Fundamentally, successful target
discrimination requires that the defense be able to anticipate many
characteristics of the threat. DOT&E, therefore, was concerned that
discrimination algorithms may not be sufficiently robust to handle
unanticipated threat scenes.
o The quality and quantity of information known prior to testing. DOT&E
was concerned that every physical property of target objects is known with
unrealistic accuracy in advance of flight tests.
Twelve of 50 recommendations in the August 2000 DOT&E report pertain to
the testing of the discrimination function. Specifically, DOT&E
recommended adding challenging yet unsophisticated countermeasures to the
target suites of integrated flight tests. DOT&E also recommended
integrating countermeasures developed by the Countermeasures Hands-On
Program11 (CHOP) into target suites of integrated flight tests. Finally,
DOT&E recommended executing flight test events-either intercept attempts
or risk reduction flights-that have a "pop quiz" component with respect to
radar discrimination. Operationally, this type of flight test is
11 The CHOP program, based at the U.S. Air Force Phillips Laboratory at
Kirtland Air Force Base in New Mexico, is an MDA-funded program chartered
to develop, build, and test countermeasures using only technology
available to emerging missile states. The program involves young
scientists, engineers, and military officers who are not specifically
trained in missile defense or countermeasures and are given access only to
the open literature and commercial off-the-shelf technology.
more representative of a true tactical engagement, because the exact
composition and type of countermeasures flown in an actual engagement are
generally unknown. Details can be found in the classified version of this
report.
Relative to the previous test program, MDA has substantially increased the
scope of work being done in discrimination. MDA is pursuing a block
approach that incrementally builds to a system-level discrimination
architecture that incorporates a network of sensors. The idea is to
observe the target suite throughout its trajectory using an array of
ground- and space-based sensors and to combine individual observations to
formulate a "discrimination solution." MDA is also investing resources to
study the discrimination problem and, for example, is moving forward with
flight test events focused on radar discrimination and large analysis
programs.
MDA has plans to conduct four Radar Certification Flights through fiscal
year 2006. These are non-intercept flight tests for comprehensively
characterizing the discrimination capability of the X-band radar and to
support the development of upgraded early warning radars. Furthermore,
these tests are expected to have a "pop quiz" component to examine radar
discrimination. MDA has not yet scheduled "pop quiz" testing in relation
to kill vehicle's capability to perform target discrimination.
MDA initiated and continues to fund analysis programs for investigating
promising technical concepts to improve its capabilities against enemy
countermeasures. For example, one such program, Project Hercules, is
focused on the development and testing of discrimination algorithms and
draws on academic, government, and industry expertise. Details can be
found in the classified version of this report.
Despite MDA's increased scope of work in the discrimination area, as
described above, the agency's specific actions pertaining to integrated
flight testing only partially address the August 2000 DOT&E
recommendations. No intercept flight tests of the current test plan, which
goes through IFT-26 (fiscal year 2007), are planned to address the
challenge posed by an enemy's use of unsophisticated but more challenging
countermeasures. Rather, agency officials told us that the technical
challenges posed by such countermeasures are being analyzed and may be
inserted into the flight test program at a later time.
DOT&E Recommendations on Programmatics
Characterization of Initial Defensive Capability
The remaining five recommendations from the August 2000 DOT&E report
pertain to concerns on programmatic issues, namely, adequacy of spares in
flight testing, and performance requirements. MDA has not provided for
adequate target or interceptor backups (hot spares) during flight tests.
MDA officials stated that additional target and interceptor spares can be
costly, but they are considering the issue. Even if implemented, MDA's
actions that address the recommendations on spares would not have a
significant impact on the actual conduct of flight tests but would reduce
schedule risk.
When DOT&E made its recommendations in August 2000, the GMD element was
being developed according to operational requirements. However, MDA is now
following a fundamentally new acquisition strategy-one that is
capability-based with no formal operational requirements developed by the
services. Hence, MDA has no plans to reexamine the reliability
requirements. Nonetheless, the current test program is addressing certain
performance issues raised by DOT&E. For example, the GMD program office is
tracking the prime contractor's progress in meeting target discrimination
goals.
Under the new acquisition strategy outlined by the Secretary of Defense in
his January 2002 memorandum, the ballistic missile defense program has
been refocused into a broad-based research and development effort managed
by MDA. The new program aims at developing layered defenses to intercept
missiles in all phases of flight and, if directed, to use developmental
prototypes and test assets to provide an early operational capability.
And, as stated above, system development is not subject to the formal
operational requirements developed by the Services.
On December 16, 2002, the President directed DOD to begin fielding the
first increment of the multi-element ballistic missile defense system in
2004. The Secretary of Defense stated the next day that "...it would be a
very preliminary, modest capability." The initial defensive capability for
defending the United States against long-range missiles would be based on
the GMD element of the Test Bed and augmented with more interceptors and
external sensors, as follows:
o GMD Element as part of the BMDS. The principal components of the GMD
element for defensive operations include a total of up to 10 interceptors
sited at Fort Greely (6) and Vandenberg Air Force Base (4); GMD fire
control nodes at Fort Greely and Schriever Air Force Base for battle
management and execution; an upgraded Cobra Dane radar at
Eareckson Air Station; and an upgraded early warning radar at Beale Air
Force Base.
o External Sensors. Existing sensors external to the GMD element would
also be available for defensive operations, including Defense Support
Program satellites for early warning of missile launches, and three
forward-deployed Aegis AN/SPY-1 radars on existing Navy destroyers for
early midcourse tracking.
The above assets comprise the initial configuration, which is scheduled
for fielding by the end of September 2004. The agency's near-term
intention is to grow this capability by adding 10 interceptors at Fort
Greely, a sea-based X-band radar, and an upgraded early warning radar at
Fylingdales, England,12 by the end of 2005.
Uncertainties of the Initial Defensive Capability
MDA is moving forward, as directed by the President, with the fielding of
an initial defensive capability by the end of the 2004 fiscal year to
protect the United States from long-range missiles. MDA cannot at this
time formulate a credible assessment of system-level effectiveness,
because critical components like the Cobra Dane radar and interceptor
boosters have yet to be developed and tested in a flight test environment,
and no initial defensive capability is available for a system-level
demonstration and evaluation.
o Cobra Dane Radar. The capabilities of the Cobra Dane radar will not be
demonstrated in flight testing before September 2004. It is an L-band
phased array radar located at Eareckson Air Station in Shemya, Alaska, at
the western end of the Aleutian chain. Its close proximity to Russia
allows it to perform its primary mission of collecting data on
intercontinental ballistic missile and submarine launched ballistic
missile test launches to the Kamchatka impact area. Since the Cobra Dane
radar is currently being used in a surveillance mode, it does not require
real time communications and data processing capabilities. After planned
software and hardware upgrades to be completed in fiscal year 2004, it
will have the additional mission to perform real-time acquisition and
tracking, functions critical for ballistic missile defense.
o Interceptor Boosters. In July 1998, the GMD prime contractor (Boeing)
began developing a new three-stage booster for its ground-based
interceptor from commercial off-the-shelf components. The contractor
12 The upgrading of the Thule early warning radar located in Greenland
will not be part of the Block 2004 Test Bed; it has been deferred to Block
2006.
encountered difficulty, and by the time the booster was flight tested in
August 2001, it was already about 18 months behind schedule. Subsequently,
to reduce risk, MDA altered its strategy for acquiring a new booster for
the GMD interceptor. Development of the original booster was transferred
to Lockheed Martin, and MDA authorized the GMD prime contractor to develop
a second source for the booster by awarding a subcontract to Orbital
Sciences Corporation. Both contractors are developing boosters for use in
the September 2004 initial defensive capability. The first demonstration
of an operational booster in an attempted intercept is scheduled for the
third quarter of fiscal year 2004.
o System-Level Testing. A system-level demonstration of the initial
defensive capability will not be conducted prior to September 2004. To
date, integrated flight tests have demonstrated basic functionality of a
representative ballistic missile defense system using surrogate and
prototype components, and have shown success in intercepting a mock
reentry vehicle in a developmental test environment. The first flight test
consisting of components closest to the configuration of the September
2004 initial defensive capability is IFT-14, which is currently scheduled
for the third quarter of fiscal year 2004. The test will incorporate Block
2004 prototypes of the interceptor booster and kill vehicle of the
configuration intended for operational use beginning in September 2004. In
addition, the first tactical build of the battle management software will
be utilized in IFT-14. However, interceptors will not be launched out of
Fort Greeley in IFT-14 and IFT-15 (the remaining integrated flight tests
to be conducted before September 2004).
In commenting on a draft of this report, MDA stated that while it cannot
address all technical concerns for the initial fielding, it has added the
following activities:
o Enhanced producibility, quality, and reliability efforts.
o Increased operational focus in the developmental program, e.g.,
military utility and effectiveness assessments.
o Expanded command and control, battle management, and operator
integration in BMDS testing to support fielding of initial defensive
capabilities in 2004.
MDA also stated that the results of these program decisions are intended
to provide a comprehensive program that demonstrates operational
effectiveness and military utility against credible threats in an
operational environment.
Effectiveness of September 2004 and December 2005 Initial Defensive
Capability
System effectiveness is characterized in terms of the following four
performance metrics: (1) defended area, (2) launch area denied, (3)
probability of engagement success, and (4) raid size breakpoint. Defended
area is the portion of the United States protected against long-range
missile attacks and, as a metric, is usually reported relative to a single
threat country or region; launch area denied simply refers to the
collection of threat countries from which the United States is protected.
The probability of engagement success is the probability that all
attacking warheads are destroyed, derived from the probabilities
associated with missile defense functions like detection, discrimination,
and hit-to-kill. Finally, raid size breakpoint is the maximum number of
warheads the system can realistically defeat in a single engagement. This
metric is highly dependent on interceptor inventory.
A detailed discussion of GMD's expected effectiveness is presented in the
classified June 2003 version of this report.
Other Factors Affecting System Performance
A notable limitation of the effectiveness of the September 2004 initial
defensive capability-and possibly the December 2005 capability- pertains
to the inability of system radars to perform target discrimination.
Neither the Cobra Dane radar nor the upgraded early warning radar at Beale
is capable of performing rigorous discrimination, a function achievable
only by the X-band radar. Rather, both radars will utilize common "target
classification" software that enables them to classify objects as
threatening or non-threatening. For example, debris would be classified as
non-threatening, but objects like deployment buses and decoy replicas
would be classified as threatening. Accordingly, the system would have to
rely solely on the kill vehicle for a final target selection.
The assessment of kill vehicle discrimination is, therefore, critical for
understanding the capability of the deployed system, a point made in the
DOT&E report. Appropriately, the GMD prime contractor tracks the
discrimination capability of the kill vehicle as a technical performance
measure. The prime contractor's December 2002 assessment rated the kill
vehicle discrimination performance as meeting expectations based on
analysis and simulation.
Lastly, measures of system suitability like availability and
vulnerability- which complement system effectiveness-are important for
characterizing the initial defensive capability as a whole. MDA is aiming
for full-time operations but faces risks in achieving this goal. Details
on system
Conclusions
Recommendations for Executive Action
availability and vulnerability are provided in our June 2003 classified
report.
Since DOT&E issued its August 2000 report, DOD has altered its approach to
the acquisition of missile defense systems to one that follows a
"capability-based" strategy. The new approach allows MDA to evolve and
demonstrate additional improvements in missile defense systems before
committing to procurement and operations. MDA's test program for all
missile defense elements, such as GMD, was also reoriented to focus on the
development and use of the BMDS Test Bed. Over time, the Test Bed should
facilitate testing that address many of DOT&E's recommendations,
especially those pertaining to flight test realism, complexity, and
artificialities. However, most of the agency's actions with respect to
DOT&E's ground testing recommendations, namely, those pertaining to
comprehensive hardware-in-the-loop testing of the kill vehicle have been
deferred. In addition, MDA is proceeding slowly with the flight testing
against certain countermeasures, which DOT&E noted are simple for an enemy
to implement. These unresolved concerns in the test program warrant
attention by DOT&E and the test community in general. Given the importance
of ground testing and discrimination testing for understanding system
effectiveness, decision makers in the Congress and Office of the Secretary
of Defense would benefit from having information on the agency's progress
in these matters as they consider investments in developing the ballistic
missile defense system. As an independent office that reviews DOD's weapon
system testing and the office that made the recommendations discussed in
this report, DOT&E would be in a good position to provide such information
to decision makers.
As a means of providing decision makers with critical information when
investments in missile defense are considered, we recommend that DOT&E
report periodically, as it deems appropriate, on the status of MDA's
actions taken or planned in response to the August 2000 recommendations.
In its review, DOT&E should include information and recommendations, as
warranted, on MDA's progress and planning (1) to improve
hardware-in-the-loop testing of the kill vehicle, (2) to test kill vehicle
components in nuclear environments, and (3) to test the GMD element's
capability to defeat likely and simple near-term countermeasures during
integrated flight tests. In the report, DOT&E can advise the Director,
MDA, on how the test program could be modified to accommodate DOT&E's
long-standing concerns.
Agency Comments
and Our Evaluation
In commenting on a classified draft version of this report, DOD agreed
with our recommendations. (See app. II for a reprinted version of DOD's
comments.) However, DOD conveyed the following concerns:
o The GMD test program as described in this report is no longer current.
o It is difficult to reconcile the dated terms of reference of the
original DOT&E recommendations with the current program strategy and
structure.
o The inherent robustness of the envisioned layered BMD System relative
to midcourse countermeasures is overlooked.
While the GMD test program has, indeed, been in a constant state of flux,
thus complicating our analysis, our report presents the latest, approved
test program information provided to us by MDA.
Despite alterations to the acquisition strategy and structure of the
ballistic missile defense system and its constituent elements, like GMD,
we believe most of the DOT&E recommendations are still relevant because
the technical challenges and uncertainty with developing, testing, and
fielding effective defensive capabilities, as identified in the August
2000 DOT&E report, remain significant. For example, the DOT&E report
issued in February 2003, FY02 Assessment of the Missile Defense Agency
Ballistic Missile Defense System, continued to highlight the need for a
comprehensive hardware-in-the-loop capability to test the kill vehicle
under the stress of real physical phenomena and to test the kill vehicle's
discrimination capability. We do recognize that a number of
recommendations for which no actions are currently planned, such as those
recommendations dealing with flight testing during Initial Operational
Test and Evaluation, are a direct result of MDA's new acquisition
approach.
The department is correct in stating that we did not address the
capability of the envisioned ballistic missile defense system as a whole
in defeating midcourse countermeasures. However, we do note that a
system-level discrimination architecture would use a network of ground-and
spacebased sensors to formulate a "discrimination solution." Also, given
the early stages of development of the envisioned layered system,
including boost-phase intercept, the value of this strategy has not been
demonstrated.
Although the department agreed that DOT&E should report periodically on
the status of MDA's actions to address the August 2000 DOT&E
recommendations, it did not believe additional reporting is required to
track their resolution. The department pointed out that our recommendation
grants DOT&E discretionary reporting authority where mandatory reporting
already exists. We believe, however, the recommendation is worded
appropriately. Existing statutory reporting requirements for DOT&E on the
adequacy and sufficiency of the missile defense test program do not
require that the August 2000 DOT&E recommendations be specifically
addressed. We worded the recommendation to highlight the areas we believe
DOT&E should address-hardware-in-the-loop testing of the kill vehicle,
testing of kill vehicle components in nuclear environments, and testing
the GMD element's capability to defeat likely and simple near-term
countermeasures-and to give DOT&E the discretion to address our
recommendation in the manner it deems appropriate. To present its
assessment, DOT&E could use existing or new reporting vehicles.
Finally, department comments pertaining to MDA actions on ground testing
are addressed in the body of this report.
As arranged with your staff, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days from
its issue date. At that time, we plan to provide copies of this report to
interested congressional committees; the Secretary of Defense; and the
Director, Missile Defense Agency. We will make copies available to others
upon request. In addition, the report will be available at no charge on
the
GAO Web site at http://www.gao.gov.
If you or your staffs have any questions concerning this report, please
contact me at (202) 512-4841. The major contributors to this report were
Randy Zounes, Stan Lipscomb, Tana Davis, and Bill Graveline.
R. E. Levin
Director
Acquisition and Sourcing Management
Appendix I: Scope and Methodology
In examining the actions taken or planned by the MDA in response to the
DOT&E recommendations, we analyzed pertinent test documents, studies, and
reports. These include the (1) GMD Element Developmental Master Test Plan
(March 2003); (2) GMD System Element Reviews; (3) MDA "immersion day"
briefing; (4) MDA written responses to our questions about MDA actions in
response to the DOT&E recommendations; (5) Secretary of Defense January
2002 Memorandum on Missile Defense Program Direction; and (6) Independent
Review Team (Welch panel) Reports. In addition, MDA officials briefed us
on GMD's program status and efforts to defeat enemy countermeasures. We
also reviewed available documentation on the schedule and purpose of the
Test Bed. These documents included studies on the enhanced test program
restructure, fiscal year 2003 budget justifications, and the request for
the contract proposal for the Block 2004 Test Bed.
To assess the effectiveness and limitations of the initial defensive
capability, we relied on the following MDA documentation: (1) GMD System
Element Review (January 2003); (2) BMDS Block 2004 Statement of Goals; and
(3) National Security Presidential Directive (NSPD-23), the President's
directive to begin fielding an initial capability. We also identified
uncertainties-based on the level of testing achieved to date-of the
potential capabilities of individual elements of the initial defensive
capability, such as the radars and interceptor boosters, as well as radar
capabilities to perform the discrimination function.
We conducted our work primarily at the MDA, located in Arlington,
Virginia, and the GMD Joint Program Office, located in Arlington,
Virginia, and Huntsville, Alabama.
We conducted our audit work for the June 2003 classified report, upon
which this unclassified version is based, from October 2001 to March 2003
in accordance with generally accepted government auditing standards.
However, reported dates of GMD flight test events given in this
unclassified version have been updated with the latest (December 2003) GMD
test schedules.
Appendix II: Comments from the Department of Defense
Appendix II: Comments from the Department of Defense
Appendix II: Comments from the Department of Defense
GAO's Mission
Obtaining Copies of GAO Reports and Testimony
The General Accounting Office, the audit, evaluation and investigative arm
of Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability of
the federal government for the American people. GAO examines the use of
public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO's Web site (www.gao.gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document files.
To have GAO e-mail this list to you every afternoon, go to www.gao.gov and
select "Subscribe to e-mail alerts" under the "Order GAO Products"
heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C.
20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
Contact:
To Report Fraud, Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in E-mail: [email protected]
Federal Programs Automated answering system: (800) 424-5454 or (202)
512-7470
Jeff Nelligan, Managing Director, [email protected] (202) 512-4800
Public Affairs U.S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
*** End of document. ***