    Missile Defense: Knowledge-Based Practices Are Being Adopted, but
    Risks Remain (30-APR-03, GAO-03-441).

    The Department of Defense (DOD) would like to build a capable
    missile defense system that paces an ever-evolving threat. This
    is an expensive and risky endeavor because it requires a diverse
    set of technologies to be quickly developed, integrated, and
    deployed across an array of platforms. DOD estimates that it will
    need $50 billion for missile defense research and development
    over the next 6 years and likely additional funds in subsequent
    years. GAO was asked to review the Missile Defense Agency's (MDA)
    strategy for this investment and determine what knowledge-based
    practices characteristic of successful programs are being adopted
    by MDA; what significant practices are not being adopted; and
    whether MDA is following the practices that it has adopted.
    -------------------------Indexing Terms-------------------------
    REPORTNUM:   GAO-03-441
        ACCNO:   A06761
      TITLE:     Missile Defense: Knowledge-Based Practices Are Being
    Adopted, but Risks Remain
         DATE:   04/30/2003
      SUBJECT:   Ballistic missiles
    	     Internal controls
    	     National defense operations
    	     National preparedness
    	     Program evaluation
    	     Strategic planning
    	     AN/SPY-1 Radar
    	     Cobra Dane Radar
    	     MDA Ballistic Missile Defense System

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO Product.                                                 **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ******************************************************************
    GAO-03-441

    Report to Congressional Requesters

    United States General Accounting Office

    GAO

    April 2003 MISSILE DEFENSE Knowledge- Based Practices Are Being Adopted,
    but Risks Remain

    GAO- 03- 441

    Our work has shown that programs are most successful when they evolve
    products over time rather than try to make big leaps in capability and
    when the programs adopt knowledge- based acquisition processes. Similarly,
    MDA is taking an evolutionary approach to developing the missile defense
    system by developing capabilities in spirals or *blocks* rather than
    attempting to deliver all desired capabilities at one time. The agency
    intends to facilitate this approach by keeping requirements flexible
    before beginning activities to integrate technologies into a planned
    block, following a knowledge- based development plan, and demonstrating
    that technologies work as intended before beginning system integration of
    a block. In addition, the agency is seeking to involve stakeholders* such
    as the military services and operational testers* early in the development
    effort.

    However, MDA has not adopted some knowledge- based practices regarding
    long- term investment decision making and, as a result, the missile
    defense program*s success could be hampered. First, MDA is not making an
    early determination of the full cost of a capability. Such an estimate
    would help decision makers more effectively evaluate which technologies to
    include because they offer the best capability for the funds invested.
    Second, DOD is not allocating a *wedge* of funds in its Future Years
    Defense Plan for system production and operations. Without this wedge, DOD
    may not have the funds needed to procure and maintain the missile defense
    system.

    In addition, the President*s directive to begin fielding a missile defense
    capability by 2004 places MDA in danger of getting off track early and
    impairing the effort over the long term. This danger is highlighted by
    MDA*s decision to not follow some of its knowledge- based practices as it
    develops the first block of the system. For example, MDA is beginning
    system integration of its first block with immature technology and limited
    testing. While doing so may help MDA meet the President*s deadline, it
    also increases the potential that some elements may not work as intended.

    Examples of Missile Defense Elements

    The Department of Defense (DOD) would like to build a capable missile
    defense system that paces an ever- evolving threat. This is an expensive
    and risky endeavor

    because it requires a diverse set of technologies to be quickly developed,
    integrated, and deployed across an array of platforms. DOD estimates that
    it will need $50 billion for missile defense research and development

    over the next 6 years and likely additional funds in subsequent years. GAO
    was asked to review the Missile Defense Agency*s (MDA) strategy for this
    investment and determine what knowledgebased practices characteristic of
    successful programs are being adopted by MDA; what significant practices
    are not being adopted;

    and whether MDA is following the practices that it has adopted. GAO is
    recommending that DOD prepare life cycle cost estimates for missile
    defense elements before beginning integration activities and explore the
    option of setting aside

    funds to produce and operate the missile defense system over the long
    term.

    www. gao. gov/ cgi- bin/ getrpt? GAO- 03- 441. To view the full report,
    including the scope and methodology, click on the link above. For more
    information, contact Robert Levin at (202) 512- 4841 or levinr@ gao. gov.
    Highlights of GAO- 03- 441, a report to

    Congressional Requesters

    April 2003

    MISSILE DEFENSE

    Knowledge- Based Practices Are Being Adopted, but Risks Remain

    Page i GAO- 03- 441 Missile Defense Letter 1 Results in Brief 3 Background
    4 Acquisition Strategy Adopts Many Knowledge- Based Practices 8 Two
    Knowledge- Based Practices Have Not Been Adopted 15 MDA Is Not Following
    Some Knowledge- Based Practices in

    Developing 2004 Capability 17 Conclusion 20 Recommendations for Executive
    Action 21 Agency Comments and Our Evaluation 22 Scope and Methodology 22
    Appendix I Comments from the Department of Defense 24

    Table

    Table 1: Events and Accomplishments within MDA*s Integrated Master Plan 12
    Figures

    Figure 1: Examples of Missile Defense Programs Transferred from Services
    into MDA 5 Figure 2: Notional Architecture of Future Ballistic Missile
    Defense

    System 7 Abbreviations

    BMDS Ballistic Missile Defense System DOD Department of Defense MDA
    Missile Defense Agency Contents

    This is a work of the U. S. Government and is not subject to copyright
    protection in the United States. It may be reproduced and distributed in
    its entirety without further permission from GAO. It may contain
    copyrighted graphics, images or other materials. Permission from the
    copyright holder may be necessary should you wish to reproduce copyrighted
    materials separately from GAO*s product.

    Page 1 GAO- 03- 441 Missile Defense

    April 30, 2003 The Honorable Carl Levin Ranking Member Committee on Armed
    Services United States Senate

    The Honorable Jack Reed United States Senate

    The threat from foreign ballistic missiles has grown steadily since the
    end of the Cold War. At least 25 countries now have or are in the process
    of acquiring missiles capable of delivering nuclear, biological, or
    chemical weapons. Countering this threat demands not only that the
    Department of Defense (DOD) develop cutting- edge technology, but also
    that it acquire and deploy complex systems more rapidly and effectively.

    DOD faces significant technical challenges in building a missile defense
    program. So far, DOD has had mixed results in achieving a *hit- to- kill*
    capability to destroy enemy warheads reliably, and DOD is still completing
    development of the components needed to detect and track a missile in all
    phases of its flight* from the boost phase, through the midcourse, and
    into the terminal phase. As it works to develop and prove these
    capabilities, DOD must also make sure that all missile defense elements
    can work together as an integrated system* a complex task in itself, given
    the number and diversity of elements involved in missile defense. Once

    these basic hurdles have been overcome, DOD still faces a far greater
    technical challenge in achieving target discrimination* that is, the
    capability to distinguish real warheads from decoys* to defeat more
    sophisticated threats.

    DOD also faces the challenge of delivering a weapon system with the
    capability promised and within the time and cost promised. For example,
    previous efforts to develop an airborne laser system and a space- based
    tracking and surveillance system encountered substantial delays, cost

    overruns, and other difficulties because DOD undertook these efforts

    United States General Accounting Office Washington, DC 20548

    Page 2 GAO- 03- 441 Missile Defense

    without fully understanding the effort that would be needed to mature
    technologies critical to developing the systems* required capabilities. 1
    Lastly, the development of missile defense weapon systems will also

    be costly. DOD estimates that it will need about $50 billion for missile
    defense research and development between fiscal years 2004 and 2009, or an
    average of over $8 billion per year. This investment does not include
    funding after 2009 to complete development of those systems, let alone the
    funding needed before and after 2009 to produce and operate the systems.

    In January 2002, facing these challenges, the Secretary of Defense
    directed the Missile Defense Agency (MDA) to orchestrate the integration
    of DOD*s diverse missile defense elements into one layered *system of
    systems* and to change its acquisition strategy as needed in doing so. In
    response, MDA has sought to implement practices that have proven
    successful in other DOD and commercial- sector programs. Taken together,
    these practices demand a high level of knowledge about a product at key
    junctures during development and that this knowledge be used to make
    prudent investment decisions. For example, developers should know that a
    technology has been demonstrated to work as intended before it is
    integrated into a product.

    The importance of making the best decisions possible is underscored by the
    time- critical nature of MDA*s efforts. In December 2002, the President
    directed DOD to begin fielding an initial capability in 2004 to protect
    the

    United States against missile attacks. To help ensure that MDA is using
    its resources wisely to develop the best possible system, you asked us to
    determine the extent to which the agency*s acquisition strategy
    incorporates the knowledge- based practices characteristic of successful
    development programs. To better answer your question, we separated it into
    three parts: (1) What knowledge- based practices are being adopted by MDA?
    (2) What significant practices are not being adopted? and (3) Is

    MDA following the knowledge- based practices that it has adopted? 1 U. S.
    General Accounting Office, Missile Defense: Knowledge- Based Decision
    Making Needed to Reduce Risks in Developing Airborne Laser, GAO- 02- 631
    (Washington, D. C.: July 12, 2002). U. S. General Accounting Office,
    Defense Acquisitions: Space- Based Infrared System- low at Risk of Missing
    Initial Deployment Date, GAO- 01- 6 (Washington, D. C.: Feb. 28, 2001).

    Page 3 GAO- 03- 441 Missile Defense

    MDA has adopted practices that offer the best opportunity to develop a
    complex weapon system successfully. Similar to the successful product
    development practices of leading commercial organizations, MDA is
    developing the missile defense system in *blocks,* rather than trying to
    make a big leap in capability. The agency intends to facilitate this
    evolutionary approach by keeping the system*s requirements flexible before
    beginning activities to integrate technologies into a planned block of the
    missile defense system, following a knowledge- based development plan, and
    maturing technology before beginning system integration of a block. In
    addition, the agency is seeking to involve stakeholders* such as the
    military services and the operational testers* early in the development
    effort.

    However, MDA has not adopted two significant practices regarding long-
    term investment decision making, and, as a result, the program*s success
    could be hampered. First, MDA is not making an early determination of the
    full cost of a capability. Specifically, before beginning system
    integration, MDA does not estimate the total costs for development,
    production, operations, and sustainment of that block. Such an estimate
    would help decision makers in evaluating which technologies to include
    because they offer the best capability for the funds invested. MDA
    officials told us that they are considering steps to provide such
    estimates earlier. Second, DOD is not allocating a *wedge* of funds in its
    Future Years Defense Plan (fiscal years 2004 through 2009) for system
    production and operations. MDA officials told us that such a wedge has

    not been set aside because MDA*s acquisition strategy does not presume
    that a decision will be made to produce and operate the system. DOD risks,
    however, that when it is ready to procure and maintain the missile defense
    system, it will not have the funds to do so unless it reduces or
    eliminates its investment in other important weapon systems. This approach
    brings little transparency to future investment choices and may constrain
    options for decision makers.

    The President*s directive to begin fielding an initial defensive
    capability in 2004 also places MDA in danger of getting off track early
    and introducing more risk into the missile defense effort over the long
    term. This danger is highlighted by MDA*s decision to not follow some of
    its knowledge- based practices as it develops the first block of the
    missile defense system, which will provide the initial capability. Because
    of time pressures, MDA must include components that have not been
    demonstrated as mature and ready for system integration into a particular
    element, let alone the block overall. For example, MDA has encountered
    considerable difficulty in

    developing a new three- stage booster and has yet to flight test
    interceptor Results in Brief

    Page 4 GAO- 03- 441 Missile Defense

    boosters in configurations planned for fielding in September 2004. Also,
    MDA*s test program has been limited to date and is under considerable
    schedule pressures. A knowledge- based approach to testing validates
    whether components (1) work individually, (2) work together as a system in
    a controlled setting, and (3) work together as a full system in a
    realistic setting. MDA*s integrated flight tests to date have used
    surrogate and prototype components and have been executed under non-
    stressing conditions. As a result, testing to date has provided only
    limited data for determining whether the system will work as intended in
    2004. Also, MDA has no plans to demonstrate through flight testing the
    upgraded primary radar in Alaska that will be used to detect and track
    enemy missiles.

    We are making recommendations to DOD for providing decision makers with
    more timely information on the cost and funding needs of missile defense.
    In commenting on a draft of this report, DOD concurred with our
    recommendations.

    In January 2002, the Secretary of Defense refocused the ballistic missile
    defense program. The Secretary delegated to MDA authority to manage all
    ballistic missile defense systems under development and shifted such
    programs controlled by the military services* such as the Army*s Theater
    High Altitude Area Defense program* to the agency. Other programs moved to
    MDA are highlighted in figure 1. These programs, previously recognized by
    DOD as major defense acquisition programs, are now considered *elements*
    and have been consolidated into one overall major program called the
    Ballistic Missile Defense System (BMDS). Future architectures of the BMDS
    may also include *stand- alone* components (primarily sensors) that would
    operate in concert with the various missile defense elements. Background

    Page 5 GAO- 03- 441 Missile Defense

    Figure 1: Examples of Missile Defense Programs Transferred from Services
    into MDA

    Source: DOD.

    In December 2002, President Bush directed the Department of Defense to
    begin fielding the first block of the ballistic missile defense system for
    operational use in 2004. That is, in addition to focusing resources on the
    development of a testbed for developmental testing of missile defense
    elements, he instructed MDA to build in an operational capability that
    would protect the United States against missile attacks. The fielding of
    such capabilities is referred to as an *initial defensive operations*
    capability, and, in a statement by the Secretary of Defense, ** would be a
    very preliminary, modest capability.* The initial capability will be based
    on the testbed and augmented with additional developmental assets.

    When fully deployed, the BMDS will include (1) space- and ground- based
    sensors to provide early warning and tracking of missile launches; (2)
    ground- based radars to identify and refine the tracks of threatening
    reentry vehicles and associated objects; (3) ground- and sea- based
    interceptors to destroy enemy missiles through *hit- to- kill* impacts;
    and (4) fire control nodes for battle management and execution of the
    ballistic missile defense mission. A notional architecture of future BMDS
    blocks is illustrated in figure 2. For example, the initial capability for
    defense of the United States against long- range missiles would come

    Page 6 GAO- 03- 441 Missile Defense

    from the Ground- Based Midcourse Defense element and BMDS sensors, as
    follows:

     Ground- Based Midcourse Defense Element. The principal components of the
    Ground- Based Midcourse Defense element for defensive operations include
    interceptors sited at Fort Greely, Alaska, and Vandenberg Air

    Force Base, California; a fire control node for battle management and
    execution located at Schriever Air Force Base, Colorado, with a backup
    node at Fort Greely; an upgraded Cobra Dane radar at Eareckson Air

    Station in Shemya, Alaska; and an upgraded early warning radar at Beale
    Air Force Base, California.  BMDS Sensors. Sensors external to the
    Ground- Based Midcourse

    Defense element and available for defensive operations include Defense
    Support Program satellites for missile warning and forward- deployed Aegis
    AN/ SPY- 1 radars on existing Navy cruisers.

    The above assets comprise the initial configuration, which is scheduled
    for fielding at the end of September 2004. The agency*s near- term
    intention is to expand this capability by adding more interceptors at Fort
    Greely, Alaska; a sea- based X- band radar deployed in the Pacific for use
    in flight testing; and an upgraded early warning radar at Fylingdales,
    England, by the end of 2005.

    Page 7 GAO- 03- 441 Missile Defense

    Figure 2: Notional Architecture of Future Ballistic Missile Defense System

    Page 8 GAO- 03- 441 Missile Defense

    To ensure the delivery of high- quality products on time and within
    budget, successful developers have adopted acquisition strategies that are
    anchored in knowledge. Specifically, they establish decision points for
    moving forward from technology development to product development and on
    to production. 2 At each decision point, decision makers ask themselves
    whether they have gained the knowledge they need to proceed into the next
    acquisition phase. For example, they determine whether the work can be
    completed with the money and time available and whether the product will
    be worth the required investment. If any of these questions are answered
    negatively, the program does not go forward. Other practices that
    facilitate successful outcomes include developing systems in stages rather
    than attempting to deliver all desired capabilities at one time, keeping
    requirements flexible so that the system can be produced within available
    resources, making sure technology is proven before incorporating it into a
    development program, and involving the right people at the right time in
    decision making. Commercial and DOD programs that have successfully
    implemented these practices have found that they help curb the incentive
    to rely on immature technologies and to

    over- promise the capability that can be delivered. Moreover, these
    programs found that keeping stakeholders involved in decision making
    ensured that the developed product better met the customers* needs.

    MDA realizes the value of these practices and is seeking to incorporate
    them into its acquisition strategy. Specifically, as discussed below, MDA
    plans to evolve the missile defense system over time, rather than trying
    to make a big leap in its capability. MDA is also planning to keep the
    system*s

    requirements flexible before beginning system integration and to follow a
    knowledge- based development plan. In addition, MDA is seeking to involve
    stakeholders* such as the military services and the operational testers*
    early in the development effort.

    2 The three acquisition phases are distinguished by the activities that
    occur during each of the three phases. During technology development,
    scientists apply scientific knowledge to a practical engineering problem
    and demonstrate that components with desired capabilities can be
    developed. Product development includes integrating those components into
    a stable system design and demonstrating that the design will result in a
    product that meets the customer*s needs and can be produced with the time
    and money available.

    Production is the manufacturing of the product. Acquisition Strategy

    Adopts Many Knowledge- Based Practices

    Page 9 GAO- 03- 441 Missile Defense

    Historically, many new development programs in DOD have sought to achieve
    a great leap ahead in capabilities. Because the technology was often not
    available to make such leaps, programs were often in development for years
    while engineers tried to develop and mature the needed technologies. As
    the time required to develop a system increased, so did the cost of the
    system.

    In contrast, development programs are most successful when they take an
    evolutionary, or phased, approach. In doing so, they establish time-
    phased plans to develop new products in increments. The first increment
    often has a limited capability because it incorporates technology that is
    already mature or can be matured quickly. As new technology is developed,
    it is incorporated into subsequent increments so that the product*s
    capability evolves over time. This approach reduces risks by introducing
    less new content and technology into a program*s design and development
    effort. An evolutionary strategy also enables developers to deliver a
    series of interim capabilities to the customer more quickly. Recognizing
    the benefits of evolving systems, DOD recently revised its acquisition
    system policy to encourage evolutionary development.

    The Missile Defense Agency*s new strategy for developing the ballistic
    missile defense system adopts evolutionary development. At the beginning
    of each block, the agency intends to predict the capability that can be
    developed given a *tool box* 3 of currently available technology and then
    to design that system. As additional technology matures, the agency can

    incorporate it into the next block being developed. Two key practices
    adopted by MDA can be expected to help the agency in taking an
    evolutionary approach. They include (1) keeping requirements flexible and
    (2) following a knowledge- based development plan with specific decision
    points and criteria for moving forward.

    Customers generally want new products that are high performance at low
    cost, delivered as soon as possible. But developing and producing such a
    product may exceed the developer*s technology or engineering expertise, or
    may be too costly and time- consuming for the customer to accept.
    Therefore, what a customer needs in a product and what a developer can
    produce given available resources must be matched to

    3 All possible elements/ components and interfaces that could be used in a
    ballistic missile defense system. Evolutionary Development

    Flexible requirements

    Page 10 GAO- 03- 441 Missile Defense

    form an achievable set of product requirements before development begins.
    To make this match, both the customer and the developer must be flexible
    so that potential gaps between needs and resources can be closed.
    Flexibility represents the customer*s ability and willingness to lower
    product expectations, coupled with the product developer*s willingness and
    ability to invest more resources to reduce technical risks before
    beginning system integration. Without flexibility, resources and needs can
    still be matched, but the options for closing the gaps between the two are
    limited to additional investments on the developer*s part. In fact, our
    past reviews have found that many traditional DOD acquisition programs
    incurred substantial cost increases and schedule delays because
    requirements by the military services were unrealistic and inflexible.

    Because of its flexibility, MDA*s new *capabilities- based approach* for
    the development of missile defense elements provides a greater opportunity
    to resolve this potential gap between resources and needs. 4 Instead of
    proceeding with the development of a solution based on firm operational
    requirements set by the services, the agency, in coordination with the
    warfighter, considers a number of system architectural options that can be
    developed given the mature technologies that are available. Decision
    makers evaluate the alternative architectures according to the potential
    military utility that each offers, given the constraints of cost and
    schedule.

    For example, decision makers would compare the regions of the world that
    each architecture could defend from missile launches (defended area) or
    each architecture*s probability of engagement success. This acquisition
    approach eliminates any possible gap between resources and needs; the

    customer (warfighter) accepts the best capability that the developer can
    deliver given available resources and assumes enhanced capabilities will
    be built into future blocks.

    In the early stages of a major defense acquisition program, DOD
    establishes a master schedule for moving through development and into
    production. Historically, once these milestones have been established,
    they have often been an impetus for moving forward even if requisite
    knowledge had not been attained* an action that invariably caused
    schedules to slip and costs to rise. By contrast, we have found that
    successful developers place more importance on capturing specific

    4 At the beginning of systems integration, MDA plans to establish system
    capability specifications. As in any organization, these specifications
    may change if testing shows that they are unattainable or that meeting
    them will be too costly or take too much time. Knowledge- based
    development plan

    Page 11 GAO- 03- 441 Missile Defense

    technology, design, and manufacturing knowledge than meeting milestones
    and they use this knowledge to make investment decisions. Moreover, these
    developers identify and use specific markers or criteria* such as
    technology readiness levels, percentage of engineering drawings released
    to the manufacturer, or the percentage of manufacturing processes under
    statistical control* to ensure that the program has sufficient knowledge
    to move forward.

    MDA has similarly adopted a structured plan, called the Integrated Master
    Plan, for moving forward with requisite knowledge. Every block would move
    through eight formal *events,* each of which would include an identified
    set of accomplishments that should be completed before the program moves
    on to the next event. (See table 1 for a list of events and their
    associated accomplishments.) As a block moves through the events,

    MDA plans to use quantitative criteria whenever possible to enhance
    decisions on whether to continue developing the block as it is or to make
    changes. At the end of a block*s development, MDA expects to recommend one
    of four alternative actions to decision makers. Officials could recommend
    that the elements be (1) transferred to the services to be produced and
    fielded in its current configuration, (2) further developed in a
    subsequent block, (3) retained as a test asset in the missile defense
    testbed, with some capability available for operational use, or (4)
    terminated.

    One such quantitative criterion adopted by MDA is technology readiness
    levels. Our reviews have found that successful developers often use
    technology readiness levels as an analytical tool to assess the maturity
    of technology being considered for inclusion in a product. There are nine
    levels of maturity. The level increases as the technology becomes closer
    in form, fit, and function to the actual system and is demonstrated in
    more realistic environments. For example, technology is least mature, or
    least ready for inclusion in a product, when it is an idea being explored
    in paper studies. Conversely, technology is most mature when it has been
    incorporated into the intended product and that product has been
    demonstrated in its intended environment. The lower the level of
    technology readiness, the more ground that must be covered to bring the
    technology to the point at which it can meet the intended product*s cost,
    schedule, and performance requirements with little risk. We found that
    most successful developers insert new technology into a product only when
    the technology has been incorporated into prototype hardware and that
    hardware has been demonstrated to work in the environment in which it is
    expected to be used.

    Page 12 GAO- 03- 441 Missile Defense

    MDA*s knowledge- base development plan incorporates the use of technology
    readiness levels at Event 1 to assess the maturity of technology proposed
    for a block configuration. The strategy calls for including new technology
    at system integration (Event 4) if that technology has been proven in
    prototype hardware that works in the environment in which it is expected
    to be used. While the incorporation of mature technology at system
    integration is MDA*s preferred approach, the strategy retains the

    flexibility to include less mature technology if it offers a significant
    benefit and the risk of including it is acceptable. In such instances, MDA
    expects to develop a plan for reducing the risk of moving forward with
    immature technology and to remove the technology from the block if the
    risk has not been reduced at subsequent decision points.

    Table 1: Events and Accomplishments within MDA*s Integrated Master Plan
    Event 0* Block Capability Alternatives

    Block planning process completed Long lead targets, tests, and exercises
    identified Affordability Analysis completed Preliminary block plan
    approved

    Event 1* Preliminary Configuration Definition

    Preliminary block description approved Technology readiness levels
    assessed Performance assessments updated Preliminary concept of operations
    and operational architecture drafted Risks assessed and mitigation
    programs established Detailed cost estimates for elements/ components
    available Cost/ benefit analysis updated Integration/ test objectives
    defined

    Element/ component preliminary design reviews completed Required funding
    identified Integrated master schedule completed

    Event 2* Configuration Definition

    Critical design reviews for all element/ component/ targets programs
    completed Performance/ cost assessments updated Risks assessed and
    mitigation programs updated Military utility characterized and concept of
    operations refined Preliminary integration test plan available Funding
    available and resources allocated Block definition updated Integrated
    master schedule updated Event 3* First Development Article

    First development article built and initial tests completed Targets built
    and initial tests completed

    Page 13 GAO- 03- 441 Missile Defense

    Test range and support planning completed Concept of operations defined
    and operational architecture available Funding and Estimate at Completion
    assessed

    Event 4* Integrated Test Readiness Review Block integration test planning
    completed Element/ component test and checkout completed Target test and
    checkout completed Ballistic missile defense system tactics, techniques,
    and procedures for designated user defined Funding and Estimate at
    Completion updated Operational characterization of each element completed

    Operational certification of element completed

    Event 5* Interim Test and Progress Review 50 percent of system test
    objectives accomplished Support systems defined Training systems defined
    Funding updated and Estimate at Completion verified Initial transition
    plans completed

    Initial operational characterization completed

    Event 6* Element/ component transition decision points

    System/ element/ component testing completed Operational characterization
    completed Support systems planned and budgeted Training systems planned
    and budgeted Transition plans completed and funded

    Production plans available Updated block definition available Element/
    component certification of military utility completed Service total
    obligation authority available

    Event 7* Block Certification of military utility

    Military utility assessed and system element/ component offered for
    transition Ballistic missile defense system capability demonstrated Life
    cycle cost estimate indicates long- term affordability Reliability,
    maintainability, and availability, and support requirements characterized
    Block certification of military utility completed Integration of declared
    block capability of ballistic missile defense system

    Source: DOD. Note: The events and accomplishments in MDA*s Integrated
    Master Plan are being revised. This set was in effect as of February 2003.

    Developers have found that if they are to be successful, all groups that
    have a stake in the product should be involved at all appropriate stages
    in the development effort. For example, cost analysts are needed to
    accurately estimate the cost of the product, experts in test and
    evaluation are needed to objectively assess the performance of product
    prototypes, Involvement of

    Stakeholders

    Page 14 GAO- 03- 441 Missile Defense

    and others are needed to enhance understanding of the customer*s needs. By
    involving these groups from the time a product design is created and
    keeping them involved throughout the product integration and demonstration
    phases, a program can ensure that it has a complete perspective. A key
    forum for stakeholder input is the Missile Defense Support Group,

    which includes representatives from the Joint Air and Missile Defense
    Organization; the Comptroller*s Office; the Director, Operational Test and
    Evaluation; and other units across DOD. The support group provides advice
    on such subjects as policy, operations, testing, acquisition, and
    resources to the Director of the Missile Defense Agency and the Under
    Secretary of Defense for Acquisition, Technology and Logistics, and
    supports the Senior Executive Council 5 in decision making. In addition,
    an analytical working group remains in close contact with MDA management
    so that it can collect information for the Missile Defense Support Group
    as well as conduct independent analyses of the missile defense program*s
    work.

    Initially, not all members of the Missile Defense Support Group and its
    working group believed that MDA*s approach to stakeholder involvement
    would be successful. Soon after the support group was established,

    members voiced concern that they were not getting sufficient access to
    agency information. Members said that communication with MDA was poor and
    that access to knowledgeable MDA individuals was limited, all of which
    made it difficult to provide timely advice. For example, in April 2002,
    the agency presented options to the support group to address capability
    shortfalls in sea- based terminal defense caused by the loss of the Navy
    Area missile defense program. 6 Members of the group questioned why a
    successor to the Navy Area program was needed. MDA planned additional
    briefings pertaining to this issue; however, the group was never briefed
    and MDA, without additional group input, went directly to the Under
    Secretary of Defense for Acquisition, Technology and Logistics to obtain
    approval for a particular approach to sea- based terminal missile

    5 The Senior Executive Council is led by the Deputy Secretary of Defense,
    and its members are the Under Secretary of Defense for Acquisition,
    Technology and Logistics, and the Secretaries of the Army, Navy, and Air
    Force. The Council provides oversight of MDA*s activities and is
    responsible for making program adjustments and deciding to transition or
    transfer a capability to the services. Furthermore, the Council approves
    MDA*s investment

    strategy and decisions. 6 DOD cancelled the Navy Area program in 2001 due
    to cost overruns.

    Page 15 GAO- 03- 441 Missile Defense

    defense. DOD officials told us that they did not seek further input from
    the group because there was insufficient time for it to fully understand
    and evaluate the issue before a decision had to be made.

    Shortly after this, support group members provided comments and
    suggestions to the Under Secretary and the MDA director on increasing
    communication, obtaining access to MDA personnel, and receiving timely
    information. Subsequently, MDA made progress in addressing the support

    group*s suggestions and concerns relating to these issues. As a result,
    according to support group members, the level of interaction,
    communication, and involvement has improved. In particular, support group
    members attend weekly system and element review meetings and have regular
    interaction with agency personnel outside of the Missile Defense Support
    Group forum.

    While MDA has adopted many of the practices of successful acquisition
    programs, it has not incorporated two particularly significant ones.
    Specifically, before beginning system integration, successful developers
    reduce their investment risk by estimating total costs and determining
    that funding is available for developing, producing, and operating the
    system. (We recently reported on the significance of DOD*s costs for
    operating its weapon systems and keeping them ready for action over many
    years. 7 )

    In MDA*s case, decision makers would benefit from having this knowledge
    available before MDA begins system integration because decision makers
    would be better positioned to consider whether to delay until subsequent
    blocks those elements that currently have unaffordable production,
    operation, or maintenance costs and whether costs might be lowered in the
    future by inserting new technology or implementing better engineering
    solutions. The information would also help decision makers to compare all
    elements* costs and decide which elements should be included in a planned
    block of the missile defense system because they offer the best capability
    for the funds invested.

    As of February 2003, MDA*s draft Integrated Master Plan did not call for
    an element*s life cycle cost to be estimated at the beginning of system 7
    U. S. General Accounting Office, Best Practices: Setting Requirements
    Differently Could Reduce Weapon Systems* Total Ownership Costs, GAO- 03-
    57 (Washington, D. C.: Feb. 11, 2003). Two KnowledgeBased

    Practices Have Not Been Adopted

    Page 16 GAO- 03- 441 Missile Defense

    integration (Event 4) but rather at the point when that element is
    considered for transfer to a military service for production, operation,
    and maintenance (Event 7). Moreover, MDA may never estimate the full cost
    of some elements because, according to officials, some elements may never
    be transferred to a military service for production and operation. For

    example, MDA plans to continually upgrade elements such as the Ground-
    Based Midcourse Defense element in the missile defense test bed even
    though it would be available for combat use. In such cases, MDA plans to
    estimate only the element*s development costs* not its production,
    operation, and maintenance costs.

    Also, DOD has not allocated funds in its Future Years Defense Plan (fiscal
    years 2004 through 2009) for the production, operation, and maintenance of
    any elements that might be transferred in the future to the military
    services. MDA has established optional decision points called *off ramps*
    where elements such as Theater High Altitude Area Defense could be
    transferred to the military services, but DOD has not set aside a *wedge*
    of funding for the element*s production and operating costs.

    MDA officials told us that the agency is considering revisions to its
    Integrated Master Plan so that it can provide decision makers with
    complete life cycle cost information on each element prior to beginning
    system integration activities for each block of the missile defense
    system.

    For example, MDA anticipates defining each element*s training and support
    systems before it begins system integration activities. The officials said
    that that they are still, however, in the process of determining the cost
    information that needs to be collected and the timing of its collection.
    In terms of setting aside a wedge of funding in the Future Years Defense
    Plan for production and operations costs, MDA officials told us that no
    such action is planned at this time because MDA*s acquisition strategy
    does not presume that a decision will be made to transfer the element to
    the military service. They told us that they expect funding would be made
    available.

    However, we believe that unplanned operation and maintenance costs could
    be a particular problem for DOD because its budget for these expenses is
    already stressed by the rising cost to operate and maintain many aging
    weapon systems. Also, when DOD is at the point of deciding whether to
    transfer elements of the missile defense system to a military service for
    production and operation, DOD could find that it does not have the funds
    available for missile defense without reducing or eliminating funding for
    other important weapon systems.

    Page 17 GAO- 03- 441 Missile Defense

    Because the President directed that a missile defense capability be
    fielded beginning in 2004, MDA will not be able to follow some knowledge-
    based practices in developing the initial capability in this brief time.
    As noted earlier, MDA*s draft Integrated Master Plan recommends that when
    a block enters system integration it include mature technology but the
    plan allows for the inclusion of less mature technology if the benefits
    are significant and if risks can be mitigated. Given the Presidential
    direction, MDA must include components in the block that have not been

    demonstrated as mature and ready for integration into a particular
    element, let alone the block overall. MDA*s plan also calls for rigorous
    testing before the agency recommends that the system or its elements be
    available for fielding. However, MDA*s test program has been limited to
    date and is under considerable time pressures.

    Our past reviews of DOD and commercial product development programs have
    shown that programs are in a much better position to succeed if components
    that incorporate new technologies are matured to a high level before being
    integrated into a product. Conversely, developers that initiated product
    developments with immature technology increased the risk that their
    products would fail tests and that some aspects of the products* design
    would have to be reworked because components did not perform as predicted.
    The overall impact of such problems was often that products did not
    deliver the promised capability or the developers had to

    spend additional time and money to develop that capability. While its
    draft Integrated Master Plan recommends that system integration begin with
    mature technologies (Event 4), MDA has begun including components into the
    Block 2004 configuration that are not yet mature. Two examples are the
    Cobra Dane radar and the boosters for the Ground- Based Midcourse Defense
    interceptors.

     The Cobra Dane radar is located at Eareckson Air Station in the western
    end of the Aleutian Islands chain in Alaska. Planned hardware and software
    upgrades intended to provide the radar with real- time acquisition and
    tracking capabilities are expected to be completed in fiscal year 2004.
    MDA has no plans, however, to demonstrate the expected functionality of
    the radar through integrated flight tests.

     MDA has encountered considerable difficulty in developing a three- stage
    booster for the Ground- Based Midcourse Defense element and has yet to
    flight test interceptor boosters in configurations planned for fielding in

    September 2004. By the time the new booster was flight tested in August
    2001, it was already about 18 months behind schedule. The first booster
    MDA Is Not

    Following Some Knowledge- Based Practices in Developing 2004 Capability

    System Integration of Block 2004 Begins with Some Immature Technologies

    Page 18 GAO- 03- 441 Missile Defense

    flight test was successful, but the second booster drifted off course and
    had to be destroyed 30 seconds after launch. Subsequently, the agency

    authorized two new contracts for developing boosters for use in the Block
    2004 capability. While this strategy should reduce risk in the program,
    the first demonstrations of these boosters will occur in the flight tests
    scheduled later this year. MDA officials told us that they could not
    deploy an initial capability in the

    timeframe directed by the President if they did not continue to develop
    the technology while designing the system. MDA officials told us that they
    expect to follow their knowledge- based development plan as they develop
    the next block (Block 2006) of the missile defense system.

    The fundamental purpose of testing is to gauge the progress being made
    when an idea or concept is translated into an actual product and,
    ultimately, to make sure the product works as intended. Leading commercial
    firms conduct testing to discover potential developmental problems early.
    The firms focus on validating that their products have reached increasing
    levels of product maturity at given points in time. The firms* products
    have three maturity levels in common: components work individually,
    components work together as a system in a controlled setting, and
    components work together as a full system in a realistic setting. Testing
    in this systematic manner helps ensure that problems are identified and
    corrected early, when the cost of solving problems is lower and more
    options are available. Over time, disciplined testing helps confirm that
    the product eventually produced will meet the customer*s needs.

    In the past, when DOD programs have been schedule* rather than event*
    driven, program managers have found it difficult to slow the program if
    problems were identified during testing. MDA has been placed in a similar,
    pressured position as it prepares to field an initial capability by
    September 2004. Also, only limited test data is available for determining
    whether a credible capability will be available at that date.

    The capability that MDA expects to deploy is essentially a collection of
    elements that are connected by battle management software. Initially, the
    mission of the software will be to hand off data from the radars that
    detect and track enemy missiles to the shooters that launch interceptors
    to kill the missiles. For example, the battle management software could
    communicate to the Ground- Based Midcourse element data on the position of
    an intercontinental ballistic missile being tracked by the Aegis Testing
    under Pressure

    and Limited to Date One system- level test is planned prior to fielding

    Page 19 GAO- 03- 441 Missile Defense

    Ballistic Missile Defense radar. MDA has begun the development of battle
    management hardware/ software and has completed some ground tests of its
    capability. However, MDA plans to deploy the block although the battle
    management software*s ability to interoperate with the elements as an
    integrated missile defense system will not be flight tested until the
    Spring of 2004.

    Integrated flight tests to date have demonstrated that the Ground- Based
    Midcourse Defense and Aegis Ballistic Missile Defense elements can defeat
    a mock warhead in a test environment. However, the tested elements did not
    include all of the same components that will be part of the elements
    deployed in 2004. Instead these elements were tested using some surrogate
    and prototype components. For example, all tests of the Ground- Based
    Midcourse element have included a surrogate booster and a prototype kill
    vehicle. In addition, tests of this element have not included the Cobra
    Dane radar that will be used in September 2004 to detect and track
    intercontinental ballistic missiles. The Cobra Dane radar will not
    actively participate in integrated flight tests at least through September
    2007.

    Element flight tests have also been executed under nonstressing conditions
    that are not fully representative of the environments that the elements
    would experience in combat. All flight tests completed to date have been
    limited to a single corridor and intercept region, that is all targets
    have been launched from Vandenberg Air Force Base, California, and
    interceptors have been launched from the Reagan Test Site in the Marshall
    Islands. As a result, flight- test engagement conditions are limited to
    those with slower closing velocities and shorter intercept ranges. Testing
    under conditions such as these significantly limit the data MDA can
    collect on system effectiveness and readiness.

    An operational test assesses the effectiveness of the system against the
    known threat and its suitability for combat use. U. S. law requires that
    such tests be carried out on major defense acquisition programs and
    assessed by DOD*s Director, Operational Test and Evaluation, before a
    full- rate production decision is made. The purpose of the Director*s
    assessment is to advise the Secretary of Defense and Congress on the
    effectiveness of the system against the known threat and its suitability
    for combat use.

    MDA does not plan to operationally test the Block 2004 Ground- Based
    Missile Defense element before it is available for initial defensive
    operations. The September 2004 fielding is not connected with a full- rate
    production decision that would clearly trigger statutory operational
    testing Element flight- test data

    is limited Operational test data is limited

    Page 20 GAO- 03- 441 Missile Defense

    requirements. Nonetheless, according to DOD officials, MDA plans to
    incorporate both developmental and some operational test requirements in
    integrated flight tests.

    The Director, Operational Test and Evaluation, will provide comments on an
    element*s operational effectiveness and suitability as demonstrated in
    these tests. 8 However, Operational Test and Evaluation officials said
    that because developmental tests are scripted, planned events, they do not
    provide the opportunity to assess how the equipment and its operators will
    function under unforeseen conditions.

    MDA is attempting to build a ballistic missile defense capability that
    paces an ever- evolving threat. This is an expensive and risky endeavor,
    because it requires a diverse set of technologies that must be quickly
    developed, integrated, and deployed across an array of land-, air-, sea-,
    and spacebased platforms. Whether MDA can successfully meet the challenge
    of quickly developing an effective and suitable missile defense system
    depends in large part on its willingness to adopt practices that have made

    other developers successful and to implement those practices as it
    develops each block.

    Certainly, the presidential directive has already caused MDA to not follow
    some of the knowledge- based practices that it had adopted as it develops
    Block 2004. Giving up this approach opens the door to greater cost and
    performance risks. Beginning system integration of Block 2004 with
    immature technology increases the potential that some element may not work
    as intended. If this happens, MDA will be faced either with fielding a
    less than credible system or likely spending more money in an attempt to
    develop the desired capability within the time allowed. In addition to the
    challenge it faces in Block 2004, MDA faces the challenge of getting its
    acquisition program back on track. Because the ballistic missile threat is

    8 The National Defense Authorization Act for Fiscal Year 2002 (P. L. 107-
    107) requires the Director of Operational Test and Evaluation to (1)
    annually assess, and report to Congress on, the adequacy and sufficiency
    of MDA*s test program during the preceding fiscal year, (2) monitor the
    development of MDA*s plan for ensuring that each critical technology for a
    missile defense program is successfully demonstrated in an appropriate
    environment before that technology enters into operational service (and
    provide the Director of MDA with appropriate comments), and (3) review, on
    an ongoing basis, the development of

    MDA*s annual program goals (including testing goals) and annual program
    plan (including schedules for flight tests and other significant testing
    activities) and provide any resulting comments on the plans to the
    Secretary of Defense and the Director of MDA. Conclusion

    Page 21 GAO- 03- 441 Missile Defense

    rapidly increasing, MDA could always believe it is operating in an
    emergency environment. Yet, it has never been proven that it takes longer
    to acquire a weapon system if a knowledge- based acquisition plan is
    followed. Instead, the opposite should be true, because such a plan
    decreases the likelihood that deadlines will be missed because critical
    elements do not work as intended.

    MDA and DOD also need to address the long- term implications of their
    investment strategy. Both are assuming increased investment risk by not
    having the right information available for decision makers at the right
    time. The level of anticipated spending magnifies this risk. MDA officials
    told us they are considering changes to MDA*s Integrated Master Plan to
    identify life cycle costs at the beginning of system integration
    activities so that tradeoff decisions can be made in a more timely manner.
    However, because DOD has not yet set aside funds to cover its long- term
    costs, the department could find that it cannot afford to procure and
    maintain that system unless it reduces or eliminates its investment in
    other important weapon systems. By setting aside funds in the Future Years
    Defense Plan, we believe DOD would bring needed visibility to the
    impending trade- offs

    between missile defense and other weapon system spending for procurement
    and operations.

    To assist MDA and DOD decision makers in determining which elements or
    components should be included in each new block of the Ballistic Missile
    Defense System, we recommend that the Missile Defense Agency, before
    beginning integration activities, prepare a life cycle cost estimate

    for configuring the element or component that the agency is considering
    including in the block.

    To help ensure that funds are available to produce and operate the
    elements of the missile defense system when a decision is made to transfer
    elements to the military services, we recommend that the Secretary of
    Defense explore the option of requiring the services to set aside funds
    for this purpose in the Future Years Defense Plan. Recommendations for

    Executive Action

    Page 22 GAO- 03- 441 Missile Defense

    In commenting on a draft of this report, the DOD concurred with our
    recommendations.

    Regarding our recommendation that MDA prepare life cycle cost estimates
    before beginning integration activities, DOD said that MDA will prepare
    its best estimate of life cycle costs based upon projected hardware life,
    operational cost drivers, and initial capability quantities prior to
    integration activities for each block.

    Regarding our recommendation that DOD set aside funds in its Future Years
    Defense Plan in anticipation of the transfer of missile defense system
    elements to the military services, DOD said that there is benefit in
    budgeting funds when such a transfer is anticipated. Doing so would
    promote budget stability and improve the likelihood that an element or
    component would actually be fielded.

    DOD also suggested technical changes, which we incorporated as
    appropriate. DOD*s comments are reprinted in appendix I.

    To address our objectives, we analyzed documents that detailed the Missile
    Defense Agency*s new acquisition practices and compared the practices to
    those of successful development programs. We also obtained detailed
    briefings from Missile Defense Agency officials regarding the agency*s
    plan for the implementation of these practices and contrasted that plan to
    the implementation plan of successful programs. In addition, we discussed
    the challenges and risks that the agency faces as it implements its new
    plan with the Institute of Defense Analyses, Alexandria, Virginia. We also
    discussed these issues with all members of the Missile Defense Support
    Group, including the Office of the Undersecretary for Acquisition,
    Technology and Logistics; Office of the Undersecretary for Policy; Office
    of the Undersecretary (Comptroller); General Counsel; Office of the
    Assistant Secretary (Command, Control, Communications, and Intelligence);
    Office of the Director, Operational Test and Evaluation; Office of the
    Director, Program Analysis and Evaluation; Office of the Director, Cost
    Analysis Improvement Group, in Washington, D. C.; and the Joint Staff;
    Department of the Army; Department of the Air Force; Department of the
    Navy; and the Missile Defense Agency in Arlington, Virginia. Agency
    Comments

    and Our Evaluation Scope and Methodology

    Page 23 GAO- 03- 441 Missile Defense

    We conducted our review from March 2002 to March 2003 in accordance with
    generally accepted government auditing standards.

    As arranged with your staff, unless you publicly announce its contents
    earlier, we plan no further distribution of this report until 30 days from
    its issue date. At that time, we plan to provide copies of this report to
    the Chairmen and Ranking Minority Members of the Senate Committee on Armed
    Services; the Senate Committee on Appropriations, Subcommittee on Defense;
    the House Committee on Armed Services; the House Committee on
    Appropriations, Subcommittee on Defense; the Secretary of Defense; and the
    Director, Missile Defense Agency. We will make copies available to others
    upon request. In addition, the report will be available at

    no charge on the GAO Web site at http:// www. gao. gov/. If you or your
    staff have any questions concerning this report, please contact me on
    (202) 512- 4841. Major contributors to this report were Katherine
    Schinasi, Barbara Haynes, Cristina Chaplain, David Hand, Alan Frazier,
    Randy Zounes, Adam Vodraska, Jose Ramos, and Greg Lagana. R. E. Levin

    Director Acquisition and Sourcing Management

    Appendix I: Comments from the Department of Defense

    Page 24 GAO- 03- 441 Missile Defense

    Appendix I: Comments from the Department of Defense

    Appendix I: Comments from the Department of Defense

    Page 25 GAO- 03- 441 Missile Defense (120135)

    The General Accounting Office, the audit, evaluation and investigative arm
    of Congress, exists to support Congress in meeting its constitutional
    responsibilities and to help improve the performance and accountability of
    the federal government for the American people. GAO examines the use of
    public funds; evaluates federal programs and policies; and provides
    analyses, recommendations, and other assistance to help Congress make
    informed oversight, policy, and funding decisions. GAO*s commitment to
    good government is reflected in its core values of accountability,
    integrity, and reliability.

    The fastest and easiest way to obtain copies of GAO documents at no cost
    is through the Internet. GAO*s Web site (www. gao. gov) contains abstracts
    and fulltext files of current reports and testimony and an expanding
    archive of older products. The Web site features a search engine to help
    you locate documents using key words and phrases. You can print these
    documents in their entirety, including charts and other graphics.

    Each day, GAO issues a list of newly released reports, testimony, and
    correspondence. GAO posts this list, known as *Today*s Reports,* on its
    Web site daily. The list contains links to the full- text document files.
    To have GAO e- mail

    this list to you every afternoon, go to www. gao. gov and select
    *Subscribe to daily E- mail alert for newly released products* under the
    GAO Reports heading.

    The first copy of each printed report is free. Additional copies are $2
    each. A check or money order should be made out to the Superintendent of
    Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
    copies mailed to a single address are discounted 25 percent. Orders should
    be sent to: U. S. General Accounting Office 441 G Street NW, Room LM
    Washington, D. C. 20548 To order by Phone: Voice: (202) 512- 6000

    TDD: (202) 512- 2537 Fax: (202) 512- 6061

    Contact: Web site: www. gao. gov/ fraudnet/ fraudnet. htm E- mail:
    fraudnet@ gao. gov Automated answering system: (800) 424- 5454 or (202)
    512- 7470 Jeff Nelligan, managing director, NelliganJ@ gao. gov (202) 512-
    4800

    U. S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.
    C. 20548 GAO*s Mission Obtaining Copies of

    GAO Reports and Testimony

    Order by Mail or Phone To Report Fraud, Waste, and Abuse in Federal
    Programs Public Affairs
    *** End of document. ***