    Defense Acquisitions: Major Weapon Systems Continue to Experience
    Cost and Schedule Problems under DOD's Revised Policy (14-APR-06,
    GAO-06-368).

    The Department of Defense (DOD) is planning to invest $1.3
    trillion between 2005 and 2009 in researching, developing, and
    procuring major weapon systems. How DOD manages this investment
    has been a matter of congressional concern for years. Numerous
    programs have been marked by cost overruns, schedule delays, and
    reduced performance. Over the past 3 decades, DOD's acquisition
    environment has undergone many changes aimed at curbing cost,
    schedule, and other problems. In order to determine if the policy
    DOD put in place is achieving its intended goals, we assessed the
    outcomes of major weapons development programs initiated under
    the revised policy. Additionally, we assessed whether the
    policy's knowledge-based, evolutionary principles are being
    effectively implemented, and whether effective controls and
    specific criteria are in place and being used to make sound
    investment decisions.
    -------------------------Indexing Terms-------------------------
    REPORTNUM:   GAO-06-368
        ACCNO:   A51657
      TITLE:     Defense Acquisitions: Major Weapon Systems Continue to
    Experience Cost and Schedule Problems under DOD's Revised Policy
         DATE:   04/14/2006
      SUBJECT:   Cost overruns
    	     Defense cost control
    	     Defense procurement
    	     Internal controls
    	     Military policies
    	     Performance measures
    	     Policy evaluation
    	     Procurement planning
    	     Procurement policy
    	     Schedule slippages
    	     Weapons research and development
    	     Weapons systems
    	     Investment management
    	     Army Future Combat Systems
    	     Joint Strike Fighter program

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO Product.                                                 **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ******************************************************************
    GAO-06-368

         * Knowledge-Based Process Not Enforced
         * Evolutionary Acquisition Is Not Being Used
         * Order by Mail or Phone

                     United States Government Accountability Office

    Report to Congressional Committees

    GAO

    April 2006

    DEFENSE ACQUISITIONS

      Major Weapon Systems Continue to Experience Cost and Schedule Problems under
                                  DOD's Revised Policy

    GAO-06-368

    DEFENSE ACQUISITIONS

    Major Weapon Systems Continue to Experience Cost and Schedule Problems
    under DOD's Revised Policy

      What GAO Found

    Changes made in DOD's acquisition policy over the past 5 years have not
    eliminated cost and schedule problems for major weapons development
    programs. Of the 23 major programs we assessed, 10 are already expecting
    development cost overruns greater than 30 percent or have delayed the
    delivery of initial operational capability to the warfighter by at least 1
    year. The overall impact of these costly conditions is a reduction in the
    value of DOD's defense dollars and a lower return on investment. The
    following table illustrates the problem.

     Cost and Schedule Outcomes Sorted by Percent of Product Development Remaining

    Percent cost Programs growtha

    Percent of Schedule growth, development in months remaining

                            Aerial Common Sensor 45% 24 85%

                            Future Combat System 48% 48 78%

                            Joint Strike Fighter 30% 23 60%

                       Expeditionary Fighting Vehicle 61% 48 49%

    C-130 Avionics Modernization Program 122% Delays anticipated Undetermined

                Global Hawk (RQ-4B) 166% Delays anticipated Undetermined

    Sources: DOD (data); GAO (analysis and presentation).

    a

    Cost growth is expressed as the percent change in program development cost
    estimates in 2005 base year dollars.

    Poor execution of the revised acquisition policy is a major cause of DOD's
    continued problems. DOD frequently bypasses key steps of the
    knowledgebased process outlined in the policy, falls short of attaining
    key knowledge, and continues to pursue revolutionary-rather than
    evolutionary or incremental-advances in capability. Nearly 80 percent of
    the programs GAO reviewed did not fully follow the knowledge-based process
    to develop a sound business case before committing to system development.
    Most of the programs we reviewed started system development with immature
    technologies, and half of the programs that have held design reviews did
    so before achieving a high level of design maturity. These practices
    increase the likelihood that problems will be discovered late in
    development when they are more costly to address. Furthermore, DOD's
    continued pursuit of revolutionary leaps in capability also runs counter
    to the policy's guidance.

    DOD has not closed all of the gaps in the policy that GAO identified
    nearly 3 years ago, particularly with regard to adding controls and
    criteria. Effective controls require decision makers to measure progress
    against specific criteria and ensure that managers capture key knowledge
    before moving to the next acquisition phase. However, DOD's policy
    continues to allow managers to approach major investment decisions with
    many unknowns. Without effective controls that require program officials
    to satisfy specific criteria, it is difficult to hold decision makers or
    program managers accountable to cost and schedule targets. In this
    environment, decisionmaking transparency is crucial, but DOD is lacking in
    this area as well.

                     United States Government Accountability Office

                                        Contents

      Abbreviations

    DOD Department of Defense GAO Government Accountability Office

    This is a work of the U.S. government and is not subject to copyright
    protection in the United States. It may be reproduced and distributed in
    its entirety without further permission from GAO. However, because this
    work may contain copyrighted images or other material, permission from the
    copyright holder may be necessary if you wish to reproduce this material
    separately.

    United States Government Accountability Office Washington, DC 20548

    April 13, 2006

    The Honorable John Ensign Chairman The Honorable Daniel K. Akaka Ranking
    Minority Member Subcommittee on Readiness and Management Support Committee
    on Armed Services United States Senate

    The Honorable Duncan L. Hunter Chairman Committee on Armed Services House
    of Representatives

    DOD's planned investment in research, development, and procurement of
    major weapon systems will total approximately $1.3 trillion between 2005
    and 2009, with over $800 billion of that investment yet to be made. DOD is
    facing a significant number of problems in managing its acquisitions.
    Military operations in Afghanistan and Iraq are consuming a large share of
    DOD resources and causing the department to invest more money sooner than
    expected to replace or fix existing weapons. Meanwhile, DOD is intent on
    transforming military operations while pursuing multiple megasystems that
    are expected to be the most expensive and complex ever. These costly
    conditions coupled with increases in spending for other national
    priorities, such as health care and social security, make it essential
    that DOD effectively leverage its investments, particularly in weapon
    system acquisitions. If DOD manages its current portfolio of weapons
    within traditional margins of error, the financial consequences could be
    dire.

    DOD's strategy for acquiring major weapon systems has traditionally been
    to plan programs that would achieve a big leap forward in capability
    within a single development program, a strategy that often results in
    major cost and schedule problems. We have assessed weapon acquisitions as
    a high-risk area for 15 years, and although U.S. weapons are among the
    best in the world, the programs to acquire them have continued to produce
    poor cost and schedule outcomes. However, the current defense acquisition
    environment continues to be characterized by cost and schedule growth, a
    lack of confidence by congressional and DOD leaders, and no appreciable
    improvement in the defense acquisition system. DOD

                                    Results in Brief

    knows what to do to achieve better outcomes. It has written into policy an
    approach that advocates that adequate knowledge be attained at critical
    junctures before DOD managers agree to invest more money in the next phase
    of weapon system development. The policy also emphasizes evolutionary
    principles for acquiring weapons rather than trying to achieve a big leap
    forward in capability within a single development program. We have
    reported in the past that DOD's revised policy does not incorporate
    adequate controls to ensure the effective implementation of a
    knowledgebased, evolutionary acquisition process. However, DOD believes
    that the policy includes the necessary controls to achieve effective
    outcomes.

    You requested that we evaluate DOD's compliance with and implementation of
    its revised acquisition policy intended to produce better cost, schedule,
    and performance outcomes for major acquisition programs. In order to
    obtain an early assessment of the cost and schedule impact of the revised
    policy, and to assess DOD's effectiveness in implementing a
    knowledge-based, evolutionary acquisition approach we assessed (1) the
    cost and schedule status of major weapons development programs initiated
    under the revised policy, (2) whether the policy's knowledge-based,
    evolutionary acquisition principles are being effectively implemented, and
    (3) whether effective controls and specific criteria are in place and
    being used to make sound investment decisions.

    In conducting our evaluation, we reviewed pertinent acquisition statutes,
    policies, and guidance; analyzed development cost and schedule data for 23
    major acquisition programs approved to start system development under
    DOD's revised acquisition policy between October 2000 and December 2004;
    conducted case study reviews of nine of those 23 programs; and interviewed
    officials from the Office of Secretary of Defense and each of the military
    services. We conducted our review from May 2005 to February 2006 in
    accordance with generally accepted government auditing standards.
    Additional information about our methodology is contained in appendix I.

    DOD's revised acquisition policy has not led to improved acquisition
    program outcomes. Programs initiated under the revised policy are already
    beginning to experience cost and schedule problems similar to programs
    managed under prior versions of the policy. Although the programs we
    reviewed have been in development for only a short period of time, nearly
    half are already estimating development cost growth greater than 30
    percent or are expecting to delay initial delivery to the warfighter by at
    least 1 year. Program officials are facing the familiar predicament of
    having to add unplanned money or time or to reduce system capabilities and
    quantities after initial business cases have been approved and system
    development has begun. As a result, DOD is reducing its buying power and
    is not achieving the return on investment it expected when the programs
    began.

    Acquisition officials are not effectively implementing the revised
    acquisition policy's knowledge-based process. They regularly bypass key
    phases of the early acquisition process, approach key decision points with
    limited knowledge about critical technologies and system design, and do
    not employ evolutionary acquisition principles. Nearly 80 percent of the
    programs we assessed were permitted to bypass the policy's initial major
    decision review and the associated systems-engineering process that are
    intended to ensure that a system's requirements match available resources
    and that a sound business case is developed prior to starting system
    development. By not consistently following key processes and strategies,
    acquisition officials are not ensuring that a solid foundation of
    knowledge about cost, schedule, and performance is established before
    allowing programs to start system development, thus resulting in
    unexecutable business cases. Although the policy explicitly states that
    programs shall increase program knowledge by maturing technologies before
    beginning system development, we found that almost three-fourths of the
    programs started since the policy was revised began development with
    immature critical technologies. Our analysis also indicates that decision
    makers are continuing to commit programs to system demonstration and
    initial manufacturing before officials have demonstrated high levels of
    design knowledge, as emphasized in the policy. In addition, programs like
    the Joint Strike Fighter and Future Combat System are still structured to
    achieve major leaps in capability within a single development program, a
    strategy that has historically proven to be problematic in terms of cost
    and schedule outcomes.

    Effective implementation of the revised policy is limited by the absence
    of effective controls that require compliance and specific criteria for
    clearly demonstrating that acceptable levels of knowledge about
    technology, design, and manufacturing have been attained at critical
    junctures during system development before making further investments in a
    program. Without effective controls, the policy cannot prevent DOD
    decision makers from starting system development even when they face
    significant unknowns about technology, design, and production. Without
    specific criteria-or standards against which a judgment or decision is
    quantifiably based-decision makers are permitted to make decisions on the
    basis of subjective judgment. We reported this condition in 2003, yet DOD
    has not

                                       Background

    closed gaps in the policy. In the absence of such controls and criteria,
    DOD faces the added problems of transparency and accountability because it
    often does not sufficiently document the rationale for its decisions to
    allow acquisition programs to advance with low levels of technology,
    design, and manufacturing knowledge.

    This report contains recommendations that DOD require programs to meet
    specific knowledge-based criteria at each key decision points in the
    acquisition process and require decision makers to provide clear and
    specific rationale for their decisions. In addition, we recommend that
    before programs enter system development they should be required to
    complete disciplined concept and technology development phases that
    include specific activities dedicated to capturing knowledge critical to
    developing an executable business. DOD partially concurred with our
    recommendations. DOD agrees that knowledge-based decision making is
    consistent with sound business practice and stated that it would continue
    to develop policy that reflects a knowledge-based approach and improves
    acquisition outcomes. DOD also agrees that acquisition decisions should be
    documented, decision makers should be held accountable, and that they
    should provide the rationale for their decisions.

    Historically, DOD's programs for acquiring major weapon systems have taken
    longer, cost more, and often delivered fewer quantities and other
    capabilities than planned. GAO has documented these problems for decades.
    In 1970, GAO reported that considerable cost growth had been and was
    continuing to occur on many current development programs. Since that
    report was issued, numerous changes have been made to DOD's acquisition
    process and environment to try to improve acquisition outcomes. Those
    changes include numerous executive branch initiatives and legislative
    actions as well as roughly 11 revisions to DOD's acquisition policy
    between 1971 and 2005. Despite these efforts, defense acquisition programs
    in the past 3 decades continued to routinely experience cost overruns,
    schedule slips, and performance shortfalls.

    Figure 1 illustrates the continued problem of development cost overruns.
    The figure depicts the combined cost overruns for large development
    programs (programs totaling more than $1 billion for research,
    development, testing and evaluation in fiscal year 2005 dollars) in each
    of the past 3 decades. The figure also identifies some of the major
    studies and improvement efforts initiated during this time frame. As the
    figure illustrates, efforts to improve acquisition outcomes have not been
    successful in curbing acquisition cost problems. Programs initiated in the
    1970s exceeded DOD's initial investment estimate by 30 percent, or $13
    billion (in fiscal year 2005 dollars), and similar outcomes continued
    during the subsequent decades despite numerous reform efforts and policy
    revisions.

    Figure 1: Development Cost Overruns by Decade (in Fiscal Year 2005 Dollars) and
                                   Key Reform Efforts

    Source: DOD (data); GAO (analysisand presentation).

    Since the mid-1990s, we have studied the best practices of leading
    commercial companies. Taking into account the differences between
    commercial product development and weapons acquisitions, we articulated a
    best practices product development model that relies on increasing
    knowledge when developing new products, separating technology development
    from product development, and following an evolutionary or incremental
    product development approach. This knowledge-based approach requires
    developers to make investment decisions on the basis of specific,
    measurable levels of knowledge at critical junctures before investing more
    money and before advancing to the next phase of acquisition. An
    evolutionary product development process defines the individual increments
    on the basis of mature technologies and a feasible design that are matched
    with firm requirements. Each increment should be managed as a separate and
    distinct acquisition effort with its own cost, schedule and performance
    baseline. An increment that excludes one of these key elements puts an
    extra burden on decision makers and provides a weak foundation for making
    development cost and schedule estimates. The knowledge-based, evolutionary
    approach in our model is intended to help reduce development risks and to
    achieve better program outcomes on a more consistent basis.

    Hoping to improve acquisition outcomes, DOD leaders initiated significant
    revisions to the department's acquisition policy again in October 2000, by
    adopting the knowledge-based, evolutionary system development approach. 1
    We reported in November 2003, that much of the revised policy agrees with
    GAO's extensive body of work and that of successful commercial firms.
    DOD's revised policy emphasizes the importance of and provides a good
    framework for capturing knowledge about critical technologies, product
    design, and manufacturing processes. If properly implemented and enforced
    this approach could help DOD's decision makers gain the confidence they
    need to make significant and sound investment decisions for major weapon
    systems. Furthermore, the policy's emphasis on evolutionary system
    development sets up a more manageable environment for achieving knowledge.
    We also noted that DOD's policy strongly suggests the separation of
    technology development from system development, a best practice that helps
    reduce technological risk at the start of a program and makes cost and
    delivery estimates much more predictable. 2 Figure 2 depicts in general
    how DOD's revised policy adopts key aspects of the best practices model.

    In addition to the acquisition policy, the process used by DOD to
    establish program funding, known as the Planning, Programming, Budgeting
    and Execution System (PPBES), and the process used to determine system
    requirements, now called the Joint Capabilities Integration and
    Development System (JCIDS), also impact program outcomes. Both processes
    are currently being studied to determine if any changes could be made to
    improve program performance as it relates to funding and requirements.

    2GAO. Defense Acquisitions: DOD's Revised Policy Emphasizes Best
    Practices, but More Controls Are Needed, GAO-04-53 ( Washington, D.C.:
    Nov. 10, 2003).

    Figure 2: Comparison of DOD's Revised Policy and Commercial Best Practices Model

    Source: DOD (data); GAO (analysisand presentation).

    Although DOD took significant steps in the right direction, its policy
    does not include controls that require program officials to meet the key
    criteria that we believe are necessary for ensuring that acceptable levels
    of knowledge are actually captured before making additional significant
    investments. We previously recommended that DOD design and implement
    necessary controls to ensure that appropriate knowledge is captured and
    used to make decisions about moving a program forward and investing more
    money at critical junctures. DOD officials acknowledged the advantages of
    using knowledge-based controls, but stated that they believed the policy
    already included enough controls to achieve effective program results. The
    officials agreed to monitor the acquisition process to assess the
    effectiveness of those controls and to determine whether additional ones
    are necessary.

      DOD's Revised Policy Has Not Improved Development Program Outcomes

    The cost and schedule outcomes being achieved by development programs
    initiated since DOD first issued its revised policy have not improved over
    those achieved by programs managed under prior versions of the policy. Of
    the 23 major programs we assessed, 10 have already reported estimated
    development cost growth greater than 30 percent or expected delays of at
    least 1 year in delivery of an initial operational capability to the
    warfighter. These programs combined represent a cost increase of $23
    billion (in fiscal year 2005 dollars) and an average delay in delivery of
    initial capability of around 2 years. Most of the other programs were
    still in the early stages as of December 2005 with over half of system
    development remaining and had not yet reported an adequate amount of cost
    or schedule data to effectively analyze their progress. Table 1 contains
    the cost and schedule increases for the 23 programs we assessed, expressed
    as a percentage of each program's development estimate.

     Table 1: Cost andSchedule Outcomes for 23 Programs Initiated under the Revised
                              Policy (as of December 2005)

    Percent growth in estimated Percent growth in estimated Program
    development costa development schedule

                         Expeditionary Fighting Vehicle 61% 70%

    Active Electronically Scanned Array radar (upgrade for F/A-18 E/F
    fighter/attack aircraft) 14% 1%

                 Global Hawk unmanned aerial vehicle 166% Undetermined

                              Joint Strike Fighter 30% 23%

                           UH-60M helicopter upgrade 151% 25%

                 C-130 Avionics Modernization Program 122% Undetermined

    C-5 Reliability Enhancement and Re-engining Program 0% 25%

                     Joint Tactical Radio System Cluster 1 31% 44%

                 Joint Tactical Radio System Waveform 44% Undetermined

                      Advanced Anti-radiation Guided Missile 7% 0%

           Multi-Platform Radar Technology Insertion Program 0% Undetermined

                              Future Combat System 48% 53%

                               E-2 Advanced Hawkeye 5% 0%

                     Warfighter Information Network-Tactical 0% 0%

                               Small Diameter Bomb 0% 0%

                                      EA-18G 7% 0%

                      Joint Tactical Radio System Cluster 5 0% 2%

                         Multi-Mission Maritime Aircraft 0% 0%

    Standard Missile-6 Extended Range Active Missile Block 1 0% 0%

                              Aerial Common Sensor 45% 36%

                         B-2 Radar Modernization Program 0% 0%

    Patriot/Medium Extended Air Defense System Combined Aggregate Program
    (fire unit) 0% 0%

                             Mission Planning System 0% 0%

    Sources: DOD (data); GAO (analysis and presentation).

    a

    Cost growth is expressed as the percent change in program development cost
    estimates in fiscal year 2005 dollars.

    The Army's Future Combat System is a case in point. Less than 3 years
    after program initiation and with $4.6 billion invested, the Army has
    already increased its development cost estimate $8.9 billion or 48 percent
    and delayed delivery of initial capability by 4 years over the original
    business case. Similarly, just over 1 year after initiating development of
    the Aerial Common Sensor aircraft, the Army has reported that severe
    weight and design problems discovered during development have stopped work
    on the program. As a result, program officials are anticipating at least a
    45 percent cost increase and a delay of 2 years in delivering an initial
    capability to the warfighter. These two Army programs are not the only
    ones experiencing problems. Table 2 contains cost and schedule data for 6
    of the 10 largest development programs initiated under the revised policy,
    including the Future Combat System and Aerial Common Sensor. As the table
    illustrates there are several programs experiencing large cost increases
    and schedule delays.

    Table 2: Cost andSchedule Outcomes for 6 of the 10 Largest Development
    Programs Sorted by Percent of System Development Remaining

    Percent development cost Delay in delivery of initial Percent of
    development Programs growth capability in months remaining

                            Aerial Common Sensor 45% 24 85%

                            Future Combat System 48% 48 78%

                            Joint Strike Fighter 30% 23 60%

                       Expeditionary Fighting Vehicle 61% 48 49%

    C-130 Avionics Modernization 122% Delays anticipated due to Undetermined
    due to program Program program restructure restructure

    Global Hawk (RQ-4B) 166% Delays anticipated due to Undetermined due to
    program program restructure restructure

    Sources: DOD (data); GAO (analysis and presentation).

    A good measure of acquisition performance is return on investment as
    expressed in acquisition program unit cost because unit cost represents
    the value DOD is getting for its acquisition dollars invested in a certain
    program. The programs listed in table 2 will not achieve the return on
    investment that DOD anticipated when they began development. In the case
    of Joint Strike Fighter, for example, DOD initially intended to purchase
    2,866 aircraft at an acquisition program unit cost of about $66 million.
    The Navy has reduced the number of Joint Strike Fighter aircraft it plans
    to buy; technology and design problems encountered during development have
    led to the significant cost growth. As a result, the acquisition program
    unit cost is now about $84 million, an increase of 27 percent. We recently
    reported that the risk of even greater increases is likely because flight
    testing has not yet started and the acquisition strategy involves
    substantial overlap of development and production. Similar problems have
    led to increases in the Future Combat System program. At program
    initiation, the Army anticipated that each of 15 units would cost about
    $5.5 billion to develop and deliver. Since that time, instability in the
    program's technologies and requirements have led to significant cost
    increases, leading to a 54 percent increase in acquisition program unit
    cost, now estimated to be $8.5 billion.

    Regarding all 23 development programs, DOD leaders originally planned to
    invest a total of about $83 billion (fiscal year 2005 dollars) for system
    development and anticipated delivering an initial operational capability
    to the warfighter in 77 months on average. However, development costs have
    grown and delivery schedules have been delayed significantly. DOD now
    expects to invest over $106 billion in those same programs, an increase of
    over $23 billion or 28 percent. The delivery of initial capability to the
    warfighter is expected to take an average of 88 months or nearly 1 year
    longer than originally planned. Figure 3 shows changes in these business
    case elements for these programs in the short time since their initiation.

              Figure3: Cost andSchedule Growth under DOD's Revised Policy

      DOD Is Not Effectively Implementing the Policy's Knowledge-Based, Evolutionary
      Approach

    DOD is not effectively implementing the knowledge-based process and
    evolutionary approach emphasized in its acquisition policy. While the
    policy outlines a specific knowledge-based process of concept refinement
    and technology development 3 to help ensure a sound business case is
    developed before committing to a new development program, almost 80
    percent of the programs we reviewed were permitted to bypass this process.
    Furthermore, the policy emphasizes the need to mature all critical
    technologies before starting system development and to demonstrate that
    the product's design is mature before beginning system demonstration.
    However, nearly three-fourths of the programs reported having immature
    critical technologies when they received approval to start development,
    and at least half of the programs had not achieved design maturity before
    holding their design review and gaining approval to enter the system
    demonstration phase of development. The policy also emphasizes the use of
    an evolutionary product development approach, yet program officials
    continue to structure major acquisition programs to achieve large advances
    in capability within a single step development program. This strategy has
    historically resulted in poor cost and schedule outcomes.

        Knowledge-Based Process Not Enforced

    DOD decision makers continue to approve programs for system development
    that have not followed key elements of the policy's suggested
    knowledge-based process. The policy requires program managers to provide
    senior decision makers with knowledge about key aspects of a system at
    critical investment points in the acquisition process. Our prior reviews
    have identified those critical points as the start of system development
    or program start (referred to as Milestone B in the DOD acquisition
    policy), design readiness review separating system integration and system
    demonstration, and production commitment (Milestone C in the DOD
    acquisition policy). The most important point occurs at program start,
    when system development begins. DOD acquisition guidance emphasizes the
    importance of the acquisition phases preceding program start, noting that
    the decisions made during those phases-concept

    3

    According to DOD Instruction 5000.2, the concept refinement phase is
    intended to refine the initial concept and develop a technology
    development strategy. Concept refinement ends when the decision authority
    approves a preferred solution resulting from the analysis of alternatives
    and approves the associated technology development strategy. After concept
    refinement, a project enters technology development at Milestone A, when
    the decision maker has approved the technology development strategy. The
    purpose of this phase is to reduce technology risk and to determine the
    appropriate set of technologies to be integrated into a full system.

    Acquisition Officials Are Not Effectively Using Early Processes to Develop
    Executable Business Cases

    refinement and technology development-generally define the nature of an
    entire acquisition program.

    Acquisition officials continue to begin system development without
    following early processes for developing executable business cases. A
    business case should provide demonstrated evidence that (1) the
    warfighter's needs are real and necessary and that they can best be met
    with the chosen concept and (2) the chosen concept can be developed and
    produced within existing resources-including technologies, design
    knowledge, funding, and the time to deliver the product when it is needed.
    Establishing a business case calls for a realistic assessment of risks and
    costs; doing otherwise undermines the intent of the business case and
    invites failure. This process requires the user and developer to negotiate
    whatever trade-offs are needed to achieve a match between the user's
    requirements and the developer's resources before system development
    begins.

    The revised policy and associated guidance emphasize the importance of
    following a sound process of systems engineering 4 and decision making
    prior to initiating a system development program. The process established
    in the policy consists of two phases, concept refinement and technology
    development, and a major decision review called Milestone A, which if
    rigorously followed, would provide acquisition officials with an
    opportunity to assess whether program officials had the knowledge needed
    to develop an executable business case. However, almost 80 percent of the
    programs we reviewed began system development without holding any prior
    decision review. Senior officials with the Office of the Secretary of
    Defense confirmed that this is a common practice among defense acquisition
    programs. This practice eliminates a key opportunity for decision makers
    to assess early product knowledge needed to establish a business case that
    is based on realistic cost, schedule, and performance expectations.

    Although program officials conduct analysis before starting a development
    program, they do not consistently follow a process to capture the critical
    knowledge needed to produce executable business cases, as evidenced by

    4

    Systems engineering is a technical management tool that provides the
    knowledge necessary to translate requirements into specific, achievable
    capabilities. By using the tools of systems engineering during these early
    phases of concept refinement and technology development acquisition
    decision makers and developers can work together to close gaps between
    requirements and available resources-well before system development
    starts.

    Page 13 GAO-06-368 DOD Acquisition Policy

    Programs Continue to Enter System Development with Immature Technologies

    the poor outcomes current programs are experiencing. Officials with the
    Office of the Secretary of Defense recognized this lack of rigor and
    discipline in acquisition process, and in February 2004, the Under
    Secretary of Defense (Acquisition, Technology and Logistics) issued a
    department-wide policy memorandum directing acquisition officials to place
    greater emphasis on systems engineering when planning and managing
    acquisition programs. The policy requires programs to develop a systems
    engineering plan that describes the programs' overall technical approach,
    including processes, resources, metrics, and applicable performance
    incentives. Although DOD's systems engineering initiative has the
    potential to improve program performance, officials have found that the
    preliminary results are mixed. Early analysis shows that implementation is
    inconsistent while program officials learn to develop and implement
    systems-engineering plans.

    DOD decision makers continue to permit programs to enter system
    development before critical technologies are mature. Our review of
    technology readiness assessments and acquisition decision memorandums for
    our nine case study programs found that seven of the nine programs were
    approved to begin development even though program officials reported
    levels of knowledge below the criteria suggested in the policy and
    associated guidance, specifically in the area of technology maturity. 5
    Those seven programs are not isolated cases. As illustrated in Figure 4,
    13 of the programs (nearly three-fourths) that received approval to enter
    system development under the revised policy did so with less than 100
    percent of their critical technologies mature to the level specified by
    DOD. Only 2 of those programs had more than 75 percent of their
    technologies mature when they began (see appendix III for technology
    maturity data for each program).

    5

    DOD's revised policy emphasizes the importance of reducing technology risk
    and demonstrating technologies in a relevant environment (technology
    readiness level 6) prior to program start. A technology readiness level of
    6 means the technology should be very close to the planned form, fit, and
    function of its physical configuration and that it has been tested or
    proven to work in a relevant environment such as a laboratory. GAO
    recommends a higher level of maturity in its best practice model based on
    best commercial practices. This would require a demonstration of the
    technology in the environment it is expected to be used.

    Programs Continue Past Design Reviews before Design Maturity is
    Demonstrated

    Figure 4: Comparison of Programs with Mature versus Immature Technologies
    at Start ofSystem Development

    Note: This figure depicts technology maturity status for 18 of the 23
    programs in our review. Technology maturity data was not available for the
    other 5 programs.

    Even though acquisition policy states that technologies shall be mature
    before beginning system development, the practice of accepting high levels
    of technology risk at program start continues to be the norm and not the
    exception. An official with Office of the Secretary of Defense responsible
    for reviewing and validating program assessments of technology maturity
    informed us that the office generally views immature critical technologies
    at the beginning of development as an acceptable risk as long as program
    officials can show that they have a plan to mature the technologies by the
    time the program reaches its design readiness review, which requires
    additional investments to move a program from system integration into
    system demonstration. Therefore, risk management plans are consistently
    viewed as acceptable substitutes for demonstrated knowledge.

    In addition to emphasizing the importance of capturing technology
    knowledge before starting system development, DOD's policy also highlights
    the importance of demonstrating design maturity before moving from the
    integration phase of system development into system demonstration and
    initial manufacturing. The policy establishes a design readiness review
    between the two phases to determine whether a product's design is mature
    and stable and whether the product is ready to move ahead. While DOD's
    policy does not require programs to demonstrate any specific level of
    design maturity, our past work has found that a key indicator of design
    maturity is the completion of 90 percent of the system's engineering
    drawings. We found that defense programs that moved forward with lower
    levels of design maturity, as indicated by drawing completion, encountered
    costly design changes and parts shortages that, in turn, caused labor
    inefficiencies, schedule delays, and quality problems. Consequently, those
    programs required significant increases in resources-time and money-over
    what was estimated at the point each program entered the system
    demonstration phase.

    We analyzed engineering drawing completion data for 8 programs initiated
    under the revised policy that have held a design review, 6 and found that
    more than half of those programs had not completed 90 percent of their
    design drawings before they received approval to enter the system
    demonstration phase of development. We also analyzed drawing-release data
    for three programs that have not yet held their design review but have
    projected the number of drawings officials anticipate will be completed
    when their reviews are held. Based on projections provided by program
    officials, 2 of those 3 programs are expected to have less than 55 percent
    of their drawings complete before they seek approval to begin system
    demonstration and initial manufacturing.

    Some programs did not report having a design readiness review but did
    report having a critical design review. Where this was the case, we
    assessed those programs' drawing data at their critical design review.

    Page 16 GAO-06-368 DOD Acquisition Policy

        Evolutionary Acquisition Is Not Being Used

    Sources: DOD (data); GAO (analysis and presentation).

    a

    Program office projections.

    Despite the revised policy's guidance that capabilities should be
    developed and delivered in individually defined and separately managed
    increments, a majority of major weapon acquisition programs we assessed
    continue to be structured to achieve revolutionary increases in capability
    within one development program. According to the policy, the objective of
    an evolutionary approach is to balance needs and available capability with
    resources and put capability into the hands of the user quickly. The
    policy states that the success of the strategy depends on consistent and
    continuous definition of requirements and the maturation of technologies
    that lead to disciplined development and production of systems that
    provide increasing capability. In this approach, requirements that cannot
    be satisfied within these limits as well as available financial resources
    must wait for future generations of the product and be managed as separate
    system development programs with separate milestones, costs, and
    schedules. In our case studies of nine acquisition programs initiated
    under the revised policy, we found only one program-the Small Diameter
    Bomb-that satisfied all of the criteria of an evolutionary approach. In
    five case studies, we found that program officials had claimed that their
    programs were evolutionary, yet our evidence shows they were not
    evolutionary in practice; 7 and in three cases, program officials chose
    not to use evolutionary acquisition from the outset. Table 3 summarizes
    our assessment of the nine case studies.

      Specific Criteria Are Needed to Ensure Disciplined and Transparent Investment
      Decisions

                 Sources: DOD (data); GAO (analysis and presentation).

    The revised acquisition policy does not contain effective controls that
    require the demonstration of product knowledge measured against specific
    criteria to ensure that acquisition officials make disciplined,
    transparent, and knowledge-based investment decisions. The lack of
    specific required criteria creates an environment in which unknowns about
    technology, design, and manufacturing processes are acceptable. Decision
    makers and program officials are left with no objective measures against
    which to gauge a program's level of knowledge, making accountability
    difficult. In the absence of criteria, transparency in acquisition
    decisions is essential to ensuring accountability, but key decision
    documents do not provide sufficient information about major decisions. DOD
    believes that acquisition decision memorandums, used to document program
    decisions, provide adequate transparency. However,

    7

    GAO, TACTICAL AIRCRAFT: Opportunity to Reduce Risks in the Joint Strike
    Fighter Program with Different Acquisition Strategy, GAO-05-271 (
    Washington, D.C.: March 15, 2005) and GAO, UNMANNED AERIAL VEHICLES:
    Changes in Global Hawk's Acquisition Strategy Are Needed to Reduce Program
    Risks, GAO-05-6 (Washington, D.C.: November 5, 2004).

    Page 18 GAO-06-368 DOD Acquisition Policy

    the decision memorandums we reviewed did not contain an explanation of the
    decision maker's rationale and rarely identify remaining risks, especially
    as they relate to the key knowledge standards emphasized in the policy.
    Further, the timeliness, accessibility, and depth, of the data contained
    in the Selected Acquisition Reports, DOD's primary means of providing
    Congress with a status report of program performance, inhibits the
    reports' usefulness as a management and oversight tool.

    In November 2003, we reported that the revised acquisition policy lacked
    many of the controls that leading commercial companies rely on to attain
    an acceptable level of knowledge before making additional significant
    investments. 8 Controls are considered effective if they are backed by
    specific criteria and if decision makers are required to consider the
    resulting data before deciding to advance a program to the next level.
    Controls used by leading companies help decision makers gauge progress in
    meeting cost, schedule, and performance goals and hold program managers
    accountable for capturing relevant product knowledge to inform key
    investment decisions. The controls we have articulated as best practices
    used by successful commercial product developers are listed below in table
    5.

    GAO, DEFENSE ACQUISITIONS: DOD's Revised Policy Emphasizes Best Practices,
    but More Controls Are Needed , GAO-04-53 ( November 10, 2003).

    Page 19 GAO-06-368 DOD Acquisition Policy

    Table 5: Types of Controls Considered Best Practices for Successful
    Product Development

               Program start (Milestone B): Start of product development

                   Demonstrate technologies to high readiness levels

    Ensure that requirements for the product are informed by the systems
    engineering process

    Establish cost and schedule estimates for product on the basis of
    knowledge from preliminary design using system engineering tools

                       Conduct decision review for program start

               Design readiness review: Beginning of system demonstration

                         Complete 90 percent of design drawings

                      Complete subsystem and system design reviews

               Demonstrate with prototype that design meets requirements

    Obtain stakeholders' concurrence that drawings are complete and producible

                    Complete the failure modes and effects analysis

                          Identify key system characteristics

                       Identify critical manufacturing processes

    Establish reliability targets and growth plan on the basis of demonstrated
    reliability rates of components and subsystems

                 Conduct decision review to enter system demonstration

         Production commitment (Milestone C): Initiation of low-rate production

                          Demonstrate manufacturing processes

                       Build production-representative prototypes

         Test production-representative prototypes to achieve reliability goal

    Test production-representative prototypes to demonstrate product in
    operational environment

                        Collect statistical process control data

    Demonstrate that critical processes are capable and in statistical control

                      Conduct decision review to begin production

    Sources: GAO (analysis and presentation).

    Some senior officials with the Office of the Secretary of Defense believe
    that the effective use of controls in DOD's policy and the establishment
    of more specific criteria for decision making would improve program
    outcomes. They note that specific criteria need to be established and that
    programs need to be held accountable to those criteria before being
    permitted to proceed into the next phase. They also note that the criteria
    for moving an acquisition effort from one phase of the process to the
    next, primarily documented in acquisition decision memorandums as exit

    criteria, are not typically specific and often do not relate to the key
    knowledge-based criteria suggested in the policy.

    We found this to be true for our nine case study programs. We reviewed
    acquisition decision memorandums in our case studies and determined that
    they were not useful in explaining the decision maker's rationale and in
    almost all of the cases they did not address the key knowledge criteria
    suggested in the acquisition policy. In most instances, the decision maker
    simply noted that the program being assessed was ready to proceed into
    system development, but did not provide an explanation of the rationale
    for the decision. Senior officials with the Office of the Secretary of
    Defense told us that they agree that a better explanation of the decision
    maker's rationale, specifically in instances where the knowledge criteria
    are not fully met, would provide transparency and ultimately allow for a
    more accountable decision-making process. The following two examples
    illustrate how decision documentation is lacking:

         o The Future Combat System program received approval to enter system
           development and demonstration in 2003, with 19 percent of its critical
           technologies mature, well below the policy's standard. The acquisition
           decision memorandum supporting this decision did not provide the
           rationale for approving the system with such a large number of
           immature critical technologies. The memo did direct an updated review
           of the decision 18 months later and that the program "remain flexible
           and open to accommodate trades in the system architecture and in the
           individual systems' designs."
         o The Joint Strike Fighter program was approved to enter system
           development in 2001. The acquisition decision memorandum did not
           address the fact that 75 percent of the program's critical
           technologies were not mature to the policy's standard. The memorandum
           did acknowledge that the program's requirements could be changed or
           modified, noting that further refinements in the requirements should
           be explored as a potential way to reduce program costs. However, the
           memorandum did not explain why the decision maker determined that the
           program should enter development without achieving the technology and
           requirements knowledge emphasized in the policy.

    The acquisition decision memorandums for most of the other programs we
    reviewed did not specifically address critical gaps in knowledge, nor did
    they effectively explain the decision makers' rationale for deeming those
    programs ready to begin system development. In memos where we found a
    reference to key knowledge principles, such as technology maturity, the

                                      Conclusions

    decision makers acknowledged that more effort was needed to meet the
    policy's suggested criteria but considered the risk acceptable to begin
    development. These memos did not explain why risks were considered
    acceptable. For example, the Navy's Multi-Mission Maritime Aircraft
    program had none of its critical technologies mature at program
    initiation. The decision maker acknowledged the need to further mature the
    critical technologies but approved the program to enter development.
    Instead of holding the program to the policy's criteria for entering
    development, the decision maker simply directed the Navy to work with the
    Office of the Secretary of Defense to implement risk mitigation and
    technology maturation plans during the integration phase of system
    development.

    In addition to the lack of transparency provided through acquisition
    decision memoranda, we also found that the data presented to Congress in
    DOD's Selected Acquisition Reports (SARs) provided only limited usefulness
    as an oversight tool. Since 1969, SARs have been the primary means by
    which DOD reports the status of major weapon system acquisitions to
    Congress. SARs are reports that are expected to contain information on the
    cost, schedule, and performance of major weapon systems in comparison with
    baseline values established at program start, full-scale development, and
    production decision points. Our analysis, as well as a previous GAO
    review, 9 of current and historical SAR data found that the timeliness,
    accessibility, and depth of the data contained in the reports limits their
    usefulness as an oversight tool. Our prior review noted that a number of
    opportunities exist for DOD to give Congress more complete information on
    the performance of major defense acquisition programs. DOD agreed that SAR
    data could be improved to make it more useful to Congress.

    Failing to consistently implement the knowledge-based process and
    evolutionary principles emphasized in the revised acquisition policy-
    coupled with a lack of specific criteria for making key investment
    decisions-are keeping DOD on its historical path of poor cost and schedule
    outcomes. Most programs are incurring the same scope of cost overruns and
    schedule delays as programs managed under prior DOD policies. More
    consistent use of the early acquisition processes would improve the
    quality and viability of program business cases by ensuring

    9

    GAO, DEFENSE ACQUISITIONS: Information for Congress on Performance of
    Major Programs Can Be More Complete, Timely, and Accessible, GAO-05-182 (
    March 28, 2005).

    Page 22 GAO-06-368 DOD Acquisition Policy

    they are founded on knowledge obtained from rigorous and disciplined
    analysis. The initiative by Office of the Secretary of Defense to
    reinstitute the use of systems engineering is a step in the right
    direction. However, in order for this initiative to be effective DOD must
    establish and enforce specific criteria at key decision points. Our past
    work has identified and recommended criteria and controls that should be
    consistently applied at major decision points. The enforcement of these
    criteria is critical to ensuring that programs have the knowledge
    necessary to successfully move forward through the acquisition process.
    DOD officials have acknowledged the advantages of using knowledge-based
    criteria and controls, but believe the policy already includes enough
    controls to achieve effective program results. However, without
    enforceable criteria, defense officials are challenged to determine
    whether adequate knowledge has been obtained for investing taxpayer
    dollars. The lack of enforceable criteria also makes it difficult to hold
    defense officials accountable for their decisions.

    DOD must ensure that appropriate knowledge is captured and used at

      Recommendations

    critical junctures to make decisions about moving a program forward and

    for Executive Action investing more money. We recommend that the Secretary
    of Defense require program officials to demonstrate that they have
    captured appropriate knowledge at three key points-program start, design
    review for transitioning from system integration to system demonstration,
    and production commitment-as a condition for investing resources. At a
    minimum those controls should require program officials to demonstrate
    that they have achieved a level of knowledge that meets or exceeds the
    following criteria at each respective decision point:

            * Program start (Milestone B): Start of product development
                 o Demonstrate technologies to high readiness levels
                 o Ensure that requirements for the product are informed by the
                   systemsengineering process
                 o Establish cost and schedule estimates for product on the basis
                   of knowledge from preliminary design using system engineering
                   tools
                 o Conduct decision review for program start
            * Design readiness review: Beginning of system demonstration
                 o Complete 90 percent of design drawings
                 o Complete subsystem and system design reviews
                 o Demonstrate with prototype that design meets requirements
                 o Obtain stakeholders' concurrence that drawings are complete
                   and producible
                 o Complete the failure modes and effects analysis
                 o Identify key system characteristics
                 o Identify critical manufacturing processes
                 o Establish reliability targets and growth plan on the basis of
                   demonstrated reliability rates of components and subsystems
                 o Conduct decision review to enter system demonstration
            * Production commitment (Milestone C): Initiation of low-rate
              production
                 o Demonstrate manufacturing processes
                 o Build production-representative prototypes
                 o Test production-representative prototypes to achieve
                   reliability goal
                 o Test production-representative prototypes to demonstrate
                   product in operational environment
                 o Collect statistical process control data
                 o Demonstrate that critical processes are capable and in
                   statistical control
                 o Conduct decision review to begin production

    Furthermore, to ensure that major decisions are transparent and that
    program officials and decision makers are held accountable, we recommend
    that the Secretary of Defense require decision makers to include written
    rationale for each major decision in acquisition decision documentation.
    The rationale should address the key knowledge-based criteria appropriate
    for milestone decisions, explain why a program's level of knowledge in
    each area was deemed acceptable if criteria have not been met and provide
    a plan for achieving the knowledge necessary to meet criteria within a
    given time frame.

                           Agency Comments and Our Evaluation

    DOD provided written comments on a draft of this report. The comments
    appear in appendix II.

    DOD partially concurred with our recommendation that the Secretary of
    Defense should establish specific controls to insure that program
    officials demonstrate that they have captured a level of knowledge that
    meets or exceeds specific criteria at three key points in the acquisition
    process: program start, design readiness review, and production
    commitment. DOD agreed that knowledge-based decision making is consistent
    with sound business practice and stated that it would continue to develop
    policy that reflects a knowledge-based approach and improves acquisition
    outcomes. DOD noted that it would consider our recommendations as it
    reassesses the DOD acquisition business model and the knowledge required
    at each decision point. We believe that DOD's plan to reassess its
    business model provides a good opportunity to establish the controls and
    specific criteria recommended in this report. Therefore, we are retaining
    our recommendation that the Secretary of Defense should establish controls
    to insure that program officials demonstrate that they have captured a
    level of knowledge that meets or exceeds specific criteria at three key
    points in the acquisition process.

    DOD also partially concurred with our recommendation that the Secretary of
    Defense require decision makers to provide written rationale in
    acquisition decision documentation for each major decision. DOD agreed
    that acquisition decisions should be documented, decision makers should be
    held accountable, and that they should provide the rationale for their
    decisions. DOD believes that the implementation of Section 801 of the
    National Defense Authorization Act for FY 2006 reinforces these processes.
    The act calls for the decision maker to certify that the program meets
    certain requirements, such as technology maturity, prior to starting a new
    development program at Milestone B. However, the act is focused on the
    decision to start a development program and does not identify specific
    criteria for programs to be measured against at design readiness review or
    production commitment. We believe our recommendation adds transparency and
    accountability to the process because it requires the decision maker to
    provide the rationale for a decision to allow a program to move forward,
    not only at Milestone B but at other key decision points as well.
    Therefore, we are retaining our recommendation that the Secretary of
    Defense require decision makers to provide written rationale for each
    major decision in acquisition decision documentation.

    We are sending copies of this report to the Secretary of Defense; the
    Secretaries of the Air Force, Army, and Navy; and the Director of the
    Office of Management and Budget. We will provide copies to others on
    request. This report will also be available at no charge on GAO's Web site
    at http://www.gao.gov.

    If you have any questions about this report or need additional
    information, please call me at (202) 512-4841 (sullivanm@gao.gov). Contact
    points for the offices of Congressional Relations and Public Affairs are
    located on the last page of this report. Key contributors to this report
    were Michael Hazard, Assistant Director; Lily Chin; Ryan Consaul;
    Christopher DePerro; Travis Masters; and Adam Vodraska.

    Michael J. Sullivan Director, Acquisition

    and Sourcing Management

                           Appendix I: Scope and Methodology

    To assess the impact of DOD's revised acquisition policy, we analyzed cost
    and schedule data for 23 major defense acquisition programs that were
    approved to begin system development under the revised policy. We did not
    assess space, missile defense, or ship programs. We collected our data
    from Selected Acquisition Reports, presidential budget documents, ongoing
    GAO work, and pertinent program officials. We utilized previous GAO
    reports related to defense acquisition policies and worked with
    knowledgeable GAO staff to ensure the use of current, accurate data. We
    also analyzed more than 150 annual Selected Acquisition Reports covering a
    36-year period from 1969 to 2005, to determine historical trends related
    to outcomes of acquisition policy implementation.

    We assessed whether the revised policy's knowledge-based, evolutionary
    acquisition principles were being effectively implemented by conducting 9
    case study reviews and analyzing design maturity data for 11 programs that
    have made engineering-drawing data available to GAO. Our case study
    programs were the Aerial Common Sensor, Multi-Platform Radar Technology
    Insertion Program, Global Hawk Unmanned Aerial Vehicle, Small Diameter
    Bomb, Future Combat System, Joint Strike Fighter, Expeditionary Fighting
    Vehicle, Multi-Mission Maritime Aircraft, and the E-2 Advanced Hawkeye. We
    interacted directly with numerous program officials to seek input on
    current developments with their programs. We studied program documents to
    assess how well programs understand and are implementing the revised
    acquisition policy. We also analyzed drawing release data for those
    programs that have either passed their design review or have provided GAO
    with estimated drawing release data for a future design review to assess
    design maturity. In several cases, we asked that program offices verify
    information in these various documents.

    We also reviewed Department of Defense (DOD) Directive 5000.1, DOD
    Instruction 5000.2, and the Defense Acquisition Guidebook. In addition we
    examined each of the military services' policy directives and guidance,
    DOD memorandums to include policy intent and DOD expectations regarding
    policy implementation as well as Joint Capabilities Integration and
    Development System documents. We interviewed relevant officials in
    Washington, D.C., from the Office of the Director, Defense Research and
    Engineering, the Joint Staff, the Office of the Secretary of Defense, and
    Army, Navy, and Air Force acquisition policy staff in order to better
    understand the content of these documents and the intent of DOD's policy.

    We conducted our review from May 2005 to February 2006 in accordance with
    generally accepted government auditing standards.

    Appendix II: Comments from the Department of Defense

    Appendix II: Comments from the Department of Defense

    Appendix II: Comments from the Department of Defense

    Appendix III: Program Data for 23 Programs Initiated under DOD's Revised
    Acquisition Policy (as of December 2005)

                       Page 31 GAO-06-368 DOD Acquisition Policy

    Appendix III: Program Data for 23 Programs Initiated under DOD's Revised
    Acquisition Policy (as of December 2005)

    Sources: DOD (data); GAO (analysis and presentation).

    Note: In this table the term "not available" means that GAO had not
    received sufficient data to make an assessment of the given program's
    design and/or technology maturity.

    a

    Milestone I was a forerunner to Milestone A, the decision review that
    currently precedes the start of technology development.

    b

    Program office projections.

    Cost growth is expressed as the percent change in program development cost
    estimates in fiscal year 2005 dollars.

    Related GAO Products

    DOD Acquisition Outcomes: A Case for Change. GAO-06-257T. Washington,
    D.C.: November 15, 2005.

    Defense Acquisitions: Stronger Management Practices Are Needed to Improve
    DOD's Software-Intensive Weapon Acquisitions. GAO-04-393. Washington,
    D.C.: March 1, 2004.

    Best Practices: Setting Requirements Differently Could Reduce Weapon
    Systems' Total Ownership Costs. GAO-03-57. Washington, D.C.: February 11,
    2003

    Best Practices: Capturing Design and Manufacturing Knowledge Early
    Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15,
    2002.

    Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices.
    GAO-02-469T. Washington, D.C.: February 27, 2002.

    Best Practices: Better Matching of Needs and Resources Will Lead to Better
    Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 2001.

    Best Practices: A More Constructive Test Approach Is Key to Better Weapon
    System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 2000.

    Defense Acquisition: Employing Best Practices Can Shape Better Weapon
    System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000.

    Best Practices: DOD Training Can Do More to Help Weapon System Programs
    Implement Best Practices. GAO/NSIAD-99-206. Washington,D.C.: August16,
    1999.

    Best Practices: Better Management of Technology Development Can Improve
    Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 1999.

    Defense Acquisitions: Best Commercial Practices Can Improve Program
    Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.

    Defense Acquisition: Improved Program Outcomes Are Possible. GAO/T-
    NSIAD-98-123. Washington, D.C.: March 17, 1998.

    Related GAO Products

    Best Practices: DOD Can Help Suppliers Contribute More to Weapon System
    Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.

    Best Practices: Successful Application to Weapon Acquisition Requires
    Changes in DOD's Environment. GAO/NSIAD-98-56. Washington, D.C.: February
    24, 1998.

    Best Practices: Commercial Quality Assurance Practices Offer Improvements
    for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 1996.

    (120448)

      GAO's Mission

    The Government Accountability Office, the audit, evaluation and
    investigative arm of Congress, exists to support Congress in meeting its
    constitutional responsibilities and to help improve the performance and
    accountability of the federal government for the American people. GAO
    examines the use of public funds; evaluates federal programs and policies;
    and provides analyses, recommendations, and other assistance to help
    Congress make informed oversight, policy, and funding decisions. GAO's
    commitment to good government is reflected in its core values of
    accountability, integrity, and reliability.

    The fastest and easiest way to obtain copies of GAO documents at no cost
    is through GAO's Web site ( www.gao.gov) . Each weekday, GAO posts GAO
    Reports and newly released reports, testimony, and correspondence on its
    Web site. To

    have GAO e-mail you a list of newly posted products every afternoon, go to
    www.gao.gov and select "Subscribe to Updates."

                                 Order by Mail or Phone

    The first copy of each printed report is free. Additional copies are $2
    each. A check or money order should be made out to the Superintendent of
    Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
    copies mailed to a single address are discounted 25 percent. Orders should
    be sent to:

    U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
    D.C. 20548

    To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
    512-6061

      To Report Fraud, Contact:

    Web site: www.gao.gov/fraudnet/fraudnet.htm

      E-mail: fraudnet@gao.gov

    Federal Programs Automated answering system: (800) 424-5454 or (202)
    512-7470

    Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S.
    Government Accountability Office, 441 G Street NW, Room 7125 Relations
    Washington, D.C. 20548

    Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800

      Public Affairs

    U.S. Government Accountability Office, 441 G Street NW, Room 7149
    Washington, D.C. 20548

        PRINTED ON

    RECYCLED PAPER
    *** End of document. ***