    Defense Acquisitions: Steps to Improve the Crusader Program's
    Investment Decisions (25-FEB-02, GAO-02-201).

    The Army wants an artillery system with greater firepower, range,
    and mobility than its current self-propelled howitzer. In 1994,
    the Army began to develop the Crusader, an advanced artillery
    system consisting of a self-propelled 155-millimeter howitzer and
    a resupply vehicle. The Department of Defense (DOD) will decide
    next year whether the Crusader program should enter its system
    development and demonstration stage, which will require the
    commitment of major resources. GAO found that the Crusader
    program has made considerable progress in developing key
    technologies and reducing its size and weight. However, more
    progress and knowledge is needed to minimize the risk of cost
    overruns, schedule delays, and performance shortfalls. The
    Crusader program will likely enter product development with most
    of its critical technologies less mature than best practices
    recommend. Most of the Crusader's critical technologies have been
    demonstrated in a relevant environment but not in the more
    demanding operational environment. Although the Army is reducing
    the Crusader's weight so that two vehicles can be deployed on a
    C-17 aircraft, the deployability advantage gained does not appear
    significant. The reduction in the Crusader system's weight would
    only decrease the number of C-17 flights needed to transport two
    complete systems and support equipment from five to four flights.
    A lighter system offers several other benefits, and knowing the
    magnitude of the deployability advantage of reduced weight would
    allow the Army to make better decisions on trade offs. An
    apparent overlap exists between the Crusader's and the Future
    Combat Systems' capabilities and schedules. The Army expects the
    Future Combat Systems to meet the same artillery missions as the
    Crusader and eventually replace it. The current schedules for
    initial fielding of the Future Combat Systems and the Crusader
    system occur in the same year, 2008. The extent of this apparent
    overlap depends more on the Future Combat Systems than the
    Crusader because less is known about the Future Combat Systems'
    technologies.
    -------------------------Indexing Terms-------------------------
    REPORTNUM:   GAO-02-201
        ACCNO:   A02801
      TITLE:     Defense Acquisitions: Steps to Improve the Crusader
    Program's Investment Decisions
         DATE:   02/25/2002
      SUBJECT:   Defense cost control
    	     Developmental testing
    	     Military cost control
    	     Performance measures
    	     Schedule slippages
    	     Strategic mobility forces
    	     Systems development life cycle
    	     Weapons research and development
    	     Weapons systems
    	     Army Brilliant Anti-Armor Submunition
    	     Program

    	     Army Crusader System
    	     Army Future Combat Systems
    	     C-17 Aircraft
    	     C-5 Aircraft
    	     Crusader Resupply Vehicle
    	     Crusader Self Propelled Howitzer
    	     Desert Storm
    	     Galaxy Aircraft
    	     Globemaster Aircraft
    	     Joint Strike Fighter
    	     Paladin Howitzer
    	     SEI Software Acquisition Capability
    	     Maturity Model

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO Product.                                                 **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ******************************************************************
    GAO-02-201

    United States General Accounting Office

                       GAO Report to Congressional Committees

    February 2002

    DEFENSE ACQUISITIONS

    Steps to Improve the Crusader Program's Investment Decisions

                                          a

    GAO-02-201

    Contents

    Letter

    Results in Brief
    Background
    Critical Technologies Need Additional Maturity to Better

    Assure Low-risk Product Development Lighter-weight Crusader May Not
    Significantly Improve Strategic Deployability Apparent Overlap of Crusader
    and Future Combat Systems

    Programs Creates Uncertainties Conclusions Recommendations for Executive
    Action Agency Comments and Our Evaluation Scope and Methodology

    1 1 4

    8

    16

    20 21 22 23 25

    Appendixes

      Appendix I: Technology Readiness Levels and Their Descriptions 29 Appendix
       II: Comments from the Department of Defense 30 Appendix III: GAO Contacts
                                                   and Staff Acknowledgments 33

    Table Table 1: Results of the Joint Crusader TRL Assessment

    Figures  Figure 1: Crusader  Howitzer 5 Figure 2:  Crusader Tracked Resupply
    Vehicle 6 Figure 3: Crusader Wheeled Resupply Vehicle 7

    A

    United States General Accounting Office Washington, D.C. 20548

    February 25, 2002

    Congressional Committees

    To address future threats, the Army has identified a requirement for an
    artillery system that has greater firepower, range, and mobility than its
    current self-propelled howitzer-the Paladin. Operation Desert Storm
    demonstrated that current howitzers were unable to keep up with our tanks
    and fighting vehicles. In 1994, the Army began to develop the Crusader, an
    advanced artillery system consisting of a self-propelled 155-millimeter
    howitzer and a resupply vehicle. The Army's total acquisition cost in the
    Crusader program is projected to be about $11 billion. In 2000, the Army
    changed its requirements and restructured the Crusader program to make the
    system lighter and more deployable. This change was in response to the
    Army's planned transformation to a future force, which will also be lighter
    and more deployable. The Army expects to use the Crusader until it is
    eventually replaced by the main component of the future force, known as the
    Future Combat Systems.

    In April 2003, Department of Defense (DOD) will decide whether the Crusader
    program should enter its system development and demonstration-or product
    development-stage, which will require the commitment of major resources to
    develop and design the Crusader system and to demonstrate its integration,
    interoperability, and utility. The opportunity to take actions that can put
    the program in a better position to succeed and, thus, minimize future cost
    and schedule increases is now, before the start of product development. As
    the Army approaches that decision point, we examined three major aspects of
    the program: (1) the progress in developing Crusader's technology and
    software, (2) the Crusader's requirement for improved deployability, and (3)
    the Army's timetables for developing the Crusader and the Future Combat
    Systems.

    Results in Brief The Crusader program has made considerable progress in
    developing key technologies and reducing its size and weight. However, with
    a 2003 decision date for committing to product development, more progress
    and knowledge will be needed to minimize risks of cost overruns, schedule
    delays, and performance shortfalls.

    * Based on current Army plans, the Crusader program will likely enter
    product development with the majority of its critical technologies less
    mature than best practices recommend. Most of the Crusader's critical

    technologies have been demonstrated in a relevant environment but not the
    more demanding operational environment. Achieving the higher level of
    technology maturity prior to beginning product development reduces the risk
    of costly schedule delays. Crusader technologies that have not reached the
    desired level of maturity for product development include the suspension,
    track, transmission, and prognostics.1 The Army made significant
    improvements to the management of the Crusader's software development
    process in response to software design problems experienced in 1998. The
    Army's continued attention to software development is essential given the
    large amount that remains to be completed.

    * Although the Army is redesigning the Crusader to reduce individual vehicle
    weight from about 60 tons to about 40 tons so that two vehicles can be
    deployed on a C-17 aircraft, the deployability advantage gained does not
    appear significant. An Army analysis conducted at our request shows that the
    reduction in the Crusader system's weight would only decrease the number of
    C-17 flights needed to transport two complete systems and support equipment
    from five flights to four flights. Moreover, the Army plans to move
    Crusaders by aircraft only under extraordinary conditions and in limited
    numbers. Also, as currently designed, the weight of two howitzers is
    projected to be very close to the C-17's weight limit and their projected
    size would make them a very tight fit in the aircraft, if they fit at all.
    The Army may need to make cost, schedule, and performance trade-offs to meet
    and maintain that weight. While, in general, a lighter system offers a
    number of other benefits, knowing the magnitude of the deployability
    advantage gained by reducing weight would enable the Army to make better
    trade-off decisions.

    * An apparent overlap exists between the Crusader's and the Future Combat
    Systems' capabilities and schedules. The Army expects the Future Combat
    Systems to eventually meet the same artillery missions as the Crusader and
    eventually replace it. The current schedules for initial fielding of the
    Future Combat Systems and the Crusader system occur in the same year, 2008.
    The extent of this apparent overlap depends more on the Future Combat
    Systems than the Crusader, because less is known about the Future Combat
    Systems' technologies.

    1  Prognostics is  a system  to forecast  potential failures  in subsystems,
    allowing the maintainers to correct them before they fail.

    More will be understood about these technologies and, thus, the Future
    Combat Systems' schedule, when the Army formally assesses their maturity in
    early 2003. Current plans call for committing to the Crusader's product
    development in the same year the Future Combat Systems' technologies are
    assessed.

    We are recommending that the Army further mature the Crusader's technologies
    before committing to product development; assess the benefits of its weight
    reduction relative to its strategic deployability; and assess the projected
    capabilities and fielding schedules for Future Combat Systems as part of the
    Crusader's milestone decision for beginning product development.

    In commenting on our report, DOD did not agree with our recommendation on
    maturing critical Crusader technologies and partially agreed with our
    recommendations on Crusader's deployability requirement and the Crusader's
    apparent overlap with the Future Combat Systems. In not agreeing, DOD noted
    that the Crusader program is using modeling and simulation to determine the
    Crusader's readiness to enter product development and stated that changing
    its acquisition strategy to further mature critical technologies would add
    significantly to the development time and expense without significantly
    reducing risk or improving performance. We agree that modeling and
    simulation are useful management tools, but believe that demonstrating
    critical technologies in an operational environment before the start of
    product development has been shown to lower program risks of significant
    cost overruns, schedule delays, and performance shortfalls. In partially
    agreeing with our recommendation to conduct an analysis of the Crusader's
    deployability requirement, DOD said that the current requirement is not
    considered a key performance parameter and, as a result, the Army is allowed
    to make trade-offs between the requirement and system cost and performance.
    DOD further stated that the Army plans to review the Crusader's requirements
    prior to the 2003 milestone B decision as required by regulations. We
    believe that an analysis to determine the importance of deploying two
    Crusader howitzers on a C-17 aircraft should be conducted as soon as
    possible to provide the Army greater flexibility and knowledge in
    considering its ongoing trade-off decisions needed to meet weight
    requirements. In partially agreeing with our recommendation to determine the
    potential capabilities and schedule of the initial version of the Future
    Combat Systems before making the decision to begin Crusader product
    development, DOD said that the Crusader's capabilities are intended to
    complement rather than be redundant to the capabilities of the Future

    Combat Systems. We continue to believe that DOD cannot determine whether the
    two systems will be complementary or redundant without knowledge of the
    initial Future Combat Systems capabilities and fielding schedule. DOD does
    not yet have this knowledge. We also continue to believe that this knowledge
    needs to be considered as part of the decision to allow the Crusader program
    to enter product development.

    Background The Army plans to invest about $11 billion developing and
    procuring the Crusader, an automated, next generation field artillery
    system. To date, the program has spent about $1.7 billion in development
    costs. It plans to procure 482 Crusader systems-each system consisting of a
    self-propelled 155-millimeter howitzer and a resupply vehicle. The Army is
    developing 2 different resupply vehicles-1 with tracks and 1 with wheels-and
    plans to procure 241 of each type. The purpose of the Crusader system is to
    overcome threats from enemy artillery and reconnaissance or surveillance
    systems as well as have the mobility needed to keep up with Army tanks and
    fighting vehicles. Figure 1 shows the planned Crusader howitzer, figure 2
    the planned tracked resupply vehicle, and figure 3 the planned wheeled
    resupply vehicle.

    Figure 1: Crusader Howitzer

                    Source: United Defense Limited Partnership.

                    Figure 2: Crusader Tracked Resupply Vehicle

    Source: United Defense Limited Partnership.

    Figure 3: Crusader Wheeled Resupply Vehicle

    Source: United Defense Limited Partnership.

    The Army restructured the Crusader program in January 2000 to align
    Crusader's design with the Army's transformation to a lighter force. The
    Army's transformation will affect all aspects of Army organization,
    training, doctrine, leadership, and strategic plans as well as the types of
    equipment and technology the Army acquires. The Army expects the
    transformation to be at least a 30-year process and has not estimated its
    full cost. The centerpiece of the lighter, more deployable future force is
    the Future Combat Systems. The Future Combat Systems concept is a system of
    ground and air, manned and unmanned weapon systems, each under 20 tons that
    is planned to replace most, if not all, of the Army's ground combat systems
    without a loss in lethality and survivability. Artillery systems are among
    those to be replaced.

    The Army expects the Crusader system to fill the existing gap in artillery
    capabilities until it is replaced by the Future Combat Systems. In keeping
    with the transformation philosophy of lightweight vehicles and ease of
    deployability, the Army is redesigning Crusader to make it lighter and more
    deployable, with the goal of reducing the weight of the self-propelled
    howitzer and tracked resupply vehicle from about 60 tons to about 40 tons
    each. Program officials said that a lighter system would enhance operational
    flexibility in employing Crusader in support of any operation.

    The Crusader is currently in the program definition and risk reduction phase
    of its development program. In April 2003, the program is scheduled for a
    milestone B review to determine whether it is ready to enter its system
    development and demonstration phase. Milestone B is the point at which DOD
    decides whether to commit major resources to develop and design the system
    and to demonstrate its integration, interoperability, and utility. The
    milestone marks the start of the program's product development. The Army
    plans to deliver the first full Crusader prototype system in October 2004,
    followed by a low-rate initial production decision in February 2006, and
    initial system fielding in April 2008.

    Critical Technologies Need Additional Maturity to Better Assure Low-risk
    Product Development

    Based on current Army plans, the Army will begin the Crusader's product
    development in April 2003 but before maturing critical Crusader technologies
    to a level considered low risk relative to best practices. These risks
    relate less to whether these technologies can be matured, but more to how
    much time and cost it will take to mature them. If, after starting product
    development, the Crusader technologies do not mature on schedule and instead
    cause delays, the Army may spend more and take longer to develop, produce,
    and field the Crusader system. Crusader performance goals may also be at
    risk. On the other hand, the Army has made improvements to the management of
    the Crusader software development process.

    Assessing Technology The maturity of a program's technologies at the start
    of product Readiness Provides development is a good predictor of that
    program's future performance. Our Opportunities to Improve past reviews of
    programs incorporating technologies into new products

    and weapon systems showed that they were more likely to meet product

    Outcomes objectives when the technologies were matured before product
    development started. For example, the Ford Motor Company's practice of
    demonstrating new technologies in driving conditions before they are

    included in a new product is essential to ensuring that the new product can
    be developed on time and within budget. Similarly, we have found that the
    early demonstration of propulsion and water-planing technologies, essential
    to the performance of the Marine Corps' Advance Amphibious Assault Vehicle,
    has been instrumental to that program's staying within 15 percent of cost
    and schedule estimates.

    Conversely, cost, schedule, and performance problems were more likely to
    occur when programs started with technologies at lower readiness levels.2
    For example, the enabling technologies for the Army's Brilliant Anti-Armor
    Submunition program were very immature at the start of the program, and
    their delays became major contributors to the program's subsequent
    88-percent cost growth and 62-percent schedule slippage. Separating
    technology development from product development into two distinct program
    phases is a best practice of both successful commercial and defense
    programs. This entails demonstrating all critical technologies at the
    component or subsystem level in an operational environment during technology
    development, prior to committing major funding to product development. Under
    this practice, the critical technologies would be demonstrated in component
    or subsystem prototypes that are nearly the right size, weight, and
    configuration needed for the intended product. Such demonstrations need not
    require a full system prototype of a Crusader vehicle, but can be done using
    surrogate vehicles.

    Technology readiness levels (TRL) are a good way to gauge the maturity of
    technologies. TRLs were pioneered by the National Aeronautics and Space
    Administration to determine the readiness of technologies to be incorporated
    into products such as weapon systems. Readiness levels are measured along a
    scale of one to nine, starting with paper studies of the basic concept,
    proceeding with laboratory demonstrations, and ending with a technology that
    has proven itself on the intended product. TRLs are based on actual
    demonstrations of how well specific technologies perform in the intended
    application. For example, a technology that has been demonstrated in an
    operational environment using subsystem prototype hardware (such as a
    complete cannon system) that is at or near the final

    2 Best Practices: Successful Application to Weapon Acquisitions Requires
    Changes in DOD's Environment (GAO/NSIAD-98-56, Feb. 24, 1998) and Best
    Practices: Better Management of Technology Development Can Improve Weapon
    System Outcomes

    (GAO/NSIAD-99-162, July 30, 1999).

    system design would be rated as a TRL 7. The individual TRL descriptions can
    be found in appendix I.

    DOD has agreed that technology readiness assessments are important and
    necessary in assisting officials who decide when and where to insert new
    technologies into weapon system programs. In January 2001, DOD issued a new
    acquisition instruction that redefined the phases in the defense acquisition
    cycle and emphasized the role of technology development in the acquisition
    process.3 Under the instruction, programs use the concept and technology
    development phase, which precedes the system development and demonstration
    phase, for developing components and subsystems that must be demonstrated
    before integration into the system. The first portion of system development
    and demonstration phase is dedicated to integrating the components and
    subsystems into the system. The instruction states that DOD prefers that
    technology be demonstrated in an operational environment but must be
    demonstrated in a relevant environment to be considered mature enough for
    product development in the system development and demonstration phase.
    According to the TRL descriptions, technology demonstrated in an operational
    environment is TRL 7 and technology demonstrated in a relevant environment
    is TRL 6.

    Maturing technology from a TRL 6 to a TRL 7 represents a major step up in
    maturity. A technology at the TRL 6 maturity level needs only to be
    demonstrated as a subsystem prototype or model in a laboratory or simulated
    operational environment. A technology at the TRL 7 maturity level must be
    demonstrated as a subsystem prototype at or near the size of the required
    subsystem outside the laboratory in an actual operational environment. For
    example, operating a prototype engine on a laboratory test stand that
    simulates the effects of the vehicle's weight on the engine would be a TRL 6
    level demonstration while operating an engine in a surrogate vehicle or
    actual prototype that weighed 50 tons, on roads and cross country, would be
    a TRL 7 demonstration.

    In June 2001, DOD issued a new acquisition regulation.4 It stated that
    technology maturity is a principal element of program risk and directed
    technology readiness assessments for critical technologies sufficiently

    3 DOD Instruction 5000.2 "Operation of the Defense Acquisition System," Jan.
    4, 2001.

    4 DOD Regulation 5000.2R "Mandatory Procedures for Major Defense Acquisition
    Programs (MDAPS) and Major Automated Information System (MAIS) Acquisition
    Programs," June 10, 2001.

    prior to selected milestone decision points-including milestone B--to
    provide useful technology maturity information to the acquisition review
    process. Although the new regulation recognizes that TRLs enable consistent,
    uniform discussions of technical maturity across different types of
    technologies and provides the definitions of TRLs used in this report, it
    permits the use of TRLs or "some equivalent assessment" when performing a
    technology readiness assessment.

    Technology Maturation Will Continue into Product Development

    In June 2001, Crusader program office engineers and we assessed the maturity
    of 16 critical Crusader technologies using TRLs.5,6 This joint assessment
    determined that 10 of the 16 critical Crusader technologies were below TRL
    7. Since the Crusader program is not scheduled to commit to product
    development until April 2003, the Army still has time to mature the 10
    critical technologies to a TRL 7 level-demonstrate them in a component or
    subsystem prototype in an operational environment. However, the Army's
    Crusader plans will result in 10 of the critical Crusader technologies
    remaining below TRL 7 at the milestone B decision and in technology
    development continuing into the product development phase. As a result, the
    Crusader program would not reach the low levels of risk that best practices
    show is needed for meeting product development cost and schedule
    commitments. Table 1 shows the results of our joint technology readiness
    assessment.

    5 We identified critical technologies as those needed to meet Crusader's key
    performance parameters.

    6 GAO has performed or is performing similar TRL assessments of the Airborne
    Laser, Comanche, Joint Strike Fighter, and Space-Based Infrared Satellite.

    Table 1: Results of the Joint Crusader TRL Assessmenta

                                     Assessment

        Key system elements             Critical Crusader Technologies     June 2001 April 2003

                                 1    Digitization (real time situational         5
                                                  awareness)
         System management       2            System prognostics                  5

                                 3    System diagnostics (including fault         5
                                                  detection)

                                 4   Cannon subsystem (including the tube,        7
                                               cooling system,
                                       laser ignition, Modular Artillery
                                                Charge System)
              Armament           5             Inductive fuzing                   6
                                 6            Automated loading                   7
                                 7        Projectile tracking system              7
       Resupply and ammunition          Automated inventory management
              handling           8          (includes recognition)                6

                                 9      Vehicle docking and transfer of           6
                                           projectiles, propellant,
                                                fuel and data

                                10    Common engine (with the Abrams tank         7
                                                   program)
                                11               Transmission                     5
              Mobility          12        Next generation suspension              7
                                13                  Track                         5
                                14              Drive by wire                     6
           Survivability        15        Integrated composite armor              7
                                16           Detection avoidance                  6

    aThe table represents a joint assessment of TRLs by GAO and the Crusader
    program office. The TRLs reflect the level of maturity of critical
    technologies at the time of the assessment in June 2001 and their expected
    level of maturity at the time of the Crusader milestone B decision in April
    2003.

    As shown in table 1, if technology develops as planned, eight critical
    technologies will be at a TRL 6 level of maturity and two will be at a TRL 5
    level of maturity at milestone B. While some technologies may embody some
    risk in meeting requirements, for the most part, the risk in the Crusader
    technologies involves the amount of time and effort needed to reach
    maturity. The planned technology maturity levels for the Crusader program at
    milestone B increase the probability that technical problems, if they occur,
    will need to be resolved in the higher cost environment of system
    development and demonstration. Confining delays in maturing technology to a
    time prior to the start of product development-in an environment where small
    teams of technologists work in laboratories and are dedicated to perfecting
    the technology-is critical to saving time and money. Conversely, if delays
    occur in product development when a large engineering force is in place to
    design and manufacture the product, delays would be much more costly. In
    fact, industry experts estimate that a delay

    during product development costs several times more than a similar delay
    that occurs before product development.

    Under the current Crusader acquisition plans, the critical technologies
    would be demonstrated in two steps after milestone B. Program officials are
    planning to demonstrate mobility component technologies first and then the
    remaining critical technologies. They recognize a risk in integrating the
    Crusader's mobility components-track, suspension, engine, and
    transmission-and plan to produce a mobility test rig to demonstrate that
    integration and to start accumulating reliability data on the mobility
    components. The mobility test rig would have the additional advantage of
    demonstrating the maturity of those technologies in an operational
    environment. The contractor is scheduled to deliver the mobility test rig in
    December 2003. The test rig would later be rebuilt as a Crusader prototype.

    The remaining critical technologies would not be demonstrated until after
    the contractor delivers the Crusader prototypes. The first Crusader system
    prototype is scheduled for delivery in October 2004 and is to enter testing
    the same month. Other prototypes would enter testing as they are delivered.
    The Army plans to award contracts for low-rate initial production long-lead
    items in March 2005-less than a fourth of the way through the
    prototype-testing schedule. This leaves little time in the Crusader's
    projected system development and demonstration schedule for solving
    unanticipated problems before the Army awards contracts for long-lead
    production items.

    The Army's approach to readying the Crusader for milestone B is to
    demonstrate progress toward achieving five of the system's requirements, two
    of which are key performance parameters7-the cannon rate of fire and the
    ability to resupply the self-propelled howitzer. For example, a Crusader key
    performance parameter is that the Crusader cannon be able to fire 10 to 12
    rounds per minute; however, the program only needs to demonstrate the
    ability to fire 6 rounds per minute before milestone B. The demonstrations,
    called exit criteria, were approved by both the Army and DOD. Among the
    demonstrations required by the exit criteria, only the cannon system is
    expected to be demonstrated in an operational

    7 A key performance parameter is a capability or characteristic that DOD
    believes is central to a system's performance.

    environment; the other critical technologies are expected to be demonstrated
    in a laboratory environment.

    Moreover, like many other DOD programs, the Crusader program is using risk
    management plans and engineering judgment, without the benefit of TRLs, to
    assess technological maturity and mitigate program risk. Risk management
    plans and engineering judgment are necessary to manage risk in any major
    development effort like the Crusader. However, we have found in our reviews
    that without an underpinning, such as TRLs, that allows transparency into
    program decisions, significant technical unknowns may be judged acceptable
    risks because a plan exists for resolving them. For example, we recently
    reported that while DOD judged the technical risks facing the Joint Strike
    Fighter as acceptable for starting product development, an analysis of TRLs
    showed that eight critical technologies were below TRL 7, with six
    technologies at TRL 4 or 5. When problems are encountered in resolving these
    unknowns, programs often fail to meet promised outcomes, as noted above with
    the Brilliant Anti-Armor Submunition program.

    Army Has Improved the Crusader's Software Development Management

    The Army has made improvements to its management of the software development
    process. Program officials stated that they would continue to aggressively
    manage the software development program to achieve and sustain the software
    process improvements.

    The automated Crusader system will be a software intensive program,
    projected to use about 1.9 million lines of code. Unlike any previous ground
    vehicle, all of the major functions of the Crusader are automated, including
    aiming, loading, and firing the cannon; managing inventory (projectiles and
    propellant); and resupplying the howitzer with ammunition and fuel. The crew
    compartment consists of a digital command center, with flat panel displays
    and re-configurable crew stations that give the crew real-time situation
    awareness, targeting information, integrated electronic technical manuals,
    decision aids, and diagnostic information.

    In 1998, the program began to experience software problems before meeting
    the software's preliminary design milestone. In June 1999, the Army decided
    that there were incomplete areas of the preliminary design and that the
    software team was not resolving design issues in a timely manner.
    Additionally, the software engineering team lacked disciplined quality
    assurance and configuration management practices, which led to some of the
    problems.

    In response, the program office tasked a software action team to identify
    problems and recommend improvements. The team drafted a recovery plan and
    recommended a number of process improvements for the prime contractor to
    implement. Program officials used the Software Development Capability
    Maturity ModelSM to define and determine the software development process
    maturity.8 The Software Engineering Institute, part of Carnegie Mellon
    University, developed the model to measure and rank an organization's
    software development and acquisition process. The contractor agreed to
    mature its software engineering processes to a level where the standard
    processes for software development, such as project and risk management, are
    documented and enforced across the organization. According to the Software
    Engineering Institute, increasing the maturity level of an organization's
    software engineering process puts the organization in better position to
    successfully develop software.

    As a result of these efforts, the Army and its prime contractor have made
    improvements to their management of the Crusader software engineering
    process. Improved areas include requirements generation and validation,
    quality assurance, configuration management, risk management, schedule and
    cost estimation, project tracking and control, and peer reviews of software
    engineering products such as design documents, code, and test plans. In
    addition, outside experts assisted in software analysis and design. Others
    were brought in to independently assess the software recovery plan. The
    contractor implemented a number of changes in the software design process,
    including the establishment of a common set of software development and
    management tools shared by all software teams and improved software testing.
    The program office has also revised the Crusader contract to provide the
    contractor monetary incentives to produce high-quality software on schedule.
    Software teams are also tracking progress and reporting it to management on
    a weekly or biweekly basis and have greatly improved their processes for
    estimating the size and schedule of the software. As a result of these
    improvements, the contractor has made more timely deliveries of software.

    8 Software Development Capability Maturity ModelSM is a service mark of
    Carnegie Mellon University and is registered in the U.S. Patent and
    Trademark Office.

    Army officials will need to continue their aggressive management approach
    because significant amounts of software remain to be developed before the
    Crusader is fully operational. Program officials stated that they would
    continue to manage the program to achieve and sustain the software process
    improvements.

    Lighter-weight Crusader May Not Significantly Improve Strategic
    Deployability

    The Army has made considerable progress over the past 2 years in redesigning
    the Crusader to substantially reduce its size and weight. In general, a
    lighter system offers a number of advantages, such as lower fuel consumption
    and easier transportation by truck and rail. However, it is uncertain that
    the requirement to deploy two Crusader howitzers on a C-17 aircraft provides
    a significant improvement in strategic deployability. Efforts to meet the
    deployability requirement will be a challenge and may require costly design
    changes and/or performance tradeoffs.

    According to an Army official, in October 1999, the Chief of Staff of the
    Army directed that the Crusader system become lighter and more deployable to
    better fit in with the Army's transformation to lighter forces. The Army
    subsequently revised the Crusader's Operational Requirements Documents to
    reflect new deployability requirements. Specifically, the documents state
    that

    * the Crusader vehicles must not exceed 42 tons at curb weight and 50 tons
    at combat weight;9

    * any combination of two Crusader vehicles, at curb weight, must be air
    transportable on both a C-5 and a C-17 aircraft;10 and

    9 The curb weight is the vehicle weight without a full load of fuel, with no
    ammunition or extra armor. The combat weight is the vehicle's weight fully
    loaded with fuel, ammunition, and armor kits. The reason for the curb weight
    limit is to allow any combination of the two Crusader vehicles to be flown
    in a C-5 aircraft to a desired range of 3,200 nautical miles and on a C-17
    with no specific range requirement.

    10 The 42-ton upper-limit on the Crusader weight is needed to accommodate
    the requirement for the C-17 to carry two Crusader vehicles. The C-17's
    maximum payload is about 85 tons. At that weight, the range of the C-17
    would be about 2,200 nautical miles.

    * both the C-5 and C-17 aircraft must be able to transport a single Crusader
    vehicle at combat weight.

    Crusader's Reduced Size and Weight May Not Provide a Significant Improvement
    in Deployability

    The main reason for the decision in January 2000 to restructure the program
    and redesign the Crusader weapon system was to reduce the system's weight
    and to improve its strategic deployability by air. However, the Army expects
    to rarely airlift the Crusader system-only during extreme emergencies-and
    that, in those circumstances, it would be likely that only small numbers of
    Crusader systems would be airlifted. Sealift would be the primary means of
    moving the Crusader system over long distances. In February 1999, the Army
    reported to Congress that the fielding of a lighter-weight Crusader would
    provide little in improved strategic deployability over a heavier version.11
    In May 2000, the DOD's Office of Program Analysis and Evaluation questioned
    the need to improve the Crusader's deployability, stating that it is unclear
    whether airlifting a small force of the heavier Crusaders, when needed,
    would be a severe burden on airlift.

    A limited Army analysis comparing the deployability by air of small numbers
    of the original heavier Crusader with that of the lighter-weight Crusader
    showed that the lighter-weight Crusader system might not significantly
    improve the system's strategic deployability. For example, this analysis
    showed that the lighter-weight Crusader system would reduce the number of
    sorties required to carry two Crusader systems and support equipment by 20
    percent-one aircraft sortie-over the system's original, heavier design. The
    study showed that it would take four C-17 sorties to airlift two of the
    lighter-weight Crusader systems and support equipment while it would take
    five sorties to airlift two of the original heavier systems and support
    equipment. In addition, the heavier Crusader howitzers and both resupply
    vehicles would arrive loaded for combat while the lighter Crusader howitzers
    and only one resupply vehicle would arrive loaded for combat. The other
    resupply vehicle would have to be manually loaded upon arrival.

    The recent analysis was done with inputs from various Army officials but has
    not been officially reviewed by the Air Force. Prior to our request, the

    11  Crusader, Advanced  Field  Artillery System,  A Report  to Congressional
    Defense Committees; U.S. Army; February 1999.

    Army had not formally analyzed the improvements in strategic deployability
    offered by a 40-ton Crusader over the earlier 60-ton Crusader.

    Army Faces Risks in Meeting Crusader's Deployability Requirement

    Meeting the requirement for carrying two Crusader howitzers on a C-17
    aircraft will be challenging. According to the Air Force, the C-17 aircraft
    is a more versatile aircraft and smaller than the C-5 aircraft. The C-5 is
    normally used for strategic deployments-into and out of the combat
    theater-while the C-17 aircraft can be used for both strategic deployments
    and tactical missions within a combat theater. According to Army and Air
    Force officials responsible for aircraft loading plans, the only possible
    way to load two Crusader howitzers on a C-17 aircraft would be back to back.
    However, they have concerns about this loading method. First, it will be a
    very tight fit with one howitzer's cannon barrel expected to be 20 inches
    from the forward bulkhead (on the edge of a crew safety zone) and the other
    howitzer's barrel expected to be within 3 inches of the stowed aft loading
    ramp. Second, according to an Air Force official, the 59 inches separating
    the two howitzers may not be enough room to properly restrain the vehicles
    with heavy chains.

    In October 2001, the Army performed a preliminary computer analysis of
    loading two Crusader howitzers on a C-17. It indicated that, if the vehicles
    dimensions remain the same through redesign, development, testing,
    production, and fielding, the two howitzers may fit. This analysis also
    showed that the loading plan would be a very tight fit and does not address
    the issue of restraining the howitzers during flight. Air Force officials
    have not reviewed this analysis. Army and Air Force officials told us that
    it is unlikely they will know if the Crusader can actually be loaded and
    carried until two lighter-weight prototypes are produced and tested in a
    C-17 aircraft.

    Army officials told us that, if carrying two Crusaders on a C-17 aircraft is
    not feasible, they will still accept the Crusader system because it is a
    much more capable system than the current self-propelled howitzer system,
    the Paladin. Program officials also told us that reducing the system's
    weight is

    desirable because it reduced  the logistics needed to support the system and
    improves, among other things, ground transportability and mobility.

    Cost and/or Performance Trade-offs May Be Needed to Meet Weight Requirement

    According to the DOD and the Army, achieving the Crusader's reduced weight
    requirement and meeting the 42-ton limit will be a difficult challenge and
    will require aggressive weight management to mitigate the risks involved
    with system weight. As of November 2001, the Crusader howitzer is projected
    to weigh 41.2 tons, which is close to the upper limit of the 42-ton curb
    weight requirement. This projection, however, is based on computer modeling
    that is still evolving. The projected weight could change considerably as
    specific components are fabricated and tested. Program office officials told
    us that, at this point in time, they have an 80-percent confidence level in
    the model's weight projection.

    The Army has already made significant changes to the Crusader system design
    to reduce the curb weights of the system's vehicles. The curb weight of the
    howitzer is expected to go from 60 tons to a projected weight of below 42
    tons. To achieve this weight reduction, the program office is redesigning
    the Crusader system by reducing the size and payload of the Crusader
    vehicles, substituting lighter weight materials for some components, and
    developing, with the Abrams tank program, a lighter weight engine.
    Additionally, the team plans to remove the heavy armor for top attack and
    road wheel protection and make it into kits that can be applied when needed
    in combat situations. To help reduce the overall weight of the Crusader
    system, the team decided to use a Palletized Load System truck carrying a
    newly designed resupply module as a second type of Crusader resupply
    vehicle-a wheeled resupply vehicle.

    Although the Army has not made vehicle weight a key performance parameter
    for the Crusader program, it has instituted an aggressive weight management
    program designed to mitigate the risks associated with maintaining the
    42-ton per vehicle weight limit. As part of the weight management program,
    the Army may have to consider the trade-offs between the system's weight and
    the program's cost, schedule, and performance requirements in order to
    achieve the required curb and combat weights. The program is also in the
    position of not being allowed any weight growth during development,
    production, fielding, and service. Before the Crusader redesign, the program
    had a 17-percent weight growth expectation for the Crusader vehicles.
    According to an Army official, if a new capability is added to the Crusader
    that increases its weight, the Army

    will  have  to find  a  way  to reduce  the  weight  of the  Crusader by  an
    equivalent amount.

    Apparent Overlap of Crusader and Future Combat Systems Programs Creates
    Uncertainties

    The Army's current schedule to begin fielding the Crusader system and its
    replacement, the Future Combat Systems, in the same fiscal year-2008-
    represents a potential risk of investing in duplicative systems to fulfill
    the same missions. However, at this time it is uncertain that the initial
    versions of the Future Combat Systems will have the capabilities to meet the
    Crusader's missions.

    The Future Combat Systems are expected to be revolutionary, lightweight
    weapon systems-20 tons or less-that involve manned and unmanned, ground and
    air systems, all of which would be digitally networked together. All the
    vehicles in the system are being designed for transport on a C-130 or
    similar aircraft-which are smaller aircraft than the C-17. Future Combat
    Systems vehicles may include command and control systems, reconnaissance
    systems, direct- and indirect-fire guns, rockets, and antitank missiles.

    The Future Combat Systems program is in an earlier stage of development than
    the Crusader-it is still in its initial 2-year concept design. Although the
    Future Combat Systems is a complex system of systems and the Army is still
    developing system concepts and technologies, the Army expects that the
    Future Combat Systems can be developed and produced in much shorter time
    frames than other weapons programs. Under the current Army schedule, the
    initial versions of the Future Combat Systems might enter the system
    development and demonstration phase as early as fiscal year 2003 and the
    first combat unit is scheduled to be equipped in 2008. Once fully fielded,
    the Future Combat Systems are intended to replace all of the Army's heavy
    weapon systems including the Crusader. Current Army plans show the Crusader
    to be in the force until 2032 or later.

    Because all the technologies needed for the Future Combat Systems may not be
    mature enough to be put into systems, the Army is planning to develop the
    initial version of the Future Combat Systems with less than its full
    capabilities and then upgrade it in a number of steps, called blocks, as the
    required technologies mature. The Army has not defined the capabilities that
    it can develop in the initial version of the Future Combat Systems, which it
    hopes will enter product development in 2003. As early as February 2002, the
    Army plans to award a contract to define these initial

    capabilities based on technologies that are mature enough to enter system
    development and demonstration in 2003.

    Eventually, the Army expects the Future Combat Systems to meet, using
    advanced technologies, the same artillery missions as the Crusader and
    eventually replace the Crusader system. While the final weapon technologies
    have not been selected for the Future Combat Systems, technologies that
    could provide the systems with capabilities to perform artillery missions
    similar to or greater than the Crusader include a multi-role armament
    system. This possible system could feature a 105-mm cannon that may have a
    non-line-of-sight capability out to a range of about 50 kilometers. Also,
    the Army is considering an advanced missile system that could be comprised
    of small-containerized missiles, known as NetFires, which are projected to
    have a range of 50 to 100 kilometers. A high-level Army official told us
    that he believes, based on recent technical briefings, that the initial
    version of the Future Combat Systems will not have the capabilities to meet
    the same artillery missions as the Crusader.

    Conclusions Moving into product development without demonstrating critical
    technologies in an operational environment increases the risk of cost
    overruns, schedule delays, and performance shortfalls. As currently planned,
    the majority of the critical Crusader technologies will have been
    demonstrated in a relevant environment but not the important operational
    environment. If the Crusader program follows the approach of moving into
    product development with less mature technologies, the program will need to
    continue to develop and demonstrate those technologies while concentrating
    on integrating subsystems into the system, testing at the subsystem and
    system levels, and preparing for production. As a result, technical
    problems, if they occur, will need to be resolved in the higher cost
    environment of system development and demonstration. On the other hand,
    demonstrating the critical technologies in an operational environment before
    entering system development and demonstration could necessitate more time
    and money than currently planned before the milestone B decision, but such
    investments would be relatively small compared to solving technical problems
    after the decision.

    The Army restructured the Crusader program to improve the system's strategic
    deployability by reducing the system's weight. The lighter-weight system,
    however, may not provide a significant improvement to strategic
    deployability. At this time, the Army is making design trade-offs to meet
    its weight requirement and it is not clear whether the Army can maintain its

    lighter weight goals throughout the development, production, and fielding of
    the Crusader system. Given the uncertainty, the Army risks making
    unnecessary cost, schedule, and performance trade-offs to meet deployability
    requirements that may not be clearly justified.

    The Army has not ruled out the possibility that it will field the Future
    Combat Systems with the ability to meet the same artillery mission as the
    Crusader in the same year the Crusader is fielded. However, the extent of
    this apparent overlap will not be clear until the potential capabilities and
    schedule of the initial version of the Future Combat Systems are determined.
    Therefore, it is important that the Army ensure that the projected
    capabilities and schedule for the initial Future Combat Systems are
    considered in the Crusader milestone decision.

    Recommendations for Executive Action

    To reduce the risk of schedule delays and increased costs in the product
    development phase of the Crusader program, we recommend that the secretary
    of defense direct the secretary of the army to dedicate the resources
    necessary to ensure that the critical Crusader technologies are
    demonstrated, at the component and subsystem level, in an operational
    environment before the program commits to product development at milestone
    B.

    To confirm the value and usefulness of the Crusader program's deployability
    requirement, we recommend that the secretary of defense direct the secretary
    of the army to conduct an analysis, before the decision to enter product
    development, to determine how important it is to deploy two Crusaders
    howitzers on a single C-17 aircraft. If it is important to the Army, we
    recommend that the secretary of defense direct the secretary of the army to
    establish, as a key performance parameter, the maximum per vehicle weight
    that would allow the C-17 aircraft to carry two Crusader howitzers. If the
    analysis determines that the redesigned Crusader does not significantly
    improve the system's military utility, we recommend that the secretary of
    defense direct the secretary of the army to reduce the priority placed on
    attaining the 42-ton weight limit.

    Finally, to ensure the Army does not invest in two weapon systems that will
    meet the same artillery missions at the same time, we recommend that the
    secretary of defense direct the secretary of the army to determine, based on
    available data, the potential capabilities and schedule of the initial
    version of the Future Combat Systems and the implication of those
    capabilities and schedule on the Crusader's utility to the Army before

    making  the  decision on  beginning  the Crusader's  system development  and
    demonstration-currently scheduled for April 2003.

                                   Agency Comments
                                 and Our Evaluation

    In written comments on a draft of this report, the director of strategic and
    tactical systems, within the Office of the Under Secretary of Defense for
    Acquisition, Technology, and Logistics, said that DOD did not agree with our
    recommendation that the Crusader technologies be demonstrated in an
    operational environment before the program commits to product development.
    DOD said that the Crusader program was a simulation-based acquisition
    program and, as such, evaluates system, component, and subsystem performance
    and technology readiness using modeling and simulation validated with test
    stands, integration laboratories, and subsystem prototypes. DOD questioned
    our definition of critical Crusader technologies and said that the track,
    for example, was selected by us as a critical technology and assessed as a
    TRL 5 despite the Army's many years of expertise in track development. DOD
    also said that the Crusader is currently demonstrating performance equal to
    or in excess of threshold requirements for the final system. Finally, DOD
    said that changing the Crusader's acquisition strategy to accommodate
    building system level prototypes required to demonstrate TRL 7 for all
    critical technologies would add significantly to the development time and
    expense without significantly reducing risk or improving performance. The
    full text of DOD's comments is included in appendix II.

    We agree that modeling and simulation is a key and accepted practice in any
    modern development program. However, we have found that programs need to
    demonstrate a high level of technology maturity before committing to product
    development. As shown by our past reviews, the best practice standard is
    that technology must be demonstrated, at the component or subsystem level,
    in an operational environment to be considered mature enough for entering
    product development. We believe that a program should use this best practice
    to assure success in meeting its cost and schedule goals.

    The determination of the critical Crusader technologies was a joint effort
    between the Crusader program office and us. We defined critical Crusader
    technologies as those required to meet the Crusader's key performance
    parameters and developed the initial critical technology list. Crusader
    program office engineers reviewed the initial list, suggested revisions, and
    agreed that the revised critical technologies list was complete and
    appropriate. Also, our analysts and program office engineers jointly arrived

    at the appropriate TRL for each critical technology. In addition, DOD's
    statement that track should not be a critical Crusader technology or should
    have been assessed at a higher TRL because of the Army's many years of
    expertise in track development underscores the value of the TRL methodology.
    Track was included as a critical Crusader technology because the Crusader
    cannot meet its mobility key performance parameter without track. The track
    was assessed at TRL 5 because the Crusader program was developing a new
    lighter-weight track. The Army plans to demonstrate it in an operational
    environment after milestone B. TRLs measure whether sufficient knowledge has
    been accumulated with respect to each application of a technology, not the
    development difficulty of the technology or whether the technology has been
    previously used in another application. The issue is not whether a
    technology like the newly developed track will ever work, but how much time
    and effort will be needed to demonstrate its maturity in this application.
    The Crusader system development and demonstration phase does not have much
    time between prototype testing and procurement of long-lead items for
    production to adjust for any delays or problems in prototype testing caused
    by technology problems. Such delays or problems could either delay the
    long-lead item procurement or reduce the amount of information available
    when committing to the procurement.

    DOD's assessment that the Crusader system is currently demonstrating
    performance equal to or in excess of threshold requirements for the final
    system is based mainly on modeling, simulations, and laboratory tests
    because the program has not produced the final system. As mentioned above,
    best practice calls for critical technologies to be demonstrated in an
    operational environment not in models, simulations, or laboratory
    environments before entering product development.

    DOD stated that building the full system prototype required to demonstrate
    TRL 7 would add significant time and expense to the program. However,
    demonstrating at TRL 7 does not require a full system prototype but only a
    prototype of the component or subsystem that contains a new technology. The
    demonstration can be accomplished by putting the new component or subsystem,
    such as an engine, on a surrogate vehicle; that is, a vehicle that already
    exists. The report's point is that using full system prototypes to
    demonstrate the maturity of critical technologies during the product
    development phase, as planned in the Crusader program, is potentially more
    costly than using component or subsystem prototypes to do so during the
    technology development phase. Problems that occur during required
    demonstrations may cause program delays in either phase, but as noted in

    the report, the delay is more expensive during the product development
    phase.

    DOD stated that it partially agreed with our recommendation to conduct an
    analysis to determine the importance of the deployability requirement and
    said that the current requirement is not considered a key performance
    parameter and, as a result, the Army is allowed to make trade-offs between
    the requirement and system cost and performance. DOD further stated that the
    Army plans to review the Crusader's requirements prior to the 2003 milestone
    B decision as required by regulations. We believe that an analysis to
    determine the importance of deploying two Crusader howitzers on a C-17
    aircraft should be conducted as soon as possible to provide the Army greater
    flexibility and knowledge in considering its ongoing trade-off decisions
    needed to meet weight requirements.

    DOD stated that it partially agreed with our recommendation to determine the
    potential capabilities and schedule of the initial version of the Future
    Combat Systems before making the decision to begin Crusader product
    development and stated that the Crusader's capabilities are intended to
    complement rather than compete with or be redundant to the capabilities of
    the Future Combat Systems. We continue to believe that DOD cannot determine
    whether the two systems will be complementary or redundant without knowledge
    of the initial Future Combat Systems capabilities and fielding schedule. DOD
    does not have this knowledge. We continue to believe that this knowledge
    needs to be considered as part of the decision to allow the Crusader program
    to enter product development. We have rewritten the recommendation to
    clarify its intent.

    Scope and To determine the readiness of the Crusader program to enter the
    system development and demonstration phase, we assessed, along with
    engineers

    Methodology from the Crusader Project Office, the current maturity of the
    critical Crusader technologies using the technology readiness level tool. We
    identified the Crusader technologies we believed were critical to meeting
    the Crusader system key performance parameters. Program engineers reviewed
    our list, suggested revisions, and agreed that the revised critical
    technologies list was complete and appropriate. After considering the
    program's plans for maturing the critical technologies before milestone B,
    we jointly determined the probable TRL levels of each of the critical
    technologies at the milestone. This determination assumed that the program
    office would successfully execute its existing plans for demonstrating some
    of the technologies before the milestone.

    To assess the status of the Crusader software development, we used project
    management criteria derived from Software Engineering Institute's Software
    Development Capability Maturity Model. We visited the Crusader prime
    contractor, met with Army and contractor officials, observed software
    development and test facilities, and examined project information. We also
    obtained and reviewed project documentation from the prime contractor and
    the Army program office.

    To assess the Crusader program's ability to meet the Crusader reduced weight
    requirements and improve the Crusader system's strategic deployability, we
    analyzed the Army's plans and requirements for reducing the weight of the
    Crusader and requested that the Army perform an analysis of the improvement
    in strategic deployability that the reduced weight Crusader system would
    provide compared to the original weight Crusader system. For this analysis,
    at our request, the Army determined the number of Crusader systems to be
    deployed, the other equipment and supplies that were required to be deployed
    with the Crusader systems, and the range of the aircraft used for the
    deployment. We reviewed the results of the Army's Crusader deployment
    analysis.

    To determine whether the Army is developing the Crusader and the Future
    Combat Systems to be fielded at the same time and to meet the same artillery
    missions, we analyzed and compared the Crusader and Future Combat Systems
    schedules and reviewed the Crusader system operational requirements
    documents. The Future Combat Systems do not have operational requirements
    documents at this stage of development. Also, we discussed with appropriate
    officials in the Army's Objective Force Task Force, the Army's artillery
    school, and the Crusader and the Future Combat Systems programs (1) the
    probability that the two programs would meet their individual schedules and
    (2) the potential technologies that might be used in the Future Combat
    Systems to provide it with artillery capabilities.

    In performing our work, we obtained documents and interviewed officials
    involved in the Crusader and the Future Combat Systems programs in the
    Office of the Deputy Chief of Staff for Operations, Washington, D.C.; U.S.
    Army Training and Doctrine Command, Fort Monroe, Virginia; U.S. Army Field
    Artillery School and Center, Fort Sill, Oklahoma; the Defense Advanced
    Research Projects Agency, Arlington, Virginia; the Military Traffic
    Management Command, Newport News, Virginia; the U.S. Air Force, Air Mobility
    Command, St. Louis, Missouri; the U.S. Air Force Aeronautical Systems
    Command, Dayton, Ohio; the Crusader Project Office, Picatinny

    Arsenal, New Jersey; and the prime contractor's Minneapolis, Minnesota,
    facility.

    We conducted our review between March 2001 and October 2001 in accordance
    with generally accepted government auditing standards.

    We also are sending copies of this report to the appropriate congressional
    committees; the director, Office of Management and Budget; and the
    secretaries of defense and the army. We will also provide copies to others
    upon request.

    If you or your staff have any questions concerning this report, please
    contact me at (202) 512-4841 or William R. Graveline at (256) 650-1414. Key
    contributors are listed in appendix III.

    James F. Wiggins
    Director
    Acquisition and Sourcing Management

    List of Committees:

    The Honorable Carl Levin Chairman The Honorable John W. Warner Ranking
    Minority Member Committee on Armed Services United States Senate

    The Honorable Daniel K. Inouye Chairman The Honorable Ted Stevens Ranking
    Minority Member Subcommittee on Defense Committee on Appropriations United
    States Senate

    The Honorable Bob Stump Chairman The Honorable Ike Skelton Ranking Minority
    Member Committee on Armed Services House of Representatives

    The Honorable Jerry Lewis Chairman The Honorable John P. Murtha Ranking
    Minority Member Subcommittee on Defense Committee on Appropriations House of
    Representatives

    Appendix I

    Technology Readiness Levels and Their Descriptions

                       Technology readiness level Description

    1.  Basic  principles  observed and  reported.  Lowest  level of  technology
    readiness.  Scientific research  begins to  be translated  into technology's
    basic properties.

    2.  Technology  concept  and/or  application Invention  begins.  Once  basic
    principles are observed, practical applications can be

    formulated. invented.  The application is speculative  and there is no proof
    or detailed analysis to  support the assumptions. Examples are still limited
    to paper studies.

    3. Analytical and experimental critical functions and/or characteristic
    proof of concept.

    Active research and development is initiated. This includes analytical
    studies and laboratory studies to physically validate analytical predictions
    of separate elements of the technology. Examples include components that are
    not yet integrated or representative.

    4. Component and/or breadboard validation in laboratory environment.

    Basic technological  components are integrated to  establish that the pieces
    will  work  together. This  is  relatively  "low fidelity"  compared to  the
    eventual  system. Examples  include integration  of "ad  hoc" hardware  in a
    laboratory.

    5. Component and /or breadboard validation Fidelity of breadboard technology
    increases significantly. The basic technological

    in relevant environment. components are integrated with reasonably realistic
    supporting elements so that the technology can be tested in a simulated
    environment. Examples include "high fidelity" laboratory integration of
    components.

    6.  System/subsystem model  or prototype  Representative model  or prototype
    system, which is well beyond the breadboard tested

    demonstration  in  a relevant  environment.  for  level 5,  is  tested in  a
    relevant   environment.  Represents   a  major   step  up   in  technology's
    demonstrated  readiness.  Examples include  testing  a prototype  in a  high
    fidelity laboratory environment or in a simulated operational environment.

    7.  System  prototype  demonstration  in an  Prototype  near  or at  planned
    operational system. Represents a major step up from level

    operational environment. 6, requiring  the demonstration of an actual system
    prototype  in  an  operational  environment. Examples  include  testing  the
    prototype in a test bed aircraft.

    8. Actual system completed and qualified Technology has been proven to work
    in its final form and under expected conditions. In

    through test and demonstration. almost all cases, this level represents the
    end of true system development. Examples include developmental test and
    evaluation of the system in its intended weapon system to determine if it
    meets design specifications.

    9. Actual system proven  through successful Actual application of technology
    in its final form and under mission conditions, such as

    mission operations.  those encountered  in operational test  and evaluation.
    Examples include using the system under operational mission conditions.

    Source: Appendix 6 of DOD Regulation 5000.2R, "Mandatory Procedures for
    Major Defense Acquisition Programs (MDAPS) and Major Automated Information
    System (MAIS) Acquisition Programs," June 10, 2001.

                                    Appendix II

                       Comments from the Department of Defense

    Appendix II
    Comments from the Department of Defense

    Appendix II
    Comments from the Department of Defense

    Appendix III

                       GAO Contacts and Staff Acknowledgments

    GAO Contacts James F. Wiggins, (202) 512-4841 William R. Graveline, (256)
    650-1414

    Staff In addition to those named above, the following individuals made
    significant contributions to this report: Robert L. Ackley; Nabajyoti

    Acknowledgments Barkakati; Paul L. Francis; Lawrence D. Gaston, Jr.; Matthew
    B. Lea; Gary L. Middleton; Madhav S. Panwar; Robert J. Stolba; and John P.
    Swain.

    GAO's Mission The General Accounting Office, the investigative arm of
    Congress, exists to support Congress in meeting its constitutional
    responsibilities and to help improve the performance and accountability of
    the federal government for the American people. GAO examines the use of
    public funds; evaluates federal programs and policies; and provides
    analyses, recommendations, and other assistance to help Congress make
    informed oversight, policy, and funding decisions. GAO's commitment to good
    government is reflected in its core values of accountability, integrity, and
    reliability.

    Obtaining Copies of GAO Reports and Testimony

    The fastest and easiest way to obtain copies of GAO documents is through the
    Internet. GAO's Web site (www.gao.gov) contains abstracts and full-text
    files of current reports and testimony and an expanding archive of older
    products. The Web site features a search engine to help you locate documents
    using key words and phrases. You can print these documents in their
    entirety, including charts and other graphics.

    Each day, GAO issues a list of newly released reports, testimony, and
    correspondence. GAO posts this list, known as "Today's Reports," on its Web
    site daily. The list contains links to the full-text document files. To have
    GAO E-mail this list to you every afternoon, go to www.gao.gov and select
    "Subscribe to daily e-mail alert for newly released products" under the GAO
    Reports heading.

    Order by Mail or Phone The first copy of each printed report is free.
    Additional copies are $2 each. A check or money order should be made out to
    the Superintendent of Documents. GAO also accepts VISA and Mastercard.
    Orders for 100 or more copies mailed to a single address are discounted 25
    percent. Orders should be sent to:

    U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C.
    20548

    To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
    512-6061

    Visit GAO's Document GAO Building

    Room 1100, 700 4th  Street, NW (corner of 4th and G Streets, NW)Distribution
    Center Washington, D.C. 20013

    To Report Fraud, Contact:

    Web site: www.gao.gov/fraudnet/fraudnet.htm,Waste, and Abuse in E-mail:
    fraudnet@gao.gov, orFederal Programs 1-800-424-5454 or (202) 512-7470
    (automated answering system).

    Public Affairs Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202)
    512-4800 U.S. General Accounting Office, 441 G. Street NW, Room 7149,
    Washington, D.C. 20548

                                 Presorted Standard
                                 Postage & Fees Paid
                                         GAO
                                   Permit No. GI00

    United States
    General Accounting Office
    Washington, D.C. 20548-0001

    Official Business
    Penalty for Private Use $300

    Address Correction Requested
    *** End of document. ***