Defense Acquisitions: Steps to Improve the Crusader Program's Investment Decisions

Defense Acquisitions: Steps to Improve the Crusader Program's
Investment Decisions (25-FEB-02, GAO-02-201).

The Army wants an artillery system with greater firepower, range,
and mobility than its current self-propelled howitzer. In 1994,
the Army began to develop the Crusader, an advanced artillery
system consisting of a self-propelled 155-millimeter howitzer and
a resupply vehicle. The Department of Defense (DOD) will decide
next year whether the Crusader program should enter its system
development and demonstration stage, which will require the
commitment of major resources. GAO found that the Crusader
program has made considerable progress in developing key
technologies and reducing its size and weight. However, more
progress and knowledge is needed to minimize the risk of cost
overruns, schedule delays, and performance shortfalls. The
Crusader program will likely enter product development with most
of its critical technologies less mature than best practices
recommend. Most of the Crusader's critical technologies have been
demonstrated in a relevant environment but not in the more
demanding operational environment. Although the Army is reducing
the Crusader's weight so that two vehicles can be deployed on a
C-17 aircraft, the deployability advantage gained does not appear
significant. The reduction in the Crusader system's weight would
only decrease the number of C-17 flights needed to transport two
complete systems and support equipment from five to four flights.
A lighter system offers several other benefits, and knowing the
magnitude of the deployability advantage of reduced weight would
allow the Army to make better decisions on trade offs. An
apparent overlap exists between the Crusader's and the Future
Combat Systems' capabilities and schedules. The Army expects the
Future Combat Systems to meet the same artillery missions as the
Crusader and eventually replace it. The current schedules for
initial fielding of the Future Combat Systems and the Crusader
system occur in the same year, 2008. The extent of this apparent
overlap depends more on the Future Combat Systems than the
Crusader because less is known about the Future Combat Systems'
technologies.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-02-201
ACCNO: A02801
TITLE: Defense Acquisitions: Steps to Improve the Crusader
Program's Investment Decisions
DATE: 02/25/2002
SUBJECT: Defense cost control
Developmental testing
Military cost control
Performance measures
Schedule slippages
Strategic mobility forces
Systems development life cycle
Weapons research and development
Weapons systems
Army Brilliant Anti-Armor Submunition
Program

Army Crusader System
Army Future Combat Systems
C-17 Aircraft
C-5 Aircraft
Crusader Resupply Vehicle
Crusader Self Propelled Howitzer
Desert Storm
Galaxy Aircraft
Globemaster Aircraft
Joint Strike Fighter
Paladin Howitzer
SEI Software Acquisition Capability
Maturity Model

******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-02-201

United States General Accounting Office

GAO Report to Congressional Committees

February 2002

DEFENSE ACQUISITIONS

Steps to Improve the Crusader Program's Investment Decisions

GAO-02-201

Contents

Letter

Results in Brief
Background
Critical Technologies Need Additional Maturity to Better

Assure Low-risk Product Development Lighter-weight Crusader May Not
Significantly Improve Strategic Deployability Apparent Overlap of Crusader
and Future Combat Systems

Programs Creates Uncertainties Conclusions Recommendations for Executive
Action Agency Comments and Our Evaluation Scope and Methodology

1 1 4

20 21 22 23 25

Appendixes

Appendix I: Technology Readiness Levels and Their Descriptions 29 Appendix
II: Comments from the Department of Defense 30 Appendix III: GAO Contacts
and Staff Acknowledgments 33

Table Table 1: Results of the Joint Crusader TRL Assessment

Figures Figure 1: Crusader Howitzer 5 Figure 2: Crusader Tracked Resupply
Vehicle 6 Figure 3: Crusader Wheeled Resupply Vehicle 7

United States General Accounting Office Washington, D.C. 20548

February 25, 2002

Congressional Committees

To address future threats, the Army has identified a requirement for an
artillery system that has greater firepower, range, and mobility than its
current self-propelled howitzer-the Paladin. Operation Desert Storm
demonstrated that current howitzers were unable to keep up with our tanks
and fighting vehicles. In 1994, the Army began to develop the Crusader, an
advanced artillery system consisting of a self-propelled 155-millimeter
howitzer and a resupply vehicle. The Army's total acquisition cost in the
Crusader program is projected to be about $11 billion. In 2000, the Army
changed its requirements and restructured the Crusader program to make the
system lighter and more deployable. This change was in response to the
Army's planned transformation to a future force, which will also be lighter
and more deployable. The Army expects to use the Crusader until it is
eventually replaced by the main component of the future force, known as the
Future Combat Systems.

In April 2003, Department of Defense (DOD) will decide whether the Crusader
program should enter its system development and demonstration-or product
development-stage, which will require the commitment of major resources to
develop and design the Crusader system and to demonstrate its integration,
interoperability, and utility. The opportunity to take actions that can put
the program in a better position to succeed and, thus, minimize future cost
and schedule increases is now, before the start of product development. As
the Army approaches that decision point, we examined three major aspects of
the program: (1) the progress in developing Crusader's technology and
software, (2) the Crusader's requirement for improved deployability, and (3)
the Army's timetables for developing the Crusader and the Future Combat
Systems.

Results in Brief The Crusader program has made considerable progress in
developing key technologies and reducing its size and weight. However, with
a 2003 decision date for committing to product development, more progress
and knowledge will be needed to minimize risks of cost overruns, schedule
delays, and performance shortfalls.

* Based on current Army plans, the Crusader program will likely enter
product development with the majority of its critical technologies less
mature than best practices recommend. Most of the Crusader's critical

technologies have been demonstrated in a relevant environment but not the
more demanding operational environment. Achieving the higher level of
technology maturity prior to beginning product development reduces the risk
of costly schedule delays. Crusader technologies that have not reached the
desired level of maturity for product development include the suspension,
track, transmission, and prognostics.1 The Army made significant
improvements to the management of the Crusader's software development
process in response to software design problems experienced in 1998. The
Army's continued attention to software development is essential given the
large amount that remains to be completed.

* Although the Army is redesigning the Crusader to reduce individual vehicle
weight from about 60 tons to about 40 tons so that two vehicles can be
deployed on a C-17 aircraft, the deployability advantage gained does not
appear significant. An Army analysis conducted at our request shows that the
reduction in the Crusader system's weight would only decrease the number of
C-17 flights needed to transport two complete systems and support equipment
from five flights to four flights. Moreover, the Army plans to move
Crusaders by aircraft only under extraordinary conditions and in limited
numbers. Also, as currently designed, the weight of two howitzers is
projected to be very close to the C-17's weight limit and their projected
size would make them a very tight fit in the aircraft, if they fit at all.
The Army may need to make cost, schedule, and performance trade-offs to meet
and maintain that weight. While, in general, a lighter system offers a
number of other benefits, knowing the magnitude of the deployability
advantage gained by reducing weight would enable the Army to make better
trade-off decisions.

* An apparent overlap exists between the Crusader's and the Future Combat
Systems' capabilities and schedules. The Army expects the Future Combat
Systems to eventually meet the same artillery missions as the Crusader and
eventually replace it. The current schedules for initial fielding of the
Future Combat Systems and the Crusader system occur in the same year, 2008.
The extent of this apparent overlap depends more on the Future Combat
Systems than the Crusader, because less is known about the Future Combat
Systems' technologies.

1 Prognostics is a system to forecast potential failures in subsystems,
allowing the maintainers to correct them before they fail.

More will be understood about these technologies and, thus, the Future
Combat Systems' schedule, when the Army formally assesses their maturity in
early 2003. Current plans call for committing to the Crusader's product
development in the same year the Future Combat Systems' technologies are
assessed.

We are recommending that the Army further mature the Crusader's technologies
before committing to product development; assess the benefits of its weight
reduction relative to its strategic deployability; and assess the projected
capabilities and fielding schedules for Future Combat Systems as part of the
Crusader's milestone decision for beginning product development.

In commenting on our report, DOD did not agree with our recommendation on
maturing critical Crusader technologies and partially agreed with our
recommendations on Crusader's deployability requirement and the Crusader's
apparent overlap with the Future Combat Systems. In not agreeing, DOD noted
that the Crusader program is using modeling and simulation to determine the
Crusader's readiness to enter product development and stated that changing
its acquisition strategy to further mature critical technologies would add
significantly to the development time and expense without significantly
reducing risk or improving performance. We agree that modeling and
simulation are useful management tools, but believe that demonstrating
critical technologies in an operational environment before the start of
product development has been shown to lower program risks of significant
cost overruns, schedule delays, and performance shortfalls. In partially
agreeing with our recommendation to conduct an analysis of the Crusader's
deployability requirement, DOD said that the current requirement is not
considered a key performance parameter and, as a result, the Army is allowed
to make trade-offs between the requirement and system cost and performance.
DOD further stated that the Army plans to review the Crusader's requirements
prior to the 2003 milestone B decision as required by regulations. We
believe that an analysis to determine the importance of deploying two
Crusader howitzers on a C-17 aircraft should be conducted as soon as
possible to provide the Army greater flexibility and knowledge in
considering its ongoing trade-off decisions needed to meet weight
requirements. In partially agreeing with our recommendation to determine the
potential capabilities and schedule of the initial version of the Future
Combat Systems before making the decision to begin Crusader product
development, DOD said that the Crusader's capabilities are intended to
complement rather than be redundant to the capabilities of the Future

Combat Systems. We continue to believe that DOD cannot determine whether the
two systems will be complementary or redundant without knowledge of the
initial Future Combat Systems capabilities and fielding schedule. DOD does
not yet have this knowledge. We also continue to believe that this knowledge
needs to be considered as part of the decision to allow the Crusader program
to enter product development.

Background The Army plans to invest about $11 billion developing and
procuring the Crusader, an automated, next generation field artillery
system. To date, the program has spent about $1.7 billion in development
costs. It plans to procure 482 Crusader systems-each system consisting of a
self-propelled 155-millimeter howitzer and a resupply vehicle. The Army is
developing 2 different resupply vehicles-1 with tracks and 1 with wheels-and
plans to procure 241 of each type. The purpose of the Crusader system is to
overcome threats from enemy artillery and reconnaissance or surveillance
systems as well as have the mobility needed to keep up with Army tanks and
fighting vehicles. Figure 1 shows the planned Crusader howitzer, figure 2
the planned tracked resupply vehicle, and figure 3 the planned wheeled
resupply vehicle.

Figure 1: Crusader Howitzer

Source: United Defense Limited Partnership.

Figure 2: Crusader Tracked Resupply Vehicle

Source: United Defense Limited Partnership.

Figure 3: Crusader Wheeled Resupply Vehicle

Source: United Defense Limited Partnership.

The Army restructured the Crusader program in January 2000 to align
Crusader's design with the Army's transformation to a lighter force. The
Army's transformation will affect all aspects of Army organization,
training, doctrine, leadership, and strategic plans as well as the types of
equipment and technology the Army acquires. The Army expects the
transformation to be at least a 30-year process and has not estimated its
full cost. The centerpiece of the lighter, more deployable future force is
the Future Combat Systems. The Future Combat Systems concept is a system of
ground and air, manned and unmanned weapon systems, each under 20 tons that
is planned to replace most, if not all, of the Army's ground combat systems
without a loss in lethality and survivability. Artillery systems are among
those to be replaced.

The Army expects the Crusader system to fill the existing gap in artillery
capabilities until it is replaced by the Future Combat Systems. In keeping
with the transformation philosophy of lightweight vehicles and ease of
deployability, the Army is redesigning Crusader to make it lighter and more
deployable, with the goal of reducing the weight of the self-propelled
howitzer and tracked resupply vehicle from about 60 tons to about 40 tons
each. Program officials said that a lighter system would enhance operational
flexibility in employing Crusader in support of any operation.

The Crusader is currently in the program definition and risk reduction phase
of its development program. In April 2003, the program is scheduled for a
milestone B review to determine whether it is ready to enter its system
development and demonstration phase. Milestone B is the point at which DOD
decides whether to commit major resources to develop and design the system
and to demonstrate its integration, interoperability, and utility. The
milestone marks the start of the program's product development. The Army
plans to deliver the first full Crusader prototype system in October 2004,
followed by a low-rate initial production decision in February 2006, and
initial system fielding in April 2008.

Critical Technologies Need Additional Maturity to Better Assure Low-risk
Product Development

Based on current Army plans, the Army will begin the Crusader's product
development in April 2003 but before maturing critical Crusader technologies
to a level considered low risk relative to best practices. These risks
relate less to whether these technologies can be matured, but more to how
much time and cost it will take to mature them. If, after starting product
development, the Crusader technologies do not mature on schedule and instead
cause delays, the Army may spend more and take longer to develop, produce,
and field the Crusader system. Crusader performance goals may also be at
risk. On the other hand, the Army has made improvements to the management of
the Crusader software development process.

Assessing Technology The maturity of a program's technologies at the start
of product Readiness Provides development is a good predictor of that
program's future performance. Our Opportunities to Improve past reviews of
programs incorporating technologies into new products

and weapon systems showed that they were more likely to meet product

Outcomes objectives when the technologies were matured before product
development started. For example, the Ford Motor Company's practice of
demonstrating new technologies in driving conditions before they are

included in a new product is essential to ensuring that the new product can
be developed on time and within budget. Similarly, we have found that the
early demonstration of propulsion and water-planing technologies, essential
to the performance of the Marine Corps' Advance Amphibious Assault Vehicle,
has been instrumental to that program's staying within 15 percent of cost
and schedule estimates.

Conversely, cost, schedule, and performance problems were more likely to
occur when programs started with technologies at lower readiness levels.2
For example, the enabling technologies for the Army's Brilliant Anti-Armor
Submunition program were very immature at the start of the program, and
their delays became major contributors to the program's subsequent
88-percent cost growth and 62-percent schedule slippage. Separating
technology development from product development into two distinct program
phases is a best practice of both successful commercial and defense
programs. This entails demonstrating all critical technologies at the
component or subsystem level in an operational environment during technology
development, prior to committing major funding to product development. Under
this practice, the critical technologies would be demonstrated in component
or subsystem prototypes that are nearly the right size, weight, and
configuration needed for the intended product. Such demonstrations need not
require a full system prototype of a Crusader vehicle, but can be done using
surrogate vehicles.

Technology readiness levels (TRL) are a good way to gauge the maturity of
technologies. TRLs were pioneered by the National Aeronautics and Space
Administration to determine the readiness of technologies to be incorporated
into products such as weapon systems. Readiness levels are measured along a
scale of one to nine, starting with paper studies of the basic concept,
proceeding with laboratory demonstrations, and ending with a technology that
has proven itself on the intended product. TRLs are based on actual
demonstrations of how well specific technologies perform in the intended
application. For example, a technology that has been demonstrated in an
operational environment using subsystem prototype hardware (such as a
complete cannon system) that is at or near the final

2 Best Practices: Successful Application to Weapon Acquisitions Requires
Changes in DOD's Environment (GAO/NSIAD-98-56, Feb. 24, 1998) and Best
Practices: Better Management of Technology Development Can Improve Weapon
System Outcomes

(GAO/NSIAD-99-162, July 30, 1999).

system design would be rated as a TRL 7. The individual TRL descriptions can
be found in appendix I.

DOD has agreed that technology readiness assessments are important and
necessary in assisting officials who decide when and where to insert new
technologies into weapon system programs. In January 2001, DOD issued a new
acquisition instruction that redefined the phases in the defense acquisition
cycle and emphasized the role of technology development in the acquisition
process.3 Under the instruction, programs use the concept and technology
development phase, which precedes the system development and demonstration
phase, for developing components and subsystems that must be demonstrated
before integration into the system. The first portion of system development
and demonstration phase is dedicated to integrating the components and
subsystems into the system. The instruction states that DOD prefers that
technology be demonstrated in an operational environment but must be
demonstrated in a relevant environment to be considered mature enough for
product development in the system development and demonstration phase.
According to the TRL descriptions, technology demonstrated in an operational
environment is TRL 7 and technology demonstrated in a relevant environment
is TRL 6.

Maturing technology from a TRL 6 to a TRL 7 represents a major step up in
maturity. A technology at the TRL 6 maturity level needs only to be
demonstrated as a subsystem prototype or model in a laboratory or simulated
operational environment. A technology at the TRL 7 maturity level must be
demonstrated as a subsystem prototype at or near the size of the required
subsystem outside the laboratory in an actual operational environment. For
example, operating a prototype engine on a laboratory test stand that
simulates the effects of the vehicle's weight on the engine would be a TRL 6
level demonstration while operating an engine in a surrogate vehicle or
actual prototype that weighed 50 tons, on roads and cross country, would be
a TRL 7 demonstration.

In June 2001, DOD issued a new acquisition regulation.4 It stated that
technology maturity is a principal element of program risk and directed
technology readiness assessments for critical technologies sufficiently

3 DOD Instruction 5000.2 "Operation of the Defense Acquisition System," Jan.
4, 2001.

4 DOD Regulation 5000.2R "Mandatory Procedures for Major Defense Acquisition
Programs (MDAPS) and Major Automated Information System (MAIS) Acquisition
Programs," June 10, 2001.

prior to selected milestone decision points-including milestone B--to
provide useful technology maturity information to the acquisition review
process. Although the new regulation recognizes that TRLs enable consistent,
uniform discussions of technical maturity across different types of
technologies and provides the definitions of TRLs used in this report, it
permits the use of TRLs or "some equivalent assessment" when performing a
technology readiness assessment.

Technology Maturation Will Continue into Product Development

In June 2001, Crusader program office engineers and we assessed the maturity
of 16 critical Crusader technologies using TRLs.5,6 This joint assessment
determined that 10 of the 16 critical Crusader technologies were below TRL
7. Since the Crusader program is not scheduled to commit to product
development until April 2003, the Army still has time to mature the 10
critical technologies to a TRL 7 level-demonstrate them in a component or
subsystem prototype in an operational environment. However, the Army's
Crusader plans will result in 10 of the critical Crusader technologies
remaining below TRL 7 at the milestone B decision and in technology
development continuing into the product development phase. As a result, the
Crusader program would not reach the low levels of risk that best practices
show is needed for meeting product development cost and schedule
commitments. Table 1 shows the results of our joint technology readiness
assessment.

5 We identified critical technologies as those needed to meet Crusader's key
performance parameters.

6 GAO has performed or is performing similar TRL assessments of the Airborne
Laser, Comanche, Joint Strike Fighter, and Space-Based Infrared Satellite.

Table 1: Results of the Joint Crusader TRL Assessmenta

Assessment

Key system elements Critical Crusader Technologies June 2001 April 2003

1 Digitization (real time situational 5
awareness)
System management 2 System prognostics 5

3 System diagnostics (including fault 5
detection)

4 Cannon subsystem (including the tube, 7
cooling system,
laser ignition, Modular Artillery
Charge System)
Armament 5 Inductive fuzing 6
6 Automated loading 7
7 Projectile tracking system 7
Resupply and ammunition Automated inventory management
handling 8 (includes recognition) 6

9 Vehicle docking and transfer of 6
projectiles, propellant,
fuel and data

10 Common engine (with the Abrams tank 7
program)
11 Transmission 5
Mobility 12 Next generation suspension 7
13 Track 5
14 Drive by wire 6
Survivability 15 Integrated composite armor 7
16 Detection avoidance 6

aThe table represents a joint assessment of TRLs by GAO and the Crusader
program office. The TRLs reflect the level of maturity of critical
technologies at the time of the assessment in June 2001 and their expected
level of maturity at the time of the Crusader milestone B decision in April
2003.

As shown in table 1, if technology develops as planned, eight critical
technologies will be at a TRL 6 level of maturity and two will be at a TRL 5
level of maturity at milestone B. While some technologies may embody some
risk in meeting requirements, for the most part, the risk in the Crusader
technologies involves the amount of time and effort needed to reach
maturity. The planned technology maturity levels for the Crusader program at
milestone B increase the probability that technical problems, if they occur,
will need to be resolved in the higher cost environment of system
development and demonstration. Confining delays in maturing technology to a
time prior to the start of product development-in an environment where small
teams of technologists work in laboratories and are dedicated to perfecting
the technology-is critical to saving time and money. Conversely, if delays
occur in product development when a large engineering force is in place to
design and manufacture the product, delays would be much more costly. In
fact, industry experts estimate that a delay

during product development costs several times more than a similar delay
that occurs before product development.

Under the current Crusader acquisition plans, the critical technologies
would be demonstrated in two steps after milestone B. Program officials are
planning to demonstrate mobility component technologies first and then the
remaining critical technologies. They recognize a risk in integrating the
Crusader's mobility components-track, suspension, engine, and
transmission-and plan to produce a mobility test rig to demonstrate that
integration and to start accumulating reliability data on the mobility
components. The mobility test rig would have the additional advantage of
demonstrating the maturity of those technologies in an operational
environment. The contractor is scheduled to deliver the mobility test rig in
December 2003. The test rig would later be rebuilt as a Crusader prototype.

The remaining critical technologies would not be demonstrated until after
the contractor delivers the Crusader prototypes. The first Crusader system
prototype is scheduled for delivery in October 2004 and is to enter testing
the same month. Other prototypes would enter testing as they are delivered.
The Army plans to award contracts for low-rate initial production long-lead
items in March 2005-less than a fourth of the way through the
prototype-testing schedule. This leaves little time in the Crusader's
projected system development and demonstration schedule for solving
unanticipated problems before the Army awards contracts for long-lead
production items.

The Army's approach to readying the Crusader for milestone B is to
demonstrate progress toward achieving five of the system's requirements, two
of which are key performance parameters7-the cannon rate of fire and the
ability to resupply the self-propelled howitzer. For example, a Crusader key
performance parameter is that the Crusader cannon be able to fire 10 to 12
rounds per minute; however, the program only needs to demonstrate the
ability to fire 6 rounds per minute before milestone B. The demonstrations,
called exit criteria, were approved by both the Army and DOD. Among the
demonstrations required by the exit criteria, only the cannon system is
expected to be demonstrated in an operational

7 A key performance parameter is a capability or characteristic that DOD
believes is central to a system's performance.

environment; the other critical technologies are expected to be demonstrated
in a laboratory environment.

Moreover, like many other DOD programs, the Crusader program is using risk
management plans and engineering judgment, without the benefit of TRLs, to
assess technological maturity and mitigate program risk. Risk management
plans and engineering judgment are necessary to manage risk in any major
development effort like the Crusader. However, we have found in our reviews
that without an underpinning, such as TRLs, that allows transparency into
program decisions, significant technical unknowns may be judged acceptable
risks because a plan exists for resolving them. For example, we recently
reported that while DOD judged the technical risks facing the Joint Strike
Fighter as acceptable for starting product development, an analysis of TRLs
showed that eight critical technologies were below TRL 7, with six
technologies at TRL 4 or 5. When problems are encountered in resolving these
unknowns, programs often fail to meet promised outcomes, as noted above with
the Brilliant Anti-Armor Submunition program.

Army Has Improved the Crusader's Software Development Management

The Army has made improvements to its management of the software development
process. Program officials stated that they would continue to aggressively
manage the software development program to achieve and sustain the software
process improvements.

The automated Crusader system will be a software intensive program,
projected to use about 1.9 million lines of code. Unlike any previous ground
vehicle, all of the major functions of the Crusader are automated, including
aiming, loading, and firing the cannon; managing inventory (projectiles and
propellant); and resupplying the howitzer with ammunition and fuel. The crew
compartment consists of a digital command center, with flat panel displays
and re-configurable crew stations that give the crew real-time situation
awareness, targeting information, integrated electronic technical manuals,
decision aids, and diagnostic information.

In 1998, the program began to experience software problems before meeting
the software's preliminary design milestone. In June 1999, the Army decided
that there were incomplete areas of the preliminary design and that the
software team was not resolving design issues in a timely manner.
Additionally, the software engineering team lacked disciplined quality
assurance and configuration management practices, which led to some of the
problems.

In response, the program office tasked a software action team to identify
problems and recommend improvements. The team drafted a recovery plan and
recommended a number of process improvements for the prime contractor to
implement. Program officials used the Software Development Capability
Maturity ModelSM to define and determine the software development process
maturity.8 The Software Engineering Institute, part of Carnegie Mellon
University, developed the model to measure and rank an organization's
software development and acquisition process. The contractor agreed to
mature its software engineering processes to a level where the standard
processes for software development, such as project and risk management, are
documented and enforced across the organization. According to the Software
Engineering Institute, increasing the maturity level of an organization's
software engineering process puts the organization in better position to
successfully develop software.

As a result of these efforts, the Army and its prime contractor have made
improvements to their management of the Crusader software engineering
process. Improved areas include requirements generation and validation,
quality assurance, configuration management, risk management, schedule and
cost estimation, project tracking and control, and peer reviews of software
engineering products such as design documents, code, and test plans. In
addition, outside experts assisted in software analysis and design. Others
were brought in to independently assess the software recovery plan. The
contractor implemented a number of changes in the software design process,
including the establishment of a common set of software development and
management tools shared by all software teams and improved software testing.
The program office has also revised the Crusader contract to provide the
contractor monetary incentives to produce high-quality software on schedule.
Software teams are also tracking progress and reporting it to management on
a weekly or biweekly basis and have greatly improved their processes for
estimating the size and schedule of the software. As a result of these
improvements, the contractor has made more timely deliveries of software.

8 Software Development Capability Maturity ModelSM is a service mark of
Carnegie Mellon University and is registered in the U.S. Patent and
Trademark Office.

Army officials will need to continue their aggressive management approach
because significant amounts of software remain to be developed before the
Crusader is fully operational. Program officials stated that they would
continue to manage the program to achieve and sustain the software process
improvements.

Lighter-weight Crusader May Not Significantly Improve Strategic
Deployability

The Army has made considerable progress over the past 2 years in redesigning
the Crusader to substantially reduce its size and weight. In general, a
lighter system offers a number of advantages, such as lower fuel consumption
and easier transportation by truck and rail. However, it is uncertain that
the requirement to deploy two Crusader howitzers on a C-17 aircraft provides
a significant improvement in strategic deployability. Efforts to meet the
deployability requirement will be a challenge and may require costly design
changes and/or performance tradeoffs.

According to an Army official, in October 1999, the Chief of Staff of the
Army directed that the Crusader system become lighter and more deployable to
better fit in with the Army's transformation to lighter forces. The Army
subsequently revised the Crusader's Operational Requirements Documents to
reflect new deployability requirements. Specifically, the documents state
that

* the Crusader vehicles must not exceed 42 tons at curb weight and 50 tons
at combat weight;9

* any combination of two Crusader vehicles, at curb weight, must be air
transportable on both a C-5 and a C-17 aircraft;10 and

9 The curb weight is the vehicle weight without a full load of fuel, with no
ammunition or extra armor. The combat weight is the vehicle's weight fully
loaded with fuel, ammunition, and armor kits. The reason for the curb weight
limit is to allow any combination of the two Crusader vehicles to be flown
in a C-5 aircraft to a desired range of 3,200 nautical miles and on a C-17
with no specific range requirement.

10 The 42-ton upper-limit on the Crusader weight is needed to accommodate
the requirement for the C-17 to carry two Crusader vehicles. The C-17's
maximum payload is about 85 tons. At that weight, the range of the C-17
would be about 2,200 nautical miles.

* both the C-5 and C-17 aircraft must be able to transport a single Crusader
vehicle at combat weight.

Crusader's Reduced Size and Weight May Not Provide a Significant Improvement
in Deployability

The main reason for the decision in January 2000 to restructure the program
and redesign the Crusader weapon system was to reduce the system's weight
and to improve its strategic deployability by air. However, the Army expects
to rarely airlift the Crusader system-only during extreme emergencies-and
that, in those circumstances, it would be likely that only small numbers of
Crusader systems would be airlifted. Sealift would be the primary means of
moving the Crusader system over long distances. In February 1999, the Army
reported to Congress that the fielding of a lighter-weight Crusader would
provide little in improved strategic deployability over a heavier version.11
In May 2000, the DOD's Office of Program Analysis and Evaluation questioned
the need to improve the Crusader's deployability, stating that it is unclear
whether airlifting a small force of the heavier Crusaders, when needed,
would be a severe burden on airlift.

A limited Army analysis comparing the deployability by air of small numbers
of the original heavier Crusader with that of the lighter-weight Crusader
showed that the lighter-weight Crusader system might not significantly
improve the system's strategic deployability. For example, this analysis
showed that the lighter-weight Crusader system would reduce the number of
sorties required to carry two Crusader systems and support equipment by 20
percent-one aircraft sortie-over the system's original, heavier design. The
study showed that it would take four C-17 sorties to airlift two of the
lighter-weight Crusader systems and support equipment while it would take
five sorties to airlift two of the original heavier systems and support
equipment. In addition, the heavier Crusader howitzers and both resupply
vehicles would arrive loaded for combat while the lighter Crusader howitzers
and only one resupply vehicle would arrive loaded for combat. The other
resupply vehicle would have to be manually loaded upon arrival.

The recent analysis was done with inputs from various Army officials but has
not been officially reviewed by the Air Force. Prior to our request, the

11 Crusader, Advanced Field Artillery System, A Report to Congressional
Defense Committees; U.S. Army; February 1999.

Army had not formally analyzed the improvements in strategic deployability
offered by a 40-ton Crusader over the earlier 60-ton Crusader.

Army Faces Risks in Meeting Crusader's Deployability Requirement

Meeting the requirement for carrying two Crusader howitzers on a C-17
aircraft will be challenging. According to the Air Force, the C-17 aircraft
is a more versatile aircraft and smaller than the C-5 aircraft. The C-5 is
normally used for strategic deployments-into and out of the combat
theater-while the C-17 aircraft can be used for both strategic deployments
and tactical missions within a combat theater. According to Army and Air
Force officials responsible for aircraft loading plans, the only possible
way to load two Crusader howitzers on a C-17 aircraft would be back to back.
However, they have concerns about this loading method. First, it will be a
very tight fit with one howitzer's cannon barrel expected to be 20 inches
from the forward bulkhead (on the edge of a crew safety zone) and the other
howitzer's barrel expected to be within 3 inches of the stowed aft loading
ramp. Second, according to an Air Force official, the 59 inches separating
the two howitzers may not be enough room to properly restrain the vehicles
with heavy chains.

In October 2001, the Army performed a preliminary computer analysis of
loading two Crusader howitzers on a C-17. It indicated that, if the vehicles
dimensions remain the same through redesign, development, testing,
production, and fielding, the two howitzers may fit. This analysis also
showed that the loading plan would be a very tight fit and does not address
the issue of restraining the howitzers during flight. Air Force officials
have not reviewed this analysis. Army and Air Force officials told us that
it is unlikely they will know if the Crusader can actually be loaded and
carried until two lighter-weight prototypes are produced and tested in a
C-17 aircraft.

Army officials told us that, if carrying two Crusaders on a C-17 aircraft is
not feasible, they will still accept the Crusader system because it is a
much more capable system than the current self-propelled howitzer system,
the Paladin. Program officials also told us that reducing the system's
weight is

desirable because it reduced the logistics needed to support the system and
improves, among other things, ground transportability and mobility.

Cost and/or Performance Trade-offs May Be Needed to Meet Weight Requirement

According to the DOD and the Army, achieving the Crusader's reduced weight
requirement and meeting the 42-ton limit will be a difficult challenge and
will require aggressive weight management to mitigate the risks involved
with system weight. As of November 2001, the Crusader howitzer is projected
to weigh 41.2 tons, which is close to the upper limit of the 42-ton curb
weight requirement. This projection, however, is based on computer modeling
that is still evolving. The projected weight could change considerably as
specific components are fabricated and tested. Program office officials told
us that, at this point in time, they have an 80-percent confidence level in
the model's weight projection.

The Army has already made significant changes to the Crusader system design
to reduce the curb weights of the system's vehicles. The curb weight of the
howitzer is expected to go from 60 tons to a projected weight of below 42
tons. To achieve this weight reduction, the program office is redesigning
the Crusader system by reducing the size and payload of the Crusader
vehicles, substituting lighter weight materials for some components, and
developing, with the Abrams tank program, a lighter weight engine.
Additionally, the team plans to remove the heavy armor for top attack and
road wheel protection and make it into kits that can be applied when needed
in combat situations. To help reduce the overall weight of the Crusader
system, the team decided to use a Palletized Load System truck carrying a
newly designed resupply module as a second type of Crusader resupply
vehicle-a wheeled resupply vehicle.

Although the Army has not made vehicle weight a key performance parameter
for the Crusader program, it has instituted an aggressive weight management
program designed to mitigate the risks associated with maintaining the
42-ton per vehicle weight limit. As part of the weight management program,
the Army may have to consider the trade-offs between the system's weight and
the program's cost, schedule, and performance requirements in order to
achieve the required curb and combat weights. The program is also in the
position of not being allowed any weight growth during development,
production, fielding, and service. Before the Crusader redesign, the program
had a 17-percent weight growth expectation for the Crusader vehicles.
According to an Army official, if a new capability is added to the Crusader
that increases its weight, the Army

will have to find a way to reduce the weight of the Crusader by an
equivalent amount.

Apparent Overlap of Crusader and Future Combat Systems Programs Creates
Uncertainties

The Army's current schedule to begin fielding the Crusader system and its
replacement, the Future Combat Systems, in the same fiscal year-2008-
represents a potential risk of investing in duplicative systems to fulfill
the same missions. However, at this time it is uncertain that the initial
versions of the Future Combat Systems will have the capabilities to meet the
Crusader's missions.

The Future Combat Systems are expected to be revolutionary, lightweight
weapon systems-20 tons or less-that involve manned and unmanned, ground and
air systems, all of which would be digitally networked together. All the
vehicles in the system are being designed for transport on a C-130 or
similar aircraft-which are smaller aircraft than the C-17. Future Combat
Systems vehicles may include command and control systems, reconnaissance
systems, direct- and indirect-fire guns, rockets, and antitank missiles.

The Future Combat Systems program is in an earlier stage of development than
the Crusader-it is still in its initial 2-year concept design. Although the
Future Combat Systems is a complex system of systems and the Army is still
developing system concepts and technologies, the Army expects that the
Future Combat Systems can be developed and produced in much shorter time
frames than other weapons programs. Under the current Army schedule, the
initial versions of the Future Combat Systems might enter the system
development and demonstration phase as early as fiscal year 2003 and the
first combat unit is scheduled to be equipped in 2008. Once fully fielded,
the Future Combat Systems are intended to replace all of the Army's heavy
weapon systems including the Crusader. Current Army plans show the Crusader
to be in the force until 2032 or later.

Because all the technologies needed for the Future Combat Systems may not be
mature enough to be put into systems, the Army is planning to develop the
initial version of the Future Combat Systems with less than its full
capabilities and then upgrade it in a number of steps, called blocks, as the
required technologies mature. The Army has not defined the capabilities that
it can develop in the initial version of the Future Combat Systems, which it
hopes will enter product development in 2003. As early as February 2002, the
Army plans to award a contract to define these initial

capabilities based on technologies that are mature enough to enter system
development and demonstration in 2003.

Eventually, the Army expects the Future Combat Systems to meet, using
advanced technologies, the same artillery missions as the Crusader and
eventually replace the Crusader system. While the final weapon technologies
have not been selected for the Future Combat Systems, technologies that
could provide the systems with capabilities to perform artillery missions
similar to or greater than the Crusader include a multi-role armament
system. This possible system could feature a 105-mm cannon that may have a
non-line-of-sight capability out to a range of about 50 kilometers. Also,
the Army is considering an advanced missile system that could be comprised
of small-containerized missiles, known as NetFires, which are projected to
have a range of 50 to 100 kilometers. A high-level Army official told us
that he believes, based on recent technical briefings, that the initial
version of the Future Combat Systems will not have the capabilities to meet
the same artillery missions as the Crusader.

Conclusions Moving into product development without demonstrating critical
technologies in an operational environment increases the risk of cost
overruns, schedule delays, and performance shortfalls. As currently planned,
the majority of the critical Crusader technologies will have been
demonstrated in a relevant environment but not the important operational
environment. If the Crusader program follows the approach of moving into
product development with less mature technologies, the program will need to
continue to develop and demonstrate those technologies while concentrating
on integrating subsystems into the system, testing at the subsystem and
system levels, and preparing for production. As a result, technical
problems, if they occur, will need to be resolved in the higher cost
environment of system development and demonstration. On the other hand,
demonstrating the critical technologies in an operational environment before
entering system development and demonstration could necessitate more time
and money than currently planned before the milestone B decision, but such
investments would be relatively small compared to solving technical problems
after the decision.

The Army restructured the Crusader program to improve the system's strategic
deployability by reducing the system's weight. The lighter-weight system,
however, may not provide a significant improvement to strategic
deployability. At this time, the Army is making design trade-offs to meet
its weight requirement and it is not clear whether the Army can maintain its

lighter weight goals throughout the development, production, and fielding of
the Crusader system. Given the uncertainty, the Army risks making
unnecessary cost, schedule, and performance trade-offs to meet deployability
requirements that may not be clearly justified.

The Army has not ruled out the possibility that it will field the Future
Combat Systems with the ability to meet the same artillery mission as the
Crusader in the same year the Crusader is fielded. However, the extent of
this apparent overlap will not be clear until the potential capabilities and
schedule of the initial version of the Future Combat Systems are determined.
Therefore, it is important that the Army ensure that the projected
capabilities and schedule for the initial Future Combat Systems are
considered in the Crusader milestone decision.

Recommendations for Executive Action

To reduce the risk of schedule delays and increased costs in the product
development phase of the Crusader program, we recommend that the secretary
of defense direct the secretary of the army to dedicate the resources
necessary to ensure that the critical Crusader technologies are
demonstrated, at the component and subsystem level, in an operational
environment before the program commits to product development at milestone
B.

To confirm the value and usefulness of the Crusader program's deployability
requirement, we recommend that the secretary of defense direct the secretary
of the army to conduct an analysis, before the decision to enter product
development, to determine how important it is to deploy two Crusaders
howitzers on a single C-17 aircraft. If it is important to the Army, we
recommend that the secretary of defense direct the secretary of the army to
establish, as a key performance parameter, the maximum per vehicle weight
that would allow the C-17 aircraft to carry two Crusader howitzers. If the
analysis determines that the redesigned Crusader does not significantly
improve the system's military utility, we recommend that the secretary of
defense direct the secretary of the army to reduce the priority placed on
attaining the 42-ton weight limit.

Finally, to ensure the Army does not invest in two weapon systems that will
meet the same artillery missions at the same time, we recommend that the
secretary of defense direct the secretary of the army to determine, based on
available data, the potential capabilities and schedule of the initial
version of the Future Combat Systems and the implication of those
capabilities and schedule on the Crusader's utility to the Army before

making the decision on beginning the Crusader's system development and
demonstration-currently scheduled for April 2003.

Agency Comments
and Our Evaluation

In written comments on a draft of this report, the director of strategic and
tactical systems, within the Office of the Under Secretary of Defense for
Acquisition, Technology, and Logistics, said that DOD did not agree with our
recommendation that the Crusader technologies be demonstrated in an
operational environment before the program commits to product development.
DOD said that the Crusader program was a simulation-based acquisition
program and, as such, evaluates system, component, and subsystem performance
and technology readiness using modeling and simulation validated with test
stands, integration laboratories, and subsystem prototypes. DOD questioned
our definition of critical Crusader technologies and said that the track,
for example, was selected by us as a critical technology and assessed as a
TRL 5 despite the Army's many years of expertise in track development. DOD
also said that the Crusader is currently demonstrating performance equal to
or in excess of threshold requirements for the final system. Finally, DOD
said that changing the Crusader's acquisition strategy to accommodate
building system level prototypes required to demonstrate TRL 7 for all
critical technologies would add significantly to the development time and
expense without significantly reducing risk or improving performance. The
full text of DOD's comments is included in appendix II.

We agree that modeling and simulation is a key and accepted practice in any
modern development program. However, we have found that programs need to
demonstrate a high level of technology maturity before committing to product
development. As shown by our past reviews, the best practice standard is
that technology must be demonstrated, at the component or subsystem level,
in an operational environment to be considered mature enough for entering
product development. We believe that a program should use this best practice
to assure success in meeting its cost and schedule goals.

The determination of the critical Crusader technologies was a joint effort
between the Crusader program office and us. We defined critical Crusader
technologies as those required to meet the Crusader's key performance
parameters and developed the initial critical technology list. Crusader
program office engineers reviewed the initial list, suggested revisions, and
agreed that the revised critical technologies list was complete and
appropriate. Also, our analysts and program office engineers jointly arrived

at the appropriate TRL for each critical technology. In addition, DOD's
statement that track should not be a critical Crusader technology or should
have been assessed at a higher TRL because of the Army's many years of
expertise in track development underscores the value of the TRL methodology.
Track was included as a critical Crusader technology because the Crusader
cannot meet its mobility key performance parameter without track. The track
was assessed at TRL 5 because the Crusader program was developing a new
lighter-weight track. The Army plans to demonstrate it in an operational
environment after milestone B. TRLs measure whether sufficient knowledge has
been accumulated with respect to each application of a technology, not the
development difficulty of the technology or whether the technology has been
previously used in another application. The issue is not whether a
technology like the newly developed track will ever work, but how much time
and effort will be needed to demonstrate its maturity in this application.
The Crusader system development and demonstration phase does not have much
time between prototype testing and procurement of long-lead items for
production to adjust for any delays or problems in prototype testing caused
by technology problems. Such delays or problems could either delay the
long-lead item procurement or reduce the amount of information available
when committing to the procurement.

DOD's assessment that the Crusader system is currently demonstrating
performance equal to or in excess of threshold requirements for the final
system is based mainly on modeling, simulations, and laboratory tests
because the program has not produced the final system. As mentioned above,
best practice calls for critical technologies to be demonstrated in an
operational environment not in models, simulations, or laboratory
environments before entering product development.

DOD stated that building the full system prototype required to demonstrate
TRL 7 would add significant time and expense to the program. However,
demonstrating at TRL 7 does not require a full system prototype but only a
prototype of the component or subsystem that contains a new technology. The
demonstration can be accomplished by putting the new component or subsystem,
such as an engine, on a surrogate vehicle; that is, a vehicle that already
exists. The report's point is that using full system prototypes to
demonstrate the maturity of critical technologies during the product
development phase, as planned in the Crusader program, is potentially more
costly than using component or subsystem prototypes to do so during the
technology development phase. Problems that occur during required
demonstrations may cause program delays in either phase, but as noted in

the report, the delay is more expensive during the product development
phase.

DOD stated that it partially agreed with our recommendation to conduct an
analysis to determine the importance of the deployability requirement and
said that the current requirement is not considered a key performance
parameter and, as a result, the Army is allowed to make trade-offs between
the requirement and system cost and performance. DOD further stated that the
Army plans to review the Crusader's requirements prior to the 2003 milestone
B decision as required by regulations. We believe that an analysis to
determine the importance of deploying two Crusader howitzers on a C-17
aircraft should be conducted as soon as possible to provide the Army greater
flexibility and knowledge in considering its ongoing trade-off decisions
needed to meet weight requirements.

DOD stated that it partially agreed with our recommendation to determine the
potential capabilities and schedule of the initial version of the Future
Combat Systems before making the decision to begin Crusader product
development and stated that the Crusader's capabilities are intended to
complement rather than compete with or be redundant to the capabilities of
the Future Combat Systems. We continue to believe that DOD cannot determine
whether the two systems will be complementary or redundant without knowledge
of the initial Future Combat Systems capabilities and fielding schedule. DOD
does not have this knowledge. We continue to believe that this knowledge
needs to be considered as part of the decision to allow the Crusader program
to enter product development. We have rewritten the recommendation to
clarify its intent.

Scope and To determine the readiness of the Crusader program to enter the
system development and demonstration phase, we assessed, along with
engineers

Methodology from the Crusader Project Office, the current maturity of the
critical Crusader technologies using the technology readiness level tool. We
identified the Crusader technologies we believed were critical to meeting
the Crusader system key performance parameters. Program engineers reviewed
our list, suggested revisions, and agreed that the revised critical
technologies list was complete and appropriate. After considering the
program's plans for maturing the critical technologies before milestone B,
we jointly determined the probable TRL levels of each of the critical
technologies at the milestone. This determination assumed that the program
office would successfully execute its existing plans for demonstrating some
of the technologies before the milestone.

To assess the status of the Crusader software development, we used project
management criteria derived from Software Engineering Institute's Software
Development Capability Maturity Model. We visited the Crusader prime
contractor, met with Army and contractor officials, observed software
development and test facilities, and examined project information. We also
obtained and reviewed project documentation from the prime contractor and
the Army program office.

To assess the Crusader program's ability to meet the Crusader reduced weight
requirements and improve the Crusader system's strategic deployability, we
analyzed the Army's plans and requirements for reducing the weight of the
Crusader and requested that the Army perform an analysis of the improvement
in strategic deployability that the reduced weight Crusader system would
provide compared to the original weight Crusader system. For this analysis,
at our request, the Army determined the number of Crusader systems to be
deployed, the other equipment and supplies that were required to be deployed
with the Crusader systems, and the range of the aircraft used for the
deployment. We reviewed the results of the Army's Crusader deployment
analysis.

To determine whether the Army is developing the Crusader and the Future
Combat Systems to be fielded at the same time and to meet the same artillery
missions, we analyzed and compared the Crusader and Future Combat Systems
schedules and reviewed the Crusader system operational requirements
documents. The Future Combat Systems do not have operational requirements
documents at this stage of development. Also, we discussed with appropriate
officials in the Army's Objective Force Task Force, the Army's artillery
school, and the Crusader and the Future Combat Systems programs (1) the
probability that the two programs would meet their individual schedules and
(2) the potential technologies that might be used in the Future Combat
Systems to provide it with artillery capabilities.

In performing our work, we obtained documents and interviewed officials
involved in the Crusader and the Future Combat Systems programs in the
Office of the Deputy Chief of Staff for Operations, Washington, D.C.; U.S.
Army Training and Doctrine Command, Fort Monroe, Virginia; U.S. Army Field
Artillery School and Center, Fort Sill, Oklahoma; the Defense Advanced
Research Projects Agency, Arlington, Virginia; the Military Traffic
Management Command, Newport News, Virginia; the U.S. Air Force, Air Mobility
Command, St. Louis, Missouri; the U.S. Air Force Aeronautical Systems
Command, Dayton, Ohio; the Crusader Project Office, Picatinny

Arsenal, New Jersey; and the prime contractor's Minneapolis, Minnesota,
facility.

We conducted our review between March 2001 and October 2001 in accordance
with generally accepted government auditing standards.

We also are sending copies of this report to the appropriate congressional
committees; the director, Office of Management and Budget; and the
secretaries of defense and the army. We will also provide copies to others
upon request.

If you or your staff have any questions concerning this report, please
contact me at (202) 512-4841 or William R. Graveline at (256) 650-1414. Key
contributors are listed in appendix III.

James F. Wiggins
Director
Acquisition and Sourcing Management

List of Committees:

The Honorable Carl Levin Chairman The Honorable John W. Warner Ranking
Minority Member Committee on Armed Services United States Senate

The Honorable Daniel K. Inouye Chairman The Honorable Ted Stevens Ranking
Minority Member Subcommittee on Defense Committee on Appropriations United
States Senate

The Honorable Bob Stump Chairman The Honorable Ike Skelton Ranking Minority
Member Committee on Armed Services House of Representatives

The Honorable Jerry Lewis Chairman The Honorable John P. Murtha Ranking
Minority Member Subcommittee on Defense Committee on Appropriations House of
Representatives

Appendix I

Technology Readiness Levels and Their Descriptions

Technology readiness level Description

1. Basic principles observed and reported. Lowest level of technology
readiness. Scientific research begins to be translated into technology's
basic properties.

2. Technology concept and/or application Invention begins. Once basic
principles are observed, practical applications can be

formulated. invented. The application is speculative and there is no proof
or detailed analysis to support the assumptions. Examples are still limited
to paper studies.

3. Analytical and experimental critical functions and/or characteristic
proof of concept.

Active research and development is initiated. This includes analytical
studies and laboratory studies to physically validate analytical predictions
of separate elements of the technology. Examples include components that are
not yet integrated or representative.

4. Component and/or breadboard validation in laboratory environment.

Basic technological components are integrated to establish that the pieces
will work together. This is relatively "low fidelity" compared to the
eventual system. Examples include integration of "ad hoc" hardware in a
laboratory.

5. Component and /or breadboard validation Fidelity of breadboard technology
increases significantly. The basic technological

in relevant environment. components are integrated with reasonably realistic
supporting elements so that the technology can be tested in a simulated
environment. Examples include "high fidelity" laboratory integration of
components.

6. System/subsystem model or prototype Representative model or prototype
system, which is well beyond the breadboard tested

demonstration in a relevant environment. for level 5, is tested in a
relevant environment. Represents a major step up in technology's
demonstrated readiness. Examples include testing a prototype in a high
fidelity laboratory environment or in a simulated operational environment.

7. System prototype demonstration in an Prototype near or at planned
operational system. Represents a major step up from level

operational environment. 6, requiring the demonstration of an actual system
prototype in an operational environment. Examples include testing the
prototype in a test bed aircraft.

8. Actual system completed and qualified Technology has been proven to work
in its final form and under expected conditions. In

through test and demonstration. almost all cases, this level represents the
end of true system development. Examples include developmental test and
evaluation of the system in its intended weapon system to determine if it
meets design specifications.

9. Actual system proven through successful Actual application of technology
in its final form and under mission conditions, such as

mission operations. those encountered in operational test and evaluation.
Examples include using the system under operational mission conditions.

Source: Appendix 6 of DOD Regulation 5000.2R, "Mandatory Procedures for
Major Defense Acquisition Programs (MDAPS) and Major Automated Information
System (MAIS) Acquisition Programs," June 10, 2001.

Appendix II

Comments from the Department of Defense

Appendix II
Comments from the Department of Defense

Appendix III

GAO Contacts and Staff Acknowledgments

GAO Contacts James F. Wiggins, (202) 512-4841 William R. Graveline, (256)
650-1414

Staff In addition to those named above, the following individuals made
significant contributions to this report: Robert L. Ackley; Nabajyoti

Acknowledgments Barkakati; Paul L. Francis; Lawrence D. Gaston, Jr.; Matthew
B. Lea; Gary L. Middleton; Madhav S. Panwar; Robert J. Stolba; and John P.
Swain.

GAO's Mission The General Accounting Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability of
the federal government for the American people. GAO examines the use of
public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to good
government is reflected in its core values of accountability, integrity, and
reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents is through the
Internet. GAO's Web site (www.gao.gov) contains abstracts and full-text
files of current reports and testimony and an expanding archive of older
products. The Web site features a search engine to help you locate documents
using key words and phrases. You can print these documents in their
entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its Web
site daily. The list contains links to the full-text document files. To have
GAO E-mail this list to you every afternoon, go to www.gao.gov and select
"Subscribe to daily e-mail alert for newly released products" under the GAO
Reports heading.

Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out to
the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:

U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C.
20548

To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061

Visit GAO's Document GAO Building

Room 1100, 700 4th Street, NW (corner of 4th and G Streets, NW)Distribution
Center Washington, D.C. 20013

To Report Fraud, Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm,Waste, and Abuse in E-mail:
fraudnet@gao.gov, orFederal Programs 1-800-424-5454 or (202) 512-7470
(automated answering system).

Public Affairs Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202)
512-4800 U.S. General Accounting Office, 441 G. Street NW, Room 7149,
Washington, D.C. 20548

Presorted Standard
Postage & Fees Paid
GAO
Permit No. GI00

United States
General Accounting Office
Washington, D.C. 20548-0001

Official Business
Penalty for Private Use $300

Address Correction Requested
*** End of document. ***