Defense Acquisitions: Assessments of Major Weapon Programs	 
(15-MAY-03, GAO-03-476).					 
                                                                 
The weapons the Department of Defense (DOD) develops have no	 
rival in superiority. How they are developed can be improved,	 
without sacrificing the superiority of the outcome. GAO's reviews
over the past 20 years have found consistent problems with weapon
investments--cost increases, schedule delays and performance	 
shortfalls--along with underlying causes, such as pressure on	 
managers to promise more than they can deliver. The best	 
practices of successful product developments offer a		 
knowledge-based approach DOD can use to improve the way it	 
develops new weapons. This report is new for GAO, and draws on	 
its work in best practices for product development. GAO's goal	 
for this report is to provide congressional and DOD decision	 
makers with an independent, knowledge-based assessment of defense
programs that identifies potential risks, and offers an 	 
opportunity for action when a program's projected attainment of  
knowledge diverges from the best practice. It can also highlight 
those programs that employ practices worthy of emulation by other
programs. GAO plans to update and issue this report annually to  
the congressional defense committees.				 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-03-476 					        
    ACCNO:   A06848						        
  TITLE:     Defense Acquisitions: Assessments of Major Weapon	      
Programs							 
     DATE:   05/15/2003 
  SUBJECT:   Best practices					 
	     Program evaluation 				 
	     Strategic planning 				 
	     Weapons						 
	     Weapons research and development			 
	     Weapons systems					 
	     Defense operations 				 
	     Army Future Combat Systems 			 
	     F-22 Aircraft					 
	     Joint Strike Fighter				 
	     National Polar-Orbiting Operational		 
	     Environmental Satellite System			 
                                                                 
	     Patriot Missile Advanced				 
	     Capability-Three Upgrade				 
                                                                 
	     SDI Theater High Altitude Area Defense		 
	     System						 
                                                                 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-03-476

                                       A

Report to Congressional Committees

May 2003 DEFENSE ACQUISITIONS Assessments of Major Weapon Programs

GAO- 03- 476

Foreword iii Letter 1

Appendixes A Knowledge- Based Approach Can Lead to Better Acquisition

Outcomes 2 Observations 5 Agency Comments 7 Scope of Our Review 7

Appendix I: Assessments of Individual Programs 9 Introduction 9

Advanced Amphibious Assault Vehicle (AAAV) 15 Airborne Laser (ABL) 17
Advanced Extremely High Frequency (AEHF) Communications Satellite 19

AN/ APG- 79 Active Electronically Scanned Array (AESA) Radar 21 AIM- 9X
Short- Range Air- to- Air Missile 23 Advanced Threat Infrared
Countermeasures/ Common Missile Warning System (ATIRCM/ CMWS) 25

Advanced Wideband Satellite (AWS) 27 Cooperative Engagement Capability
(CEC) 29 CH- 47F Improved Cargo Helicopter 31 RAH- 66 Comanche 33 EX- 171
Extended Range Guided Munition (ERGM) 35 Excalibur Artillery Round 37 F/
A- 18E/ F Super Hornet 39 F/ A- 22 Raptor 41 Joint Air- to- Surface
Standoff Missile (JASSM) 43 Joint Common Missile 45 Joint Primary Aircraft
Training System (JPATS) 47 F- 35 Joint Strike Fighter (JSF) 49 Joint
Standoff Weapon (JSOW) 51 National Polar- orbiting Operational
Environmental Satellite System (NPOESS) 53

Patriot Advanced Capability 3 (PAC- 3) Program 55 Space Based Infrared
System (SBIRS) High 57 Theater High Altitude Area Defense (THAAD) 59
Tactical Tomahawk Missile 61 V- 22 Osprey 63 Wideband Gapfiller Satellite
(WGS) Communications System 65

Appendix II: Methodology 67 System Profile Assessment 67 Product Knowledge
Assessment 68

Appendix III: GAO Contact and Acknowledgments 71 Figures Figure 1:
Knowledge Build at Key Points in Product Development

Reduces the Risk of Unknowns 4 Figure 2: Graphic Depiction of Best
Practices for Technology, Design, and Production Knowledge 10

Figure 3: Depiction of a Notional Weapon System Program*s Knowledge as
Compared with Best Practices 12

Abbreviations

DOD Department of Defense GAO General Accounting Office USAF United States
Air Force USN United States Navy USMC United States Marine Corps

This is a work of the U. S. Government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. It may contain
copyrighted graphics, images or other materials. Permission from the
copyright holder may be necessary should you wish to reproduce copyrighted
materials separately from GAO*s product.

Congressional Committees systems. May 15, 2003 Foreword

Recent military operations in Iraq have soundly demonstrated the
superiority of United States military capabilities. The Department of
Defense (DOD) develops weaponry that is unmatched in levels of
technological sophistication and lethality. Despite their superiority,
weapon systems routinely take much longer to field, cost more to buy, and
require more support than investment plans provide for. In a constrained
funding environment, unforeseen cost growth in weapon systems forecloses
other investment choices for the government, both within and outside of
DOD. DOD*s investment in major weapon systems is expected to grow
considerably in the future as DOD works to keep legacy systems while
investing in future capabilities such as unmanned aircraft, satellite

networks, and information communication systems. For example, the
investment in weapons from fiscal years 2003 through 2009 will exceed $1
trillion. Such an investment clearly requires DOD to be as efficient and

effective as possible in the development and acquisition of weapon In the
last several years, we have undertaken a body of work that examines weapon
acquisition issues from a different, more cross- cutting perspective* one
that draws lessons learned from best product development practices to see
if they apply to weapon system development. We found that programs managed
with a knowledge- based approach*

where product knowledge is demonstrated at critical points in a
development cycle* place themselves on a low- risk path to production.
These programs are more likely to be executed within cost and schedule

estimates. We believe that by employing this approach, DOD can still field
superior weapons without attendant cost and schedule growth.

This report is a new product for GAO. It provides decision makers with a
snapshot of program performance and risk and is focused on each system*s
developmental progress vis a vis best practices. Each assessment is

summarized in an easy to read, visually descriptive 2- page format that
provides a fact- based analysis of each program*s cost, schedule, and
development status. We plan to issue this report annually in early spring,
and we intend to increase the number of systems reviewed each year. We
have briefed numerous committee staff on the product and received

positive feedback regarding the report*s utility and breath of coverage.

The continuing war on terrorism, regional instability, the challenge of
transforming the military, as well as the federal government*s short- and
long- term budget pressures have created a challenging environment for
DOD. It faces a number of difficult missions that will put its strategies
and resources under enormous strain. Consequently, it is important that

weapon systems be acquired using a knowledge- based approach to ensure
that their development is within cost, schedule, and performance
parameters. We believe that this report can provide useful insights on key
risks in development, allow decision makers to take corrective actions,
and thereby place programs in a better position to succeed.

David M. Walker Comptroller General of the United States

May 15, 2003 Letter

Congressional Committees The Department of Defense (DOD) is on the
threshold of several major investments in programs that are likely to
dominate budget and doctrinal debates well into the next decade. These
programs include, among others, the Missile Defense Agency*s suite of
land, sea, air, and space defense systems; the Army*s Future Combat
System; and the Air Force and Navy*s Joint Strike Fighter. In fiscal year
2003, the Congress appropriated $127 billion to DOD for the research,
development, and procurement of

weapon systems. Funding for weapon systems is projected to continue
growing to $182 billion in fiscal year 2009* an increase of over 43
percent. In total, the investment in weapons from fiscal years 2003
through 2009 will exceed $1 trillion. Thus, it is essential that sound
foundations for these and other weapon system investments be laid now so
that the resulting

programs can be executed within estimates of available resources. The
challenge of putting new programs on a better footing than their
predecessors is a daunting one. Clearly, the acquisition process produces
superior weapons. But it does so at a high price. Weapon systems routinely
take much longer to field, cost more to buy, and require more support than
investment plans provide for. These consequences reduce the buying power
of the defense dollar, delay capabilities for the warfighter, and force
unplanned* and possibly unnecessary* trade- offs among programs.

DOD has undertaken a number of acquisition reforms over the preceding two
decades in response to those problems, but while there have been
individual successes, these reforms have not yet yielded consistent
improvements in program outcomes. More recently, DOD leadership has
embraced an evolutionary acquisition approach, coupled with time- phased
requirements. This approach supports developing weapons in smaller, more
predictable iterations of increasing capabilities, rather than the past

approach of attempting to achieve a weapon*s maximum capability in one
design leap. DOD is also striving to give programs, such as missile
defense, more flexibility to make trade- offs between cost, schedule, and
performance that can lead to better investment decisions. It is also
currently looking at how to revise its planning, programming, and
budgeting process that has been in place for over 40 years.

Key to any effort to improve weapon system outcomes is using the lessons
that can be learned from the best practices of successful commercial and

defense product development programs. We have found that these practices
can be collectively described as a knowledge- based approach whose success
depends on the timely attainment and use of a product*s technology,
design, and production maturity. In this report, we compare the knowledge
gained on 26 DOD weapon system programs with best practices. Our objective
is to provide decision makers a means to quickly gauge the progress and
potential risks* based on demonstrated knowledge* of the individual weapon
system programs. A Knowledge- Based

All product development efforts, whether for a car, a plane, a missile, or
a Approach Can Lead to

satellite, go through a process of building knowledge. Ultimately, this
process brings together and integrates all of the technologies,
components, Better Acquisition

and subsystems needed for the product to work and to be reliably Outcomes

manufactured. The product development process can be characterized as the
reduction of risk and the resolution of unknowns through the attainment of
knowledge.

About 7 years ago, at the request of the Senate Committee on Armed
Services, we began an extensive body of work identifying best practices in
product development, both in DOD and in the commercial sector. Of
particular interest were cases in which increasingly sophisticated
products were being developed in significantly less time and at lower cost
than their predecessors. A major reason for these successes was the use of
a product development process that was anchored in knowledge. Product
developers employed specific practices to ensure that a high level of
knowledge regarding critical facets of the product was achieved at key
junctures in development. We have characterized these junctures as three
knowledge points. We have also identified key indicators that can be used
to assess the attainment of each knowledge point. When tied to major
events on a program*s schedule, they can disclose whether gaps or
shortfalls exist in demonstrated knowledge, which can presage future cost,
schedule, and performance problems. These knowledge points and associated
indicators are defined as follows.

 Knowledge point 1: Resources and needs are matched. This level of
knowledge is attained when a match is made between a customer*s needs and
the developer*s technical, financial, and other resources.

Technology maturity is a particularly important indicator of resource
availability. A best practice is to achieve a high level of technology
maturity at the start of product development. This means that the

technologies needed to meet essential product requirements have been
demonstrated to work in their intended environment.  Knowledge point 2:
The product design is stable. This level of

knowledge is attained when the product*s design demonstrates its ability
to meet the customer*s requirements. A best practice is to achieve design
stability at the system- level critical design review, usually held midway
through development. Completion of engineering drawings at the system
design review provides tangible evidence that the design is stable.

 Knowledge point 3: Production processes are mature. This level of
knowledge is attained when it is demonstrated that the product can be
manufactured within cost, schedule and quality targets. A best practice is
to achieve production maturity at the start of production. This means

that all key manufacturing processes produce output within statistically
acceptable limits for quality. As illustrated in figure 1, the process is
building block in nature as the

attainment of each successive knowledge point builds on the proceeding
one. While the knowledge itself builds continuously without clear lines of
demarcation, the attainment of knowledge points is sequential. In other
words, production maturity cannot be attained if the design is not mature,
and design maturity cannot be attained if the key technologies are not

mature.

Figure 1: Knowledge Build at Key Points in Product Development Reduces the
Risk of Unknowns

For the most part, all three knowledge points are eventually attained on a
completed product. The difference between highly successful product
developments* those that deliver superior products within cost and
schedule projections* and problematic product developments is how this
knowledge is built and how early in the development cycle each knowledge

point is attained. When knowledge is built more slowly than these points
suggest, less knowledge is on hand at key decisions or events, such as the
decisions to start a development program, hold the critical design review,
and start production. This invites greater cost, schedule, and performance
risks because (1) problems are more likely to be discovered late in the
process and will therefore be more difficult and costly to correct and (2)
a variety of pressures encourage program managers to underestimate the
difficulties.

It is important to note that successful product developers treat
technology development as a different and separate effort that precedes
product development. This treatment of technology development is key to
reaching the first knowledge point at the start of product development, as
it is a prerequisite for capturing design and production knowledge early
in product development. This approach to attaining knowledge puts program
managers* and programs* in a better position to succeed.

Observations When programs proceed with less knowledge than suggested by
best practices, cost, schedule, and performance problems often result. To
varying degrees, all the programs we assessed proceeded with lower levels

of knowledge at critical junctures and thus attained key elements of
product knowledge later in development. In some programs, the consequences
of proceeding with early knowledge deficits have already been felt. For
example:

 The F- 22 Fighter began product development with key technologies
immature* deferring knowledge point 1* and subsequently had only a quarter
of the desired amount of engineering drawings completed at the critical
design review* deferring knowledge point 2. The program has experienced
substantial cost increases and schedule delays in the latter stages of
development.

 The Patriot Advanced Capability missile also reached knowledge points 1
and 2 later than best practices. The seeker technology did not demonstrate
maturity until close to the production decision and the design remains
unstable. Each seeker still needs to be reworked about 3 times on average
before it passes quality inspections. The cost of the

seeker has increased by 76 percent and contributed to a 2- year delay in
the program*s schedule.  The Extended Range Guided Munition program began
with only one of

its 20 critical technologies mature* deferring knowledge point 1. While
progress has been made, program officials do not expect to achieve
maturity on all technologies until after the design review. The lack of
mature technologies contributed to subsequent test failures, cost
increases, and schedule delays.

If programs attain more knowledge as suggested by best practices, they are
in a better position to succeed in meeting cost, schedule, and performance
expectations. We found some programs that did attain key product knowledge
earlier than most. For example:

 The National Polar- orbiting Operational Environmental Satellite System
program ensured that its pacing technologies were demonstrated before
committing to product development. The program plans to demonstrate three
critical sensors on a demonstrator satellite prior to their inclusion on
the new satellite.

 The Theater High Altitude Area Defense System made significant strides
in product development, following a problematic preliminary development
phase. In 2000, we reported that the program*s delayed demonstration of
technologies and components and reliance on fullsystem

testing to discover problems, was a very costly method to mature the
system*s design and nearly caused the cancellation of the program. The
program has since structured a product development phase that places a
much greater emphasis on early demonstration of components, a testing
program that incorporates sufficient time between tests for learning, and
a plan to achieve design stability by

releasing 90 percent of engineering drawings by the time of the critical
design review* knowledge point 2.

In general, we found that the greatest absence of knowledge was in the
area of production. Almost no programs collected statistical process
control data, the indicator for production maturity. Unlike technology
readiness levels, which can be applied at any time, and engineering
drawing release data, which is captured on all programs, few programs
collected statistical process control data. While the absence of this data
does not necessarily mean that production processes were immature,
attained knowledge could not be assessed against an objective standard.
Other indicators of production maturity, such as scrap and rework rates,
can indicate positive trends, but are not prospective* that is, they are
not useful in guiding preparations for production. To some extent,
statistical process control data is not being collected because DOD has
been delegating more responsibility to prime contractors and reducing the
amount of data requested. The lack of such data may put program offices in
a disadvantaged position to gain insights about a contractor*s production
progress. We have recently issued a report that recommends that DOD

collect statistical process control data on its weapon system programs and
DOD has agreed with this recommendation. 1

We conducted our review from September 2002 through May 2003 in accordance
with generally accepted government auditing standards.

1 U. S. General Accounting Office, Best Practices: Capturing Design and
Manufacturing Knowledge Early Improves Acquisition Outcomes, GAO- 02- 701
(Washington, D. C.: July 15, 2002).

Agency Comments DOD did not provide general comments on a draft of this
report, but did provide technical comments on individual assessments.
These comments,

along with program office comments, are included with each individual
assessment as appropriate.

Scope of Our Review We selected programs for the assessments based on
several factors, including (1) high dollar value, (2) stage in
acquisition, and (3)

congressional interest. The majority of the 26 programs covered in this
report are considered major defense acquisition programs by DOD. A program
is defined as major if its estimated research and development costs exceed
$365 million or its procurement exceeds $2.19 billion in fiscal year 2000
constant dollars. We plan to include more programs in subsequent years,
with a greater

focus on programs early enough in development that the assessments can be
used to improve the program*s prospects for success, and issue this report
annually to the congressional defense committees. The individual
assessment of each program can be found in appendix I. Appendix II
contains detailed information on our methodology.

We are sending copies of this report to interested congressional
committees; the Secretary of Defense; the Secretaries of the Army, Navy,
and Air Force; and the Director, Office of Management and Budget. We will
also make copies available to others upon request. In addition, the report
will be available at no charge on the GAO Web site at http:// www. gao.
gov. If you have any questions on this report, please contact me at (202)
512- 4841 or Paul Francis at (202) 512- 2811. Major contributors to this
report are

listed in appendix III. Jack L. Brock, Managing Director Acquisition and
Sourcing Management

List of Congressional Committees

The Honorable John W. Warner Chairman The Honorable Carl Levin Ranking
Member Committee on Armed Services United States Senate The Honorable Ted
Stevens Chairman The Honorable Daniel K. Inouye Ranking Member
Subcommittee on Defense Committee on Appropriations United States Senate
The Honorable Duncan Hunter Chairman The Honorable Ike Skelton Ranking
Minority Member Committee on Armed Services House of Representatives The
Honorable Jerry Lewis Chairman The Honorable John P. Murtha Ranking
Minority Member Subcommittee on Defense Committee on Appropriations House
of Representatives

Appendi Appendi xes x I

Assessments of Individual Programs Introduction For the 26 programs, each
assessment provides the historical and current program status and offers
the opportunity to take early corrective action when a program*s projected
attainment of knowledge diverges significantly from the best practices.
The assessments also identify programs that are employing practices worthy
of emulation by other programs. If a program is attaining the desired
levels of knowledge, it has less risk* but not zero risk* of future
problems. Likewise, if a program shows a gap between demonstrated
knowledge and best practices, it indicates an increased risk* not a
guarantee* of future problems. The real value of the assessments is
recognizing gaps early, which provides opportunities for constructive
intervention* such as adjustments to schedule, trade- offs in
requirements, and additional funding* before cost and schedule
consequences mount.

Our assessment of each program is summarized in two components*( 1) a
system profile and (2) a product knowledge assessment.

The system profile presents a general description of the product in
development; a picture of the product or a key element of the product; a
schedule timeline identifying key dates in the program; a table
identifying the prime contractor; the program office location, and the
fiscal year 2004 requested funding if available; and a table summarizing
the cost, schedule and quantity changes to the program.

The rest of the assessment analyzes the extent to which product knowledge
at the three key knowledge points has been attained. We depict the extent
of knowledge in a stacked bar graph and provide a narrative summary at the
bottom of the first page. The second page is devoted to a narrative
assessment of technology, design and production maturity, as well as other
program issues identified and comments from the program office.

The product knowledge figure is based on the three knowledge points and
the key indicators for the attainment of knowledge. A *best practice* line
is drawn based on the ideal attainment of the three types of knowledge at
the three knowledge points (see fig. 2).

Figure 2: Graphic Depiction of Best Practices for Technology, Design, and
Production Knowledge

The first major point on the best practice line represents two facts: a
commitment to a new product development has been made and the key
technologies needed for the new product are mature.

When all critical technologies have reached a technology readiness level
7, technology maturity* and thus knowledge point 1* has been attained. In
our assessment, the technologies that have reached technology readiness
level 7, a prototype demonstrated in an operational environment, are
considered mature and those that reach technology readiness level 6, a

prototype demonstrated in a relevant environment, are assessed as
attaining 50 percent of the desired level of knowledge. Satellite
technologies that achieved technology readiness level 6 were assessed as
fully mature due to the difficulty of demonstrating maturity in an
operational environment* space. (Technology readiness levels are more
fully explained in appendix II.) The second major point on the best
practice line captures technology maturity plus design maturity* knowledge
point 2. A design is considered mature when 90 percent of the engineering
drawings have been released or deemed releasable to manufacturing. In the

successful programs we have studied, design maturity is attained about
halfway through the product development phase. The third major point on
the best practice line captures the sum of technology maturity, design
maturity, and production maturity. Production is considered mature when
all key production processes are in statistical control. Ideally, this
occurs before the first products for delivery to the customer are
manufactured. As can be seen, knowledge about the technology, design, and
production of a

new product builds over time. While the knowledge itself builds
continuously without clear lines of demarcation, the attainment of
knowledge points is sequential. In other words, production maturity cannot
be attained if the design is not mature, and design maturity cannot be
attained if the key technologies are not mature.

Data for a given weapon system program is then plotted against the best
practices line. In the assessments that follow, a brown bar indicates the
technology knowledge attained by a weapon system program. The actual
technology readiness levels attained for a program*s key technologies are
measured at the start of development* normally milestone II or milestone B
in the Department of Defense*s (DOD) acquisition process. The closer a
program*s attained knowledge is to the best practice line, the more likely
the weapon will be delivered within its estimated cost and schedule. A
knowledge deficit at this point* indicated by a gap between the technology
knowledge attained by the weapon system and the best practices line* means
the program proceeded with immature technologies and may face a greater
likelihood of cost and schedule increases as technology risks are
discovered and resolved. A green bar indicates the design knowledge
attained by a weapon system program. This is calculated by measuring the
percent of engineering drawings released to manufacturing. The green bar
is stacked on top of the brown bar to indicate whether any cumulative gap*
considering both technology and design* exists at the halfway point

of product development. A blue bar indicates the production knowledge
attained by a weapon system program. This is calculated by measuring the
percentage of key production processes in statistical control. The blue
bar is stacked on top of the brown and green bars to indicate whether any
cumulative technology, design, and production gaps exist at the time
production begins. In some cases, we obtained projections from the program
office of future knowledge attainment. These projections are depicted as
dashed bars.

Figure 3 depicts an example of an assessment for a notional weapon system.

Figure 3: Depiction of a Notional Weapon System Program*s Knowledge as
Compared with Best Practices

An interpretation of this notional example would be that the product
development began with key technologies immature, thereby missing
knowledge point 1. Knowledge point 2 was not attained at the design review
as some technologies were still not mature and only a small percentage of
engineering drawings had been released. Projections for the

production decision show that the program is expected to achieve a greater
level of maturity, but will still fall short. It is likely that this
program would have had significant cost and schedule increases.

We also found three situations in which programs were unable to provide
key knowledge indicators. We used three types of labels in the knowledge
figures to depict those situations. Programs with these labels are
distinguished from those that have elected not to collect data that can be
used to assess progress against best practices. First, a few programs are
planning to collect the relevant knowledge indicator, but they have not
yet begun collecting it. In these situations, we annotate the graph with
the phrase *Data unavailable.* Second, a few programs have not followed
the

traditional acquisition model. For example, one program combined the

development start decision with the production decision. Another program
used commercial off- the- shelf components, which negated the need to
monitor production processes. In these situations, we annotate the graph
with the phrase *Not applicable.* Finally, some programs were unable to
provide or reconstruct the relevant knowledge indicator because the event
happened too many years ago. In these situations, we annotate the graph
with the phrase *Not assessed.*

Our assessments of the 26 systems follow.

Advanced Amphibious Assault Vehicle (AAAV)

The Marine Corps* AAAV is designed to transport troops from ships to shore
at higher speeds and from farther distances than the existing AAV- 7. It
is designed to be more mobile, lethal, reliable, and effective in all
weather conditions. AAAV will have two variants* a troop carrier for 17
Marines and a command vehicle to manage combat operations in the field.

Prime contractor: General Dynamics

Program office: Woodbridge, Va.

FY 2004 funding request: FY 2003 dollars in millions Approved Latest
Percent R& D $240.7 million

12/ 00 12/ 01 change

Procurement $97.9 million

Research & development cost $1, 395.7 $1,616. 4 15.8 Quantity 0 vehicles

Procurement cost $6, 256.7 $6,741. 7 7.8 Total program cost $7, 732.7
$8,440. 9 9.2

Program unit cost $7. 544 $8.235 9.2

Total quantities 1,025 1,025 0.0 Acquisition cycle time (months) 138 150
8.7

AAAV demonstrated most technology and design knowledge at critical
junctures in the program. At the start of the program, all but one of the
critical technologies were mature. The design was close to meeting best
practice standards at the design review, signifying the design was stable.
Early development of fully functional prototypes and

other design practices facilitated design stability. However, late
maturation of the remaining technology may lead to some redesign. Also,
the demonstration of production maturity remains a concern because the
program is currently uncertain about requiring the contractor to use
statistical process controls to achieve quality objectives. The AAAV
production decision is not scheduled until

September 2005. Remaining efforts include developmental, operational, live
fire, and reliability testing.

AAAV Program

Other Technology Maturity

The Marine Corps has recently restructured the AAAV program to add 12
additional months of testing before Four of the five critical technologies
had

the September 2005 production decision. This change demonstrated an
acceptable level of maturity at the more than doubles the number of
vehicle test months start of product development. The remaining

previously planned. The change also moves the initial technology, moving
map navigation, is not expected to operational capability date from
September 2007 to achieve maturity until the spring of 2003. Program

September 2008. The program estimates a $480 million officials stated that
maturing this technology is increase in acquisition costs*$ 101 million
for added contingent on developing and testing system

testing, $75 million for development, and $304 million hardware. As a
backup, program officials said they

for recurring production. could carry out the AAAV mission using existing
technology, but it would not provide full vehicle- tovehicle

Program Office Comments

situational awareness. AAAV program officials concurred with our

Design Maturity

assessment. The AAAV design is essentially complete. However, late
maturation of the new mapping system may lead to some redesign, if testing
identifies any problems. At the critical design review, AAAV had completed
77 percent of the drawings* not up to the best practice standard of 90
percent, but higher than many

DOD programs. Early engineering prototypes* fully integrated and
functional* allowed the program to demonstrate that the design worked as
required. These early prototypes have completed over 4,000 hours of
testing that resulted in design improvements for subsequent prototypes.

To complete development, program plans call for building and testing nine
development prototypes and one live fire test vehicle. These prototypes
will be production representative vehicles for developmental, operational,
live fire and reliability testing. The first

prototype is scheduled to be available by May 2003. Production Maturity

Program officials are developing a production readiness plan to ensure
vehicles will meet cost, schedule, and quality objectives. At this time,
they are uncertain whether this plan will require the contractor to use
statistical process controls, the best practice standard. As the prime
contractor currently produces the nine developmental prototype vehicles,
it is not tracking statistical process control data. Instead, it is using
postproduction inspections, considered less

efficient and effective than statistical process controls to achieve
quality.

Airborne Laser (ABL)

The Missile Defense Agency*s ABL is designed to destroy enemy ballistic
missiles almost immediately after their launch. The system, carried aboard
a highly modified Boeing 747 aircraft, uses a highenergy chemical laser to
rupture the skin of enemy missiles; a beam control/ fire control subsystem
to guide the laser beam through the aircraft, focus the beam on the
target, and maintain the beam*s quality as it travels through the
atmosphere; and a battle

management subsystem to plan and execute the engagement. We assessed all
components.

Prime contractor: Boeing

Program office: Albuquerque, NM

FY 2004 funding request: FY 2003 dollars in millions Approved Latest
Percent R& D $610.0 million 1/ 97 12/ 02 change

Procurement $0 million

Research & development cost $2, 400.5 $4,415. 7 84 Quantity 0 systems

Procurement cost $3, 170.6 NA NA Total program cost $5, 571.1 NA NA

Program unit cost $795.9 NA NA

Total quantities 7 NA NA Acquisition cycle time (months) 118 NA NA Note:
Latest costs only through FY 2007. Procurement funding, quantities, and
the initial capability date have yet to be determined. NA = not
applicable.

Only one of ABL*s critical subsystems has demonstrated acceptable levels
of maturity. The Missile Defense Agency is developing an initial ABL
system to demonstrate technology critical to the system*s design and plans
to begin development of a second improved demonstration aircraft in 2003.
Either of these aircraft, or later improved configurations, could be given
to the Air Force for

operational testing and production if system- level tests show that any
one of them is capable of destroying a threat missile at an operational
range. Although the agency*s development strategy incorporates some
knowledge- based practices, it is

difficult to see how the discipline of a knowledgebased approach can be
achieved when uncertainty exists about whether the effort is a technology
development or a product development.

ABL Program

Technology Maturity

Only one of ABL*s five critical subsystems, the aircraft itself,
represents mature technology. A second subsystem, which directs the laser
energy through the aircraft, consists of several technologies that have
been tested in a simulated environment. However, three other subsystems
consist of low- fidelity

prototype technologies that have only been tested in a laboratory
environment. They include the laser, the battle management subsystem, and
the ground support subsystem.

Problems associated with maturing technology have consistently been a
source of cost and schedule growth throughout the life of the program. DOD
analysts attribute this growth to the increased complexity of designing
laser subsystems, substantial increases in engineering analysis and
design, and greater than anticipated aircraft engineering complexity.

The program is managed under the Missile Defense Agency*s new
capabilities- based acquisition strategy. This approach develops an
operational system

through a series of block upgrades. The agency plans to use the first two
blocks, block 2004 and block 2008, to demonstrate critical technologies,
but if tests show either configuration has any battlefield utility, that
configuration could be deployed in the event of an emergency.

The 2004 configuration will have a 6- module laser, rather than the 14
modules planned for the production system. The optical components can
withstand the heat produced by a 6- module laser, but the agency would
have to redesign optical components for the system to withstand the heat
associated with an increase in laser power. In addition, the 2004
configuration is far too heavy to allow the addition of laser modules that
will likely be needed in an operational ABL system. To accommodate more
modules, a weight reduction

program has begun that includes redesigning many components and the
increased use of composite materials. The program is considering whether
to use a different aircraft configuration that would allow the system*s
weight to be moved forward to relieve stress on the airframe. However, its
use would require additional design changes.

The Missile Defense Agency has made changes that are expected to improve
its ability to evolve ABL*s critical technologies, including adopting a
flexible requirements setting process, providing additional time and
facilities to develop and test these technologies, and attaining the
knowledge to match the warfighters* needs with demonstrated technology. On
the other hand, it is not clear whether the start of a

block represents a technology development or a product development. This
uncertainty may hamper the application of knowledge standards and forfeit
the

discipline necessary to ensure successful product development.

Program Office Comments

In commenting on a draft of this assessment, program officials
reemphasized their commitment to spiral development and capabilities-
based acquisition. They plan to use this strategy to improve the critical
aspects of the system by allowing the pace of technological development to
dictate the introduction of improved capabilities into the system. They
believe this strategy

is not inconsistent with knowledge- based acquisition. They also mentioned
that laser power depends not only on the number of laser modules but also
on module efficiency, optics, and pointing precision. They admit that the
laser subsystem should be operated in flight before any production
decision is made. Program officials are conducting emergency operational
capability planning to support a possible emergency ABL deployment. This
decision will be based on the potential threat and an assessment of the
capabilities ABL may provide.

The program office indicated that all but one of the battle management
components have been tested in an operational environment. This component
is the active ranger system, which provides crucial angle measurements and
range data for engaging ballistic

missiles.

Advanced Extremely High Frequency (AEHF) Communications Satellite

The Air Force*s AEHF is a satellite system intended to replace the
existing Milstar system with improved, survivable, jam- resistant,
worldwide, secure communication capabilities at lower launch costs. First
launch of an AEHF satellite is expected in 2006. The system also includes
a mission control segment with service- specific terminals to process
satellite information. DOD is negotiating international partner
participation in the program. We assessed the satellites and mission
control segments.

Prime contractor: Lockheed Martin

Program office: El Segundo, Calif.

FY 2004 funding request: FY 2003 dollars in millions Approved Latest
Percent R& D $778.1 million 10/ 01 12/ 02 change

Procurement $0 million Research & development cost $4, 339.3 $4,608. 4 6.2
Quantity 0 satellites

Procurement cost $1, 286.2 $509. 1 -60.4 Total program cost $5, 625.5
$5,117. 5 -9.0

Program unit cost $1, 125.1 $1,705. 8 51.6

Total quantities 5 3 -40.0 Acquisition cycle time (months) 111 118 6.3

The AEHF satellite program demonstrated most technology knowledge at
development start. Eleven of the 12 critical technologies were mature,
according to best practice standards. The remaining

technology is not projected to be mature prior to the critical design
review, nor does it have a backup technology. However, some elements of
this technology are mature. The program expects to complete 90 percent of
its drawings by the critical design review. The manufacture of the

communications and transmission security subsystem is a major challenge
facing the program as upgrades are being added into the new cryptological
equipment. If production of this subsystem slips, first launch could slip
correspondingly as no backup exists.

AEHF Program

security subsystem*s functions as it is integrated into the AEHF
satellite's communications subsystem.

Technology Maturity

However, continued complications in fabrication Eleven of the 12 critical
technologies have reached

could potentially place the entire program at cost, maturity according to
best practice standards. The

schedule, and performance risk. program does not project achieving
maturity on the

Other Program Issues

remaining technology* the phased array antenna* by the design review in
June of 2004, nor does it have a

In December 2002, the Deputy Secretary of Defense backup capability.
However, some elements of this decided to change the acquisition strategy
of AEHF technology have been demonstrated in an operational

from a five- satellite program to a three- satellite environment.

program. Under the revised strategy, full capability may no longer be
satisfied by an AEHF- only

Design Maturity

constellation. The program has completed 150 or more of the 6,000

Program Office Comments

total drawings for release to manufacturing. Program officials project
completing 90 percent of drawings by

In commenting on a draft of this assessment, program the system critical
design review in June 2004. The officials stated that the program is
executing very well program has completed key segment level preliminary

since contract definitization in August 2002, with cost design reviews and
is expected to complete all design and schedule variance at less than 1
percent. reviews by the second quarter of fiscal year 2004.

Currently, at approximately 33 percent complete Program officials consider
the design and

toward first launch, the total program is on track and development of the
satellite subcomponents low risk estimated to finish on time and on
budget. The system because those components have been used on other

preliminary design review has been completed. space systems. However, the
integration of these

Critical design reviews are on track for completion by subcomponents into
a subsystem, such as the phasedarray Spring 2004. Funding cuts have, in
the past, caused antenna, has yet to be successfully

schedule slips and cost increases. Given the focus on demonstrated at the
AEHF satellite frequencies.

the critical design review, the impacts of changing Program officials
assessed the software development requirement will have increasing
deleterious effects.

for the mission control system as moderate risk and The program remains
focused on addressing critical

have developed a risk mitigation strategy. This risks that threaten cost,
schedule, and performance.

strategy includes consulting with the National New system security
requirements recently received Software Engineering Institute and the
Aerospace

from the National Security Agency for the space, Corporation and
conducting a software development

mission control, and terminal segments are being capability evaluation.
Also, the program office has evaluated. After aggressive risk management,
the most

incorporated spiral development and the use of likely impacts include
additional testing, verification,

software emulators so users and developers can see and program
documentation. The program has also how the software will look and work.
Until these

begun developing engineering models for all of the actions are completed,
software may be at risk for

critical subsystems. These efforts are on track and unplanned cost and
schedule growth.

proceeding well.

Production Maturity

Any future problems with the fabrication of the communications and
transmission security microprocessor, a component designed to limit access
to satellite transmissions to authorized users, could delay the production
schedule and the launch of the first satellite planned for December 2006.
Program officials have started a number of risk reduction efforts,
including a chip emulator whose purpose is to simulate the communications
and transmission

AN/ APG- 79 Active Electronically Scanned Array (AESA) Radar

The Navy*s AESA radar is one of the top upgrades for the F/ A- 18E/ F
aircraft. It is to be the aircraft*s primary search/ track and weapon
control radar and is designed to correct deficiencies in the current
radar. According to the Navy, the AESA radar is key to maintaining the
Navy*s air- to- air fighting

advantage and will improve the effectiveness of the air- to- ground
weapons. When completed, the radar will be inserted in new production
aircraft and retrofitted into existing aircraft.

Prime contractor: Raytheon

Program office: Patuxent River, Md.

FY 2003 funding request: FY 2003 dollars in millions Approved Latest
Percent R& D $107.1 million

6/ 01 12/ 01 change

Procurement $24.06 million

Research & development cost $518.9 $494. 2 -4.7 Quantity 0 radars

Procurement cost NA NA NA Total program cost NA NA NA

Program unit cost NA NA NA

Total quantities NA NA NA Acquisition cycle time (months) 69 68 -1.4 Note:
NA = not applicable.

The AESA radar*s demonstrated knowledge is difficult to characterize. The
fact that almost all of the engineering drawings have been completed
suggests design stability. However, until the technologies are
demonstrated, the potential for design changes remains. The AESA radar is
also dependent on other programs that could pose significant risk to the
radar*s cost, schedule, and technical performance. The technology and
design risks are significant given that the AESA radar is only a few
months from a production decision. The Navy is currently reassessing the
radar*s technology maturity. Although many of the F/ A- 18E/ F aircraft
will be retrofitted with the AESA radar, full funding for the retrofitting
has not been budgeted. If the radar is not ready for production as
scheduled, more aircraft will have to be retrofitted.

AESA Radar Program

planned total buy of 548 F/ A- 18E/ F aircraft will not receive the radar
as they are being produced. Plans are

Technology Maturity

to retrofit the radar onto 136 aircraft at a projected The AESA program*s
four critical technologies were

cost of $3.14 million each. This cost does not include not mature at the
start of development in February

the cost of new APG- 73 mechanical scanned radars 2001, and they were not
mature as of May 2002. The

that will be installed in the aircraft until AESA radars Navy is currently
reassessing the maturity of these

are available for retrofit. If delays occur in the AESA technologies. At
the time of its last assessment, two of

radar deliveries, retrofit costs will increase. the technologies had been
tested using simulation and The AESA radar is projected to weigh about two
had been tested in the laboratory. Program

270 pounds more than the current radar and will officials indicated that
they have several options for

require a more capable cooling system than the one dealing with immature
technologies, including

currently on the aircraft. The Navy expects some utilizing backup
technologies. Initial flight tests of the minor degradation in aircraft
performance, such as radar in an aircraft are scheduled for June 2003*

slightly decreased range, as a result of the increased concurrent with the
production decision. All four

weight and new cooling system. technologies are not expected to be mature
until

The AESA program is linked to a number of other late 2004.

corporate and Navy programs. For example, the radar

Design Maturity

will use a 32- port fiber channel fabric module developed by Boeing as a
commercial venture. At the design review, 67 percent of the currently
Technical difficulties with the module have caused projected total
drawings were completed. In the

schedule delays and may impact cost and performance period between June
2002 and December 2002, the of the radar. Also, Raytheon is developing
some number of total expected drawings increased by 21 hardware and
software for the radar with company percent. Program officials stated that
the increase was

funds or in coordination with other programs. due to new or modified
drawings for systems Disruptions in these efforts could adversely impact
the supporting the radar such as the radome, shield, and

AESA program. aircraft airframe. Program officials indicated that they
currently have 98 percent of the drawings complete;

Program Office Comments

however, the technology maturation process may lead The AESA program did
not provide a general to more design changes.

statement in response to our review but did provide

Production Maturity

technical comments that were incorporated where appropriate. We could not
assess the AESA program*s production maturity against best practices, as
statistical control data was not available.

Other Program Issues

Program officials estimate that the first low- rate production unit will
exceed its cost target by 27 percent. Subcontractor development cost was
considered to be the biggest contributor to this increase. The effects of
the cost increase may be minimized in low- rate production lots 1- 3
because of firm fixed price contract options. Program officials stated
that cost reduction initiatives were underway to reduce the cost overruns
by half by full- rate

production. Delivery of the first production AESA radars for insertion
into F/ A- 18E/ F aircraft on the production line is scheduled for fiscal
year 2005. As a result, 254 of the

AIM- 9X Short- Range Air- to- Air Missile

The AIM- 9X is a follow- on version of the existing AIM- 9M short- range
missile for Air Force and Navy fighters. The AIM- 9X is designed to be a
highly maneuverable, launch- and- leave missile; capable of engaging
targets using passive infrared guidance to provide full day/ night
operations and improved resistance to countermeasures and expanded target
acquisition. The full capabilities of the AIM- 9X will not be achieved
without completing development of the helmet mounted cueing system* a
separate

development program that we did not assess.

Prime contractor: Raytheon Missile System Company

Program office: Patuxent River, Md.

FY 2004 funding request: FY 2003 dollars in millions Approved Latest

Percent R& D $2.7 million 1/ 97 12/ 01 change

Procurement $104.9 million

Research & development cost $577.0 $594.6 3.0 Quantity 531 missiles

Procurement cost $2, 116.1 $2, 055.1 -2.9 Total program cost $2, 693.0 $2,
649.6 -1.6

Program unit cost $0. 268 $0. 261 -2.5

Total quantities 10, 049 10, 142 0.9 Acquisition cycle time (months) 92
105 14.1

The AIM- 9X program entered production in September 2000 without assuring
that the manufacturing processes were mature. However, because the missile
is a follow- on to the AIM- 9M

missile, program officials believe that they have significant production
knowledge. The program ensured, prior to entering low- rate initial
production, that the missile design was stable. The program did attain
knowledge early in development by using proven technologies from
predecessor systems and other programs, as well as testing numerous
prototype versions of the missile. As a result, the program released the
majority of its engineering drawings at the design review.

AIM- 9X Program

rework data. The AIM- 9X acquisition cost and schedule history shows the
program has been able to

Technology Maturity

meet its goals. All of the AIM- 9X critical technologies are mature

Program Office Comments

because they have been demonstrated in developmental tests using actual
hardware in realistic

In commenting on a draft of this assessment, program conditions.
Specifically, the program used prototypes

officials acknowledged they did not contractually to test new technologies
and existing missile

require collection of statistical process control data on components that
are being employed in a new

critical manufacturing processes. Program officials operational
environment.

stated their strategy for demonstrating manufacturing process maturity
includes building, testing, and

Design Maturity

evaluating production representative missiles; The design of the AIM- 9X
is complete, and 100 percent

conducting multiple readiness reviews; utilizing lowrate of the drawings
have been released to manufacturing. initial production to test production
processes; The AIM- 9X program built and tested 43 prototypes of

and maturing production processes before full- rate various configurations
during development to help

production. mature the missile's design. Hardware and software performance
was assessed at subsystem and system levels, and design changes were
incorporated into the prototypes until a mature and stable missile
configuration was demonstrated. The AIM- 9X program held design reviews
for the 11 subsystems between

October 1997 and March 1998. The early design reviews, prototypes, and
early testing allowed the program to achieve a stable design at the system
design review in March 1998. At that time, the contractor had released 94
percent of its engineering drawings to manufacturing.

Production Maturity

The AIM- 9X program does not contractually require collection of
statistical process control data on critical manufacturing process, but it
has undertaken an acquisition strategy to incentivize the contractor to

reach cost and quality goals. However, the contractor and program
officials believe that they have significant knowledge about producing the
missile. The AIM- 9X is a variant of the AIM- 9M missile and uses
components

produced for other weapon systems, providing the program with significant
production knowledge. In addition, to improve the production capabilities,
the contractor built developmental units on production equipment. Program
officials believe this practice has allowed them to mature the
manufacturing processes. According to program officials, most of the
critical processes on the AIM- 9X are at the subcontractor level and a
process exists to attain cost and quality goals. This is accomplished
primarily by postproduction inspections to track production yield, scrap,
and

Advanced Threat Infrared Countermeasures/ Common Missile Warning System

The Army*s and Special Operations* ATIRCM/ CMWS is a component of the
Suite of Integrated Infrared Countermeasures planned to defend U. S.
aircraft from advanced infrared- guided missiles. The system will be
employed on Army and Special Operations

aircraft. ATIRCM/ CMWS includes an active infrared jammer, a missile
warning system, and a countermeasure dispenser capable of loading and
employing expendables, such as flares, chaff, and smoke.

Prime contractor: BAE Systems

Program office: Huntsville, Ala.

FY 2004 funding request: FY 2003 dollars in millions Approved Latest
Percent R& D $7.2 million

3/ 96 12/ 01 change

Procurement $75.7 million

Research & development cost $568.0 $540. 1 -4.9 Quantity 2 units

Procurement cost $2, 338.9 $1,971. 9 -15.7 Total program cost $2, 906.9
$2,512. 0 -13.6

Program unit cost $0. 940 $2.330 148.0

Total quantities 3,094 1,078 -65.2 Acquisition cycle time (months)
Classified Classified NA Note: NA = not available.

The ATIRCM/ CMWS is scheduled to enter production in May 2003 with no
assurance that production processes are in control. The CMWS portion of
the ATIRCM/ CMWS program entered limited production in February 2002 to
meet an urgent need. Full- rate production for ATIRCM/ CMWS was delayed
because of reliability problems, which may indicate that production
processes were not in control. These problems are, at least in part, a
consequence of design proceeding with known shortfalls in knowledge: key
technologies were demonstrated late in development and only a small number
of design drawings were completed by design review. Resolving these
knowledge shortfalls has led to cost and schedule increases. While the key
technologies appear mature, reliability and producibility issues could
necessitate design changes.

ATIRCM/ CMWS Program

The program delayed the production decision for the combined system an
additional year to the currently Technology Maturity

scheduled May 2003 date primarily due to reliability The five critical
technologies for the system are

issues. Reliability testing was halted because of mature, but they did not
mature until after the system

numerous failures with the ATIRCM subsystem. design review. Most of the
early technology

Reliability failure can be an indicator of producibility development
effort was focused on the application to

and process control problems. The program plans to rotary wing aircraft.
However, when product

build and develop six additional subsystems during development began in
1995, the requirements were 2002 and 2003. The full- rate production
decision for expanded by Office of the Secretary of Defense

the complete system is now scheduled for 2005. direction to include Navy
and Air Force fixed wing

Other Program Issues

aircraft. According to program officials, they did not fully anticipate
the additional technology needed to

The Army procured an initial 32 systems in fiscal year meet these much
more demanding requirements. This 2002 that only included the CMWS. The
Army plans to change caused problems that largely contributed to procure a
total of 99 ATIRCM/ CMWS systems to outfit cost increases of more than 150
percent to the special operations aircraft between fiscal year 2002
development contract. The Navy and the Air Force and 2009. subsequently
dropped out of the program, rendering the extra effort needless.

Program Office Comments Design Maturity

In commenting on a draft of this assessment, program officials stated that
the Army eliminated the program's The basic design of the system is
complete, with

funding for fiscal years 2002 and 2003. In fiscal year 100 percent of the
drawings released to

2003, the Special Operations Command funded the manufacturing. However,
reliability and producibility

urgent procurement of 32 CMWSs. Subsequently, the issues could require
design changes. The design was

Army reinstated the program for fiscal years 2004* particularly immature
at the critical design review, 2009. The program office stated that the
loss of with only 22 percent of the drawings complete. A

funding in fiscal year 2003 slowed the program major cause was that the
technology requirements

markedly. The program's acquisition strategy remains were not well
understood until the system design

to equip Special Operations forces before equipping review, leading to the
discovery that a major redesign the remainder of the Army. was needed to
meet requirements. It was not until

The system was modified in 2002 to address ATIRCM 2 years after the design
review that 90 percent of the reliability, producibility, and built- in-
test issues. Six drawings were released and the design was

ATIRCM systems are being manufactured and tested considered stable.
According to program officials, the

to demonstrate and verify the enhancements. ATIRCM immature design caused
inefficient manufacturing, is scheduled to begin low- rate initial
production in rework, and testing and contributed to the 3- year

May 2003, and CMWS is scheduled to begin low- rate schedule delay. initial
production in January 2004. The program office

Production Maturity

stated that low- rate production is required to maintain a production
base. The system's operational testing is The ATIRCM/ CMWS program does
not collect

planned for March 2005. According to the program statistical control data
on its critical manufacturing

office, the prime contractor indicated that statistical processes. Program
officials have identified the

process control is not within its corporate philosophy, absence of
statistical process control data as a particularly for a program with such
low production weakness and believe it should be instituted. Despite

rates and quantities. this shortfall in knowledge, the Army entered
limited CMWS subsystem production in February 2002 to

meet an urgent need of the U. S. Special Operations Command.

Advanced Wideband Satellite (AWS)

The Advanced Wideband Satellite system is designed to provide improved,
survivable, jamresistant, worldwide, secure and general purpose
communications to support the National Aeronautics and Space
Administration, DOD and the intelligence community. It will replace the
current Milstar satellite system and supplement the AEHF satellite system,
reviewed elsewhere in this report. It will be the cornerstone of a DOD

architecture that includes the multiple satellite systems.

Prime contractor: In competition

Program office: El Segundo, Calif.

FY 2004 funding request: FY 2003 dollars in millions President*s budget
Latest Percent R& D $439.3 million 02/ 03 00/ 00 change

Procurement $0.0 million Research & development cost $5, 814.5 $0.0 0.0
Quantity 0 satellites

Procurement cost $2, 343.9 $0.0 0.0 Total program cost $8, 158.4 $0.0 0. 0

Program unit cost $2, 039.6 $0.0 0. 0

Total quantities within budget 4 0 0 Acquisition cycle time (months) 75 0
0 Note: Program costs and schedule have yet to be formally approved by
DOD. Costs only through FY 2009.

The AWS/ TSAT program is scheduled to enter product development with only
one of its five critical technologies mature, according to best practices.
The initial product development period will likely require concurrent
technology and product development activities to maintain schedule.
Although the new draft space acquisition guidance allows this approach, it
is contrary to the best practice of separating technology development from
product development.

AWS/ TSAT Program

Technology Maturity

Of the five AWS/ TSAT key space segment technologies, one is mature while
the other four are scheduled to reach maturity by January 2006, more than
2 years after development start. Three of the four immature technologies
have a backup technology available in case of development difficulties.
However, use of these technologies would degrade system overall
performance. The Single Access Laser

Communications technology has no backup and, according to program
officials, any delay in maturing this technology would result in a slip in
the expected launch date.

Other Program Issues

The program plans a development cycle that is, according to DOD
documentation, aggressive. The satellite development cycle is planned to
be 75 months: 27 months for technology development; 15 months for product
development; and 33 months for satellite build, test and launch. This
period of time

is substantially shorter than the development cycle for the AEHF satellite
(118 months vs. 75 months), though the AWS/ TSAT system is expected to
provide a transformational leap in satellite communications capability.
The program is managed under the new National

Security Space Acquisition process, which makes no clear distinction
between the end of technology development and the start of product
development. Therefore, the AWS/ TSAT acquisition strategy may allow the
system's technology development and

product development to be conducted concurrently prior to the production
decision. DOD*s acquisition system policy states that one of the entrance
criteria for the system development and demonstration phase

is technology maturity. The AWS/ TSAT acquisition strategy does not ensure
that technology maturity will be achieved prior to the start of product
development consistent with best practices.

Program Office Comments

In commenting on a draft of this assessment, program officials stated that
the National Security Space Acquisition Policy was developed to streamline
the decision- making framework and to tailor it for space systems, in
order to more efficiently field systems that incorporate rapidly changing
technology advances.

Cooperative Engagement Capability (CEC)

The Navy*s CEC is designed to connect radar systems to enhance detection
and engagement of air targets. Ships and planes equipped with their
version of CEC hardware and software will share real- time data to create
composite radar tracks, essentially allowing the battle group to see the
same radar picture. A CEC- equipped ship will then be able to detect and
launch missiles against targets its radar cannot see. We assessed block 1
of the CEC. The Navy is developing a more advanced block 2 CEC.

Prime contractor: Raytheon Systems Corporation

Program office: Washington, D. C.

FY 2004 funding request FY 2003 dollars in millions Approved Latest
Percent R& D $72.5 million

5/ 95 4/ 02 change

Procurement $128.6 million

Research & development cost $1, 154.0 $2,052. 8 77.9 Quantity 21 systems

Procurement cost $1, 292.5 $2,127. 6 64.6 Total program cost $2, 493.9
$4,180. 4 67.6

Program unit cost $13.6 $15. 4 12.8

Total quantities 183 272 48.6 Acquisition cycle time (months) 16 16 0. 0

The technologies and design of the CEC program block 1 are fully mature.
The program*s production maturity could not be assessed. The program lacks
the necessary data primarily because the government does not collect it on
the commercially available portions. However, program and

contractor officials consider the processes to be capable of producing a
quality product on time and on cost. Block 1 of the CEC program was
approved

in April 2002 for full- rate production for the shipboard version and
continued low- rate initial production for the airborne version.

CEC Program

Some solutions for interoperability and integration issues will also be
assessed in follow- on testing.

Technology Maturity

However, many of these issues are expected to be In January 2002, the
Office of Naval Research assessed resolved through the introduction of
block 2. The plan CEC*s six critical technologies. Five of the

was approved in April 2002. Block 2 is expected to technologies assessed
as mature were incorporated provide cost, performance, and functional into
the shipboard version when it successfully

improvements over the current system, though its completed the operational
evaluation in May 2001. The

details are yet to be defined. Among the anticipated sixth technology, a
data processor, was not assessed

characteristics of block 2 is interoperability with as part of the
operational evaluation but was

legacy combat systems. determined to be mature.

Program Office Comments Design Maturity

In commenting on a draft of this assessment, program CEC*s basic design
appears complete, as all of the

officials stated that a production readiness review drawings needed to
build the shipboard version have

conducted in October 2001 found CEC production to been released to
manufacturing.

be mature. They evaluated all areas of production, including quality,
configuration management, CEC program officials noted that new drawings

processes and procedures, drawings, and testing. They continue to be
released. They explained that as

stated that the contractor is delivering systems on commercially available
technologies, which comprise

schedule and within cost. To date, 29 systems over 5 approximately 60
percent of CEC*s hardware, become

years have been successfully delivered, installed, more advanced, portions
of the system will need to be

tested, and many have been deployed. Following and redesigned to
incorporate those advances.

operational testing and evaluation, the Navy found CEC to be operationally
suitable and effective and the

Production Maturity

DOD Director for Operational Test and Evaluation found CEC demonstrated
the highest reliability of any We could not assess the CEC program*s
production system tested so far of comparable complexity. maturity against
the best practice as data were not

According to program officials, CEC*s use of available. According to
program officials, the

commercial off- the- shelf components enables the noncommercially
available portions of CEC do not

program to select mature cost- effective components involve any critical
manufacturing processes. Officials

from industry, instead of manufacturing them inhouse. indicated that they
do not have insight into

In recognition of the above, DOD approved the manufacturing processes for
the commercially program for full- rate production in April 2002.
available portions, including whether these processes are critical and
whether the contractor has them under statistical control.

The program officials and the contractor are confident that a quality
product can be delivered on time and within cost based on the contractor*s
adherence to industry standards and past performance on the lowrate

initial production contracts for the shipboard version.

Other Program Issues

Battle group- level interoperability, integration, and built- in- test
false alarm rates were identified as areas needing improvement following
the operational

evaluation. Program officials expect a solution for the alarm rates to be
in place for a follow- on operational test and evaluation planned for
2004.

CH- 47F Improved Cargo Helicopter

The Army*s CH- 47F heavy lift helicopter is intended to provide
transportation for tactical vehicles, artillery, engineer equipment,
personnel, and logistical support equipment. It is expected to operate in
both day and night. The purpose of the

CH- 47F program is to improve the performance and extend the useful life
of the CH- 47. This effort includes installing a digitized cockpit,
rebuilding the airframe, and reducing aircraft vibration through airframe
stiffening.

Prime contractor: Boeing

Program office: Huntsville, Ala.

FY 2004 funding request: FY 2003 dollars in millions Approved Latest
Percent R& D $0 million 5/ 98 10/ 02 change

Procurement $516.0 million Research & development cost $148.0 $169. 6 14.6
Quantity 16 aircraft

Procurement cost $2, 613.9 $5,927. 7 126.8 Total program cost $2, 761.9
$6,097. 3 120.7

Program unit cost $9. 145 $17.986 96.7

Total quantities 302 339 12.3 Acquisition cycle time (months) 82 99 20.7

The CH- 47F helicopter began low- rate production in December 2002,
although key production processes were not in control. Program officials
believe that CH- 47F production is low risk because no new technology is
being inserted into the aircraft, two prototypes have been produced, and
the production process has been demonstrated during the

development phase. The CH- 47F technologies and design appear mature,
although a low percentage of engineering drawings were released at the
design review. Production unit costs have more than doubled due to
contractor rate increases, increases in system capabilities, and initial
underestimation of program cost.

CH- 47F Program Technology Maturity Although we did not assess technology
maturity in detail, the CH- 47F is a modification of the existing CH- 47D
helicopter. Program officials believe that all critical technologies are
mature and have been demonstrated prior to integration into the CH- 47F
development program.

Design Maturity

The CH- 47F design is complete, with 100 percent of the drawings released
to manufacturing. However, at the design review only 37 percent of the
system*s engineering drawings were complete. Since that time, the number
of drawings completed has increased substantially. The majority of the new
drawings were

instituted to correct wire routing and installation on the aircraft,
changes program officials believed could not be determined until after the
first prototype was developed.

Production Maturity

CH- 47F production maturity could not be determined because the program
does not use statistical process control to ensure that production
processes are stable. Program officials believe the production is low risk
because two prototypes have been produced and the production processes
have been demonstrated during the development phase. The Army plans to
conduct

operational testing in fiscal year 2004 to demonstrate its readiness to
proceed into full- rate production. Prior to that decision, the Army plans
to complete a risk assessment for the CH- 47F to eliminate any production
risk that remains.

Other Program Issues

Both the total cost and the program unit cost for the CH- 47F production
program have more than doubled. This growth triggered a Nunn- McCurdy
breach (see 10 U. S. C. 2433) in December 2001, requiring a review by the
Secretary of Defense and a report to Congress. As a result, the Secretary
of Defense has certified to Congress that the CH- 47F is essential for
national

security, there are no alternatives, the new cost estimates are
reasonable, and the management structure is in place to continue to keep
costs under control. According to the program office, the cost increases
were due to (1) prime contractor labor rate increases and material cost
growth, (2) additional system capabilities required by the Army, (3)
recapitalization of 36 Special Operations aircraft,

and (4) initial underestimation of program costs. According to the program
manager, the Army has fully funded the program*s cost growth of about $2.5
billion (then- year dollars). This increase in program cost necessitated
rebaselining the CH- 47F program. The Army approved the CH- 47F
acquisition program baseline.

Program Office Comments

The CH- 47F program office generally concurred with this assessment.

RAH- 66 Comanche

The Army*s Comanche is a multi- mission helicopter intended to perform
tactical armed reconnaissance. It is designed to operate in adverse
weather across a wide spectrum of threat environments and provide improved
speed, agility, reliability, maintainability, and low observability over
existing helicopters. It is also expected to lower operating costs through
the use of integrated diagnostics, a composite airframe, and a

bearingless rotor system. It will replace the AH- 1, OH- 6, and OH- 58
helicopters.

Prime contractor: Boeing- Sikorsky

Program office: Huntsville, Ala.

FY 2004 funding request: FY 2003 dollars in millions Approved Latest
Percent R& D $1. 1 billion 7/ 00 10/ 02 change

Procurement $0 million

Research & development cost $8, 886.4 $12,556. 3 41.3 Quantity 0 aircraft

Procurement cost $30,550.9 $21,939. 5 -28.2 Total program cost $39,824.0
$34,545. 0 -13.2

Program unit cost $32.831 $53.146 61.9

Total quantities 1,213 650 -46.4 Acquisition cycle time (months) 222 250
14.9

Most of the Comanche*s critical technologies have demonstrated acceptable
levels of maturity, and the program appears very close to meeting the best
practice standard for a stable design. This level of maturity follows many
years of difficult development. Since the program*s first cost estimate
was originally approved in 1985, the research and development cost has
almost quadrupled and the time to obtain an initial capability has
increased from 9 years to over 21 years. The program has recently
undergone another major restructuring to incorporate an

evolutionary acquisition approach and reduce concurrency and lower overall
risk. This restructuring shows promise of being a knowledgebased program
that matches program resources with user requirements.

RAH- 66 Comanche

program requirements with force requirements and program risks. Weight
issues were addressed through

Technology Maturity

increased engine performance. Initial operational Seven of the Comanche*s
eight critical technologies

capability was moved from December 2008 to are considered mature. Only one
critical technology, September 2009 to reduce risk and significantly the
radar cross- section needed for low observability, increase the amount of
testing conducted. requires additional development. The Army expects

Program Office Comments

that this technology will reach maturity in fiscal year 2005, a year
before the production decision.

In commenting on a draft of this assessment, program officials generally
concurred with our assessment.

Design Maturity

They added that in October 2002, the Office of the The Comanche program
has released 73 percent of the Secretary of Defense approved the Comanche
engineering drawings to manufacturing. The program program as an
evolutionary acquisition approach. The

has improved its ability to reach design maturity by Comanche quantity was
reduced from 1213 to 650 rescheduling the design review from July 2002 to
April

based on emerging results of the Comanche*s role in 2003. The program
estimates that it will complete

the Objective Force. This reduction in quantities, 90 percent of the
drawings by the design review under

combined with the research and development cost the proposed plan, instead
of the former 59 percent

growth, resulted in a program acquisition unit cost under the previous
program.

increase of approximately 62 percent. Excluding impacts of the quantity
reduction, the average Critical technologies have not yet been integrated
and

procurement unit cost increased 18 percent and the demonstrated on the
Comanche airframe. Prior to the

program acquisition unit cost increased 23 percent. proposed program
restructure, integration of critical technologies was considered high
risk, even though most of the technologies had reached maturity on other
platforms. Program officials believe that the restructured program reduces
integration risks and that the longer development schedule will allow for
reduced concurrent development and additional integration time and
facilities, thereby reducing critical risks. The longer schedule also
provides additional time for

near- term development testing, use of a production representative
aircraft for initial operational testing, and full qualification testing.
Additionally, the phasing of development and operational tests was revised
and expanded to reduce overall program risk.

Other Program Issues

Continuing cost and schedule issues have led to the most recent
restructuring of the program. In October 2002, the Office of the Secretary
of Defense approved the Comanche program to continue under

an evolutionary acquisition approach. However, because of uncertainties
with future funding and capabilities, quantities were reduced from 1213 to
650 aircraft. This reduction in quantities, combined with the research and
development cost growth, resulted in a unit cost increase of approximately
62 percent.

Program officials stated that the restructuring added a more robust
internal review process and balanced

EX- 171 Extended Range Guided Munition (ERGM)

The Navy*s ERGM is a rocket- assisted projectile that is fired from a gun
aboard ships. It can be guided to land targets at ranges of between about
10 and 50 nautical miles to provide fire support for ground troops. ERGM
is expected to offer increased range and accuracy compared to the Navy*s
current gun range of 13 nautical miles. ERGM requires modifications to
existing 5- inch guns, a new munitions- handling system (magazine), and a
new fire control system. We assessed the projectile only.

Prime contractor: Raytheon

Program office: Washington, D. C.

FY 2004 funding request: FY 2003 dollars in millions Approved Latest
Percent R& D $28.6 million

4/ 97 12/ 02 change

Procurement $3.8 million

Research & development cost $77.6 $326. 1 320.5 Quantity 0 rounds

Procurement cost $308.4 $159. 4 -48.3 Total program cost $386.0 $485. 5
25.8

Program unit cost $0. 045 $0.150 233.8

Total quantities 8,570 3230 -62.3 Acquisition cycle time (months) 50 121
146.9

The ERGM program began product development with very few of its critical
technologies mature according to the best practices standards. While
significant progress has been made in the past 7 years, program officials
do not expect to achieve maturity on all critical technologies until after
the design review. No production representative engineering drawings had
been released at the time of our assessment, and none are projected by the
system design review. The program office currently expects to release
these 1 year later. In June 2002, the program conducted a successful test
of a guided tactical round under realistic launch conditions. This test
did not evaluate the performance of a new

warhead design.

ERGM Program

fiscal years 2002 and 2003, stretched out program milestones and will
delay deployment of ERGM until Technology Maturity

2006. Fourteen of ERGM*s 20 critical technologies have

Other Program Issues

demonstrated technological maturity. The remaining 6 technologies are
approaching maturity, and program

Future program costs are not accurately reflected in officials expect that
all 20 critical technologies will be the latest program cost estimate and
the fiscal year demonstrated in an operational environment by the

2004/ 2005 budget request. The cost estimate is based end of 2003,
approximately 7 months after the design

on a much lower production quantity than is contained review. Three of the
technologies yet to reach maturity in either the approved or the current
draft revision of are part of the new unitary warhead design, and a

the ERGM acquisition program baseline. The budget fourth is related to
this change. Program officials request does not fully fund the 80
operational test recently identified the unitary warhead*s safe/ arm

rounds currently required. device and fuze as a critical technology, after
a Navy

Two testing issues could affect the program. The safety review concluded
that it needed to be

Director of Operational Test and Evaluation has raised redesigned to meet
applicable DOD standards.

a concern about test range restrictions that could limit The ERGM program
began development with only one

realistic operational testing. Finally, the project of its critical
technologies mature. Having only one manager stated that the availability
of a fully capable critical technology mature at the start of product ship
to support development testing could be an issue development has caused
cost and schedule problems. due to funding shortfalls for magazine
modifications For example, when the program began, none of the

on these ships. components of the rocket motor had been integrated Some of
the cost increases and schedule slippages to into an ERGM representative
design. Subsequent

date may be attributed to the fact that the contractor problems with the
performance reliability of the motor

relocated the program in 1998, resulting in a loss of resulted in cost
growth of more than $13 million.

trained personnel and development inertia.

Design Maturity Program Office Comments

None of ERGM*s approximately 127 production In commenting on a draft of
this assessment, program representative engineering drawings have been

officials stated that although production released to manufacturing. The
program office plans representative drawings will not be available at the
for all of these drawings to be released in June 2004, design review, the
entire ERGM design would be

about one year after the design review. In the under configuration
control. Design maturity at that meantime, the design review will be used
to validate

time will be sufficient to produce all- up rounds for the design of the
development test rounds. The June

land and ship- based development testing. Based on 2004 drawing release,
which will reflect knowledge data obtained from flight and qualification
tests in gained from 8 of 18 flight tests and some qualification fiscal
year 2004, minor revisions to the ERGM tests, will be used to build the 80
production

technical data package may be made. Production representative operational
test rounds. Program representative drawings will be finalized by June
2004. officials pointed out that progress has been made in

Program officials stated that they are highly maturing the design. For
example, the main elements

encouraged by the significant progress in ERGM of the design were
validated during the guided gunfire development activities over the last
18 months. They

test in June 2002. further stated that they have a high degree of In
January 2002, in order to meet lethality and safety

confidence that ERGM will meet all performance requirements, the Navy
decided to make a significant

requirements, while meeting the production cost goals change to the
warhead design, moving from a

specified in the acquisition program baseline. multiple- submunition
design to a single explosive* or unitary* warhead. This decision, coupled
with the decision to stay within planned funding levels for

Excalibur Artillery Round

The Army*s Excalibur is a family of extended range, precision, 155- mm
artillery projectiles. It is designed to increase soldier survivability by
allowing the Future Combat Systems* nonline of sight cannon to fire from
farther away and defeat threats more quickly, while reducing logistic
support. It also is intended to be more effective when fired at urban
targets, through a combination of altered trajectory and global
positioning system accuracy.

Prime contractor: Raytheon Missile Systems

Program office: Picatinny Arsenal, NJ.

FY 2004 request: FY 2003 dollars in millions Approved Latest Percent R& D
$134 million 4/ 97 2/ 03 change

Procurement $0 million

Research & development cost $59.8 $662. 8 1, 008.7 Quantity 0 projectiles

Procurement cost $676.2 $4,135. 9 511.6 Total program cost $736.0 $4,798.
7 552.0

Program unit cost $0. 004 $0.062 1, 578.8

Total quantities 200,000 77,677 -61.2 Acquisition cycle time (months) 160
136 -15.0

The Excalibur program*s three critical technologies are not fully mature,
even though product development began over 5 years ago. The technologies
appear to be approaching maturity, and program officials project
demonstrating

technology and design maturity before the design review in 2005.
Currently, 13 percent of the drawings are at the level that could be
released to

manufacturing. Program officials expect to have a stable design by the
design review. The program has undergone a major restructuring effort. It
has encountered a number of challenges since development began, including
a substantial decrease in planned quantities, a relocation of the
contractor*s plant, limited early funding, technical problems, changes in
program direction, and a merger with another program.

Excalibur Program

Program Office Comments Technology Maturity

In commenting on a draft of this assessment, program officials generally
agreed with the information in this None of the Excalibur*s three critical
technologies are report. However, they provided the following fully mature
according to best practice standards.

clarifying comments. According to program officials, all three have been
Concerning the Excalibur design maturity, program demonstrated in a
relevant environment and are

officials stated that approximately 600 drawings are expected to reach
maturity before the design review in

anticipated at the subsystem level. But because the March 2005. The
Excalibur*s design and requisite

program is still in research and development, no technologies have changed
since product drawings have been officially released to development was
started. The three critical manufacturing. The program is fabricating
hardware technologies for the current design are the guidance

in a research and development environment. control system, the airframe,
and the warhead. The

warhead was not considered a critical technology in 1997 because the
Excalibur design called for a warhead that was under production for other
munitions. Based on Army direction, the program has undertaken development
of a different warhead that is currently undergoing testing.

Design Maturity

About 13 percent of the Excalibur*s engineering drawings are at a level
that could be released to manufacturing. The program office plans to have
all of its drawings complete and released to manufacturing by the design
review in March 2005. However, program officials could not estimate the
total number

of drawings expected.

Other Program Issues

The program has gone through many changes since the beginning of product
development in May 1997. It was almost immediately restructured due to
limited funding, and it was restructured again in 2001. In response to
congressional direction, the program was restructured to merge with the
joint Swedish/ U. S. program known as Trajectory Correctable Munitions.
The merger should help the program deal with design challenges, including
issues related to its folding fin

design. Also, in May 2002, the Office of the Secretary of Defense directed
the program to develop the Excalibur for the Future Combat Systems nonline
of sight cannon and to field it in fiscal year 2008.

Although program officials have not yet released the new cost and schedule
estimates, the net effect of these changes has been to increase the
program*s

schedule and to substantially decrease planned procurement quantities. As
a result, the program*s overall costs and unit costs have dramatically
increased.

aircraft designed to meet fighter escort, Navy's F/ A- 18C/ D and F- 14
aircraft.

Prime contractor: Program office: Boeing Patuxent River, Md.

FY 2004 funding request: R& D $179.0 million

Procurement $3.0 billion

Quantity 42 aircraft F/ A- 18E/ F Super Hornet

The Navy*s F/ A- 18E/ F is a multi- mission tactical interdiction, fleet
air defense, and close air support mission requirements. The program was
approved as a major modification to earlier F/ A- 18 aircraft in

1992. It is intended to complement and replace the

FY 2003 dollars in millions Approved

Research & development cost Acquisition cycle time (months) 6/ 92 Latest
Program unit cost $67.390 102 12/ 01 Procurement cost $61,215.3 $41,368. 2
Total program cost $67,389.7 $47,549. 8 112 Percent change

$6, 174.4 $6,181. 6 0.1 -32.4 -29.4

$86.770 28.8

Total quantities 1,000 548 -45.2

The F/ A- 18E/ F went into full- rate production in June 2000. Although
the program proceeded

9.8

without obtaining full product knowledge at key decision points, it
embraced the concepts of attaining design and manufacturing knowledge
early in development. The program released just over half of its
engineering drawings by its design review. When low- rate production
began, nearly all of the drawings were released and about 75 percent of
the manufacturing processes were in control. The Navy reduced some program
risk because aviation electronics from an earlier version of the F/ A- 18
were incorporated into the baseline F/ A- 18E/ F. Furthermore, focus was
placed on commonality between the F/ A- 18 C/ D and the F/ A- 18 E/ F,
which further reduced risk.

F/ A- 18E/ F Program

Technology Maturity

We did not assess the technology maturity of the F/ A- 18E/ F program
because it is already in full- rate production. Nevertheless, we did not
identify any technical challenges during the development of this aircraft
in our previous reviews.

Design Maturity

The F/ A- 18E/ F design appears complete. The program has released 100
percent of the design drawings to manufacturing. At the time of the
critical design review in July 1994, 56 percent of the engineering

drawings were completed and released to manufacturing for aircraft
structure and systems. According to program officials, they decided to
proceed despite the low level of completed drawings because the knowledge
gathered from earlier F/ A- 18C/ D models gave them confidence that the

design was stable. By the time of the low- rate initial production
decision, 99 percent of the drawings had been released.

Production Maturity

According to program officials, they currently have 100 percent of their
critical manufacturing processes under control, according to the best
practice standard. Therefore, they are no longer tracking processes using
statistical process control. However, defects are still

monitored through inspections, failures, and age exploration testing, and
during maintenance. If production problems are identified, the program
would resume statistical process control analysis where necessary.

Program officials estimate that about 75 percent of key manufacturing
processes were in control at the low- rate production decision in March
1997. Program officials stated that they concentrated on maturing their
manufacturing processes before starting

production. As a result of these efforts, labor efficiency rates have
steadily improved.

Other Program Issues

The F/ A- 18E/ F will not reach its full potential until after the
incorporation of several preplanned upgrades* the Active Electronically
Scanned Array (AESA) radar, the Joint Mounted Helmet Cueing System coupled
with the AIM- 9X missile, and the Advanced Targeting Forward Looking
Infrared sensor.

The level of effort and timing to incorporate some of the sensors* the
AESA radar and the Advanced Targeting sensor* may prove to be a challenge.
We have assessed the AESA radar elsewhere in this

report.

Program Office Comments

In commenting on a draft of this assessment, program officials stated
initial schedule delays were due to a procurement reduction of 10 aircraft
in a 1998 Program Objective Memorandum. Since that time, the contractor
has consistently delivered aircraft ahead of

schedule. Program officials also noted that the aircraft are demonstrating
two to three times the quality of the F/ A- 18C/ D and have provided
measurable

improvements to squadron readiness. In addition, all F/ A- 18E/ F
preplanned upgrades continue to track to their program schedules. The
Joint Mounted Helmet Cueing System has completed operational evaluation,
and the system has been incorporated into lot 24 of the aircraft
(deliveries of which began in September 2001). Deliveries of the Advanced
Targeting Forward Looking Infrared Sensor production units began in

April 2002, and the units were deployed in January 2003. Finally, program
officials stated that the AESA radar program continues to execute as
planned, and the program has received the first engineering and
manufacturing development unit.

F/ A- 22 Raptor

The Air Force*s F/ A- 22, originally planned to be an air superiority
fighter, will also have air- to- ground attack capability. It is being
designed with advanced

features, such as stealth characteristics, to make it less detectable to
adversaries and capable of high speeds for long ranges. It also has
integrated aviation electronics (avionics) designed to greatly

improve pilots' awareness of the situation surrounding them. It is
designed to replace the Air Force*s F- 15 aircraft.

Prime contractor: Lockheed Martin

Program office: Dayton, Ohio

FY 2004 funding request: FY 2003 dollars in millions Approved Latest
Percent R& D $936.5 million

2/ 92 12/ 02 change

Procurement $4.2 billion

Research & development cost $20,938.2 $30,836. 1 47.3 Quantity 22 aircraft

Procurement cost $54,272.2 $39,049. 2 -28.0 Total program cost $75,461.2
$70,469. 4 -6.6

Program unit cost $116.5 $253. 5 117.7

Total quantities 648 278 -57.1 Acquisition cycle time (months) 203 230
13.3

Because the F/ A- 22 Program Office stopped collecting process control
data in 2000, the program began production in 2001 with no proof that

processes were in control, as defined by best practice standards.
Technology appears mature and the design appears stable; however, problems
with the vertical tail and the avionics have been discovered recently,
which require design

modifications. Delays in capturing technology, design, and production
knowledge and these latest problems contributed to cost increases and
schedule delays. The potential exists for further cost increases and
schedule delays as a significant amount of the test program remains,
including operational tests. Also, the latest production cost estimate is
likely to increase because of several factors, and the estimate assumes
over $25 billion in offsets from cost reduction plans.

F/ A- 22 Program

practice standards. In September 2001, the Air Force awarded a contract
for 10 aircraft to begin F/ A- 22 Technology Maturity

production. Although we did not assess the F/ A- 22 key

Other Program Issues

technologies using technology readiness levels, the three critical
technologies (supercruise, stealth, and

In September 2001, the Air Force acknowledged an integrated avionics)
appear mature. Two of these

estimated production cost increase of $5.4 billion technologies,
integrated avionics and stealth, were (then- year dollars) over the
congressional cost limit. late to mature. It was not until September 2000,
or

We believe conditions exist that makes it likely over 9 years into product
development, that the

production costs will increase even further. In integrated avionics
reached maturity. During

addition, the Air Force is counting on over $25 billion development, the
integrated avionics was a source of

in cost reduction plans to offset estimated cost growth schedule delays
and cost growth. Since 1997, avionics

and enable the program to meet the production cost software development
and flight- testing have been

estimate. If these cost reduction initiatives are not delayed, and the
cost of avionics development has

achieved as planned, production costs could increase. increased by over
$980 million. Moreover, the Air

Further, the contractor has yet to demonstrate it can Force did not
complete an evaluation of stealth

efficiently build the development aircraft, and technology on a full-
scale version of the aircraft until

estimates of the cost to build the production aircraft several years into
product development.

continue to increase.

Design Maturity

In December 2002, DOD estimated development costs would increase by $876
million and that the funding The basic design of the F/ A- 22 is
essentially complete,

necessary to cover this cost increase would be as engineering drawings are
complete. However, transferred from production funding. Avionics design
changes have been necessary as a result of

problems discovered in flight- testing are the primary flight tests and
structural tests. For example, problems

contributor to a six- month extension to the with excessive movement of
the vertical tails and

development program. avionics failures in flight tests were discovered,
and they will require costly design modifications. The Air

Program Office Comments

Force still has to complete a significant amount of development testing
and operational testing. Until In commenting on a draft of this
assessment, program initial operational testing is completed as planned in

officials stated that the report implies that had the F/ A22 June 2004,
the possibility of additional design changes

deferred product development until engineering remains. and testing were
accomplished, at a level providing higher product knowledge, substantial
cost increases Design knowledge for the F/ A- 22 was built slowly.

and schedule delays would have been prevented. The Only 26 percent of the
total drawings were released at

issues cited as examples do not pose a substantial risk the 1995 design
review. The program released

to either cost or schedule and have either been fixed 90 percent of the
drawings over 3 years later, after the through minor design change or are
anticipated to be first two development aircraft had been delivered.

resolved without major impact to continued testing Late drawing release
contributed to parts shortages

and production. A program of this nature is expected and work performed
out of sequence during assembly,

to have both design and technological maturities to which drove up costs
and contributed to delaying overcome and there will be some element of
risk flight tests by 83 months.

throughout its development and into the production process. Production
Maturity

The program office stopped collecting process control information in
November 2000. The contractor estimated that nearly half of the key
processes had reached a marginal level of control, but not up to best

Joint Air- to- Surface Standoff Missile (JASSM)

JASSM is a joint Air Force and Navy program designed to attack surface
targets outside of the range of area defenses. JASSM will be delivered by
a variety of aircraft, including the F- 16 C/ D, the B- 52H, the F/ A-
18E/ F, the B- 2, and the B- 1B. The

system includes the missile, software, and software interfaces with the
host aircraft and mission planning system.

Prime contractor: Lockheed Martin

Program office: Eglin, Fla.

FY 2004 funding request FY 2003 dollars in millions Approved

Latest Percent R& D $56.3 million 11/ 98 12/ 02 change

Procurement $102.5 million Research & development cost $863.6 $1,209. 5
40.1 Quantity 250 missiles

Procurement cost $1, 078.7 $2,568. 4 138.1 Total program cost $1, 962.9
$3,777. 8 92.5

Program unit cost $0. 795 $0.852 7.2

Total quantities 2,469 4,434 79.6 Acquisition cycle time (months) 75 87
16.0

The JASSM program entered production in December 2001 without ensuring
that production processes were in control, according to best practice
standards. However, program officials indicated that they have
demonstrated the production processes and that they sample statistical
data at the subsystem level. The program ensured that the technology was
mature and that

the design was stable at critical points in development, closely tracking
best practice standards. Redesign remains one area of concern because
recent test failures have led to the delay of

operational tests. The program has identified fixes to the problems, and a
retrofit plan is in progress. The contractor*s ability to attain a higher
production rate is another area of concern.

JASSM

capacity and expanding facilities to support full- rate production plus
anticipated foreign military sales.

Technology Maturity

Program officials believe that none of the The JASSM program used existing
technologies and so manufacturing processes that affect critical system
its level of technology maturity is high. Although none

characteristics are problematic, although there are of the subsystems
involve new technologies, three

key production processes that have cost implications, critical
technologies are new applications of existing

such as the bonding for the low observable materials technologies. These
three technologies are the global

and the painting/ coating application. positioning system anti- spoofing
receiver module, the

Program Office Comments

low observable technology, and the composite materials. The program office
reports these

In commenting on a draft of this assessment, program technologies to be
mature.

officials stated that JASSM has established a new benchmark for missile
development by ensuring

Design Maturity

weapon system design maturity and production The contractor has released
100 percent of the capability were demonstrated during development

drawings to manufacturing. The two remaining prior to entering low- rate
initial production. JASSM*s concerns are the software for the missile and
the

acquisition strategy incorporated existing technology status of
integration with aircraft, although program to reduce program risk and
speed up delivery of the officials believe the risks are low. weapon to
the warfighter. The officials further stated

that JASSM*s development cycle is 33 percent faster Recent failures in
development and operational tests than comparable weapon systems, with
production have led to the delay of the remaining JASSM unit prices 50
percent less than weapon systems with operational tests. During an
operational test on

less capability. The contractor was contracted to October 10, 2002, the
missile flew its planned route

produce 82 all- up production prove- out test rounds and penetrated the
target, but it failed to detonate.

during development on the production line prior to According to program
officials, this failure occurred

low- rate initial production missile delivery. Program because the
requested test methodology was

officials noted that establishment of production experimental and exceeded
original design representative hardware during development was key
requirements for the fuze. On October 24, 2002, during to the contractor*s
ability to prove out all production the last of 11 developmental tests,
the missile went out

processes. The contractor has a capitalization plan to of control and
crashed at the test site. According to

meet full- rate production quantities. program officials, this failure was
due to a failed actuator. Program officials believe they have identified
the problems in both cases and have a retrofit plan.

Retrofits will be tested in spring 2003. However, if additional problems
occur, they will have to be corrected while JASSM is in production, which
may require additional retrofitting of missiles already produced.

Production Maturity

Program officials do not collect production process control data at the
system level. However, they stated that all production processes had been
demonstrated and that statistical data is collected at the subsystem level
and is sampled as required. Program officials indicated that the
contractor will produce at the rates required for the first production lot
and 76 missiles will be delivered. A contract for the second lot, 100
missiles, has been signed. Production concerns remaining include achieving
full- rate production

Joint Common Missile

The Joint Common Missile is an air- launched and potentially a ground-
launched missile designed to target tanks; light armored vehicles; missile
launchers; command, control, and communications vehicles; bunkers; and
buildings. It is designed to provide line of sight and beyond line- of-
sight

capabilities. It can be employed in a fire- and- forget mode* providing
maximum survivability* or a precision attack mode, providing the greatest
accuracy. The Joint Common Missile will be a joint Army and Navy program
with USMC participation.

Prime contractor: In competition

Program office: Huntsville, Ala.

FY 2004 funding request: FY 2003 dollars in millions

Latest Percent R& D $183.8 million Approved 11/ 02 change

Procurement $0 million

Research & development cost NA $563.92 0 Quantity 0 missiles

Procurement cost NA $1,597. 3 0 Total program cost NA $2,161. 2 0

Program unit cost NA TBD 0

Total quantities NA 8,425 0 Acquisition cycle time (months) NA 60 0 Note:
Funding from FY 2004 President*s Budget. Total Army and Navy Acquisition
Objective is 77,400. Official cost position to be finalized by 6/ 2003. NA
= not applicable.

The Joint Common Missile is scheduled to enter product development before
any of its critical technologies are fully mature, according to best
practices. Furthermore, program officials currently project that the
critical technologies will not reach maturity until a year after the
design review. The Army will initially focus development on an airlaunched
version.

Joint Common Missile System

risk. Prototype testing of a multi- mode seeker (tower and captive
flight), a multi- purpose warhead (heavy

Technology Maturity

armor and building structures), and a rocket motor None of the Joint
Common Missile*s three critical

(high maximum to minimum thrust profiles over technologies have
demonstrated full maturity. These operational temperatures) is currently
being critical technologies include a multi- mode seeker for

conducted with results to be available in sufficient increased
countermeasure resistance, a boost- sustain

time to support the milestone decision to begin the propulsion for
increased standoff range, and a multipurpose development phase. warhead
for increased lethality capability.

Program officials noted that many of the components of these technologies
are currently in production on other missile systems, but that they have
not been fully integrated. While backup technologies exist for each of the
critical technologies, substituting any of them would result in degraded
performance or increased costs.

Design Maturity

Program officials project that full integration of the subsystems into the
Joint Common Missile will be mature one year after the system design
review, which is scheduled for July 2004.

Other Program Issues

The current cost estimates are from the fiscal year 2004 President*s
budget. This cost estimate will be updated at the conclusion of the Army's
formal estimating process. The formal estimating process began in January
2003 for presentation at the milestone decision review in September 2003.
According to program officials the Army's acquisition objective is 54, 400
missiles and the Navy's acquisition objective is 23,000. Program officials
also indicated that the modular design will reduce life- cycle costs,
including demilitarization, and will enable continuous

technology insertion to ensure improvements against advancing threats.

Program Office Comments

In commenting on a draft of this assessment, program officials stated that
they plan to demonstrate the technological maturity required by DOD
acquisition system policy before beginning the development phase in
September 2003. Program officials further stated that the technological
maturity projected represents a major achievement in the technology's
demonstrated readiness in a relevant environment and provides the critical
technologies the maturity necessary to accomplish system integration of

demonstrated subsystems, thereby reducing program

Joint Primary Aircraft Training System (JPATS)

JPATS is a joint acquisition by the Air Force and the Navy to replace the
aging primary trainer aircraft fleet. JPATS is a variant of the Beech
Pilatus PC- 9 commercial aircraft, but it has been modified significantly
to incorporate military unique requirements. The JPATS program includes
the aircraft; the ground- based training system (simulators, course
materials), and an integrated training management system. We assessed the
aircraft.

Prime contractor: Raytheon

Program office: Wright Patterson AFB, Ohio

FY 2004 funding request: FY 2003 dollars in millions Approved Latest
Percent R& D $0 million

8/ 95 12/ 01 change

Procurement $283.0 million Research & development cost $349.1 $294. 2
-15.7 Quantity 52 aircraft

Procurement cost $2, 720.3 $4,316. 8 58.7 Total program cost $3, 138.6
$4,674. 8 49.0

Program unit cost $4. 408 $5.970 35.4

Total quantities 712 783 10.0 Acquisition cycle time (months) 97 113 16.5

The JPATS aircraft entered full production in December 2001 without
ensuring that the manufacturing processes were mature. The aircraft
entered limited production in 1995 before achieving design stability. DOD
considered the aircraft a

mature commercial product that did not require extensive product
development. However, program officials underestimated the number of
design changes needed to accommodate the military unique requirements. The
design has subsequently changed about 70 percent from the commercial
baseline. The JPATS initial operating capability occurred in 2002, 2 years
later than originally planned.

JPATS Program

Program Office Comments Technology Maturity

In commenting on a draft of this assessment, program officials disagreed
with our analysis of production Although we did not assess the JPATS
aircraft key

maturity. They stated that statistical process control is technologies,
the aircraft is a derivative of a

not the only determinant of maturity. The production commercial aircraft
and the technologies appear

line was certified by the International Organization for mature.

Standardization in 1994 and by the Federal Aviation Administration in
1999, and is currently producing

Design Maturity

aircraft according to these guidelines. Assembly labor The basic design of
the aircraft is currently complete. hours per aircraft are on a 78 percent
learning curve, However, the military unique design was only about and
they have decreased 65 percent since the first

5 percent complete shortly after the program was operational aircraft was
delivered. The production line

approved to enter limited production in 1995. The rate increased to five
aircraft per month by the end of design has changed about 70 percent from
its

2002, and remains there still, even as design changes commercial baseline.
Testing has revealed tangible

are incorporated into the production line. After initial examples of
design immaturity. Several subsystems,

production difficulties, over the past year the including the engine, the
UHF radio, and the

contractor has been delivering aircraft ahead of environmental control
system, have required schedule while incorporating engineering changes to

extensive modification or redesign. These and other increase the
suitability of the system. Program

problems have delayed both aircraft testing and the officials also stated
that the cycle time should be

production decision. reduced by 6 months because the JPATS program was

unable to award a contract or proceed with contract In November 2001,
operational testers concluded that

performance pending the disposition of several bid JPATS was operationally
effective but not

protests. operationally suitable. They cited concerns about the aircraft's
reliability, availability, and maintainability. GAO Comments

They also reported that the full JPATS had not yet been tested due to
uncorrected deficiencies in the

Our prior work has shown that leading commercial aircraft and the
immaturity of the software- intensive

firms rely on statistical control data as the best training information
management system. The

indicator of production readiness. Despite its contractor is incorporating
changes to the aircraft as a

commercial origins, the JPATS program entered result of operational test
issues. Operational testers

limited and full production without this information. expressed concern
that some changes may adversely Subsequent testing has uncovered numerous
problems impact other critical subsystems. Despite these issues,

that require modification and retrofit. Although the the Air Force
proceeded into full- rate production the

aircraft has been production certified by the Federal following month.
Aviation Administration, its regulations merely require

the contractor to maintain a generic quality control

Production Maturity

system and do not provide assurance that the components can be built
within cost and on schedule. Production maturity remains at issue because
We used DOD official documents to determine information about the
contractor's manufacturing

acquisition cycle time. process controls is not available. The Air Force
did not require this information because the aircraft was considered a
commercial derivative.

Other factors could affect production maturity. In 2002, two key
modifications* the environmental control system and the UHF radio* began
to be incorporated on the aircraft. The program office has also identified
additional retrofit requirements and is evaluating a replacement for the
collision warning system. The rework associated with these changes may
affect aircraft production efficiencies.

F- 35 Joint Strike Fighter (JSF)

The JSF program goals are to develop and field a family of stealthy,
strike fighter aircraft for the Navy, Air Force, Marine Corps, and U. S.
allies, with maximum commonality to minimize life- cycle costs. The
carrier suitable version will complement the

Navy F/ A- 18 E/ F. The Air Force version will primarily be an air- to-
ground replacement for the F- 16 and the A- 10 and complement the F/ A-
22. The short take- off and vertical landing version will

replace the Marine Corps F/ A- 18 and AV- 8B. Significant foreign military
purchases are expected.

Prime contractor: Lockheed Martin

Program office: Arlington, Va.

FY 2004 funding request: FY 2003 dollars in millions Approved Latest

Percent

R& D $4. 4 billion 10/ 01 12/ 01 change

Procurement $0 million

Research & development cost $32,788.6 $32,880. 8 0.3 Quantity 0 aircraft

Procurement cost $145,733.8 $147,604. 7 1.3 Total program cost $180,047.0
$180,485. 5 0.2

Program unit cost $62.8 $63. 0 0.2

Total quantities (U. S. only) 2,866 2,866 0.0 Acquisition cycle time
(months) 185 185 0.0

The JSF program entered the development phase without demonstrating that
its eight critical technologies had reached maturity according to best
practice standards. Two technologies, propulsion and critical fabrication
techniques, were very close to maturity. DOD conducted an independent
review in 2001 and concluded that the technology maturity was sufficient
to proceed into product development. The JSF program no longer focuses on
the previous 8 technology areas, instead it uses a different method of
integration and risk management that currently tracks 23 program level
risks. We were unable to assess the new risk areas, but program data
indicates that the majority are

moderate risk. The program expects to have 80 to 90 percent of its
critical build- to- packages completed by the final design review in 2005.

JSF Program

JSF preliminary design review in late March 2003. We were unable to review
the results of those meetings

Technology Maturity

prior to the release of this report, but program office During its concept
development phase, the Joint

data indicates the discovery of higher risk levels for Strike Fighter had
eight critical technologies: short

the propulsion system and overall aircraft weight. take- off vertical
landing/ integrated flight propulsion Other Program Issues

control, prognostic and health management, integrated support systems,
subsystems technology, Due to the highly complex nature of the JSF design,
integrated core processor, radar, mission systems

the Director, Operational Test and Evaluation, expects integration, and
manufacturing. We reported in

numerous test challenges for the program. These May 2000 and again in
October 2001 that low levels of

challenges include the integration of highly advanced maturity in these
technologies could increase the sensors with the avionics systems,
vertical thrust likelihood of program cost and schedule growth.

capability for the Marine Corps version, and performance and maintenance
requirements of the The program experienced cost growth and schedule low
observable capabilities. The program has received concerns during the
concept demonstration phase,

authority for its low- rate production quantity to reach prior to starting
product development in October 2001.

15 percent* 427 aircraft* of the total production run. This included
manufacturing delays for hardware used

on the propulsion system for the Marine Corps

Program Office Comments

version. To reduce cost and schedule delays, the program eliminated
planned risk- reduction efforts and

In commenting on a draft of this assessment, program delayed other
technology demonstrations until after

officials stated that, prior to the start of the product development
began.

development phase, JSF*s key technologies had gone through an extensive
series of tests and An independent review performed by DOD in 2001,

demonstrations, culminating in four experimental using a different method
than technology readiness

aircraft proving flight capabilities for each service levels, concluded
that the overall technology maturity

variant in over 200 hours of flight. An independent of the JSF program was
sufficient to enter into DOD review concluded that JSF had demonstrated
product development. Today, the program no longer

sufficient technical maturity for low risk entry into the monitors the
eight specific technologies from the

development phase. For this phase, the program previous phase. Instead,
the program is using

officials stated that JSF has adapted the contractor's Lockheed Martin*s
Key System Development

risk mitigation approach. Risk mitigation assessments Integration approach
to monitor overall technology

in February 2003 indicated that most program level development and design
integration. Further, the

risks were rated moderate using the contractor*s program tracks 23 program
level risk areas and has approach. Cost and schedule planning for the
assessed 19 as moderate and 2 as high. Five of eight

development phase has evolved as the services critical technologies from
the concept development

iterated system operational requirements with life phase are contained
within elements of these program

cycle cost. The JSF air system preliminary design level risks. We have not
evaluated the current JSF

review is scheduled in March 2003, and the first of technique for
assessing risks.

three critical design reviews is to occur in April 2004. Finally, program
officials stated that the program is

Design Maturity

being executed in accordance with its cost, schedule, The program has
committed time and funding to the and technical baselines. system
development and demonstration phase that should improve its chances for
success. Specifically, the new program structure will now include
additional test aircraft, increased software on the aircraft, and a
greater number of flight test hours. Program documents indicate that the
1996 estimated cost and schedule for JSF*s development phase have
increased by 56 percent and 40 percent, respectively, due to

changes in program scope. Meetings were held for the

Joint Standoff Weapon (JSOW)

JSOW is a joint Air Force and Navy guided bomb to attack targets from
outside of the range of most enemy air defenses. There are three JSOW
variants that use a common air vehicle. Two variants (JSOW A and B) carry
submunitions to attack soft targets or armored vehicles. The unitary
variant (JSOW C) uses a seeker, autonomous targeting acquisition software,
and a single warhead to attack targets. We assessed the unitary variant
and the common air

vehicle.

Prime contractor: Raytheon Systems Company

Program office: Patuxent River, MD

FY 2004 funding request: FY 2003 dollars in millions Approved Latest
Percent R& D $0.8 million

6/ 92 12/ 01 change

Procurement $65.89 million

Research & development cost $325.2 $311. 5 -4.2 Quantity 175 missiles

Procurement cost $3, 871.4 $803. 8 -79.2 Total program cost $4, 196.6
$1,115. 3 -73.4

Program unit cost $0. 538 $0.372 -30.9

Total quantities 7,800 3,000 -61.5 Acquisition cycle time (months) 89 112
25.8

The JSOW program is scheduled to begin low- rate production in March 2003
without knowing that production processes are in control, according to
best practice standards. The program instead relies on an after-
production process of inspection to discover defects. Immature technology
at the start of development at least partially delayed design maturity,
and developmental testing of the seeker is not complete.

JSOW Program Program Office Comments Technology Maturity

In commenting on a draft of this assessment, program officials stated that
the contractor has completed The JSOW unitary*s technology appears mature.
The

17 consecutive months of on- schedule deliveries, program office
identified the imaging infrared seeker increasing the inventory to over
850 combat ready with the autonomous acquisition software as the only
assets. In addition, program officials noted that the Air critical
technology for the system. The seeker was not

Force has upgraded its JSOW inventory to mission mature at the start of
development, but it did

ready as a result of a successful resolution of demonstrate maturity in
October 2001* over threefourths remaining manufacturing, navigation, and
vibration through development* when it was flown

tolerance issues. The JSOW unitary continues aboard an aircraft in a
captive flight test. Program

development and its performance is being monitored officials stated that
in three free- flight tests, the

by the program office. seeker's performance substantially exceeded
requirements.

Design Maturity

The JSOW unitary variant*s basic design appears complete. At the system
design review in May 2002, the program office had completed 99 percent of
the drawings. The Navy included nine developmental tests in its
development program* three sled tests with the warhead, three free flights
with the seeker, and three combined warhead/ seeker tests. The Navy has

completed two of the warhead sled tests and the seeker free- flight tests.

Production Maturity

JSOW production maturity could not be determined because the contractor
does not use statistical process controls to ensure that production
processes are stable and units are produced with few, if any, defects.
Rather, the contractor uses a process of postproduction inspection to
control production quality. The contractor collects this postproduction
data on a

factorywide basis that includes JSOW production but is not specific to it.

According to the program office, the contractor delivered end items in the
past that included manufacturing defects. The program office attributes
these defects at least partially to suppliers and to reorganization and
relocation of the prime contractor to Tucson, Arizona. To mitigate the
risk of further manufacturing problems, the Navy has instituted a series
of reviews of major suppliers. The Navy will conduct an additional
production readiness review after the low- rate production is approved.
Program officials report that the contractor is meeting its revised
production schedule and that the scrap and rework rates remain low.

National Polar- orbiting Operational Environmental Satellite System
(NPOESS)

The NPOESS is a joint National Oceanic and Atmospheric Administration
(NOAA), DOD, and National Aeronautics and Space Administration satellite
program to monitor the weather and

environment. Current NOAA and DOD satellites will be merged into a single
national system (NPOESS), with projected savings of at least $1.3 billion.
The program consists of five segments: space; command, control, and
communications; interface data processing; launch; and field terminals.
Prime contractor: Northrop Grumman Space Technology

Program office: Silver Spring, Md.

FY 2004 funding request: FY 2003 dollars in millions Approved Latest
Percent R& D $544.4 million 8/ 02 9/ 02 change

Procurement $0 million

Research & development cost $4, 029.2 $4,431. 6 10.0 Quantity 0 satellites

Procurement cost $1, 155.6 $1,264. 5 9.4 Total program cost $5, 628.2
$6,183. 4 9.9

Program unit cost $938.0 $1,030. 6 9.9

Total quantities 6 6 0.0 Acquisition cycle time (months) 172 174 1.2

The NPOESS program entered product development in August 2002 with most of
its technologies mature. The program also completed a significant portion
of the engineering drawings well in advance of the design review; however,
the total number has yet to be determined. Over 5 years ago, program
officials considered the program to have

several high- risk areas. Since then, officials have implemented several
efforts, which are expected to reduce all program areas to low risk by the
first NPOESS launch, currently scheduled for the 2008- 2009 time frame.
Perhaps the most significant step taken to reduce risk was to put the
pacing space sensor technologies into full development in advance of the
satellite system itself.

NPOESS Program

radiometer suite, the cross- tracked infrared sounder, the advanced
technology microwave sounder and the

Technology Maturity

ozone mapper/ profiler suite. This satellite will provide The NPOESS
spacecraft and the sensors under

the program office and the data processing centers development consist of
14 key technologies; twelve

with an early opportunity to work with the sensors, were mature at the
start of development in August

ground control, and data processing systems, thus 2002. allowing lessons
learned to be incorporated into the NPOESS satellites. In 1997, the
program office determined that the space segment had high cost and
technical risks and that the

Program Office Comments

interface data processing segment and overall system The NPOESS integrated
program office concurred

integration effort had high cost, schedule, and with this assessment.

technical risks. To reduce the risk to the data processing segment, two
contractors selected for program definition and risk reduction each
conducted four ground- based demonstrations of the data processing
hardware and

software components. Therefore, the program office expects the data
processing segment to be relatively mature before product development.

Program officials indicated that they achieved maturity by concentrating
on the early development of key individual sensors. The acquisition
strategy focused on maturing key sensor technologies using individual
development contracts structured to demonstrate the maturity of each
sensor through a component- level design review prior to the systemlevel
design review. The two technologies that are not mature are needed for two
key sensors* the crosstrack infrared sounder and the conical microwave
imager/ sounder. However, program officials project that those two
technologies will be mature by the system design review in 2005.

Design Maturity

Although the total number of engineering drawings has yet to be
determined, program officials indicated that at least 52 percent of the
6,829 currently identified drawings were completed and released to
manufacturing by the end of January 2003. Program officials further
project that all of the currently identifiable drawings will be complete
by the system

design review in 2005. The program is taking advantage of a unique
opportunity to demonstrate design maturity. The NPOESS Preparatory
Project, a planned demonstration satellite, is to be launched in 2006,
about 2 to 3 years before the first NPOESS satellite launch. The
demonstration satellite is scheduled to carry four critical sensors* the
visible/ infrared imager

Patriot Advanced Capability 3 (PAC- 3) Program

The Army*s Patriot system is a long- range, highmedium altitude air and
missile defense system. PAC- 3 is designed to enhance the Patriot*s
ability to detect and identify missiles and other targets, increase system
computer capabilities and the number of missiles in each launcher, improve
communications, and incorporate a new hit- to- kill missile. The PAC- 3
system has two primary components, the fire unit and the missile. We
assessed both components.

Prime contractor: Raytheon (prime) Lockheed Martin (missile segment)

Program office: Huntsville, Ala.

FY 2003 dollars in millions Approved Latest Percent FY 2004 funding
request: 2/ 95 12/ 02 change

R& D $174.5 million

Research & development cost $2, 760.7 $4,476. 2 62.1 Procurement $561.6
million

Procurement cost $3, 721.9 $7904. 9 112.4 Quantity 108 missiles

Total program cost $6, 482.6 $12,381. 2 91.0

Program unit cost $5. 170 $10.326 99.8

Total quantities 1,254 1,199 -4.4 Acquisition cycle time 66 136 106.1

The PAC- 3 program currently has only about onefourth of its critical
production processes under statistical control using best practice
standards. Continuing problems with producing and testing the missiles are
partially explained by the absence of process control and partially a
consequence of maturing PAC- 3*s design late in development. Technical and
design challenges disrupted the early part of product development, causing
cost and

schedule increases and delays in attaining production knowledge. PAC- 3*s
basic design is now complete and the technology appears mature. However,
the contractor must increase production earlier than planned because DOD
decided to

accelerate deliveries. This decision may present new production challenges
because the contractor must find and train additional personnel.

Patriot PAC- 3 Program

performance was adversely affected by PAC- 3 missile reliability and
launch failures. According to program

Technology Maturity

officials, there were several anomalies caused by Although we did not
assess the PAC- 3 technologies

manufacturing practices, software, and test hardware. using technology
readiness levels, the system's critical

However, they believe there are no systemic issues technologies appear
mature. However, a key and the anomalies have been corrected. A flight
test to technology, the Ka band seeker, was particularly late validate
these corrections is scheduled for the spring

to mature. The seeker did not mature until 1999, close of 2003.

to the low- rate production decision. Problems The program has adopted an
evolutionary acquisition

experienced during development increased the approach, with production
decisions every 2 to 3

seeker's cost by 76 percent and delayed the contractor years. In October
2002, DOD decided to buy 208

in attaining design and production knowledge. missiles covering the next 2
years. DOD plans to

accelerate the production rate immediately by adding

Design Maturity

a second manufacturing shift and test equipment. PAC- 3*s basic design is
complete, with 100 percent of Because production was not expected to be
the drawings released to manufacturing. Only

accelerated to this level this early in production, the 21 percent of the
drawings were complete when the contractor must expeditiously find and
train qualified program held its design review, which led to a number

personnel. The accelerated plan requires additional of problems. For
example, the contractor attributed a

funding of $239 million for fiscal years 2003 and 2004. $101 million cost
increase to first- time manufacturing

Program Office Comments

problems, such as some subsystems not fitting together properly and some
not passing ground or In commenting on a draft of this assessment, program
environmental tests. These problems were a major

officials stated that they believe production processes contributor to a
2- year schedule delay. To reduce

are in control. Program officials stated that they have missile costs, the
contractor has identified several meticulously and methodically examined
every critical major design changes, which will be incorporated into

process from a labor and inspection standpoint to help the design in 2004.
ensure a consistent and quality product. Despite the less than fully
successful operational tests, they also

Production Maturity

believe that they have the most successful The program has 23 percent of
the key manufacturing development flight test program in the history of
processes used to assemble the missile and the seeker missile development.
They provided technical

under control. Production maturity has deteriorated comments, which were
incorporated as appropriate.

from the 35 percent that was in control at the October 1999 low- rate
production decision. A switch in the manufacturing facilities may have
played a role. According to program officials, the program entered
production before process control was emphasized to the contractor. The
contractor is still having difficulties building the missile. For example,
each seeker still needs to be reworked about three times on

average before it passes quality inspections. Program officials have added
quality tests of components, which have improved the situation, but the
contractor

has not yet demonstrated that these tests will eliminate the need for
seeker rework in the future.

Other Program Issues

The Army conducted four operational tests in 2002; none were completely
successful. The PAC- 3 system defeated half of the targets in flight-
testing. System

Space Based Infrared System (SBIRS) High

SBIRS High will consist of a constellation of four satellites plus one
spare, two sensors on a nonSBIRS satellite, and associated ground
stations. SBIRS High is to provide missile warning and missile defense
information and will be used to support the technical intelligence and
battlespace characterization missions. The first launch of SBIRS High is
scheduled for fiscal year 2007.

Prime contractor: Lockheed Martin

Program office: El Segundo, Cal.

FY 2004 funding request: FY 2003 dollars in millions Approved Latest
Percent R& D: $617.2 million 3/ 98 6/ 02 change

Procurement: $95.4 million Research & development cost $3, 378.4 $6,077. 3
79.9 Quantity: 1 satellite Procurement cost $558.1 $1,417. 4 154.0

Total program cost $4, 127.0 $8,241. 2 99.7

Program unit cost $825.4 $1,648. 2 99.7

Total quantities 5 5 0. 0 Acquisition cycle time (months) NA NA NA Note:
NA = not available.

The SBIRS High program*s critical technologies have demonstrated
acceptable levels of maturity. This level of maturity follows many years
of difficult

development. The level of design stability is unknown since the contractor
was unable to provide information on the total number of releasable
drawings at specific milestones.

Similarly, production maturity could not be determined because the
contractor does not collect statistical control data. The SBIRS High
program is building the first two satellites using research and
development funding with a first launch expected in fiscal year 2007. The
program also recently underwent a major restructuring to reduce program

risk.

SBIRS High Program

10 U. S. C. 2433) occurred on December 31, 2001, requiring a review by the
Secretary of Defense and a

Technology Maturity

report to Congress. As a result, DOD certified to The SBIRS High program*s
three critical

Congress in May 2002 that the SBIRS High program is technologies* the
infrared sensor, thermal essential for national security, there are no
management, and the on- board processor* are now

alternatives that provide equal or greater capability at mature. Program
officials indicated that the hardware

less cost, cost estimates are reasonable, and the was built and tested in
a thermal vacuum chamber management structure is in place to continue to
keep

under expected flight conditions. When the program costs under control.

began product development in 1996, none of its

Program Office Comments

critical technologies were mature, according to best practice standards.

Program officials generally concurred with our assessment and provided
technical comments, which

Design Maturity

we have incorporated where appropriate. Program Program officials do not
know how many total

officials added that the fiscal year 2004 budget fully drawings are
expected for SBIRS High, and thus do funds their restructured program and
directs the not track the number of releasable drawings. As a

satellite procurement to begin in fiscal year 2006. result, we could not
assess design stability relative to best practices. Program officials did
state that the current number of releasable drawings is 2,342, about twice
the number at the time of the design review. This means that at most, no
more than half of the drawings could have been releasable at the design
review. Design stability has been an issue for this program. During
development, the satellite was redesigned to

maintain key performance parameters. Redesign efforts resulted in a 6-
month slip to the spacecraft and increased the requirement for ground
processing.

On the other hand, the two sensors that will be aboard non- SBIRS
satellites are considered stable with subsystem qualification nearing
completion, and integration and delivery of the flight payload are

expected within the year. The first of these sensors is scheduled for
delivery in May 2003* three months behind schedule. This delay is
attributed to problems

with radio waves emitted by the sensor's electronics that interfere with
the host satellite. Despite these integration difficulties, data shows
that the sensors will perform much better than expected.

Production Maturity

We could not assess the SBIRS High production maturity relative to best
practice standards because the contractor does not use statistical process
control

to ensure that production processes are stable.

Other Program Issues

The total unit cost of the SBIRS High program rose more than 25 percent in
1 year. The notification to Congress of the Nunn- McCurdy breach (see

Theater High Altitude Area Defense (THAAD)

THAAD is an element of the terminal defense segment of the Ballistic
Missile Defense System. Its mission is to defend against short and medium
range ballistic missiles. THAAD*s ability to intercept

inside and outside the atmosphere makes effective countermeasures more
difficult and allows multiple intercept opportunities. The system includes
missiles, launchers, radars, command and control/ battle management (C2/
BM), and THAAD support equipment.

Prime contractor: Lockheed Martin

Project office: Huntsville, Ala.

FY 2004 funding request: FY 2003 dollars in millions Approved Latest
Percent R& D $730.6 million 1/ 92 2/ 03 change

Procurement $0 million

Research & development cost $4, 382.7 $10,548. 0 138.0 Quantity 0 missiles

Procurement cost NA NA NA Total program cost NA NA NA

Program unit cost NA NA NA

Total quantities (U. S. only) NA NA NA Acquisition cycle time (months) 114
NA NA Note: Procurement schedule, funding, and quantities have yet to be
determined. The THAAD schedule no longer includes production milestones.
NA = not applicable.

Most of THAAD*s critical technologies have demonstrated acceptable levels
of maturity and the program appears close to meeting the best practice
standard for a stable design. The program*s launcher and radar have
essentially attained technological maturity, but the missile and the
command and control/ battle management components are somewhat less
mature. This level

of maturity follows many years of difficult development. It appears that
the THAAD program has mostly recovered from initial problems driven by an
early fielding requirement and poor quality control. The current THAAD
acquisition strategy shows a much greater emphasis on attaining knowledge.
The program expects to reach technological maturity and design stability
by February 2004.

THAAD Program

missile in the more stressing flight environment inside the atmosphere,
and (3) prepares the system for initial

Technology Maturity

operational test and evaluation. THAAD program officials assessed 47
technologies in

Program Office Comments

four major elements* command and control/ battle management; missile
interceptor; launcher; and radar. In commenting on a draft of this
assessment, program Of the four elements, the radar is currently the most

officials stated that, to ensure the highest probability mature, followed
by the launcher, command and

of success in flight- testing, a substantial amount of control/ battle
management, and the missile. The

ground testing is being conducted in the next year and program has made
progress on technology maturity

a half. This testing includes exhaustive engineering since it began
development despite early failures in

and qualification level testing on all flight components. intercept
attempts. Early flight- test failures were

Program officials further stated that the extensive caused by a
combination of the compressed test

design, fabrication, and test preparation activity has schedule and
quality control problems. The program

been very successful to date, and the program remains was restructured
twice, before the first successful

healthy with a slightly ahead- of- schedule and undercost intercept
occurred in 1999. The research and status. development cost grew from $4.4
to $10.5 billion prior to the program's transfer to the Missile Defense
Agency, partially as a result of these problems.

The current program strategy appears geared to obtaining the necessary
knowledge by providing more time for maturing the technology before flight
tests and placing greater emphasis on risk reduction efforts. This
strategy includes utilizing technology readiness levels to assess
technological maturity.

Design Maturity

The program has released about 82 percent of total drawings. Program
officials expect to release about 91 percent of the drawings by the
system- level design review in February 2004. The Missile Defense Agency
is redesigning the missile to be more reliable and testable, with
significantly fewer parts than the previous version. The first flight test
of the redesigned missile is not scheduled to occur until at least 6
months after the system design review. Depending on the outcome, flight
tests could require more design changes and delay achieving design
stability.

Other Program Issues

THAAD was recently transferred from the Army to the Missile Defense
Agency, which has restructured and modified the contract to a block
upgrade approach. Therefore, limited information is currently available on
the total projected costs of this program.

In response to the prior program setbacks, the THAAD project office is
accelerating some risk reduction activities, and it has planned a series
of flight tests that (1) tests the missile in a less stressing intercept
environment outside the atmosphere, (2) tests the

Tactical Tomahawk Missile

The Navy*s Tactical Tomahawk (block IV) is a major upgrade to the Tomahawk
Land Attack Missile (block III). The Tactical Tomahawk missile will
provide ships and submarines with enhanced capability to attack targets on
land. New features include improved antijamming global positioning system,
in- flight retargeting, and ability to transmit battle damage imagery. The
system includes the missile, the weapon control system, and the mission

planning system. We assessed only the missile.

Prime contractor: Raytheon Systems Company

Program office: Patuxent River, Md.

FY 2004 funding request: FY 2003 dollars in millions Approved Latest
Percent R& D $71.4 million

9/ 99 12/ 01 change

Procurement $277.6 million

Research & development cost $559.5 $584. 1 4.4 Quantity 267 missiles

Procurement cost $1, 236.4 $1,546. 0 25.0 Total program cost $1, 795.9
$2,130. 0 18.6

Program unit cost $1. 316 $1.235 -6.1

Total quantities 1,365 1,725 26.4 Acquisition cycle time (months) 58 69
19.0

The Tactical Tomahawk missile entered low- rate production without
ensuring that production processes were in control. Program officials
indicated that they plan to collect production process control data over
the next year, prior to

award of the full- rate production contract in fiscal year 2004. At that
time, program officials expect over 80 percent of the low- rate production
missiles to be in various stages of assembly. The technology and design
have reached acceptable levels of maturity. While engineering drawings
have improved to 96 percent, the program only had about

half of its drawings released at the design review. Program plans call for
a full- rate production decision in May 2004.

Tactical Tomahawk Program

awarded in mid- January 2003 for 167 units. Program officials stated that
total quantities have increased to

Technology Maturity

2,396. We did not assess the technology readiness levels of

Program Office Comments

the key technologies for the Tactical Tomahawk missile. At the time of our
review, critical technologies

In commenting on a draft of this assessment, program were mature.
According to the program office, the

officials stated that two development test flights, critical technologies
for the key subsystems* antijam

conducted prior to low- rate production awards, global positioning system,
digital scene matching area demonstrated that the Tactical Tomahawk
missile correlator, and cruise engine* were modified design met or
exceeded technical and key derivatives from other programs or upgrades to

performance parameters. They also noted that, due to existing Tomahawk
subsystems and consequently

the stability of the design and successful completion already mature. To
date, subsystem and the majority

of all component and flight qualification testing, the of missile- level
qualification testing has been

Navy*s operational test agency issued a favorable completed successfully.
operational assessment, stating that the Tactical Tomahawk missile is
potentially suitable and

Design Maturity

potentially operationally effective. The basic design of the Tactical
Tomahawk missile is essentially complete. The critical design review
occurred in June 2000. At that time, approximately

47 percent of the drawings had been released to manufacturing. In October
2002, at the first low- rate initial production award, 723 of 750 total
drawings, or about 96 percent, had been released.

Production Maturity

Officials plan to collect statistical control data at the start of the
manufacturing process but do not expect to have meaningful statistical
data until sometime in 2004. Manufacture of the Tactical Tomahawk missile
is scheduled to begin at the subcontractor's facility in 2003 and missile
assembly in 2004. Although two lowrate production contracts have been
awarded, program officials stated that data regarding manufacturing
process controls currently is very limited. Program officials told us that
it is too soon to

know what percentage of critical manufacturing processes will be under
statistical control when the full- rate production contract is awarded in
mid- 2004, but that they plan to start collecting production process
control data over the next year.

Other Program Issues

The Tactical Tomahawk missile successfully completed its first
developmental flight test in August 2002, and the first low- rate
production contract for 25 units was awarded in October 2002. A second and
final low- rate production contract was

V- 22 Ospr ey

The V- 22 Osprey is a tilt- rotor, vertical takeoff and landing aircraft
designed to meet the amphibious/ vertical assault needs of the Marine
Corps, longrange missions of Special Operations forces, and combat search
and rescue needs of the Navy. The

V- 22 will replace the CH- 46E and the CH- 53A/ D in the Marine Corps; the
H- 53 and H- 60 will augment the C- 130 in the Air Force and the Special
Operations Command; and supplement the H- 60 in the Navy. We assessed the
block A version.

Prime contractor: Bell- Boeing

Program office: Patuxent River, Md.

FY 2004 funding request: FY 2003 dollars in millions Approved Latest
Percent R& D $543.9 million 2/ 87 12/ 01 change

Procurement $1.11 billion

Research & development cost $3, 568.5 $10,253. 5 187.3 Quantity 11
aircraft

Procurement cost $29,499.2 $32,312. 7 9.5 Total program cost $33,264.7
$42,617. 5 28.1

Program unit cost $36.434 $93.051 155.4

Total quantities 913 458 -49.8 Acquisition cycle time (months) 117 261
123.1

The V- 22 program plans to enter full- rate production without ensuring
that the manufacturing processes are mature. Redesign of the aircraft*s
hydraulic and electric system, and software changes have been made to
address safety, reliability, maintainability,

and logistics supportability. These design changes and others are
undergoing developmental testing to ready the aircraft for an operational
test and evaluation test period in late 2004 through early 2005 to
determine if the V- 22 is operationally suitable and effective. The design
changes, however, have not been incorporated into the lowrate

production aircraft currently being produced. The value of contract
modifications needed to address the cost of these design changes is not
yet known. Also, parts shortages and quality issues are currently
effecting low- rate production costs. Some key performance requirements
have been eliminated.

V- 22 Program

and incorporated into production aircraft. Delivery of block A aircraft is
expected to start in the fourth

Technology Maturity

quarter of fiscal year 2003. However, the cost of Although we did not
specifically assess the V- 22*s contract modifications needed to
reconfigure already technology maturity, the program office believes key

produced aircraft and aircraft still on the assembly technologies to be
mature. An operational test report, line to the bock A configuration has
not been dated November 2000, determined that the V- 22 was

negotiated. not operationally suitable because of poor reliability,
Program Office Comments

maintainability, availability, human factors, and interoperability
problems. Immature technology, in

In commenting on draft of this assessment program part, contributed to
this assessment.

officials stated that they have restructured the program to gather more
technical knowledge through

Design Maturity

a more rigorous *event- driven* flight test program. As a result of a
crash in December 2000, the V- 22 has

Program officials strongly disagreed that the program undergone several
design changes. Specifically, the

plans to enter full- rate production without ensuring aircraft*s hydraulic
and electrical lines were that manufacturing process are mature. V- 22s
are

redesigned to improve safety, reliability, currently being manufactured at
a minimum sustaining rate (11 aircraft per year). A May 20 th
maintainability, and logistics supportability. The V- 22 defense
acquisition board review is scheduled to flight control system software
was also redesigned.

consider increasing this rate. Manufacturing processes The program office
estimates that redesign of the V- 22 and tooling are in place and being
continually analyzed resulted in 1, 755 additional drawings, increasing
the

and improved. Both companies utilize statistical total number of drawings
to 7,490. To date, all of these

process control techniques and numerous metrics to drawings are complete.

assess program performance. They do not use the The success of these
design changes will be process capability index, the only metric that GAO
determined as the aircraft undergoes additional

uses as a basis for their assessment. Program officials developmental
testing through 2005. Testing will

are also undertaking an affordability review to reduce address many
issues, including high rate of descent,

the aircraft unit cost to $58 million by 2010. High unit handling
qualities, austere environment operations, costs are driven by the current
low production and ship operations. The operational assessment of

quantities and will remain the norm until production these characteristics
will not occur until late 2004 or quantities increase. early 2005. Recent
decisions to defer some V- 22 operational requirements previously
considered

GAO Comments

critical until later blocks will void the need for some design changes in
the block A.

Our prior work has shown that leading commercial firms rely on statistical
control data, specifically, the

Production Maturity

process capability index, as the best indicator of production readiness.
The V- 22 program entered lowrate Neither V- 22 contractor collects
statistical process

production without this information and has control data on its critical
manufacturing processes. A

experienced production quality problems. recent program management
assessment rated V- 22 production as cautionary. Part shortages and
quality problems caused inefficiencies in shop and assembly operations, as
well as scrap, rework, repair, and schedule delays. Other Program Issues

Low- rate production of the V- 22 continues. V- 22s are being fabricated
and partially assembled, but not delivered until the first set of
upgrades* referred to as block A* needed to bring the V- 22 to a safe
operational and suitable configuration are approved

Wideband Gapfiller Satellite (WGS) Communications System

The Wideband Gapfiller Satellite system is a joint Air Force and Army
program intended to provide communications to the U. S. warfighters,
allies, and Coalition Partners during all levels of conflict short

of nuclear war. It is the next generation wideband component in the DOD*s
future Military Satellite Communications architecture.

Prime contractor: Boeing Satellite Systems (BSS)

Program office: El Segundo, Calif.

FY 2004 funding request: FY 2003 dollars in millions Estimate Latest

Percent R& D $36.7 million 12/ 01 12/ 02 change

Procurement $34.6 million Research & development cost $181. 2 $244.8 35.1
Quantity 0 satellites

Procurement cost $831. 5 $1, 389.8 67.2 Total program cost $1,012. 7 $1,
634.6 61.4

Program unit cost $337. 6 $326.9 -3.2

Total quantities 3 5 66.7 Acquisition cycle time (months) 50 55 10.0

The WGS program*s critical technologies and design are mature, while its
production processes are nearly mature. DOD plans to rely on commercial
technologies that will not require extensive product development. However,
two of these processes use statistical control rates that are below the
level prescribed by best practice standards. The program recently added
two satellites to better support intelligence, surveillance, and
reconnaissance missions in the future.

WGS Program

The WGS program*s two critical technologies* the digital channelizer and
the phased array antenna* are mature. Most of these technologies are
commercial derivatives. For this reason, many of the satellite
technologies selected were already at high levels of maturity. In fact,
the program is leveraging commercial technology and practices by modifying
commercial satellites to better support unique military requirements.

Design Maturity

The WGS design is essentially complete, as the program has released
approximately 95 percent of the expected drawings.

Production Maturity

The contractor has six of its eight key manufacturing processes under
control, according to the best practice standards. Program officials
indicated that they are bringing the remaining processes under statistical
control.

Program Office Comments

In commenting on a draft of this assessment, program officials stated that
while critical technology areas being applied to WGS are fairly mature,
the manufacturing of the systems using these technologies is relatively
new for the contractor. Risk of production problems was to be reduced due
to other commercial

satellite system developments and production ahead of WGS in the
development and production schedule. However, due to the drastic loss of
commercial satellite orders, only one commercial satellite with

similar technologies as WGS is now leading WGS in the manufacturing
schedule. Recently identified problems found on the *leader* program will
impact the WGS manufacturing schedule, and a first launch schedule delay
of 4 to 6 months can be expected due to time needed to resolve the
*leader* program manufacturing problems. Satellites four and five have
been directed by DOD to be launched in fiscal year 2009 and fiscal year
2010, respectively. These dates are outside the allowable dates of the WGS
contract

option clauses and will require renegotiation to finalize their cost. The
cost is expected to increase to compensate for loss of learning curve from
over a 3-

year break in production, parts obsolescence, and inflation.

Appendi x II

Methodology In conducting our work, we evaluated performance and risk data
from each of the programs included in this report. We summarized our
assessments of each individual program in two components* a system profile
and a product knowledge assessment. We did not validate or verify the data
provided by DOD. However, we took several steps to address data quality.
Specifically, we reviewed the data and performed various quality checks,

which revealed some discrepancies in the data. We discussed these
discrepancies with program officials and adjusted the data accordingly.
System Profile

In the past 3 years, DOD revised its policies governing weapon system
Assessment acquisitions and changed the terminology used for major
acquisition events. In order to make DOD*s acquisition terminology more
consistent across the 26 program assessments, we standardized the
terminology for key program events. In the individual program assessments,
program start refers to the initiation of a program; DOD usually refers to
program start as milestone I or milestone A, which begins the concept and
technology development phase. Similarly, development start refers to the
commitment to product development that coincides with either milestone II
or milestone B, which begins DOD*s system development and demonstration

phase. The production decision generally refers to the decision to enter
the production and deployment phase, typically with low- rate initial
production. Initial capability refers to the initial operational
capability, sometimes also called first unit equipped or required asset
availability.

The funding request information presented refers to the President*s fiscal
year 2004 budget request, except where noted. The program cost comparisons
are the latest estimates provided by the individual programs. The
quantities listed refer to total quantities, including both procurement
and development quantities.

To assess the cost, schedule, and quantity changes of each program, we
reviewed DOD*s selected acquisition reports or obtained data directly from
the program offices. In general, we compared the latest available selected
acquisition report information with a baseline for each program. For

systems that have started product development* those that are beyond
milestone II or B* we compared the latest available Selected Acquisition
Report to the development estimate from the first Selected Acquisition
Report issued after the program was approved to enter development. For
systems that have not yet started product development, we compared the
latest available data to the planning estimate issued after milestone I or
A. For systems not included in selected acquisition reports, we attempted
to

obtain comparable baseline and current data from the individual program
offices.

All cost information is presented in base year 2003 dollars, unless
otherwise noted, using Office of the Secretary of Defense approved
deflators to eliminate the effects of inflation. We have depicted only the

programs* main elements of acquisition cost* research and development, and
procurement, however the total program costs displayed also include
military construction and acquisition operation and maintenance costs.
Because of rounding and these additional costs, in some situations the
total cost may not match the exact sum of the research and development and
procurement costs. The program unit costs are calculated by dividing the
total program cost by the total quantities planned. These costs are often
referred to as program acquisition unit costs.

The schedule assessment is based on acquisition cycle time, defined as the
number of months between the program start, usually milestone I or A, and
the achievement of initial operational capability or an equivalent
fielding date.

The intent of these comparisons is to provide an aggregate or overall
picture of a program*s history. These assessments represent the sum total
of the federal government*s actions on a program, not just those of the
program manager and the contractor. DOD does a number of detailed analyses
of changes that attempt to link specific changes with triggering events or
causes. Our analysis does not attempt to make such detailed distinctions.

Product Knowledge To assess the product development knowledge of each
program at key

Assessment points in development, we submitted a data collection
instrument to each

program office. The results are graphically depicted in each two- page
assessment. The methodology used to generate each graph is discussed at
the beginning of appendix I. We also reviewed pertinent program
documentation, such as the operational requirements document, the
acquisition program baseline, test reports, and major program reviews.

To assess technology maturity, we asked program officials to apply a tool,
referred to as technology readiness levels, for our analysis. The National
Aeronautics and Space Administration originally developed technology
readiness levels, and the Army and Air Force Science and Technology
research organizations use them to determine when technologies are ready

to be handed off from science and technology managers to product
developers. Technology readiness levels are measured on a scale of one to
nine, beginning with paper studies of a technology*s feasibility and
culminating with a technology fully integrated into a completed product.

Our best practices work has shown that a technology readiness level of 7*
demonstration of a technology in an operational environment* is the level
of technology maturity that constitutes a low risk for starting a product

development program. In most cases, we did not validate the program
offices* selection of critical technologies or the determination of the
demonstrated level of maturity. We sought to clarify the technology
readiness levels in those cases where

information existed that raised concerns. If we were to conduct a detailed
review, we might adjust the critical technologies assessed, the readiness
level demonstrated or both. It was not always possible to reconstruct the
technological maturity of a weapon system at key decision points after the
passage of many years.

To assess design maturity, we asked program officials to provide the
percentage of engineering drawings completed or projected for completion
by the design review, the production decision, and as of our current
assessment. Completed engineering drawings were defined as the number of
drawings released or deemed releasable to manufacturing that can be
considered the *build to* drawings.

To assess production maturity, we asked program officials to identify the
number of critical manufacturing processes and, where available, to
quantify the extent of statistical control achieved for those processes.
We used a standard called the Process Capability Index, which is a process

performance measurement that quantifies how closely a process is running
to its specification limits. 1 The index can be translated into an
expected product defect rate and we have previously found it to be a best
practice. We sought other data, such as scrap and rework trends in those
cases

where quantifiable statistical control data was unavailable. 1 Process
Capability Index provides assurance that production processes are under
100 percent statistical control. A high index value equates to fewer
defects per part based on statistical process control data. The general
rule of thumb used by the manufacturing industry states that if the index
value for a process is less than 1.33, then the process is not capable of
producing a part with acceptable consistency.

Although the knowledge points provide excellent indicators of potential
risks, by themselves, they do not cover all elements of risk that a
program encounters during development, such as funding instability. Our
detailed reviews on individual systems normally provide for a fuller
treatment of risk elements.

Appendi x III

GAO Contact and Acknowledgments GAO Contact Paul Francis (202) 512- 2811
Acknowledgments David B. Best, James A. Elgas, and James L. Morrison made
key

contributions to this report and were largely supported by GAO*s
Acquisition and Sourcing Management staff. The following staff were
responsible for individual programs.

Staff System

Michael Hazard Advanced Amphibious Assault Vehicle (AAAV) Steven Martinez
Advanced Extremely High Frequency

(AEHF) Communications Satellite David Hubble Advanced Wideband Satellite/
Transformational Communications

Satellite (AWS/ TSAT) Gaines Hensley AN/ APG- 79 Active Electronically
Scanned

Array (AESA) Radar Marvin Bonner AIM- 9X Short- Range Air- to- Air Missile
Thomas Gordon Airborne Laser (ABL) Dana Solomon/ Carrie

Advanced Threat Infrared Wilson/ Danny Owens

Countermeasures/ Common Missile Warning System (ATIRCM/ CMWS) Johanna
Ayers Cooperative Engagement Capability (CEC) Leon Gill CH- 47F Improved
Cargo Helicopter Wendy Smythe RAH- 66 Comanche Marti Dey/ Ron Schwenn EX-
171 Extended Range Guided Munition

(ERGM) Larry Gaston Excalibur Artillery Round Cheryl Andrew F/ A- 18E/ F
Super Hornet Donald Springman F- 22 Raptor Beverly Breen/ Lynn

Joint Air- to- Surface Standoff Missile (JASSM) Lavalle Danny Owens Joint
Common Missile Rae Ann Sapp/ Art Cobb Joint Primary Aircraft Training
System (JPATS)

Brian Mullins/ Ron F- 35 Joint Strike Fighter (JSF) Schwenn

Staff System

Carol Mebane Joint Standoff Weapon (JSOW) Bruce Thomas National Polar-
orbiting Operational

Environmental Satellite System (NPOESS) Matthew Lea Patriot Advanced
Capability 3 Program

(PAC- 3) Tana Davis Tactical Tomahawk Missile William Lipscomb/ Tana

Theater High Altitude Area Defense (THAAD) Davis Maricela Cherveny/ Nancy
Space Based Infrared Satellite- High (SBIRSHigh) Rothlisberger Jerry Clark
V- 22 Osprey Art Gallegos/ Tony

Wideband Gapfiller Satellite Communication Beckham

System

(120185)

GAO*s Mission The General Accounting Office, the audit, evaluation and
investigative arm of Obtaining Copies of GAO Reports and Testimony

To Report Fraud, Waste, and Abuse in Federal Programs

Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability of
the federal government for the American people. GAO examines the use of
public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO*s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO*s Web site (www. gao. gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as *Today*s Reports,* on its
Web site daily. The list contains links to the full- text document files.
To have GAO e- mail this

list to you every afternoon, go to www. gao. gov and select *Subscribe to
GAO Mailing Lists* under *Order GAO Products* heading.

Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO

also accepts VISA and Mastercard. Orders for 100 or more copies mailed to
a single address are discounted 25 percent. Orders should be sent to:

U. S. General Accounting Office 441 G Street NW, Room LM Washington, D. C.
20548 To order by Phone: Voice: (202) 512- 6000 Contact:

TDD: (202) 512- 2537 Fax: (202) 512- 6061 Web site: www. gao. gov/
fraudnet/ fraudnet. htm E- mail: fraudnet@ gao. gov Automated answering
system: (800) 424- 5454 or (202) 512- 7470 Public Affairs Jeff Nelligan,
Managing Director, NelliganJ@ gao. gov (202) 512- 4800 U. S. General
Accounting Office, 441 G Street NW, Room 7149 Washington, D. C. 20548

a

GAO United States General Accounting Office

GAO assessed 26 defense programs ranging from the Marine Corps* Advanced
Amphibious Assault Vehicle to the Missile Defense Agency*s Theater High
Altitude Area Defense system. GAO*s assessments are anchored in a
knowledge- based approach to product development that reflects best
practices of successful programs. This approach centers on attaining high
levels of knowledge in three elements of a new product or weapon*
technology, design, and production. If a program is not attaining this
level of knowledge, it incurs increased risk of technical problems,
accompanied by cost and schedule growth (see figure). If a program is
falling short in one element, like technology maturity, it is harder to
attain knowledge in succeeding elements.

All of the programs GAO assessed proceeded with less knowledge at critical
junctures than suggested by best practices, although several came close to
meeting best practice standards. GAO also found that programs generally
did not track statistical process control data, a key indicator for
production maturity. Program stakeholders can use these assessments to
recognize the gaps in knowledge early and to take advantage of
opportunities for constructive intervention* such as adjustments to
schedule, trade- offs in requirements, and additional funding. GAO has
summarized the results of its assessments in an easy to read twopage

format. Each two- page assessment contains a profile of the product that
includes a description; a timeline of development; a baseline comparison
of cost, schedule, and quantity changes to the program; and a graphical
and narrative depiction of how the product development knowledge of an
individual program compared to best practices. Each program office
submitted comments and they are included with each individual assessment
as appropriate. The weapons the Department of

Defense (DOD) develops have no rival in superiority. How they are
developed can be improved, without sacrificing the superiority of the
outcome. GAO*s reviews over the past 20 years have found consistent
problems with weapon investments* cost increases,

schedule delays and performance shortfalls* along with underlying causes,
such as pressure on managers to promise more than they can deliver. The
best practices of successful product developments offer a knowledgebased
approach DOD can use to improve the way it develops new weapons.

This report is new for GAO, and draws on its work in best practices for
product development. GAO*s

goal for this report is to provide congressional and DOD decision makers
with an independent, knowledge- based assessment of

defense programs that identifies potential risks, and offers an
opportunity for action when a program*s projected attainment of knowledge
diverges from the best practice. It can also highlight those

programs that employ practices worthy of emulation by other programs. GAO
plans to update and issue this report annually to the congressional
defense committees.

GAO makes no recommendations. Program office comments are included in the
assessments of each individual program. DEFENSE ACQUISITIONS

Assessments of Major Weapon Programs

www. gao. gov/ cgi- bin/ getrpt? GAO- 03- 476. To view the full report,
including the scope and methodology, click on the link above. For more
information, contact Paul Francis at (202) 512- 4841 or francisp@ gao.
gov. Highlights of GAO- 03- 476, a report to

Congressional Committees

May 2003

Page i GAO- 03- 476 Acquisition Trends and Risks

Contents

Contents

Page ii GAO- 03- 476 Acquisition Trends and Risks

Page iii GAO- 03- 476 Acquisition Trends and Risks United States General
Accounting Office Washington, D. C. 20548

Comptroller General of the United States A

Page iv GAO- 03- 476 Acquisition Trends and Risks

Page 1 GAO- 03- 476 Acquisition Trends and Risks United States General
Accounting Office Washington, D. C. 20548

Page 1 GAO- 03- 476 Acquisition Trends and Risks

A

Page 2 GAO- 03- 476 Acquisition Trends and Risks

Page 3 GAO- 03- 476 Acquisition Trends and Risks

Page 4 GAO- 03- 476 Acquisition Trends and Risks

Page 5 GAO- 03- 476 Acquisition Trends and Risks

Page 6 GAO- 03- 476 Acquisition Trends and Risks

Page 7 GAO- 03- 476 Acquisition Trends and Risks

Page 8 GAO- 03- 476 Acquisition Trends and Risks

Page 9 GAO- 03- 476 Acquisition Trends and Risks

Appendix I

Appendix I Assessments of Individual Programs

Page 10 GAO- 03- 476 Acquisition Trends and Risks

Appendix I Assessments of Individual Programs

Page 11 GAO- 03- 476 Acquisition Trends and Risks

Appendix I Assessments of Individual Programs

Page 12 GAO- 03- 476 Acquisition Trends and Risks

Appendix I Assessments of Individual Programs

Page 13 GAO- 03- 476 Acquisition Trends and Risks

Appendix I Assessments of Individual Programs Page 14 GAO- 03- 476
Acquisition Trends and Risks

Appendix I Common Name: AAAV

GAO- 03- 476 Acquisition Trends and Risks Page 15

Page 16 GAO- 03- 476 Acquisition Trends and Risks

Common Name: AAAV Appendix I

Appendix I Common Name: ABL

GAO- 03- 476 Acquisition Trends and Risks Page 17

Page 18 GAO- 03- 476 Acquisition Trends and Risks

Common Name: ABL Appendix I

Appendix I Common Name: AEHF

GAO- 03- 476 Acquisition Trends and Risks Page 19

Page 20 GAO- 03- 476 Acquisition Trends and Risks

Common Name: AEHF Appendix I

Appendix I Common Name: AESA

GAO- 03- 476 Acquisition Trends and Risks Page 21

Page 22 GAO- 03- 476 Acquisition Trends and Risks

Common Name: AESA Appendix I

Appendix I Common Name: AIM- 9X GAO- 03- 476 Acquisition Trends and Risks
Page 23

Page 24 GAO- 03- 476 Acquisition Trends and Risks

Common Name: AIM- 9X Appendix I

Appendix I Common Name: ATIRCM/ CMWS

GAO- 03- 476 Acquisition Trends and Risks Page 25

Page 26 GAO- 03- 476 Acquisition Trends and Risks

Common Name: ATIRCM/ CMWS Appendix I

Appendix I Common Name: AWS

GAO- 03- 476 Acquisition Trends and Risks Page 27

Page 28 GAO- 03- 476 Acquisition Trends and Risks

Common Name: AWS Appendix I

Appendix I Common Name: CEC GAO- 03- 476 Acquisition Trends and Risks Page
29

Page 30 GAO- 03- 476 Acquisition Trends and Risks

Common Name: CEC Appendix I

Appendix I Common Name: CH- 47F

GAO- 03- 476 Acquisition Trends and Risks Page 31

Page 32 GAO- 03- 476 Acquisition Trends and Risks

Common Name: CH- 47F Appendix I

Appendix I Common Name: Comanche

GAO- 03- 476 Acquisition Trends and Risks Page 33

Page 34 GAO- 03- 476 Acquisition Trends and Risks

Common Name: Comanche Appendix I

Appendix I Common Name: ERGM

GAO- 03- 476 Acquisition Trends and Risks Page 35

Page 36 GAO- 03- 476 Acquisition Trends and Risks

Common Name: ERGM Appendix I

Appendix I Common Name: Excalibur

GAO- 03- 476 Acquisition Trends and Risks Page 37

Page 38 GAO- 03- 476 Acquisition Trends and Risks

Common Name: Excalibur Appendix I

Appendix I Common Name: F/ A- 18E/ F

GAO- 03- 476 Acquisition Trends and Risks Page 39

Page 40 GAO- 03- 476 Acquisition Trends and Risks

Common Name: F/ A- 18E/ F Appendix I

Appendix I Common Name: F/ A- 22

GAO- 03- 476 Acquisition Trends and Risks Page 41

Page 42 GAO- 03- 476 Acquisition Trends and Risks

Common Name: F/ A- 22 Appendix I

Appendix I Common Name: JASSM

GAO- 03- 476 Acquisition Trends and Risks Page 43

Page 44 GAO- 03- 476 Acquisition Trends and Risks

Common Name: JASSM Appendix I

Appendix I Common Name: Joint Common Missile

GAO- 03- 476 Acquisition Trends and Risks Page 45

Page 46 GAO- 03- 476 Acquisition Trends and Risks

Common Name: Joint Common Missile Appendix I

Appendix I Common Name: JPATS

GAO- 03- 476 Acquisition Trends and Risks Page 47

Page 48 GAO- 03- 476 Acquisition Trends and Risks

Common Name: JPATS Appendix I

Appendix I Common Name: JSF

GAO- 03- 476 Acquisition Trends and Risks Page 49

Page 50 GAO- 03- 476 Acquisition Trends and Risks

Common Name: JSF Appendix I

Appendix I Common Name: JSOW

GAO- 03- 476 Acquisition Trends and Risks Page 51

Page 52 GAO- 03- 476 Acquisition Trends and Risks

Common Name: JSOW Appendix I

Appendix I Common Name: NPOESS

GAO- 03- 476 Acquisition Trends and Risks Page 53

Page 54 GAO- 03- 476 Acquisition Trends and Risks

Common Name: NPOESS Appendix I

Appendix I Common Name: PAC- 3

GAO- 03- 476 Acquisition Trends and Risks Page 55

Page 56 GAO- 03- 476 Acquisition Trends and Risks

Common Name: PAC- 3 Appendix I

Appendix I Common Name: SBIRS High

GAO- 03- 476 Acquisition Trends and Risks Page 57

Page 58 GAO- 03- 476 Acquisition Trends and Risks

Common Name: SBIRS High Appendix I

Appendix I Common Name: THAAD

GAO- 03- 476 Acquisition Trends and Risks Page 59

Page 60 GAO- 03- 476 Acquisition Trends and Risks

Common Name: THAAD Appendix I

Appendix I Common Name: Tactical Tomahawk

GAO- 03- 476 Acquisition Trends and Risks Page 61

Page 62 GAO- 03- 476 Acquisition Trends and Risks

Common Name: Tactical Tomahawk Appendix I

Appendix I Common Name: V- 22

GAO- 03- 476 Acquisition Trends and Risks Page 63

Page 64 GAO- 03- 476 Acquisition Trends and Risks

Common Name: V- 22 Appendix I

Appendix I Common Name: WGS

GAO- 03- 476 Acquisition Trends and Risks Page 65

Page 66 GAO- 03- 476 Acquisition Trends and Risks

Common Name: WGS Appendix I

Page 67 GAO- 03- 476 Acquisition Trends and Risks

Appendix II

Appendix II Methodology

Page 68 GAO- 03- 476 Acquisition Trends and Risks

Appendix II Methodology

Page 69 GAO- 03- 476 Acquisition Trends and Risks

Appendix II Methodology

Page 70 GAO- 03- 476 Acquisition Trends and Risks

Page 71 GAO- 03- 476 Acquisition Trends and Risks

Appendix III

Appendix III GAO Contact and Acknowledgments

Page 72 GAO- 03- 476 Acquisition Trends and Risks

United States General Accounting Office Washington, D. C. 20548- 0001
Official Business Penalty for Private Use $300 Address Service Requested

Presorted Standard Postage & Fees Paid

GAO Permit No. GI00
*** End of document. ***