Best Practices: Better Acquisition Outcomes Are Possible If DOD  
Can Apply Lessons from F/A-22 Program (11-APR-03, GAO-03-645T).  
                                                                 
Over the next 5 years, DOD's overall investments are expected to 
average $150 billion a year to modernize and transition our	 
forces. In addition, DOD must modernize its forces amid competing
demands for federal funds, such as health care and homeland	 
security. Therefore, it is critical that DOD manage its 	 
acquisitions in the most cost efficient and effective manner	 
possible. DOD's newest acquisition policy emphasizes the use of  
evolutionary, knowledge-based concepts that have proven to	 
produce more effective and efficient weapon systems outcomes.	 
However, most DOD programs currently do not employ these	 
practices and, as a result, experience cost increases, schedule  
delays, and poor product quality and reliability. This testimony 
compares the best practices for developing new products with the 
experiences of the F/A-22 program.				 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-03-645T					        
    ACCNO:   A06641						        
  TITLE:     Best Practices: Better Acquisition Outcomes Are Possible 
If DOD Can Apply Lessons from F/A-22 Program			 
     DATE:   04/11/2003 
  SUBJECT:   Best practices					 
	     Defense cost control				 
	     Defense procurement				 
	     Military research and development			 
	     Strategic planning 				 
	     Weapons systems					 
	     AIM-9X Missile					 
	     C-17 Aircraft					 
	     F/A-22 Aircraft					 
	     Global Hawk Unmanned Aerial Vehicle		 
	     Joint Direct Attack Munition			 
	     Joint Strike Fighter				 
	     Patriot PAC-3					 
	     V-22 Aircraft					 
	     Sidewinder Missile 				 
	     Globemaster Aircraft				 
	     Osprey Aircraft					 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-03-645T

Testimony Before the Subcommittee on National Security, Emerging Threats,
and International Relations, Government Reform Committee, House of
Representatives

United States General Accounting Office

GAO For Release on Delivery Expected at 10: 00 a. m. EDT Friday, April 11,
2003 BEST PRACTICES

Better Acquisition Outcomes Are Possible If DOD Can Apply Lessons from F/
A- 22 Program

Statement of David M. Walker, Comptroller General of the United States

GAO- 03- 645T

GAO*s reviews of commercial best practices have identified key enablers to
the success of product development programs and focused on how DOD can
better leverage its investments by shortening the time it takes to field
new capabilities at a more predictable cost and schedule. First,
commercial firms use an approach that evolves a product to its ultimate
capabilities on the

basis of mature technologies and available resources. This approach allows
only the product features and capabilities achievable with available
resources in the initial development. Further product enhancements are
planned for subsequent development efforts when technologies are proven to
be mature and other resources are available. Second, commercial firms
ensure that a high level of knowledge exists at key junctures during a
product*s development. The knowledge- based process includes three points:

Before a program is launched, successful programs match customer needs and
available resources* technology, engineering knowledge, time, and funding.

About midway through development, the ability of the product*s design is
demonstrated to be stable and meet performance requirements.

Before production begins, programs must show that a product can be
manufactured within cost, schedule, and quality targets.

In contrast, the F/ A- 22 program illustrates what can happen when a major
acquisition program is not guided by the principles of evolutionary,
knowledge- based acquisition. When the program was started, several key
technologies were not mature. Program managers proceeded through
development without the requisite knowledge to effectively manage program
risk and, at the start of production, key manufacturing processes were not
under control. The F/ A- 22 program has undergone significant cost
increases. Instead of fielding early capabilities to the war fighter, the
development cycle has extended to 19 years, so far, and original
quantities have been significantly reduced, raising concerns about the
capability the program will eventually deliver.

DOD recognizes the need to get better weapon system outcomes, and its
newest acquisition policy emphasizes the use of evolutionary, knowledge-
based acquisition concepts proven to produce better outcomes in developing
new products. However, policy changes alone are not enough. Leadership
commitment and attention to putting the policy into practice for
individual programs is needed to avoid the problems of the past. DOD will
have many opportunities to do so over the next several years with its
force modernization investments. Over the next 5 years, DOD*s

overall investments are expected to average $150 billion a year to
modernize and transition our forces. In addition, DOD must modernize its
forces amid competing demands for federal funds, such as health care and
homeland security. Therefore, it is

critical that DOD manage its acquisitions in the most cost efficient and
effective manner possible.

DOD*s newest acquisition policy emphasizes the use of evolutionary,
knowledge- based concepts that have proven to produce more effective and
efficient weapon systems outcomes. However, most DOD programs currently do
not

employ these practices and, as a result, experience cost increases,
schedule delays, and poor product

quality and reliability. This testimony compares the best practices for
developing new products with the experiences of the F/ A- 22 program. GAO
is not making recommendations in this testimony. However, in a number of
prior reports, GAO has recommended that DOD adopt policies with metrics
for technology, design, and manufacturing maturity to support

knowledge- based decision making. These policies should apply when making
decisions on individual weapons programs.

www. gao. gov/ cgi- bin/ getrpt? GAO- 03- 645T. To view the full report,
including the scope and methodology, click on the link above. For more
information, contact Katherine Schinasi at (202) 512- 4841 or schinasik@
gao. gov.

Highlights of GAO- 03- 645T, a testimony before the Subcommittee on
National Security, Emerging Threats, and International Relations,
Government Reform Committee, House of Representatives

April 2003

BEST PRACTICES

Better Acquisition Outcomes Are Possible If DOD Can Apply Lessons From F/
A- 22 Program

Page 1 GAO- 03- 645T Best Practices

Mr. Chairman and Members of the Subcommittee: I am pleased to be here
today to participate in the Subcommittee*s hearing on how the Department
of Defense (DOD) can* and must* get better outcomes from its weapon system
investments. DOD is on the threshold of several major investments in
acquisition programs that are likely to dominate budget and doctrinal
debates well into the next decade. These programs include such systems as
the Missile Defense Agency*s suite of land, sea, air, and space defense
systems; the Army*s Future Combat Systems; and the Air Force*s and Navy*s
Joint Strike Fighter. Over the next 5 years, DOD*s overall investments are
expected to average $150 billion a year as DOD works to keep legacy
systems as well as modernize and transform our national defense
capabilities for the future. Therefore, to meet these challenges, it is
essential that sound foundations for these and other weapon system
investments be laid now so that the resulting programs can be executed
within estimates of available resources.

Any discussion of improvements to DOD*s modernization efforts must be set
in the context of overall expected budget availability. There are
important competing priorities. Health care costs are growing at
doubledigit rates, and spending on homeland security will likely grow as
we seek to defeat terrorism worldwide. We face an oncoming demographic
tidal wave, and by 2035 the number of people who are 65 or older will have
doubled, creating much larger demands on the federal budget. The demand of
funding for entitlement programs continues to grow, creating increasing
pressures on discretionary funding for other federal priorities like
education and defense. Therefore, it is critical that DOD manage its
acquisitions in the most cost efficient and effective manner possible.

My testimony today is about improving the outcomes of major weapon system
acquisitions by using best practices to capture and use the right product
knowledge at the right time for better decision making during product
development. As per your request, I will compare acquisition practices and
decisions made for the F/ A- 22 with these best practices for developing
new products. The divergence between F/ A- 22 experiences and best product
development practices, we believe, largely explains why the F/ A- 22 has
been in development for over 16 years and its cost has grown
substantially. It is also a primary contributor to other performance
issues that are currently faced by the program. My testimony will also
include observations on what can be done at this time to limit further
negative outcomes in the F/ A- 22 program. Lastly, I will discuss the need
for enforcing DOD*s newest acquisition policy, which on paper embraces
best

Page 2 GAO- 03- 645T Best Practices

practices but in operation does not always do so, if DOD really expects to
get improved outcomes in its major weapon system acquisitions.

Clearly, the acquisition process has produced superior weapons, but it
does so at a high price. Weapon systems routinely take much longer time to
field, cost more to buy, and require more support than investment plans
provide for. These consequences reduce the buying power of the defense
dollar, delay capabilities for the war fighter, and force unplanned* and
possibly unnecessary* trade- offs in desired acquisition quantities and an
adverse ripple effect among other weapons programs or defense needs.
Because of the lengthy time to develop new weapons, many enter the field
with outdated technologies and a diminished supply base needed for system
support. Frequently, this requires upgrades to the capability as soon as
the new system is fielded. As previously noted, these inefficiencies have
often led to reduced quantities of new systems. In turn, legacy systems
remain in the inventory for longer periods, requiring greater operations
and support cost that pull funds from other accounts, including
modernization. DOD is facing these problems with its tactical air force

assets now. We believe DOD can learn lessons from the experiences with the
F/ A- 22 program as it frames the acquisition environment for its many
transformational investments.

DOD recognizes the need to get better weapon system outcomes, and its
newest acquisition policy emphasizes the use of evolutionary,
knowledgebased acquisition concepts proven to produce more effective and
efficient outcomes in developing new products. It incorporates the
elements of a knowledge- based acquisition model for developing new
products, which we have recommended in our reviews of commercial best
practices. Our body of work focuses on how DOD can better leverage its
investments by shortening the time it takes to field new capabilities at a
more predictable cost and schedule. However, policy changes alone will not
guarantee success. Unless written policies are consistently implemented in
practice through timely and informed decisions on individual programs,
outcomes will not change. This requires sustained leadership and
commitment and attention to the capture and use of key product knowledge
at critical decision points to avoid the problems of the past. A key
enabler to the success of commercial firms is using an approach that

evolves a product to its ultimate capabilities on the basis of mature
technologies and available resources. This approach allows commercial
companies to develop and produce more sophisticated products faster and
Improving Major

Weapon System Acquisition Outcomes The Case for an Evolutionary Product
Development Environment

Page 3 GAO- 03- 645T Best Practices

less expensively than their predecessors. Commercial companies have found
that trying to capture the knowledge required to stabilize the design of a
product that requires significant amounts of new technical content is an
unmanageable task, especially if the goal is to reduce development cycle
times and get the product to the marketplace as quickly as possible.
Therefore, product features and capabilities not achievable in the initial
development are planned for subsequent development efforts in future
generations of the product, but only when technologies are proven to be
mature and other resources are available. DOD*s new policy embraces the
idea of evolutionary acquisition. Figure 1 compares evolutionary and
single step (* big bang*) acquisitions.

Page 4 GAO- 03- 645T Best Practices

Figure 1: Comparison of Evolutionary and Big Bang Acquisition Approaches

Page 5 GAO- 03- 645T Best Practices

An evolutionary environment for developing and delivering new products
reduces risks and makes cost more predictable. While the customer may not
receive an ultimate capability initially, the product is available sooner,
with higher quality and reliability, and at lower, more predictable cost.

Improvements are planned for future generations of the product. Leading
commercial firms expect that their program managers will deliver high-
quality products on time and within budgets. Doing otherwise could result
in losing a customer in the short term and losing the company in the
longer term. Thus, in addition to creating an evolutionary product
development environment that brings risk in control, these firms have
adopted practices that put their individual program managers in a good
position to succeed in meeting these expectations on individual products.
Collectively, these practices ensure that a high level of knowledge exists
about critical facets of the product at key junctures during its
development. Such a knowledge- based process enables decision makers to be
reasonably certain about critical facets of the product under development
when they need to be.

The knowledge- based process followed by leading firms is shown in detail
in table 1, but in general can be broken down into three knowledge points.
First, a match must be made between the customer*s needs and the available
resources* technology, engineering knowledge, time, and funding* before a
program is launched. Second, a product*s design must demonstrate its
ability to meet performance requirements and be stable about midway
through development. Third, the developer must show that the product can
be manufactured within cost, schedule, and quality targets and is
demonstrated to be reliable before production begins. The following table
illustrates more specifically what we have learned about how successful
programs gather knowledge as they move through product development. The
Case for KnowledgeBased

Product Development Process

Page 6 GAO- 03- 645T Best Practices

Table 1: Highlights of Specific Best Practices for Acquisitions Knowledge
Point 1 (Should occur before program launch)

Separate technology from product development. Have clear measures and high
standards for assessing technology maturity* technology readiness levels.
Use a disciplined systems engineering process for translating and
balancing customer*s desires with product developer*s technology, design,
and production limitations; in other words, bring the right knowledge to
the table when laying down a program*s foundation.

Identify the mismatches between desired product features and the product
developer*s knowledge and either (1) delay the start of the new product
development until knowledge deficit can be made up or (2) reduce product
features to lessen their dependence on areas where knowledge is
insufficient (evolutionary acquisition). The main opportunities for
trading off design features to save time and money occur here, before a
program is started.

When do you know you have achieved this knowledge point? When technologies
needed to meet essential product requirements have been demonstrated to
work in their intended environment and the producer has completed a
preliminary design of the product.

Knowledge Point 2 (Should occur midway between system integration and
demonstration)

Hold a major decision review between system integration and system
demonstration that determines that the product design is stable and
includes specific criteria to move into the system demonstration phase.

Use integrated engineering prototypes to demonstrate design stability and
prove with testing that the design meets the customer*s requirements. It
is important that this happen before initial manufacturing begins* a point
when investments are increased to produce an item.

Identify critical manufacturing processes and establish a plan to bring
these under statistical control by the start of production; also establish
reliability goals and a growth plan to achieve these by production. This
facilitates the achievement of process control and reliability goals at
the completion of knowledge point 3. When do you know you have achieved
this knowledge point? When 90 percent of engineering drawings are
releasable to manufacturing organizations. Drawings are the language used
by engineers to communicate to the manufacturers the details of the new
product* what it looks like, how its components interface, how to build it
and the critical materials and processes needed to

fabricate it. This makes drawings a key measure of whether the design is
stable or not.

Knowledge Point 3 (Should occur before production)

Demonstrate that all critical manufacturing processes are under
statistical control and consistently producing items within the quality
standards and tolerances for the overall product before production begins.
This is important, since variation in one process can reverberate to
others and result in defective parts that need to be repaired or reworked.

Demonstrate product reliability before the start of production. This
requires testing to identify the problems, design corrections, and retest
the new design. Commercial firms consider reliability important and its
achievement a measure of design maturity.

When do you know you have achieved this knowledge point? When all key
manufacturing processes have come under statistical control and product
reliability has been demonstrated.

Page 7 GAO- 03- 645T Best Practices

DOD programs often do not employ these practices. We found that if the
evolutionary, knowledge- based acquisition concepts were not applied, a
cascade of negative effects became magnified in the product development
and production phases of an acquisition program. These led to acquisition
outcomes that included significant cost increases and schedule delays,
poor product quality and reliability, and delays in getting new capability
to the war fighter. This is often the case in DOD programs as shown in our

past work on systems like F/ A- 22 fighter, C- 17 airlifter, V- 22
tiltrotor aircraft, PAC- 3 missile, BAT antitank munition, and others. We
did find some DOD programs that employed best practice concepts and have
had more successful program outcomes to date. These included the Global
Hawk unmanned vehicle, AIM- 9X missile, and Joint Direct Attack Munitions
guided bomb. Figure 3 shows a notional illustration of the different paths
and effects of a product development.

Page 8 GAO- 03- 645T Best Practices

Figure 2: Different Paths That A Product*s Development Can Take

It is clear that knowledge about the product*s technology, design, and
processes captured at the right time can reduce development cycle times
and deliver a more cost effective, reliable product to the customer sooner
than programs that do not capture this knowledge.

In applying the knowledge- based approach, the most leveraged decision
point of the three, is matching the customer*s needs with the developer*s
resources* technology, design, timing, and funding. This initial decision

Page 9 GAO- 03- 645T Best Practices

sets the stage for the eventual outcome* desirable or problematic. The
match is ultimately achieved in every development program, but in
successful development programs, it occurs prior to program launch. In
successful programs, negotiations and trade- offs occur before a product
development is launched to ensure that a match exists between customer

expectations and developer resources. The results achieved from this match
are balanced and achievable requirements, sufficient investment to
complete the development, and a firm commitment to deliver the product.
Commercial companies we have visited usually limit product development
cycle- time to less than 5 years.

In DOD, this match is seldom achieved. It is not unusual for DOD to bypass
early trade- offs and negotiations, instead planning to develop a product
based on a rigid set of requirements that are unachievable within a
reasonable development time frame. This results in cost and schedule

commitments that are unrealistic. Although a program can take as long as
15 years in DOD, the program manager is expected to develop and be
accountable for precise cost and schedule estimates made at the start of
the program. Because of their short tenures, it normally takes several
program managers to complete product development. Consequently, the
program manager that commits to the cost and schedule estimate at the
beginning of the program is not the same person responsible for achieving
it. Therefore, program accountability is problematic. Ironically, this

outcome is rational in the traditional acquisition environment. The
pressures put on program managers to get programs approved encourage
promising more than can be delivered for the time and money allotted.

They are not put in a position to succeed. The differences in the
practices employed by successful commercial firms and DOD reflect the
different demands imposed on programs by the environments in which they
are managed. Specific practices take root and are sustained because they
help a program succeed in its environment.

The way success and failure are defined for commercial and defense product
developments differs considerably, which creates a different set of
incentives and evokes different behaviors from managers. Attempts at
reforming weapon system acquisitions have not succeeded because they did
not change these incentives. All of the participants in the acquisition
process play a part in creating incentives. The F/ A- 22 program,
advertised as a flagship of acquisition reform in its early days, failed
to establish this match before program launch and today we are discussing
the resulting outcomes to- date.

Page 10 GAO- 03- 645T Best Practices

The F/ A- 22 provides an excellent example of what can happen when a major
acquisition program is not guided by the principles of evolutionary,
knowledge- based acquisition. The program failed to match requirements
with resources and make early trade- offs and took on a number of new and
unproven technologies. Instead of fielding early capability and then
evolving the product to get new capabilities to the war fighter sooner,
the Air Force chose a *big bang* product development approach that is now
planned to take about 19 years. This created a challenging and risky
acquisition environment that has delayed the war fighter the capabilities
expected from this new aircraft. Program leaders did not capture the
specific knowledge identified as key for each of the three critical
knowledge points in product development. Instead, program managers
proceeded through the F/ A- 22*s development without the requisite
knowledge necessary for reducing program risk and achieving more
successful program outcomes. Now the optimism underlying these decisions
has resulted in significant cost increases, schedule delays, tradeoffs*
making do with less than half the number of originally desired aircraft*
and concerns about the capability to be delivered.

Since the F/ A- 22 acquisition program was started in October 1986, the F/
A22 cost and schedule estimates have grown significantly to where, today,
the Air Force estimates the total acquisition unit cost of a single
aircraft is $257.5 million. 1 This represents a 74 percent increase from
the estimate at

the start of development and a commensurate loss in the buying power of
the defense dollar. Intended to replace the aging F- 15 fighter, the F/ A-
22 program is now scheduled to reach its initial operational capability in
December 2005* making its development cycle about 19 years. During this
cycle, the planned buy quantity has been reduced 63 percent from 750 to
276 aircraft 2 . In addition, since fiscal year 2001, funding for F/ A- 22
upgrades has dramatically increased from $166 million to $3.0 billion,
most of which is to provide increased ground attack capability, a
requirement that was added late in the development program.

1 All references to F/ A- 22 costs in this testimony are in then- year
dollars in order to maintain consistent reporting with our prior reports
on the F/ A- 22. 2 Between 1986 and the start of engineering and
manufacturing development in 1991, the quantity was reduced from 750 to
648 aircraft. F/ A- 22 Did Not

Employ Evolutionary or Knowledge- Based Process F/ A- 22 Program Outcomes

Page 11 GAO- 03- 645T Best Practices

The F/ A- 22 acquisition strategy from the outset was to achieve full
capability in a *big bang* approach. By not using an evolutionary
approach, the F/ A- 22 took on significant risk and onerous technological
challenges. While the big bang approach may have allowed the Air Force to
more successfully compete for early funding, it hamstrung the program

with many new undemonstrated technologies, preventing the program from
knowing cost and schedule ramifications throughout development. Cost,
schedule, and performance problems resulted. The following table

summarizes the F/ A- 22 program*s attainment of critical knowledge and key
decision junctures during the development program and the changes in
development cost and cycle time at each point.

Table 2: Knowledge Attainment in the F/ A- 22 Program Program start* 1986
Design review* 1995 Production start* 2001 Best practice Attain knowledge
point 1.

Separate technology and product development, deliver mature technology,
and have preliminary design.

Attain knowledge point 2. 90 percent of systems and structures engineering
drawings releasable and subsystem design reviews completed.

Attain knowledge point 3. 100% of critical manufacturing processes in
statistical control and reliability goals demonstrated.

F/ A- 22 practice Knowledge point 1 not attained. Failed to separate
technology and product development. Three critical technologies immature:
Low- observable materials, propulsion, and integrated avionics. Knowledge
point 1 not attained until September 2000. Knowledge points 1 and 2 not

attained. Only 26 percent of drawings released at the critical design
review in February 1995. Knowledge point 2 not attained until September
1998, after

delivery of second test aircraft. Knowledge point 3 not attained.

Less than 50 percent of critical manufacturing processes in control. Only
22 percent of reliability goal demonstrated with many outstanding
deficiencies.

F/ A- 22*s estimated development cost a $12.6 billion $21.2 billion

(68 percent increase) $28.7 billion

(128 percent increase)

Estimated development cycle time 9.4 years 18.1 years

(93 percent increase) 19.2 years

(104 percent increase) a The development estimate includes all F/ A- 22
RDT& E costs.

Technology* The F/ A- 22 did not have mature technology at the start of
the acquisition program. The program included new low- observable
(stealth) materials, integrated avionics, and propulsion technology that
were not mature at this time. The Air Force did not complete an evaluation
of stealth technology on a full- scale model of the aircraft until several

years into development. It was not until September 2000, or 9 years into
F/ A- 22 Did Not Use Evolutionary Acquisition

or Capture Knowledge Required at Key Decision Junctures

Page 12 GAO- 03- 645T Best Practices

development, that the integrated avionics reached a maturity level
acceptable to begin product development. During development, the
integrated avionics was a source of schedule delays and cost growth. Since
1997, avionics software development and flight- testing have been delayed,
and the cost of avionics development has increased by over $980 million
dollars. Today, the avionics still has problems affecting the ability

to complete developmental testing and begin operational testing, and the
Air Force cannot predict when a solution will be found.

Design* The effects of immature technologies cascaded into the F/ A- 22
development program, making it more difficult to achieve a stable design
at the right time. The standard measure of design stability is 90 percent
of

design drawings releasable by the critical design review. The F/ A- 22
achieved only 26 percent by this review, taking an additional 43 months to
achieve the standard. Moving ahead in development, the program experienced
several design and manufacturing problems described by the

F/ A- 22 program office as a *rolling wave* effect throughout system
integration and final assembly. These effects included numerous design
changes, labor inefficiencies, parts shortages, out of sequence work, cost
increases, and schedule delays.

Production* At the start of production, the F/ A- 22 did not have
manufacturing processes under control and was only beginning testing and
demonstration efforts for system reliability. Initially, the F/ A- 22 had
taken steps to use statistical process control data to gain control of
critical manufacturing processes by full rate production. However, the
program abandoned this best practice approach in 2000 with less than 50
percent of its critical manufacturing processes in control. In March 2002,
3 we recommended that the F/ A- 22 program office monitor the status of
critical manufacturing process as the program proceeds toward high rate

production. The reliability goal for the F/ A- 22 is 3 hours of flying
time between maintenance actions. The Air Force estimated that in late
2001, when it entered production, it should have been able to demonstrate
almost 2 flying hours between maintenance actions. Instead, it could fly
an average of only 0.44 hours between maintenance actions. Since then
there has

3 U. S. General Accounting Office, Tactical Aircraft: F- 22 Delays
Indicate Initial Production Rates Should Be Lower to Reduce Risks, GAO-
02- 298 (Washington, D. C.: Mar. 5, 2002).

Page 13 GAO- 03- 645T Best Practices

been a decrease in reliability. As of November 2002, development test
aircraft have been completing only 0.29 hours between maintenance actions.
Additionally, the program was slow to fix and correct problems that had
affected reliability. At the time of our review in July 2002, program
officials had identified about 260 different types of failures and had
identified fixes for less than 50 percent of the failures. To achieve
reliability goals will require additional design changes, testing, and
modifications. Therefore, additional problems and costs can be expected if
the system is fielded with the level of reliability achieved to date.

The F/ A- 22 did not take advantage of evolutionary, knowledge- based
concepts up front and now, the best it can hope for is to limit cost
increases and performance problems by not significantly increasing its
production until development is complete* signified by developmental and
operational testing and reliability demonstrations. To that end, we

have recommended that the Air Force reconsider its decision to increase
the aircraft production rate beyond 16 aircraft per year. 4 The program is
nearing the end of developmental testing and plans to start initial
operational testing in October 2003. If developmental testing goes as
planned, which is not guaranteed, operational testing is expected to be
completed around September 2004. By the end of this fiscal year, 51 F/
A22s will be on contract as low rate production began in 2001. Our March
2003 report identifies various problems still outstanding that

could have further impacts on cost, schedule, and delivered performance
that are in addition to undemonstrated reliability goals. The problems
identified are of particular concern, given Air Force plans to increase
production rates and make a full rate production decision in 2004. The
problems include:

 unexpected shutdowns (instability) of the avionics,  excessive movement
of the vertical tails,  overheating in rear portions of the aircraft, 
separations of the horizontal tail material,  inability to meet airlift
support requirements, and  excessive ground maintenance actions.

4 U. S. General Accounting Office, Tactical Aircraft: DOD Should
Reconsider Decision to Increase F/ A- 22 Production Rates While
Development Risks Continue, GAO- 03- 431 (Washington, D. C.: Mar. 14,
2003). It Is Too Late for the F/ A22

Program to Gain Full Benefit of a KnowledgeBased Process

Page 14 GAO- 03- 645T Best Practices

These problems are still being addressed, and not all of them have been
solved as yet. For example, Air Force officials stated they do not yet
understand the problems associated with the avionics instability well
enough to predict when they would be able to resolve them, and certain
tests to better understand the vertical tail problem have not yet begun.
Despite remaining testing and outstanding problems, the Air Force plans to
continue acquiring production aircraft at increasing annual rates and make
the full rate production decision in 2004. This is a very risky strategy,
given outstanding issues in the test program and the system*s less than
expected reliability. The Air Force may encounter higher production costs
as a result of acquiring significant quantities of aircraft before
adequate testing and demonstrations are complete. In addition, remaining
testing could identify problems that require costly modifications in order
to achieve satisfactory performance.

In a February 28, 2003 report to Representative John Tierney, 5 we found
that F/ A- 22 production costs are likely to increase more than the latest
$5.4 billion cost growth recently estimated by the Air Force and the
Office of Secretary of Defense (OSD). First, the current OSD production
estimate does not include $1.3 billion included in the latest Air Force
acquisition plan. Second, schedule delays in developmental testing could
further postpone the start of the first F/ A- 22 multiyear contract, which
has already been delayed until fiscal year 2006. This could result in
lower cost savings from multiyear procurement. Last, we found several risk
factors that may increase future production costs, including the
dependency of certain cost reduction plans on Air Force investments that
are not being made to

improve production processes, the availability of funding, and a reduction
in funding for support costs. In addition, DOD has not informed Congress
about the quantity of aircraft that can be procured within existing
production cost limits, which we believe could be fewer than the 276
currently planned. Further details on F/ A- 22 cost growth and the Air
Force*s attempt to offset it are provided in appendix I.

5 U. S. General Accounting Office, Tactical Aircraft: DOD Needs to Better
Inform Congress about Implications of Continuing F/ A- 22 Cost Growth,
GAO- 03- 280 (Washington, D. C.: Feb. 28, 2003).

Page 15 GAO- 03- 645T Best Practices

While DOD*s new acquisition policy is too late to influence the F/ A- 22
program, it is not too late for other major acquisition programs like the
Missile Defense Agency*s suite of land, sea, air, and space defense
systems; the Army*s Future Combat Systems; and the Air Force and Navy*s
Joint Strike Fighter. DOD*s revised acquisition policy represents tangible
leadership action to getting better weapon system acquisition outcomes,
but unless the policies are implemented through decisions on individual
programs, outcomes are not likely to change. Further, unless pressures are
alleviated in DOD to get new acquisition programs approved and funded on
the basis of requirements that must stand out, programs will continue to
be compromised from the outset with little to no chance of successful
outcomes. If new policies were implemented properly, through decisions on
individual programs, managers would face less pressure to promise

delivery of all the ultimate capabilities of a weapon system in one *big
bang.*

Both form and substance are essential to getting desired outcomes. At a
tactical level, we believe that the policies could be made more explicit
in several areas to facilitate such decisions. First, the regulations
provide little or no controls at key decision points of an acquisition
program that force a program manager to report progress against knowledge-
based metrics. Second, the new regulations, once approved, may be too
general and may no longer provide mandatory procedures. Third, the new
regulations may not provide adequate accountability because they may not
require knowledge- based deliverables containing evidence of knowledge at
key decision points.

At a strategic level, some cultural changes will be necessary to translate
policy into action. At the very top level, this means DOD leadership will
have to take control of the investment dollars and to say *no* in some
circumstances if programs are inappropriately deviating from sound

acquisition policy. In my opinion, programs should follow a knowledgebased
acquisition policy* one that embraces best practices* unless there is a
clear and compelling national security reason not to. Other cultural
changes instrumental to implementing change include:

 Keeping key people in place long enough so that they can affect
decisions and be held accountable.  Providing program offices with the
skilled people needed to craft

acquisition approaches that implement policy and to effectively oversee
the execution of programs by contractors. Real Change in

Acquisition Outcomes Requires Disciplined Enforcement of Acquisition
Policy

Page 16 GAO- 03- 645T Best Practices

 Realigning responsibilities and funding between science and technology
organizations and acquisition organizations to enable the separation of
technology development from product development.  Bringing discipline to
the requirements- setting process by demanding a

match between requirements and resources.  Requiring readiness and
operating cost as key performance parameters

prior to beginning an acquisition.  Designing and implementing test
programs that deliver knowledge when

needed, including reliability testing early in design. Ultimately, the
success of the new acquisition policy will be seen in individual program
and resource decisions. Programs that are implementing knowledge- based
policies in their acquisition approaches should be supported and
resourced, presuming they remain critical to national needs and affordable
within current and projected resource levels. Conversely, if programs that
repeat the approaches of the past are approved and funded, past policies*
and their outcomes* will be reinforced with a number of adverse
implications.

DOD will continue to face challenges in modernizing its forces with new
demands on the federal dollar created by changing world conditions.
Consequently, it is incumbent upon DOD to find and adopt best product
development practices that can allow it to manage its weapon system
programs in the most efficient and effective way. Success over the long

term will depend not only on policies that embrace evolutionary,
knowledge- based acquisition practices but also on DOD leadership*s
sustaining its commitment to improving business practices and ensuring
that those adopted are followed and enforced.

DOD*s new acquisition policy embraces the best practice concepts of
knowledge- based, evolutionary acquisition and represents a good first
step toward achieving better outcomes from major acquisition programs. The
F/ A- 22 program followed a different path at its beginning, a big bang,
highrisk approach whose outcomes so far have been increased cost, quality
and reliability problems, growing procurement reductions, and delays in
getting the aircraft to the war fighter. Since this program is nearing the
end of development and already into production, it is too late to adopt a
knowledge approach, but it can limit further cost increases and adverse
actions by not ramping up production beyond current levels until
developmental and operational testing are completed and reliability goals
have been demonstrated. Regardless of the F/ A- 22*s current predicament,
the new policy can and should be used to manage all new acquisition
Conclusions

Page 17 GAO- 03- 645T Best Practices

programs and should be adapted to those existing programs that have not
progressed too far in development to benefit. At a minimum, the F/ A- 22
should serve as a lesson learned from which to effect a change in the
future DOD acquisition environment. The costs of doing otherwise are
simply too high for us to tolerate.

Mr. Chairman, this concludes my prepared statement. I would be happy to
respond to any questions that you or other members of the Subcommittee may
have.

Page 18 GAO- 03- 645T Best Practices

Over the last 6 years, DOD has identified about $18 billion in estimated
production cost growth during the course of two DOD program reviews. As a
result, the estimated cost of the production program currently exceeds the
congressional cost limit. The Air Force has implemented cost reduction
plans designed to offset a significant amount of this estimated cost
growth. But the effectiveness of these cost reduction plans has varied.
During a 1997 review, the Air Force estimated cost growth of $13.1
billion. 1 The major contributing factors to this cost growth were
inflation,

increased estimates of labor costs and materials associated with the
airframe and engine, and engineering changes to the airframe and engine.
These factors made up about 75 percent of the cost growth identified in
1997.

In August 2001, DOD estimated an additional $5.4 billion in cost growth
for the production of the F/ A- 22, bringing total estimated production
cost to $43 billion. The major contributing factors to this cost growth
were again due to increased labor costs and airframe and engine costs.
These factors

totaled almost 70 percent of the cost growth. According to program
officials, major contractors* and suppliers* inability to achieve the
expected reductions in labor costs throughout the building of the
development and early production aircraft has been the primary reason for
estimating this additional cost growth.

The Air Force was able to implement cost reduction plans and offset cost
growth by nearly $2 billion in the first four production contracts
awarded. As shown in table 3, the total offsets for these contracts
slightly exceeded earlier projections by about $. 5 million.

1 Based on a plan to procure 438 aircraft. Appendix I: F/ A- 22 Production
Cost Growth

Mixed Success With Cost Reduction Plans

Page 19 GAO- 03- 645T Best Practices

Table 3: Comparison of Planned Versus Implemented Cost Reduction Offsets
for Awarded Production Contracts

Dollars in millions Production lot Planned offsets Implemented

offsets Difference

Fiscal year 1999 (2 aircraft) $199.0 $200.5 $1.5 Fiscal year 2000 (6
aircraft) 329.3 336.4 7.1 Fiscal year 2001 (10 aircraft) 580.2 611.1 30.9
Fiscal year 2002 (13 aircraft) 827.2 788.2 (39.0)

Total $1,935.7 $1,936.2 $0.5

Source: Air Force.

Cost reduction plans exist but have not yet been implemented for
subsequent production lots planned for fiscal years 2003 through 2010
because contracts for these production lots have not yet been awarded. If
implemented successfully, the Air Force expects these cost reduction plans
to achieve billions of dollars in offsets to estimated cost growth and to
allow the production program to be completed within the current production
cost estimate of $43 billion. 2 However, this amount exceeds the
production cost limit of $36.8 billion. In addition, while the Air Force
has been attempting to offset costs

through production improvement programs (PIPs), recent funding cutbacks
for PIPs may reduce their effectiveness. PIPs focus specifically on
improving production processes to realize savings by using an initial
government investment. The earlier the Air Force implements PIPs, the
greater the impact on the cost of production. Examples of PIPs previously
implemented by the Air Force include manufacturing process improvements
for avionics, improvements in fabrication and assembly processes for the
airframe, and redesign of several components to enable lower production
costs.

As shown in figure 3, the Air Force reduced the funding available for
investment in PIPs by $61 million for lot 1 and $26 million for lot 2 to
cover

2 The F/ A- 22 President*s budget for fiscal year 2004 would transfer $876
million in production funding to help fund estimated cost increases in
development. As a result, the current production cost estimate is $42.2
billion.

Page 20 GAO- 03- 645T Best Practices

cost growth in production lots 1 and 2 3 . As a result, it is unlikely
that PIPs covering these two lots will be able to offset cost growth as
planned.

Figure 3: Planned Versus Actual F/ A- 22 Production Improvement Program
Investment for Production Lots 1 (Fiscal Year 2001) and 2 (Fiscal Year
2002)

Figure 4 shows the remaining planned investment in PIPs through fiscal
year 2006 and the $3. 7 billion in estimated cost growth that can
potentially be offset through fiscal year 2010 if the Air Force invests as
planned in

these PIPs. 3 Production lot 1 was awarded in fiscal year 2001 and
production lot 2 was awarded in fiscal year 2002.

Page 21 GAO- 03- 645T Best Practices

Figure 4: Planned Offsets to Cost Growth From Investing in and
Implementing PIPs In the past, Congress has been concerned about the Air
Force*s practice of requesting fiscal year funding for these PIPs but then
using part of that funding for F/ A- 22 airframe cost increases. 4
Recently, Congress directed the Air Force to submit a request if it plans
to use PIP funds for an alternate purpose.

4 Report 107- 298, Nov. 19, 2001.

Page 22 GAO- 03- 645T Best Practices

Best Practices: Setting Requirements Differently Could Reduce Weapon
Systems* Total Ownership Costs. GAO- 03- 57. Washington, D. C.: February
11, 2003.

Best Practices: Capturing Design and Manufacturing Knowledge Early
Improves Acquisition Outcomes. GAO- 02- 701. Washington, D. C.: July 15,
2002.

Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices.
GAO- 02- 469T. Washington, D. C.: February 27, 2002. Best Practices:
Better Matching of Needs and Resources Will Lead to

Better Weapon System Outcomes. GAO- 01- 288. Washington, D. C.: March 8,
2001.

Best Practices: A More Constructive Test Approach Is Key to Better Weapon
System Outcomes. GAO/ NSIAD- 00- 199. Washington, D. C.: July 31, 2000.

Defense Acquisitions: Employing Best Practices Can Shape Better Weapon
System Decisions. GAO/ T- NSIAD- 00- 137. Washington, D. C.: April 26,
2000.

Best Practices: DOD Training Can Do More to Help Weapon System Programs
Implement Best Practices. GAO/ NSIAD- 99- 206. Washington, D. C.: August
16, 1999.

Best Practices: Better Management of Technology Development Can Improve
Weapon System Outcomes. GAO/ NSIAD- 99- 162. Washington, D. C.: July 30,
1999.

Best Practices: DOD Can Help Suppliers Contribute More to Weapon System
Programs. GAO/ NSIAD- 98- 87. Washington, D. C.: March 17, 1998. Best
Practices: Successful Application to Weapon Acquisition Requires Changes
in DOD*s Environment. GAO/ NSIAD- 98- 56. Washington, D. C.:

February 24, 1998.

Best Practices: Commercial Quality Assurance Practices Offer Improvements
for DOD. GAO/ NSIAD- 96- 162. Washington, D. C.: August 26, 1996. Related
GAO Products

Page 23 GAO- 03- 645T Best Practices

Tactical Aircraft: Status of the F/ A- 22 Program. GAO- 03- 603T.
Washington, D. C.: April 2, 2003.

Tactical Aircraft: DOD Should Reconsider Decision to Increase F/ A- 22
Production Rates While Development Risks Continue. GAO- 03- 431.
Washington, D. C.: March 14, 2003.

Tactical Aircraft: DOD Needs to Better Inform Congress about Implications
of Continuing F/ A- 22 Cost Growth. GAO- 03- 280. Washington, D. C.:
February 28, 2003. Tactical Aircraft: F- 22 Delays Indicate Initial
Production Rates Should

Be Lower to Reduce Risks. GAO- 02- 298. Washington, D. C.: March 5, 2002.

Tactical Aircraft: Continuing Difficulty Keeping F- 22 Production Costs
within the Congressional Limitation. GAO- 01- 782. Washington, D. C.: July
16, 2001.

Tactical Aircraft: F- 22 Development and Testing Delays Indicate Need for
Low- Rate Production. GAO- 01- 310. Washington, D. C.: March 15, 2001.
Defense Acquisitions: Recent F- 22 Production Cost Estimates Exceeded

Congressional Limitation. GAO/ NSIAD- 00- 178. Washington, D. C.: August
15, 2000.

Defense Acquisitions: Use of Cost Reduction Plans in Estimating F- 22
Total Production Costs. GAO/ T- NSIAD- 00- 200. Washington, D. C.: June
15, 2000.

Budget Issues: Budgetary Implications of Selected GAO Work for Fiscal Year
2001. GAO/ OCG- 00- 8. Washington, D. C.: March 31, 2000.

F- 22 Aircraft: Development Cost Goal Achievable If Major Problems Are
Avoided. GAO/ NSIAD- 00- 68. Washington, D. C.: March 14, 2000. Defense
Acquisitions: Progress in Meeting F- 22 Cost and Schedule Goals. GAO/ T-
NSIAD- 00- 58. Washington, D. C.: December 7, 1999.

Fiscal Year 2000 Budget: DOD*s Procurement and RDT& E Programs. GAO/
NSIAD- 99- 233R. Washington D. C.: September 23, 1999.

Budget Issues: Budgetary Implications of Selected GAO Work for Fiscal Year
2000. GAO/ OCG- 99- 26. Washington, D. C.: April 16, 1999.

Page 24 GAO- 03- 645T Best Practices

Defense Acquisitions: Progress of the F- 22 and F/ A- 18E/ F Engineering
and Manufacturing Development Programs. GAO/ T- NSIAD- 99- 113.
Washington, D. C.: March 17, 1999.

F- 22 Aircraft: Issues in Achieving Engineering and Manufacturing
Development Goals. GAO/ NSIAD- 99- 55. Washington, D. C.: March 15, 1999.
F- 22 Aircraft: Progress of the Engineering and Manufacturing

Development Program. GAO/ T- NSIAD- 98- 137. Washington D. C.: March 25,
1998.

F- 22 Aircraft: Progress in Achieving Engineering and Manufacturing
Development Goals. GAO/ NSIAD- 98- 67. Washington, D. C.: March 10, 1998.
Tactical Aircraft: Restructuring of the Air Force F- 22 Fighter Program.
GAO/ NSIAD- 97- 156. Washington, D. C.: June 4, 1997.

Defense Aircraft Investments: Major Program Commitments Based on
Optimistic Budget Projections. GAO/ T- NSIAD- 97- 103. Washington, D. C.:
March 5, 1997. F- 22 Restructuring. GAO/ NSIAD- 97- 100BR. Washington, D.
C.: February 28, 1997.

Tactical Aircraft: Concurrency in Development and Production of F- 22
Aircraft Should Be Reduced. GAO/ NSIAD- 95- 59. Washington, D. C.: April
19, 1995.

Tactical Aircraft: F- 15 Replacement Issues. GAO/ T- NSIAD- 94- 176.
Washington, D. C.: May 5, 1994.

Tactical Aircraft: F- 15 Replacement Is Premature as Currently Planned.
GAO/ NSIAD- 94- 118. Washington, D. C.: March 25, 1994.

(120231)

This is a work of the U. S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
*** End of document. ***