Best Practices: Successful Application to Weapon Acquisitions Requires
Changes in DOD's Environment (Chapter Report, 02/24/98, GAO/NSIAD-98-56).

Pursuant to a congressional request, GAO reviewed whether commercial
practices offer ways to improve Department of Defense's (DOD) process
for transitioning weapons from development to production, focusing on:
(1) a comparison of DOD's practices for preparing a weapon system for
production with best commercial practices; (2) whether differences in
commercial and DOD environments for developing new products affect
practices; and (3) environmental changes that are key to the success of
DOD initiatives for improving the transition of weapons from development
to production.

GAO noted that: (1) commercial firms gained more knowledge about a
product's technology, performance, and producibility much earlier in the
product development process than DOD; (2) product development in
commercial ventures was a clearly defined undertaking for which firms
insisted on having the technology in hand to meet customer requirements
before starting; (3) once underway, these firms demanded--and
got--specific knowledge about a new product before production began; (4)
the process of discovery--the accumulation of knowledge and the elements
of unknowns--was completed for the best commercial programs well ahead
of production; (5) in contrast, DOD programs allowed more technology
development to continue into product development; (6) consequently, the
programs proceeded with much less knowledge--and thus more risk--about
required technologies, design capability, and producibility; (7)
although DOD accepted more unknowns on its programs than commercial
firms, it understated the risks present; (8) the commercial and defense
environments created different incentives and elicited different
behaviors from the people managing the programs; (9) specific practices
took root and were sustained because they helped a program succeed in
its environment--not because they were textbook solutions; (10) the
success of commercial product developments was determined when
production items were sold; (11) DOD programs began without needed
technology in hand; rather, they were encouraged to include undeveloped
technology; (12) because these programs ran much longer, production
concerns did not play as big a role and were not as critical to success
in the early stages; (13) the definition of success was more complicated
in DOD; (14) during most of product development, success was defined as
getting DOD and Congress as the customer, rather than the firm--to fund
the development annually; (15) commercial practices for gaining
knowledge and assessing risks can help produce better outcomes on DOD
acquisitions; (16) indeed, DOD has several commercial-like initiatives
underway; (17) at least two factors are critical to fostering an
environment conducive to such practices: (a) program launch decisions
must be relieved of the need to overpromise on technical performance and
resource estimates; and (b) once a program is underway, it must become
acceptable for program managers to identify unknowns as high risks so
that they can be aggressively worked on earlier in development; and (18)
currently, identifying a high risk on a DOD program is perceived as
inviting criticism and the loss of funding.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  NSIAD-98-56
     TITLE:  Best Practices: Successful Application to Weapon 
             Acquisitions Requires Changes in DOD's Environment
      DATE:  02/24/98
   SUBJECT:  Advanced weapons systems
             Research and development
             Defense cost control
             Comparative analysis
             Procurement evaluation
             Concurrency
             Cost effectiveness analysis
             Defense procurement
             Testing
IDENTIFIER:  C-17 Aircraft
             F-22 Aircraft
             AIM-9X Missile
             Joint Direct Attack Munition
             Boeing 777-300 Aircraft
             HS 702 Satellite
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Report to the Subcommittee on Acquisition and Technology, Committee
on Armed Services, U.S.  Senate

February 1998

BEST PRACTICES - SUCCESSFUL
APPLICATION TO WEAPON ACQUISITIONS
REQUIRES CHANGES IN DOD'S
ENVIRONMENT

GAO/NSIAD-98-56

Best Practices

(707188)


Abbreviations
=============================================================== ABBREV

  AIM - Air Intercept Missile
  CAIV - cost as an independent variable
  CATIA - computer-aided three-dimensional interactive application
  CDR - critical design review
  DOD - Department of Defense
  EMD - engineering and manufacturing development
  GAO - General Accounting Office
  IPT - integrated product team
  JDAM - Joint Direct Attack Munitions
  SPC - statistical process control

Letter
=============================================================== LETTER


B-275842

February 24, 1998

The Honorable Rick Santorum
Chairman
The Honorable Joseph I.  Lieberman
Ranking Minority Member
Subcommittee on Acquisition and Technology
Committee on Armed Services
Unites States Senate

As you requested, this report assesses whether commercial product
development practices offer ways to improve the process used at the
Department of Defense (DOD) to prepare weapon systems for production. 
It also determines how differences in commercial and DOD environments
for developing new products affect the corresponding practices.  We
make recommendations to the Secretary of Defense and present matters
for congressional consideration, all of which are intended to create
the necessary environment to ensure DOD's future success in improving
the transition of weapons from development to production. 

We are sending copies of this report to other congressional
committees; the Secretaries of Defense, the Army, the Navy, and the
Air Force; and the Director, Office of Management and Budget.  We
will also make copies available to others on request. 

This report was prepared under the direction of Katherine V. 
Schinasi, Associate Director of Defense Acquisitions Issues, who may
be reached at (202) 512-4841 if you or your staff have any questions. 
Other major contributors to this report are listed in appendix II. 

Henry L.  Hinton, Jr.
Assistant Comptroller General


EXECUTIVE SUMMARY
============================================================ Chapter 0


   PURPOSE
---------------------------------------------------------- Chapter 0:1

The Department of Defense (DOD) continues to state a need to
modernize weapons for the armed forces at a faster pace within
relatively level funds.  It has a budget of over $40 billion in
fiscal year 1998 to acquire and upgrade weapons, and may not
reasonably expect to receive much more than that amount in future
years.  Therefore, it must find new ways to modernize more
economically.  The challenge of the Soviet Union has been replaced by
a challenge to maintain weapons that have a leading technological
edge; however, such technologies are increasingly being developed in
the commercial sector.  In this period of lessened tensions, DOD has
an opportunity to revamp the practices it uses to acquire weapon
systems, avoid the technological obsolescence that comes with 15-year
and longer product development cycles, speed the pace of
modernization in the face of budgetary pressures, and meet defense
needs with sufficient flexibility.  The practices employed by some
commercial firms to reduce the time and money to develop new
products--by as much as 50 percent--can illuminate ways for DOD to
make similar improvements. 

In response to a request from the Chairman and Ranking Minority
Member, Subcommittee on Acquisition and Technology, Committee on
Armed Services, GAO assessed whether commercial practices offer ways
to improve DOD's process for transitioning weapons from development
to production.  Specifically, this report (1) compares DOD's
practices for preparing a weapon system for production with best
commercial practices, (2) determines how differences in commercial
and DOD environments for developing new products affect practices,
and (3) discusses environmental changes that are key to the success
of DOD initiatives for improving the transition of weapons from
development to production. 


   BACKGROUND
---------------------------------------------------------- Chapter 0:2

Commercial firms make a distinction between product and technology
development.  Product development entails the design and manufacture
of a product, such as an airplane, a car, or a satellite, as an end
item for delivery to a customer.  The basic features of product
development include a definable market in terms of customer needs or
wants, the ability to design a product for that market opportunity,
the ability to produce that product, and the investment capital
needed to fund the development effort.  Technology development
fosters technological advances for potential application to a product
development.  In DOD, the acquisition cycle for weapons includes
technology development, product development, and production.  Thus,
the distinction between product and technology development on an
individual program is much less clear. 

Among the key sources of information GAO relied on in this review
were individual DOD acquisition programs and several firms recognized
as being among the best in developing and manufacturing new products. 
Much commonality existed among these firms' practices, even though
their products spanned a range of technological sophistication.  In
this report, GAO highlights the best commercial practices in product
development based on its fieldwork.  As such, they are not intended
to describe all commercial industry or suggest that commercial firms
are without flaws. 


   RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3

Commercial firms gained more knowledge about a product's technology,
performance, and producibility much earlier in the product
development process than DOD.  Product development in commercial
ventures was a clearly defined undertaking for which firms insisted
on having the technology in hand to meet customer requirements before
starting.  Once underway, these firms demanded--and got--specific
knowledge about a new product before production began.  The process
of discovery--the accumulation of knowledge and the elimination of
unknowns--was completed for the best commercial programs well ahead
of production.  Not having this knowledge when demanded constituted a
risk the firms found unacceptable.  Immature or undeveloped
technology could not meet these demands and was kept out of
commercial product development programs; this technology was managed
separately until it could meet the demands for product development. 

In contrast, DOD programs allowed more technology development to
continue into product development.  Consequently, the programs
proceeded with much less knowledge--and thus more risk--about
required technologies, design capability, and producibility.  The
programs' discovery process persisted much longer, even after the
start of production.  Turbulence in program outcomes--in the form of
production problems and associated cost and schedule increases--was
the predictable consequence as the transition to production was made. 
Although DOD accepted more unknowns on its programs than commercial
firms, it understated the risks present. 

The commercial and defense environments created different incentives
and elicited different behaviors from the people managing the
programs.  Specific practices took root and were sustained because
they helped a program succeed in its environment--not because they
were textbook solutions.  The success of commercial product
developments was determined when production items were sold.  Until
that point, the firm's own money was at risk.  Failure was seen as
both clear and painful if the customer walked away.  This definition
of success, coupled with the realization that production startup was
relatively close at hand, made production concerns a key factor in
program start and subsequent decisions and provided strong incentives
to identify risks both early and realistically.  DOD programs began
without needed technology in hand; rather, they were encouraged to
include undeveloped technology.  Because these programs ran much
longer, production concerns did not play as big a role and were not
as critical to success in the early stages.  The definition of
success was more complicated in DOD.  During most of product
development, success was defined as getting DOD and the Congress, as
the customers, to fund the development annually.  Optimistic
assessments of performance and cost helped ensure this kind of
success; realistic risk assessments did not. 

Commercial practices for gaining knowledge and assessing risks can
help produce better outcomes on DOD acquisitions.  Indeed, DOD has
several commercial-like initiatives underway, such as using cost or
price as a means for forcing technology tradeoffs, that it believes
are having promising results.  For such practices to work on a broad
scale, however, the DOD environment must be conducive to such
practices.  At least two factors are critical to fostering such an
environment.  First, program launch decisions must be relieved of the
need to overpromise on technical performance and resource estimates. 
The pressure to amass broad support at launch creates incentives for
new programs to embrace far more technology development than
commercial programs.  Although technology development clearly must be
undertaken by DOD and supported by the Congress, its objectives, as
well as what can be demanded of its knowledge and estimates, differ
from those of product development.  Second, once a program is
underway, it must become acceptable for program managers to identify
unknowns as high risks so that they can be aggressively worked on
earlier in development.  Currently, identifying a high risk on a DOD
program is perceived as inviting criticism and the loss of funding. 
Studies sponsored by DOD and the defense industry call for the kinds
of changes that could help shape a better environment for managing
weapon acquisitions.  The challenge for DOD and the Congress is to
foster the environment that provides program managers with incentives
for applying best practices. 


   PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:4


      LEADING COMMERCIAL FIRMS
      ATTAIN MORE KNOWLEDGE
      EARLIER IN PRODUCT
      DEVELOPMENT
-------------------------------------------------------- Chapter 0:4.1

To minimize the amount of technology development that occurs during
product development, the companies GAO visited employ a disciplined
process to match requirements with technological capability before
the product development process begins.  This process is grounded in
production realities that demand proof that the technology will work
and can be produced at an acceptable cost, on schedule, and with high
quality.  The companies bring proven technological knowledge to the
requirements process in the form of current, high fidelity
information from predecessor programs, people with first-hand
experience on those programs, and new technologies deemed mature as a
result of having "graduated" from a disciplined technology
development and screening process.  Also, they communicate
extensively with customers to match their wants and needs to the
firms' available technology and ability to manufacture the desired
product.  They do not stray far from their technological foundation. 

For the programs GAO reviewed, DOD did not get as good a match
between what customers require and what technology could confidently
deliver.  Even though DOD examines the potential for available
technology to meet product requirements, it allows requirements to
make technology reach beyond what is proven.  DOD has less knowledge
about the ability of the design to deliver required performance and
be produced at this point in the program than commercial firms.  The
knowledge that the match has been achieved between requirements and
technology is often not attained until testing is completed, late in
development or after production begins.  Commercial and DOD decisions
made on a lightweight aircraft material--aluminum lithium--are
illustrative of the differences between the two sectors.  Boeing had
initially decided to use the alloy on its 777-200 aircraft but
rejected its use early in development because it was expensive, its
manufacturing processes were not well understood, and its
availability was limited.  DOD accepted these risks and used the
alloy on the C-17 aircraft.  The alloy is now being phased out of the
program as some of its unknowns became actual problems. 

Before the halfway point of product development, the commercial firms
GAO visited achieved near certainty that their product designs would
meet customer requirements and had gone a long way to ensure that the
product could be produced.  Both DOD and commercial firms hold a
critical design review to determine whether the design is mature and
essentially complete for production purposes.  Engineering drawings,
which document the schematics of the product, along with its
performance, required materials and production processes, comprise
much of the source material for the critical design review. 
Commercial firms typically had over 90 percent of these drawings
available for the review.  In comparison, the C-17 program had less
than 60 percent and the F-22 program less than one-third of the
drawings available for the review.  Over one-fifth of the C-17's
drawings became available after production began, and the aircraft
experienced a number of problems in production as difficulties with
the design were worked out.  Several key technologies are still
unproven on the F-22, and some will not be proven until after 40
aircraft have entered production.  Nonetheless, the risks of
proceeding with the rest of development as planned at the time of the
critical design reviews for both programs were deemed acceptable. 

The companies GAO visited reached the point at which they knew that
manufacturing processes would produce a new product conforming to
cost, quality, and schedule targets before they began fabricating
production articles.  Reaching this point meant more than knowing the
product could be manufactured; it meant that all key processes were
under control, such that the quality, volume, and cost of their
output were proven and acceptable.  The DOD programs demanded less
proof of a design's producibility.  The C-17 program began production
in 1989 and still has less than 13 percent of its key manufacturing
processes in control.  The F-22 program is currently faring better
than the C-17:  the contractor believes it has almost 40 percent of
its key manufacturing processes in control, 2 years before production
is scheduled to begin.  However, the program does not plan to have
all key processes in control until about
4 years into production. 


      DIFFERENCES IN MILITARY AND
      COMMERCIAL PRACTICES REFLECT
      DIFFERENT ENVIRONMENTS
-------------------------------------------------------- Chapter 0:4.2

The commercial firms GAO contacted launch a product development only
when a solid business case can be made.  The business case basically
revolves around the ability to produce a product with the right
features to meet the market opportunity on schedule, with limited
investment capital, and at a predictable unit cost so that the
product will sell well enough to make an acceptable return on
investment.  Because success is determined in production, the
business case for launching a program considers production realities
and builds in natural curbs to overreaching for performance, cost, or
schedule.  A company demands considerable proof that the product will
fulfill all of the business case factors and then provides full
support for the program to succeed.  The business case provides a
very solid decision-making framework from the outset and throughout
the program.  Commercial companies build relatively short cycle
times, keyed to meeting market demands, into their decisions to begin
a product's development.  These short timeframes, together with the
responsibility for protecting the business case, encourage program
managers to identify risks and enable them to say "no" to pressures
to accept unknowns.  The companies are conservative in their
estimates and aggressive in reducing risk.  The abundance of reliable
data and experienced people from predecessor programs provides a
solid factual basis for defining unknowns and assessing risks. 

Because DOD launches programs earlier, the knowledge commercial firms
insist on is generally not available for a DOD program until years
later.  Some knowledge, such as high-fidelity information from
numerous closely related predecessor programs, is simply unavailable. 
Even though less information about a new DOD product development is
available at the time of launch, the competition for funding requires
detailed projections to be made from the information that does exist. 
Although DOD is attempting to ease the technical requirements of
programs, a new product development must possess performance features
that distinguish it from either the systems to be replaced or rival
candidates.  Consequently, aspiring DOD programs are encouraged to
include performance features and design characteristics that rely on
immature technologies.  Untempered by knowledge to the contrary, the
risks associated with these technologies are deemed acceptable. 
Production realities, critical to matching technological capabilities
with customer requirements on commercial programs, are too far away
from the DOD launch decision to have the same curbing effect on
technology decisions. 

In this environment, risks in the form of ambitious technology
advancements and tight cost and schedule estimates are accepted as
necessary for a successful launch.  Problems or indications that the
estimates are decaying do not help sustain the program in subsequent
years, and thus their admission is implicitly discouraged.  An
optimistic production cost estimate makes it easier to launch a
product development and sustain annual approval; admission that costs
are likely to be higher could invite failure.  There are few rewards
for discovering and recognizing potential problems early in the DOD
product development, given the amount of external scrutiny the
programs receive.  The behavior of tolerating unknowns and not
designating them the same risk value as in the commercial environment
is manifested in the DOD environment because there is little
incentive to admit to high risks before it is absolutely necessary. 
This is not to say that DOD program managers and other sponsors act
irrationally or with bad intentions.  Rather, they see the programs
under their purview as aligned with the national interest and do what
they believe is right, given the pressures they face. 


      THE RIGHT ENVIRONMENT WILL
      BE KEY TO THE SUCCESS OF DOD
      INITIATIVES TO IMPROVE
      WEAPON ACQUISITIONS
-------------------------------------------------------- Chapter 0:4.3

DOD has embarked on several acquisition reform initiatives that draw
lessons from commercial practices.  These initiatives include using
cost or price targets as vehicles for forcing requirements or
technology tradeoffs, evaluating contractors' past performance as a
factor in selecting sources for new contracts, removing
specifications that told contractors how to design or build a product
in favor of specifying what performance was required of the end item,
and using multidisciplinary teams to make decisions and tradeoffs on
individual weapon systems.  DOD reports that programs are showing
potential reductions in cycle time and staff resources as a result of
these initiatives.  DOD has also recently set up a funding reserve to
offset unexpected cost growth to mitigate the effect of unknowns on
programs.  In past acquisitions, changing the mechanics of a weapon's
development, without changing aspects of its environment that
determine its incentives, did not produce desired results. 
Nonetheless, DOD's current initiatives could help the transition of
weapons into production if the environment for launching programs and
appraising risks can be changed to better approximate the commercial
environment. 

Studies sponsored by DOD and the defense industry call for changes
that could help shape such an environment.  A May 1996 study by the
Defense Science Board recommended replacing the current weapon
acquisition process with one that emphasizes incremental technology
advancement, coupled with much shorter product development cycle
times.  The study also noted that technology development should be
aggressively pursued outside the realm of weapon system development
programs.  In April 1996, the National Center for Advanced
Technologies also proposed a change in the DOD weapons acquisition
process to reduce cycle time to the 3- to 5-year range by following
an incremental technology advancement approach.  The report noted
that although concepts such as DOD's initiatives are constructive,
there is no reason to assume that these concepts should succeed when
previous good ideas did not. 

A December 1994 study by the Defense Systems Management College made
several recommendations to create an environment for weapon systems
development to encourage realism in reporting program status.  A
significant recommendation is that the individual military services
transfer control of their acquisition organizations and people to the
Under Secretary of Defense for Acquisition and Technology.  The study
noted that the Under Secretary would then be empowered to reward
candor and realism in reporting through the use of assignments and
promotions. 


   RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5

GAO makes several recommendations to the Secretary of Defense.  These
recommendations are intended to (1) ensure that sound standards for
the timing and quality of production-related knowledge are applied to
individual programs, (2) separate technology development from weapon
system programs to enable DOD to meet higher knowledge standards on
those programs, and (3) use decisions on individual weapon systems to
foster the right environment for identifying risks early and send the
right message to other programs concerning what practices will work
in that environment.  These recommendations appear in full in chapter
5. 


   MATTERS FOR CONGRESSIONAL
   CONSIDERATION
---------------------------------------------------------- Chapter 0:6

Because of its critical role in creating the environment for what
constitutes program success and which practices will work, the
Congress may wish to consider supporting the Secretary of Defense's
efforts to create such an environment through changes to the
acquisition process that provide program managers clear incentives
for gaining sufficient knowledge at key points in weapon acquisition
programs.  The best commercial practices described in this report
suggest what may constitute "sufficient" levels of knowledge.  If DOD
takes steps to manage technology development efforts outside the
bounds of individual weapon system programs, the Congress may wish to
consider providing the funds needed for such efforts.  The Congress
could also help create the right incentives on individual programs by
favorably considering DOD funding requests to mitigate high risks
early in a program and cautiously considering late requests for funds
to resolve problems that should have been addressed earlier. 


   AGENCY COMMENTS
---------------------------------------------------------- Chapter 0:7

DOD agreed with the report and all of the recommendations.  A
discussion of DOD's actions appears on pages 75-77.  DOD's comments
appear in appendix I. 


INTRODUCTION
============================================================ Chapter 1

The Department of Defense (DOD) continues to state a need to
modernize weapons for the armed forces at a faster pace within
relatively level funds.  It has a budget of over $40 billion in
fiscal year 1998 to acquire and upgrade weapons, and may not
reasonably expect to receive much more than that amount in future
years.  Therefore, it must find new ways to modernize more
economically.  The challenge of the Soviet Union has been replaced by
a challenge to maintain weapons that have a leading technological
edge; however, such technologies are increasingly being developed in
the commercial sector.  In this period of lessened tensions, DOD has
an opportunity to revamp the practices it uses to acquire weapon
systems, avoid the technological obsolescence that comes with 15-year
and longer product development cycles, speed the pace of
modernization in the face of budgetary pressures, and meet defense
needs with sufficient flexibility.  When dealing with some similar
problems, the best commercial firms have focused on the early stages
of the product development cycle to speed up product developments,
get them into production sooner, and achieve cost goals. 


   PRODUCT DEVELOPMENT IN THE
   COMMERCIAL AND DEFENSE SECTORS
---------------------------------------------------------- Chapter 1:1

For commercial firms, developing a product entails the design and
manufacture of an end item, such as an airplane, car, or satellite,
for delivery to a customer.  The basic features of product
development include a definable market in terms of customer needs or
wants, the ability to design a product for that market opportunity,
the ability to produce that product, and the investment capital
needed to fund development.  Product development is therefore very
concerned with production realities and cost and schedule targets. 
It is distinct from technology development, which furthers the
advancement of technology to see if it has potential application to a
product.  For example, a firm that makes satellites may study new
materials and their properties to see if they can reduce the weight
and size of solar array batteries, thus improving a satellite's
carrying capacity.  If the technology can be perfected and is
feasible to use, it is ready to be included in a new product
development.  More failures are expected in a technology development,
since it is a process of discovery that deals with many unknowns. 

In DOD, the same functions take place on the weapon systems DOD
develops, but they occur within the broader context of the
acquisition cycle.  The entire cycle includes technology development,
product development, and production.  DOD's development process is
designed to manage a program through sequential phases, all followed
by major milestone decisions in which decisionmakers approve or
disapprove the acquisition strategy for a program and its move into
the next phase based on progress made as presented by the program
manager.  The phases of the acquisition cycle, as shown in figure
1.1, are:  concept exploration; program definition and risk
reduction; engineering and manufacturing development (EMD); and
production, fielding/deployment, and operational support.  The
concept exploration phase decides what kind of weapon, if any, is the
best solution to a military need.  A new program actually starts with
the program definition and risk reduction phase. 

   Figure 1.1:  DOD's Acquisition
   Process

   (See figure in printed
   edition.)

   Source:  DOD.

   (See figure in printed
   edition.)

Within the acquisition cycle for a given weapon system, technology
development, product development, and production activities take
place over a number of years.  The delineation of these activities is
not always clearly discernable.  Thus, even though the product
development activities described for the commercial sector would
occur largely in a weapon's EMD phase, technology development and
production activities also occur in this phase.  Likewise, some
product development activities for a weapon system take place during
production.  The acquisition process is embodied in DOD's Regulation
5000.2, Mandatory Procedures for Major Defense Acquisition Programs,
which was revised in 1996.  In addition, DOD's 1985 Manual 4245.7-M,
Transition From Development to Production, contains risk-reduction
guidance based on best practices. 

At any given time, DOD has hundreds of products in various stages of
development and production, each with a program manager from one of
the three services or other procuring agencies within DOD.  Although
DOD maintains oversight for all of these acquisitions and controls
their funding, the actual management practices employed are specific
to the program, the contractors, and the relevant government
organization.  We have issued a number of reports on weapon system
acquisitions since the 1970s. 

The program manager and staff for a new product's development in
either sector face a formidable task.  They are responsible for
balancing the requirements for the product's performance against its
established cost and schedule targets.  This responsibility includes
preparing an acquisition strategy for funding, testing, and
manufacturing the product and ensuring that the strategy is carried
out.  Throughout the development process, the program manager is
expected to identify and manage high-risk areas, which means making
decisions about tradeoffs among cost, schedule, and performance to
deliver a product that will satisfy the customer.  The program
manager and staff must therefore have expertise in many areas, such
as estimating techniques, the budget process, simulation, testing,
engineering, logistics, and production.  The program manager is the
one person who ultimately must know whether there is a match among
financial, time, and personnel resources provided by upper management
and what the product development effort can deliver.  In product
development, the financial investment is substantial and the stakes
are high; program failure is not taken lightly. 


   THE CASE FOR A SMOOTHER AND
   FASTER DEVELOPMENT TO
   PRODUCTION CYCLE
---------------------------------------------------------- Chapter 1:2

Products that proceed more quickly and smoothly through development
into production require lower financial investment costs to develop. 
Also, a smoother and faster process enhances the product's impact on
customers or users and benefits the producer by providing improved
profitability and market share.  Commercial firms that have been
recognized as industry leaders have significantly reduced the time it
takes to complete the product development cycle.  They have put new
products into production more quickly within unit production cost
targets and improved the products' performance features and quality. 
For example, Boeing will reduce development time by 40 percent on its
777-300 airplane compared with its 777-200 development, Hughes
develops satellites in
26 months and will soon reduce the time to 12 months, and Chrysler
has reduced its new vehicle development times from over 5 years to
less than 30 months. 

If DOD made reductions of the same magnitude in the time it takes to
develop a weapon, it could reap similar benefits.  A compelling
reason for DOD to pursue such reductions is to resolve long-standing
difficulties in controlling cost and schedule outcomes of major
weapon system programs.  According to cost and schedule data from 93
major acquisitions started since 1975, the acquisitions overran
original schedule estimates by an average of 24 percent.  Recent DOD
studies have concluded that weapon system development programs
typically overrun costs by 20 to 40 percent and that acquisitions
average 16 to 18 years.  A 1993 Rand study concluded that there had
been no improvement in controlling cost growth on the average weapon
system.  These results have persisted despite the implementation of
various initiatives to mitigate cost risk and growth, including
significant risk management guidelines DOD instituted in 1985 to
improve the transition to production. 

Another reason for DOD pursuing faster and smoother development to
production cycles is to lower its investment costs, enabling the
services to modernize at a faster pace within existing funding
levels.  According to DOD, modernization in recent years has been
sacrificed to improve readiness in the forces.  DOD believes that
modernization should proceed more quickly and wants to increase the
annual investment in procurement by $20 billion, but such increases
have not materialized.  Shorter development cycles would also help
mitigate the possibility that technology will become obsolete while a
weapon is still in development.  Technological obsolescence has
become a significant challenge for DOD acquisition programs whose
developments span many years.  Finally, faster product developments
would deliver improved products and better capabilities to the
military forces sooner. 

DOD recognizes the importance of reducing product development costs
and cycle times.  DOD's 1996 Technology Area Plan recognizes that the
prevailing acquisition environment made affordability and rapid cycle
times for acquisition of new systems key to maintaining an
appropriate mix of systems and forces that are ready to respond to
the defense missions of the future.  The plan recognized that short
cycle times invariably create cost reductions that would have clear
implications for readiness and modernization.  It also noted that
world-class commercial companies have demonstrated overall product
development and production cost reductions as well as cycle time
reductions of about 50 percent.  The plan states that, despite the
major differences in products and requirements, cost and cycle time
savings in that same range are feasible for military programs as
well. 

DOD is attempting to improve the product development process by
introducing acquisition reform initiatives that mirror world-class
companies' product development practices.  DOD believes that these
initiatives will result in better outcomes.  A recent Defense Science
Board study\1 concluded that the research and development phase of
military systems should be revamped and should adopt best commercial
practices.  The aerospace industry has also offered suggestions based
on commercial practices for improving DOD's development process for
weapons as a basis for change. 


--------------------
\1 A Streamlined Approach to Weapon Systems Research, Development and
Acquisition:  The Application of Commercial Practices, a Report from
the Defense Science Board Task Force on Defense Acquisition Reform,
May 1996. 


   OBJECTIVES, SCOPE, AND
   METHODOLOGY
---------------------------------------------------------- Chapter 1:3

The Chairman and the Ranking Minority Member, Subcommittee on
Acquisition and Technology, Senate Committee on Armed Services,
requested that we examine various aspects of the acquisition process
to identify best practices that might be useful or should be
strengthened.  This report covers one aspect of the acquisition
process, the transition from development to production.  Our overall
objective was to determine whether commercial practices offer ways to
improve DOD's product development process as it relates to the
transition to production.  Specifically, this report (1) compares
DOD's practices for preparing a weapon system for production with
best commercial practices, (2) determines how differences in the
commercial and DOD environments for developing new products affect
specific practices, and (3) discusses environmental changes that are
key to the success of DOD initiatives for improving the transition of
weapons from development to production. 

Even though we selected firms with product lines of varying
complexity, we did not concentrate only on firms whose products had
the most in common with weapon systems.  Such an approach would have
limited our ability to include firms recognized as the best at
getting new products into production.  In our analysis, we
concentrated on the criteria and knowledge used to support product
decisions.  Although the approach from product to product may vary,
the basic processes and standards the best commercial firms applied
to product decisions were consistent.  We were limited in our ability
to obtain and present some relevant data that commercial companies
considered proprietary in nature.  This information included funding
amounts for investing in product development or for recurring
production costs; the exact number and cost of changes to engineering
drawings on a product; and specific costs of scrap, rework, and
repair on products. 

To determine historical cost and schedule outcomes on past DOD
acquisitions, we examined Rand's database of Selected Acquisition
Reports for over 200 acquisitions since the 1960s.  To determine
current DOD policy and practices and identify new initiatives, we
interviewed and obtained documents from officials of the Office of
the Secretary of Defense in Washington, D.C.  For information from
acquisitions on specific risk management practices and results, we
interviewed officials and reviewed acquisition and risk management
documentation from the following two major DOD acquisition programs,
which are in different stages of the process: 

  -- The C-17 aircraft, an air refuelable, four-engine jet transport,
     is being developed by McDonnell Douglas Corporation (since
     acquired by Boeing).  It received full-rate production authority
     in November 1995.  The Air Force estimates the cost to develop
     and produce 120 C-17 aircraft at about $43 billion, amounting to
     a program unit cost of $358 million.  Procurement unit cost is
     estimated by the Air Force at $298 million. 

  -- The F-22 fighter, the next-generation air superiority fighter,
     is being developed by Lockheed Martin Aeronautical Systems
     Company.  It is currently in EMD.  The Air Force estimates the
     cost to develop and produce 339 F-22s at about $62.2 billion,
     amounting to a program unit cost of $183 million.  Procurement
     unit cost is estimated at $127 million. 

We also reviewed the following newer DOD acquisition programs that
have implemented some of DOD's acquisition reform initiatives;
because these programs are newer, our review was more limited: 

  -- The AIM-9X Sidewinder Missile, a joint Navy/Air Force,
     launch-and-leave air intercept missile, is being developed by
     Hughes Corporation.  The program is currently in EMD.  The
     services estimate the cost to develop and produce 10,049 AIM-9X
     missiles at about $3.2 billion, amounting to a program unit cost
     of $320,000.  Procurement unit cost is estimated at $264,000. 

  -- The Joint Direct Attack Munitions (JDAM), a joint Navy/Air Force
     tail kit to be attached to a 2,000-pound free-fall bomb and
     converting it to a guided munition, is being developed by
     McDonnell Douglas Corporation (since acquired by Boeing).  The
     JDAM program is currently in EMD.  The services estimate the
     cost to develop and produce 87,500 JDAM kits at about $3.39
     billion, amounting to a program unit cost of $38,700. 
     Procurement unit cost is estimated at $32,900. 

To understand the product development process and best practices from
the commercial sector, we conducted general literature searches and
focused those searches as the assignment progressed.  We also met
with several experts in the area of product development, including
representatives of Indiana University, Bloomington, Indiana; the
University of Michigan, Ann Arbor, Michigan; and the Navy's Center of
Excellence for Best Manufacturing Practices, College Park, Maryland. 
On the basis of our literature searches and discussions with experts,
we identified a number of commercial firms as having innovative
development processes and practices that resulted in successful
transitions to production.  We developed a data collection instrument
to assist us in gathering uniform, quantifiable measurements about
each firm's product development process and practices and the results
they accomplished.  We visited the following commercial firms, all
identified in our literature searches, and followed the same agenda
with each one: 

  -- Boeing Commercial Airplane Group (airplane manufacturer),
     Everett, Washington;

  -- Chrysler Corporation (automobile manufacturer), Auburn Hills,
     Michigan;

  -- Cummins Engine Company (engine manufacturer), Columbus, Indiana;

  -- Ford Motor Company (automobile manufacturer), Dearborn,
     Michigan;

  -- Honda Motor Company (automobile manufacturer), Raymond, Ohio;
     and

  -- Hughes Space and Communications (satellite and spacecraft
     manufacturer), Los Angeles, California. 

Our report highlights the best commercial practices in product
development based on our fieldwork.  As such, they are not intended
to describe all commercial industry and practices or suggest that
commercial firms are without flaws. 

We conducted our review between October 1996 and September 1997 in
accordance with generally accepted government auditing standards. 


KNOWLEDGE AT KEY JUNCTURES IS
CRITICAL TO A SUCCESSFUL
TRANSITION TO PRODUCTION
============================================================ Chapter 2

Product development can be characterized as the reduction of risk and
resolution of unknowns through the acquisition of knowledge. 
Compared with the DOD programs we reviewed, commercial firms gain
more knowledge about a product's performance, producibility, and
ability to meet customer requirements much earlier in the product
development process.  The best firms will not launch a new product
development unless they have high confidence that they have achieved
a match between what the customer wants and what the firms can
deliver.  The firms' keen knowledge of their technological
capabilities and limitations frames the design features they are
willing to offer.  Before the midway point in product development,
the firms attain enough knowledge to ensure that the product works. 
By the end of product development, but before fabrication of
production items begins, commercial firms prove that their production
methods will yield the desired volume and quality of the product
within the desired unit cost.  Throughout product development, the
firms adhere to their own proven standards as to what constitutes an
acceptable level of knowledge.  By the time actual production begins,
few risks or unknowns remain. 

The DOD programs we reviewed allowed more technology development to
carry over into product development.  Consequently, they proceeded
through product development with much less knowledge about required
technologies, design capability, and producibility than the
commercial firms we visited.  Two programs--the C-17 and the
F-22--did not or will not attain the same level of knowledge of the
design's ability to perform or be produced until late in development
or early production.  Attaining the match between product design and
customer requirements will not be certain until testing is completed
late in development, a practice that is not acceptable in commercial
product developments.  Proof that production methods can yield the
desired volume and quality within the desired unit cost will occur
after production begins.  As a result, the DOD programs tend to have
more unknowns and greater risks to manage when production fabrication
begins.  DOD's review mechanisms tended to overstate knowledge or
understate risks, as borne out by problems or unknowns discovered
later in product development.  Two programs we reviewed--JDAM and the
AIM-9X--have attempted to make a closer match up front between
demonstrated technology and customer requirements.  They are thus in
a better position to gain critical knowledge sooner in the remainder
of their development phases. 


   COMMERCIAL FIRMS DEVELOP
   KNOWLEDGE SOONER IN THE PRODUCT
   DEVELOPMENT CYCLE
---------------------------------------------------------- Chapter 2:1

The successful management of cost, schedule, and performance risk in
transitioning a new product from development to production is related
to how soon full knowledge about key dimensions of the product is
attained.  In this sense, knowledge is not the same as information. 
Knowledge, in this context, means that program managers and
decisionmakers have reached virtual certainty about an aspect of the
product being developed, such as a critical manufacturing process. 
It is the inverse of risk.  Information is essential to attaining
knowledge, but not all information produces the same level of
certainty.  For example, a study that shows a new manufacturing
process should work does not produce the same degree of certainty
that actual use of the process on another product does.  For the
purposes of comparing commercial and DOD product development cycles,
we have characterized the point at which virtual certainty of some
aspect of a product is achieved as a "knowledge point."

The commercial and military programs we reviewed did not all follow
the same processes or steps in their development cycles.  However, at
some point, full knowledge was or will be achieved about a completed
product, regardless of what development approach was taken. 
Knowledge on product developments can be broken down into three
points or junctures:  when a match is made between the customer's
requirements and the available technology; when the product's design
is determined to be capable of meeting performance requirements; and
when the product is determined to be producible within cost,
schedule, and quality targets.  Program launch or start is the point
at which organizations define a product's performance, cost, and
schedule estimates and commit to making the financial investment
needed to complete development and bring the product into production. 
Figure 2.1 illustrates the three knowledge points and the differences
between the commercial and military product developments in terms of
when they attain knowledge. 

   Figure 2.1:  Comparison of
   Three Key Knowledge Points for
   Commercial and Military Product
   Developments

   (See figure in printed
   edition.)

Because military programs tend to start product development with more
unknowns, it takes them additional time, sometimes until well after
production begins, to actually discover and capture enough solid
information to attain full product knowledge and thereby virtually
eliminate risk.  The programs continue to resolve risks about the
performance of the product design and its producibility after they
begin production.  Commercial firms attain full product knowledge
before fabrication of production units begins. 

Decisions made in matching requirements with technology have a direct
impact on what is required to reach subsequent knowledge points. 
Synergy exists when a decision early in product development, such as
establishing product requirements, reduces the amount of unknowns
that have to be resolved before full knowledge of product design
performance and producibility is attained.  Conversely, a decision to
reach for a technological advance can increase the effort required to
attain full design performance knowledge, which, in turn, can delay
the attainment of producibility knowledge. 


   KNOWLEDGE POINT 1: 
   REQUIREMENTS AND TECHNOLOGICAL
   CAPABILITY ARE MATCHED
---------------------------------------------------------- Chapter 2:2

To minimize the amount of technology development that occurs during
product development, the commercial companies we visited employ
disciplined processes to match requirements with technological
capability before product development begins.  The process reflects
production realities and demands proof that the technology will work
and can be produced at an acceptable cost, on schedule, and with high
quality.  The companies bring solid technological knowledge to the
requirements process in the form of current, high-fidelity
information from predecessor programs, people with first-hand
experience on those programs, or new technologies deemed mature as a
result of having "graduated" from a disciplined technology
development and screening process.  In addition, the companies
communicate extensively with customers to match their wants and needs
to the firm's available technology and its ability to manufacture an
appropriate product.  They do not stray far from their technological
foundation.  In the commercial world, this process is sometimes
referred to as Quality Functional Deployment, a technique used to
convert complex or unclear customer requirements into design and
manufacturing requirements. 

For the programs we reviewed, DOD did not get as good a match between
customer requirements and the technology that could be confidently
delivered.  Usually, the individual services identify a need for a
capability and translate that need into specific weapon performance
objectives and thresholds.  Even though DOD examines the potential
for available technology to meet product requirements, it allows
requirements to drive technology and make it reach beyond what is
proven.  DOD has less knowledge than commercial firms--in the form of
predecessor programs, people with experience on those programs, and
technology maturity--about the ability of the design to deliver
required performance and be produced at the point it starts a
program.  The knowledge that the match has been achieved between
required performance and the capability of technology is often not
attained until after production begins, when operational testing is
completed. 


      COMMERCIAL PRACTICES
-------------------------------------------------------- Chapter 2:2.1

We found examples of best commercial practices for matching
requirements to technology at Boeing, Hughes, and Ford.  Boeing
communicated with the airlines to set achievable requirements for the
777-200 airplane and tested the design early.  Hughes used a
technology development process that graduated new technologies from
concept into a product development program, enabling the firm to make
what it saw as quantum performance increases with mature
technologies.  Ford uses its technology deployment process to
separate immature technology from a new product's development.  It
enforces a decision point in product development in which new
technology must show proof that it will meet cost and performance
expectations to be accepted onto a product's development program. 

Boeing established requirements for the 777-200 airplane after
extensive communications with several airlines over an 18-month
period before beginning product development.  Originally, the
airlines thought that a 767 derivative would serve their purposes. 
However, during the course of discussions, Boeing and the airlines
realized that a "family" of airplanes that would provide the
airlines' range and payload flexibility was required.  During that
time, Boeing matched customer's needs to its technological capability
using an airline working group made up of airline representatives to
establish market and product requirements and extensive performance
testing to these requirements.  Boeing tested all of the airplane's
various systems together as a single, integrated entity in simulated
flight conditions very early in product development.  In addition,
Boeing completed an initial design review that ensured the design's
performance, identified all risks, made plans for their resolution,
and selected a final configuration. 

The diligence Boeing applied to matching requirements and
technological capability on the 777 was a practice that resulted from
painful experiences on two predecessor programs.  On one program,
Boeing had erred on the side of proposing too many new technologies
and design features on a new aircraft design.  Boeing halted the
program before the launch point, after one airline customer
complained that it contained "too much technology for technology's
sake." In retrospect, Boeing was relieved because the design
contained too much risk to make the sound business case needed for
the launch decision.  On another program, Boeing had erred too much
on the side of allowing customers to add requirements after program
launch.  This program was completed, and aircraft were delivered;
however, delays occurred in the development phase due to reworking
the design, and the early production aircraft experienced problems
with reliability rates.  The requirements process for the 777-200
applied the lessons learned from both programs.  Figure 2.2 shows the
Boeing 777 airliner. 

   Figure 2.2:  Boeing 777
   Airliner

   (See figure in printed
   edition.)

   Lessons learned from previous
   programs helped Boeing match
   technology with requirements
   before launching the 777.

   (See figure in printed
   edition.)

   Source:  Boeing Commercial
   Airplane Group.

   (See figure in printed
   edition.)

Because of early communication and testing, Boeing met the airlines'
requirement with plans for a family of airplanes in the medium-lift
category that would meet the airlines' needs in a timely manner.  The
777-200 model was first, followed by models that would increase
capacity and range.  Boeing also found that it could meet
requirements for the 777-200 using mostly existing technologies. 
When new technologies were needed, such as digital avionics and
advanced materials, they had to be demonstrated with laboratory
testing, simulation, and modeling before introducing them to the
development program.  Boeing also decided against some new technology
on the 777 as the result of its matching of requirements to
capability.  For example, Boeing originally designed the airplane
using a new, lightweight material--aluminum lithium--for some
portions.  However, at the design review that preceded the start of
development, manufacturing members on the design/build team argued
that the material was expensive and hard to find and that the
fabrication processes for it were not yet fully understood.  Despite
the increase in weight that would result, the team decided against
using the material.  Boeing made the customers aware of the problem
and worked with them to reduce the weight.  Ultimately, the final
777-200 configuration was
1,500 pounds under its originally proposed weight. 

According to Hughes Space and Communications, its newest product, the
HS-702 satellite, was a quantum leap in satellite technology.  It
provided twice as much power and capacity as its predecessor;
however, it was made with mature technology.  Hughes used its
technology development process to demonstrate new technology before
inserting it into the HS-702's development.  Specifically, it uses
Technology Roadmaps to prioritize technology against business needs
and corporate strategies, and investments are made accordingly. 
Product development managers will not use the resultant technology
until it proves capable of meeting product performance, cost,
schedule, and quality requirements and manufacturing processes are in
place and in control. 

Hughes identified three technologies it considered high risk when it
began development of the HS-702 satellite:  the gallium arsenide
solar cells and solar concentrators, the xenon-ion propulsion system,
and the deployable radiator needed to radiate additional heat
resulting from the increased power.  The new solar cell and
propulsion system technologies had been through the corporate
technology development process and had been applied on predecessor
programs, so there was significant knowledge about them at the
outset.  The deployable radiator was high risk because Hughes did not
believe it had the technology for this capability.  Hughes found a
Russian supplier that was already producing the required technology,
greatly reducing technical risk.  At Hughes, technology that is not
proven through predecessor programs or the development process is not
allowed onto a new product.  For example, Hughes wanted to include
lithium ion batteries and advanced receiver technologies on the
HS-702 satellite to decrease weight and improve its capacity and
power, but it did not include them because these technologies had not
met cost, schedule, performance, and quality targets. 

Ford reduces risk from new technology during product development with
its Technology Deployment Process.  This process prioritizes,
matures, and introduces new or emerging technologies for product
development, reducing risk from the time the concept is proposed to
its ultimate implementation on a new product.  During this process,
technologists agree to deliver new technology to vehicle centers for
product development.  Managers of product development are customers
to technologists and can agree or refuse to sponsor technology, which
creates a shared understanding of deliverables between the
technologist and the product manager.  Each technology project has a
concept readiness milestone, which includes a demonstration of the
hardware and its required manufacturing processes, as well as testing
and analysis results.  Also, each technology has an implementation
readiness milestone, which includes proof that the technology is
ready for product development with identified and manageable risks. 
At this point, the new technology's cost, schedule, and quality
targets must be met. 

Ford enforces a "Wall of Invention" at the start of each new
product's development.  Beyond this point, no new technologies or
design features are allowed to be added to the vehicle.  From then
on, the product manager takes responsibility for successfully
developing and producing the new vehicle.  Product managers are
veterans of previous production programs, and their production
knowledge makes them very discriminating in deciding what new
technology will be allowed onto the vehicle. 


      DOD PRACTICES
-------------------------------------------------------- Chapter 2:2.2

DOD programs did not attain a match between technology and
requirements at the time of launch.  DOD accepted varying--but
consistently higher--degrees of technological risk on four of the
product development programs we reviewed.  Technologies ranged from
comparatively high-risk advances for the F-22's avionics and low
observability to low-risk advances for mostly off-the-shelf
components being used to develop the AIM-9X missile and JDAM.  The
C-17 program was developed using mostly existing technology, with
some demanding features required of the airframe design.  DOD
practices for matching product requirements to available technology
on the AIM-9X and JDAM appeared to do a better job of reducing risks. 

The F-22 development program has reduced risk from new technologies
across most of its key design features through extensive,
high-fidelity modeling and analysis.  However, the program's product
development includes new low-observable materials, avionics, and
propulsion technology whose performance is undemonstrated and raises
some concern about producibility.  Low observability is one of the
F-22's most important characteristics and is dependent on many
factors, such as the shape of the airframe, its flight
characteristics, and airframe materials and coatings.  Although the
program has done pole model testing of the aircraft to determine
whether it meets its low-observability requirements, the aircraft
includes 10 newly developed derivatives of existing materials that
are important to its low-observability feature.  The materials will
have had 3 years of flight testing before they are applied to the
first production aircraft; however, the performance and
maintainability of these materials cannot be completely verified
until radar signatures can be tested in varying climates.  This
testing is scheduled to be completed 2 years after production begins. 

Similarly, much remains to be proven on the F-22's avionics
technology before it is certain that the technology is a match for
the performance requirements.  The avionics software features a level
of integration not previously achieved in a fighter.  One major
technical challenge is perfecting the way in which various avionics
subsystems will interface.  The program has incorporated a series of
risk reduction activities designed to mitigate these software
integration problems.  For example, it has defined and replicated all
of the avionics interfaces at this point in development, but this
testing will not prove that the various subsystems will interact in
identifying and prioritizing threats.  The program has begun testing
in an avionics integration laboratory and on a flying avionics test
bed mounted on a Boeing 757 and will further test the avionics
capabilities for prioritizing threats at very high densities and
conditions once the test facility is ready.  However, until that type
of testing and flight testing begin to produce results--in 1999 at
the earliest--there are significant unknowns concerning the
capability of integrated avionics on the F-22.  The program plans
three incremental updates to the software, completing its development
in 2002.  By this time, Lockheed plans to have at least
40 production F-22s, an investment of at least $8 billion, finished
or in the manufacturing process. 

The F-22 engines have many advanced features to meet aggressive
performance requirements.  One such requirement is for
supercruise--the ability to sustain supersonic speeds without the use
of afterburners.  Some of these advanced features resulted in
components that are more difficult to produce than originally
anticipated and are resulting in higher-than-planned recurring
production costs.  The producibility risk is largely driven by
unknowns concerning the hollow fan blades, some components fabricated
from composite materials, the turbine exhaust case, and a few
features of the exhaust nozzle.  A product cost reduction process has
been implemented to mitigate these risks, but program officials
acknowledged that the user's overall performance requirements for the
engines remain very challenging, and it is still unclear whether the
engines will meet all of the requirements. 

For the most part, the C-17 was developed using nondevelopmental
items or commercial parts.  However, performance requirements for
short landing and takeoff, payload, and range required design
innovations that significantly increased technical risk on the
program.  A new technology that added risk to the program was
aluminum lithium, a new, unproven alloy that the program used for its
strength and weight savings.  The first production aircraft contained
2,200 pounds of the alloy.  Its application was unsuccessful due to a
lack of knowledge about the technology's characteristics.  The
program discontinued its use because of its potential to warp during
handling, cracking and chipping during installation, the need to keep
it separate from other materials, and concerns over sources of
supply.  It is being phased out of the program and the 51st
production airplane will be free of it.  The C-17 program replaced
aluminum lithium with aluminum alloy 7050, a less expensive and
easier-to-handle material with several sources of supply.  Boeing
made the same decision at the outset of its 777-200 development
program based on similar conclusions reached by its design/build
team.  This is a good illustration of different knowledge standards
applied in making design decisions.  Figure 2.3 shows the C-17
aircraft. 

   Figure 2.3:  C-17 Cargo
   Aircraft

   (See figure in printed
   edition.)

   Although the C-17 was developed
   mostly with nondevelopmental
   items and commercial parts,
   performance requirements
   significantly increased
   technical risk.

   (See figure in printed
   edition.)

   Source:  DOD.

   (See figure in printed
   edition.)

Also, the software development effort to meet avionics performance
requirements turned out to be significantly more complex than the Air
Force thought.  When the C-17 development program began in 1985, for
example, the Air Force had identified 4 subsystems with about 164,000
lines of code that had to be developed.  By 1990, this number had
increased to 56 subsystems and about 1,356,000 lines of code,
including approximately 643,000 newly developed lines of code. 

The JDAM and AIM-9X development programs have been more attentive to
matching customer requirements with technological capabilities.  Both
programs chose designs that relied strongly on predecessor data, and
both programs emphasized affordability as a key product requirement. 
JDAM used modified variants of proven product lines for its guidance
component and global positioning system.  It also used mature,
existing components from other proven manufacturing processes for its
own system for controlling tail fin movements.  The designs for the
battery and the tail housing both use mature technology and will be
built using mostly existing tooling and processes.  The program
office and contractor have been able to further minimize risk by
conducting extensive development, qualification, and flight testing
to demonstrate performance, quality, and reliability on some key
subsystems. 

Except for its tracking technology, the AIM-9X air-to-air missile
will use technology largely from manufacturing processes already in
place or from the United Kingdom's Advanced Short-Range Air-to-Air
Missile production program.  For example, the sensor is currently
being developed for the United Kingdom missile, and over 2,500
sensors will have been produced before the first AIM-9X is assembled. 
The AIM-9X airframe is adapted from a past Air Force flight test
program and is made up of proven technology.  The rocket motor,
engine, warhead, and fuse are all taken directly from AIM-9M missiles
and are already in production.  The AIM-9X tracking system is the
primary area of new technology.  It is a sophisticated, computerized
component that translates the light image obtained by the seeker into
electronic signals that can be viewed by the pilot, discriminate
against countermeasures, and guide the missile to a particular point
on the target aircraft.  It provides the seeker a wider, higher
fidelity view in which to find targets.  This new technology has been
in development for some time and is made up of off-the-shelf
components.  Hughes believes this risk is manageable because of
modeling and simulation that preceded the development program. 


   KNOWLEDGE POINT 2:  THE DESIGN
   WILL PERFORM AS REQUIRED
---------------------------------------------------------- Chapter 2:3

The completion of engineering drawings and their release to
manufacturing organizations signify that program managers are
confident in their knowledge that the design performs acceptably and
can be considered mature.  The drawings are critical to documenting
this knowledge because they are not only precision schematics of the
entire product and all of its component parts--they also reflect the
results of testing and describe the materials and manufacturing
processes to be used to make each component.  Both DOD and commercial
companies consider the design to be essentially complete when about
90 percent of the engineering drawings are completed.  Both sectors
schedule a critical design review (CDR) to review the drawings,
confirm the design is mature, and "freeze" it to minimize changes in
the future.  Figure 2.4 compares what knowledge, in the form of
released drawings, was in hand at the time of CDR for the commercial
and DOD programs we reviewed. 

   Figure 2.4:  Comparison of When
   Commercial and DOD Programs
   Achieve Knowledge About Their
   Product's Design

   (See figure in printed
   edition.)


      COMMERCIAL PRACTICES
-------------------------------------------------------- Chapter 2:3.1

The commercial firms we visited released over 90 percent of the
products' engineering drawings at CDR.  At that point--about midway
through development--the firms had near certainty that their product
designs would meet requirements and had gone a long way to ensure
that the product could be produced. 

Boeing had full knowledge of the 777-200 design's capability at its
final CDR.  It began releasing engineering drawings after a
preliminary design review when product development began in 1990 and
completed this release process in 1992, less than 2 years later.  CDR
at Boeing is generally done by peer organizations, such as engineers
from another program.  Outside consultants are sometimes used. 
Boeing officials told us that, once the design drawings were released
to the manufacturing process, they averaged less than two changes per
drawing, well below their average on past programs and exceeding
their goal for the 777-200 program.  Once CDR was complete, Boeing
strictly enforced the design freeze for the 777-200.  For example,
Boeing incorporated a customer requirement to include folding
wingtips, along with the supporting bulkheads, into the 777 design at
a cost of nearly $40 million.  Later, the customer decided the
folding wingtips were not necessary; however, Boeing left the
bulkheads in the wings anyway because all of the engineering drawings
were completed and the risk of introducing changes, even though the
changes would have saved weight, was considered too high relative to
cost and schedule targets. 

Hughes, which designs both military and commercial satellites in the
same facility, completed two design reviews for development of the
HS-702.  First, it passed an internal review to satisfy the
corporation that its basic design was producible.  Second, it
reiterated design reviews on individual, customized satellites as
they were ordered by customers.  During the internal development
process, Hughes began releasing engineering drawings at preliminary
design review and completed releasing drawings at CDR, about 15
months into a 26-month product development process.  Hughes' goal was
to complete and release all engineering drawings at CDR; however,
representatives told us the drawings were close to 95 percent
released at that point. 

Chrysler described several practices for ensuring mature designs.  It
uses information from predecessor products to quickly develop focused
updates and innovative derivatives to existing products.  This
predecessor data, along with its use of platform teams, allows
Chrysler to take advantage of existing information and expertise in
new product developments.  It is able to simultaneously develop and
test the product's design and its required processes, which it
described as true concurrent engineering at the right time.  Chrysler
uses prototypes to refine the product and capture design knowledge
early.  It also employs a pilot production process and extensive
testing facilities at its technical center to prove the design's
performance and producibility before it is released to manufacturing
facilities. 


      DOD PRACTICES
-------------------------------------------------------- Chapter 2:3.2

The C-17 and the F-22 programs had less knowledge--in the form of
test results or engineering drawings--about their designs than
commercial companies did at the time they held their CDRs.  The
programs did not get or were not projected to get to the same level
of completion on the drawings until later in the development cycle,
which placed greater reliance on the lesser information available at
CDR.  This is important because the review is a major event that
represents a point of departure from detail design to manufacture of
the product.  The risks of proceeding with CDR and the rest of
development as planned are increased without the requisite drawings
in hand.  This was deemed acceptable for both programs. 

The C-17 program had released 10,229 of its engineering drawings--56
percent--at CDR in August 1988, just past the midway point in
development.  After CDR, an additional 8,161 drawings were released,
over 4,000 of which were released after assembly of the first
production aircraft had started.  We estimate that the C-17 program
did not release 95 percent of its initial drawings until December
1991, more than 3 years after CDR and after seven production aircraft
had been delivered to DOD.  The C-17 encountered numerous technical
problems during testing that resulted in redesigns, cost increases,
and schedule delays.  For example, flight tests in 1991 revealed that
an innovative system of blown wing flaps--essential to the C-17's
ability to fly steep landing approaches at slow speeds to access
shorter runways--suffered heat damage and acoustical cracks.  To
correct this problem, the contractor had to completely redesign the
slats to include titanium skin and substructure.  This knowledge was
not obtained until the program was at least two-thirds into
development.  Also, in 1992 both wings on the static test article
failed at approximately 128 percent during a 150-percent limit load
test and had to be redesigned. 

Software development and management problems also contributed to C-17
cost and schedule problems.  The Air Force lacked specific knowledge
about software development problems as they occurred.  The first C-17
aircraft achieved first flight 19 months later than planned and did
not include many of the mission-critical software functions required
for a fully operational aircraft.  Problems such as these caused
completion of testing on the C-17 to be delayed by well over 1 year. 

The F-22 program had released 3,070 initial structures and systems
drawings, less than one-third of the eventual drawings, at its CDR
held at the midway point in development.  Since then, an additional
6,032 drawings have been completed and released.  The drawing release
process is not yet complete for the F-22, so we cannot determine if
the program has yet released 95 percent of its drawings.  F-22
production is scheduled to begin in June 1999. 

Even though it is still too early to predict outcomes on the AIM-9X
missile and the JDAM development programs, their prospects appear
promising because they have chosen mostly proven technology from
existing programs to achieve performance requirements.  They appear
to have robust, mature designs that will use manufacturing processes
already identified and demonstrated to be stable or in control.  JDAM
held its CDR in August 1995.  At that time, it had released 65
percent of its drawings.  In addition, it demonstrated how the
product would meet requirements using data, analysis, and a physical
display of the product.  The time it took to build up and load a JDAM
was also demonstrated.  AIM-9X has not yet held its CDR, but program
officials indicated that they will have already built a prototype
before CDR and hope to freeze the design based on that prototype. 


   KNOWLEDGE POINT 3:  PRODUCTION
   UNITS WILL MEET COST, QUALITY,
   AND SCHEDULE OBJECTIVES
---------------------------------------------------------- Chapter 2:4

The companies we visited reached the point at which they knew that
manufacturing processes would produce a new product according to
cost, quality, and schedule targets before they began fabricating
production articles.  This meant more than knowing the product could
be manufactured; it meant that all key processes were under control,
so the quality, volume, and cost of their output were proven and
acceptable.  The C-17, in production for 7 years, still does not have
all of its processes under control.  The F-22 program is not
scheduled to have all of its processes under control until the 4th
year of production.  The AIM-9X and JDAM programs appear to be doing
better in this regard.  Both programs are using the same approach as
commercial firms to attempt to get their critical manufacturing
processes in control before the programs enter production.  Figure
2.5 illustrates the difference between the commercial HS-702 and
777-200 programs and the military C-17 and F-22 programs concerning
when full knowledge about the production processes was achieved. 

   Figure 2.5:  Comparison of When
   Commercial and DOD Programs
   Achieve Knowledge That
   Processes Can Produce An
   Acceptable Product

   (See figure in printed
   edition.)


      COMMERCIAL PRACTICES
-------------------------------------------------------- Chapter 2:4.1

The commercial firms relied on existing manufacturing processes and
statistical process control (SPC) data to achieve this knowledge in a
timely manner and, in fact, had all their key processes under
statistical control when production began.  SPC is established by
monitoring processes to see if they are consistently producing output
that is within the quality standards and tolerances set for the
overall product.  Statistics concerning the quality of each process
output are analyzed, and when the output is out of tolerance, process
owners search for causes.  Once a process is producing consistently
high-quality output, the process is considered to be in statistical
control, and inspections can be reduced.  The knowledge gained using
SPC is significant in transitioning from development to production
because it helps companies ensure that cost, schedule, quality, and
reliability targets will satisfy the customer and the business case. 

The ability to establish SPC for key processes before production
began was the culmination of all the practices employed to identify
and reduce risk.  The criteria commercial companies had established
throughout their product development processes forced program
managers to prove that the product's design was capable and
producible early in the process.  For example, Boeing carried out
more than 25 technical and manufacturing reviews during the
development process leading up to production to ensure that the
product's design was producible.  Boeing identified computer-aided
three-dimensional interactive application software (CATIA) and its
design/build teams as crucial to ensuring that processes were in
control and the transition to production went smoothly.\1 CATIA
software allowed workers to share design and process information
about the 777 in real time and different locations.  Manufacturing
engineers could check on a part's producibility at any time in the
development process. 

Hughes told us that it knew that all of the processes to manufacture
the HS-702 were in control because the processes were either already
in place from previous programs and under SPC or they had been proven
to be in control before being released to product development by
demonstrations during the corporation's overall technology
development process. 
Figure 2.6 shows the Hughes HS-702 satellite.  Ford had similar
practices to Hughes, often demonstrating SPC on technology before it
was inserted into a product development, even before production
begins. 

   Figure 2.6:  Hughes HS-702
   Satellite

   (See figure in printed
   edition.)

   Hughes has achieved statistical
   process control on the
   low-volume HS-702 satellite
   before beginning production.

   (See figure in printed
   edition.)

   Source:  Hughes Space and
   Communications.

   (See figure in printed
   edition.)

Chrysler's use of platform teams, CATIA software, and its technical
center all contribute to a smooth transition to production and ensure
that processes are in control for every new product.  Perhaps as
important, synergy develops with a disciplined product development
process that accumulates so much knowledge up to this point.  All of
the companies we visited agreed that knowledge about technology and
design up front in the process makes the control of processes
possible and the transition to production smooth. 


--------------------
\1 CATIA allows designers and manufacturers the opportunity to view
design drawings as three dimensional and provides real-time
information to everyone in the process about the impact a change to a
drawing will have on the overall design and manufacture of the
product. 


      DOD PRACTICES
-------------------------------------------------------- Chapter 2:4.2

The DOD programs we reviewed demanded less proof than commercial
firms of a design's producibility before a product transitions to
production.  The C-17 did not achieve SPC on all key processes before
production and will not reach that point until well into production. 
The F-22 is not slated to achieve control on all key processes until
after production begins.  Moreover, basic producibility problems were
not discovered until late in the F-22's development and after
production began on the C-17.  These risks went unrecognized even
though both programs had established criteria for ensuring that risks
were acceptable and enough knowledge had been gained before
proceeding to the next program phase.  These exit criteria were light
on production-related requirements, even at the decision point for
proceeding with production.  For example, the F-22 program's exit
criteria for moving to full production does not require information
from the final production readiness review until after the decision,
and the C-17 program's exit criteria did not require proof of
producibility.  Neither set of criteria contained requirements about
drawing releases nor the schedule for achieving control on key
processes. 

The C-17 program began production in 1989 with less than 13 percent
of its key manufacturing processes in control, despite completing
production readiness reviews that were intended to reduce
producibility problems.  The program has produced and delivered
nearly 40 aircraft and has now identified 420 key manufacturing
processes.  Of those, only 56 are under statistical control.  This
inability to control manufacturing processes was caused, in part, by
an immature design.  For example, the C-17 program discovered major
design changes of the wings, flaps, and slats were required after
CDR, as prototype aircraft were being built.  In addition to causing
high rates of drawing changes, these late design changes caused
problems on the production line.  They necessitated costly changes to
processes; forced the manufacturers to develop workaround plans that
led to labor inefficiencies; and resulted in high rates of scrap,
rework, and repair.  The C-17 program estimates scrap, rework, and
repair costs on the 34 delivered aircraft to be over $200 million. 

The F-22 program is ahead of the C-17 at this point regarding SPC,
perhaps due in part to its use of CATIA.  Of 926 key manufacturing
processes identified, almost 40 percent of its key processes are in
control now,
2 years before production is scheduled to begin.  Nonetheless, the
production preparations for the F-22 illustrate the limitation of a
review mechanism when a substantial amount of knowledge is
unattained.  The program's initial production readiness review, held
in 1995 when only about one-third of the engineering drawings were
released, did not report any high risks in manufacturing or
producibility, although clearly much was unknown at the time and
drawing release was slower than called for by DOD guidance.  In 1996,
an independent team mandated by the Air Force reviewed the program
and discovered numerous manufacturing and producibility problems,
such as underestimated complexity in manufacturing processes,
understated labor requirements for building the aircraft, immature
definition of avionics flight test requirements, and concerns about
software integration.  These and other concerns led to increases to
development and production cost estimates of $2.1 billion and $13
billion, respectively.  Air Force officials believe that these
additional costs can be offset by reprogramming existing funds and by
implementing a number of cost avoidance measures in the future. 

The JDAM program office completed manufacturing assessments at the
prime contractor and all major suppliers in February 1997 in such
areas as process controls and plant capacities.  These assessments
found that all of the production tooling was in place to build JDAM. 
The program office has identified 84 manufacturing processes that
must be in control to meet the product's key performance
characteristics and has achieved statistical control on 69 percent of
the processes at this point in the program, about
1 year before full-rate production begins.  The AIM-9X program is
still very early in its EMD phase but has already identified all of
the critical manufacturing processes that must be in control to meet
the product's key performance characteristics.  The contractor told
us that all of those processes are stable at this point in the
program, and most are in statistical control. 

Although the F-22 program may not be characterizing and controlling
its manufacturing processes as quickly as commercial firms do, it
offers perspective on some of the strides that DOD has made in
preparing weapons for production since the 1980s.  In 1985, we
reported that several weapons had encountered substantial problems in
early production because they used hand-built development prototypes
made in special shops by expert personnel.\2

Production articles were built in different facilities with different
processes and people.  There was no real transition to production,
but rather a sudden shift from development to production.  One of the
programs at that time, the F-16 fighter, had a much better
experience.  We noted that development prototypes were built in the
production facility using as many production processes and people as
possible.  This made the transition to production much more gradual
and was one of several factors that helped prepare the F-16 for
production.  Although the F-22 is attempting to make a bigger
technological leap than the F-16 did, the development prototypes are
being built in the production facility using some production
processes and people. 


--------------------
\2 Why Some Weapon Systems Encounter Production Problems While Others
Do Not:  Six Case Studies (GAO/NSIAD-85-34, May 24, 1985). 


DIFFERENCES IN MILITARY AND
COMMERCIAL PRACTICES REFLECT
DIFFERENT ENVIRONMENTS
============================================================ Chapter 3

The differences in the practices employed by the best commercial
firms and DOD are not necessarily explainable by differences in
tools, techniques, or talent.  Rather, the differences in the actual
practices reflect the different demands imposed on programs by the
circumstances or environment in which they were managed.  Indeed, the
way success and failure are defined for commercial and defense
product developments differs considerably, which creates a different
set of incentives and different behaviors from the people managing
the programs.  Specific practices take root and are sustained because
they help a program succeed in its environment.  In this sense,
practices are adopted because they work--not because they are
textbook solutions. 

In general, the success of a commercial product development is
determined on the basis of production items sold; similarly, failure
is clearly defined as the customer walking away and purchasing the
product of a competitor.  This definition of success, coupled with
shorter cycle times, makes production concerns a main determinant in
the initial business case to launch a commercial product development
and in subsequent decisions to manage risk and make technology
tradeoffs.  This environment encourages the early identification of
unknowns with realistic risk assessments so that risks are not
allowed to jeopardize success.  Strong incentives, both positive and
negative, stress getting the product development right. 

DOD product developments are launched much sooner--with greater
technology, cost, and schedule unknowns--and extend much longer. 
Production concerns do not play as big a role early in these programs
because production is so far off and thus is not as critical to
success.  The definition of success is more complicated in DOD. 
During most of product development, success is defined in large part
as getting DOD and the Congress to fund the development on an annual
basis.  Optimistic assessments of design performance and cost help
ensure this kind of success; realistic risk assessments for unknowns
do not.  By the time production nears, the risk of failure is much
reduced because a vested customer is not likely to walk away. 


   COMMERCIAL PRACTICES ARE DRIVEN
   BY THE CUSTOMER'S ACCEPTANCE OF
   THE FINISHED PRODUCT
---------------------------------------------------------- Chapter 3:1

The commercial firms we contacted launch a product development
program only when a solid business case can be made.  The business
case basically revolves around the ability to produce a product that
will sell well enough to make an acceptable return on investment. 
The point of sale occurs after product development is complete;
program success is determined in production when the customer buys
the finished product.  If the firm has not made a sound business
case, or has been unable to deliver on one or more of the business
case factors, it faces a very real prospect of failure.  Production
is a dominant concern throughout the product development process and
forces discipline and tradeoffs in the design process.  This
environment encourages realistic assessments of risks and costs;
doing otherwise would threaten the business case and invite failure. 
For the same reasons, the environment places a high value on
knowledge for making decisions.  Incentives favor identifying
unknowns early, designating them an appropriate high risk, and
aggressively eliminating them.  Practices, such as achieving
statistical process control before production, are adopted because
they help ensure success. 


      PRODUCTION CONSIDERATIONS
      FIGURE PROMINENTLY IN THE
      PROGRAM LAUNCH DECISION
-------------------------------------------------------- Chapter 3:1.1

For the commercial firms we contacted, the main focus of a product
development program is to produce and sell the right product at the
right time.  The success of a program depends on the product's sales
and not just on its successful development.  Therefore, the business
case for launching a program considers production realities and
builds in natural curbs to overreaching for performance, cost, or
schedule.  Although the corporations demand considerable proof in the
business plan that the product will meet market expectations, they
then provide full support--including funding--for the plan to
succeed.  This provides a program with a very solid baseline from the
outset that continues to serve as a good decisionmaking framework as
the program progresses and problems arise.  The day of
reckoning--when program success or failure is realized--comes clearly
and swiftly after development when the customer takes delivery of the
product and is prominent throughout the product development process. 

Boeing described the business case for the 777-200 product
development as a "money wheel" that must be balanced across all of
its factors.  These factors include a market opportunity, a product
whose technical features can satisfy the market, available investment
capital, a cycle time short enough to get the product to market on
time, and a unit production cost that will yield an acceptable return
on investment.  Boeing informed us that if any factor gets out of
line, either through estimating errors or changing conditions, the
"wheel" will not turn, and profitability--and perhaps corporate
reputation--could be lost.  The program manager is judged by these
standards, unlike in DOD. 

Before Boeing committed to development of the 777-200, it first
determined that a market opportunity existed for a family of at least
10,000 new, medium-lift airplanes and that the airlines would buy
these airplanes at a certain unit price.  The company then had to
make the business case--that it could develop a product with the
right features, in time to meet the market opportunity, and at an
acceptable unit cost.  Boeing did not commit to offer the new
aircraft for sale until the commercial group's president presented
solid evidence, on the basis of predecessor programs, testing,
simulations, and analysis, to the board of directors that all of the
business case factors were achievable.  The board then approved
investment of capital based on its assessment of the risk involved
with meeting the business case.  Boeing program officials pointed
out, however, that it was not the board meeting that provided the
incentive to get the business case right.  Rather, they said that the
commercial group president knew that the company's investment capital
was at stake on such a major product development.  An oversold
business case would lead to failure and could have a devastating
effect on the company's future.  This had a very sobering effect on
the people that constructed the factors in the business case and
curbed the tendency to overstate the case. 

Hughes presented a very similar scenario for the corporate decision
to go forward with the development of the HS-702 satellite.  Hughes
was very comfortable with the level of technical risk at the outset
because its technology development process had matured the required
technologies before inclusion in the program.  In fact, the biggest
risk in its business case was ensuring that the market for the
product was profitable.  To resolve this risk, Hughes spent
considerable time educating its customers about the benefits of the
proposed program.  Only after potential customers were convinced did
Hughes have a solid business case for going forward with development. 

The other companies we contacted--Chrysler, Cummins, and Ford--had
similar frameworks established for making decisions about product
development.  Business realities, such as capital investment, cycle
time, execution in production, and the customer's sensitivity to
purchase price, created a high-stakes environment.  The decision to
begin a new development program was typically made at the highest
corporate level.  However, the business case, with all of its
competing factors, provided a framework and the incentives for
program managers to make practical decisions about product
development based on facts and data.  This framework held the
achievement of each factor as essential to success and committed the
program managers to assess all associated risks realistically. 
Getting a program approved that embodied false or weak information
was viewed as a failure because the customer might not buy the
finished product.  One commercial program manager informed us that
his firm used to routinely underestimate the cost and schedule for
developing a new product and overestimate its sales volume and profit
margin.  He described this practice as a "death cycle" that nearly
bankrupted the company. 


      SHORTER CYCLE TIMES AND
      STRONG INCENTIVES FOSTER
      BEHAVIORS THAT KEEP PROGRAMS
      ON TRACK
-------------------------------------------------------- Chapter 3:1.2

Once a company decides to launch a product development, strong
incentives--both positive and negative--serve to keep the programs on
track.  To meet market demands, leading commercial companies build
relatively short cycle times into decisions to begin a product's
development.  Boeing's 777-200 went to production less than 5 years
after development began, Hughes' HS-702 took about 26 months, and
Chrysler developed its new sport utility vehicle in less than 30
months.  These short timeframes make the day of reckoning, in terms
of customer acceptance and return on investment, close at hand. 
Consequently, production--on time, at rate, at cost, and with
quality--looms as a near-term reality that continues to greatly
influence subsequent design and configuration decisions within the
framework of the business case.  The incentives that operate in the
commercial environment encourage program managers to want risks
identified early, be intolerant of unknowns, and not rely on testing
as the main vehicle for discovering the performance characteristics
of the product.  By protecting the business case as the key to
success, program managers are conservative in their estimates and
aggressive in risk reduction.  Ultimately, preserving the business
case strengthens the ability to say "no" to pressures to accept risks
or unknowns. 

Boeing believes a number of factors serve as incentives to keep a
program on track.  First, because the fate of the program (and to a
large extent the fate of the company) is in the hands of the
program's design/build teams and their leaders, the incentive is for
them to convince themselves that the business case is solid rather
than to "sell" program estimates to corporate management.  Second,
the company's investment capital is fixed.  Thus, development cost
increases on any one program that can be absorbed are limited
because, at some point, the overrun will hurt investment on other
programs.  Third, the program's ultimate performance becomes part of
a scorecard on the program manager's impact on corporate
profitability.  A failure that hurts the company financially will
remain associated with the program manager and other people involved. 
Fourth, if the product does not live up to its performance
characteristics (which include operation and support costs), the
company will lose money on product guarantees.  Finally, the program
management staff understand that more is at stake than the program at
hand; a serious miscue on one program can hurt the company's
reputation and damage other programs as well. 

Although these factors could be seen as negative incentives, Boeing's
environment also provides strong positive incentives for keeping a
program on track.  The board approval to launch a product development
is a commitment to completely develop and produce the item.  There is
no turning back unless something occurs that would cause a serious
decay in the business case.  The investment funding is approved up
front, the corporation stands firmly behind the product development,
and the program manager has control over the product's development
process.  When problems do occur, the corporate support does not
waver, and problem solving takes place within a well-understood
framework--anchored in the business case--for making tradeoffs.  For
example, Boeing realized early in the 777-200 program that investment
costs would be higher than anticipated, largely due to cultural
changes it was trying to accomplish with full use of CATIA and
design/build teams.  Boeing's approach to resolving the problem began
with the premise that the contractual commitment to deliver the
aircraft, with all of its features, on time, and at the same price,
would be met.  The problem was then limited to minimizing the
increase in investment cost to keep the business case intact. 

Boeing said that it is predisposed to identifying and solving
problems early and paying the up-front costs to do it rather than
accept the production cost consequences of the problems at the end. 
Its decisions on the use of aluminum lithium and the bulkhead for the
777-200 wing tip are demonstrative of behaviors that value minimizing
production risks after a program is underway.  Although the aluminum
lithium could have been designed into the aircraft and offered
performance improvements, it was rejected because of the unknowns it
presented for material handling on the production line and for
long-term performance under operational conditions.  Similarly,
Boeing could have taken the unneeded bulkhead out of the wing tip to
reduce weight, but that idea was rejected because the redesign would
have had a ripple effect on drawings and production preparations. 

Chrysler officials discussed a similar environment for managing the
product development process, emphasizing the need for discipline to
come from within the program itself.  Chrysler establishes a target
unit price, specific performance objectives, and cycle time as part
of the business case for each product's development.  These factors
are strongly enforced--within a fixed-price, firm schedule
environment--as a discipline to manage risk and facilitate product
tradeoffs.  The platform teams operate under a "zero-sum" basis,
meaning that if the price of one component is higher than expected,
then savings must be found elsewhere.  The platform teams are totally
empowered to make all decisions concerning their product as long as
periodic reviews by product assurance teams indicate that all of the
performance objectives are being met.  If the objectives are not
being met, the team receives increased scrutiny and outside
direction.  This practice creates a strong incentive for product
managers and platform teams to keep product risk to a minimum by
using known product and process technologies and discovering problems
as early as possible. 

Chrysler explained an important aspect of the program manager's role
is to say no to things, such as immature technologies, that may
disrupt the product's cost, schedule, or performance targets. 
Chrysler believes that, without the "bottom line" discipline from the
program manager, cost targets will not work.  Any facet of product
development that cannot be shown to be knowledge, rather than
projection, is considered an unknown and designated a high risk. 
Design and production issues are thus equalized; a suspension
component that can improve vehicle performance and reduce complexity
will be rejected if the proposing team cannot provide proof that this
component can be produced at the rate and cost called for by the
business case.  Figure 3.1 shows Chrysler's Plymouth Prowler. 

   Figure 3.1:  Plymouth Prowler

   (See figure in printed
   edition.)

   Shortening product development
   cycles and other incentives
   help Chrysler managers identify
   risks and say no when
   necessary. 

   (See figure in printed
   edition.)

   Source:  Chrysler Corportation.

   (See figure in printed
   edition.)

In this environment, every team member is considered to have high
risk until each can demonstrate otherwise.  Team members are
encouraged to volunteer information early on risks or problems they
are having because the other team members will step in to help,
particularly if they have savings that can offset price increases. 
The team members know that if one of them fails, they all fail
because the business case decays.  They also know that if they help
someone, they increase the willingness of others to help them.  By
the same token, the team member that withholds problems until late in
the development is ostracized for putting the entire team in a
position to fail. 

Ford officials believed that the product development environment at
the firm had gotten to the point at which people treated the
company's money as if it were their own.  In our discussions with
these officials, we discerned several factors that converged to keep
estimates intrinsically sound during the product development process. 
These factors were:  (1) the vast amount of predecessor data from
previous product developments that can be used as a reality check for
a new program's estimates; (2) the amount and quality of the
information being generated for the program at hand, including the
technique being used to generate data; and (3) the consequences of
weak or overstated data on the program's success.  Figure 3.2 shows
the Ford Motor Company's Lincoln Navigator. 

   Figure 3.2:  Lincoln Navigator

   (See figure in printed
   edition.)

   Ford reduced the amount of
   unknowns on the new Lincoln
   Navigator by limiting product
   requirements to proven
   technologies.

   (See figure in printed
   edition.)

   Source:  Ford Motor Company.

   (See figure in printed
   edition.)

The individual practices employed by commercial firms are consistent
with the incentives of their environment.  The high reliance on CATIA
shortens cycle time and reduces risk by generating knowledge that
previously had to await the building and testing of physical
articles.  Insistence on having near 100-percent release of drawings
at the CDR drives risks down as the production line is geared up. 
Achieving statistical process control before production begins helps
create certainty that production costs, schedule, and quality will
satisfy the customer.  In short, these practices are adopted because
they work by helping to ensure success in the commercial environment. 


   DOD PRACTICES REFLECT THE NEED
   TO SUCCEED IN FUNDING AND
   MANAGING THE DEVELOPMENT EFFORT
---------------------------------------------------------- Chapter 3:2

DOD must make a defendable case before launching a program.  The
program's merits generally relate to providing a needed capability
within the limits of affordability.  DOD typically defines and
launches a program years earlier in the process than a commercial
product development, and thus the case for the product is made when
much less is known about technology, cost, and schedule.  In a very
real sense, the point of sale begins much earlier on a DOD program
and continues throughout development as the customer (DOD and the
Congress) pays for the product on an annual installment basis from
program launch.  Success, then, for most of the product development
cycle, is measured in terms of ability to secure the next
installment.  Because this approval must be won every year, it
creates incentives to make the program's case look attractive. 

By the time production begins, the customer is deeply invested and
unlikely to walk away.  As a result, success, in terms of program
continuance, is substantially ensured before end items are produced. 
For these reasons, and because production is often many years from
the launch decision, it is difficult for production realities and
concerns to exert as much influence on a DOD product development. 
Instead, design features and performance are more dominant.  More
unknowns are accepted on a DOD program, and their attendant risks are
often understated.  This combination, which can be devastating to a
business case, can work in a weapon development if it helps the
program get launched and survive.  Practices, such as deferring
prove-out of key processes until production begins, are compatible
with this definition of success. 


      EARLY LAUNCHES AND LONG-TERM
      PROGRAMS PUT FOCUS ON
      TECHNOLOGY DEVELOPMENT
-------------------------------------------------------- Chapter 3:2.1

For a major product development to be launched, DOD's acquisition
guidance calls for a strong case to be made that includes a firm
need, superior product performance, affordable cost, and feasible
technology.  A need can stem from several sources, including an
increase in threat capabilities, the obsolescence of an existing
weapon system, or the potential for a new or expanded capability made
possible by technology advances.  Traditionally, needs for new
weapons have been generated by individual branches within each
service.  In recent years, DOD has been experimenting with other
vehicles for determining needs, such as advanced concept technology
demonstrators to evaluate the utility of mature advanced
technologies.\1 Once a need has been established, a product
development vying for launch faces intense competition for initial
funding. 

A key distinction between DOD and commercial product developments is
the timing of the launch.  The launch decision on a DOD product
development is often made years before that on a commercial product. 
In DOD, this decision is made when the program is approved to enter
the program definition/risk reduction phase.  The knowledge required
to make the business case to launch a commercial product development
is generally not available for a DOD program until well into the EMD
phase.  By way of analogy, the F-22 fighter program, 11 years after
launch and about midway through the EMD phase, may now be at the
point that it would be ready for launch in the commercial
environment.  Thus, when a DOD product development competes for
launch funding, it is generally dealing with far greater unknowns
than its commercial counterpart.  Figure 3.3 shows an F-22 fighter
plane. 

   Figure 3.3:  F-22 Fighter Plane

   (See figure in printed
   edition.)

   The F-22, launched earlier than
   commercial products, needed
   significant advances in
   propulsion, low-observables,
   and avionics technologies to
   meet performance requirements.

   (See figure in printed
   edition.)

   Source:  DOD.

   (See figure in printed
   edition.)

Even though less information about a new DOD product development is
available at the time of launch, the competition for funding requires
detailed projections to be made from what information does exist. 
Although DOD is attempting to ease the technical requirements of
programs, a new product development is encouraged to possess
performance features that distinguish it from other systems.  For
example, the F-22 will be faster than the F-117 and stealthier than
the F-15; the RAH-66 Comanche helicopter will be faster, stealthier,
and smaller than the AH-64 Apache.  Consequently, aspiring DOD
programs have incentives to include performance features and design
characteristics that rely on immature technologies. 

Untempered by knowledge to the contrary, the risks associated with
these technologies are deemed acceptable.  We have often reported on
the subsequent realization of such risks in the form of problems
later in the product development.  In a 1993 study of seven weapon
systems, Rand found that, despite policies to the contrary, the
technology underlying several programs was clearly not fully
developed and ultimately caused substantial difficulties.\2
Production realities, critical to matching technological capabilities
with customer requirements on commercial programs, are too far away
from the DOD launch decision to have the same curbing effect on
technology decisions.  Indeed, in support of the Defense
Manufacturing Council, a multi-discipline team from industry and
government pinpointed a major cause of acquisition problems in DOD as
the imbalance between product goals and the maturity of engineering
and manufacturing processes used to reach those goals. 

Other pressures on DOD programs at launch make tough demands for
knowledge that does not yet exist.  A product development deemed
worthy cannot be launched unless development and production funding
is available over the right time period.  The product's development
and production cost, as well as timing, must fall within available
funding.  Because DOD relies largely on forecasts of cost, schedule,
and performance that are comparatively soft at this stage, success in
funding competition encourages the cost and schedule estimate to be
squeezed into profiles of available funding.  Additional product
requirements, such as high reliability and maintainability, serve to
make the fit even tighter. 

Ultimately, the demands of successfully competing for launch funds
make for a much different business case on a DOD product development. 
Compared with commercial programs, the DOD environment launches
product developments that embody more technical unknowns and less
knowledge about the performance and production risks they entail. 
These unknowns place a much greater focus on demonstrating and
proving out technology during the remainder of product development
than we found on commercial programs.  Equally important, the DOD
program is launched with customer funding.  Even though the
competition for funding will continue throughout the program's
development, the point of sale begins with program launch.  When a
commercial product development is launched, the customer's
willingness to buy is still undemonstrated and thus remains a
significant risk and a powerful incentive. 


--------------------
\1 DOD defines advanced concept technology demonstrators as a means
of demonstrating the use of mature technology to address urgent
military needs.  The demonstrators are not acquisition programs, but
are designed to provide a residual, usable capability upon
completion.  If the user determines that additional units are
desired, the additional buys would constitute an acquisition program. 

\2 Barriers to Managing Risk in Large Scale Weapons System
Development Programs, Rand, 1993. 


      INCENTIVES ON DOD PROGRAMS
      ENCOURAGE DIFFERENT
      BEHAVIORS FROM MANAGERS
-------------------------------------------------------- Chapter 3:2.2

As a product development proceeds in DOD, its success is still
measured in terms of the funding it receives.  Success translates
into getting the funding requested each year; failure can mean
anything from a significant funding cut to cancellation.  In a sense,
the day of reckoning for a DOD product development comes every year
with the budget process.  The most important issues facing the
program at the time will dominate the attention of program managers
and decisionmakers.  These issues tend to be related to the
successful demonstration of technology.  In contrast, the day of
reckoning for commercial programs is later in the program cycle:  at
the sale of the produced item.  The business case with the production
conclusion sustains the commercial product development's focus
through future decisions and problem solving.  Again, shorter cycle
times are key to sustaining this focus. 

Unlike commercial programs, DOD programs do not receive full
corporate support throughout development.  Even though many
individual programs compete for funding at any given time,
competition also exists at the aggregate level, among the services
for their portion of the budget.  In addition, DOD programs face
scrutiny by service executives, the Office of the Secretary of
Defense, independent cost estimating and test agencies, audit
agencies, and several committees and subcommittees of the Congress. 
Given this amount of competition and oversight, the detection of a
problem on an individual program makes that program vulnerable to
criticism and possible loss of funding support.  Ironically, it is
these same pressures that encourage overreaching at the time of
program launch.  This situation contrasts with that of the commercial
product development, which enjoys full corporate support once it is
launched; in turn, the program's estimates are kept realistic by the
knowledge that the program's success may determine the firm's future. 

The pressures and incentives in the DOD environment explain why the
behaviors of managers and other sponsors of product developments
differ from those in commercial programs.  According to a 1994 study
done for the Under Secretary of Defense for Acquisition, government
program managers found their formal role of objective program
management at odds with their informal role of program advocates.\3
According to the study: 

     "A feeling of responsibility for program advocacy appears to be
     the primary factor causing government managers to search
     aggressively and optimistically for good news relating to their
     programs, and to avoid bad news, even when it means discrediting
     conventional management tools that forecast significant negative
     deviations from plan."

In this environment, risks in the form of ambitious technology
advancements and tight cost and schedule estimates are accepted as
necessary for a successful launch.  Problems or indications that the
estimates are decaying do not help sustain the program in subsequent
years, and thus their admission is implicitly discouraged.  An
optimistic production cost estimate makes it easier to launch a
product development and sustain annual approval; admission that costs
are likely to be higher could invite failure.  There are few rewards
for discovering and recognizing potential problems early in the DOD
product development.  For commercial product developments, an
optimistic production cost estimate will mean failure of sales or
profit; admission of cost increases early invites aggressive
problem-solving behaviors to restore the business case.  The behavior
of tolerating unknowns and not designating them the same risk value
as in the commercial environment is rational in the DOD environment
because there is little incentive to admit to high risks before it is
absolutely necessary, as long as the resulting estimates are accepted
by DOD and the Congress.  In fact, admitting risk may doom the
program. 

Behaviors toward testing follow a similar logic.  On commercial
product developments, much more is known about the product's
performance at the beginning of development.  Testing is used to
confirm knowledge and identify weaknesses or limits in the product. 
It is consistent with a firm's anxiety to eliminate unknowns to
preclude failure in production.  DOD product developments are much
more dependent on testing to discover technical performance
characteristics and answer the question of whether the product will
work.  DOD tests serve more than the purpose of discovering or
confirming performance characteristics--they are examinations on
which the program must get good grades or face failure in the form of
withdrawal of support.  Good test results can help a program, whereas
negative test results are equated with failure.  Unknowns, then,
present a safer course of action; if testing does not occur until
late in the product development, forecasts of product performance
will serve as the only information available. 

Cycle time is an important determinant of what pressures the
environment brings to bear on a product development.  In a DOD
product development, cycle times can be 10 to 15 years, whereas the
tenure of a program manager is more likely to be 2 to 4 years.  One
senior commercial program manager informed us that it is unreasonable
to expect that a program manager can truly focus on anything that is
more than 3 years off; events beyond that timeframe are not powerful
motivators.  Thus, it is hard for production success to present the
same reality for the first few managers of a DOD program and the
decisions they are involved in.  To some extent, DOD may attempt to
counteract the disparity between the length of a program manager's
tenure and that of the program being managed through formal reviews. 
Production readiness reviews are an example; they do not have a
commercial counterpart because readiness for production is intrinsic
to the commercial product development.  The production readiness
review does not provide the same incentives for success because it is
also prey to the pressure to accept unknowns and assess them as
tolerable risks.  DOD's practices, which allow key production
knowledge to be gained concurrent with production, are a logical
consequence of the DOD environment.  If problems arise in production,
there is not nearly the risk of failure that a commercial product
faces; by the time a weapon enters production, the point of sale to
the customer has already occurred. 

We did not attempt to make a formal comparison between the
capabilities of DOD and commercial program managers.  We believe that
commercial program managers have one advantage in that they are
likely to have more experience with repeated product developments
than a DOD program manager.  Aside from that advantage, we did not
observe that commercial managers were somehow better or more ethical
than their DOD counterparts.  As we previously reported,\4 DOD
program managers and other sponsors do not act irrationally or with
bad intentions.  Rather, they see the acquisition of the weapons
under their purview as aligned with national interests.  They do what
they believe is right, given the pressures they face.  The difference
is that the definition of program success determines what is right,
and success in the DOD environment is different from success in the
commercial environment. 


--------------------
\3 Critical Issues in the Defense Acquisition Culture, Defense
Systems Management College--Executive Institute, December 1994. 

\4 Weapons Acquisition:  A Rare Opportunity for Lasting Change
(GAO/NSIAD-93-15, Dec.  1992). 


CREATING THE RIGHT ENVIRONMENT IS
KEY TO THE SUCCESS OF INITIATIVES
TO IMPROVE WEAPON ACQUISITIONS
============================================================ Chapter 4

DOD has embarked on several initiatives that draw lessons from
commercial practices, such as cost as an independent variable (CAIV),
integrated product teams (IPT), use of past performance data, and
performance specifications.  DOD has also recently set up a funding
reserve to offset unexpected cost growth to mitigate the effect of
unknowns on acquisition programs.  These initiatives could have a
positive effect on the transition of weapons into production if the
environment for launching programs and appraising risks can be
changed to provide the right incentives.  Studies sponsored by DOD
and the defense industry call for changes that could help shape such
an environment.  A 1996 study by the Defense Science Board Task Force
on Defense Acquisition Reform calls for replacing the DOD acquisition
process with a process that combines the best elements of practices
from commercial and defense-unique activities.  A major tenet of the
recommended process is more emphasis on incremental technology
advancement, coupled with much shorter product development cycle
times.  These themes are also echoed in a study by a National Center
for Advanced Technologies task force comprised of defense firms. 

Examples from past acquisitions serve as reminders that changing the
mechanics of a weapon's development, without changing aspects of its
environment that determine its incentives, may not produce desired
results.  DOD's guidance for preparing weapons for a successful
transition to production, some of which is 10 years old, already has
much in common with commercial best practices.  Thus, the challenge
for DOD and congressional decisionmakers may not lie so much in the
"how to" aspects of product development as in creating the
incentives--or the reasons why best practices will work for program
managers. 


   DOD'S RISK REDUCTION POLICY IS
   CONSISTENT WITH COMMERCIAL
   PRACTICES
---------------------------------------------------------- Chapter 4:1

DOD 4245.7-M, Transition From Development to Production, a risk
reduction policy manual written in 1985, provides metrics and tools
for minimizing design and production risk during a product's
development.  In addition, DOD's new procedures for major defense
acquisition programs contained in the new 5000 series of acquisition
policies emphasize best practices in assessing and mitigating risk
during development.  Figure 4.1 shows DOD's main policy guidance on
preparing weapons for production. 

   Figure 4.1:  DOD Guidance on
   Preparing Weapons for
   Production

   (See figure in printed
   edition.)

   DOD has provided ample guidance
   to program managers on how to
   properly manage weapon system
   development to successful
   production. 

   (See figure in printed
   edition.)

DOD 4245.7-M contains templates that provide guidance for when and
how to perform trade studies, design reviews, and producibility
reviews.  It contains criteria for trade studies that require an
evaluation of new technology before selecting it for a product's
development and suggests that cost, producibility, and quality should
be considered equally with performance.  The templates also suggest
that a CDR should be held when 95 percent of the product's design is
complete in terms of engineering plans and drawings and that the
design should be frozen and released directly after CDR.  The
templates also state that CDR results and critical drawing release
schedules and approvals are essential data sources for design
release.  Finally, the templates suggest that (1) the effect of the
design on current manufacturing processes should be measured during
the design process, (2) any new manufacturing processes should be
"proofed" during development to ensure producibility, (3) proofing
simulates actual production conditions and environments, and (4) SPC
data are considered essential program data sources for qualifying the
manufacturing processes during product development. 

DOD revised its 5000 series procurement policies in March 1996 with
the intent of defining an acquisition environment that makes DOD a
smart and responsive buyer of the best goods and services.  Several
themes described in the new 5000 series documents resonate with
commercial practices: 

  -- The commercial marketplace must be researched to determine its
     potential to meet system performance requirements and results
     documented in the initial operational requirements document. 

  -- The acquisition process must consider both performance
     requirements and fiscal constraints as embodied in CAIV. 

  -- Acquisition of commercial processes and practices provides rapid
     and affordable application of new technologies to meet DOD
     mission needs. 

  -- Future acquisitions must take into account customary commercial
     best practices in developing acquisition strategies and
     contracting arrangements. 

  -- A streamlined management structure and event-driven management
     process at DOD would emphasize risk management and affordability
     and would explicitly link milestone decisions to demonstrated
     accomplishments. 

DOD's 5000 series documents emphasize the interrelationship between
establishing requirements, managing the development process, and
making funding decisions with the objective of translating users'
needs into products with affordability as a key discriminator.  The
policy also emphasizes the consideration of producibility early in a
product's development.  It states that producibility is key to
managing risk and that existing manufacturing processes should be
capitalized on when possible.  It also states that production should
not be approved until the design has been stabilized, manufacturing
processes have been proven, and facilities and equipment are in
place.  DOD has emphasized that programs should meet users' needs
with mature technology, when possible, by using demonstrations of
technology and incremental or evolutionary product developments,
thereby taking advantage of existing information about cost and
performance. 


   DOD INITIATIVES ADAPT SOME
   COMMERCIAL PRACTICES
---------------------------------------------------------- Chapter 4:2

DOD's more recent initiatives are intended to reduce the cost and
duration of major weapon system acquisitions.  In establishing the
framework in which cost is considered an independent variable, DOD is
attempting to make cost a prime reason for trading off performance
requirements.  Once cost and performance tradeoffs have been made,
CAIV establishes cost as the overriding constraint for obtaining the
needed military capability from the new system.  For CAIV to work
properly, program managers will need sufficient knowledge of
technological capability and associated program development and
production costs, along with a sound mechanism and forum for making
tradeoffs.  Whether tradeoffs are made will depend on the incentives
in place to make them.  DOD points to the JDAM as an example of how
effective CAIV can be because use of this tradeoff technique reduced
expected unit production prices from $40,000 to about $14,000 per
unit.  Figure 4.2 shows the JDAM. 

As the program proceeds through development, the framework for making
decisions is to be similar to the zero sum approach an auto
manufacturer follows after establishing a price target for a new
vehicle.  Today, however, the career incentives for program managers
in DOD are not consistent with those of the auto manufacturers. 

   Figure 4.2:  Joint Direct
   Attack Munitions System

   (See figure in printed
   edition.)

   Use of cost as an independent
   variable and integrated product
   teams enabled the projected
   unit price of the JDAM to be
   reduced from $40,000 to
   $14,000.

   (See figure in printed
   edition.)

IPTs are a means to integrate the development of a product and its
manufacturing processes by using multidisciplinary teams that
represent a variety of product functions, such as engineering,
manufacturing, purchasing, and accounting.  Traditionally, these
functions have been separate organizations that tended to get
involved sequentially in the product development process.  Through
IPTs, the functions are brought together to make tradeoffs that will
optimize the design, manufacturing, and supportability processes for
a weapon system.  They are important to making techniques such as
CAIV work.  IPTs have analogies in Boeing's design/build teams and
Chrysler's platform teams.  Many defense contractors have already
adopted the IPT approach.  Several factors affect the success of
IPTs, including reasonableness of the program goals, the knowledge
the members bring to the team, the authority the teams have to make
tradeoffs to achieve the goals, and the right disciplines represented
on the teams. 

DOD has an initiative to use information on the past performance of a
contractor to ensure that DOD has access to a globally competitive
industrial base capable of supplying the best value goods and
services.  DOD believes that past contractor performance information
is an important strategic tool that when properly used, will allow
DOD to evaluate the risk of using poor or nonperforming contractors
as well as the potential for using excellent contractors.  This
information will allow DOD to make better decisions during source
selection.  In the commercial sector, the use of past performance
information for suppliers was generally found to be integral to
improving product developments. 

DOD has also made the decision to use performance specifications on
weapon system developments.  Previously, many military specifications
told defense contractors specifically how they were to accomplish
certain tasks.  Performance specifications tell the contractor only
what performance is desired of the end product and not how to get
that performance.  Again, there is a commercial analogy in the sense
that customers are generally not concerned with how a product does
something, such as get better fuel economy, as long as performance is
as advertised. 

DOD is tracking the effects of these acquisition reform initiatives
on seven programs, designated as Defense Acquisition Pilot Programs. 
These programs were afforded early statutory and regulatory relief
under the provisions of the Federal Acquisition Streamlining Act of
1994 to set the example for acquisition reform.  JDAM is one of the
pilot programs.  According to DOD, the successful application of
commercial practices enabled these programs to demonstrate
significant improvements.  DOD projects that acquisition programs
that have benefitted from acquisition reform could reduce cycle time
by 25 percent.  Some of DOD's pilot programs have significantly
reduced the amount of military specifications and standards.  JDAM
has eliminated all 87 of the military specifications and standards
that were in its baseline contract in favor of commercial practices. 
A 1997 DOD report on the pilot programs credits the use of IPTs for
reducing program office size and significantly decreasing government
contract administrative hours.  The same report states that the
improvements indicated by the pilot programs appear to be applicable
across DOD. 

DOD has recently set up a reserve of funds to be used to mitigate the
effect of unforeseen technical problems that might threaten to upset
an acquisition program's schedule.  This initiative emerged from
DOD's Quadrennial Defense Review, which stated that

     "complex, technologically advanced programs all bear some risk
     of costing more than planned.  When unforeseeable growth in
     costs occurs, offsets from other programs must be found, which
     in turn disrupts the overall modernization program.  Our
     programming process must provide sufficient flexibility in the
     form of program reserves to address this risk."

DOD plans to begin accumulating the risk reduction fund in fiscal
year 2000 and expects it to grow to about $1 billion by fiscal year
2003.  It will be accumulated through contributions from the Office
of the Secretary of Defense and the services. 

We have not evaluated this new funding mechanism or the way DOD plans
to implement it.  Nonetheless, DOD's use of the reserve has the
potential for communicating to program managers which practices will
be encouraged and which ones will not.  For example, if the fund is
used primarily to pay for problems that are revealed in late
development or early production, the fund could reinforce existing
incentives for not dealing with risks until they become problems. 
Conversely, if the fund is used to resolve risks early and preclude
problems, it could encourage risks to be revealed earlier in
programs. 


   EXPERIENCE ON PAST PROGRAMS
   UNDERSCORES THE NEED TO MATCH
   TECHNIQUE WITH THE ENVIRONMENT
---------------------------------------------------------- Chapter 4:3

A discussion of lessons learned or difficulties encountered on past
weapon acquisitions must recognize that the DOD acquisition process
has produced the best weaponry in the world.  Nonetheless, the
experiences of some weapon systems serve to point out the importance
of making the environmental changes necessary for good practices to
work.  These techniques then will have the chance to offer better
outcomes consistently in terms of cost, schedule, and performance. 

In the 1980s, the Army attempted to acquire the Sergeant York Air
Defense Gun using a strategy that included several features similar
to those in the Defense Science Board's recommended process.  These
features included a competition between two contractors to use
components based on previously demonstrated technology and integrate
them into a weapon system, coupled with the freedom to trade off
flexible performance requirements to save costs and shorten cycle
time.  The resulting system failed in operational tests after it had
entered the 4th year of production and was subsequently canceled. 
The risks associated with integrating the technologies had been
underestimated, and the discovery process was protracted into
production.  The techniques to foster tradeoffs, reduce risks, and
give the competing contractors the freedom to design the system did
not, by themselves, change the Sergeant York's environment into one
that encouraged the early discovery of problems. 

Fluctuations in funding cause perturbations in weapon programs and
are a complication that poses less of a problem for the commercial
firms we contacted.  However, stable funding, by itself, will not
ensure program success.  In the 1980s, baselining a program was a
technique aimed at enhancing program stability, whereby a program
office "contracted" with top management to develop a system that met
basic performance, cost, and schedule requirements in exchange for
stable funding and minimal interference.  The 1986 Packard Commission
cited the B-1B bomber as an example of a baselined system.  Despite
enjoying stable funding and congressional support, the B-1B
experienced significant performance shortfalls and continues to
require substantial additional funding to correct these and other
problems.  Similarly, the F-117A program experienced cost increases,
schedule delays, and substantial modifications despite stable
funding, continuous congressional support, and streamlined
management.  In view of the many pressures that characterized the
acquisition environment for these programs, stable funding and
support did not prevent other problems, such as those stemming from
ambitious performance requirements and concurrent schedules. 

When a stated program goal is to trade performance for cost and
operational support reductions, the acquisition process must
encourage those tradeoffs.  This was not the case for the Army's
Comanche helicopter.  This program was initiated in the early 1980s
by senior Army management as a family of lightweight, multipurpose
helicopters whose justification centered on practicality rather than
the threat.  The program was expected to replace a fleet of
Vietnam-era helicopters with new helicopters that would be up to 50
percent less expensive to operate and support.  Within these
economical confines, the new helicopters were to offer as good a
technical performance as was possible.  However, specific
requirements were subsequently developed through a
performance-oriented requirements process.  The program emerged as a
threat-based program to yield the next-generation, high-performance
helicopter at a cost significantly higher than that of the Apache,
the Army's most advanced and costly helicopter. 

The lesson to be drawn from these experiences is not that a weapon's
transition to production cannot be improved through the use of better
practices.  Rather, these experiences show that attempts to improve
individual acquisitions require more than a new technique or
approach; they also require the knowledge needed to make key
tradeoffs and risk assessments, as well as the recognition by
decisionmakers that such information provided early is critical to
program success.  We believe that the JDAM and AIM-9X have promise in
this regard because they have utilized mostly existing technology in
setting requirements and, when they did not use existing technology,
have reduced risk through other means.  JDAM represents a new
munition built with mostly commercial parts, and the AIM-9X is an
upgrade to an existing missile system that largely relies on its
predecessor's technology.  Moreover, the launch decisions for the
JDAM and the AIM-9X missile have encouraged performance tradeoffs to
save time and money.  Figure 4.3 shows the AIM-9X missile. 

   Figure 4.3:  AIM-9X Missile

   (See figure in printed
   edition.)

   Tradeoffs early in the program
   may enable requirements for the
   AIM-9X to be met with mature
   technologies.

   (See figure in printed
   edition.)

   Source:  DOD.

   (See figure in printed
   edition.)

The challenge for these programs will be to maintain their integrity
during subsequent decisions as knowledge--and problems--are
discovered.  It also remains to be seen to what extent large
programs, such as new tank or aircraft development programs, can
apply these techniques in the perhaps higher pressure environment in
which they will be managed.  DOD's 1997 report on the Defense
Acquisition Pilot Programs notes that, to fully achieve the benefits
of acquisition reform, a complete cultural change must be affected. 
The report notes that despite significant top-level management
emphasis on changing the acquisition process, cultural resistance to
various reform initiatives has been encountered on both industry and
government teams. 


   RECENT STUDIES PROMOTE SHORTER
   CYCLE TIMES AND OTHER
   ENVIRONMENTAL CHANGES
---------------------------------------------------------- Chapter 4:4

In May 1996, the Defense Science Board completed a study that
concluded "force modernization at low cost and with short cycles--and
with ability to draw on world-class commercial firms--requires a new
weapons R&D [research and development] process." Some of the features
of the process recommended by the study directly address key
environmental differences we noted between commercial and DOD product
developments.  Specifically, the study recommended that DOD's process

  -- aggressively pursue high-risk technology before inclusion in a
     weapon research and development program;

  -- employ product solutions chosen jointly by the Under Secretary
     of Defense for Acquisition and Technology and the Vice Chairman
     of the Joint Chiefs of Staff;

  -- maintain competing alternatives for solutions throughout product
     design;

  -- make use of already developed next-generation technology, and
     concentrate on evaluations of existing subsystems as the
     building blocks for the concepts selected to meet the need;

  -- use flexible performance requirements with fixed-price and firm
     schedule requirements; and

  -- rely on competitive forces and price-based contracting versus
     regulations and cost-based contracting. 

In the opinion of the Defense Science Board, adopting such an
acquisition process would supply effective hardware in small
quantities, producible and supportable at affordable cost, with cycle
times reduced by one-half. 

In April 1996, the National Center for Advanced Technologies also
proposed a change in the DOD weapons acquisition process to reduce
cycle time by drawing on lessons learned from successful military and
commercial programs.  The center notes that concepts such as
integrated product/process development and CAIV are constructive but
that previous good ideas did not succeed.  Its proposal for change
calls for a new culture that relies on an affordable, incremental
approach that could reduce product development cycle times by 3 to 5
years.  The new culture features

  -- an incremental approach to performance, with a threshold or
     minimum performance for the initial battle group with
     incremental upgrades and

  -- requirements that would be managed through cost tradeoffs to
     keep performance and cost in balance, avoid grand designs, and
     mitigate risk. 

The December 1994 study by the Defense Systems Management College,
Critical Issues in the Defense Acquisition Culture, made several
recommendations to create an environment for weapon systems
development that encourages realism in reporting program status
information.  Perhaps the most significant recommendation is that the
individual military services transfer control of their acquisition
organizations and people to the Under Secretary of Defense for
Acquisition and Technology.  The study noted that, by doing so, the
Under Secretary would then be empowered to reward candor and realism
in reporting through the use of assignments, transfers, and
promotions. 


CONCLUSIONS AND RECOMMENDATIONS
============================================================ Chapter 5


   CONCLUSIONS
---------------------------------------------------------- Chapter 5:1

Product development in commercial ventures is a clearly defined
undertaking for which firms insist on high levels of knowledge before
starting.  Once underway, these firms demand--and get--specific
knowledge about a new product before manufacturing begins.  The
process of discovery--the accumulation of knowledge and the
elimination of risks or unknowns--is completed for the best
commercial programs well before production units are made.  In our
analysis, we characterized this knowledge in terms of three points or
junctures:  the match between requirements and technology, the
ability of the design to perform as expected, and the ability to
produce the product on time and at the right price.  Not having this
knowledge when demanded constitutes risks that the firms find
unacceptable.  Immature or undeveloped technology cannot meet these
demands.  It is essentially kept out of commercial product
development programs because of the risks it poses to a product's
price and schedule.  Increases in price and schedule devalue a
commercial product's worth.  Immature technology is managed
separately until it can meet the demands for product development. 

DOD does not insist on the same level of knowledge when it decides to
begin a weapon development program.  In the DOD acquisition process,
a clear delineation is not made between technology development and
product development.  In fact, programs are launched in the
technology development phase, and the pursuit of new technology--the
discovery process--continues through EMD and even into production. 
Not having the same level of knowledge as commercial firms explains
much of the turbulence in DOD program outcomes as the transition to
production is made.  It is a predictable consequence that can be
forecast early by the use of knowledge points or other metrics.  DOD
has had guidance in place that provides policies and metrics that
have much in common with what commercial firms demand in terms of
knowledge.  However, the way the tools for implementing such guidance
are used, such as CDRs and production readiness reviews, understates
the risks present.  Ironically, commercial firms that have weeded out
unknowns and minimized the discovery process still identified some
risks as high that needed to be resolved in their product
developments.  DOD programs, which accepted more unknowns and
technical advances, did not assess risks as high. 

Environmental differences underlie the differences in the product
development practices we observed between the DOD programs and
commercial firms.  At the core of these differences are the
definition of success and the time span of programs.  In commercial
programs, success is defined by the sale of production units to the
customer; product development activities exist only for production. 
Until the sale is made, the investment money of the product developer
is at risk.  Product development cycle times are short enough that
production realities are a major factor in assessing and eliminating
risk in the commercial sector.  Behaviors, such as candor about risk
and an intolerance for unknowns, and practices, such as ensuring
design performance and demonstration of all critical production
processes by CDR, are encouraged because they facilitate success. 

In DOD, programs are launched with immature technology, product
development cycle times are much longer, and production realities
figure less prominently in early tradeoff decisions.  Success is
defined more as the ability to secure annual funding installments
from the customer through the budgetary process to sustain the
development effort.  Thus, the point of sale essentially occurs when
the program is launched.  By the time production begins, the sale of
the product has, to a large extent, already occurred.  DOD's
acceptance of unknowns and practices that understate risk are
consistent with program success as defined in its environment. 

Commercial practices for gaining knowledge and assessing risks can
help produce better outcomes on DOD acquisitions.  For such practices
to work, however, the knowledge they produce must help a DOD program
succeed in its environment.  Thus, the DOD environment must become
conducive to such practices.  At least two factors are critical to
fostering such an environment.  First, program launch decisions must
be relieved of the need to overpromise on performance and resource
estimates.  The pressure to amass broad support at launch creates
incentives for new programs to embrace far more technology
development than commercial programs.  The objectives of technology
development, as well as what is demanded of knowledge and estimates,
differ from those of product development.  Clearly, DOD has to
develop technology, particularly the technology that is unique to
military applications.  However, by separating technology development
from weapon programs, DOD could insist on higher standards for
knowledge on its programs and get better results when those programs
transition to production. 

Second, once a program is underway, it must become acceptable to
identify unknowns as high risks so that they can be aggressively
worked on earlier in development.  Commercial firms insist on
knowledge measured against a criterion for assessing risk.  Fixed
prices and firm schedules help form the basis for such assessments. 
Firms then make decisions to preserve the business case by
eliminating risks.  The result is discipline provided from within. 
In contrast, the identification of high risk is more likely to invite
damage to the funding case on DOD programs.  Unknowns are more likely
to be accepted, and knowledge is more likely to be traded to preserve
resource estimates.  Releasing drawings behind schedule indicated the
attainment of less knowledge about design and manufacturing on the
F-22 program, but production readiness reviews did not identify any
areas of high risk.  It took an external team--discipline from
outside--to assess production risks as having significant cost
consequences. 

Decisions made by the Office of the Secretary of Defense and the
services to advance programs with significant unknowns or advance
programs with known high risks but without sufficient resources, send
signals to acquisition managers that current practices work and are
acceptable.  These decisions define what success means in DOD and
what practices contribute to success.  The newly created risk reserve
could become one vehicle for signaling what behaviors and practices
are encouraged on individual programs. 


   RECOMMENDATIONS
---------------------------------------------------------- Chapter 5:2

To close the gap between policy and practice, we recommend that the
Secretary of Defense take steps to ensure that sound standards for
the timing and quality of production-related knowledge are applied to
individual weapon system programs and used as a basis for assessing
production risks and for making tradeoffs.  These standards, which
can already be found to some extent in existing DOD guidance, should
draw from commercial practices and could include the release of
engineering drawings, identification of key production processes,
demonstration of risky or new production processes, and achievement
of statistical process control.  Such standards could enable
manufacturing representatives on IPTs to flag something as high risk
if it would delay drawing release and the achievement of statistical
process control beyond the standard of acceptability.  Identifying
the impact of such deficits in production knowledge could help
program managers to say "no" to proposals to accept unknowns and
could force tradeoffs in the design. 

To make the environment for DOD product developments more conducive
to the techniques used by commercial firms, we also recommend the
Secretary of Defense redefine the point for launching programs as the
point at which technology development ends and product development
begins.  This recommendation is made without prejudice toward the
necessity of technology development but rather with the intent that
programs could be better managed if such development was conducted
outside their purview.  Thus, concomitant with defining the launch
point later in the acquisition cycle must be the willingness of
decisionmakers in DOD and the Congress to support research and
development efforts that will advance technology and establish the
basis for determining which technologies and subsystems have the
mettle to meet the performance, production, and precision estimating
demands of product development.  If extenuating circumstances
necessitate including technology development in a program, this
should be recognized as a departure from sound practices, accorded a
high risk, and funded and paced accordingly. 

Finally, we recommend that on individual program decisions, the
Secretary of Defense or his designee send the signals that create
incentives for acquisition managers to identify unknowns and
ameliorate their risks early in development.  Incentives could take
the form of a decision to fully fund one program's efforts to
mitigate a high risk identified early or requiring another program in
which risks are revealed late to absorb the associated financial
consequences. 


   MATTERS FOR CONGRESSIONAL
   CONSIDERATION
---------------------------------------------------------- Chapter 5:3

Because of its critical role in creating the environment for what
constitutes program success and which practices will work, the
Congress may wish to consider supporting the Secretary of Defense's
efforts to create such an environment through changes to the
acquisition process that provide program managers clear incentives
for gaining sufficient knowledge at key points in weapon acquisition
programs.  The best commercial practices described in this report
suggest what may constitute "sufficient" levels of knowledge.  If DOD
takes steps to manage technology development efforts outside the
bounds of individual weapon system programs, the Congress may wish to
consider providing the funds needed for such efforts.  The Congress
could also help create the right incentives on individual programs by
favorably considering DOD funding requests to mitigate high risks
early in a program and cautiously considering late requests for funds
to resolve problems that should have been addressed earlier. 


   AGENCY COMMENTS AND OUR
   EVALUATION
---------------------------------------------------------- Chapter 5:4

DOD concurred with a draft of this report and all of its
recommendations (see app.  I).  DOD noted that, although it accepts
technical risks to provide superior weapons that often differ from
risks in commercial industry, it agreed that changes were needed in
its environment to adopt practices that industry has used to become
leaner and more flexible.  DOD stated that it was pursuing such
practices through the Defense Reform Initiative, the National
Performance Review Reinvention Center, Management Reform Memorandums,
and numerous acquisition reform initiatives. 

DOD also highlighted actions that it believes are consistent with our
recommendations.  Regarding redefining the point for launching
programs, DOD noted that its EMD phase was equivalent to the
commercial product launch and that it was taking steps to ensure that
technology development was separated from this phase.  DOD stated
that, to create incentives for acquisition managers to identify
unknowns and risks early in development, it had established goals for
reducing program cycle time and obviating cost growth.  DOD said it
was striving to identify metrics to be used in assessing program
risks and cited several existing sources for such metrics.  In
response to the matter that the Congress favorably consider DOD
proposals for fully funding technology development efforts, DOD
observed that full funding can, in some instances, reduce the
flexibility needed to manage these risky efforts.  DOD also provided
technical comments, which we incorporated where appropriate. 

As DOD takes actions to improve weapon system outcomes by better
identifying and eliminating program risks early, we underscore the
need for an environment that creates incentives for such actions to
succeed.  It was the right environment that gave rise to the
excellent practices we found in leading commercial firms.  Given the
competitive pressures new programs face, a weapon system program that
is started during the technology development phase (Program
Definition and Risk Reduction) is encouraged to accept significant
technological risks, which are not necessarily reflected in cost and
schedule projections.  When that program reaches the EMD point, it is
difficult to back off of the original projections and put the program
on a footing in which requirements, available technology, funding,
and time are matched to finish development and start production. 
Yet, it is from that point on, despite the defense-unique
technological risks previously accepted, that a weapon system program
should have much in common with a commercial program, including the
standards for knowledge that are demanded. 

Likewise, once a program is underway, perhaps the clearest way of
encouraging the right behaviors--specifically, program managers' and
other decisionmakers' willingness to identify and deal with risks
early--is through signals sent by individual decisions made on
individual programs.  Absent such incentives, mechanisms that already
exist for identifying and reporting risks have not worked.  While we
support the search for improved metrics for production-related
knowledge, it is the effective application of good metrics that is
more pressing.  Such metrics should be applied to programs at key
junctures in a way that will enforce standards for knowledge like
those we found at commercial companies.  We did not intend that
providing full funding for technology development efforts should
impede the flexibility needed to manage such efforts.  We have
clarified the wording on that matter. 




(See figure in printed edition.)APPENDIX I
COMMENTS FROM THE DEPARTMENT OF
DEFENSE
============================================================ Chapter 5



(See figure in printed edition.)



(See figure in printed edition.)


MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II


   NATIONAL SECURITY AND
   INTERNATIONAL AFFAIRS DIVISION,
   WASHINGTON, D.C. 
-------------------------------------------------------- Appendix II:1

Louis J.  Rodrigues
Paul L.  Francis
Gordon W.  Lusby
Maria J.  Santos


   CHICAGO FIELD OFFICE
-------------------------------------------------------- Appendix II:2

Michael J.  Sullivan
Marvin E.  Bonner


   OFFICE OF THE GENERAL COUNSEL
-------------------------------------------------------- Appendix II:3

John A.  Carter


*** End of document. ***