Best Practices: Better Management of Technology Development Can Improve
Weapon System Outcomes (Chapter Report, 07/30/1999, GAO/NSIAD-99-162).
The Pentagon plans to boost its investment in new weapons to about $60
billion in fiscal year 2001--a 40-percent increase over fiscal year
1997. The military has high expectations for this investment: that new
weapons will be better and less expensive than their predecessors and
will be developed in half the time. However, the Defense Department
(DOD) will not meet these expectations using its traditional management
approach. Leading commercial firms have changed their practices for
developing products and have achieved the kinds of results DOD seeks.
Maturing new technology before it is included in products is one of the
main determinants of these firms' successes. This practice holds promise
for DOD, for immature technologies have been a main source of problems
on weapon systems. This report assesses (1) the impact of technology
maturity on product outcomes, (2) best practices for managing new
technologies and incorporating them into products, and (3) ways DOD can
adapt these practices to get better outcomes on weapon system programs.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: NSIAD-99-162
TITLE: Best Practices: Better Management of Technology
Development Can Improve Weapon System Outcomes
DATE: 07/30/1999
SUBJECT: Private sector practices
Defense procurement
Comparative analysis
Defense capabilities
Weapons research and development
Weapons systems
Testing
IDENTIFIER: Comanche Helicopter
DOD/NASA Integrated High Performance Turbine Engine
Technology
DOD Defense Acquisition Pilot Program
Defense Reform Initiative
Brilliant Anti-Armor Submunition
DOD Airborne Laser Program
DOD Advanced Amphibious Assault Vehicle Program
Forward Looking Infrared System
HS 702 Satellite
Seawolf Attack Submarine
Army Future Scout and Cavalry System
F-15 Aircraft
F-117 Aircraft
F-22 Aircraft
DOD Advanced Concept Technology Demonstration Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO/NSIAD-99-162
A
Report to the Chairman and Ranking Minority Member, Subcommittee on
Readiness and Management Support, Committee on Armed Services, U. S. Senate
July 1999 BEST PRACTICES Better Management of Technology Development Can
Improve Weapon System Outcomes
National Security and International Affairs Division
B- 280233 Letter July 30, 1999 The Honorable James Inhofe Chairman The
Honorable Charles Robb Ranking Minority Member Subcommittee on Readiness and
Management Support Committee on Armed Services United States Senate
As you requested, this report assesses how best practices offer improvements
to the way the Department of Defense (DOD) incorporates new technology into
weapon system programs. It also assesses the factors that can make it
difficult to mature technologies before they are included on
weapon system programs and what can be done about them. We make
recommendations to the Secretary of Defense on how advanced technologies can
be better managed so they pose less risk when they are included in weapon
system designs.
We are sending copies of this report to the Honorable William S. Cohen,
Secretary of Defense; the Honorable Louis Caldera, Secretary of the Army;
the Honorable Richard Danzig, Secretary of the Navy; the Honorable F.
Whitten Peters, Acting Secretary of the Air Force; the Honorable Jacob J.
Lew, Director, Office of Management and Budget; and to interested
congressional committees. We will also
make copies available to others upon request. If you have any questions
regarding this report, please call me at (202) 512- 4841. Other key contacts
are listed in appendix III.
Katherine V. Schinasi Associate Director Defense Acquisitions Issues
Executive Summary Purpose The Department of Defense (DOD) plans to increase
its investment in new
weapons to about $60 billion in fiscal year 2001- a 40- percent increase
over fiscal year 1997. DOD has high expectations from this investment: that
new weapons will be better and less expensive than their predecessors and
will be developed in half the time. With its traditional management
approach- which has produced superior weapons, but at much greater cost and
time than planned- DOD will not meet these expectations.
Leading commercial firms have changed their practices for developing
products and have achieved the kinds of results DOD seeks. Maturing new
technology before it is included in products is one of the main determinants
of these firms' successes. This practice holds promise for DOD, for immature
technologies have been a main source of problems on weapon
systems. In response to a request from the Chairman and the Ranking Minority
Member, Subcommittee on Readiness and Management Support, Senate Committee
on Armed Services, GAO assessed (1) the impact of technology maturity on
product outcomes, (2) best practices for managing new technologies and
incorporating them into products, and (3) ways DOD can adapt these practices
to get better outcomes on weapon system programs.
Background GAO reviewed commercial and DOD experiences in incorporating 23
different technologies into new product and weapon system designs.
The technologies were drawn from (1) six commercial firms recognized for
their success in developing technically advanced products more quickly than
the products' predecessors and (2) five DOD weapon system
programs that incorporated advanced technologies, including some that did
not encounter problems and some that did. GAO asked the managers of these
technologies to assess the maturity of the technologies at the point they
were included in product development by applying a tool, referred to as
technology readiness levels (TRLs). The National Aeronautics and Space
Administration and the Air Force Research Laboratory use TRLs to determine
the readiness of technologies to be incorporated into a weapon
or another type of system. Readiness levels are measured along a scale of
one to nine, starting with paper studies of the basic concept, proceeding
with laboratory demonstrations, and ending with a technology that has proven
itself on the intended product. The Air Force Research Laboratory considers
TRL 6 an acceptable risk for a weapon system entering the program definition
stage, the point at which
DOD launches its weapon programs, and TRL 7 an acceptable risk for the
engineering and manufacturing development stage. This is an important
distinction because leading commercial firms launch a new product later than
DOD, after technology development is complete. They refer to this point as
the beginning of product development, the point at which they
commit to developing and manufacturing the product. Typically, technology is
still being developed when weapon system programs are launched; the point at
which a weapon system is far enough along to compare to a commercial product
development is likely to be at or after the start of engineering and
manufacturing development.
Results in Brief The experiences of DOD and commercial technology
development cases GAO reviewed indicate that demonstrating a high level of
maturity before new technologies are incorporated into product development
programs
puts those programs in a better position to succeed. The TRLs, as applied to
the 23 technologies, reconciled the different maturity levels with
subsequent product development experiences. They also revealed when gaps
occurred between a technology's maturity and the intended product's
requirements. For technologies that were successfully incorporated into a
product, the gap was recognized and closed before product development began,
improving the chances for successful cost and schedule outcomes.
The closing of the gap was a managed result. It is a rare program that can
proceed with a gap between product requirements and the maturity of key
technologies and still be delivered on time and within costs.
Two conditions were critical to closing the maturity gap. First, the right
environment for maturing technologies existed. Key to this environment was
making a science and technology organization, rather than the program or
product development manager, responsible for maturing technologies to a high
TRL. When a maturity gap persisted, managers were given the flexibility to
take the time to mature the technology or decrease product requirements so
that they could use another, already mature technology. Second, both
technology and product managers were supported with the disciplined
processes, readily available information,
readiness standards, and authority to ensure technology was ready for
products. This support enabled these managers to safeguard product
development from undue technology risks. On the other hand, immature
technologies were sometimes incorporated into products for reasons such as
inflexible performance requirements, increasing the likelihood of cost
overruns and delays in product development. Product managers had little
choice but to accept the technologies and hope that they would mature
successfully. However, the pressures of product development made for an
environment less conducive to maturing technology.
For several reasons, DOD is likely to move technologies to product
development programs before they are mature. Science and technology
organizations, which traditionally operate within fixed budget levels, do
not necessarily have the funds to mature technology to the higher TRLs.
Programs are more able to command the large budgets necessary for reaching
these levels. The pressures exerted on new programs to offer unique
performance at low cost encourage acceptance of unproven technologies. The
technologies GAO reviewed indicate these conditions can be overcome on
individual cases. DOD has several initiatives underway, such as advanced
technology demonstrations, that could make
it more feasible for science and technology organizations to mature
technology before it is moved to product development programs. The challenge
will be whether the lessons learned from these cases and initiatives offer
an approach that has a DOD- wide application.
GAO makes recommendations to the Secretary of Defense on ways to pursue
advanced technologies while lessening their potential for causing problems
on weapon system programs.
Principal Findings Maturity of Technology at The 23 technologies GAO
reviewed spanned a wide range of readiness Program Start Is an
levels- from a low of TRL 2 to a high of TRL 9- when they were included
Important Determinant of
in product development programs. Programs with key technologies at Success
readiness levels 6 to 8 at the time of program launch met or were meeting
cost, schedule, and performance requirements. All of the commercial
technologies and a few of the DOD technologies fell into this category. For
example, Ford managed its voice- activated control technology to TRL 8- a
10- year effort- before introducing it on the 1999 Jaguar. Similarly, the
Defense Advanced Research Projects Agency matured a revolutionary periscope
technology to TRL 9 before it was included on the Virginia class attack
submarine. DOD programs that accepted technologies at a readiness level of 5
or less experienced significant cost and schedule increases due, in part, to
problems with the technologies. DOD's acceptance of technologies at level 4
or lower was not unusual. For example, the key technologies for the Army's
brilliant antiarmor submunition were at levels 2
and 3 when weapon system development began. At these levels, DOD had a
significant gap in technology maturity at the start of the program. The gap
was not closed until well into the development program, and problems with
the technologies were a main contributor to the program's 88- percent cost
growth and 62- percent slip in schedule.
Controllable Conditions Closing the technology development gap before
beginning product Affect How Well a development was the result of good
technology maturation practices and Technology's Inclusion on a
sound methods for moving technologies to products. The more successful
Product Can Be Managed
of the 23 technologies were managed by science and technology organizations
until they reached at least TRL 6 and more, often TRL 8 or higher. This
environment was an important condition for successfully maturing
technologies, as it allowed room for unexpected results such as test
“failures,” which are considered normal events in developing
technologies. To match technology maturity and product requirements,
managers also had the option of waiting until technologies matured or
changing product requirements so that an already mature technology could
be used. For example, Hughes deferred the development of the HS- 702
satellite until critical solar cell technology had matured- a process that
took over 10 years. Also, Navy managers accepted an existing weapon ejection
system on the Virginia class attack submarine when technology failed to
mature as expected. In contrast, performance requirements for the Comanche
helicopter were inflexible; requirements mandated the inclusion of advanced
sensors and avionics technologies, despite their immaturity. The Comanche
program has experienced cost growth and schedule delays, partly attributable
to the inclusion of these technologies.
In the more successful cases, technology and product managers were given the
authority and tools to move technology only when it was at high readiness
levels. Disciplined processes provided managers credible information on the
status of technologies and high standards for assessing
readiness. Science and technology managers developed technologies to
standards acceptable to product managers who could reject those technologies
that fell short. For example, Ford's science and technology managers use
agreed- upon standards for judging technology readiness, and all new
technologies follow the same maturation process. Ford's product managers are
also empowered to say no when technologies are not deemed
mature. Recently, the Jaguar vehicle team rejected night vision technology
at TRL 8 because it did not meet cost objectives. DOD program managers that
had to accept immature technologies had less information available to
guide them. For example, key technologies for the brilliant antiarmor
submunition program bypassed Army science and technology organizations,
forcing the program manager to accept the technologies with little
information about their readiness. Often, the tools used to assess the
technologies' status failed to identify high risks; the TRLs indicate that
risks on the problematic technologies were often high. Also, the greater
pressures to meet cost and schedule goals in product development provided a
less forgiving environment for fledgling technologies. Impediments to
Adopting Leading commercial firms have put the organizations, tools, and
other Best Practices for practices in place to foster technology development
and improve the Technology Inclusion in
outcomes of product developments as a matter of necessity. The large DOD Are
Surmountable investment required for a new product and the risks to that
investment if the product does not meet customer needs reinforce these
practices. The DOD cases that followed a similar approach- the Advanced
Amphibious
Assault Vehicle and the Virginia class attack submarine- have so far avoided
problems with key technologies. Yet these cases are not the norm for DOD
programs. DOD programs operate under conditions that make it more difficult-
and less rewarding- to separate technology from product development and to
allow technology to reach high maturity before being included in an
acquisition program.
It is easier for weapon system programs to fund technology development at
higher readiness levels because they attract much bigger budgets than
science and technology projects. DOD typically does not fund science and
technology organizations to take technology past the feasibility stage- TRL
5. As a practical matter, it is often necessary to move immature technology
to a weapon system program to get needed funds and management support. New
programs are pressured to include immature
technologies that offer significant performance gains. These pressures come
from the user's perception of the threat, technologists that see the program
as an opportunity to apply a new technology, and funding competition that
rewards weapon systems with unique features.
DOD and the services have several initiatives for improving the technology
development process and reducing weapon system cycle times. These include
defense technology objectives, advanced technology
demonstrations, advanced concept technology demonstrations, the Army's new
scout/ cavalry vehicle, and the Air Force's Integrated High Performance
Turbine Engine Technology Program. These initiatives are aimed at putting
the science and technology organizations and funding in
place to bring technologies to higher readiness levels before they are
included in weapon system programs.
Recommendations GAO recommends that the Secretary of Defense adopt a
disciplined and knowledge- based approach of assessing technology maturity,
such as
TRLs, DOD- wide, and establish the point at which a match is achieved
between key technologies and weapon system requirements as the proper point
for committing to the development and production of a weapon system. GAO
also recommends that the Secretary (1) require that technologies needed to
meet a weapon's requirements reach a high
readiness level (analogous to TRL 7) before making that commitment, (2)
extract lessons from successful technology inclusion cases for application
to future technology inclusion efforts, and (3) empower program managers to
refuse to accept key technologies with low levels of maturity by making
decisions on individual programs that reinforce a best practice approach to
technology maturation and inclusion. These recommendations appear in full in
chapter 5.
Agency Comments DOD generally agreed with the report and its
recommendations. A detailed discussion of DOD's comments appear in appendix
I.
Letter Executive Summary 2 Chapter 1
Separating Technology Development From Product Development Introduction
Is a Best Practice 12 Technology and Product Development Conducted at the
Same Time
Within DOD 15 Shorter Acquisition Cycle Times Are Needed 17 Objectives,
Scope, and Methodology 18
Chapter 2 Technology Maturity Can Be Measured and Its Consequences for
Maturity of Technology Products Can Be Forecast 22
Technologies With High Readiness Levels at Launch Were Better at Program
Start Is an Able to Meet Product Objectives 25 Important Determinant of
Success
Chapter 3 Providing the Right Environment Is Critical to the Successful
Controllable Maturation of Technology 35 Good Technology Handoff Decisions
Depend on the Tools and
Conditions Affect How Authority Given to Managers 41
Well a Technology's Inclusion on a Product Can Be Managed
Chapter 4 Several Factors Make It Difficult to Mature Technologies Before
Impediments to They Are Included on Weapon Systems 50
Services Encouraged to Use Best Practices 54 Adopting Best
Two Unique DOD Projects May Provide Lessons on How to Enable Practices for
S& T Organizations to Manage Technology Further 56 Technology Inclusion in
DOD Are Surmountable
Chapter 5 Conclusions 61
Conclusions and Recommendations 63
Agency Comments and Our Evaluation 65 Recommendations
Appendixes Appendix I: Technology Readiness Levels and Their Definitions 68
Appendix II: Comments From the Department of Defense 69 Appendix III: GAO
Contacts and Staff Acknowledgments 73
Related GAO Products 76 Tables Table 2.1: Cost and Schedule Experiences on
Product
Developments 27 Table 3.1: TRLs of Technologies Managed by S& T
Organizations 36
Figures Figure 1.1: Cycle for Providing Users a Product With Better
Capabilities 13
Figure 1.2: DOD's Weapon System Acquisition Cycle 16 Figure 1.3: Allocation
of DOD's Fiscal Year 1999 Research and
Development Funds 17 Figure 2.1: Using TRLs to Match Technology With Product
Launch
Requirements 24 Figure 2.2: Readiness Levels of Technologies at the Time
They Were Included in Product Designs 26
Figure 2.3: Time Line for Ford's Development of Voice Activated Controls
Technology 28 Figure 2.4: Jaguar 29 Figure 2.5: Virginia Class Attack
Submarine 30 Figure 2. 6: Brilliant Anti- Armor Submunition 32 Figure 2.7:
Comanche Helicopter 33 Figure 3.1: Hughes Solar Cell Arrays 38 Figure 3.2:
Integrated Avionics for Comanche Helicopter 40 Figure 3.3: Process for
Closing the Gap Between the Readiness of
Adaptive Cruise Control Technology and Jaguar Requirements 43 Figure 3.4:
Process for Closing the Gap Between the Readiness of
Propulsion Technologies and AAAV Requirements 46 Figure 3.5: AAAV 47 Figure
3.6: Assimilation of New Technology Into the BAT Program 48 Figure 4.1:
Airborne Laser 52 Figure 4.2: Comparison of Traditional Technology
Development
Process With the Army's Fast Track Approach 57 Figure 4.3: Future Scout and
Cavalry System 59
Abbreviations
AAAV Advanced Amphibious Assault Vehicle ABL Airborne Laser ACTD Advanced
Concept Technology Demonstration ATD Advanced Technology Demonstration BAT
Brilliant Anti- Armor Subminition DARPA Defense Advanced Research Projects
Agency DEAL Deliverables Agreement Log DOD Department of Defense DTO defense
technology objective FLIR forward- looking infrared NASA National
Aeronautics and Space Administration S& T science and technology TRL
technology readiness level
Chapt er 1
Introduction A central piece of the National Military Strategy is the
military capability represented by advanced weaponry. The Department of
Defense (DOD) plans to increase its annual investment in new weapons to
about $60 billion by fiscal year 2001- a 40- percent increase over fiscal
year 1997. DOD has high expectations from this investment: that new weapons
will be better and less expensive than their predecessors and will be
developed in half the time. These expectations frame a great challenge for
managers of programs. The traditional management approach- which has
produced
superior weapons but at much greater cost and time than planned- will not
meet these expectations. Cycle times- the time to develop a new weapon- can
be so long that the technology a weapon is designed with becomes obsolete
before it can be produced. Costs of new weapons have reached the point that
significantly fewer can be bought than planned.
These are not new issues, but they have become more pressing as the pace and
sophistication of foreign and commercial technology have increased,
complicating a national security environment of unknown threats.
Leading commercial firms have changed the way they develop products and have
achieved the kinds of results DOD seeks, often yielding more sophisticated
products in half the time formerly needed. Industry experts
estimate that resolving technology problems before product development
begins results in 10 times the savings compared to correcting problems
afterward. In this sense, technology maturity breeds product success. The
practices leading firms use to mature and transition technology to products
hold promise for DOD, for immature technologies have been main sources of
problems on weapon systems. We have previously reported on the different
elements of knowledge firms insist on to get better products to market
faster. Of these, no element is more important than having technology,
advanced enough to meet requirements but also mature enough to be
predictably managed, available at the start of the product development
cycle. Maturing new technology before it is included on a
product is perhaps the most important determinant of the success of the
eventual product- or weapon system. It is the topic of this report.
Separating Technology The cycle for placing better capabilities in the hands
of users- both
Development From military and commercial- can be described as consisting of
technology
Product Development Is a Best Practice
development, product development, and production. In a 1998 report, 1 we
characterized the knowledge needed on a new product as consisting of three
knowledge points: when a match is made between a customer's requirements and
the available technology; when the product's design is
determined to be capable of meeting performance requirements; and when the
product is determined to be producible within cost, schedule, and quality
targets (see fig. 1.1). We found that this knowledge, when obtained at the
right time and in the right sequence- technology, design, and manufacturing-
was a best practice. This practice lowered product development risks,
reduced cycle times and costs, and resulted in smoother production programs.
Figure 1.1: Cycle for Providing Users a Product With Better Capabilities
Technology Product
Production Development
Development Product launch Knowledge
Knowledge Knowledge
Point 1 Point 2
Point 3 Technology Design
Control of readiness
maturity manufacturing
process
Leading commercial firms recognize a distinct difference between technology
development and product development; accordingly, they develop technology
before introducing it into product development programs. They minimize risk,
improve cost and schedule outcomes, reduce cycle time, and improve quality
during product development by
1 Best Practices: Successful Application to Weapon Acquisitions Requires
Changes in DOD's Environment (GAO/ NSIAD- 98- 56, Feb. 24, 1998).
gaining significant knowledge about a technology before launching the
product development. Scientists and technologists- different people than
those that manage product developments- manage the development of technology
until it is ready to be included in the design of a product. Program launch
is the point at which a firm defines a product's
performance, cost, and schedule estimates and begins making a large
investment in human capital, facilities, and materials- an investment that
increases continuously as the product approaches the point of
manufacture. It includes a commitment to manufacture the product. Therefore,
program launch and the start of product development are synonymous within
commercial firms. Protecting this investment provides a strong incentive for
firms to minimize the potential for technology development problems during
the product phase and cause delays.
Confining delays in maturing technology to a time prior to launch- in an
environment where small teams of technologists work in laboratories and are
dedicated to perfecting the technology- is critical to saving time and
money. If delays occur during product development, when a large engineering
force is in place to design and manufacture the product, they would be much
more costly. In fact, industry experts estimate that identifying and
resolving a problem before product development can reap a
10- fold savings compared to correcting the problem after launch and that
correcting the same problem in the manufacturing stage would be even more
costly. Leading commercial firms have found that managing technology
development separately from and before product development is a major reason
they have been able to reduce product cycle times. As a whole,
50 to 70 percent reductions in cycle times are not unrealistic achievements
by leading commercial firms. For instance, leaders in the automobile
industry have reduced cycle times from 7 years to 2 years, or by about
70 percent. The consumer electronics industry has recently reduced its cycle
time from 2 years to 6 months, and the commercial aircraft industry has
achieved reductions of 50 percent. Leading commercial firms have found that
reducing the product development cycle time brings products to market
faster, results in an increased market share, and helps to keep products
from becoming technologically obsolete.
Technology and DOD's process for developing and manufacturing weapon systems
is Product Development
described as a cycle consisting of phases. These phases are concept
exploration, program definition and risk reduction, engineering and
Conducted at the Same
manufacturing development, and production and fielding. The basic Time
Within DOD process of gathering knowledge about technology, design, and
manufacturing is followed, but in practice, the DOD cycle does not make a
clear distinction between technology development and product development.
The launch of a program in DOD usually takes place several years before the
beginning of product development does in leading commercial firms. In fact,
a new weapon system program is normally launched at the start of the program
definition and risk reduction phase, which is often in the midst of
technology development, while most product development activities do not
begin until the engineering and manufacturing development phase.
Consequently, technology, design, and manufacturing knowledge is attained
concurrently-- in the higher cost environment that characterizes product
development-- throughout the weapon system phases. In our February 1998
report, we noted that such technology development problems are a major cause
of cost increases and schedule delays on DOD
weapon system programs. The phases in DOD's weapon system acquisition cycle
and the knowledge gathering process, as it is typically followed, are shown
in figure 1.2.
Figure 1.2: DOD's Weapon System Acquisition Cycle
Concept Program Engineering and
Production exploration definition and
manufacturing and fielding risk reduction
development Program launch Begin product development
Technology maturity Knowledge
Design maturity attainment
Manufacturing processes controlled
DOD's process also has organizational and budgetary implications. Activities
accomplished in the first three phases of the acquisition cycle use research
and development funds whereas production programs use procurement funds.
Generally, DOD's science and technology (S& T) community is responsible for
basic research, applied research, and advanced technology development to
produce generic, rather than weapon- specific, technologies. Its goal is to
conduct research, develop technology, and farm these efforts for potential
military application, such as a weapon system. The S& T community also uses
research and development funds, but its work generally precedes the
acquisition cycle. Weapon system program managers, who receive most of DOD's
research and development budget, apply generic technologies to specific
weapon systems. However, they often become responsible for completing
development of generic technologies as well. The allocation of DOD's fiscal
year 1999 research and development funds to these categories is shown in
figure 1.3.
Figure 1.3: Allocation of DOD's Fiscal Year 1999 Research and Development
Funds 3%
Basic research - $1. 1 billion
9%
Applied research - $3. 2 billion
9%
Advanced technology development - $3. 5 billion
Weapon specific
79%
development - $29.6 billion Source: DOD
S& T officials stated their role is to show that technology is feasible
through laboratory experiments or demonstrations. It is often at this point
that the technology's military potential will be identified and the
technology will be harvested for inclusion on a weapon system. Because the
technology is still not mature, its development will be completed as part of
the weapon
system's design and development, under the authority of the weapon system
manager and apart from the S& T community.
Shorter Acquisition DOD's weapon acquisition cycle times average between 10
to 15 years- far Cycle Times Are
longer than the cycle time for commercial products. To an extent, DOD's
cycle times are longer because they start earlier than commercial cycles
Needed
and often entail more complex products. Compounding the length of the weapon
system development cycle is its unpredictability. Over the years, we have
issued numerous reports highlighting cost overruns and schedule delays
during the product development cycle, for which technology
development problems were a major cause. These problems require additional
technology development activities to take place at a time when
the product should be undergoing design and manufacturing development. As a
result, the pace of technology advances outruns the time to develop a weapon
system and some of the more mature components designed into a weapon system
become obsolete before the weapon is manufactured. For example, the F- 22
will have almost 600 obsolete components by fiscal year
2000 while the aircraft is still in development. The longer a weapon
system's development cycle, the more prone the program is to management and
funding changes. According to DOD, an 11- year development program
historically encounters a 30- percent cost growth over time. Based on
historical averages, DOD calculates that the typical program will have four
different program managers, eight defense acquisition executives, and seven
Secretaries of Defense- all of who are major influences and decisionmakers
on the program. In addition, the program will have gone through 11 annual
budget cycles in which funding changes could have occurred and affected the
program's content.
The Under Secretary of Defense for Acquisition and Technology has stated
that cycle time reduction is necessary to meet DOD's goals of delivering
emerging technologies to warfighters in less time and at lower costs. The
Under Secretary has set a goal to reduce the average acquisition cycle time
for all program starts in fiscal year 1999 and beyond by 50 percent over
historical averages. Reductions in cycle times will (1) allow for earlier
fielding of increased capabilities, (2) reduce costs, (3) free up funds for
more programs, (4) reduce the potential for components becoming obsolete,
and (5) take more frequent advantage of technology advances found in the
commercial world. An emphasis on shorter cycle times may also reduce the
tendency to add technological advances that are unproven and immature into
weapon acquisition programs. To help achieve this goal, DOD is working on
several efforts such as Defense Acquisition Pilot Programs, the Defense
Reform Initiatives, and many acquisition reform
projects. The Under Secretary has also advocated adopting the practices of
leading commercial firms and taking a more evolutionary approach to
developing weapon systems, which would lessen the amount of technology
development initially attempted within a weapon system program. Objectives,
Scope, and
The Chairman and the Ranking Minority Member, Subcommittee on Methodology
Readiness and Management Support, Senate Committee on Armed Services,
requested that we examine various aspects of the acquisition process to
determine whether the application of best practices can improve program
outcomes. To date, we have issued reports on advanced quality
concepts, earned value management, management of a product from development
to production, and management of key suppliers (see related GAO products).
This report covers the inclusion of technology into weapon system programs,
and is, in a sense, a prequel to our report on product development. Our
overall objective was to determine whether best practices offer methods to
improve the way DOD matures new technology
so that it can be assimilated into weapon system programs with less
disruption. Specifically, we assessed (1) the impact of technology maturity
on product outcomes, (2) best practices for managing new technologies and
incorporating them into products, and (3) ways DOD can adapt best practices
to achieve better outcomes on weapon system programs.
Our methodology consisted of analyzing 23 commercial and DOD technologies
that had transitioned or attempted to transition into product development
programs. The technologies were drawn from six commercial firms recognized
for their success in developing technically advanced products more quickly
than their predecessors and five weapon system programs that incorporated
advanced technologies, including some that did not encounter problems and
some that did. We asked the managers of these technologies to apply a tool,
referred to as technology readiness levels (TRLs), for our analysis. The
managers used TRLs to judge the maturity of the technologies at the time
they had entered product
development or were included in programs. The National Aeronautics and Space
Administration (NASA) originally developed TRLs, and the Air Force Research
Laboratory uses them to determine when technologies are ready to be handed
off from S& T managers to product development managers. We held discussions
with the DOD and NASA users of TRLs to better understand their applicability
to our review. They stated that TRLs can be
used as general indicators of a technology's readiness level and associated
risk of including the technology into a product development program, given
its TRL at that time. TRLs are more fully explained in chapter 2. To
understand the best practices the commercial sector used to include
technologies in product development programs, we conducted literature
searches and focused those searches as the review progressed. On the basis
of the searches, we identified a number of commercial firms with innovative
technology development processes for including new or
advanced technologies into new products. We used structured interview
questions sent in advance of our visits to gather uniform and consistent
information about each firm's process and practice and the results
achieved. In addition, we examined four specific technology cases- Ford's
night vision, adaptive cruise control, and voice activated controls and
Hughes' solar cell array- to better understand their processes and
practices. The commercial firms we visited were
Ethicon- Endo Surgery (medical device manufacturer), Division of Johnson and
Johnson, Cincinnati, Ohio; Ford Motor Company (automobile manufacturer),
Dearborn, Michigan; Harris Semiconductor (semiconductor manufacturer),
Melbourne,
Florida; Hughes Space and Communications (satellite and spacecraft
manufacturer), Los Angeles, California;
3M (commercial products manufacturer), St. Paul, Minnesota; and Motorola
Corporate Research Headquarters (communications technology manufacturer),
Schaumburg, Illinois, and Motorola Land Mobile Products Sector, Plantation,
Florida.
We also attended and participated in conferences and workshops with
recognized leaders in the acquisition field to obtain information on how
organizations are improving their acquisition processes. Finally, we
interviewed officials from trade organizations concerning the application of
commercial practices to DOD operations.
To better understand DOD's technology inclusion process, we selected 19
advanced technologies that had been included in 5 DOD weapon system programs
that were in various stages of the acquisition process. We collected
technical reports, acquisition management, and risk management documentation
about the technologies. In addition, we interviewed S& T
and acquisition program management officials about each technology's
development history, costs, and current status. The technologies and
programs reviewed were acoustic sensor, infrared seeker, inertial
measurement unit, tandem
shaped charge warhead, and processor technologies from the Army's Brilliant
Anti- Armor Submunition (referred to as BAT) Program; rotor, engine,
integrated avionics, forward looking infrared, and helmet mounted display
technologies from the Army's Comanche Helicopter Program;
nonpenetrating periscope and weapon ejection system technologies from the
Navy's Virginia class attack submarine program; high speed planing craft,
power dense diesel engine, lightweight
composite armor, high power water jet, moving map and advanced navigation
technologies from the Marines' Advanced Amphibious Assault Vehicle (AAAV)
Program; and
laser and beam control technologies from the Air Force's Airborne Laser
(ABL) Program. To determine relevant DOD policy and initiatives, we obtained
documents and interviewed officials of the Office of the Secretary of
Defense; the Defense Advanced Research Projects Agency (DARPA); and Army,
Navy and Air Force Science and Technology organizations. We also had
discussions with former DOD officials and industry experts about DOD
acquisition policies and practices.
Even though we selected firms with product lines of varying complexity, we
did not concentrate only on firms whose products had the most in common with
weapon systems. Such an approach would have limited our ability to include
firms recognized as the best at including new, advanced technologies into
programs. In our analysis, we concentrated on the
criteria and knowledge used to support technology readiness decisions.
Although the approach from product to product may vary, the basic processes
and standards leading commercial firms applied to technology inclusion
decisions were consistent. We were limited, however, in our ability to
obtain and present some relevant data that commercial
companies considered proprietary in nature. This information included
funding amounts for investing in technology development, details on
technological innovations, and some specific data from recent technology
inclusion successes. Our report highlights the best commercial practices for
including technology into product development programs. As such, they are
not intended to describe all commercial industry practices or to suggest
that commercial firms do not have any flaws. We conducted our review between
March 1998 and June 1999 in accordance with generally accepted government
auditing standards.
Maturity of Technology at Program Start Is an
Chapt er 2
Important Determinant of Success The experiences of the DOD and commercial
technology development cases we reviewed indicate that demonstrating a high
level of maturity before allowing new technologies into product development
programs puts those programs in a better position to succeed. Simply put,
the more mature technology is at the start of the program, the more likely
the program will succeed in meeting its objectives. Technologies that were
included in a product development before they were mature later contributed
to cost increases and schedule delays in those products.
We found an analytical tool- TRLs- that can assess the maturity level of
technology as well as the risk that maturity poses if the technology is
included in a product development. The tool associates different TRLs with
different levels of demonstrated performance, ranging from paper studies
to proven performance on the intended product. The value of using the tool
is that it can presage the likely consequences of incorporating a technology
at a given level of maturity into a product development, enabling
decisionmakers to make informed choices. TRLs proved to be reliable
indicators of the relative maturity of the 23 technologies reviewed, both
commercial and military, and their eventual success after they were included
in product development programs.
Technology Maturity Successful technologies progress from initial concept to
proven
Can Be Measured and performance, whether they are developed in the
laboratory or in the
factory, by commercial industry or DOD. The Air Force Research Its
Consequences for
Laboratory has adapted and uses TRLs to measure the key steps in this
Products Can Be progression toward inclusion into weapon systems. TRLs are
measured Forecast along a scale of one to nine, starting with paper studies
of the basic concept and ending with a technology that has proven itself in
actual usage on the intended product. A detailed description of TRLs is
provided in appendix II, but the following hypothetical example about an
airborne communications radio can illustrate the readiness levels.
First, the idea for a new radio is conceived. The idea reaches TRL 3 when
analytical studies and some tests of the technology's elements, such as a
circuit, back it up. When initial hand- built versions of all of the radio's
basic elements are connected and tested together, the radio reaches TRL 5.
This is sometimes referred to as a “breadboard” article;
although it may function
like a radio, it does not look like one because the individual parts are
attached to plywood and hand- wired together. When the technology is built
into a generic model, which is well beyond the breadboard tested in TRL 5,
and demonstrated in a laboratory environment, the radio reaches TRL 6.
This model represents the last level of demonstration before the radio
becomes tailored for application to a specific aircraft. When the components
are assembled inside a case that resembles the final radio design and are
demonstrated aboard a surrogate for the intended aircraft, the radio reaches
TRL 7. TRL 8 is reached when the radio is put in its final form, installed
in the intended aircraft's cockpit, and tested in conjunction with the other
aircraft equipment with which it must interface. TRL 9 is achieved when the
radio is successfully operated on the aircraft through
several test missions. Unexpected problems can arise at every level, and
effort must be expended to overcome them. This effort takes time and can
delay the progress to the next readiness level.
Once a technology's readiness level has been established, the risks of
including that technology in a product development can be assessed. Unlike
S& T projects, for which the main objective is to develop knowledge, a
product development's objective is to deliver products that meet strict
cost, schedule, and performance targets. We found that most leading
commercial firms, after they had translated their own methods of assessing
risk into TRLs, determined that a TRL 8 was required before they allowed a
new technology into a product development. 1 DOD launches a program in the
program definition and risk reduction phase- much earlier than the
leading commercial firms do. According to the Air Force Research Laboratory,
a TRL 6 is required for a technology to be an acceptable risk for a program
in that phase. When weapon system development reaches the engineering and
manufacturing development phase, it more nearly approximates the point at
which a commercial product development program would start. The Air Force
Research Laboratory depicts a
technology at TRL 7 as an acceptable risk for this phase- technologies at
lower levels would be considered high risks.
The lower the level of technology readiness, the more ground must be covered
to bring the technology to the point at which it can meet the intended
product's cost, schedule, and performance requirements with little risk (see
fig. 2.1).
1 An exception to this is space systems technology. Space- based
technologies are generally included on a development program once they have
been prototyped and ground tested- a TRL 6, the highest level attainable
short of space operation.
Figure 2.1: Using TRLs to Match Technology With Product Launch Requirements
High risk for Low risk for
Product
product launch product launch
Requirements 8 9
Risks or
6 7
unknowns
4 5 TR L
1 2 3
The gap between the maturity of the technology and the product's
requirements represents the risks or unknowns about the technology. As each
succeeding level of readiness is demonstrated, unknowns are replaced by
knowledge and the gap becomes smaller. Ideally, the gap is closed before a
new technology is included in a new product's design,
although the Air Force Research Laboratory accepts the amount of risk at TRL
7 for a program entering engineering and manufacturing development.
Technologies that reach TRL 7 or higher at the start of product development
allow product managers to focus their attention on integrating the
technologies and proving out the product design. Technologies that are
included at lower maturity levels require more of the product managers'
attention and resources, as basic knowledge about those technologies must
still be gained.
Thus, a major purpose served by TRLs is to reveal the gap between a
technology's maturity and the maturity demanded for successful inclusion in
the intended product. With TRLs as guides, the options available to
decisionmakers can be framed. Given that a key determinant of achieving cost
and schedule outcomes for a product development is the technology's maturity
at product launch, decisionmakers can either (1) delay product
development until the technology is matured to a high enough readiness level
or (2) reduce the product's requirements so that a less advanced, but more
mature, technology can suffice. If it is perceived that the requirements of
the product cannot be lowered and the product launch cannot be delayed until
the requisite technology is of a sufficient readiness level, then the
remaining option is to launch the product development with the immature
technology. If this option is chosen, then the success of the product
development will depend heavily on the product manager's ability to
simultaneously close the technology maturity gap and develop the product for
manufacture, which is a very challenging task.
TRLs do not represent strictures that must be adhered to without exception.
According to the people in DOD who have used TRLs, there are occasions when
a lower than expected TRL can be accepted, such as when the product
development's schedule and resources are generous enough that the technology
will have enough time to mature. In other instances, a higher than expected
TRL may be required, such as if the technology in question is the linchpin
for the entire product. Nonetheless, we found that TRLs ably reconciled the
different maturity levels and product experiences
of the 23 technologies reviewed. Technologies With The 23 technologies
reviewed spanned a wide range of readiness levels at High Readiness Levels
the time they were included in product development programs. The least
mature reached TRL 2 at the time it was included in a product at Launch Were
Better
development, while the most mature had reached TRL 9 at the point of Able to
Meet Product
inclusion. We observed a general relationship between TRLs and the
Objectives
technologies' inclusion on the intended product developments. Those products
whose technologies reached high TRLs at the time they were included were
better able to meet cost, schedule, and performance
requirements. In fact, commercial firms informed us that maturing the
technology separately from and ahead of the product was a main reason they
were able to reduce cycle times on their products. An official from one of
the firms termed the approach as “moving discovery to the left.”
Those technologies with low TRLs at inclusion encountered maturation
difficulties and contributed to problems the products experienced. Other
problems, such as funding and schedule changes unrelated to the
technologies, also contributed to problems in the product developments.
Figure 2.2 shows the TRLs when each of the 23 technologies was included in a
product design, whether at product development launch (for commercial
technologies) or at program launch (for DOD technologies).
Figure 2.2: Readiness Levels of Technologies at the Time They Were Included
in Product Designs
Non- penetrating periscope Adaptive cruise control
Night vision Voice activated control
Solar cell array High- speed planing craft
High power water jet Weapon ejection system
Diesel engine Helicopter rotor Lightweight composite armor
Helicopter engine Moving map and navigation
Hemical oxygen iodin laser Beam control system Helmet mounted display
Helicopter forward linking infrared
Integrated avionics Data processor Inertial measurement unit
Warhead Infrared seeker Acousting targeting sensor
1 2 3 4 5 6 7 8 9 Technology Readiness Levels
Commercial technologies DoD technologies
The cost and schedule experiences of some of the products or programs that
inherited the technologies are shown in table 2.1.
Table 2.1: Cost and Schedule Experiences on Product Developments Product
development TRL at Product development and
program associated technologies
launch Cost growth Schedule slippage
Comanche helicopter 101 percent a 120 percent a Engine
5 Rotor
5 Forward looking infrared
3 Helmet mounted display
3 Integrated avionics
3 BAT
88 percent 62 percent Acoustic sensor
2 Infrared seeker
3 Warhead
3 Inertial measurement unit
3 Data processors 3
Hughes HS- 702 satellite None None Solar cell array 6 Ford Jaguar
None None Adaptive cruise control
8 Voice activated controls 8 a The Comanche, in particular, has experienced
a great deal of cost growth and schedule slippage for
many reasons, of which technology immaturity is only one. Other factors,
such as changing the scope, funding, and pace of the program for
affordability reasons, have also contributed.
Data for three weapon system development programs, the Virginia class attack
submarine, AAAV, and the ABL, were not included in the table because they
had not been in the product development phase long enough to report actual
consequences. To date, AAAV and the submarine have stayed within 15 percent
of their cost and schedule estimates for development. The ABL, for which key
technologies were much less mature at program launch, still faces challenges
with these technologies. Ford's
night vision technology was excluded because the firm decided not to include
the technology on a product. Details on the Comanche, BAT, the Virginia
class attack submarine, and Ford technology and product experiences follow.
Technology and Product The key technologies for the Ford Jaguar and the
Virginia class attack Experiences on Ford and
submarine followed the pattern of increasing TRLs until they demonstrated
Virginia Class Attack a low risk for transition to the product. Two examples
are Ford's voice
Submarine activated controls development and DARPA's nonpenetrating
periscope
development for the submarine. In both cases, the technologies were
validated, operational prototypes demonstrated, and the technologies had
demonstrated the form, fit, and function of the final article by the
beginning of product development. Ford's voice activated controls
technology, which allows a driver to control
certain functions such as windows and the radio through verbal commands, was
under development in the technology base for over 10 years, being pushed by
the firm's technology leaders. It was not until 1993 that Ford found that
(1) other complementary technologies, such as processor speeds and low cost
memory, had become available and
(2) customers wanted more features and functions but less distractions from
driving. Given this market information, Ford decided to pursue voice
technology as a strategic technology in terms of product differentiation,
recognizing the importance of being first to market with this enabling
technology. Figure 2.3 shows the time line for developing this technology.
Figure 2.3: Time Line for Ford's Development of Voice Activated Controls
Technology
1983 1993
1995 1999 Ford decides to pursue
Technology is linked Technology is ready to
Technology featured voice activated controls
to a specific vehicle. transition into a product
on model year 1999 technology. Technology
Cost and performance development program.
Jaguar designs. under early development
requirements are Technology meets all in technology base.
defined. cost and schedule targets for the product.
TRL 3 - 5 TRL 6 - 7
TRL 8 TRL 9
Between 1993 and 1994, based on discussions with customers, Ford developed
cost and performance requirements for the technology. Ford has
never relaxed them. By September 1995, when Ford allowed the technology into
the development program for a new Jaguar design, voice activated controls
had been demonstrated as an integrated system in the appropriate form and
fit for the Jaguar. Ford officials stated that the product has met all
cost and cycle time targets established at the outset of its development.
Figure 2.4 shows the Jaguar.
Figure 2.4: Jaguar
Ford demonstrated voice activated control technology in the appropriate form
and fit before incorporating it into the Jaguar.
Source: Ford Motor Company.
DARPA began developing the nonpenetrating periscope technology as part of
its submarine technology development efforts after recognizing, in 1988,
along with the Navy, that the nonpenetrating technology would enhance
operator visibility, provide greater submarine design flexibility, and be
stealthier than conventional masts and periscopes. At the time, the Virginia
class attack submarine program had not been initiated. Once the decision
was made to include the nonpenetrating periscope, it became a key feature of
the submarine and was a major design driver for the submarine's overall
configuration.
Nonpenetrating refers to the fact that the periscope is essentially a group
of sensors that are linked to the submarine via fiber optic and other
cables. This technology uses infrared imaging and advanced sensors to
replace conventional periscopes and frees up physical space compared with a
conventional periscope. A conventional periscope relies on a series of
telescoping shafts and reflecting surfaces to see above the water's surface.
When the periscope is retracted, the shafts take up a column of space from
the top of the submarine to the bottom, through all decks. Its location
virtually dictates the design and placement of the control and other rooms.
If the nonpenetrating periscope technology did not become available, then
the submarine would have to be drastically redesigned to accommodate the
space required by a conventional periscope.
The new nonpenetrating periscope and photonics mast technology underwent
land testing in 1991- a TRL 5. The Navy actually tested the new technology
at sea on the U. S. S. Memphis in 1992 and 1993. According to program
officials, these sea trials demonstrated the highest level of technology
readiness: proving the actual system through successful mission operations.
This readiness equated to a TRL 9. Yet, this technology was not included in
the Virginia class attack submarine requirements until 1995. Figure 2.5
shows an artist's concept of the Virginia class attack submarine.
Figure 2.5: Virginia Class Attack Submarine
The Navy demonstrated a key technology at the highest readiness level before
including it as a requirement for the Virginia class attack submarine.
Source: DOD. The high readiness level of the nonpenetrating periscope
afforded the Navy the opportunity to develop an improved version of the
periscope to a TRL 9. This was a relatively low- risk endeavor as the
baseline periscope was sufficient to meet the submarine's requirements.
Program officials believe that having knowledge about key technologies, such
as the nonpenetrating periscope, for the Virginia class attack submarine at
program launch made a short program definition and risk reduction phase
possible. This phase for the Virginia class attack submarine was about
75 percent shorter than those of previous acquisition programs. Based on its
demonstrated maturity, we anticipate that the nonpenetrating periscope to be
less likely to impact the cost and schedule of the submarine's development
program. There are, however, several other technologies that
are critical to the submarine program. We did not examine these technologies
and cannot predict their likely outcomes.
BAT and Comanche Cases Key technologies for the BAT and Comanche programs
had much lower readiness levels at the time the product developments were
launched.
Consequently, they did not reduce the gap between their demonstrated
maturity and the maturity needed to meet product requirements until after
program launch. For some technologies, the gap was not closed until well
into the product development program. For others, the gap has still not been
closed. Five key technologies included in the Army's BAT program had low
TRLs when they were included on the program. The level of readiness for most
of
these technologies at program launch was characterized by the program office
as experimental in nature but with major uncertainty remaining- a TRL 3. The
acoustic targeting technology was the most important enabling technology
needed to meet the weapon's performance requirements. This technology
provides BAT the capability to locate targets from great distances based on
the sounds generated by the target, such as moving tanks and vehicles. At
the time the program was launched, the Army knew little about the
feasibility of using this technology on this program. In fact, the
technology was still being defined in paper studies- a TRL 2. The
Army did not prototype this technology until after the program had entered
the engineering and manufacturing development phase, more than 6 years after
program launch. As of December 1998, the BAT had experienced significant
development cost and schedule increases, which program officials attribute
at least, in part, to unknowns about the new technologies. Figure 2.6 shows
the BAT.
Figure 2.6: Brilliant Anti- Armor Submunition
At the time the program was launched, the Army knew relatively little about
the performance of several key technologies for the BAT
Source: DOD.
Two technologies key to meeting the Comanche helicopter's
requirements– integrated avionics and forward- looking infrared (FLIR)
technologies- were included on the program when they were still conceptual
in nature. The integrated avionics technology replaces individual radios,
navigation, and other communication equipment with a modular system that
shares a common processor. The FLIR is a second- generation version that
uses infrared sensors to improve the pilot's ability to see at night and in
bad
weather. Program officials stated that both had TRLs of 3 when the
helicopter program was started. Despite the low readiness levels of the
technologies, the Army included the technologies on the program to meet
weight, cost, and performance requirements.
The development of these technologies has taken longer than the Army
expected it would. The contractor for the integrated avionics has had
difficulties in getting the multiple avionics modules to work simultaneously
within required size and weight parameters, and the FLIR technology has
undergone several design and performance requirement changes. As of
September 1998- approximately 10 years after program launch- neither
the integrated avionics nor the FLIR technology had advanced past a TRL 5.
Problems with the maturation of these technologies have contributed to the
program's cost and schedule increases. In contrast, the advanced rotor and
engine technologies, which were the most mature of the Comanche technologies
we reviewed, have experienced fewer problems in maturation and have not
contributed significantly to the program's cost and schedule increases.
Figure 2.7 shows the Comanche.
Figure 2.7: Comanche Helicopter
The Army included two key technologies in the Comanche when they were still
considered conceptual to meet weight, cost, and performance requirements
Source: DOD.
Controllable Conditions Affect How Well a Technology's Inclusion on a
Product Can Be
Chapt er 3
Managed Closing the gap between technology maturity and product requirements
before a product is launched- and baselines are set- distinguished the more
successful cases. Notably, closing the gap before product launch was a
managed result; it put product managers in a better position to succeed. Two
conditions were critical to achieving this kind of result. First was an
environment that put the primary responsibility for maturing technology in
the hands of S& T managers and provided them considerable flexibility to
make decisions. Second was having the quality information and standards
needed to make good technology handoff decisions, coupled with giving the
product manager the authority to refuse new technology that did not meet
product requirements. When these conditions were not present, the handoff to
the product manager was compromised, with negative
consequences for both technology and product. In each of the successful
cases, S& T organizations played major roles in bridging the gap between
technology maturity and product requirements. Flexibility provided by
requirements communities and resource providers enabled S& T and product
managers to delay the inclusion of technology if it was not ready or to
reduce product requirements to match what mature technology could deliver.
This environment was better suited to the
unexpected results and delays that accompany technology development.
Moreover, technology maturation was managed within a disciplined process
that provided good information to be judged against clear and high
standards, like TRLs. Armed with the tools and the authority to make
technology inclusion decisions, both S& T and product managers functioned as
gatekeepers to safeguard the product development.
In the more problematic cases, S& T organizations disengaged much earlier,
and product managers had little choice but to accept immature technologies.
Accordingly, less information about the technologies was available at the
point of inclusion. Often, the tools used to assess the technologies' status
failed to identify high risks. In retrospect, TRLs indicated that risks were
in fact high and perhaps unacceptable from a
product standpoint. Also, pressures to meet cost and schedule estimates in
product development provided a less forgiving environment for technologies
in the discovery process.
Providing the Right While most new technologies- commercial and military-
are initially
Environment Is Critical managed by the S& T community, the more successful
cases we reviewed continued to be managed by S& T organizations until they
reached at least
to the Successful TRL 6 and more often TRL 8 or higher. These technologies
were provided Maturation of
the environmental advantages an S& T project has over a product Technology
development. This environment availed S& T managers and product managers of
the less risky options of waiting or trading to get the match between
technology maturity and product requirements- rather than forcing the
product launch and gambling on the completion of technology maturity. In
contrast, the more problematic technologies did not have as benign an
environment. Often, the technologies were handed off early by S& T
organizations because inflexible performance requirements for the product
demanded their inclusion. Product development managers launched the product
development and hoped that the technology development would succeed. Once in
a product development environment, external pressures to keep
the program moving become dominant, such as preserving cost and schedule
estimates to secure budget approval. For example, DOD policies require that
a program be funded in the current year and that funds be made available
over the next 6 years in the DOD planning cycle. If, during the program
definition phase, a program manager were to decide that an additional year
was needed to overcome unexpected technology problems to reach the desired
level of maturity, the delay could push the start of engineering and
manufacturing development back. This delay could
jeopardize the funding for that phase, thus risking the funding support for
the entire program. Consequently, the program manager may be more likely to
accept the risk of not getting the technology to the desired level of
maturity and starting the engineering and manufacturing development phase as
planned, rather than risk the rest of the program. These conditions compete
with and detract from the needs of technology development. One acquisition
official stated that these conditions cause the weapon system program
“to pull double duty,” inventing new technology while
integrating it into a product. In general, he believed there is an equal
amount of
difficulty in both tasks. Technologies Matured by
In the most successful cases that we reviewed, S& T organizations bridged S&
T Organizations Made
the gap between immature technology and the maturity needed for either
Smooth Transitions into program start in DOD (TRL 6) or product development
(TRL 7 or higher).
Product Developments These cases and the responsible organizations are shown
in table 3.1.
Table 3.1: TRLs of Technologies Managed by S& T Organizations TRL at
Responsible Receiving product Technology handoff S& T organization
development program
Nonpenetrating 9 DARPA Virginia class attack periscope submarine program
Adaptive cruise control 8 Ford Advanced Vehicle
Jaguar vehicle team Technology Office Voice activated controls 8 Ford
Advanced Vehicle
Jaguar vehicle team Technology Office Solar cell array 6 Hughes Laboratories
HS- 702 satellite
program Weapon ejection 6 Office of Naval
Virginia class attack system Research submarine program Diesel powered
engine 6 Office of Naval
AAAV program Research High- speed planing
6 Office of Naval AAAV program craft Research High- power water jet 6 Office
of Naval
AAAV program Research Despite the different circumstances between the
commercial and DOD
sectors and among the DOD cases themselves, the results were similar: having
S& T organizations bridge the maturity gap reduced technology- related
problems in the products. For the leading commercial firms we visited, it is
standard practice to have S& T organizations responsible for the bridge. In
the DOD cases shown in table 3.1, the S& T organizations played atypical
roles in managing the bridge between technology and product by delivering
the technology to a TRL 6 or higher. Different pressures and incentives that
are brought to bear on the commercial and DOD product developments explain
why DOD product managers become responsible for more technology development
than their commercial counterparts. These influences are discussed in
chapter 4.
Having an S& T organization manage a technology to maturation means more
than just having a different group of people involved than a product
development. S& T projects operate in a different environment than product
developments. The process of developing technology culminates in discovery
and must, by its nature, allow for unexpected results. S& T
provides a more forgiving environment in which events- such as test
“failures,” new discoveries, and delays in the attainment of
knowledge- are considered normal. It is also a less costly environment,
making external pressures to develop knowledge on a schedule less keenly
felt. On the
other hand, the process of developing a product culminates in delivery, and
thus gives great weight to design and production. The same events and
unexpected results that are considered normal for technology development
represent problems in the product environment; they can jeopardize
achievement of cost and schedule objectives and draw criticism to the
product. The ups and downs and the resource changes associated with the
technology discovery process do not mesh well with a program's need to meet
cost, schedule, and performance goals. This situation has been described as
attempting to “schedule inventions.”
Successful Cases Afforded In the early 1980s, Hughes Space and
Communications began developing
Flexibility to dual junction solar cell technology that had the potential of
greatly Decisionmakers
increasing the electrical power on satellites. By 1985, a Hughes laboratory
had demonstrated the technology by ground testing prototypes, a TRL 6, which
is considered an acceptable level of demonstration for space- based
technology. Nonetheless, Hughes was not satisfied that the supporting
infrastructure (materials, reactors, and test equipment) was mature enough
to sustain development and production of the new technology on a satellite.
The infrastructure was seen as critical to meeting the cost and schedule
requirements of a product. As a result, Hughes did not hand off the
technology to a product. Instead, the firm kept it in a research
environment, away from cost and schedule pressures.
In the early 1990s, Hughes established requirements for a new satellite- the
HS- 702- that would use the solar cell technology to leapfrog the
competition. After a laboratory demonstration in 1993, Hughes successfully
used the new technology on a high- powered version of its existing HS- 601
satellite before it began product development on the HS- 702 satellite. By
1994, it had determined that the business base was available to sustain
development and production of the HS- 702 satellite. In all, the firm waited
10 years for the demonstrated technology to meet the requirements. This
experience closely resembled that of Ford's voice activated control
technology because, in both cases, the new technology took 10 years to
mature enough for product readiness. Thus, the firms' approach was not to
accelerate technology development but to shorten product development by
maturing the technology first. Figure 3.1 shows the solar cell arrays
installed on the HS- 702.
Figure 3.1: Hughes Solar Cell Arrays
Hughes successfully proved solar cell array technology on a predecessor
satellite before beginning product development of the HS- 702 Source: Hughes
Space and Communications.
The Navy made trade- offs in choosing a technology for the weapon ejection
system, which is used to deploy weapons like torpedoes, of the Virginia
class attack submarine. Because of quietness, weight, and cost requirements,
the Navy preferred a new elastomeric (rubber- based) technology. However,
this technology failed endurance testing, and product
managers determined that the technology was too risky to be included in the
first product. Product managers could have declined this technology and its
attendant risk without delaying the submarine's schedule because the Navy
accepted marginal increases to the cost and weight requirements for the
system so that the proven Seawolf ejection technology could be used as a
substitute. Using proven technology on the first submarine has allowed the
Navy S& T community to continue developing the elastomeric
technology, which is to be incorporated into the new system on the fourth
production submarine. As discussed earlier, decisionmakers also had the
flexibility to wait for the nonpenetrating periscope technology to reach TRL
9 before including it in the submarine's requirements.
Problematic Cases Provided According to Army officials, the FLIR and
integrated avionics technologies Little Flexibility to required for the
Comanche helicopter were critical for providing an
Managers increased operational capability over existing Army helicopters.
The
advanced FLIR technology was needed to meet the user's requirements for
increased targeting range and for improved piloting capabilities in bad
weather and at night. It represented a quantum leap from existing
capabilities. Integrated avionics technology was expected to replace
separate radios, navigation systems, and other communication equipment on
the helicopter with a modular system that uses central processors.
These technologies were needed to meet weight and size requirements for the
aircraft as well as improve communications. Both were critical elements of a
mission equipment package that was supposed to reduce the pilot's workload
while improving capabilities. Requirements were inflexible. Thus,
requirements managers informed us they were unwilling to accept the product
manager's request to trade requirements that was prompted by his concerns
that the technologies could not advance in time to meet the program's
schedule. They believed the product manager to be too risk averse and said
they would not take no for an answer. Not only did the user consider the
technologies nontradable, they became even more confined by weight and cost
restrictions that were placed on the program. For example, a more mature
FLIR technology that could possibly meet performance requirements but also
weighed more was rejected.
The technological solutions that could meet the strict requirements were
limited. According to Army officials, the only viable option was to develop
the new technologies, which were in a very immature state, to the required
performance levels because no suitable back- up technologies existed. When
the Comanche acquisition program was launched, the FLIR and integrated
avionics technologies had a TRL 3, barely demonstrated in a laboratory. This
level placed the burden on the Comanche program manager to complete their
development during the acquisition program. The only ways for the program
manager was to slip the schedule or increase development costs. Figure 3.2
shows an early model of the integrated avionics component for the Comanche.
Figure 3. 2: Integrated Avionics for Comanche Helicopter
The Army launched the Comanche program with immature technologies, placing
the burden on the program manager to complete technology development
Source: DOD.
Similarly, the acoustic sensor technology on the BAT was critical to the
submunition's performance because it provided breakthrough improvements in
the capability for precision attack of targets at ranges of up to 500
kilometers and in most weather. There was no flexibility for the program
manager to ease requirements to substitute a more mature technology because
the Army had no existing capability to perform this mission. Thus, the
technology, which had a TRL 2 at program launch, was
the only solution for locating and acquiring targets. Its feasibility was
based on an engineering analysis in the form of studies. Key challenges for
the acoustic sensor were to reduce noise to an acceptable level, develop
microphones with sufficient range, and reduce the size of the sensor so it
would fit into the BAT delivery system. The technology development that was
necessary to have the sensor meet requirements had to be accomplished during
the schedule- driven, delivery- oriented product
development program. The development program encountered technical problems
that left the program manager with no choice but to slip the schedule and
increase the cost. By the start of the engineering and manufacturing
development phase, program officials stated that the acoustic sensor had a
TRL 5- still a high risk using the Air Force Research Laboratory's criteria.
Good Technology With the right environment as a precondition, managers on
the successful Handoff Decisions
cases benefited from disciplined technology development processes that
linked the technologies to products and provided credible information on
Depend on the Tools
the status of technologies. They also had standards that were both clear and
Authority Given to
and high for assessing readiness. Once a technology's feasibility and
Managers
usefulness were demonstrated, it was linked to a product through an early
agreement with the product developer to use it if it could be fully
developed. Ideally, as technologies approached the higher readiness levels
associated with the bridge, S& T managers and receiving product managers
agreed to more specific terms for accepting or rejecting a technology. These
agreements were early links to the product that were needed for the
technology to succeed. If a product manager was not willing to make such an
agreement, then the investment to bring the technology to higher readiness
levels might not be made. S& T managers were responsible for
ensuring that information at key junctures was sufficient and that the
technology was ready for inclusion on a product. They saw their role as to
screen and develop technologies to standards acceptable to product managers.
Product managers were responsible for ensuring that the product could be
developed and brought to market within cost and performance targets. They
saw as their role to encourage the successful
development of new technology but to decline the handoff if it did not meet
product performance, cost, and schedule requirements. When an S& T
organization disengages from a technology at a low TRL, the S& T manager
gives up much of the ability to be a gatekeeper. In the event that
unyielding requirements or other pressures force product managers to accept
technologies before they have matured, they are weakened in their
ability to safeguard the product development from technology risks. For the
cases in which technologies had problems transitioning to products,
decisionmakers were disadvantaged by the incomplete information available to
them, yet were not empowered to say no to the handoff. Their situation was
further degraded by risk assessments that embodied lower
standards for accepting undemonstrated technology readiness. In the case of
the BAT, the S& T community was bypassed altogether, as the weapon system
and its enabling technologies were proposed by a contractor and assigned
directly to a program manager. Successful Cases Benefited
All new technologies at Ford, regardless of whether they are proposed by
From Strong Gatekeepers, inside or outside sources, take essentially the
same path and gates into Disciplined Processes, and products. Initially,
technology proposals pass through a process that High Maturity Standards
prioritizes them according to customer needs. The proposals are then passed
on to the Advanced Vehicle Technology Office, an S& T organization
that determines the readiness of the proposed technology and fits it into
Ford's path of technology demonstration. Once approved, the technology
follows a structured process that includes two development phases:
concept ready and implementation ready. This process results in a smooth
transition from the technology development environment into a product, once
the technology is mature. Ford's adaptive cruise control technology went
through this process, as shown in figure 3.3.
Figure 3.3: Process for Closing the Gap Between the Readiness of Adaptive
Cruise Control Technology and Jaguar Requirements
Technology Bridge between technology
Product feasibility
and product development
Advanced vehicle technology office Jaguar vehicle team
Enabling technologies Interest expressed
Agreement signed Product launch
Model introduced pursued in the technology
by vehicle team between S& T and
base vehicle team
TRL 5 TRL 7
TRL 8 1993 1995 1996
1997 1999
According to Ford officials, technologies for adaptive cruise control
existed as separate projects in the technology base from about 1993 to 1995,
when the Jaguar vehicle team identified a strong demand for the capability.
Ford's S& T community inventoried the ongoing projects and demonstrated the
technology as a laboratory breadboard- a TRL 5. By August 1996, the
technologists had built a prototype that could
demonstrate the technology in a relevant environment- a TRL 7. This work
comprised the concept ready phase, in which the technology was taken from
concept to where its feasibility was demonstrated to potential users. At the
end of this phase, S& T representatives proved that it could work, and cost,
schedule, and performance targets were established. Also, a target product
and sponsor were identified, linking the technology to a product. The
sponsor agreed that it would accept this technology if specific cost,
quality, schedule, and performance targets were met. The medium for
this acceptance was the Deliverables Agreement Log (DEAL), which was signed
in September 1996 by Jaguar's chief engineer based on the prototype
demonstration.
Ford uses the DEAL as a tool to maintain visibility over a new technology as
it progresses through the development process and to assess its readiness
and acceptability for inclusion in a vehicle program before handing it to a
sponsor, the vehicle center, or team. The DEAL formalizes
the content of the two development phases and establishes agreements between
the technologists managing the project and those with authority to accept
the technology into a product. According to Ford officials, the DEAL is
important to this process because it is a contract between the parties that
addresses the technology's performance, cost, quality, weight,
producibility, and maintainability targets that must be met before the end
of each phase.
It has been invaluable in getting parties to agree on what is expected by
the giver and receiver of a technology during the process. Once these
targets are established, the technology moves to the
implementation ready phase. For the Jaguar, the Advanced Vehicle Technology
Office matured the technology to a high level of readiness by prototyping it
in demonstrator vehicles- a TRL 8. The technology passed the implementation
ready milestone in February 1997. At that point, the vehicle team accepted
the technology for inclusion on a Jaguar product development.
Ford used this decision- making process to develop the night vision
technology, but with a different result. Since 1991, Ford has been working
on this technology to provide a wide field of view and depth perception for
the driver at night, similar to that provided by a FLIR. By 1998, the
Advanced Vehicle Technology Office brought the technology to a TRL 8.
However, the vehicle center did not agree to include the technology on a
product because the technology did not meet the cost targets established in
the DEAL.
Other companies we visited had similar practices for supporting technology
inclusion decisions. For example, 3M takes technology from its technology
base when it believes it has a customer need. The gatekeeper
responsible for moving technology into a concept phase- analogous to TRL 3
or 4- is the S& T organization of a business unit. That business unit
monitors the technology's progress until a new product requirement is
identified and decides whether there is interest from a product center to
“pull” it. If an interest exists, it begins a feasibility phase
that refines requirements through quality functional deployment and builds
working prototypes of the new product- a stage that would be analogous to
TRL 7 or 8. This phase culminates with an agreement between the
technologists and the product developers- the receivers- as to the specific
cost and
schedule targets that must be met for the technology to be included into a
product. To help facilitate the transition, 3M establishes a product
development team that includes people from research and development,
marketing, manufacturing, and other functions that transfer with the new
technology and ensure it is integrated into the new product. 3M also has
high standards for measuring the readiness of a technology before the
product developer accepts it. For example, 3M officials told us that they
are developing a fuel cell technology for which they have built 15
prototypes for testing purposes- a TRL 7 or higher. However, because the
technology has not yet met all of the cost, schedule, and performance
targets for product development, they have not allowed it to be included on
a new product, despite demand from the marketplace. Among the DOD cases, the
process followed and the roles played on the
AAAV program had several features that enabled good technology inclusion
decisions. For almost 3 decades, the Marine Corps has stated a need for an
amphibious vehicle with far greater capabilities than the current vehicle.
Specifically, the requirement to achieve a speed of 20 to 25 knots in the
open ocean made advances in propulsion technology key enablers for the AAAV
program. For a vehicle of the planned size and weight of the AAAV, this
requirement meant achieving 2,700 horsepower with a relatively
compact engine that must operate on land and in water. The Corps had been
exploring propulsion technologies for such a vehicle in its technology base
for many years. Despite this, the Office of Naval Research, an S& T
organization, assessed the propulsion technology and advised that it was not
mature enough to warrant inclusion on a program. Based on this assessment,
Marine Corps and Navy decisionmakers delayed program launch from 1991 to
1995, until the technology could be brought to higher readiness levels.
Figure 3.4 illustrates the process used to transition this technology.
Figure 3.4: Process for Closing the Gap Between the Readiness of Propulsion
Technologies and AAAV Requirements
Technology Bridge between
Product feasibility
technology and program development Office of Naval Research
AAAV program office Engine proof
Agreements reached between Program launched in
Prototype of concept
S& T and program office program definition engine
demonstrated phase
demonstrated TRL 3
TRL 6 TRL 7
1988 1991 1995 1999
The S& T community and the product managers agreed on what had to be done
before the program could be launched. The S& T community then took the lead
in maturing the engine to a TRL 6- a level the Air Force Research Laboratory
considers acceptable for starting the program definition and risk reduction
phase. Thus, the assessment by the Office of Naval Research provided both
the information and the criteria that enabled decisionmakers to say no to
launching the program given the low readiness of the propulsion technology.
This was coupled with the flexibility to wait for the technology to mature
and the decision to give an S& T organization
responsibility for managing the bridge to product readiness. Figure 3.5
shows the AAAV.
Figure 3.5: AAAV
The Marine Corps and Navy delayed program launch by 4 years to develop key
technologies to a higher readiness level
Source: DOD.
Even with an urgent need for the AAAV, the Marine Corps remained disciplined
in its development approach, allowing the technology to mature to the level
of the requirement. Two years before program launch, a Navy S& T
organization demonstrated the technology in a full- scale prototype engine.
By program launch in 1995, the required 2,700 horsepower was demonstrated by
a near prototype engine- a TRL 6. The remaining risk was limited to marginal
weight and size reductions, although the demonstrator engine could be used
as a backup if the size and weight reductions could not be obtained. In
early 1999, the AAAV program office demonstrated a
prototype engine at 2,700 horsepower that met size and weight requirements-
a TRL 7.
Technology Handoffs Were In the BAT program, neither the S& T community nor
the product manager
Compromised When had the opportunity to act as a gatekeeper between product
requirements Managers Had Limited and the maturity of enabling technologies.
All of the technologies for the
Information and Authority BAT came to the program after the contractor, in
1985, had proposed a
weapon concept for carrying out unmanned, deep strike missions to attack
enemy armored vehicles. Army leadership accepted the concept and drafted
requirements for the BAT, and the acquisition program was launched after the
proposal was accepted. Thus, the technology for the
weapon came directly from the contractor's technology base into the
acquisition program, with little or no review by the Army's S& T
organization. The process, information, and standards that were critical to
successful technology inclusion decisions in other cases were not employed
on the BAT. The process followed is shown in figure 3.6.
Figure 3.6: Assimilation of New Technology Into the BAT Program
Technology maturation Technology concept and product development
Contractor technology base BAT program office
Program launched in Start engineering,
program definition phase manufacturing and
development phase TRL 2- 3
TRL 5 1985
1991 The program office accepted the acoustic sensor, infrared seeker, and
navigation technologies included on the BAT program. In retrospect, the
levels of demonstration at the time posed high risks to the product
development because the acoustic sensor technology had a TRL of 2 and
the infrared seeker and navigation technologies had TRLs of 3. Program
officials stated that a significant amount of technology development was
required during product development due to the lack of visibility over
technology readiness before program launch. As a result, the development
program's cost and schedule significantly increased over original estimates.
An interesting sidelight to the BAT experience concerns the inertial
measurement unit, a navigation component of the submunition. When the
contractor first proposed the BAT concept, the design included a mature
inertial measurement unit in production on other systems. However, after the
program was launched, the contractor substituted a new quartz rate
technology. At the request of the BAT program manager, the Army's Missile
Research and Development Engineering Center, an S& T organization, assessed
the maturity of the quartz rate technology. The Center concluded that the
new technology had not demonstrated a high enough level of readiness and
recommended that a more proven existing technology be used in the program.
Eventually, the new technology was dropped, and an existing technology that
was at a higher readiness level was used.
We observed additional cases in which decisionmakers relied on comparatively
low standards for including technologies. The Army assessed the FLIR,
integrated avionics, and helmet mounted display technologies as having
moderate risk when they were included in the Comanche program. Army
officials stated that they required only the
existence of an ongoing S& T technology project as acceptable, as long as
the technology was projected to be ready by the engineering and
manufacturing development phase. According to program officials,
demonstrated maturity was considered but not required; proof that the
projects were progressing as scheduled was enough. These technologies,
however, had TRLs of 3 at the time of launch- a high risk for the program
definition and risk reduction phase. This risk assessment is more consistent
with the actual experience of the technologies' maturation in the
program. The standards used for accepting the laser technology into the ABL
program also appeared low when compared with the standards used on the more
successful cases. While the Air Force had established demonstration
standards for the laser to meet prior to program launch, these standards
were met if scale models of the laser technology in a laboratory
demonstrated they had the potential to produce the energy needed for an
operational system. This level of technical demonstration equated to a TRL
of 4, representing a high risk for inclusion into an acquisition program.
Impediments to Adopting Best Practices for Technology Inclusion in DOD Are
Chapt er 4
Surmountable Although product developments- commercial or defense- fare
better when key technologies are matured before they are included in the
product design, the more traditional approach within DOD is to mature
technology during a product's development. Rational explanations are behind
this tradition. S& T organizations, operating within fixed budget levels,
are not necessarily accustomed or equipped to manage the bridge between
technology feasibility and product readiness. Programs are more able to
command the large budgets necessary for reaching higher levels of technology
readiness than S& T projects. Also, pressures are exerted on new programs to
offer unique performance and acceptable cost and schedule projections, which
encourage premature acceptance of unproven technologies. The Under Secretary
of Defense for Acquisition and Technology not only
supports shorter cycle times and a more aggressive pursuit of technology
outside of programs, but also use of commercial best practices to get these
results. DOD has several initiatives underway that could make conditions
more favorable for S& T organizations to mature a technology further before
it is included in a product development. One Army project calls for an S& T
organization to manage all technology maturation and integration
tasks for a new combat vehicle up to the engineering and manufacturing
development phase. Other initiatives may make the S& T community a more
integral participant in matching user requirements with technology and tying
S& T projects more closely to product development paths. Whether these
efforts are effective and can be applied on a broader scale remains to
be seen. Several Factors Make
Budgetary, organizational, and other factors within DOD make it difficult to
It Difficult to Mature bring technologies to high readiness levels before
being included in weapon systems. These factors encourage S& T organizations
to disengage Technologies Before
from technology development too soon and weapon system program They Are
Included on managers to accept immature technology. Factors other than these
Weapon Systems
encourage leading commercial firms to keep technology development out of the
product developers' hands and in those of S& T organizations. The
differences in these factors and in the management of technology development
stem from differences in what helps commercial and DOD programs to succeed.
They do not stem from capabilities commercial firms possess that DOD does
not.
Budget and Organizational Budget realities within DOD- the fact that weapon
system programs Factors attract higher levels of funding than S& T projects-
make these programs a more advantageous setting for funding technology
development to the
higher readiness levels. As a practical matter, it is often necessary to
move immature technology to a weapon system program to get needed funds and
management support for maturation. Normally, DOD S& T organizations do not
see their role as going beyond demonstrating the feasibility of a technology
for generic- versus product specific- application (a TRL 5). However, as
seen in several of the cases we reviewed, even this level often
is not reached before a product development organization takes over. The S&
T organizations that helped to bridge the gap from technology feasibility to
product readiness on the more successful cases had gone beyond their typical
role. One of the reasons that S& T organizations disengage relatively early
is that S& T work is traditionally funded as a percentage of the overall DOD
research and development budget. S& T organizations receive about $8 billion
annually, or about 20 percent, of DOD's research and development budget.
This money funds several thousand projects, providing less than $1 million
per project on average. As a result, a project needing $100 million or more
to mature technology to higher readiness
levels than normal- not unreasonable sums- would command a fairly large
share of an S& T organization's budget, thereby reducing funds available for
other projects. Under the current scenario, the remaining 80 percent of
DOD's research and development funds, approximately $30 billion, is spread
out over a much smaller number of specific weapon programs. A typical weapon
system program can receive several hundred million dollars annually and
occasionally over $1 billion to fund development. A major program, such as
the F- 22, can command $15 billion
or more in total for product development, receiving sometimes more than $2
billion in a year.
Events on the Air Force's ABL program illustrate these realities.
Originally, the Air Force had planned the ABL as a technology development
project to be managed to high readiness levels by an S& T organization. The
project was started in 1992 as an advanced technology transition
demonstration to design, fabricate, and test a single demonstrator weapon
system and was to take 8 years to complete. The pacing technologies, the
laser and the beam
control, were to be matured to a high level- equivalent to TRL 6 or 7-
before being included in a product development program. Requirements had not
been fixed. In other words, the planned approach resembled what
we have described as the more successful cases in our review. Figure 4.1
shows the ABL.
Figure 4. 1: Airborne Laser
The Air Force used relatively low readiness level standards to include a key
technology into ABL Source: DOD.
In 1996, the Air Force abandoned this approach and decided to launch ABL as
a weapon system development program, not because technologies were
sufficiently mature but because of funding and sponsorship concerns. At
this time, the two key technologies were at TRLs 3 and 4. According to the
retired manager of the S& T project, a product development program was
deemed necessary to make the technology development effort appear real to
the users and not a scientific curiosity. Within the Air Force, the
perceived lack of support by the users placed the project in a constant
state of funding jeopardy. This perception was important because the S& T
project was costly, with a total estimated cost of $800 million, with some
annual funding requirements approaching $200 million. The annual funding
requirements would encompass a large percentage of the Air Force's S& T
budget unless additional funds were made available from weapon system
budgets or elsewhere. By transitioning to a weapon system program linked to
user requirements, the ABL was more likely to get these funding levels.
This approach was successful- the program won user support and the desired
funding. However, sacrifices were made in technology development. According
to the former project manager, the new program focused less on the elemental
technology hurdles and more on meeting all
user requirements. More expensive demonstrations were necessary to meet
these broader requirements without necessarily doing more to demonstrate
basic technology readiness. It became a more traditional program with
technology and product development proceeding at the same
time, with attendant higher risks. In March 1999, we reported that, while
the ABL has made progress in developing these technologies, it still faced
technical challenges. 1 Other Incentives Pressures exerted on weapon system
programs can make it advantageous
to include in their design immature technologies that offer significant
performance gains. One traditional source has been the perceived threat.
Users can demand performance improvements that necessitate the application
of unproven technologies, particularly when a fielding date is mandated, to
stay ahead of the threat. Another source is technologists,
whether from S& T organizations or contractors, who see a new weapon system
as an opportunity to apply a new technology. Also, the competition for funds
can encourage performance features- and requisite technologies- that
distinguish the new weapon system from competitors. The F- 22 was justified
as being faster, stealthier, and more lethal than other
fighters, such as the F- 15 and F- 117, were. As a result, the F- 22 is
being designed with several advanced technologies, including a very
sophisticated suite of avionics that is critical to its performance features
that distinguish it from the other fighters. However, at the time the F- 22
program was launched in 1986, the avionics technologies were immature;
they have since been a source of problems on the program. We recently
reported that the development of the F- 22's integrated avionics systems
continues to experience cost growth and schedule delays, more than
12 years into the program. 2 A different set of incentives causes leading
commercial firms to make their S& T organizations responsible for maturing
technologies to higher readiness levels. Commercial firms are aware of the
risks associated with the high investment that product development requires.
They have a strong incentive in the realization that if a product is late,
costs more, or performs 1 Defense Acquisitions: DOD Efforts to Develop Laser
Weapons for Theater Defense (GAO/ NSIAD- 99- 50, Mar. 31, 1999). 2 F- 22
Aircraft: Issues in Achieving Engineering and Manufacturing Development
Costs (GAO/ NSIAD- 99- 55, Mar. 15, 1999).
less than expected, the customer could walk away from the product and the
investment would be lost. Minimizing the possibility of technology being the
cause of such problems is thus a top priority. Having their S& T
organizations reduce those risks is essential to putting product
developments in the best position to succeed. DOD does not have the same
incentives. DOD programs are not penalized if a product is late, costs more,
or performs less than expected, because the customer does not walk away.
Services Encouraged Over the past several years, DOD has encouraged the
services to use best to Use Best Practices
practices to streamline the current process for acquiring new weapon systems
in order to make them faster, cheaper, and better. Shorter acquisition cycle
times are seen as critical to making the best use of advances in technology.
To encourage change, DOD has set a goal to reduce
the average acquisition cycle time for all program starts in fiscal year
1999 and beyond by 50 percent over historical averages. DOD has several
initiatives to improve its technology development process and to move
technologies to the warfighter faster and less expensively than the
traditional means. The initiatives also attempt to put the organizations and
funding in place to bring technologies to higher readiness levels before
they
are included in programs. These initiatives- defense technology objectives,
advanced technology demonstrations, and advanced concept technology
demonstrations- call for S& T organizations to play a bigger role in
managing technologies closer to the point of product readiness, matching
requirements to technology projects, and making better use of demonstration
standards. Defense Technology
Defense technology objectives (DTO) are used to bring more discipline to
Objectives S& T projects and to link them more closely with weapon system
development programs. A DTO typically involves a particular technology
advance, such as high temperature materials for turbine engines and high
fidelity infrared sensors. It can also group several technologies into a
larger demonstration. Each DTO identifies a specific technology advancement
that will be developed or demonstrated, the anticipated date of the
technology availability, the ultimate customer, and the specific benefits
resulting from the technology. It places a corporate attention and
commitment on the technology project by having the technologists, product
developer, and customer involved in the project.
According to DOD, the focus of its S& T investment is enhanced and guided
through DTOs. Each DTO must go through a formal review and approval
process within DOD and must be directly related to advancing the operational
concepts depicted in DOD's “Joint Vision 2010” planning
document. According to DOD officials, those requirements have helped to
eliminate instances in which technologists work on projects of particular
interest to them, but with no military application, because the projects
should be linked to a specific warfighter need. For fiscal year 1999, DOD
established approximately 350 DTOs, which accounted for $3 billion, or less
than 50 percent, of the funds DOD had allocated to S& T projects. The
remaining funds were allocated to projects under the jurisdiction of each
military service or other defense agencies and did not go through the same
review and approval process.
Advanced Technology Advanced technology demonstrations (ATD) are intended to
more rapidly
Demonstrations evolve and demonstrate new technologies so they can be
incorporated into
a product, if warranted. An ATD has four characteristics that distinguish it
from a conventional S& T project. They (1) require large- scale resources;
(2) involve the user; (3) use specific cost, schedule, and performance
metrics; and (4) identify a target product for inclusion. An ATD is managed
by an S& T organization and should conclude with an operational
demonstration of the potential capabilities of the technology, equating to a
TRL 5 or 6. The original approach to the ABL was essentially an ATD
approach. Most ATDs use laboratory hardware to demonstrate the potential
capability of nonproduct specific technologies and not prototype hardware.
If the technology is determined to be feasible and provides some military
use, then it may proceed to the program definition and risk
reduction phase of an acquisition program. From that point, the product
developer completes the technology development for a specific product.
Advanced Concept
In 1994, DOD initiated Advanced Concept Technology Demonstrations Technology
Demonstrations
(ACTD) to help expedite the transition of mature technologies from the
developers to the warfighters. ACTDs are intended to help the DOD
acquisition process adapt to budget constraints while developing
technology more rapidly. The purpose of an ACTD is to assess the military
use of a capability, such as a weapon, comprised of mature technologies.
Typically, ACTDs last 2 to 4 years and consist of building and demonstrating
a prototype to provide a warfighter the opportunity to assess a prototype's
capability in realistic operational scenarios. From this demonstration, the
warfighter can refine operational requirements, develop
an initial concept of operation, and determine the military use of the
technology before it proceeds to the product development process.
According to DOD, ACTDs, which are managed by S& T organizations, will be a
key mechanism to ensure technology development is separated from product
development. In related work on unmanned air vehicles, we found that ACTDs
provided decisionmakers credible data that they used to terminate efforts or
transition the demonstrator to an acquisition program. In these cases, ACTDs
put decisionmakers in a better position to be
gatekeepers. However, we have reported that the ACTD program needs to be
improved. 3 We found that DOD's process for selecting program candidates
does not include adequate criteria for assessing the maturity of proposed
technology and has resulted in the approval of projects that included
immature technologies. We found that the use of specific criteria for
determining maturity was a best practice in the most successful
technology development cases we examined. Two Unique DOD
Two DOD projects are using S& T organizations to manage technology Projects
May Provide development to higher readiness levels. One, the Army's Future
Scout and
Cavalry System, is using a modified ATD to mature technologies and make
Lessons on How to
performance trade- offs in the more flexible environment provided by Enable
S& T S& T. The other is a joint government and industry program, which the
Air Organizations to
Force Research Laboratory is managing to reduce the risks associated with
new jet engine technologies. These projects may provide insights on how
Manage Technology S& T organizations could routinely play a bigger role in
maturing
Further technologies enough for safe inclusion on weapon system programs.
They
may also clarify the concern that playing a bigger role in technology
maturation could cause S& T organizations to do less basic research and
technology development.
Future Scout and Cavalry In fiscal year 1997, the Army began piloting a
variation of an ATD that is System
designed to help bridge the gap between technology development and product
development by expanding the S& T community's role in managing technologies
further into the development cycle. The Army's initiative, called Fast
Track, is intended to reduce cost and cycle time by bypassing the program
definition and risk reduction phase of the DOD acquisition
process. The Army is testing this concept with its Future Scout and Cavalry
System project. In this project, the Army will design, develop, and build a
demonstrator vehicle to show the technical feasibility of the weapon. All of
3 Defense Acquisition: Advanced Concept Technology Demonstration Program Can
Be Improved (GAO/ NSIAD- 99- 4, Oct. 15, 1998).
these tasks will be done under the management of the Army's S& T
organization. The Army believes that a more extensive S& T project will make
the program definition phase unnecessary and estimates that this concept
will reduce the development process by as much as 4 years and
save about $400 million. We did not review the Future Scout project in terms
of its affordability, feasibility, or any impacts it may have on the Army's
S& T budget. Figure 4.2 compares the Fast Track development process with the
traditional approach.
Figure 4.2: Comparison of Traditional Technology Development Process With
the Army's Fast Track Approach
Technology feasibility Technology maturation and product development
Traditional S& T organization
Weapon system program office approach 9 years
Program launched in Production
program definition phase TRL 3- 5
Technology feasibility Bridge built to Technology maturation
increase technology and product development maturity
Future Scout Army S& T organization Weapon system program office and Cavalry
System 4.5 years
Program office Program launch
Production staff joins S& T
in engineering, project office
manufacturing and development phase.
TRL 6
While we do not necessarily agree that the first phase of the acquisition
cycle can be omitted, so far the Future Scout project is emulating
technology development practices like those we observed in the successful
cases. First, it has established demonstration criteria that must be met
before the technology enters product development. Second, it has also
established forums that involve key players on the technology path to keep
them informed of the technology's development progress. For example, the
acquisition program manager will be integrated into the development project
during the final 1.5 years of the S& T program. This should provide a good
link between the technology development and product development, allowing
the program manager to fully understand the technology before product
development begins. Finally, by allowing an S& T organization the
flexibility to manage technologies further into the development cycle, Army
officials believe they will be able to make trade- offs among cost,
schedule, and performance requirements before program launch, without
raising concerns about the state of the project or breaching baselines that
had been set without enough knowledge.
While this concept comes closer to the most successful technology
development cases we reviewed, it still embodies greater technical risk. The
Army expects to demonstrate some performance capabilities of the vehicle
before the product development phase begins. However, the demonstrator
vehicle will only be about 75 percent of a complete prototype, which means
some key technologies will not be demonstrated to high readiness levels
before that phase begins. Nonetheless, the project manager equated the
expected overall technical maturity of the vehicle at transition to a TRL 6.
The Army considers this a medium or acceptable
level of risk, and it is willing to enter product development with some
immature technologies. If, however, product development begins at
engineering and manufacturing development, this risk could be assessed as
high, based on TRLs. Figure 4.3 shows an artist's concept of the Future
Scout and Cavalry System.
Figure 4. 3: Future Scout and Cavalry System
An Army S& T organization is maturing technologies and proposing performance
trade- offs for the Future Scout and Cavalry System before program launch
Source: DOD.
Integrated High The Integrated High Performance Turbine Engine Technology
program- a Performance Turbine joint government and industry effort- is
focused on developing Engine Technology Program
technologies for more affordable and higher performance turbine engines for
both missiles and aircraft. It is a technology validation program and is
managed by an S& T organization to perform demonstrations of various engine
technologies to higher readiness levels than most S& T projects.
After the demonstrations, the technologies enter a product development
program. The program takes the technology through a series of tests that
range from individual component tests to full- scale engine demonstrations.
The program has established strong links with the acquisition programs for
which the technologies are intended. For example, Air Force Research
Laboratory officials informed us that they established formal technology
transition plans with the F- 22 and Joint Strike Fighter programs that
document agreements on what technology development activities will be
performed to support the programs. Representatives from each program office
are invited to all technology demonstrations and are kept informed about
demonstrated progress.
As part of the program, the Air Force developed a set of standards to assess
the readiness levels of technologies similar to NASA's TRLs. According to
the Air Force, the S& T organization uses these standards to determine when
the project has been completed. These standards were the first application
of readiness levels by the Air Force Research Laboratory. There are five
technology readiness levels ranging from component- level tests in a
laboratory to the highest level involving actual flight tests of engines.
The program typically does not take technologies to the highest readiness
level (flight test) because of the high cost. The program stops when it has
been determined the technology is well defined within acceptable boundaries
and a good correlation exists between test results and engineering
predictions. This readiness level would translate to a TRL of 5 or 6, as
used in this report. The final step of the technology development is left to
the product developer who determines if the technology can be packaged and
integrated into the final product.
Chapt er 5
Conclusions and Recommendations Conclusions Clearly, DOD's continued
advancement of new technologies is essential to the continued superiority of
its weaponry. The leading edge military capabilities the United States
possesses today, such as stealth aircraft, precision munitions, and
intelligence- gathering satellites, bear witness to the effects of such
technical advances. At the same time, the incorporation of advanced
technologies before they are mature has been a major source of cost
increases, schedule delays, and performance problems on weapon systems. As
DOD contemplates increasing its annual investment in new weapons to $60
billion, the expectations on program managers are great: they must develop
and field weapons of superior capability more quickly and less expensively
than in the past. The way advanced technologies are matured and included in
weapon systems will play a central role in meeting these expectations.
Although different ways to better assimilate new technologies into weapons
are legitimate topics for debate, that it has to be done better is not.
The leading commercial firms' practices have produced results that resemble
those sought by DOD: more technically advanced, higher quality products,
developed in significantly less time, and less expensively than their
predecessors. Managing the development of advanced technology differently--
and separately-- from the development of a product has been key to these
results. The firms insist that advanced technology reach a high
level of maturity, the point at which the knowledge about that technology is
essentially complete, before allowing it into a product development. By
separating the two, the firms lessen the product manager's burden and place
that person in a better position to succeed in delivering the product. These
practices may not necessarily accelerate the pace at which technology
matures. In fact, several of the commercial technologies we reviewed took 10
years or more to get to market. The clear beneficiaries of the practices are
the product developments, for which the investments are much larger, and
time translates into significantly more resources than in a technology
project. Adapting these practices on its weapon system programs can help DOD
to reduce costs and the time from product launch to fielding, and use
technology advances as they become available more frequently.
Separating technology development from product development calls for a new
approach to managing technology development. Two conditions are essential to
such an approach. First, the right environment for maturing technologies
must exist. A practice that is instrumental in providing this environment is
maturing technology to achieve product readiness before it
is constrained by the rules of an acquisition program. In the successful DOD
cases we reviewed, this environment was provided by S& T organizations or a
team of S& T and product developers who managed technologies to high
readiness levels before they were included in an acquisition program. These
organizations provided an environment more conducive to the ups and downs
normally associated with the discovery process. A corollary practice is
agreeing on what level of knowledge is needed about a new technology before
it is considered for inclusion in a
product design. When that knowledge level does not exist, the flexibility
for S& T organizations and product managers to either take the time to
mature the technology or trade off product requirements until they can be
met with mature technology is essential. It is a rare program that can
proceed with a
gap between product requirements and maturity of key technologies and still
be delivered on time and within costs. Second, S& T and product managers
must be provided with the disciplined processes, information, standards, and
authority to make good handoffs of technology to product. Prepared with the
tools and authority to make sound handoff decisions, both S& T and product
managers can function as gatekeepers to safeguard the product development
from undue technology risks.
Leading commercial firms have adopted this approach as a matter of necessity
and have used the organizations, tools, and other practices to foster
technology development and improve the outcomes of product developments. The
high stakes stemming from the large investment required for a new product
and the risks if the product does not meet customer needs reinforce this
approach in leading commercial firms. The
DOD cases that followed a similar approach were realizing better program
outcomes, at least in the sense that the programs avoided key technology
development problems. Yet, these cases are not the norm for DOD programs for
several reasons.
More typically, the commitment to develop and produce a weapon system is
made before a match between technology and weapon system requirements
exists.
DOD programs operate under different conditions that make it more difficult-
and less rewarding- to separate technology development from product
development. Budget realities make it more difficult for S& T organizations
to carry technologies to the high readiness levels needed to meet product
requirements; such resources are more available within product developments.
The pressures to show the unequalled performance necessary to win funding
encourage including promising, but immature, technologies in weapon system
designs. It will take procedural, organizational, and cultural changes
within DOD's acquisition process to foster an environment in which (1)
technologies can be successfully matured outside the purview of weapon
system programs, (2) programs can be relieved of the pressures to include
immature technologies and the unrealistic expectations that the technologies
will conform to tight cost and schedule projections, and (3) technology
advances will not stall due to inadequate funding or lack of identification
with a product in the later, more expensive stages of demonstration.
Experience has shown that such an approach can work within DOD on individual
cases. DARPA played a primary role in managing the transition of the
nonpenetrating photonics mast technology to the Virginia class attack
submarine. The Integrated High Performance Turbine Engine Technology program
has carried advanced jet engine technologies to TRLs of between 5 and 6 for
successful inclusion into programs. In the Future Scout program, an Army S&
T organization, augmented by product development staff, is managing an ATD
to lower the risk of key
technologies before a product development program is launched. However, it
remains to be seen whether the Army will be successful in using large and
expensive S& T projects, such as the Future Scout program, without affecting
other Army S& T projects. A challenge for DOD will be whether the lessons
learned from these individual cases offer an approach that has
DOD- wide application. Meeting this challenge is essential to fielding
technologically superior weapons more quickly and within predicted costs.
Recommendations We have previously recommended that DOD separate technology
development from weapon system programs. That recommendation was made
without prejudice toward the necessity of technology development
but rather with the intent that programs could be better managed if such
development was conducted outside of a program manager's purview. Similarly,
the recommendations that follow are made without prejudice toward- or the
intention of compromising- the basic research and other activities that S& T
organizations perform. We recognize that implementation of these
recommendations will have organizational, funding, and process implications
and will require the cooperation of the
Congress.
To help ensure that new technologies are vigorously pursued and successfully
moved into weapon system programs, we recommend that the Secretary of
Defense adopt a disciplined and knowledge- based method for
assessing technology maturity, such as TRLs, DOD- wide. This practice should
employ standards for assessing risks of handoff to program managers that are
based on a technology's level of demonstration and its criticality to
meeting the weapon system's requirements.
With these tools in hand, we recommend that the Secretary (1) establish the
place at which a match is achieved between key technologies and weapon
system requirements as the proper time for committing to the cost, schedule,
and performance baseline for developing and producing that weapon system and
(2) require that key technologies reach a high maturity level- analogous to
TRL 7- before making that commitment. This would approximate the launch
point for product development as practiced by leading commercial firms. We
recommend that the Secretary find ways to ensure that the managers
responsible for maturing the technologies and designing weapon systems
before product development are provided the more flexible environment that
is suitable for the discovery of knowledge, as distinct from the delivery of
a product. Providing more flexibility will require the cooperation of
requirements managers and resource managers so that rigid requirements or
the threat of jeopardizing the funding planned to start product development
will not put pressure on program managers to accept
immature technologies. Such an environment may not be feasible if the
program definition and risk reduction phase remains the effective launch
point for an entire weapon system program.
An implication of these recommendations is that S& T organizations will have
to play a greater role in maturing technologies to higher levels and should
be funded accordingly. Therefore, we recommend that the Secretary of Defense
evaluate the different ways S& T organizations can play a greater role in
helping technologies reach high levels of maturity before product
development begins. For example, given that a technology has sufficient
potential for application to a weapon system, at a minimum, an S& T
organization should be responsible for taking a technology to TRL 6 before
it is handed off to a program office at the program definition and risk
reduction phase. During this phase, the program manager would be responsible
for maturing the technology to TRL 7 before it is included in an engineering
and manufacturing development program. In a situation where a single,
design- pacing technology is to be developed for a known
application- like the nonpenetrating periscope- an S& T organization should
be required to mature that technology to TRL 7 before it is turned over to a
product development manager. S& T organizations could play a similar role
when a significant new technology is being prepared for
insertion into an existing weapon system. Finally, when multiple new
technologies are to be merged to create a weapon system, S& T organizations
should be required to bring key technologies to TRL 6 and then become part
of a hybrid organization with product developers to integrate the
technologies and bring them to TRL 7 before handing full responsibility to a
product development manager.
To help guard against the possibility that the more basic research and
technology development activities would be compromised by having S& T
organizations routinely take key technologies to TRL 6 or higher, we
recommend that the Secretary extract lessons from the nonpenetrating
periscope, the AAAV, and the Army's Future Scout programs, and other ATD and
ACTD programs. Specifically, the Secretary should assess whether the
resources needed to enable S& T organizations to play a leading role in
the development of technologies and, in some cases, preliminary system
design, detracted from or displaced more basic research and technology
development programs. Finally, we recommend that the Secretary empower
managers of product development programs to refuse to accept key
technologies with low levels of demonstrated maturity. The Secretary can
encourage this behavior through supportive decisions on individual programs,
such as by denying proposals to defer the development of key technologies
and by favoring proposals to lengthen schedules or lessen requirements to
reduce technological risk early.
Agency Comments and DOD generally concurred with a draft of this report and
its
Our Evaluation recommendations, noting that the traditional path to new
weapon system
development is no longer affordable or necessary (see app. I). DOD stated
that it has embarked upon a “Revolution in Business Affairs”
that will enable new technologies to be developed more efficiently and
effectively. It believes that the first steps in this direction have already
been taken but agrees that more progress needs to be made. DOD agreed that
TRLs are
necessary in assisting decisionmakers in deciding on when and where to
insert new technologies into weapon system programs and that weapon system
managers should ensure that technology is matured to a TRL 7 before
insertion occurs. DOD concurred that S& T organizations should be
involved in maturing technologies to high levels, such as TRL 6, before
transitioning to the engineering and manufacturing development phase and
agreed to assess the impact of this involvement on other S& T resources. We
note that the best practice is to mature technology to at least a TRL 7
before starting the engineering and manufacturing development phase, whether
the technology is managed by an S& T organization, a weapon
system program manager, or a hybrid of the two organizations. DOD noted that
while TRLs are important and necessary, the increasing projected life for
new weapon systems, total ownership costs, and urgency based upon threat
assessments are also important considerations for system development
decisions. We agree and note that our recommendations are not intended to
cover all aspects of weapon system development decisions or to suggest that
technology maturity is the only factor in such decisions. Rather, the
recommendations are in keeping with the purpose of the report, “to
determine whether best practices offer methods to improve the way DOD
matures new technology so that it can be assimilated into weapon system
programs with less disruption.” We believe that a knowledge- based
approach to maturing technology, such as TRLs,
can benefit other considerations as well. For example, decisions on what
technologies to include in a weapon system and when to include them can have
a significant bearing on its total ownership costs. DOD stated that there
should be an established point for the transition of technologies and that
it plans to supplement its milestone review process with additional guidance
in the next revisions to DOD Directive 5000.2R. It also stated that its
policy on the evolutionary approach to weapon acquisitions should be
developed in consonance with the technology transition strategy. We cannot
comment on the revisions to the directive or
the evolutionary acquisition policy because they have yet to be published.
However, under the current milestone review process, the pressures placed on
a program during the program definition and risk reduction phase- when much
technology development occurs- can operate against the flexibility and
judgments that are needed to mature technologies. If the revisions to the
directive supplement the current milestones without relieving the pressures
brought to bear on programs as they are launched in the program definition
and risk reduction phase, it will remain difficult to discourage the
acceptance of immature technologies in the design of new
weapon systems. To relieve these pressures, we encourage DOD, as it develops
the directive and the evolutionary acquisition policy, to separate
technology development from product development and to redefine the launch
point for a program as the point at which enough knowledge has
been gained to ensure that a match is reached between the maturity of key
technologies and weapon system requirements.
DOD also stated that program managers already have the ability to reject
inappropriately mature technologies, and to the extent technology immaturity
affects acquisition baselines, to advise acquisition executives of feasible
alternatives. We did not find this to be the case in our review. Rather, we
found that the program managers' ability to reject immature technologies is
hampered by (1) untradable requirements that force acceptance of
technologies despite their immaturity and (2) reliance on tools for judging
technology maturity that fail to alert the managers of the high risks that
would prompt such a rejection. As noted in the report, once a weapon system
program begins, the environment becomes inflexible and deviations to program
baselines can attract unwanted attention. This reality limits the program
managers' ability to reject immature technologies.
Technology Readiness Levels and Their Appendi I x Definitions Technology
readiness level Description
1. Basic principles observed and reported. Lowest level of technology
readiness. Scientific research begins to be translated into applied research
and development. Examples might include paper studies of a technology's
basic properties.
2. Technology concept and/ or application Invention begins. Once basic
principles are observed, practical applications can be formulated. invented.
The application is speculative and there is no proof or detailed analysis to
support the assumption. Examples are still limited to paper studies. 3.
Analytical and experimental critical function Active research and
development is initiated. This includes analytical studies and and/ or
characteristic proof of concept. laboratory studies to physically validate
analytical predictions of separate elements of the technology. Examples
include components that are not yet integrated or
representative. 4. Component and/ or breadboard validation in Basic
technological components are integrated to establish that the pieces will
work laboratory environment. together. This is relatively “low
fidelity” compared to the eventual system. Examples include
integration of “ad hoc” hardware in a laboratory. 5. Component
and/ or breadboard validation in Fidelity of breadboard technology increases
significantly. The basic technological relevant environment. components are
integrated with reasonably realistic supporting elements so that the
technology can be tested in a simulated environment. Examples include
“high fidelity” laboratory integration of components. 6. System/
subsystem model or prototype Representative model or prototype system, which
is well beyond the breadboard demonstration in a relevant environment.
tested for TRL 5, is tested in a relevant environment. Represents a major
step up in a technology's demonstrated readiness. Examples include testing a
prototype in a high fidelity laboratory environment or in simulated
operational environment.
7. System prototype demonstration in an Prototype near or at planned
operational system. Represents a major step up from operational environment.
TRL 6, requiring the demonstration of an actual system prototype in an
operational environment, such as in an aircraft, vehicle or space. Examples
include testing the prototype in a test bed aircraft.
8. Actual system completed and “flight qualified” Technology has
been proven to work in its final form and under expected conditions. through
test and demonstration. In almost all cases, this TRL represents the end of
true system development. Examples include developmental test and evaluation
of the system in its intended weapon system to determine if it meets design
specifications. 9. Actual system “flight proven” through Actual
application of the technology in its final form and under mission
conditions, successful mission operations. such as those encountered in
operational test and evaluation. In almost all cases, this is the end of the
last “bug fixing” aspects of true system development. Examples
include using the system under operational mission conditions.
Appendi I I x Comments From the Department of Defense
Now on pp. 7 and 63- 64. Now on pp. 7 and 63- 64. Now on pp. 7 and 63- 64.
Now on pp. 7 and 63- 64.
Now on pp. 7 and 65. Now on pp. 7 and 65.
Appendi I I I x GAO Contacts and Staff Acknowledgments GAO Contacts Louis
Rodrigues, (202) 512- 4841 Paul Francis, (202) 512- 2811 Acknowledgments In
addition to those named above, Michael Sullivan, Jeffrey Hunter,
Matthew Lea, Maria Santos, Rae Ann Sapp, and Katrina Taylor made key
contributions to this report.
Related GAO Products Best Practices: Commercial Quality Assurance Practices
Offer Improvements for DOD (GAO/ NSIAD- 96- 162, Aug. 26, 1996).
Major Acquisitions: Significant Changes Underway in DOD's Earned Value
Management Process (GAO/ NSIAD- 97- 108, May 5, 1997).
Best Practices: Successful Application to Weapon Acquisitions Requires
Changes in DOD's Environment (GAO/ NSIAD- 98- 56, Feb. 24, 1998).
Best Practices: DOD Can Help Suppliers Contribute More to Weapon System
Programs (GAO/ NSIAD- 98- 87, Mar. 17, 1998).
Defense Acquisition: Improved Program Outcomes Are Possible (GAO/ T- NSIAD-
98- 123, Mar. 18, 1998).
Defense Acquisitions: Best Commercial Practices Can Improve Program Outcomes
(GAO/ T- NSIAD- 99- 116, Mar. 17, 1999).
GAO United States General Accounting Office
GAO/ NSIAD- 99- 162
United States General Accounting Office Washington, D. C. 20548
Lett er
Page 2 GAO/ NSIAD- 99- 162 Best Practices
Lett er
Executive Summary Page 3 GAO/ NSIAD- 99- 162 Best Practices
Lett er
Executive Summary Page 4 GAO/ NSIAD- 99- 162 Best Practices
Executive Summary Page 5 GAO/ NSIAD- 99- 162 Best Practices
Executive Summary Page 6 GAO/ NSIAD- 99- 162 Best Practices
Executive Summary Page 7 GAO/ NSIAD- 99- 162 Best Practices
Page 8 GAO/ NSIAD- 99- 162 Best Practices
Contents
Contents Page 9 GAO/ NSIAD- 99- 162 Best Practices
Contents Page 10 GAO/ NSIAD- 99- 162 Best Practices
Page 11 GAO/ NSIAD- 99- 162 Best Practices
Page 12 GAO/ NSIAD- 99- 162 Best Practices
Chapter 1
Chapter 1 Introduction
Page 13 GAO/ NSIAD- 99- 162 Best Practices
Chapter 1 Introduction
Page 14 GAO/ NSIAD- 99- 162 Best Practices
Chapter 1 Introduction
Page 15 GAO/ NSIAD- 99- 162 Best Practices
Chapter 1 Introduction
Page 16 GAO/ NSIAD- 99- 162 Best Practices
Chapter 1 Introduction
Page 17 GAO/ NSIAD- 99- 162 Best Practices
Chapter 1 Introduction
Page 18 GAO/ NSIAD- 99- 162 Best Practices
Chapter 1 Introduction
Page 19 GAO/ NSIAD- 99- 162 Best Practices
Chapter 1 Introduction
Page 20 GAO/ NSIAD- 99- 162 Best Practices
Chapter 1 Introduction
Page 21 GAO/ NSIAD- 99- 162 Best Practices
Page 22 GAO/ NSIAD- 99- 162 Best Practices
Chapter 2
Chapter 2 Maturity of Technology at Program Start Is an Important
Determinant of Success
Page 23 GAO/ NSIAD- 99- 162 Best Practices
Chapter 2 Maturity of Technology at Program Start Is an Important
Determinant of Success
Page 24 GAO/ NSIAD- 99- 162 Best Practices
Chapter 2 Maturity of Technology at Program Start Is an Important
Determinant of Success
Page 25 GAO/ NSIAD- 99- 162 Best Practices
Chapter 2 Maturity of Technology at Program Start Is an Important
Determinant of Success
Page 26 GAO/ NSIAD- 99- 162 Best Practices
Chapter 2 Maturity of Technology at Program Start Is an Important
Determinant of Success
Page 27 GAO/ NSIAD- 99- 162 Best Practices
Chapter 2 Maturity of Technology at Program Start Is an Important
Determinant of Success
Page 28 GAO/ NSIAD- 99- 162 Best Practices
Chapter 2 Maturity of Technology at Program Start Is an Important
Determinant of Success
Page 29 GAO/ NSIAD- 99- 162 Best Practices
Chapter 2 Maturity of Technology at Program Start Is an Important
Determinant of Success
Page 30 GAO/ NSIAD- 99- 162 Best Practices
Chapter 2 Maturity of Technology at Program Start Is an Important
Determinant of Success
Page 31 GAO/ NSIAD- 99- 162 Best Practices
Chapter 2 Maturity of Technology at Program Start Is an Important
Determinant of Success
Page 32 GAO/ NSIAD- 99- 162 Best Practices
Chapter 2 Maturity of Technology at Program Start Is an Important
Determinant of Success
Page 33 GAO/ NSIAD- 99- 162 Best Practices
Page 34 GAO/ NSIAD- 99- 162 Best Practices
Chapter 3
Chapter 3 Controllable Conditions Affect How Well a Technology's Inclusion
on a Product Can Be Managed
Page 35 GAO/ NSIAD- 99- 162 Best Practices
Chapter 3 Controllable Conditions Affect How Well a Technology's Inclusion
on a Product Can Be Managed
Page 36 GAO/ NSIAD- 99- 162 Best Practices
Chapter 3 Controllable Conditions Affect How Well a Technology's Inclusion
on a Product Can Be Managed
Page 37 GAO/ NSIAD- 99- 162 Best Practices
Chapter 3 Controllable Conditions Affect How Well a Technology's Inclusion
on a Product Can Be Managed
Page 38 GAO/ NSIAD- 99- 162 Best Practices
Chapter 3 Controllable Conditions Affect How Well a Technology's Inclusion
on a Product Can Be Managed
Page 39 GAO/ NSIAD- 99- 162 Best Practices
Chapter 3 Controllable Conditions Affect How Well a Technology's Inclusion
on a Product Can Be Managed
Page 40 GAO/ NSIAD- 99- 162 Best Practices
Chapter 3 Controllable Conditions Affect How Well a Technology's Inclusion
on a Product Can Be Managed
Page 41 GAO/ NSIAD- 99- 162 Best Practices
Chapter 3 Controllable Conditions Affect How Well a Technology's Inclusion
on a Product Can Be Managed
Page 42 GAO/ NSIAD- 99- 162 Best Practices
Chapter 3 Controllable Conditions Affect How Well a Technology's Inclusion
on a Product Can Be Managed
Page 43 GAO/ NSIAD- 99- 162 Best Practices
Chapter 3 Controllable Conditions Affect How Well a Technology's Inclusion
on a Product Can Be Managed
Page 44 GAO/ NSIAD- 99- 162 Best Practices
Chapter 3 Controllable Conditions Affect How Well a Technology's Inclusion
on a Product Can Be Managed
Page 45 GAO/ NSIAD- 99- 162 Best Practices
Chapter 3 Controllable Conditions Affect How Well a Technology's Inclusion
on a Product Can Be Managed
Page 46 GAO/ NSIAD- 99- 162 Best Practices
Chapter 3 Controllable Conditions Affect How Well a Technology's Inclusion
on a Product Can Be Managed
Page 47 GAO/ NSIAD- 99- 162 Best Practices
Chapter 3 Controllable Conditions Affect How Well a Technology's Inclusion
on a Product Can Be Managed
Page 48 GAO/ NSIAD- 99- 162 Best Practices
Chapter 3 Controllable Conditions Affect How Well a Technology's Inclusion
on a Product Can Be Managed
Page 49 GAO/ NSIAD- 99- 162 Best Practices
Page 50 GAO/ NSIAD- 99- 162 Best Practices
Chapter 4
Chapter 4 Impediments to Adopting Best Practices for Technology Inclusion in
DOD Are
Surmountable Page 51 GAO/ NSIAD- 99- 162 Best Practices
Chapter 4 Impediments to Adopting Best Practices for Technology Inclusion in
DOD Are
Surmountable Page 52 GAO/ NSIAD- 99- 162 Best Practices
Chapter 4 Impediments to Adopting Best Practices for Technology Inclusion in
DOD Are
Surmountable Page 53 GAO/ NSIAD- 99- 162 Best Practices
Chapter 4 Impediments to Adopting Best Practices for Technology Inclusion in
DOD Are
Surmountable Page 54 GAO/ NSIAD- 99- 162 Best Practices
Chapter 4 Impediments to Adopting Best Practices for Technology Inclusion in
DOD Are
Surmountable Page 55 GAO/ NSIAD- 99- 162 Best Practices
Chapter 4 Impediments to Adopting Best Practices for Technology Inclusion in
DOD Are
Surmountable Page 56 GAO/ NSIAD- 99- 162 Best Practices
Chapter 4 Impediments to Adopting Best Practices for Technology Inclusion in
DOD Are
Surmountable Page 57 GAO/ NSIAD- 99- 162 Best Practices
Chapter 4 Impediments to Adopting Best Practices for Technology Inclusion in
DOD Are
Surmountable Page 58 GAO/ NSIAD- 99- 162 Best Practices
Chapter 4 Impediments to Adopting Best Practices for Technology Inclusion in
DOD Are
Surmountable Page 59 GAO/ NSIAD- 99- 162 Best Practices
Chapter 4 Impediments to Adopting Best Practices for Technology Inclusion in
DOD Are
Surmountable Page 60 GAO/ NSIAD- 99- 162 Best Practices
Page 61 GAO/ NSIAD- 99- 162 Best Practices
Chapter 5
Chapter 5 Conclusions and Recommendations
Page 62 GAO/ NSIAD- 99- 162 Best Practices
Chapter 5 Conclusions and Recommendations
Page 63 GAO/ NSIAD- 99- 162 Best Practices
Chapter 5 Conclusions and Recommendations
Page 64 GAO/ NSIAD- 99- 162 Best Practices
Chapter 5 Conclusions and Recommendations
Page 65 GAO/ NSIAD- 99- 162 Best Practices
Chapter 5 Conclusions and Recommendations
Page 66 GAO/ NSIAD- 99- 162 Best Practices
Chapter 5 Conclusions and Recommendations
Page 67 GAO/ NSIAD- 99- 162 Best Practices
Page 68 GAO/ NSIAD- 99- 162 Best Practices
Appendix I
Page 69 GAO/ NSIAD- 99- 162 Best Practices
Appendix II
Appendix II Comments From the Department of Defense
Page 70 GAO/ NSIAD- 99- 162 Best Practices
Appendix II Comments From the Department of Defense
Page 71 GAO/ NSIAD- 99- 162 Best Practices
Appendix II Comments From the Department of Defense
Page 72 GAO/ NSIAD- 99- 162 Best Practices
Page 73 GAO/ NSIAD- 99- 162 Best Practices
Appendix III
Page 74 GAO/ NSIAD- 99- 162 Best Practices
Page 75 GAO/ NSIAD- 99- 162 Best Practices
Page 76 GAO/ NSIAD- 99- 162 Best Practices
(707336) Lett er
Ordering Information The first copy of each GAO report and testimony is
free. Additional copies are $2 each. Orders should be sent to the following
address, accompanied by a check or money order made out to the
Superintendent of Documents, when necessary, VISA and
MasterCard credit cards are accepted, also. Orders for 100 or more copies to
be mailed to a single address are discounted 25 percent.
Orders by mail: U. S. General Accounting Office P. O. Box 37050 Washington,
DC 20013
or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U. S.
General Accounting Office Washington, DC
Orders may also be placed by calling (202) 512- 6000 or by using fax number
(202) 512- 6061, or TDD (202) 512- 2537.
Each day, GAO issues a list of newly available reports and testimony. To
receive facsimile copies of the daily list or any list from the past 30
days, please call (202) 512- 6000 using a touchtone phone. A recorded menu
will provide information on how to obtain these lists.
For information on how to access GAO reports on the INTERNET, send an e-
mail message with “info” in the body to: info@ www. gao. gov or
visit GAO's World Wide Web Home Page at: http:// www. gao. gov
United States General Accounting Office Washington, D. C. 20548- 0001
Official Business Penalty for Private Use $300
Address Correction Requested Bulk Rate
Postage & Fees Paid GAO Permit No. GI00
*** End of document. ***