Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes (07/15/2002, GAO-02-701}

-------------------------Indexing Terms-------------------------
REPORTNUM:   GAO-02-701
    TITLE:   Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes
     DATE:   07/15/2002



This is the accessible text file for GAO report number GAO-02-701 
entitled 'Best Practices: Capturing Design and Manufacturing Knowledge 
Early Improves Acquisition Outcomes' which was released on July 15, 
2002.



This text file was formatted by the U.S. General Accounting Office 

(GAO) to be accessible to users with visual impairments, as part of a 

longer term project to improve GAO productsï¿½ accessibility. Every 

attempt has been made to maintain the structural and data integrity of 

the original printed product. Accessibility features, such as text 

descriptions of tables, consecutively numbered footnotes placed at the 

end of the file, and the text of agency comment letters, are provided 

but may not exactly duplicate the presentation or format of the printed 

version. The portable document format (PDF) file is an exact electronic 

replica of the printed version. We welcome your feedback. Please E-mail 

your comments regarding the contents or accessibility features of this 

document to [email protected].



Report to the Subcommittee on Readiness and Management Support, 

Committee on Armed Services, 

U.S. Senate:



July 2002:



BEST PRACTICES:



Capturing Design and Manufacturing Knowledge Early Improves Acquisition 

Outcomes:



GAO-02-701:



Letter:



Executive Summary:



Purpose:



Background:



Results in Brief:



Principal Findings:



Recommendations for Executive Action:



Agency Comments:



Chapter 1:



Best Practices of Leading Commercial Companies:



DODï¿½s Traditional Approach to Product Development:



DODï¿½s Adoption of Best Practices:



Objectives, Scope, and Methodology:



Chapter 2:



DOD Programs Had Better Outcomes When Design and Manufacturing 

Knowledge Was Captured at Key Program Junctures:



Chapter 3:



Leading Commercial Companies Use Evolutionary Product Development 

Framework to Reduce Development Risks:



Leading Commercial Companies Use a Product Development Process to 

Capture Design and Manufacturing Knowledge for Decision Making:



When DOD Programs More Closely Approximated Best Practices, Outcomes 

Were Better:



Chapter 4:



Acquisition Policy Lacks Specific Implementation Criteria:



Incentives in the DOD Acquisition Environment Do Not Favor Capture of 

Design and Manufacturing Knowledge Early

Enough:



Chapter 5:



Conclusions:



Recommendations for Executive Action:



Agency Comments and Our Evaluation:



Appendixes:



Appendix I: Comments from the Department of Defense:



Appendix II: GAO Staff Acknowledgements:



Acknowledgments:



Related GAO Products:



Tables:



Table 1: Activities That Enable the Capture of Design and Manufacturing 

Knowledge:



Table 2: Attainment of Design and Manufacturing Knowledge in DOD 

Programs and the Program Outcomes:



Table 3: Activities to Capture Design Knowledge and Make Decisions:



Table 4: Examples of Prototypes Used by Cummins Inc. at Various Stages 

of Product Development:



Table 5: Activities to Capture Manufacturing Knowledge and Make 

Decisions:



Table 6: Cpk Index and Probability of a Defective Part:



Table 7: Analysis of DOD Acquisition Policy for Inclusion of Best 

Practices for Knowledge-based Design and Manufacturing Decisions:



Figures:



Figure 1: Research, Development, Test and Evaluation, and Procurement 

Funding for Fiscal Years 1995 to 2007:



Figure 2: Knowledge-based Process for Applying Best Practices to the 

Development of New Products:



Figure 3: Notional Illustration Showing the Different Paths That a 

Productï¿½s Development Can Take:



Figure 4: DODï¿½s Concurrent Approach to Weapon System Development:



Figure 5: Notional Single-Step and Evolutionary Approaches to 

Developing New Products:



Figure 6: Achieving Stability on AIM-9X Missile Program by Knowledge 

Point 2:



Figure 7: History of Drawing Completion for the F-22 Program:



Figure 8: PAC-3 Design Knowledge at Critical Design Review:



Figure 9: Illustration to Show How the Best Practice Model Would Apply 

to DODï¿½s Acquisition Process:



Letter July 15, 2002:



The Honorable Daniel Akaka

Chairman

The Honorable James Inhofe

Ranking Minority Member

Subcommittee on Readiness and Management Support

Committee on Armed Services

United States Senate:



As you requested, this report examines how best practices offer 

improvements to the way the Department of Defense develops new weapon 

systems, primarily the design and manufacturing aspects of the 

acquisition process. It examines the attainment of design and 

manufacturing knowledge and its use at critical junctures to make 

decisions about weapon systemsï¿½ readiness to move forward in the 

acquisition process. We make recommendations to the Secretary of 

Defense for improvements to weapon system acquisition policy to better 

align design and manufacturing activities with best practices that have 

shown that the capture and use of key knowledge can result in better 

cost, schedule, and performance outcomes.



We are sending copies of this report to the Secretary of Defense; the 

Secretary of the Army; the Secretary of the Navy; the Secretary of the 

Air Force; the Director of the Office of Management and Budget; the 

Director, Missile Defense Agency; and interested congressional 

committees. We will also make copies available to others upon request. 

In addition, the report will be available at no charge on the GAO Web 

site at http://www.gao.gov.



If you have any questions regarding this report, please call me at 

(202) 512-4841. Other contacts are listed in appendix II.



Katherine V. Schinasi

Director

Acquisition and Sourcing Management:

Signed by Katherine V. Schinasi:



[End of section]



Executive Summary:



Purpose:



Historically, the Department of Defense (DOD) has taken much longer and 

spent much more than originally planned to develop and acquire its 

weapon systems, significantly reducing the departmentï¿½s buying power 

over the years. Clearly, it is critical to find better ways of doing 

business and, in particular, to make sure that weapon systems are 

delivered on time and cost-effectively. This is especially true given 

the vast sums DOD is spending and is expected to spend on weapons 

acquisition--$100 billion alone in 2002 and an anticipated $700 billion 

over the next 5 years. DOD has recognized the nature of this problem 

and has taken steps to address it, including advocating the use of best 

practices for product development from commercial companies. Leading 

commercial companies have achieved more predictable outcomes from their 

product development processes because they identify and control design 

and manufacturing risks early and manage them effectively. While DOD 

has made some progress in recent years, GAOï¿½s recent weapon system 

reviews show that persistent problems continue to hinder acquisition 

cost, schedule, and performance outcomes. For this reason, GAO has 

continued a body of work to identify the lessons learned by best 

commercial companies to see if they apply to weapon system 

acquisitions.



This report addresses how DOD can manage its weapon system acquisition 

process to ensure important knowledge about a systemï¿½s design, critical 

manufacturing processes, and reliability is captured and used to make 

informed and timely decisions before committing to substantial 

development and production investments. It identifies best practices to 

facilitate this decision making at two critical junctures--transition 

from system integration to system demonstration during product 

development and then transition into production. Ultimately, this 

should improve cost, schedule, and quality outcomes of DOD major weapon 

system acquisitions. In response to a request from the Chairman and the 

Ranking Minority Member, Subcommittee on Readiness and Management 

Support, Senate Committee on Armed Services, GAO (1) assessed the 

impact of design and manufacturing knowledge on DOD program outcomes, 

(2) compared best practices to those used in DOD programs, and (3) 

analyzed current weapon system acquisition guidance for applicability 

of best practices to obtain better program outcomes.



Background:



In any new product development program there are three critical points 

that require the capture of specific knowledge to achieve successful 

outcomes. The first knowledge point occurs when the customerï¿½s 

requirements are clearly defined and resources--proven technology, 

design, time, and money--exist to satisfy them. Commercial companies 

insist that technology be mature at the outset of a product development 

program and, therefore, separate technology development from product 

development. The second knowledge point is achieved when the productï¿½s 

design is determined to be capable of meeting product requirements--the 

design is stable and ready to begin initial manufacturing of 

prototypes. The third knowledge point is achieved when a reliable 

product can be produced repeatedly within established cost, schedule, 

and quality targets. GAOï¿½s prior work on best practices covers 

achieving the first knowledge point.[Footnote 1] This report examines 

best practices for achieving the second and third knowledge points.



Commercial companies understand the importance of capturing design and 

manufacturing knowledge early in product development, when costs to 

identify problems and make design changes to the product are 

significantly cheaper. In a knowledge-based process, the achievement of 

each successive knowledge point builds on the preceding one, giving 

decision makers the knowledge they need--when they need it--to make 

decisions about whether to invest significant additional funds to move 

forward with product development. Programs that follow a knowledge-

based approach typically have a higher probability of successful cost 

and schedule outcomes. Problems occur in programs when knowledge builds 

more slowly than commitments to enter product development or 

production. The effects of this delay in capturing knowledge can be 

debilitating. If a decision is made to commit to develop and produce a 

design before the critical technology, design, or manufacturing 

knowledge is captured, problems will cascade and become magnified 

through the product development and production phases. Outcomes from 

these problems include increases in cost and schedule and degradations 

in performance and quality.



Results in Brief:



The success of any effort to develop a new product hinges on having the 

right knowledge at the right time. Knowledge about a productï¿½s design 

and producibility facilitates informed decisions about whether to 

significantly increase investments and reduces the risk of costly 

design changes later in the program. Every program eventually achieves 

this knowledge; however, leading commercial companies GAO visited have 

found that there is a much better opportunity to meet predicted cost, 

schedule, and quality targets when it is captured early, in preparation 

for critical investment decisions. A product development process 

includes two phases followed by production--integration phase and 

demonstration phase. The commercial companies GAO visited achieved 

success in product development by first achieving a mature, stable 

design supported by completed engineering drawings during an 

integration phase and then by demonstrating that the productï¿½s design 

was reliable and critical manufacturing processes required to build it 

were in control before committing to full production. The more 

successful DOD programs GAO reviewed--the AIM-9X and the FA-18-E/F 

programs--had achieved similar knowledge as the commercial companies, 

resulting in good cost and schedule outcomes. In contrast, the DOD 

programs, which had completed about one-quarter of their drawings when 

they transitioned to the demonstration phase and had less than half of 

their manufacturing processes in control when entering production, 

experienced poor cost and schedule outcomes.



Leading commercial companies employed practices to capture design and 

manufacturing knowledge in time for making key decisions during product 

development. Two were most prominent. First, the companies kept the 

degree of the design challenge manageable before starting a new product 

development program by using an evolutionary approach to develop a 

product. This minimized the amount of new content and technologies on a 

product, making it easier to capture the requisite knowledge about a 

productï¿½s design before investing in manufacturing processes, tooling, 

and facilities. Second, the companies captured design and manufacturing 

knowledge before the two critical decision points in product 

development: when the design was demonstrated to be stable--the second 

knowledge point--and when the product was demonstrated to be producible 

at an affordable cost--the third knowledge point. A key measure of 

design stability was stakeholdersï¿½ agreements that engineering drawings 

were complete and supported by testing and prototyping when necessary. 

A key measure of producibility was whether the companiesï¿½ critical 

manufacturing processes were in control and product reliability was 

demonstrated. Most DOD programs GAO reviewed did not complete 

engineering drawings prior to entering the demonstration phase, nor did 

they bring critical manufacturing processes in control or demonstrate 

reliability prior to making a production decision.



DOD has made changes to its acquisition policy[Footnote 2] in an 

attempt to improve its framework for developing weapon systems, but the 

policy does not require the capture of design or manufacturing 

knowledge or sufficient criteria to enter the system demonstration and 

production phases. In addition, it does not require a decision review 

to enter the demonstration phase of product development. Further, there 

is little incentive for DOD program managers to capture knowledge early 

in the development process. Instead, the acquisition environment 

emphasizes delaying knowledge capture and problem identification since 

these events can have a negative influence on obtaining annual program 

funding--a key to success for DOD managers. In contrast, commercial 

companies encourage their managers to capture product design and 

manufacturing knowledge to identify and resolve problems early in 

development, before making significant increases in their investment.



GAO is making recommendations to the Secretary of Defense on ways to 

improve DODï¿½s acquisition process to achieve better outcomes by 

incorporating best practices to capture design and manufacturing 

knowledge and then use this knowledge as a basis for decisions to 

commit significant additional time and money as an acquisition program 

progresses through system demonstration and into production.



Principal Findings:



Timely Design and Manufacturing Knowledge Is Critical to Program 

Success:



Knowledge that a productï¿½s design is stable early in the program 

facilitates informed decisions about whether to significantly increase 

investments and reduces the risk of costly design changes that can 

result from unknowns after initial manufacturing begins. Likewise, 

later knowledge that the design can be manufactured affordably and with 

consistent high quality prior to making a production decision ensures 

that targets for cost and schedule during production will be met. 

Leading commercial companies do not make significant investments to 

continue a product development or its production until they have 

knowledge that the productï¿½s design works and it can be manufactured 

efficiently within cost and schedule expectations.



DOD programs that captured knowledge similar to commercial companies 

had more successful outcomes. For example, the AIM-9X and the F/A-18E/

F captured design and manufacturing knowledge by key decision points 

and limited cost increases to 4 percent or less and schedule growth to 

3 months or less. In fact, the AIM-9X had 95 percent of its drawings 

completed at its critical design review. The F/A-18E/F had 56 percent 

of its drawings completed and also had over 90 percent of its higher 

level interface drawings completed, adding confidence in the system 

design. Both took steps to ensure that manufacturing processes were 

capable of producing an affordable product by the time the programs 

made production decisions.



On the other hand, the F-22, PAC-3, and Advanced Threat Infrared 

Countermeasures/Common Missile Warning System (ATIRCM/CMWS) programs 

did not capture sufficient knowledge before significant investments to 

continue the programs and experienced cost growth that ranged from 23 

to 182 percent and schedule delays that ranged from 

18 months to over 3 years. None of these programs had completed more 

than 26 percent of their engineering drawings for their critical design 

reviews, and only the F-22 and PAC-3 programs attempted to track the 

capability of their critical manufacturing processes prior to 

production.



Best Practices Enable Timely Capture of Design and Manufacturing 

Knowledge:



Leading commercial companies developed practices that enabled the 

timely capture of design and manufacturing knowledge. First, they used 

an evolutionary approach to product development by establishing time-

phased plans to develop a new product in increments based on 

technologies and resources achievable now and later. This approach 

reduced the amount of risk in the development of each increment, 

facilitating greater success in meeting cost, schedule, and performance 

requirements. The commercial companies GAO visited used the 

evolutionary approach as their method for product development. Each 

company had a plan for eventually achieving a quantum leap in the 

performance of its products and had established an orderly, phased 

process for getting there, by undertaking continuous product 

improvements as resources became available. For the most part, DOD 

programs try to achieve the same leap in performance but in just one 

step, contributing to development times that can take over 15 years to 

deliver a new capability to the military user.



Second, each leading commercial company had a product development 

process that was prominent and central to its success. The process was 

championed by executive leadership and embraced by product managers and 

development teams as an effective way to do business. Critical to the 

product development process were activities that enabled the capture of 

specific design and manufacturing knowledge and decision reviews to 

determine if the knowledge captured would support the increased 

investment necessary to move to the next development phase or into 

production. These activities provided knowledge that the product design 

was stable at the decision point to start initial manufacturing 

(exiting the integration phase) as demonstrated by the completion of 90 

percent of the engineering drawings. They also captured knowledge that 

a product was ready to begin production (exiting the demonstration 

phase) as demonstrated by proof that critical processes were in control 

and product reliability was achievable. The activities that enabled the 

capture and use of this knowledge to make decisions are listed in table 

1.



Table 1: Activities That Enable the Capture of Design and Manufacturing 

Knowledge:



[See PDF for Image]



[End of table]



DOD programs that had more successful outcomes used key best practices 

to a greater degree than others. For example, the AIM-9X missile 

program completed 95 percent of its engineering drawings at the 

critical design review because it made extensive use of prototype 

testing to demonstrate the design met requirements coupled with design 

reviews that included program stakeholders. The F/A-18-E/F program 

eliminated over 40 percent of the parts used to build predecessor 

aircraft to make the design more robust for manufacturing and 

identified critical manufacturing processes, bringing them under 

control before the start of production. Both programs developed 

products that evolved from existing versions, making the design 

challenge more manageable.



On the other hand, DOD programs with less successful outcomes did not 

apply best practices to a great extent. At their initial manufacturing 

decision reviews, the F-22, PAC-3, and ATIRCM/CMWS had less than one-

third of their engineering drawings, in part, because they did not use 

prototypes to demonstrate the design met requirements before starting 

initial manufacturing. On the F-22 program, it was almost 3 years after 

this review before 90 percent of the drawings needed to build the F-22 

were completed. Likewise, at their production decision reviews, these 

programs did not capture manufacturing and product reliability 

knowledge consistent with best practices. For example, the PAC-3 

missile program had less than 40 percent of its processes in control 

and, as a result, the missile seekers had to be built, tested, and 

reworked on average 4 times before they were acceptable. The F-22 

entered production despite being substantially behind its plan to 

achieve reliability goals. As a result, the F-22 is requiring 

significantly more maintenance actions than planned.



A Better Match of Policy and Incentives Is Needed to Ensure Capture of 

Design and Manufacturing Knowledge:



DODï¿½s acquisition policy establishes a good framework for developing 

weapon systems; however, more specific criteria, disciplined adherence, 

and stronger acquisition incentives are needed to ensure the timely 

capture and use of knowledge and decision making. DOD recently changed 

its acquisition policy to emphasize evolutionary acquisition and 

establish separate integration and demonstration phases in the product 

development process. Its goal was to develop higher quality systems in 

less time and for less cost. While similar to the leading commercial 

companiesï¿½ approach, the policy lacks detailed criteria for capturing 

and using design and manufacturing knowledge to facilitate better 

decisions and more successful acquisition program outcomes. It also 

lacks a decision review to proceed from the integration phase to the 

demonstration phase of product development.



While the right policy and criteria are necessary to ensure a 

disciplined, knowledge-based product development process, the 

incentives that influence the key players in the acquisition process 

will ultimately determine whether they will be used effectively. In 

DOD, current incentives are geared toward delaying knowledge so as not 

to jeopardize program funding. This undermines a knowledge-based 

process for making product development decisions. Instead, program 

managers and contractors push the capture of design and manufacturing 

knowledge to later in the development program to avoid the 

identification of problems that might stop or limit funding. They focus 

more on meeting schedules than capturing knowledge. On the other hand, 

commercial companies must develop high-quality products quickly or they 

may not survive in the marketplace. Because of this, they encourage 

their managers to capture product design and manufacturing knowledge to 

identify and resolve problems early in development, before making 

significant increases in their investment. Instead of a schedule-driven 

process, their process is driven by events that bring them knowledge: 

critical design reviews that are supported by completed engineering 

drawings and production decisions supported by reliability testing and 

statistical process control data. They do not move forward without the 

design and manufacturing knowledge needed to make informed decisions.



Recommendations for Executive Action:



GAO recommends that the Secretary of Defense revise policy and guidance 

on the operation of the defense acquisition system to include (1) a 

requirement to capture specific design knowledge to be used as exit 

criteria for transitioning from system integration to system 

demonstration and (2) a requirement that the current optional interim 

progress review between system integration and demonstration be a 

mandatory decision review requiring the program manager to verify that 

design is stable and that this be reported in the programï¿½s Defense 

Acquisition Executive Summary and Selected Acquisition Report. The 

policy and guidance should also be revised to include (1) a requirement 

to capture and use specific manufacturing knowledge at the production 

commitment point as exit criteria to transition from system 

demonstration into production and (2) a requirement to structure major 

weapon system contracts to ensure the capture and use of knowledge for 

DOD to make investment decisions at critical junctures when 

transitioning from system integration to system demonstration and then 

into production.



Agency Comments:



DOD generally agreed with the report and its recommendations. A 

detailed discussion of DODï¿½s comments appears in appendix I.



[End of section]



Chapter 1: Introduction:



The Department of Defense (DOD) spends close to $100 billion annually 

to research, develop, and acquire weapon systems, and this investment 

is expected to grow substantially. Over the next 5 years, starting in 

fiscal year 2003, DODï¿½s request for weapon system development and 

acquisition funds is estimated to be $700 billion (see fig. 1).



How effectively DOD manages these funds will determine whether it 

receives a good return on its investment. Our reviews over the past 20 

years have consistently found that DODï¿½s weapon system acquisitions 

take much longer and cost much more than originally anticipated, 

causing disruptions to the departmentï¿½s overall investment strategy and 

significantly reducing its buying power. Because such disruptions can 

limit DODï¿½s ability to effectively execute war-fighting operations, it 

is critical to find better ways of doing business.



In view of the importance of DODï¿½s investment in weapon systems, we 

have undertaken an extensive body of work that examines DODï¿½s 

acquisition issues from a different, more cross-cutting perspective--

one that draws lessons learned from the best commercial product 

development efforts to see if they apply to weapon system acquisitions. 

This report looks at the core of the acquisition process, specifically 

product development and ways to successfully design and manufacture the 

product. Our previous reports looked at such issues as how companies 

matched customer needs and resources, tested products, assured quality, 

and managed suppliers and are listed in related GAO products at the end 

of the report.



Figure 1: Research, Development, Test and Evaluation, and Procurement 

Funding for Fiscal Years 1995 to 2007:



[See PDF for image]



Source: DOD.



[End of figure]



Best Practices of Leading Commercial Companies:



Leading commercial companies expect their program managers to deliver 

high-quality products on time and within budget. Doing otherwise could 

result in the customer walking away. Thus, the companies have created 

an environment and adopted practices that put their program managers in 

a good position to succeed in meeting these expectations. Collectively, 

these practices ensure that a high level of knowledge exists about 

critical facets of the product at key junctures during development. 

Such a knowledge-based process enables decision makers to be reasonably 

certain about critical facets of the product under development when 

they need this knowledge.



To ensure the right level of knowledge at each key decision point in 

product development, leading commercial companies separate technology 

from product development and take steps to ensure the product design is 

stabilized early so product performance and producibility can be 

demonstrated before production. The process followed by leading 

companies, illustrated in figure 2, can be broken down into the 

following three knowledge points.



* Knowledge point 1 occurs when a match is made between the customerï¿½s 

needs and the available resources--technology, design, time, and 

funding. To achieve this match, technologies needed to meet essential 

product requirements must be demonstrated to work in their intended 

environment. In addition, the product developer must complete a 

preliminary product design using systems engineering to balance 

customer desires with available resources.



* Knowledge point 2 occurs when the productï¿½s design demonstrates its 

ability to meet performance requirements. Program officials are 

confident that the design is stable and will perform acceptably when at 

least 90 percent of engineering drawings are complete. Engineering 

drawings reflect the results of testing and simulation and describe how 

the product should be built.



* Knowledge point 3 occurs when the product can be manufactured within 

cost, schedule, and quality targets and is reliable. An important 

indicator of this is when critical manufacturing processes are in 

control and consistently producing items within quality standards and 

tolerances. Another indicator is when a productï¿½s reliability is 

demonstrated through iterative testing that identifies and corrects 

design problems.



Figure 2: Knowledge-based Process for Applying Best Practices to the 

Development of New Products:



[See PDF for image]



Source: GAOï¿½s analysis.



[End of figure]



This report focuses on best practices for achieving knowledge points 2 

and 3, particularly at how successful companies design and manufacture 

a product within established cost, schedule, and quality targets. The 

concepts discussed build on our previous reports, which looked at the 

earlier phases of an acquisition, including matching customer needs and 

available resources.



A key success factor evident in all our work is the ability to obtain 

the right knowledge at the right time and to build knowledge to the 

point that decision makers can make informed decisions about moving 

ahead to the next phase. Programs that do this typically have 

successful cost and schedule outcomes. Programs that do not typically 

encounter problems that eventually cascade and become magnified through 

the product development and production phases. As shown in figure 3, 

the effects of not following a knowledge-based process can be 

debilitating.



Figure 3: Notional Illustration Showing the Different Paths That a 

Productï¿½s Development Can Take:



[See PDF for image]



Source: GAOï¿½s analysis.



[End of figure]



DODï¿½s Traditional Approach to Product Development:



DOD has historically developed new weapon systems in a highly 

concurrent environment that usually forces acquisition programs to 

manage technology, design, and manufacturing risk at the same time. 

This environment has made it difficult for either DOD or congressional 

decision makers to make informed decisions because appropriate 

knowledge has not been available at key decision points in product 

development. DODï¿½s common practice for managing this environment has 

been to create aggressive risk reduction efforts in its programs. Cost 

reduction initiatives that typically arise after a program is 

experiencing problems are common tools used to manage these risks. 

Figure 4 shows the overlapping and concurrent approach that DOD uses to 

develop its weapon systems. This figure shows that DOD continues to 

capture technology, design, and manufacturing knowledge long after a 

program passes through each of the three knowledge points when this 

knowledge should have been available for program decisions.



Figure 4: DODï¿½s Concurrent Approach to Weapon System Development:



[See PDF for image]



Source: GAOï¿½s analysis.



[End of figure]



More important, the problems created by this concurrent approach on 

individual programs can profoundly affect DODï¿½s overall modernization 

plans. It is difficult to prioritize and allocate limited budgets among 

needed requirements when acquisition programsï¿½ cost and schedule are 

always in question. Programs that are managed without the knowledge-

based process are more likely to have surprises in the form of cost and 

schedule increases that are accommodated by disrupting the funding of 

other programs. Because of these disruptions, decision makers are not 

able to focus on a balanced investment strategy.



DODï¿½s Adoption of Best Practices:



DOD is taking steps to change the culture of the acquisition community 

with actions aimed at reducing product development cycle times and 

improving the predictability of cost and schedule outcomes. DOD 

recently made constructive changes to its acquisition policy that 

embrace best practices. These changes focused primarily on (1) ensuring 

technologies are demonstrated to a high level of maturity before 

beginning a weapon system program and (2) taking an evolutionary, or 

phased, approach to developing new weapon systems. Because these 

changes occurred in 2000 and 2001, it is too early to determine how 

effectively they will be put into practice. While these are good first 

steps, further use of best practices in product development would 

provide a greater opportunity to improve weapon system cost and 

schedule outcomes.



Objectives, Scope, and Methodology:



Our overall objective was to determine whether best practices offer 

methods to improve the way DOD ensures that the design is stable early 

in the development process and whether having manufacturing processes 

in control before production results in better cost, schedule, and 

quality outcomes in DOD major acquisition programs. Specifically, we 

identified best practices that have led to more successful product 

development and production outcomes, compared the best practices to 

those used in DOD programs, and analyzed current weapon system 

acquisition guidance for applicability of best practices.



To determine the best practices for ensuring product design and 

manufacturing maturity from the commercial sector, we conducted general 

literature searches. On the basis of our literature searches and 

discussions with experts, we identified a number of commercial 

companies as having innovative development processes and practices that 

resulted in successful product development. We visited the following 

commercial companies:



* Caterpillar designs and manufactures construction and mining 

equipment, diesel and natural gas engines, and industrial gas turbines. 

In 2001, it reported sales and revenues totaling $20.45 billion. We 

visited its offices in Peoria, Illinois.



* Cummins Inc. (Engine Business group) designs and manufactures diesel 

and natural gas engines ranging in size from 60 to 3,500 horsepower for 

mining, construction, agriculture, rail, oil and gas, heavy and medium-

duty trucks, buses, and motor homes. In 2001, the Engine Business Group 

reported sales of $3.1 billion. We visited its offices in Columbus, 

Indiana.



* General Electric Aircraft Engines designs and manufactures jet 

engines for civil and military aircraft and gas turbines, derived from 

its successful jet engine programs, for marine and industrial 

applications. In 2001, it reported earnings totaling $11.4 billion. We 

visited its offices in Evendale, Ohio.



* Hewlett Packard designs and manufactures computing systems and 

imaging and printing systems for individual and business use. In 2001, 

it reported revenues totaling $45.2 billion. We visited its offices 

involved in the design and manufacturing of complex ink jet imaging 

equipment in Corvallis, Oregon.



* Xerox Corporation designs and manufactures office equipment, 

including color and black and white printers, digital presses, 

multifunction devices, and digital copiers designed for offices and 

production-printing environments. In 2001, it reported revenues 

totaling $16.5 billion. We visited its offices in Rochester, New York.



At each of the five companies, we conducted structured interviews with 

representatives to gather uniform and consistent information about each 

companyï¿½s new product development processes and best practices. During 

meetings with these representatives, we obtained a detailed description 

of the processes and practices they believed necessary and vital to 

mature a product design and get manufacturing processes under control. 

We met with design engineers, program managers, manufacturing and 

quality engineers, and developers of the knowledge-based processes and 

policies.



During the past 5 years, we have gathered information on product 

development practices from such companies as 3M, Boeing Commercial 

Airplane Group, Chrysler Corporation, Bombardier Aerospace, Ford Motor 

Company, Hughes Space and Communications, and Motorola Corporation. 

This information enabled us to develop an overall model to describe the 

general approach leading commercial companies take to develop new 

products.



Our report highlights several best practices in product development 

based on our fieldwork. As such, they are not intended to describe all 

practices or suggest that commercial companies are without flaws. 

Representatives from the commercial companies visited told us that the 

development of their best practices has evolved over many years and 

that the practices continue to be improved based on lessons learned and 

new ideas and information. They admit that the application and use of 

these have not always been consistent or without error. However, they 

strongly suggested that the probability of success in developing new 

products is greatly enhanced by the use of these practices. Further, 

because of the sensitivity to how data that would show the actual 

outcomes of new product development efforts might affect their 

competitive standing, we did not obtain specific cost, schedule, and 

performance data. Most examples provided by these companies were 

anecdotal. However, the continued success of these companies over time 

in a competitive marketplace indicated that their practices were 

important and key to their operations. Furthermore, based on our 

observations during meetings at these companies, it was apparent that 

because of the level of detailed process tools developed for their 

managers and executive leadership these best practices were a 

centerpiece of their operations.



Next, we compared and contrasted the best practices with product 

development practices used in five DOD major acquisition programs. 

Below is a brief description of each program we examined:



* The F-22 fighter aircraft program. This aircraft is designed with 

advanced features to allow it to be less detectable to adversaries, 

capable of high speeds for long ranges, and able to provide the pilot 

with improved awareness of the surrounding situation through the use of 

integrated avionics. The F-22 program began in 1986 and entered limited 

production in 2001. The Air Force expects to buy 341 at a total 

acquisition cost (development and procurement) estimated at $69.7 

billion.



* The Patriot Advanced Capability (PAC-3) missile program. This program 

is intended to enhance the Patriot system, an air-defense, guided 

missile system. PAC-3 is designed to enhance the Patriot radarï¿½s 

ability to detect and identify targets, increase system computer 

capabilities, improve communications, increase the number of missiles 

in each launcher, and incorporate a new ï¿½hit-to-killï¿½ missile. The 

ï¿½hit-to-killï¿½ missile capabilities represent a major part of the 

development program, as these are not capabilities included in prior 

versions of the Patriot system. The missile program began in 1994 and 

entered limited production in 1999. The Army plans to buy 1,159 

missiles at a total acquisition cost estimated at $8.5 billion.



* The Advanced Threat Infrared Countermeasures/Common Missile Warning 

System (ATIRCM/CMWS) program. ATIRCM/CMWS is a defensive countermeasure 

system for protection against infrared guided missiles. The common 

missile warning system detects missiles in flight, and the advanced 

threat infrared countermeasure defeats the missile with the use of a 

laser. The combined system is designed for helicopter aircraft. The 

common missile warning system is also designed for tactical aircraft 

such as fighters. The program began in 1995 and is expected to start 

limited production in 2002. The Army and the Special Operations Command 

plan to buy 1,078 systems at a total acquisition cost estimated at $2.9 

billion.



* The AIM-9X missile program. AIM-9X is an infrared, short range, air-

to-air missile carried by Navy and Air Force fighter aircraft. The AIM-

9X is an extensive upgrade of the AIM-9M. The AIM-9X is planned to have 

increased resistance to countermeasures and improved target acquisition 

capability. A key feature is that it will have the ability to acquire, 

track, and fire on targets over a wider area than the AIM-9M. The AIM-

9X program began in 1994 and entered limited production in 2000. DOD 

plans to buy 10,142 missiles at a total acquisition cost estimated at 

$3 billion.



* The F/A-18 E/F fighter aircraft program. This aircraft is intended to 

complement and eventually replace the current F/A-18 C/D aircraft and 

perform Navy fighter escort, strike, fleet air defense, and close air 

support missions. It is the second major model upgrade since the F/A-18 

inception. The development program began in 1992. The program entered 

limited production in 1997 and full rate production in 2000. The Navy 

plans to buy 548 aircraft at a total acquisition cost estimated at 

$48.8 billion.



We selected these programs for review based on cost, schedule, and 

performance data presented in the Selected Acquisition Reports[Footnote 

3] for each program. We also selected these programs because we 

considered them to be in two basic categories--successful and 

unsuccessful cost and schedule performance outcomes. This basis for 

selection was to compare and contrast the development practices used on 

each with best practices used by the commercial companies. For each 

program, we interviewed key managers and design and manufacturing 

engineering representatives. In some cases, we discussed design and 

manufacturing issues with representatives of the primary contractor for 

the specific program to obtain information on the practices and 

procedures used by the program to ready the product design for initial 

manufacturing and testing as well as for production. We also discussed 

the use and potential application of best practices that we identified. 

In addition to discussions, we analyzed significant amounts of data on 

engineering drawings, design changes, labor efficiencies, 

manufacturing processes, quality indicators, testing, and schedules. We 

did not verify the accuracy of the data but did correlate it to other 

program indicators for reasonableness. Our analysis of the data was 

used as a basis to develop indicators of each programï¿½s development 

efficiencies and detailed questions to discuss product design and 

manufacturing practices.



We conducted our review between May 2001 and April 2002 in accordance 

with generally accepted government auditing standards.



[End of section]



Chapter 2 Timely Design and Manufacturing Knowledge Is Critical to 

Program Success:



The success of any effort to develop a new product hinges on having the 

right knowledge at the right time. Every program eventually achieves 

this knowledge; however, leading commercial companies we visited have 

found that there is a much better opportunity to meet predicted cost, 

schedule, and quality targets when it is captured early, in preparation 

for critical decisions. Specifically, knowledge that a productï¿½s design 

is stable early in the program facilitates informed decisions about 

whether to significantly increase investments and reduces the risk of 

costly design changes that can result from unknowns after initial 

manufacturing begins. This knowledge comes in the form of completed 

engineering drawings before transitioning from the system integration 

phase to the system demonstration phase of product development. Best 

practices suggest that at least 90 percent of the drawings for a 

productï¿½s design be completed before a decision to commit additional 

resources is made. Likewise, later knowledge that the design can be 

manufactured affordably and with consistent high quality prior to 

making a production decision ensures that cost and schedule targets 

will be met. This knowledge comes in the form of evidence from data 

that shows manufacturing processes are in control and system 

reliability is achievable. Leading commercial companies rely on 

knowledge obtained about critical manufacturing processes and product 

reliability to make their production decisions.



The Department of Defense (DOD) programs we reviewed captured varying 

amounts of design and manufacturing knowledge in the form of completed 

engineering drawings and statistical process control data. We found a 

correlation between the amount of knowledge each captured and their 

cost and schedule outcomes. Programs that were able to complete more 

engineering drawings and control their critical manufacturing processes 

had more success in meeting cost and schedule targets established when 

they began.



DOD Programs Had Better Outcomes When Design and Manufacturing 

Knowledge Was Captured at Key Program Junctures:



Conceptually, the product development process has two phases: a system 

integration phase to stabilize the productï¿½s design and a system 

demonstration phase to demonstrate the product can be manufactured 

affordably and work reliably. The system integration phase is used to 

stabilize the overall system design by integrating components and 

subsystems into a product and by showing that the design can meet 

product requirements. When this knowledge is captured, knowledge point 

2 has been achieved. It should be demonstrated by the completion of at 

least 90 percent of engineering drawings, which both DOD and leading 

commercial companies consider to be the point when a productï¿½s design 

is essentially complete. In the DOD process, this knowledge point 

should happen by the critical design review, before system 

demonstration and the initial manufacturing of production 

representative products begins. The system demonstration phase is then 

used to demonstrate that the product will work as required and can be 

manufactured within targets. When this knowledge is captured, knowledge 

point 3 has been achieved. Critical manufacturing processes are in 

control and consistently producing items within quality standards and 

tolerances for the overall product. Also, product reliability has been 

demonstrated. In the DOD process, like with the commercial process, 

this knowledge point should happen by the production commitment 

milestone. Bypassing critical knowledge at either knowledge point will 

usually result in cost, schedule, and performance problems later in 

product development and production.



We found that the most successful programs had taken steps to gather 

knowledge that confirmed the productï¿½s design was stable before the 

design was released to manufacturing organizations to build products 

for demonstration. They had most of the detailed design complete, 

supported by the completion of a large percentage of engineering 

drawings to manufacturing. Again, engineering drawings are critical 

because they include details on the parts and work instructions needed 

to make the product and reflect the results of testing. These drawings 

allowed manufacturing personnel to effectively plan the fabrication 

process and efficiently build production representative prototypes in 

the factory so manufacturing processes and the productï¿½s performance 

could be validated before committing to production. The most successful 

DOD programs also captured the knowledge that manufacturing processes 

needed to build the product would consistently produce a reliable 

product by the end of system demonstration, before making a production 

decision. On these programs, the initial phase of production--sometimes 

known as low-rate initial production--was able to focus on building 

operational test articles and improving the production processes, 

instead of continuing the productï¿½s design and development.



Problematic programs moved forward into system demonstration without 

the same knowledge from engineering drawings that successful cases had 

captured. They increased investments in tooling, people, and materials 

before the design was stable. In these programs, only a small 

percentage of the drawings needed to make the products had been 

completed at the time the designs were released to manufacturing 

organizations for building production representative prototypes. In 

doing so, these programs undertook the difficult challenge of 

stabilizing the designs at the same time they were trying to build and 

test the products. This design immaturity caused costly design changes 

and parts shortages that, in turn, caused labor inefficiencies, 

schedule delays, and quality problems. Consequently, these programs 

required significant increases in resources--time and money--over what 

was estimated at the point each program began the system demonstration 

phase.



The most problematic programs also started production before design and 

manufacturing development work was concluded. In these cases, programs 

were producing items for the customers while making major product 

design and tooling changes, still establishing manufacturing processes, 

and conducting development testing. These programs encountered 

significant cost increases, schedule delays, and performance problems 

during production.



Table 2 shows the relationship between design stability and 

manufacturing knowledge at key junctures and the outcomes for the DOD 

programs we reviewed. To measure design stability at the start of the 

system demonstration phase, knowledge point 2, we determined the 

percentage of the productï¿½s engineering drawings that had been 

completed by the critical design review. In DOD programs, after the 

critical design review, the system design is released to manufacturing 

to begin building the production representative prototypes for the 

system demonstration phase. To measure producibility at the production 

decision, knowledge point 3, we determined whether the critical 

manufacturing processes were in statistical control at that time. We 

compared this information with best practices. The cost and schedule 

experiences of the program since the start of system demonstration are 

also shown.



Table 2: Attainment of Design and Manufacturing Knowledge in DOD 

Programs and the Program Outcomes:



Weapon system: Best practice; Percentage of drawings completed prior to 

manufacturing: At least 90 percent of drawings completed; Percentage of 

critical manufacturing processes in control at production: All critical 

processes in statistical control; Program experience since system 

demonstration started: Meet cost and schedule targets.



Weapon system: AIM-9X (air to air missile); Percentage of drawings 

completed prior to manufacturing: 95 percent; Percentage of critical 

manufacturing processes in control at production: Unknown[A]; Program 

experience since system demonstration started: 4 percent unit cost 

increase,; 1-month production delay.



Weapon system: FA-18 E/F fighter; Percentage of drawings completed 

prior to manufacturing: 56 percent[B]; Percentage of critical 

manufacturing processes in control at production: 78 percent; Program 

experience since system demonstration started: No unit cost increase,; 

3-month production delay.



Weapon system: F-22 fighter; Percentage of drawings completed prior to 

manufacturing: 26 percent; Percentage of critical manufacturing 

processes in control at production: 44 percent; Program experience 

since system demonstration started: 23 percent unit cost increase,; 18-

month production delay.



Weapon system: Patriot Advanced Capability (PAC-3) missile; Percentage 

of drawings completed prior to manufacturing: 21 percent; Percentage of 

critical manufacturing processes in control at production: 35 percent; 

Program experience since system demonstration started: 159 percent unit 

cost increase,; 39-month production delay.



Weapon system: Advanced Threat Infrared Countermeasures/Common Missile 

Warning System (ATIRCM/CMWS); Percentage of drawings completed prior to 

manufacturing: 21 percent; Percentage of critical manufacturing 

processes in control at production: 0; Program experience since system 

demonstration started: 182 percent unit cost increase,; 34-month 

production delay.



[A] While AIM-9X used statistical process control on a limited basis, 

we believe other factors contributed to a successful production outcome 

to date. Other factors included early achievement of design stability, 

early identification of key characteristics and critical manufacturing 

processes, use of established manufacturing processes for components 

common to other weapon systems, design trade-offs to enhance 

manufacturing capability, and a product design less vulnerable to 

variations in manufacturing processes.



[B] F/A-18 E/F had 56 percent of drawings completed but also had 

completed most of the higher-level assembly drawings. The combination 

of these drawings with the fact the aircraft was a variant of 

previously fielded F-18 aircraft models provided the program a 

significant amount of knowledge that the design was stable at the start 

of system demonstration.



Source: DOD program offices and Selected Acquisition Reports.



[End of table]



As shown in the table, the AIM-9X and FA-18 E/F programs had captured a 

significant amount of design knowledge at the start of system 

demonstration and manufacturing knowledge by the start of production. 

In each of those programs, product developers had the advantage of 

prior versions of the systems. These programs came very close to 

meeting their original cost and schedule estimates for product 

development. The other three programs, F-22, PAC-3, and ATIRCM/CMWS, 

had less knowledge at each key junctures. Their development cost and 

schedule results significantly exceeded estimates. Specific details on 

the AIM-9X, F-22, and ATIRCM/CMWS program experiences follow.



AIM-9X Program Experience:



The AIM-9X program began in 1994, continuing the long-term evolution of 

the AIM-9 series of short-range air-to-air missiles. In 1999, after 

developing and testing a number of engineering prototype missiles, the 

program held a critical design review to determine if the program was 

ready to begin initial manufacturing of a production representative 

prototype for system demonstration. At this review, about 95 percent of 

the eventual engineering drawings were completed--a stable design by 

best practices. Because 

AIM-9X was the next generation in this family of missiles, the program 

had significant knowledge on how to produce the missile. At the 1999 

critical design review, the estimated development and production costs 

totaled $2.82 billion. As of December 2001, the estimate was $2.96 

billion, less than a 5 percent increase.



F-22 Program Experience:



The F-22 program began detailed design efforts in 1991 when it entered 

a planned 8-year product development phase. In 1995, about the expected 

midpoint of the phase, the program held its critical design review to 

determine if the design was stable and complete. Despite having only 

about a quarter of the eventual design drawings completed for the 

system, the program declared the design to be stable and ready to begin 

initial manufacturing. At that time, the program office had estimated 

the cost to complete the development program at $19.5 billion. However, 

the program did not complete 90 percent of its drawings for the 

aircraft until 1998, 

3 years into the system demonstration phase. During the building of the 

initial aircraft, several design and manufacturing problems surfaced 

that affected the deliveries of major sections of the aircraft. Large 

sections were delivered incomplete to final assembly and had to be 

built out of the planned assembly sequence.



In 1997, an independent review team examined the program and determined 

the product development effort was underestimated. The team found that 

building the first three aircraft was taking substantially more labor 

hours than planned. Between 1995 and 1998, the development estimate for 

the F-22 increased by over $3.3 billion and the schedule slipped by a 

year. Achieving design stability late has contributed to further cost 

increases. As of December 2001, the estimated development cost was 

$26.1 billion, a 34 percent increase since the critical design review 

was held in 1995.



While the program attributes some production cost increases to a 

reduction in F-22 quantities, it has been significantly affected by 

design and manufacturing problems that started during development. The 

independent review team evaluated the cost impact on the production 

aircraft that would likely occur because of cost and schedule problems 

in development and found that production aircraft would have to begin 

later, at a slower pace, and cost more than expected. The team 

estimated that production costs could increase by as much as $13 

billion if savings were not found. The Air Force subsequently increased 

the estimate to more than $19 billion in cost savings required to avoid 

cost increases. In 2001, when the F-22 limited production decision was 

made, the program had less knowledge about the aircraftï¿½s reliability 

and manufacturing processes than more successful cases. For example, at 

its limited production decision, it had only 44 percent of its critical 

manufacturing processes in control. In September 2001, the program 

reported that overall production cost would likely increase by more 

than $5.4 billion. This estimate was based on the effort needed so far 

to build the aircraft during product development.



ATIRCM/CMWS Program Experience:



Since it began in 1995, the ATIRCM/CMWS program has had significant 

cost growth and schedule delays during product development. The product 

developer held a major design review in 1997. Like the F-22, the review 

demanded less proof about the productï¿½s design in the form of 

engineering drawings before deciding to begin initial manufacturing. At 

that time, only 21 percent of the engineering drawings had been 

completed, and it was still unknown whether the design would meet the 

requirements. In fact, the program knew that a major redesign of a 

critical component was needed. Despite this, the program office deemed 

the risk acceptable for moving the program forward to begin 

manufacturing prototypes. Over the next 2 years, the program 

encountered numerous design and manufacturing problems. It was not 

until 1999, about 2 years after the critical design review, that 

program officials felt that the design had stabilized; however, by this 

time, the product development cost had increased 160 percent and 

production had been delayed by almost 3 years.



ATIRCM/CMWS is scheduled to begin limited production in early 2002, but 

without the same degree of assurance as the more successful programs 

that the product can be manufactured within cost, schedule, and quality 

targets. The program has not yet determined if manufacturing processes 

needed to build the product are in control. Many of the development 

units were built by hand, in different facilities, and with different 

processes and personnel. Program officials stated that because they did 

not stabilize the design until late in development, manufacturing 

issues were not adequately addressed. Since 1997, the estimated unit 

cost for the system has increased by 182 percent.



[End of section]



Chapter 3 Best Practices Enable Timely Capture of Design and 

Manufacturing Knowledge:



Leading commercial companies have been successful in achieving product 

development goals because they have found ways to enable the capture of 

design and manufacturing knowledge about the products they are 

developing in a timely way. We found two practices that allowed leading 

commercial companies to capture necessary knowledge for product 

development. First, they established a framework of evolutionary 

product development that limited the amount of design and manufacturing 

knowledge that had to be captured. This framework limited the design 

challenge for any one new product development by requiring risky 

technology, design, or manufacturing requirements to be deferred until 

a future generation of the product. Second, each company (1) employed a 

disciplined product development process that brought together and 

integrated all of the technologies, components, and subsystems required 

for the product to ensure the design was stable before entering product 

demonstration and (2) demonstrated the product was reliable and 

producible using proven manufacturing processes before entering 

production.



The product development process includes tools that both capture 

knowledge and tie this knowledge to decisions about the productï¿½s 

design and manufacturing processes before making commitments that would 

significantly affect company resources. For example, during system 

integration, each leading commercial company used various forms of 

prototypes and information from predecessor products to stabilize the 

productï¿½s design and identify critical processes, then used a decision 

review that required agreements from key stakeholders that the 

requisite design knowledge was captured in making a decision to move 

into system demonstration. During system demonstration, each company 

used statistical process control and reliability testing to ensure the 

product could be produced affordably and would be reliable, then used a 

similar decision review that required agreements from key stakeholders 

that the requisite knowledge was captured when deciding to move into 

production.



The Department of Defense (DOD) programs that we reviewed used some of 

these practices to varying degrees and experienced predictable 

outcomes. For example, the AIM-9X and F/A-18 E/F programs were 

evolutionary in nature, modifications of existing products with a 

manageable amount of new technological or design challenges. They also 

gathered design and manufacturing knowledge, although not to the extent 

we found at commercial companies. Finally, they held program reviews 

and ensured that the design and manufacturing knowledge was captured 

before moving forward. They had relatively successful outcomes. The 

other DOD programs--the F-22, ATIRCMS, and PAC-3--did not closely 

approximate best practices in capturing design or manufacturing 

knowledge during product development. They took on greater design 

challenges, had program reviews that were not supported by critical 

design and manufacturing knowledge, and made decisions to advance to 

the next phases of development without sufficient design and 

manufacturing knowledge.



Leading Commercial Companies Use Evolutionary Product Development 

Framework to Reduce Development Risks:



A key to the success of commercial companies was using an evolutionary 

approach to develop a product. This approach permitted companies to 

focus more on design and development with a limited array of new 

content and technologies in a program. It also ensured that each 

company had the requisite knowledge for a productï¿½s design before 

investing in the development of manufacturing processes and facilities. 

Companies have found that trying to capture the knowledge required to 

stabilize the design of a product that requires significant amounts of 

new content is an unmanageable task, especially if the goal is to 

reduce cycle times and get the product into the marketplace as quickly 

as possible. Design elements not achievable in the initial development 

were planned for subsequent development efforts in future generations 

of the product, but only when technologies were proven to be mature and 

other resources were available.



Commercial companies have implemented the evolutionary approach by 

establishing time-phased plans to develop new products in increments 

based on technologies and resources achievable now and later. This 

approach reduces the amount of risk in the development of each 

increment, facilitating greater success in meeting cost, schedule, and 

performance requirements. In effect, these companies evolve products, 

continuously improving their performance as new technologies and 

methods allow. These evolutionary improvements to products eventually 

result in the full desired capability, but in multiple steps, 

delivering a series of enhanced interim capabilities to the customer 

more quickly.



Historically, DODï¿½s approach has been to develop new weapon systems 

that often attempt to satisfy the full requirement in a single step, 

regardless of the design challenge or the maturity of technologies 

necessary to achieve the full capability. Under this single-step 

approach, a war fighter can wait over 15 years to receive any improved 

capability. Figure 5 shows a notional comparison between the single-

step and evolutionary approaches.



Figure 5: Notional Single-Step and Evolutionary Approaches to 

Developing New Products:



[See PDF for image]



Source: GAOï¿½s analysis and DOD acquisition guidance.



[End of figure]



Each commercial company we visited used the evolutionary approach as 

the primary method of product development. General Electric builds on 

the basic capability of a fielded product by introducing proven 

improvements in capability from its advanced engineering development 

team. General Electric considers the introduction of immature 

technologies into fielded products or new engine development programs 

as a significant cost and schedule risk. Its new product development 

process is primarily focused on reducing and managing risk for design 

changes and product introductions. Cummins and Hewlett Packard managers 

indicated that, in the past, their companies learned the hard way by 

trying to make quantum leaps in product performance and by including 

immature technologies. Now, both companies have new product development 

processes that actively manage the amount of new content that can be 

placed on a new product development effort. Caterpillar also limits new 

content on its new products as a way to more successfully and cost-

effectively develop new, but evolutionary, products. Even during the 

development of its 797 mining truck, which it considered a major design 

challenge, it did not require the truck to achieve capabilities--such 

as prognostics for better maintenance--that it could not demonstrate or 

validate in the design in a timely manner.



Of the five DOD programs we reviewed, two--the F/A-18-E/F and the 

AIM-9X--were variations of existing products--the F/A-18-C/D and the 

AIM-9M--and the programs made a commitment to use existing technologies 

and processes as much as possible. These two programs had relatively 

successful cost and schedule outcomes. They represented an exception to 

the usual practice in DOD. The overwhelming majority of DODï¿½s major 

acquisitions today require major leaps in capability over their 

predecessors or any other competing weapon systems, with little 

knowledge about the resources that will be required to design and 

manufacture the systems. Decisions are continually made throughout 

product development without knowing the cost and schedule 

ramifications.



Leading Commercial Companies Use a Product Development Process to 

Capture Design and Manufacturing Knowledge for Decision Making:



Leading commercial companies we visited had spent significant amounts 

of time and resources to develop and evolve new product development 

processes that ensured design and manufacturing knowledge was captured 

at the two critical decision points in product development: when the 

productï¿½s design was demonstrated to be stable--knowledge point 2--and 

when the product was demonstrated to be producible at an affordable 

cost--knowledge point 3. The process established a disciplined 

framework to capture specific design and manufacturing knowledge about 

new products. Companies then used that knowledge to make informed 

decisions about moving forward in a new product development program. 

Commercial companies tied this knowledge to decisions about the 

productsï¿½ design and manufacturing processes before making commitments 

that would significantly impact company resources. Each commercial firm 

we visited had a new product development process that was prominent and 

central to the firmï¿½s successes. It included three aspects: (1) 

activities that led to the capture of specific design knowledge, (2) 

activities that led to the capture of specific manufacturing and 

product reliability knowledge, and (3) decision reviews to determine if 

the appropriate knowledge was captured to move to the next phase.



Design Knowledge Should Be Captured before Entering Product 

Demonstration:



To ensure that the productï¿½s design was stable before deciding to 

commit additional resources to product demonstration, commercial 

companies demanded knowledge, either from existing product information 

or by building engineering prototypes. They also used a disciplined 

design review process to examine and verify the knowledge that had 

culminated at the end of product integration, This design review 

process required agreement from stakeholders that the product design 

could be produced and would satisfy the customerï¿½s requirements. 

Stakeholders included design engineers, manufacturing or production 

personnel, and key supplier representatives who used engineering 

drawings, supported by test results and engineering data, as a key 

indicator of the designï¿½s stability. Once the program achieved a stable 

design, the certainty of their cost and schedule estimates was 

substantially increased, allowing them to plan the balance of the 

product development program with high confidence. Table 3 shows the 

activities required to capture design knowledge that leads to executive 

decisions about whether to transition to the next phase of development.



Table 3: Activities to Capture Design Knowledge and Make Decisions:



Knowledge: Design is stable and performs as expected (knowledge point 

2): Activities to Achieve Stable Design Knowledge; * Limit design 

challenge - The initial design challenge is limited to a product that 

can be developed and delivered quickly and provide the user with an 

improved capability. A time-phased plan is used to develop improved 

products--future generations--in increments as technologies and other 

resources become available.; * Demonstrate design meets requirements -

The productï¿½s design is demonstrated to meet the userï¿½s requirements. 

For a new product that is not based on an existing product, prototypes 

are built and tested. If the product is a variant of an existing 

product, companies often used modeling and simulation or prototypes at 

the component or subsystem level to demonstrate the new productï¿½s 

design.; * Complete critical design reviews - Critical design reviews 

are used to assess whether a productï¿½s design meets requirements and is 

ready to start initial manufacturing. They are conducted for the 

system, subsystems, and components to assess design maturity and 

technical risk.; * Stakeholders agree drawings complete and producible 

- The agreement by stakeholders (engineers, manufacturers, and other 

organizations) is used to signify confidence that the design will work 

and the product can be built.; * Executive level review to begin 

initial manufacturing - Corporate stakeholders meet and review relevant 

product knowledge, including design stability, to determine whether a 

product is ready to initiate manufacturing of production representative 

prototypes used during system demonstrations. The decision is tied to 

the capture of knowledge..



[End of table]



Demonstrating the Design Helped Achieve Stability:



A key tool used by each company to ensure that a productï¿½s design was 

stable by the end of the product integration phase was a demonstration 

that the design would meet requirements. The companies visited 

indicated that prototypes at various system levels were the best way to 

demonstrate that the productï¿½s design would work. If the product under 

development was an incremental improvement to existing products, such 

as the next generation of a printer or engine, these companies used 

virtual prototypes for any components that were being used for the 

first time. If the product included more new content or invention, 

fully integrated prototypes were frequently used to demonstrate that 

the design met requirements. Prototypes at this stage in development 

were typically not built in a manufacturing facility. This allowed 

demonstrations of the design before the companies made more costly 

investments in manufacturing equipment and tooling to build production 

representative prototypes for the demonstration phase. Table 4 shows an 

example of the types and purposes for various kinds of prototypes used 

by Cummins Inc. depending on the amount of knowledge it needed to 

capture and the point it was in the development process. Prototypes 

were used by commercial companies throughout the product development 

process and not just during product integration.



Table 4: Examples of Prototypes Used by Cummins Inc. at Various Stages 

of Product Development:



Prototype; Product integration: Engineering prototypes (virtual or 

physical); Product demonstration: Production representative 

prototypes; Production: Initial products.



Purpose; Product integration: Demonstrate form, fit and function, and a 

stable design; Product demonstration: Demonstrate the product is 

capable, reliable, and manufacturing processes in statistical control; 

Production: Demonstrate ready for full rate production.



Build environment; Product integration: Engineering; Product 

demonstration: Manufacturing; (1st set of production tooling); 

Production: Production (all rate tooling).



[End of table]



Cummins, the world sales leader in diesel engines over 200 horsepower, 

effectively uses prototypes to ensure that a design is stable and 

believes in the value of prototyping throughout product development. A 

Cummins representative stated that not using prototypes becomes a 

matter of ï¿½pay me now or pay me later,ï¿½ meaning that it is far less 

costly to demonstrate a productï¿½s design early in development with 

prototypes, concepts, and analyses than to incur the cost of 

significant design changes after a product has entered production--a 

much more costly environment to make changes. Cummins built and tested 

12 engineering concept prototype engines for its Signature 600 engine, 

a new concept, 600 horsepower, overhead cam diesel engine that 

represented a quantum leap in performance beyond Cumminsï¿½ existing 

products. These prototypes were built using production-like tooling and 

methods using production workers. In addition to using engineering 

prototypes during the product integration phase of product development, 

Cummins and other companies we visited used other prototypes--such as 

production representative prototypes--in the remaining product 

development phases before production, as shown in table 4, to 

demonstrate product reliability and process control. Prior to reaching 

production for its Signature 600 engine, Cummins used many prototypes 

to complete hundreds of thousands of test hours, accumulating millions 

of test miles.



Caterpillar, a major manufacturer of heavy equipment, has a continuous 

product improvement philosophy. That is, it tries to develop new 

products that increase the capabilities of existing product lines, but 

it limits the amount of new content on any one product development 

because new content inherently increases design risk. In evolving its 

products this way, Caterpillar is able to use modeling and simulation 

prior to initial manufacturing because it has existing products to 

provide a baseline of knowledge and a good benchmark for assessing the 

simulated performance. In addition, with knowledge of existing 

components, it can focus attention on maturing the new content, the 

higher risk element of the new product. When Caterpillar developed the 

797 mining truck, a new 360-ton payload truck design, it demonstrated 

design stability by identifying the critical components and building 

engineering prototypes of them for reliability testing and 

demonstration of the design before beginning initial manufacturing. 

This knowledge, coupled with vast experience in manufacturing trucks, 

ensured the stability of the 797-truck design before initial 

manufacturing started. Caterpillar was able to deliver this design in 

18 months after the product development was started.



Disciplined Reviews and Stakeholder Agreements Supported the Capture of 

Design Knowledge:



The commercial companies we visited understood the importance of having 

disciplined design reviews and getting agreement from the stakeholders 

that the productï¿½s design had been demonstrated to meet requirements 

before beginning initial manufacturing. Each company had a design 

review process that began at the component level, continued through the 

subsystem level, and culminated with a critical design review of the 

integrated system to determine if the product was ready to progress to 

the next phase of development. In addition to design engineers, a 

cross-functional team of stakeholders in the process included key 

suppliers, manufacturing representatives, and service and maintenance 

representatives. From past experience, commercial companies have 

discovered that cross-functional teams provide a complete perspective 

of the product. While design engineers bring important skills and 

experience to creating a product design, they may not be aware of 

manufacturing issues, available technologies, or manufacturing 

processes, and they may design a product that the company cannot afford 

to produce or maintain.



The productï¿½s design is stable when all stakeholders agree that 

engineering drawings are complete and that the design will work and can 

be built. A commercial company considers engineering drawings[Footnote 

4] to be a good measure of the demonstrated stability of the productï¿½s 

design because they represent the language used by engineers to 

communicate to the manufacturers the details of a new product design--

what it looks like, how its components interface, how it functions, how 

to build it, and what critical materials and processes are required to 

fabricate and test it. The engineering drawing package released to 

manufacturing includes items such as the schematic of the productï¿½s 

components, interface control documents, a listing of materials, 

notations of critical manufacturing processes, and testing 

requirements. It is this package that allows a manufacturer to build 

the product in the manufacturing facility.



In developing the Signature 600, Cummins used cross-functional design 

teams that included stakeholders from suppliers, machine tool 

manufacturers, foundry and pattern makers, purchasing, finance, 

manufacturing engineering, design engineering, and other technical 

disciplines. Signature 600 components were designed with the key 

suppliers co-located at the Cummins design facility. Likewise, 

Caterpillar said that early supplier and manufacturing involvement was 

critical to success and that engineering drawings were signed by design 

and manufacturing stakeholders. Caterpillar representatives said that 

signing the drawings was a certification that the design could be 

manufactured the next day, if necessary.



Executive Level Reviews Were Required to Begin Initial Manufacturing:



Each commercial company, after capturing specific design knowledge, had 

an executive level review at the decision point to determine if the 

product design had sufficiently progressed to permit a transition from 

product integration to product demonstration. This decision point used 

the knowledge captured as exit criteria for moving to the next phase of 

development. For example, to demonstrate the product design was stable 

and ready to move from integration to demonstration, the design had to 

be demonstrated, at least 90 percent of the engineering drawings had to 

be completed, design reviews had to be completed, and stakeholders had 

to agree the design was complete and producible. If the design team 

could not satisfy the exit criteria, then other options had to be 

considered. Options included canceling the development program, 

delaying the decision until all criteria were met, or moving ahead with 

a detailed plan to achieve criteria not met by a specific time when 

leadership would revisit the other options. One company emphasized that 

if a major milestone is delayed, an appropriate adjustment should be 

made to the end date of the program, thereby avoiding compressing the 

time allotted for the rest of product development and managing the 

risks that subsequent milestones will be missed.



This decision point coincides with the companiesï¿½ need to increase 

investments in the product development and continue to the next phase. 

For this reason, the decision point was considered critical to 

achieving success in product development and could not be taken 

lightly. For example, transitioning from the integration to the 

demonstration phase requires a significant investment to start building 

and testing production representative prototypes in a manufacturing 

environment. This requires establishing a supplier base and purchasing 

materials. In addition, establishing tooling and manufacturing 

capability is also required. After a product passes this decision point 

and added investments are made, the cost of making changes to the 

product design also increases significantly. Therefore, commercial 

companies strive to firm the design as early in the process as possible 

when it is significantly cheaper to make changes.



Manufacturing and Product Reliability Knowledge Should Be Captured 

before Starting Production:



We found that leading commercial companies used two tools to capture 

knowledge that a productï¿½s design was reliable and producible within 

cost, schedule, and quality targets before making a production 

decision. These tools are (1) a quality concept that uses statistical 

process control to bring critical manufacturing processes under control 

so they are repeatable, sustainable, and consistently producing parts 

within the quality tolerances and standards of the product and (2) 

product tests in operational conditions that ensure the system would 

meet reliability goals-the ability to work without failure or need of 

maintenance for predictable intervals. Company officials told us that 

these two tools enabled a smooth transition from product development to 

production, resulting in better program outcomes. Companies employed 

these tools on production representative prototypes, making the 

prototypes a key ingredient to successful outcomes. Table 5 shows the 

activities required to capture manufacturing knowledge that leads to 

executive decisions about whether to transition from product 

development into production.



Table 5: Activities to Capture Manufacturing Knowledge and Make 

Decisions:



Knowledge: Product can be produced within cost, schedule, and quality 

targets (knowledge point 3): Activities to Achieve Manufacturing 

Knowledge; * Identify key system characteristics and critical 

manufacturing processes - Key product characteristics and critical 

manufacturing processes are identified. Because there can be thousands 

of manufacturing processes required to build a product, companies focus 

on the critical processes--those that build parts that influence the 

productï¿½s key characteristics such as performance, service life, or 

manufacturability.; * Determine processes in control and capable - 

Statistical process control is used to determine if the processes are 

consistently producing parts. Once control is established, an 

assessment is made to measure the processï¿½s ability to build a part 

within specification limits as well as how close the part is to that 

specification. A process is considered capable when it has a defect 

rate of less than 1 out of every 15,152 parts produced.; * Conduct 

failure modes and effects analysis - Bottom-up analysis is done to 

identify potential failures for product reliability. It begins at the 

lowest level of the product design and continues to each higher tier of 

the product until the entire product has been analyzed. It allows early 

design changes to correct potential problems before fabricating 

hardware.; * Set reliability growth plan and goals - A productï¿½s 

reliability is its ability to perform over an expected period of time 

without failure, degradation, or need of repair. A growth plan is 

developed to mature the productï¿½s reliability over time through 

reliability growth testing so that it has been demonstrated by the time 

production begins.; * Conduct reliability growth testing -Reliability 

growth is the result of an iterative design, build, test, analyze, and 

fix process for a productï¿½s design with the aim of improving the 

productï¿½s reliability over time. Design flaws are uncovered and the 

design of the product is matured.; * Conduct executive level review to 

begin production - Corporate stakeholders meet and review relevant 

product knowledge, including manufacturing and reliability knowledge, 

to determine whether a product is ready to begin production. The 

decision is tied to the capture of knowledge..



[End of table]



Statistical Process Control Is Important to Controlling Critical 

Manufacturing Processes:



Commercial companies rely on statistical process control data to track, 

control, and improve critical manufacturing processes before production 

begins. Bringing processes under statistical control reduces variations 

in parts manufactured, thus reducing the potential for defects. Product 

variation has been called the ï¿½silent killerï¿½ on the manufacturing 

floor because it can result in defects that require additional 

resources to either rework or scrap the product. Products fielded with 

defects may have degraded performance, lower reliability, or increased 

support costs. Experience has taught commercial companies that it is 

less costly--in terms of time and money--to eliminate product variation 

by controlling manufacturing processes than to perform extensive 

inspection after a product is built. Because thousands of manufacturing 

processes can be required to build a product, companies focus on the 

critical processes--those that build parts that influence the productï¿½s 

performance, service life, or manufacturability. Therefore, when design 

engineers are designing the new product, they must identify its key 

characteristics so that manufacturing engineers can identify and 

control critical manufacturing processes. Key product characteristics 

and critical manufacturing processes are noted on the engineering 

drawings and work instructions that are released to manufacturing.



Once critical processes are identified, companies perform capability 

studies to ensure that a process will produce parts that meet 

specifications. These studies yield a process capability index (Cpk), a 

measure of the processï¿½s ability to build a part within specified 

limits. The index can be translated into an expected product defect 

rate. The industry standard is to have a Cpk of 1.33 or higher, which 

equates to a probability that 99.99 percent of the parts built on that 

process will be within the specified limits. Four of the five[Footnote 

5] companies we visited wanted their critical processes at a minimum of 

a 1.33 Cpk and many had goals of achieving higher Cpks. Table 6 shows 

various Cpk values and the defect rate associated with each value. The 

table also shows the higher the Cpk, the lower the defect rate.



Table 6: Cpk Index and Probability of a Defective Part:



Manufacturing process capability (Cpk): Cpk - .67 (not capable); 

Associated defect rate: 1 in 22 parts produced.



Manufacturing process capability (Cpk): Cpk - 1.0 (marginally capable); 

Associated defect rate: 1 in 370 parts produced.



Manufacturing process capability (Cpk): Cpk - 1.33 (industry standard); 

Associated defect rate: 1 in 15,152 parts produced.



Manufacturing process capability (Cpk): Cpk - 2.0 (industry growth 

goal); Associated defect rate: 1 in 500,000,000 parts produced.



[End of table]



Cpk values also have an additive effect on various individual parts 

when each part is integrated into the final product. For example, a 

product composed of 25 parts, where each part is produced on a 

manufacturing process with a Cpk of 0.67, has a 95.5 percent 

probability that each part will be defect free. However, when all 25 

parts are assembled into the final product, the probability that the 

final product will be defect free is only 

32 percent. In comparison, if the same parts are produced with 

manufacturing processes at a Cpk of 1.33, the probability of each part 

being defect free is 99.99 percent. When these same 25 parts are 

assembled into the final product, the probability that the final 

product will be defect free is

99.8 percent. This comparison illustrates the impact that having 

manufacturing processes in control has on the amount of rework and 

repair that would be needed to correct defects and make the product 

meet its specifications.



Cummins uses statistical process control data to measure a productï¿½s 

readiness for production. In developing the new Signature 600 diesel 

engine, Cummins included manufacturing engineers and machine tool and 

fixture suppliers in the design decision process as the engine concept 

was first being defined. Cummins built production representative 

prototypes of its engines to demonstrate that the design and the engine 

hardware would perform to requirements. These prototypes represented 

the first attempt to build the product solely using manufacturing 

personnel, production tooling, and production processes. Cummins used 

the knowledge captured from these and subsequent prototypes to refine 

and eventually validate the manufacturing processes for the engine. 

This process of employing statistical process control techniques on 

prototype engines verified that the manufacturing processes were 

capable of manufacturing the product to high quality standards within 

established cost and schedule targets.



Other companies we visited emphasized the importance of controlling 

manufacturing processes before committing to production. For example, 

Xerox captures knowledge about the producibility of its product early 

in the design phase. By production, it strives to have all critical 

manufacturing processes for the product--including key suppliersï¿½ 

processes--in control with a Cpk index of at least 1.33. Xerox achieves 

this by building production representative prototypes and by requiring 

suppliers of key components and subassemblies to produce an adequate 

sample of parts to demonstrate the suppliersï¿½ processes can be 

controlled, usually before the parts are incorporated into the 

prototypes. General Electric Aircraft Engines has digitally captured, 

and made available to design engineers, Cpk data on almost all of its 

manufacturing processes and it strives to have critical processes in 

control to a point where they will yield no more than 

1 defect in 500 million parts, a Cpk of 2.0. Other companies, such as 

Caterpillar and Hewlett Packard, told us that getting manufacturing 

processes in control prior to production is key to meeting cost, 

schedule, and quality targets. Each of the companies visited used this 

as an indicator of the productï¿½s readiness for production and 

emphasized the importance of having critical manufacturing processes 

under control by the start of production.



Demonstrating Product Reliability Indicates the Product Is Ready for 

Production:



A product is reliable when it can perform over a specified period of 

time without failure, degradation, or need of repair. Reliability is a 

function of the specific elements of a productï¿½s design. Making design 

changes to achieve reliability requirements after production begins is 

inefficient and costly. Reliability growth testing provides visibility 

over how reliability is improving and uncovers design problems so fixes 

can be incorporated before production begins.



In general, reliability growth is the result of an iterative design, 

build, test, analyze, and fix process. Prototype hardware is key to 

testing for reliability growth. Initial prototypes for a complex 

product with major technological advances have inherent deficiencies. 

As the prototypes are tested, failures occur and, in fact, are desired 

so that the productï¿½s design can be made more reliable. Reliability 

improves over time with design changes or manufacturing process 

improvements. The earlier this takes place, the less impact it will 

have on the development and production program. Companies we visited 

matured a productï¿½s reliability through these tests and demanded proof 

that the product would meet the customerï¿½s reliability expectations 

prior to making a production decision.



Improvements in the reliability of a productï¿½s design can be measured 

by tracking a key reliability metric over time. This metric compares 

the productï¿½s actual reliability to a growth plan and ultimately to the 

overall reliability goal. Several commercial companies we visited began 

gathering this data very early in development and tracked it throughout 

development. The goal was to demonstrate the product would meet 

reliability requirements before starting full rate production.



Caterpillar establishes a plan to grow and demonstrate the productï¿½s 

reliability before fabrication of a production representative prototype 

begins. Before Caterpillar starts making parts, it estimates the 

productï¿½s reliability in its current stage of development based on 

knowledge captured from failure modes and effects analysis,[Footnote 6] 

component prototype testing, and past product experience. This 

information marks the starting point for the productï¿½s reliability 

growth plan and is the basis for assessing whether the plan is 

achievable by production. If Caterpillar believes the risks are too 

high and the goal cannot be achieved on time, decision makers assess 

trade-offs between new and existing components to reduce the risks to a 

more manageable level. Trade-offs might be made if the productï¿½s 

performance still fails to meet requirements. If trade-offs are not 

possible, decision makers may decide not to go forward with the 

development. Once Caterpillar has established this plan, it tracks 

demonstrated reliability against it as a management tool to measure 

progress. It sets an interim reliability milestone and expects to be at 

least halfway toward the expected goal by the time it begins to build 

production units. Caterpillar has learned from experience that it will 

achieve the full reliability goal by full production if it meets the 

interim goal by the time it produces pilot production units. If the 

reliability is not growing as expected, then decisions about changing 

or improving the design must be addressed.



Caterpillar improves the productï¿½s reliability during development by 

testing prototypes, uncovering failures, and incorporating design 

changes. According to Caterpillar officials, the production decision 

will be delayed if they are not on track to meeting their reliability 

goal. These officials told us that Caterpillar maintains the philosophy 

of first getting the design right, then producing it as quickly and 

efficiently as possible. They emphasized that demonstrating reliability 

before production minimized the potential for costly design changes 

once the product is fielded.



Executive Level Reviews Are Conducted to Begin Production:



The commercial companies, after capturing specific manufacturing 

knowledge, had executive level reviews to determine if the product 

development had sufficiently progressed to permit a transition into 

production. Executives used the knowledge captured as exit criteria for 

the transition. For example, to demonstrate the product was ready for 

production, critical processes had to be in control and testing should 

have demonstrated the product reliability. If the design team could not 

satisfy the exit criteria, then other options had to be considered. The 

production decision led to increased investments for materials and 

resources such as additional tooling to build the product at a planned 

rate, facilities, people, training and support.



When DOD Programs More Closely Approximated Best Practices, Outcomes 

Were Better:



Our analysis of DOD programs showed that those more closely 

approximating best practices had better outcomes. The F/A-18 E/F 

fighter and the AIM-9X missile were based extensively on predecessor 

programs and employed similar tools to capture design and manufacturing 

knowledge at critical program junctures. These programs had 

demonstrated a significantly higher degree of design stability prior to 

entering system demonstration and committing to initial manufacturing 

when compared to other DOD weapon programs in our review. They also 

gained control of most of their manufacturing processes and 

demonstrated that the products were reliable before entering 

production. The success of these programs is best demonstrated by the 

fact that they have been close to meeting cost, schedule, and 

performance objectives. On the other hand, the PAC-3 missile, F-22 

fighter, and ATIRCM/CMWS programs did not use these best practices. 

These programs were not based on predecessor products or evolutionary 

in nature, and each productï¿½s full capability was expected in one step, 

with the first product off the production line. With this daunting 

task, these programs failed to demonstrate a stable design before 

committing to initial manufacturing, causing quality and labor 

problems. These programs also had much less knowledge about the 

manufacturability of their design when they entered production. As a 

result, they experienced significant increases in development costs and 

production delays usually at the expense of other DOD programs. Details 

on the five DOD programs follow.



AIM-9X Missile Program:



The AIM-9X development practices closely paralleled best practices used 

by the commercial companies we visited. The program achieved design 

stability before moving into system demonstration by incorporating 

mature technologies and components from other missiles and munitions, 

using engineering prototypes to demonstrate the design, holding a 

series of design reviews prior to the system level critical design 

review, and completing and releasing 95 percent of the engineering 

drawings at that time. Figure 6 shows the building of knowledge 

required to achieve a stable design on the AIM-9X.



Figure 6: Achieving Stability on AIM-9X Missile Program by Knowledge 

Point 2:



[See PDF for image]



Source: GAOï¿½s analysis.



[End of figure]



The AIM-9X program made extensive use of engineering prototypes to 

stabilize the missileï¿½s design before building production 

representative prototypes. Program officials stated that testing of 

engineering prototypes uncovered problems with missile design and 

manufacturing tooling early in the development, during system 

integration, allowing time to re-design and re-test in follow-on 

configurations. According to program officials, this not only helped 

stabilize the design before entering initial manufacturing but grew 

system reliability and reduced total ownership costs. The program also 

held design reviews for each of the major subsystems, allowing the 

program to achieve and demonstrate a stable design in July 1999, before 

beginning initial manufacturing of production representative 

prototypes.



While the AIM-9X used statistical process control only to a limited 

extent, other factors have allowed it to have a more successful 

production outcome to date. Program officials took steps to ensure that 

manufacturing aspects of the product were included in the design, 

including empowering a product leader with a manufacturing background, 

identifying the key characteristics and critical manufacturing 

processes early, making design trade-offs to enhance manufacturing 

capability, and demonstrating a robust design to make the product less 

vulnerable to variations in manufacturing process. In addition, the 

ability to achieve design stability at the critical design review 

allowed program officials to focus the system demonstration phase on 

maturing the manufacturing processes. Prior to committing to 

production, the program demonstrated that the product could be 

efficiently built using production processes, people, tools, and 

facilities to build prototypes. According to the former program 

manager, these steps gave the officials knowledge that a reliable 

product could be produced within cost and schedule targets prior to 

entering production. To date, the AIM-9X program has largely met its 

production targets.



F/A-18 E/F Program:



The F/A-18 E/F aircraft development program was able to take advantage 

of knowledge captured in developing and manufacturing prior versions of 

the aircraft. This evolutionary approach significantly contributed to 

the cost and schedule successes of this program. Because the F/A-18 E/

F was a variant of the older F/A-18 aircraft, the developer had prior 

knowledge of design and manufacturing problems. This knowledge, coupled 

with the use of modeling and computer-aided design software, helped 

create a design that was easier to manufacture. While the program did 

not fully use each of the best practices, it did embrace the concepts 

of capturing design and manufacturing knowledge early in the program.



During the programï¿½s critical design review, about 56 percent of the 

drawings were completed and, while the program did not meet the best 

practice of 90 percent complete, it did have additional drawing data of 

the F/A-18 E/F assemblies available for review at the critical design 

review. The Navy used early versions of the F/A-18 aircraft to 

demonstrate new component designs and new materials. In addition, the 

aircraft was designed to have 42 percent fewer parts than its 

predecessor, making its design more robust. The program also identified 

the critical manufacturing processes and collected statistical process 

control data early in product development. At the start of production, 

78 percent of these critical processes were in control. Unit costs for 

the F/A-18 E/F program have not grown since the critical design review 

and its schedule has been delayed by only 3 months.



F-22 Fighter Program:



The F-22 program is structured to provide the productï¿½s full capability 

with the first product off the production line--an extreme design 

challenge. This required the product design to include many new and 

unproven technologies, designs, and manufacturing processes. It did not 

demonstrate design stability until about 3 years after it held its 

critical design review. The program completed 3,070 initial engineering 

drawings at its critical design review in 1995, about 26 percent of the 

eventual drawings needed. It did not complete 90 percent of the 

necessary engineering drawings until 1998, after the first two 

development aircraft were delivered. Figure 7 shows the drawing 

completion history for the program.



Figure 7: History of Drawing Completion for the F-22 Program:



[See PDF for image]



Source: GAOï¿½s analysis.



[End of figure]



After its critical design review, the F-22 program encountered several 

design and manufacturing problems that resulted in design changes, 

labor inefficiencies, cost increases, and schedule delays. For example, 

delivery of the aft fuselage--the rear aircraft body section--was late 

for several of the test aircraft and two ground test articles because 

of late parts and difficulties with the welding process. According to 

the F-22 program office, design maturity and manufacturing problems 

caused a ï¿½rolling waveï¿½ effect throughout system integration and final 

assembly. Late engineering drawing releases to the factory floor 

resulted in parts shortages and work performed out of sequence. These 

events contributed to significant cost overruns and delays to aircraft 

deliveries to the flight test program.



The F-22 program initially had taken steps to use statistical process 

control data during development and gain control of critical 

manufacturing processes by the full rate production decision. In 

1998,[Footnote 7] we reported that the program had identified 926 

critical manufacturing processes and had almost 40 percent in control 2 

years before production was scheduled to begin. Although this did not 

match the standard set by commercial companies, it offered major 

improvements over what other DOD programs had attempted or achieved. 

Unfortunately, citing budgetary constraints and specific hardware 

quality problems that demanded attention, the program abandoned this 

best practices approach in 2000 with less than 50 percent of it 

critical manufacturing processes in control. Currently, the program is 

using post-assembly inspection to identify and fix defects rather than 

statistical process control techniques to prevent them. In March 

2002,[Footnote 8] we recommended that the F-22 program office monitor 

the status of critical manufacturing processes as the program proceeds 

toward high rate production. The program stated that it would assess 

the processes status as the program moves forward.



The program entered limited production despite being substantially 

behind its plan to achieve reliability goals. A key reliability 

requirement for the 

F-22 is mean time between maintenance, defined as the number of 

operating hours for the aircraft divided by the number of maintenance 

actions. The reliability goal for the F-22 is a 3-hour mean time 

between maintenance. The Air Force estimated that in late 2001, when 

the F-22 entered limited production, it should have been able to 

demonstrate almost 2 flying hours between maintenance actions. However, 

when it actually began limited production it could only fly an average 

of 0.44 hours between maintenance actions. In other words, the F-22 is 

requiring significantly more maintenance actions than planned. 

Additionally, the program has been slow to fix and correct problems 

that have affected reliability. To date, the program has identified 

about 260 different types of failures, such as main landing gear tires 

wearing out more quickly than planned, fasteners being damaged, and 

canopy delaminating. It has identified fixes for less than 50 percent 

of these failures. Ideally, the design fixes for the failures should be 

corrected prior to manufacturing production units.



PAC-3 Missile Program:



The PAC-3 missile did not achieve design stability until after the 

building of production representative prototypes for system 

demonstration began. At the programï¿½s critical design review, the PAC-

3 program had completed 980 engineering drawings--21 percent of the 

eventual drawings needed for the missile. Since then, almost 3,700 more 

drawings have been completed. The total number of drawings expected to 

represent the completed design grew from about 2,900 at the critical 

design review to almost 4,700 as of July 2001. This uncertainty in the 

expected drawings not only indicates that the design was not stable 

when initial manufacturing began but also shows that there was a 

significant lack of knowledge about the design. Figure 8 shows the 

design knowledge at the critical design review, when the decision was 

made to commit to initial manufacturing of the missile.



Figure 8: PAC-3 Design Knowledge at Critical Design Review:



[See PDF for image]



Source: GAOï¿½s analysis.



[End of figure]



Prototypes of the product design were not built before the critical 

design review or before initial manufacturing started to show that the 

design would work. Therefore, because of the immature design, initially 

manufactured development missiles were hand-made, took longer to build 

than planned, and suffered from poor quality. As a result, many design 

and manufacturing problems surfaced during system demonstration. 

Subsystems did not fit together properly, and many failed ground and 

environmental tests the first time. The contractor attributed $100 

million of additional cost to first time manufacturing problems.



Prior to entering limited production in 1999, the program had less than 

40 percent of the critical manufacturing processes in control for 

assembling the missile and the seeker. According to program officials, 

there was little emphasis during development or initial production on 

using statistical control on critical manufacturing processes. Most of 

the development missiles were built in specialty shops rather than in a 

manufacturing environment. The result was a lack of knowledge about 

whether the critical manufacturing processes could produce the product 

to established cost, schedule, and quality targets. This uncertainty is 

reflected in contractor estimates that more than 50 percent of the time 

charged to build the initial production missiles will be for 

engineering activities. Actual production labor is expected to account 

for about 30 percent of the charged time.



To further understand the problems on the PAC-3 program, we focused on 

its seeker subsystem, which is key to acquiring and tracking targets 

and represents a large percentage of the missileï¿½s cost. Currently, 

despite being in production, it is unclear whether the supplier of the 

seeker can produce it within cost, schedule, and quality targets. 

During development, the supplier had difficulty in designing and 

manufacturing this subsystem. It was not uncommon for seekers to be 

built, tested, and reworked seven or eight times before they were 

acceptable. The program entered production, despite these producibility 

issues. Now, even with 2 years of production experience, the supplier 

continues to have difficulty producing the seeker with acceptable 

quality. Data provided by the supplier in October 2001 showed that less 

than 25 percent of the seekers were being manufactured properly the 

first time and the rest had to be reworked, on average, four times.



ATIRCM/CMWS Program:



According to program officials, ATIRCM/CMWS did not have a stable 

design until about 2 years after the critical design review. A 

contributing factor to this was a lack of understanding about the full 

requirements for the new system at the critical design review in 1997. 

This led to a major redesign of the common missile warning systemï¿½s 

sensor. At the critical design review, only 21 percent of a productï¿½s 

engineering drawings had been completed. It did not complete 90 percent 

drawings--the best practice--until 1999. The immature design caused 

inefficiencies in manufacturing, rework, and delayed deliveries. In 

addition, between 1995 and 1999, the development contract target price 

increased by 165 percent.



The ATIRCM/CMWS program did not begin reliability growth testing until

4 years after its critical design review, leaving only 1 year to test 

the system prior to scheduled production. Program officials said that 

an immature design limited their ability to begin reliability testing 

earlier in development. About one-third of the way through the 

reliability growth test program, testing was halted because too many 

failures occurred in components such as the power supply, the high 

voltage electrical system, and the cooling system. According to a 

program official, the inability to demonstrate system reliability 

contributed to a production delay of about 

1 year. The program plans to build, develop, and test six additional 

development units during 2002 and 2003 that will incorporate design 

changes to fix the system failures. ATIRCM/CMWS plans to enter limited 

production in the early part of 2002 with significantly less knowledge 

about the designï¿½s producibility than commercial companies. The 

contractor does not use statistical process control and has not 

identified critical manufacturing processes. A production readiness 

review identified the lack of statistical process control as a major 

weakness that needs to be corrected.



[End of section]



Chapter 4 A Better Match of Policy and Incentives Is Needed to Ensure 

Capture of Design and Manufacturing Knowledge:



The Department of Defenseï¿½s (DOD) acquisition policy[Footnote 9] 

establishes a good framework for developing weapon systems; however, 

disciplined adherence, more specific criteria, and stronger acquisition 

incentives are needed to ensure the timely capture and use of knowledge 

in decision making. DOD changed its acquisition policy to emphasize 

evolutionary acquisition and establish separate integration and 

demonstration phases in the product development process. Its goal was 

to develop higher quality systems in less time and for less cost. 

However, DODï¿½s acquisition policy lacks detailed criteria for capturing 

and using design and manufacturing knowledge to facilitate better 

decisions and more successful acquisition program outcomes. As 

demonstrated by successful companies, using these criteria can help 

ensure that the right knowledge is collected at the right time and that 

it will provide the basis for key decisions to commit to significant 

increases in investment as product development moves forward.



While the right policy and criteria are necessary to ensure a 

disciplined, knowledge-based product development process, the 

incentives that influence the key players in the acquisition process 

will ultimately determine whether they will be used effectively. In 

DOD, current incentives are geared toward delaying knowledge so as not 

to jeopardize program funding. These incentives undermine a knowledge-

based process for making product development decisions. Instead, 

program managers and contractors push the capture of design and 

manufacturing knowledge to later in the development program to avoid 

the identification of problems that might stop or limit its funding. 

They focus more on meeting schedules than capturing and having the 

knowledge necessary to make the right decisions at those milestones. 

Such an approach invariably leads to added costs because programs are 

forced to fix problems late in development.



By contrast, commercial companies must develop high-quality products 

quickly or they may not survive in the marketplace. Because of this, 

they encourage their managers to capture product design and 

manufacturing knowledge to identify and resolve problems early in 

development, before making significant increases in their investment. 

Instead of a schedule-driven process, their process is driven by events 

that bring them knowledge: critical design reviews that are supported 

by completed engineering drawings and production decisions that are 

supported by reliability testing and statistical process control data. 

They do not move forward without the design and manufacturing knowledge 

needed to make informed decisions.



Acquisition Policy Lacks Specific Implementation Criteria:



Greater emphasis on evolutionary acquisitions and structuring the 

product development process into two phases--system integration and 

system demonstration--were good first steps for DOD to achieve its 

goals of buying higher quality systems in less time and for lower 

costs. However, DOD policy still lacks criteria to be used to capture 

specific design and manufacturing knowledge and does not require the 

use of that knowledge as exit criteria at key decision points to 

transition from system integration to system demonstration and then 

into production. In three of the five DOD program examples in chapter 

3, managers decided to move forward in development, even when 

developers had failed to capture design and manufacturing knowledge to 

support increased investments. As a result, these programs encountered 

significant increases in acquisition costs as well as delays in 

delivering capabilities to the war fighter.



Table 7 illustrates key criteria used by commercial companies that are 

currently lacking in DODï¿½s policy. The table shows the design and 

manufacturing knowledge needed to make more informed decisions. The 

capture of some of the important manufacturing and reliability 

knowledge should begin in the integration phase in order to have the 

full knowledge needed to make decisions at the end of the demonstration 

phase for transitioning into production.



Table 7: Analysis of DOD Acquisition Policy for Inclusion of Best 

Practices for Knowledge-based Design and Manufacturing Decisions:



Commercial criteria: X; Best practices to capture design knowledge by 

decision point to enter system demonstration phase: Use of key 

indicator to show design stability (90 percent of drawings completed); 

DOD criteria: [Empty].



Commercial criteria: X; Best practices to capture design knowledge by 

decision point to enter system demonstration phase: Limit design 

challenge prior to entering system integration; DOD criteria: X.



Commercial criteria: X; Best practices to capture design knowledge by 

decision point to enter system demonstration phase: Demonstrate the 

design meets requirements; DOD criteria: X.



Commercial criteria: X; Best practices to capture design knowledge by 

decision point to enter system demonstration phase: Complete critical 

design reviews; DOD criteria: [Empty].



Commercial criteria: X; Best practices to capture design knowledge by 

decision point to enter system demonstration phase: Obtain stakeholder 

agreements that drawings complete and producible; DOD criteria: 

[Empty].



Commercial criteria: X; Best practices to capture design knowledge by 

decision point to enter system demonstration phase: Hold decision 

review to begin initial manufacturing; DOD criteria: [Empty].



Best practices to capture design knowledge by decision point to enter 

system demonstration phase: Commercial criteria : Best practices to 

capture product knowledge by decision point to enter production; DOD 

criteria: Commercial criteria : [Empty].



Commercial criteria: X; Best practices to capture design knowledge by 

decision point to enter system demonstration phase: Use of key 

indicators to show the product is ready for production (processes in 

statistical control and product reliability demonstrated); DOD 

criteria: [Empty].



Commercial criteria: X; Best practices to capture design knowledge by 

decision point to enter system demonstration phase: Identify key system 

characteristics and manufacturing processes; DOD criteria: [Empty].



Commercial criteria: X; Best practices to capture design knowledge by 

decision point to enter system demonstration phase: Determine critical 

processes are in control and capable; DOD criteria: [Empty].



Commercial criteria: X; Best practices to capture design knowledge by 

decision point to enter system demonstration phase: Conduct failure 

modes and effects analysis; DOD criteria: [Empty].



Commercial criteria: X; Best practices to capture design knowledge by 

decision point to enter system demonstration phase: Set reliability 

growth goals; DOD criteria: [Empty].



Commercial criteria: X; Best practices to capture design knowledge by 

decision point to enter system demonstration phase: Conduct reliability 

growth testing; DOD criteria: [Empty].



Commercial criteria: X; Best practices to capture design knowledge by 

decision point to enter system demonstration phase: Hold decision 

review to begin production; DOD criteria: X.



[End of table]



According to DODï¿½s current acquisition policy, the system integration 

phase of an acquisition normally begins with the decision to launch a 

program. The policy states that, during this phase, a systemï¿½s 

configuration should be documented and the system should be 

demonstrated using prototypes in a relevant environment. While these 

are noteworthy activities and resemble best practices, the policy does 

not provide criteria for what constitutes the level of knowledge 

required for completing this stage, nor does it require a decision--

based on those criteria--as to whether a significant, additional 

investment should be made. Commercial companies demand knowledge from 

virtual or engineering prototypes, 90 percent of required engineering 

drawings for the product supported by test results, demonstration that 

the product meets customer requirements, a series of disciplined design 

reviews, and stakeholder agreement that the design is stable and ready 

for product demonstration before a commitment is made to move forward 

and invest in product demonstration. Under DODï¿½s revised policy, it is 

still difficult to determine if a product should enter product 

demonstration with a stable design.



DODï¿½s current acquisition policy also states that the system 

demonstration phase begins after prototypes have been built and 

demonstrated in a relevant environment during system integration. 

According to the policy, a system must be demonstrated before the 

department will commit to production. The low-rate initial production 

decision occurs after this phase of product development. Like the end 

of system integration, the policy fails to provide specific criteria 

for what constitutes the knowledge required to support the decision to 

move into production. For example, the policy states there should be 

ï¿½no significant manufacturing risksï¿½ but does not define what this 

means or how it is measured. Without criteria for building knowledge 

during the demonstration phase, the production decision is often based 

on insufficient knowledge, creating a higher probability of 

inconsistent results and cost and schedule problems. On the other hand, 

commercial companies demand proof that manufacturing processes are in 

control and product reliability goals are attained before committing to 

production. With more specific knowledge in hand at the end of 

development, decision makers can make a more informed decision to move 

into production with assurances that the product will achieve its cost, 

schedule, and quality outcomes.



Finally, while DODï¿½s policy separates product development into a two-

stage process--integration and demonstration--it does not require a 

decision milestone to move from one stage to the next. The policy 

states that an interim progress review should be held between the two 

stages, but the review has no established agenda and no required 

outputs of information unless specifically requested by the decision 

maker. Its purpose is to confirm that the program is progressing as 

planned. On the other hand, commercial companies consider this review a 

critical decision point in their product development process because it 

precedes a commitment to significantly increase their investment. 

Therefore, they use specific, knowledge-based standards and criteria to 

determine if the product is ready to enter the next phase and they hold 

decision makers accountable for their actions. These decision reviews 

are mandatory and are typically held at the executive level of the 

commercial firm.



Figure 9 illustrates the commercial model for knowledge to be captured 

and delivered during product integration and product demonstration and 

the possible application of that model to DODï¿½s acquisition process. 

Without a similar decision review to bring accountability to the DOD 

process, acquisition programs can--and do--continue to advance into 

system demonstration without a stable design. As shown in our case 

studies, this provides for a high probability of cost growth and 

schedule delays to occur.



Figure 9: Illustration to Show How the Best Practice Model Would Apply 

to DODï¿½s Acquisition Process:



[See PDF for image]



Source: GAOï¿½s analysis.



[End of figure]



Incentives in the DOD Acquisition Environment Do Not Favor Capture of 

Design and Manufacturing Knowledge Early Enough:



The incentives for program managers and product developers to gather 

knowledge and reduce risk are also critical to DODï¿½s ability to adopt 

best practices for product development. In DOD, incentives are centered 

on obtaining scarce funding on an annual basis in a competitive 

environment to meet predetermined and typically optimistic program 

schedules. These incentives actually work against the timely capture of 

knowledge, pushing it off until late in the process to avoid problems 

that might keep a program from being funded. Because design and 

manufacturing knowledge is not captured, key decision points intended 

to measure and ensure that a weapon system has sufficiently matured to 

move forward in the process risk becoming unsupported by critical 

knowledge. In leading commercial companies, the opposite is true. 

Because companies know they have to deliver high-quality products 

quickly and affordably, they limit the challenge for their program 

managers and provide strong incentives to capture design and 

manufacturing knowledge early in the process. Program managers are 

empowered to make informed decisions before big investments in 

manufacturing capability are required.



DODï¿½s current acquisition environment is driven by incentives to make 

decisions while significant unknowns about the systemï¿½s design and 

manufacturability persist. This environment results in higher risks and 

a greater reliance on cost-reimbursement[Footnote 10] contracts for 

longer periods of time during product development. Because events that 

should drive key decisions, such as critical design reviews, interim 

progress reviews, and production decision reviews, are based on 

inadequate design and manufacturing knowledge, they do not support 

decisions to invest more and move to the next phase of the acquisition 

process. Nevertheless, this approach has proven effective in securing 

funds year to year. For example, the F-22, PAC-3, and ATIRCMS/CMWS 

programs had less than one-third of their engineering drawings 

completed at their critical design review, but each obtained the 

funding necessary to move onto the initial manufacturing of production 

representative prototypes. That funding allowed a significant increase 

in investment to develop a manufacturing capability before critical 

knowledge had been captured. The incentive to capture funding for the 

program was greater than the incentive to wait, capture knowledge, and 

reduce the risk of moving forward. Each of these programs encountered 

significant cost increases and schedule delays.



The incentives are quite different for leading commercial companies. 

For them, the business case centers on the ability to produce a product 

that the customer will buy and that will provide an acceptable return 

on investment. If the firm has not made a sound business case, or has 

been unable to deliver on one or more of the business case factors, it 

faces a very real prospect of failure--the customer may walk away. 

Also, if one product development takes more time and money to complete 

than expected, it denies the firm opportunities to invest those 

resources in other products. For these reasons, commercial companies 

have strong incentives to capture product knowledge early in the 

process to assess the chances of making the business case and the need 

for further investments.



Production is a dominant concern in commercial companies throughout the 

product development process and forces discipline and trade-offs in the 

design process. This environment encourages realistic assessments of 

risks and costs since doing otherwise would threaten the business case 

and invite failure. For the same reasons, the environment places a high 

value on knowledge for making decisions. Program managers have good 

reasons to identify risks early, be intolerant of unknowns, and not 

rely on testing late in the process as the main vehicle for discovering 

the performance characteristics of the product. By adhering to the 

business case as the key to success, program managers in leading 

commercial companies are conservative in their estimates and aggressive 

in risk reduction. Ultimately, adherence to the business case 

strengthens the ability to say ï¿½noï¿½ to pressures to accept high risks 

and unknowns. Practices such as prototyping, early manufacturing and 

supplier involvement, completing 90 percent of engineering drawings by 

critical design review, demonstrating product reliability, and 

achieving statistical control of critical manufacturing processes by 

production are adopted because they help ensure success.



In DODï¿½s current acquisition environment, the customer is willing to 

trade time and money for the highest performing weapon system possible. 

That willingness drives the business case. This creates strong 

incentives for the program office to take significant risks with 

technologies and designs to ensure it can offer the customer a weapon 

system that is a quantum leap above the competition. In addition, 

because funding is secured on an annual basis in DOD, strong incentives 

exist for the program office to make optimistic assumptions about 

development cost and schedule. Because the customer is willing to wait 

and funding is never certain, an environment exists where program 

managers have good reasons to avoid the capture of knowledge and delay 

testing. Since the business case in DOD places very little premium on 

meeting cost and schedule targets, but a very high premium on 

performance, programs succeed at the point where sunk costs make it 

difficult--if not prohibitive--for decision makers to cancel them.



The practices commercial companies use to capture knowledge are not 

currently used in this environment because the business case does not 

favor them. Instead, DODï¿½s product development environment relies on 

cost-type contracting throughout the entire product development 

process. Once in production, programs will cut quantities to maintain 

funding or once fielded, they rely on the operations and maintenance 

budget to pay for reliability problems not solved in development.



[End of section]



Chapter 5 Conclusions and Recommendations:



Conclusions:



The Department of Defenseï¿½s (DOD) planned $700 billion investment in 

weapon systems over the next 5 years requires an approach that keeps 

cost, schedule, and performance risks to a minimum. This approach means 

adopting and implementing an evolutionary approach to developing new 

weapon systems, improving policy to more closely approximate a 

knowledge-based product development process, and creating incentives 

for capturing and using knowledge for decision making. Without an 

evolutionary approach as its foundation, the ability to capture design 

and manufacturing knowledge early in the development process is 

significantly reduced. Programs, in turn, take on too much new unproven 

content to meet their objectives and risks invariably increase. DOD has 

made improvements in its acquisition policy by incorporating guidance 

for evolutionary acquisition, creating guidelines for the development 

of a basic product that can be upgraded with additional capabilities as 

technologies present themselves. However, evolutionary acquisition has 

yet to be consistently implemented with success on individual weapon 

system acquisitions.



Regardless of whether DOD emphasized greater use of evolutionary 

acquisition, acquisition programs are not capturing sufficient design 

and manufacturing knowledge to make good decisions at key investment 

points. The current policy establishes a good framework to develop a 

product, but the policy still lacks specific criteria required to move 

a program forward and does not tie knowledge to decisions for 

increasing investments in the program as it moves from system 

integration to system demonstration. As a result, programs often pass 

through each development phase and into production with an unstable 

design and insufficient knowledge about critical manufacturing 

processes and product reliability. This results in greater likelihood 

for inconsistent and poor results and cost and schedule problems later 

in the program.



Additionally, DOD does not provide the proper incentives to encourage 

the use of best practices in capturing knowledge early in its 

development programs. Currently, managers are focused more on the 

annual exercise of obtaining funding needed to keep their programs 

viable and alive. The importance of capturing design and manufacturing 

knowledge early gives way to the pressures of maintaining funding, 

often resulting in the acceptance of greater risks. Raising problems on 

a program early because design and manufacturing knowledge is 

discovered can cause extra oversight and questions that threaten a 

systemï¿½s survival. The prevailing culture is to accept greater risks 

upfront and then fix problems later in the development program.



We found that leading commercial companies over the years had found 

ways to overcome these problems and had identified best practices that 

resulted in the early capture of and use of design and manufacturing 

knowledge. This was done by a combination of four key elements. First, 

they established and used an evolutionary approach to develop products 

that made the capture of design and manufacturing knowledge a more 

manageable task. This framework limited the design challenge for any 

one new product development by allowing risky technology, design, or 

manufacturing requirements to be deferred until a future generation of 

the product. DODï¿½s current policy addresses this; however, it has not 

had sufficient time to show how this will be implemented.



Second, each company we visited used the same basic product development 

process and criteria for bringing together and integrating all of the 

technologies, components, and subsystems required for the product to 

ensure the design was stable and then demonstrating that the product 

was producible and reliable using proven manufacturing processes. DODï¿½s 

policy lacks the criteria to measure design stability and process 

controls. Third, successful companies used tools to capture design and 

manufacturing knowledge about the product and decide about whether to 

invest further based on that knowledge. Their new product development 

process included key, high-level decision points before moving into 

product demonstration, and again before making the production decision 

that required specific, knowledge-based exit criteria. DODï¿½s policy 

does not require a decision to move from system integration to system 

demonstration. Finally, leading companies created an environment for 

their managers that emphasized capturing design and manufacturing 

knowledge early, before committing substantial investments in a product 

development that made cancellation a more difficult decision to make. 

DODï¿½s environment encourages meeting schedule milestones instead of 

capturing design and manufacturing knowledge to make decisions.



Recommendations for Executive Action:



DOD should take steps to close the gaps between its current acquisition 

environment and best practices. To do this, it should ensure that its 

acquisition process captures specific design and manufacturing 

knowledge, includes decisions at key junctures in the development 

program, and provides incentives to use a knowledge-based process. Such 

changes are necessary to obtain greater predictability in weapon system 

programsï¿½ cost and schedule, to improve the quality of weapon systems 

once fielded, and to deliver new capability to the war fighter faster. 

More specifically, we recommend that the Secretary of Defense:



* Require the capture of specific knowledge to be used as exit criteria 

for decision making at two key points--when transitioning from system 

integration to system demonstration and from system demonstration into 

production. The knowledge to be captured when moving from system 

integration into system demonstration should include the following:



* Completed subsystem and system design reviews.



* Ninety percent of drawings completed.



* Demonstration that design meets requirements--prototype or variant 

testing.



* Stakeholdersï¿½ (cross functional design team that includes design 

engineers, manufacturing, key supplier) assurance that drawings are 

complete.



* Completed failure modes and effects analysis.



* Identification of key system characteristics.



* Identification of critical manufacturing processes.



* Set reliability targets and growth plan.



The knowledge to be captured when moving from system demonstration into 

production should include the following:



* Demonstrated manufacturing processes.



* Built production representative prototypes.



* Tested prototypes to achieve reliability goal.



* Tested prototypes to demonstrate product in operational environment.



* Collected statistical process control data.



* Demonstration that critical processes are capable and in control.



* Require that the interim progress review, currently identified in 

DODï¿½s policy as that point in the process between system integration 

and system demonstration, be a mandatory decision review. At this 

point, the design should be demonstrated to be stable so that during 

the next phase of development attention can be focused on demonstrating 

manufacturing processes and product reliability. The program manager 

should have proof--based on the exit criteria for moving out of system 

integration in the above recommendation--that the product design is 

stable. The exit criteria should be demonstrated and verified by the 

program manager before the program can make the substantial investments 

needed to begin manufacturing production representative prototypes in 

the next phase of development--system demonstration. To ensure 

visibility of demonstrated exit criteria to decision makers, the 

criteria and the programï¿½s status in achieving them should be included 

in each programï¿½s Defense Acquisition Executive Summary and Selected 

Acquisition Reports. If the program does not meet the exit criteria, 

investments should be delayed until such time as the criteria are 

satisfied. To proceed without completing the required demonstrations 

should require approval by the decision authority.



* Expand exit criteria for the Milestone C decision to include the 

knowledge to be captured during the system demonstration phase as 

identified in recommendation one. This will require that the program 

office demonstrate that the critical manufacturing processes are under 

statistical control and that product reliability has been demonstrated 

before entering production of the new weapon system. These are best 

practices and indicate that the product design is mature and the 

program is ready to begin production of units for operational use that 

will meet the cost, schedule, and quality goals of the program.



* To ensure that contracts support a knowledge-based process, we 

further recommend that DOD structure its contracts for major weapon 

system acquisitions so that (a) the capture and use of knowledge 

described in recommendation one for beginning system demonstration is a 

basis for DODï¿½s decision to invest in the manufacturing capability to 

build initial prototypes and (b) the capture and use of manufacturing 

and reliability knowledge discussed in recommendation one for moving 

from system demonstration to production is a basis for DODï¿½s decision 

to invest in production.



Agency Comments and Our Evaluation:



DOD concurred with a draft of this report and agreed with the benefits 

of using design and manufacturing knowledge to make informed decisions 

at key points in a system acquisition program. DOD had some comments 

with regard to the details contained in the recommendations, which are 

summarized below. DOD concurred with our recommendation to add exit 

criteria at two key points in the acquisition process--when 

transitioning from system integration to system demonstration and from 

system demonstration into production. DOD believes, however, that the 

milestone decision authority needs to retain flexibility in applying 

the knowledge requirement for drawings. Flexibility and judgment are 

management prerogatives that should exist in any decision process. We 

agree there may be circumstances, such as in the development of 

software, when it makes good sense to progress with less than the best 

practice standard for drawings, but the DOD policy should maintain the 

requirement to achieve 90 percent drawings by the completion of the 

system integration phase.



DOD also concurred that critical manufacturing processes must be 

demonstrated using statistical process control techniques before 

production, but believes that achieving this at Milestone C, the low 

rate production decision, is unlikely. It believes the criteria would 

be better applied to the full rate production decision or when low rate 

production quantities extend beyond 10 percent of the planned weapon 

system buy. This is a reasonable approach when processes are new or 

unique. However, not all critical processes will be new or unique to a 

specific weapon system. Some will have been used to manufacture parts 

or components for other systems or products. At a minimum, it should be 

possible to demonstrate these by Milestone C. For other critical 

processes that may require additional production experience to bring 

under statistical process control, a program manager should have a 

reasonable plan at the Milestone C decision review to bring those 

processes into control by the full rate production decision, but no 

later than completion of 10 percent of the planned buy.



[End of section]



Appendixes:



Appendix I: Comments from the Department of Defense:



ACQUISITION, TECHNOLOGY AND LOGISTICS:



OFFICE OF THE UNDER SECRETARY OF DEFENSE:



3000 DEFENSE PENTAGON WASHINGTON, DC 20301-3000:



19 JUN 2002:



Ms. Katherine V. Schinasi:



Director, Acquisition and Sourcing Management U.S. General Accounting 

Office:



441 G Street, N.W. Washington, D.C. 20548:



Dear Ms. Schinasi:



This is the Department of Defense (DOD) response to the GAO draft 

report, ï¿½BEST PRACTICES: Capturing Design and Manufacturing Knowledge 

Early Improves Acquisition Outcomes,ï¿½ dated May 15, 2002 (GAO Code 

120054/GAO-02-701).



The Department concurs with the objectives of GAOï¿½s DRAFT report and we 

agree with the benefits of using important information about a systemï¿½s 

design and critical manufacturing processes to make informed decisions. 

However, we have some comments with regard to the details contained in 

the recommendations.



In response to previous GAO reports, DoD has structured its acquisition 

process to make progress through the acquisition life-cycle dependent 

on the knowledge available at key decision points. This report adds 

robustness to our already disciplined process.



We look forward to discussing these issues with the GAO. Detailed 

comments are provided in the enclosure.



My action officer for this effort is Mr. Richard Sylvester, (703) 697-

6399.



Sincerely,



Donna S. Richbourg:



Director, Acquisition Initiatives:



Signed by Donna S. Richbourg:



Enclosure: As stated:



GAO DRAFT REPORT - DATED MAY 15, 2002 GAO CODE 120054/GAO-02-701:



ï¿½BEST PRACTICES: Capturing Design and Manufacturing Knowledge Early 

Improves Acquisition Outcomesï¿½:



DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS:



RECOMMENDATION 1: The GAO recommended that the Secretary of Defense 

require the capture of specific knowledge to be used as exit criteria 

for decision making at two key points - when transitioning from system 

integration to system demonstration and from system demonstration into 

production. The knowledge to be captured when moving from system 

integration into system demonstration should include the following:



*Completed subsystem and system design reviews:



*90% of drawings completed:



*Demonstration that design meets requirements - prototype or variant 

testing:



*Stakeholdersï¿½ (cross functional design team that includes design 

engineers, manufacturing, key supplier) assurance that drawings are 

complete *Completed failure modes and effects analysis:



*Identification of key system characteristics:



*Identification of critical manufacturing processes:



*Set reliability targets and growth plan:



The knowledge to be captured when moving from system demonstration into 

production should include the following:



*Demonstrated manufacturing processes:



*Built production representative prototypes:



*Tested prototypes to achieve reliability goal:



*Tested prototypes to demonstrate product in operational environment:



*Collected statistical process control data:



*Demonstration that critical processes are capable and in control (pgs. 

69-70/GAO Draft Report):



DOD RESPONSE: Concur. DoD agrees with the benefits of identifying 

specific design and manufacturing information to support decision 

making at key decision points. However, we believe that the milestone 

decision authority needs to retain the flexibility to determine 

application of the knowledge requirement for drawings (e.g., software 

systems will not have drawings, 85% of completed drawings may be enough 

when measured against urgency of requirement, etc.). However, in no 

case should a system proceed without a substantial percentage of the 

drawings completed. We believe the specific measures should be set by 

the MDA at MS B and that the MDA should have the option to add specific 

criteria to reflect specific systems.



RECOMMENDATION 2: The GAO recommended that the Secretary of Defense 

require that the interim progress review, currently identified in DoD 

policy as that point in the process between system integration and 

system demonstration, be a mandatory decision review. At this point, 

the design should be demonstrated to be stable so that during the next 

phase of development attention can be focused on demonstrating 

manufacturing processes and product reliability. The program manager 

should have proof - based on the exit criteria for moving out of system 

integration in the above recommendation - that the product design is 

stable. The exit criteria should be demonstrated and verified by the 

program manager before the program can make the substantial investments 

needed to begin manufacturing production representative prototypes in 

the next phase of development - system demonstration. To ensure 

visibility of demonstrated exit criteria to decision makers, the 

criteria and the programï¿½s status in achieving it should be included in 

each programï¿½s Defense Acquisition Executive Summary and Selected 

Acquisition Reports. If the program does not meet the exit criteria, 

investments should be delayed until such time as the criteria is 

satisfied. To proceed without completing the required demonstrations 

should require approval by the decision authority. (p. 70/GAO Draft 

Report):



DOD RESPONSE: Concur. DOD agrees with the requirement to demonstrate 

design stability at a point between system integration and system 

demonstration and to satisfy criteria reflecting that stability 

established by the MDA at MS B. The PM should be able to demonstrate 

exit criteria satisfaction by providing a report to the Milestone 

Decision Authority. That data can then be considered in the context of 

other key indications of program progress. A mandatory review would 

only be considered if the exit criteria are not satisfied.



RECOMMENDATION 3: The GAO recommended that the Secretary of Defense 

expand exit criteria for the Milestone C decision to include the 

knowledge to be captured during the system demonstration phase as 

identified in recommendation 1. This will require that the program 

office demonstrate that the critical manufacturing processes are under 

statistical control and that product reliability has been demonstrated 

before entering production of the new weapon system. These are best 

practices, and indicate that the product design is mature and the 

program is ready to begin production of units for operational use that 

will meet the cost, schedule and quality goals of the program. (p. 70/

GAO Draft Report):



DOD RESPONSE: Concur. DoD agrees that demonstration of critical 

manufacturing processes must occur prior to rate production. However, 

it is unlikely that the criteria suggested by the GAO could be 

satisfied at MS C. MS C authorizes the program to proceed to Low Rate 

Initial Production in support of Operational Test. That decision 

precedes the Full Rate Production Decision where approval is given for 

production in support of operational use. The criteria suggested by the 

GAO would be better applied to the Full Rate Production Decision or 

when LRIP extends beyond 10 percent of the total planned buy..



RECOMMENDATION 4: To ensure that contracts support a knowledge-based 

process, the GAO recommended that DoD structure its contracts for major 

weapon system acquisitions so that (a) the capture and use o^ knowledge 

described in Recommendation 1 for beginning system demonstration is a 

basis for DoDï¿½s decision to invest in the manufacturing capability to 

build initial prototypes, and (b) the capture and use of manufacturing 
and 

reliability knowledge discussed in Recommendation 1 for moving from 

system demonstration into production is a basis for DoDï¿½s decision to 

invest in production. (pgs. 70-71/GAO Draft Report):



DOD RESPONSE: Concur. See additional comments in response to 

recommendations 1 and 3. DoD agrees with the benefits of applying both 

design and manufacturing criteria at key points in the acquisition 

process and with capturing those criteria in program contracts. The 

specific criteria would be most effective if tied to established breaks 

(such as contract line items) designed into the contract structure.



[End of section]



Appendix II: GAO Staff Acknowledgments:



Acknowledgments:



Cheryl Andrew, Cristina Chaplain, Michael Hazard, Matthew Lea, Gary 

Middleton, Michael Sullivan, Katrina Taylor, and Adam Vodraska:



[End of section]



Related GAO Products:



Defense Acquisitions: DOD Faces Challenges in Implementing Best 

Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.



Best Practices: Better Matching of Needs and Resources Will Lead to 

Better Weapon System Space Outcomes. GAO-01-288. Washington, D.C.: 

March 8, 2001.



Best Practices: A More Constructive Test Approach Is Key to Better 

Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 

2000.



Defense Acquisition: Employing Best Practices Can Shape Better Weapon 

System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000.



Best Practices: DOD Training Can Do More to Help Weapon System Programs 

Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August 

16, 1999.



Best Practices: Better Management of Technology Development Can Improve 

Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 

1999.



Defense Acquisition: Best Commercial Practices Can Improve Program 

Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.



Defense Acquisition: Improved Program Outcomes Are Possible. 

GAO/T-NSIAD-98-123. Washington, D.C.: March 18, 1998.



Best Practices: DOD Can Help Suppliers Contribute More to Weapon System 

Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.



Best Practices: Successful Application to Weapon Acquisitions Requires 

Changes in DODï¿½s Environment. GAO/NSIAD-98-56. Washington, D.C.: 

February 24, 1998.



Major Acquisitions: Significant Changes Underway in DODï¿½s Earned Value 

Management Process. GAO/NSIAD-97-108. Washington, D.C.: May 5, 1997.



Best Practices: Commercial Quality Assurance Practices Offer 

Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 

1996.



FOOTNOTES



[1] U.S. General Accounting Office, Best Practices: Better Matching of 

Needs and Resources Will Lead to Better Weapon System Outcomes, 

GAO-01-288 (Washington, D.C.: Mar. 8, 2001) and Best Practices: Better 

Management of Technology Development Can Improve Weapon System 

Outcomes, GAO/NSIAD-99-162 (Washington, D.C.: July 30, 1999).



[2] DOD Directive 5000.1, The Defense Acquisition System (Oct. 23, 

2000), DOD 

Instruction 5000.2, Operation of the Defense Acquisition System (Apr. 

5, 2002), and DOD Regulation 5000.2-R, Mandatory Procedures for Major 

Defense Acquisition Programs (MDAPS) and Major Automated Information 

System (MAIS) Acquisition Programs (Apr. 5, 2002).



[3] The Selected Acquisition Report provides standard, comprehensive 

summary reporting of cost, schedule, and performance information for 

major defense acquisition programs to the Congress.



[4] Engineering drawings can include the standard two-dimensional 

drawings or newer three-dimensional drawings that are the product of 

computer-aided design software systems.



[5] The fifth company wanted its critical manufacturing processes at a 

minimum of 1 Cpk.



[6] Failure modes and effects analysis is a bottom-up approach to 

failure identification. It should begin at the lowest level of the 

product design. Through analysis potential failure modes are identified 

allowing early design change to correct potential problems before 

fabricating hardware--a more cost-effective time to identify and fix 

problems.



[7] U.S. General Accounting Office, Best Practices: Successful 

Application to Weapon Acquisition Requires Changes in DODï¿½s Environment 

GAO/NSIAD-98-56 (Washington, D.C.: Feb. 24, 1998).



[8] U.S. General Accounting Office, Tactical Aircraft: F-22 Delays 

Indicate Initial Production Rates Should Be Lower to Reduce Risks 

GAO-02-298 (Washington, D.C.: Mar. 5, 2002).



[9] DOD Directive 5000.1, The Defense Acquisition System (Oct. 23, 

2000), DOD

Instruction 5000.2, Operation of the Defense Acquisition System (Apr. 

5, 2002), and DOD Regulation 5000.2-R, Mandatory Procedures for Major 

Defense Acquisition Programs (MDAPS) and Major Automated Information 

System (MAIS) Acquisition Programs (Apr. 5, 2002).



[10] Cost-reimbursement contracts provide for payment of allowable 

incurred costs, to the extent prescribed in the contracts. They are 

suitable for use only when uncertainties involved in contract 

performance, such as research and development work, do not permit costs 

to be estimated with sufficient accuracy. In contrast, fixed-priced 

contracts, except those subject to price adjustment, provide for a 

preestablished firm price, place maximum risk and full responsibility 

for all costs and resulting profit or loss on the contractor, and 

provide maximum incentive for the contractor to control costs and 

perform effectively.



GAOï¿½s Mission:



The General Accounting Office, the investigative arm of Congress, 

exists to support Congress in meeting its constitutional 

responsibilities and to help improve the performance and accountability 

of the federal government for the American people. GAO examines the use 

of public funds; evaluates federal programs and policies; and provides 

analyses, recommendations, and other assistance to help Congress make 

informed oversight, policy, and funding decisions. GAOï¿½s commitment to 

good government is reflected in its core values of accountability, 

integrity, and reliability.



Obtaining Copies of GAO Reports and Testimony:



The fastest and easiest way to obtain copies of GAO documents at no 

cost is through the Internet. GAOï¿½s Web site ( www.gao.gov ) contains 

abstracts and full-text files of current reports and testimony and an 

expanding archive of older products. The Web site features a search 

engine to help you locate documents using key words and phrases. You 

can print these documents in their entirety, including charts and other 

graphics.



Each day, GAO issues a list of newly released reports, testimony, and 

correspondence. GAO posts this list, known as ï¿½Todayï¿½s Reports,ï¿½ on its 

Web site daily. The list contains links to the full-text document 

files. To have GAO e-mail this list to you every afternoon, go to 

www.gao.gov and select ï¿½Subscribe to daily E-mail alert for newly 

released productsï¿½ under the GAO Reports heading.



Order by Mail or Phone:



The first copy of each printed report is free. Additional copies are $2 

each. A check or money order should be made out to the Superintendent 

of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. 

Orders should be sent to:



U.S. General Accounting Office



441 G Street NW,



Room LM Washington,



D.C. 20548:



To order by Phone: 	



	Voice: (202) 512-6000:



	TDD: (202) 512-2537:



	Fax: (202) 512-6061:



To Report Fraud, Waste, and Abuse in Federal Programs:



Contact:



Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: [email protected]



Automated answering system: (800) 424-5454 or (202) 512-7470:



Public Affairs:



Jeff Nelligan, managing director, [email protected] (202) 512-4800 U.S.



General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.



20548: