Best Practices: Increased Focus on Requirements and Oversight	 
Needed to Improve DOD's Acquisition Environment and Weapon System
Quality (01-FEB-08, GAO-08-294).				 
                                                                 
A Senate report related to the National Defense Authorization Act
for Fiscal Year 2007 asked GAO to compare quality management	 
practices used by the Department of Defense (DOD) and its	 
contractors to those used by leading commercial companies and	 
make suggestions for improvement. To do this, GAO (1) determined 
the impact of quality problems on selected weapon systems and	 
prime contractor practices that contributed to the problems; (2) 
identified commercial practices that can be used to improve DOD  
weapon systems; (3) identified problems that DOD must overcome;  
and (4) identified recent DOD initiatives that could improve	 
quality. GAO examined 11 DOD weapon systems with known quality	 
problems and met with quality officials from DOD, defense prime  
contractors, and five leading commercial companies that produce  
complex products and/or are recognized for quality products.	 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-08-294 					        
    ACCNO:   A80369						        
  TITLE:     Best Practices: Increased Focus on Requirements and      
Oversight Needed to Improve DOD's Acquisition Environment and	 
Weapon System Quality						 
     DATE:   02/01/2008 
  SUBJECT:   Best practices					 
	     Contract administration				 
	     Contract oversight 				 
	     Contract performance				 
	     Cost overruns					 
	     Defense capabilities				 
	     Defense cost control				 
	     Defense procurement				 
	     Department of Defense contractors			 
	     Performance measures				 
	     Prime contracts					 
	     Quality control					 
	     Requirements definition				 
	     Schedule slippages 				 
	     Systems evaluation 				 
	     Weapons systems					 
	     Government agency oversight			 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-08-294

   

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to [email protected]. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Committees: 

United States Government Accountability Office: 
GAO: 

February 2008: 

Best Practices: 

Increased Focus on Requirements and Oversight Needed to Improve DOD's 
Acquisition Environment and Weapon System Quality: 

GAO-08-294: 

GAO Highlights: 

Highlights of GAO-08-294, a report to congressional committees. 

Why GAO Did This Study: 

A Senate report related to the National Defense Authorization Act for 
Fiscal Year 2007 asked GAO to compare quality management practices used 
by the Department of Defense (DOD) and its contractors to those used by 
leading commercial companies and make suggestions for improvement. 
To do this, GAO (1) determined the impact of quality problems on 
selected weapon systems and prime contractor practices that contributed 
to the problems; (2) identified commercial practices that can be used 
to improve DOD weapon systems; (3) identified problems that DOD must 
overcome; and (4) identified recent DOD initiatives that could improve 
quality. GAO examined 11 DOD weapon systems with known quality problems 
and met with quality officials from DOD, defense prime contractors, and 
five leading commercial companies that produce complex products and/or 
are recognized for quality products 

What GAO Found: 

Problems related to quality have resulted in major impacts to the 11 
DOD weapon systems GAO reviewedï¿½billions in cost overruns, years-long 
delays, and decreased capabilities for the warfighter. For example, 
quality problems with the Expeditionary Fighting Vehicle program were 
so significant that DOD extended development 4 years at a cost of $750 
million. The F-22A fighter aircraft experienced cracks in the planeï¿½s 
canopy that grounded the flight test aircraft, and initial operating 
capability for the Wideband Global SATCOM satellite was delayed 18 
months because a supplier installed some fasteners incorrectly. GAOï¿½s 
analysis of 11 DOD weapon systems illustrates that defense contractorsï¿½ 
poor practices for systems engineering activities as well as 
manufacturing and supplier quality problems contributed to these 
outcomes. Reliance on immature designs, inadequate testing, defective 
parts, and inadequate manufacturing controls are some of the quality 
problems that GAO found. Senior prime contractor officials GAO met with 
generally agreed with GAOï¿½s assessment of the causes of the quality 
problems. 

In contrast, leading commercial companies GAO contacted use more 
disciplined systems engineering, manufacturing, and supplier quality 
practices. For example, rather than wait to discover defects after the 
fact, Boeing Commercial Airplanes tries to design parts that can be 
assembled only one way. Effective use of many systems engineering 
practices has helped Space Systems/Loral, a satellite producer, improve 
overall quality, for example, by allowing the company to operate its 
satellites for more than 80 million consecutive hours in orbit with 
just one failure. Companies also put significant effort into validating 
product design and production processes to catch problems early on, 
when problems are less costly to fix. They conduct regular audits of 
their suppliers and hold them accountable for quality problems. 

DOD faces its own set of challengesï¿½setting achievable requirements for 
systems development and providing effective oversight during the 
development process. In conducting systems development, DOD generally 
pays the allowable costs incurred for the contractorï¿½s best efforts. 
These conditions contribute to an acquisition environment that is not 
conducive for incentivizing contractors to build high-quality weapon 
systems and DOD, which typically uses cost-reimbursement contracts to 
develop weapon systems, assumes most of the risks and pays contractors 
to fix most of the problems. 

DOD has taken steps to improve its acquisition practices by 
experimenting with a new concept decision review practice, selecting 
different acquisition approaches according to expected fielding times, 
and establishing panels to review weapon system configuration changes 
that could adversely affect program cost and schedule. None of these 
initiatives focus exclusively on quality issues, and none specifically 
address problems with defense contractorsï¿½ practices. 

What GAO Recommends: 

GAO recommends that the Secretary of Defense take actions to set 
achievable requirements for new weapon system development, oversee and 
expand initiatives that could improve quality, and use data to assess 
contractor performance and weapon system quality. DOD partially agreed 
with the recommendations, stating that its current practices or planned 
actions are appropriate. We believe our recommendations remain valid 
and can improve weapons systems quality. 

To view the full product, including the scope and methodology, click on 
[hyperlink, http://wwww.GAO-08-294]. For more information, contact 
Michael Sullivan at (202) 512-4841 or [email protected]. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

DOD Weapon Systems Experience Quality Problems Due to Prime 
Contractors' Inconsistent Practices: 

Leading Commercial Companies Use Disciplined Quality Management 
Practices: 

Different Environments Create Different Incentives to Improve Quality: 

DOD Efforts to Improve Acquisition Outcomes: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Quality Problems for 11 DOD Weapon Systems: 

Appendix III: Comments from the Department of Defense: 

Appendix IV: GAO Contact and Staff Acknowledgments: 

Related GAO Products: 

Figures: 

Figure 1: Weapon System Quality Problems and Impact: 

Figure 2: Advanced SEAL Delivery System: 

Figure 3: Advanced Threat Infrared Countermeasure/Common Missile 
Warning System: 

Figure 4: Expeditionary Fighting Vehicle: 

Figure 5: F-22A: 

Figure 6:Global Hawk Unmanned Aircraft System: 

Figure 7: Joint Air-to-Surface Standoff Missile: 

Figure 8: LPD 17 Amphibious Transport Dock: 

Figure 9: MH-60S Fleet Combat Support Helicopter: 

Figure 10: Patriot Advanced Capability-3: 

Figure 11: V-22 Joint Services Advanced Vertical Lift Aircraft: 

Figure 12: Wideband Global SATCOM: 

Abbreviations: 

DOD: Department of Defense: 

RDT&E: research, development, test and evaluation: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

February 1, 2008: 

Congressional Committees: 

The Department of Defense (DOD) plans to invest about $1.5 trillion (in 
2007 dollars) in its current portfolio of major weapon systems. 
However, the cost of designing and developing these systems could 
continue to exceed estimates by billions of dollars if DOD continues to 
employ the same acquisition practices, including those for quality, as 
it has in the past. Excessive scrap, rework, and repair costs, as well 
as reliability problems impact overall quality and could ultimately 
present serious consequences on a weapon system's long-term support 
costs and affordability. 

Like DOD, commercial companies collectively invest trillions to develop 
their products. Fundamentally, they know they must meet customer 
expectations to ensure continued growth. Many leading companies follow 
a knowledge-based approach for product development and rely on proven 
practices to attain high-quality products, control costs, and make a 
profit. While commercial companies are not without flaws and can 
produce poor-quality products, the demands of the marketplace force 
them to place a high priority on quality. 

This report examines how DOD and its defense contractors can improve 
the quality of major weapon systems. A Senate report related to the 
John Warner National Defense Authorization Act for Fiscal Year 2007 
asked GAO to compare quality management practices used by DOD and its 
contractors to those used by leading commercial companies and make 
suggestions as to how DOD's practices could be improved. We (1) 
determined the impact of quality problems on selected DOD weapon 
systems and defense contractors' practices that contributed to the 
problems; (2) identified practices used by leading commercial companies 
that can be used to improve the quality of DOD weapon systems; (3) 
identified problems DOD faces in terms of improving quality; and (4) 
identified recent DOD initiatives that could improve quality. 

To do this, we considered quality activities that take place from the 
time requirements are established for a product until it is fielded. 
This includes virtually all key design and engineering elements during 
development, the transition to production, and production itself. We 
examined 11 DOD weapon systems with known deficiencies that are in 
later phases of development or production to determine the impact of 
quality problems and assessed the effectiveness of defense contractors' 
quality practices for these systems. The 11 weapon systems were chosen 
to demonstrate the types of problems DOD weapon systems experience and 
to help focus our discussions with leading commercial companies on 
aspects of development that caused DOD major quality problems. We also 
met with representatives from the Office of the Under Secretary of 
Defense (Acquisition, Technology and Logistics), each of DOD's military 
services, selected commands, the Defense Contract Management Agency, 
and six of DOD's largest prime contractors--BAE Systems, Boeing 
Integrated Defense Systems, General Dynamics, Lockheed Martin, Northrop 
Grumman, and Sikorsky Aircraft--to discuss quality practices used to 
build DOD weapon systems and to obtain related documentation.[Footnote 
1] These prime contractors are involved in a little over $1.1 trillion, 
or about 76 percent, of DOD's expected $1.5 trillion expenditure on 
weapon systems in its current portfolio. To identify leading commercial 
companies' quality practices, we interviewed and obtained documentation 
from quality management personnel at five companies: Boeing Commercial 
Airplanes; Cummins Inc., a manufacturer of diesel and natural gas- 
powered engines; Siemens Medical Solutions, a producer of ultrasound 
systems; Space Systems/Loral, a producer of satellite systems; and 
Kenworth, a trucking company. Much of the information we obtained from 
these companies is anecdotal, due to the proprietary nature of the data 
that could affect their competitive standing. We also met with 
officials from American Airlines and Intelsat, a satellite 
communications company, to understand the commercial customers' role in 
acquiring high-quality products. 

We compared leading commercial company practices with those used by DOD 
and the prime contractors we reviewed to identify both potential areas 
for improvement and practices that could improve the quality of DOD 
weapon systems. We also reviewed recent DOD initiatives aimed at 
improving acquisitions to determine if they have the potential to 
improve weapon system quality. Appendix I includes additional details 
about our scope and methodology. We conducted this performance audit 
from September 2006 to December 2007 in accordance with generally 
accepted government auditing standards. Those standards require that we 
plan and perform the audit to obtain sufficient, appropriate evidence 
to provide a reasonable basis for our findings and conclusions based on 
our audit objectives. We believe that the evidence obtained provides a 
reasonable basis for our findings and conclusions based on our audit 
objectives. 

Results in Brief: 

Quality problems have caused cost overruns, schedule delays, and 
reduced weapon system availability on the 11 DOD weapon systems we 
reviewed. The Expeditionary Fighting Vehicle is a case in point. Just 
as it was scheduled to move into production, DOD extended the program's 
development by 4 years at an estimated cost of $750 million when the 
prime contractor could not meet interim reliability goals. In another 
example, the Air Force temporarily grounded the F-22A's flight test 
aircraft when its first-of-a-kind canopy suffered cracks near the 
mounting holes because of problems with the prime contractor's 
manufacturing practices. In addition, the Wideband Global SATCOM 
communications satellite's initial operating capability date was 
delayed by 18 months because a supplier had installed some fasteners 
incorrectly and 1,500 fasteners on each of the first three satellites 
had to be reinspected. We found many other problems on other programs 
we reviewed as well--a laser jammer that did not work as intended, 
peeling coating on ships, deficient welding, and nonconforming parts-- 
that added to DOD's costs and schedules. Prime contractors' poor 
practices related to systems engineering, manufacturing, and supplier 
quality contributed to these problems. Senior prime contractor quality 
officials generally agreed with our assessment of the causes of 
problems in the systems we reviewed. 

Like DOD prime contractors, leading commercial companies rely on many 
practices related to systems engineering, manufacturing, and supplier 
quality, but the companies we reviewed apply more discipline and more 
rigorous, institutionalized processes to ensure product quality. The 
companies set well-defined product requirements and performed 
appropriate testing, which are critical systems engineering practices. 
For example, recent satellite components designed and developed by 
Space Systems/Loral, a satellite producer, have over 80-million hours 
of in orbit experience with only one failure, a greater than 99 percent 
availability rate. Space Systems/Loral accomplished this by focusing on 
reliability requirements during development and using reliability 
assessments and extensive testing to identify weak links before 
production started. Likewise, leading commercial companies focus on 
getting manufacturing processes in control prior to production. Cummins 
builds prototype engines to validate its manufacturing processes and 
Kenworth uses electronic versions of installation work processes to 
ensure that there is configuration control over the installation 
process and reduce rework. The companies also conducted regular audits 
of their suppliers and tracked supplier performance related to parts 
delivery and quality. For example, Boeing Commercial Airplanes requires 
its highest-rated suppliers to meet a 99 percent rate for parts 
conformance. For these companies, using disciplined processes and 
continuous improvement was essential to producing high-quality products 
and sustaining their competitive position in the commercial 
marketplace. 

DOD's acquisition environment does not provide incentives to prime 
contractors to use best practices to efficiently build high-quality 
weapon systems. The department faces challenges setting achievable 
requirements for systems development and providing effective oversight 
during the development process. In conducting systems development, DOD 
generally pays the allowable costs incurred for the contractor's best 
efforts and accepts most of the financial risks associated with 
development because of technical uncertainties. However, DOD and its 
contractors often enter into development contracts before requirements 
have been analyzed with disciplined systems engineering practices. This 
introduces significant cost and schedule risk to a development program, 
risk that is not borne by the prime contractor, but by DOD. Contractors 
have little incentive to utilize the best systems engineering, 
manufacturing, and supplier quality practices to control costs. DOD 
also has limited oversight of prime contractor activities and does not 
aggregate quality data in a manner that helps decision makers assess or 
identify systemic quality problems. In contrast, commercial companies 
we visited operate in an environment that requires their own investment 
of significant funds to develop new products before they are able to 
sell them and recoup that investment. This high-cost environment 
creates incentives for reasonable requirements that have been analyzed 
and proven achievable, the use of best practices, and continuous 
improvement in systems engineering, manufacturing, and supplier quality 
activities. 

In response to the John Warner National Defense Authorization Act for 
Fiscal Year 2007, the Under Secretary of Defense for Acquisition, 
Technology and Logistics has identified several initiatives DOD 
recently started that might eventually help improve weapon system 
quality. Some of its new initiatives address problems we noted in this 
report, such as placing greater emphasis on setting achievable 
requirements before starting development. However, DOD has not taken 
actions that would address problems related to prime contractor systems 
engineering, manufacturing, and supplier quality practices we found in 
our review of the 11 weapon systems. 

We are making recommendations that the Secretary of Defense improve 
weapons system quality by setting achievable requirements at the start 
of weapon system development, overseeing and expanding initiatives that 
could improve quality, and using data to assess prime contractor 
performance and weapon system quality. DOD partially agreed with each 
of the recommendations, stating that it believes the current practices 
or actions it plans to take are appropriate. In response to DOD's 
comments, we added more detail to one recommendation and acknowledged 
that the department is taking steps that could improve weapons system 
quality. Nevertheless, we believe our recommendations remain valid for 
improving weapons system quality. 

Background: 

In general, a quality product is one that is delivered on time, 
performs as expected, and can be depended on to perform when needed, at 
an affordable cost. This applies whether the customer is an individual 
purchasing a simple consumer good, such as a television, a hospital 
purchasing medical imaging equipment to help doctors treat cancer 
patients, or DOD purchasing sophisticated weapons for its warfighters 
to use on the battlefield. 

For about 3 decades, DOD based its quality requirements on a military 
standard known as MIL-Q-9858A, and its quality assurance practices were 
oriented toward discovering defects through inspections. In 1994, the 
Secretary of Defense announced that commercial quality standards should 
replace MIL-Q-9858A. The intent was to remove military-unique 
requirements that could present barriers to DOD in accessing the 
commercial supplier base. Currently, responsibilities for quality 
policy and oversight fall under the Systems and Software Engineering 
organization, within the Office of the Secretary of Defense. 

Over the past 20 years, commercial companies have had to dramatically 
improve quality in response to increased competition. Many companies 
moved from inspection-oriented quality management practices--where 
problems are identified and corrected after a product is produced--to a 
process in which quality is designed into a product and manufacturing 
processes are brought in statistical control to reduce defects. Many 
companies have also adopted commercial quality standards, such as ISO 
9001.[Footnote 2] This standard was developed by the International 
Organization for Standardization, a non-governmental organization 
established in 1947 to facilitate the international coordination and 
unification of industrial standards. Similar to DOD's MIL-Q-9858A, ISO 
9001 includes requirements for controlling a product's design and 
development, and production, as well as processes for oversight and 
improvement. Some industries, such as the automotive and aerospace 
industries, also have standards specific to their sector based on the 
ISO 9001.[Footnote 3] Because supplier parts account for a substantial 
amount of the material value of many companies' products, companies may 
require their suppliers to adopt the same standards. 

In practice, DOD and its prime contractors both participate in 
activities that contribute to weapon system quality. DOD plays a large 
role in quality when it sets key performance parameters, which are the 
most important requirements DOD wants prime contractors to focus on 
during development. For example, if reliability is one of those key 
performance parameters, then prime contractors are expected to focus on 
it during weapon system design. Prime contractors employ quality 
assurance specialists and engineers to assess the quality and 
reliability of parts they receive from suppliers, as well as the 
overall weapon system. DOD has its own quality specialists within the 
Defense Contract Management Agency and the military services, such as 
the Navy's Supervisor of Shipbuilding organization. DOD's quality 
specialists oversee prime contractors' design, manufacturing, and 
supplier management activities; oversee selected supplier manufacturing 
activities; and conduct final product inspections prior to acceptance. 

GAO previously reported on DOD quality practices in 1996.[Footnote 4] 
At that time, we reported that numerous weapon system programs had 
historically had quality problems in production because designs were 
incomplete. The B-2 bomber program and the C-17 Airlifter program, for 
example, encountered major manufacturing problems because they went 
forward with unstable designs and relied on inspections to find defects 
once in production. Since 1996, GAO has recommended several times that 
DOD adopt a knowledge-based acquisition approach used by leading 
commercial companies to develop its weapon systems. Under this 
approach, high levels of knowledge are demonstrated at critical 
decision points in the product development process, which results in 
successful product development outcomes. Systems engineering is a key 
practice that companies use to build quality into new products. 
Companies translate customers' broad requirements into detailed 
requirements and designs, including identifying requisite 
technological, software, engineering, and production capabilities. 
Systems engineering also involves performing verification activities, 
including testing, to confirm that the design satisfies requirements. 
Products borne out of a knowledge-based approach stand a significantly 
better chance to be delivered on time, within budget, and with the 
promised capabilities. Related GAO products, listed at the back of this 
report, provide detailed information about the knowledge-based 
approach. 

DOD Weapon Systems Experience Quality Problems Due to Prime 
Contractors' Inconsistent Practices: 

Although major defense contractors have adopted commercial quality 
standards in recent years, quality and reliability problems persist in 
DOD weapon systems. On the 11 weapon systems GAO reviewed, these 
problems have resulted in billions of dollars in cost overruns, years 
of schedule delays, and reduced weapon system availability. Prime 
contractors' poor systems engineering practices related to requirements 
analysis, design, and testing were key contributors to these quality 
problems. We also found problems with manufacturing and supplier 
quality that contributed to problems with DOD weapon systems. Senior 
officials from the prime contractor companies we contacted said that 
they agreed with our assessment of the causes of the quality problems 
of weapon system programs we reviewed and that disciplined processes 
help improve overall quality. 

Case Studies Illustrate Impact of DOD Weapon System Quality Problems: 

Quality problems caused significant cost and/or schedule delays in the 
11 weapon systems we reviewed. Figure 1 shows the types of problems we 
found and the resulting impacts. Appendix II contains detailed 
information about each of the programs' quality problems. Quality 
problems occurred despite the fact that each of the prime contractors 
for these programs is certified to commercial quality standards and 
most provided us with quality plans that address systems engineering 
activities such as design, as well as manufacturing, and supplier 
quality. However, quality problems in these areas point to a lack of 
discipline or an inconsistency in how prime contractors follow through 
on their quality plans and processes. 

Figure 1: Weapon System Quality Problems and Impact: 

[See PDF for image] 

This figure is a table illustrating Weapon System Quality Problems and 
Impact. For each system, there is an accompanying photograph. The 
following data is depicted: 

System: Advanced SEAL Delivery System[A]; 
Source of quality problem, Systems engineering: [Check]; 
Source of quality problem, Manufacturing: [Empty]; 
Source of quality problem, Supplier quality: [Check]; 
Impact of quality problem, Cost (in millions of dollars): $87; 
Impact of quality problem, Schedule: Program halted. 

System: Advanced Threat Infrared Countermeasure/Common Missile Warning 
System; 
Source of quality problem, Systems engineering: [Check]; 
Source of quality problem, : [Check]; 
Source of quality problem, Supplier quality: [Empty]; 
Impact of quality problem, Cost (in millions of dollars): $117; 
Impact of quality problem, Schedule: 5-year delay. 

System: Expeditionary Fighting Vehicle; 
Source of quality problem, Systems engineering: [Check]; 
Source of quality problem, Manufacturing: [Empty]; 
Source of quality problem, Supplier quality: [Empty]; 
Impact of quality problem, Cost (in millions of dollars): $750; 
Impact of quality problem, Schedule: 4-year extension to system 
development. 

System: F-22A; 
Source of quality problem, Systems engineering: [Check]; 
Source of quality problem, : [Check]; 
Source of quality problem, Supplier quality: [Check]; 
Impact of quality problem, Cost (in millions of dollars): $400; 
Impact of quality problem, Schedule: No schedule impact to program. 

System: Global Hawk[A]; 
Source of quality problem, Systems engineering: [Check]; 
Source of quality problem, Manufacturing: [Empty]; 
Source of quality problem, Supplier quality: [Check]; 
Impact of quality problem, Cost (in millions of dollars): $239; 
Impact of quality problem, Schedule: 4-month production slip for sensor 
suite. 

System: Joint Air-to-Surface Standoff Missile; 
Source of quality problem, Systems engineering: [Empty]; 
Source of quality problem, : [Check]; 
Source of quality problem, Supplier quality: [Check]; 
Impact of quality problem, Cost (in millions of dollars): $39; 
Impact of quality problem, Schedule: Program deferred. 

System: LPD 17 Amphibious Transport Dock[A]; 
Source of quality problem, Systems engineering: [Check]; 
Source of quality problem, Manufacturing: [Check]; 
Source of quality problem, Supplier quality: [Check]; 
Impact of quality problem, Cost (in millions of dollars): $846; 
Impact of quality problem, Schedule: 3-year delay. 

System: MH-60s Fleet Combat Support Helicopter; 
Source of quality problem, Systems engineering: [Empty]; 
Source of quality problem, : [Check]; 
Source of quality problem, Supplier quality: [Empty]; 
Impact of quality problem, Cost (in millions of dollars): No cost 
impact to program; 
Impact of quality problem, Schedule: 6-month production slip. 

System: Patriot Advanced Capability-3; 
Source of quality problem, Systems engineering: [Check]; 
Source of quality problem, Manufacturing: [Check]; 
Source of quality problem, Supplier quality: [Check]; 
Impact of quality problem, Cost (in millions of dollars): $26; 
Impact of quality problem, Schedule: 6-month delay. 

System: V-22 Joint Services Advanced Vertical Lift Aircraft; 
Source of quality problem, Systems engineering: [Check]; 
Source of quality problem, : [Empty]; 
Source of quality problem, Supplier quality: [Empty]; 
Impact of quality problem, Cost (in millions of dollars): $165; 
Impact of quality problem, Schedule: Flight operations halted for 17-
months. 

System: Wideband Global SATCOM; 
Source of quality problem, Systems engineering: [Empty]; 
Source of quality problem, Manufacturing: [Check]; 
Source of quality problem, Supplier quality: [Check]; 
Impact of quality problem, Cost (in millions of dollars): $10; 
Impact of quality problem, Schedule: 18-month delay for initial 
operating capability. 

Source: GAO analysis of DOD and prime contractor data. 

[A] Cost and schedule figures are not solely attributable to quality 
problems. 

[End of figure] 

Lack of Systems Engineering Discipline Early in Programs Leads to 
Significant Quality Problems Later: 

GAO's past work has identified systems engineering as a key practice 
for ensuring quality and achieving successful acquisition 
outcomes.[Footnote 5] Systems engineering is a sequence of activities 
that translates customer needs into specific capabilities and 
ultimately into a preferred design. These activities include 
requirements analysis, design, and testing in order to ensure that the 
product's requirements are achievable and designable given available 
resources, such as technologies. In several of the DOD weapon programs 
we reviewed, poor systems engineering practices contributed to quality 
problems. Examples of systems engineering problems can be found on the 
Expeditionary Fighting Vehicle, Advanced Threat Infrared 
Countermeasure/Common Missile Warning System, and Joint Air-to-Surface 
Standoff Missile programs. 

Design problems have hampered the development of the Marine Corps' 
Expeditionary Fighting Vehicle. The system, built by General Dynamics, 
is an amphibious vehicle designed to transport troops from ships 
offshore to land at higher speeds and from farther distances than its 
predecessor. According to program officials, prime contractor design 
and engineering changes were not always passed to suppliers, resulting 
in supplier parts not fitting into assemblies because they were 
produced using earlier designs. Systems engineering problems have also 
contributed to poor vehicle reliability, even though reliability was a 
key performance parameter. Consequently, the prime contractor was only 
able to demonstrate 7.7 hours between mission failures, which was well 
short of the 17 hours it needed to demonstrate in pre-production 
testing. Subsequently, the vehicle's development phase has been 
extended. Program officials estimate that this extension, which will 
primarily focus on improving reliability, will last an additional 4 
years at an estimated cost of $750 million. 

For several other weapon systems, inadequate testing was another 
systems engineering problem. The Army's Advanced Threat Infrared 
Countermeasure/Common Missile Warning System program, developed by BAE 
Systems, is designed to defend U.S. aircraft from advanced infrared- 
guided missiles. Reliability problems related to the Advanced Threat 
Infrared Countermeasure jam head forced the Army to initiate a major 
redesign of the jam head in fiscal year 2006, and fielding of the 
subsystem has been delayed until fiscal year 2010. According to a prime 
contractor official, the reliability problems were caused, at least in 
part, by inadequate reliability testing. Likewise, the Joint Air-to- 
Surface Standoff Missile program, developed by Lockheed Martin, has 
experienced a number of flight test failures that have underscored 
product reliability as a significant problem. Ground testing, which 
prime contractor officials said could have identified most of the 
failure modes observed in flight testing, did not occur initially. 
Prime contractor officials indicated that ground testing was not 
considered necessary because the program was a spin-off of a previous 
missile program and there was an urgent need for the new missile. As a 
result of the test failures, the program has initiated a reliability 
improvement effort that includes ground and flight testing. A program 
official reported that the cost of reliability improvements for fiscal 
years 2006 and 2007 totaled $39.4 million. 

Manufacturing Problems Are Often Caused by Lack of Process Controls: 

GAO's past work addresses the importance of capturing manufacturing 
knowledge in a timely manner as a means for ensuring that an 
organization can produce a product within quality targets.[Footnote 6] 
Prime contractor activities to capture manufacturing knowledge should 
include identifying critical characteristics of the product's design 
and then the critical manufacturing processes to achieve these 
characteristics. Once done, those processes should be proven to be in 
control prior to production. This would include making work 
instructions available, preventing and removing foreign object debris 
in the production process, and establishing criteria for workmanship. 
However, prime contractors' lack of controlled manufacturing processes 
caused quality problems on several DOD weapon programs, including the F-
22A and LPD 17 programs. 

The F-22A, a fighter aircraft with air-to-ground attack capability 
being built by Lockheed Martin, entered production with less than 50 
percent of the critical manufacturing processes in control. In 2000, 
citing budgetary constraints and specific hardware quality problems 
that demanded attention, the Air Force abandoned its efforts to get 
manufacturing processes in control prior to the start of production. 
Subsequently, the contractor experienced a scrap, rework, and repair 
rate of about 30 percent on early-production aircraft. The contractor 
also experienced major problems with the aircraft canopy. According to 
program officials, the aircraft uses a first-of-a-kind canopy, with an 
external metallic stealth layer. The contractor did not bring its 
manufacturing processes in control and the canopy cracked near the 
mounting holes. This problem was discovered in March 2000 and 
temporarily grounded the flight test aircraft. In addition, in 2006 a 
pilot was trapped in an F-22A for 5 hours when a defective activator 
prevented him from opening the canopy. According to the Air Force, when 
production began in 2001, the prime contractor should have been able to 
demonstrate that the F-22A could achieve almost 2 flying hours between 
maintenance. However, at that time, the contractor could demonstrate 
only about 40 minutes. Six years later, the contractor increased the 
flying hours to 97 minutes mean time, short of the Air Force's current 
3-hour requirement. The program now has budgeted an additional $400 
million to improve the aircraft's reliability and maintainability. 

Northrop Grumman, the prime contractor for the LPD 17, the first ship 
of a new class of amphibious transport dock ships, delivered the ship 
to the Navy in 2005 with many quality problems resulting from poor 
manufacturing practices. For example, the program experienced problems 
with non-skid coating applications because the company did not keep the 
boat surface free from dirt and debris when applying the coating, which 
caused it to peel. As of late 2007, the problem was not fixed. In 
addition, the ship encountered problems with faulty welds on piping 
used in some of the ship's hydraulic applications. According to the 
prime contractor, they could not verify that welds had been done 
properly. This problem required increased rework to correct the 
problems and reinspect all the welds. Had the problem not been 
discovered and weld failure had occurred, the crew and the ship could 
have been endangered. These problems, as well as many others, 
contributed to a 3-year delay and cost increase of $846 million in 
delivering the ship to the Navy. In June 2007, the Secretary of the 
Navy sent a letter to the Chairman of the Board of Northrop Grumman 
expressing his concerns about the contractor's ability to construct and 
deliver ships that meet Navy quality standards and to meet agreed-to 
cost and schedule commitments. 

Supplier Quality Problems Can Result in Higher Product Cost: 

Management of supplier quality is another problem area for DOD weapon 
systems. Supplier quality is particularly important because more that 
half of the cost of a weapon system can be attributed to material 
received by the prime contractor from its supplier base. While DOD 
prime contractors told us that they manage and control the quality of 
parts and material they receive from their suppliers with the help of 
performance reviews and process audits, we found supplier quality 
problems on seven of the weapon systems we reviewed. Two examples are 
the Wideband Global SATCOM and Patriot Advanced Capability-3 programs. 

Boeing Integrated Defense Systems is the prime contractor for the Air 
Force and Army's Wideband Global SATCOM communications satellite. 
Boeing Integrated Defense Systems discovered that one of its suppliers 
had installed certain fasteners incorrectly. As a result, 1,500 
fasteners on each of the first three satellites had to be inspected or 
tested, and 148 fasteners on the first satellite had to be reworked. 
The DOD program office reported that the resulting 15-month schedule 
slip would add rework and workforce costs to the program and delay 
initial operating capability by 18 months. A prime contractor official 
estimated the cost to fix the problem was about $10 million. 

In 2006, a supplier for the Patriot Advanced Capability-3 program, a 
long-range system that provides air and missile defense for ground 
combat forces, accepted non-conforming hardware for a component for the 
missile's seeker. The seeker contractor had to re-inspect components 
and some were returned for rework. As a result of this and other 
problems involving poor workmanship and inadequate manufacturing 
controls, the supplier facility was shut down for 7 months, delaying 
delivery of about 100 missiles. 

Prime Contractors' Observations on Quality: 

We met with senior quality officials at the prime contractor companies 
we included in this review to discuss the problems we found. For the 
most part, they agreed with our assessment, and that the discipline 
with which a company implements its processes is a key contributor to 
quality outcomes. The officials discussed the importance of quality and 
how they are attempting to improve quality across their companies. This 
includes the use of Six Sigma, a tool for measuring defects and 
improving quality, as well as independent program reviews and improving 
design processes.[Footnote 7] 

The senior quality officials also identified factors they believe 
affect the quality of DOD weapon systems, including insufficient 
attention to reliability by DOD during development and the prime 
contractor's lack of understanding of weapon system requirements, 
including those for testing. 

Leading Commercial Companies Use Disciplined Quality Management 
Practices: 

While there are similarities between the quality management practices 
of DOD prime contractors and leading commercial companies in our 
review, the discipline with which leading companies implement their 
practices contributes to the high quality of their products. According 
to company officials we contacted, reliability is a paramount concern 
for them because their customers demand products that work, and the 
companies must develop and produce high-quality products to sustain 
their competitive position in the marketplace. Leading commercial 
companies use disciplined, well-defined, and institutionalized 
practices for (1) systems engineering to ensure that a product's 
requirements are achievable with available resources, such as 
technologies; (2) manufacturing to ensure that a product, once 
designed, can be produced consistently with high quality and low 
variability; and (3) supplier quality to ensure that their suppliers 
are capable of delivering high-quality parts. These practices, which 
were part of the companies' larger product development processes, and 
other tools such as Six Sigma, provided an important foundation for 
producing quality products and continually improving performance. 

Adherence to Systems Engineering Practices Leads to Clear Requirements 
and Reliable Designs: 

Several of the companies we met with discussed how they use systems 
engineering as a key practice for achieving quality outcomes. As part 
of Siemens Medical Solutions' standard product development process, the 
company validates that product requirements are sufficiently clear, 
precise, measurable, and comprehensive. They ensure that requirements 
address quality, including requirements for reliability and readiness 
prior to making a commitment to developing and building a new product. 
Officials with Boeing Commercial Airplanes say they have shifted their 
view of quality into a more proactive approach, which includes a focus 
on "mistake-proofing" designs so that they can be assembled only one 
way. To help assess the producibility of critical parts designs, the 
company has also developed a tool that rates different attributes of 
the design, including clarity of engineering requirements, consequences 
of defects on performance or manufacturability, and verification 
complexity. Company officials say they use the tool's ratings to modify 
designs to ensure that parts will be less prone to manufacturing and 
assembly error, and that its use has resulted in lower costs for scrap, 
rework, and repair and fewer quality problems. 

Space Systems/Loral also relies on well-defined and disciplined 
processes to develop and produce satellites. Because the company's 
customers expect satellites to perform for up to 15 years, product 
reliability is paramount and company officials say that using systems 
engineering to design reliability into a satellite is essential. As 
part of its systems engineering activities, the company performs 
reliability assessments to verify that satellite components and 
subsystems will meet reliability requirements and to identify potential 
hardware problems early in the design cycle. Space Systems/Loral 
officials also discussed testing and its importance to developing 
products. For significant new product developments, Space Systems/Loral 
employs highly accelerated life testing to find weak links in a design 
and correct them to make the product more robust before going into 
production. As a result of the company's disciplined quality management 
practices, new satellite components--such as lithium-ion batteries, 
stationary plasma thrusters, and a satellite control system--have over 
80 million hours of operation in orbit with only one component failure, 
according to company data. 

Effective Manufacturing Process Controls Reduce Variability and 
Defects: 

Several company officials discussed the importance of having controlled 
manufacturing processes, and described several approaches to reduce 
variability and the likelihood of defects. These approaches greatly 
increase the likelihood that a product, once designed, can be produced 
consistently and with high quality and low variability. In this way, 
they reduce waste and increase a product's reliability in the field. 

Early in its product development process, Cummins, a manufacturer of 
diesel and natural gas-powered engines, establishes a capability growth 
plan for manufacturing processes. This increases the probability that 
the manufacturing process will consistently produce parts that meet 
specifications. Prior to beginning production, Cummins completes what 
it calls "alpha" and "beta" builds, which are prototypes intended to 
validate the product's design and production processes. Cummins 
officials noted that these activities allow them to catch problems 
earlier in development, when problems are less costly to fix. 

Officials from Kenworth, a manufacturer of heavy-and medium-duty 
trucks, described several initiatives it uses to improve manufacturing 
process controls. For example, the company has a new electronic system 
for process documents. Workers on the manufacturing floor used to rely 
on paper installation instructions, and sometimes workers used outdated 
instructions. Kenworth officials say that converting to an electronic 
system ensures that all workers use the most current process 
configuration and reduces rework. For a selected number of processes, 
Kenworth has also developed documents that include pictures as well as 
engineering specifications to ensure that workers follow the correct 
processes, and performs audits to assess whether workers are properly 
trained and know where to go if they have questions regarding the 
process. 

Companies Hold Suppliers Accountable to Deliver High Quality Parts for 
the Product: 

At several of the companies we visited, officials reported that 
supplier parts accounted for a substantial amount of the overall 
product value. Companies we met with systematically manage and oversee 
their supply chain through such activities as regular supplier audits 
and performance evaluations of quality and delivery, among other 
things. Several officials noted that their supplier oversight focuses 
on first-tier suppliers, with limited interaction and oversight of 
lower-tier suppliers. However, Kenworth officials said they hold their 
first-tier suppliers accountable for quality problems attributable to 
lower-tier suppliers. 

Leading commercial companies we met with set high expectations for 
supplier quality. Boeing Commercial Airplanes categorizes its suppliers 
by rates of defective parts per million. To achieve the highest rating 
level, a supplier must exhibit more than 99 percent part conformance, 
and company officials said they have been raising their supplier 
quality expectations over time. The organization has taken steps to 
reduce the number of direct suppliers and retain higher-performing 
suppliers in the supply base. Similarly, suppliers of major components 
for Siemens Medical Solutions' ultrasound systems must provide 
conforming products 98 percent of the time, and the company will levy 
financial penalties against suppliers that do not meet this standard. 
Other companies also financially penalized suppliers for providing 
nonconforming parts. 

Disciplined Processes and Continuous Quality Improvement Are a Focus at 
Several Companies: 

Several company officials discussed how a focus on improving product 
development processes and product quality served as the foundation for 
their systems engineering, manufacturing, and supplier quality 
practices. Officials with Space Systems/Loral discussed how they 
adopted a more disciplined product development process following 
quality problems in the 1990s with some of its satellites. This 
included creating companywide product development processes, adopting a 
formal program that institutionalized an iterative development process, 
and implementing strict documentation requirements and pass/fail 
criteria. The company also established an oversight organization to 
ensure that processes are followed. As a result, the first-year failure 
rate for Space Systems/Loral's satellites decreased by approximately 50 
percent from 2000 through 2006. Likewise, Cummins officials told us 
that quality problems following the initial release of their ISX engine 
were a major factor in the implementation of their current product 
development process. This includes review gates to ensure process 
compliance and management reviews that use knowledge-based approaches 
for evaluating projects. 

Cummins and Kenworth also use tools such as Six Sigma to define, 
measure, analyze, control, and continually improve their processes. For 
example, Cummins applies Six Sigma to its technology development, 
design, and production activities. The company also expects its 
critical suppliers to implement Six Sigma programs to improve quality 
and customer satisfaction. As a result of implementing initiatives such 
as Six Sigma, Cummins officials reported that the company's warranty 
costs have declined substantially in the last several years. Kenworth 
also uses Six Sigma to drive efficiencies into the organization's work 
processes, particularly in the design phase of new product development 
and in controlling manufacturing processes. Kenworth requires its first-
tier suppliers to participate in a Six Sigma program. Company officials 
estimated that Six Sigma projects saved its Chillicothe, Ohio, facility 
several million dollars in 2006. 

In addition, each of the commercial companies we met with collected and 
used data to measure and evaluate their processes and products. This 
helps them gauge the quality of their products and identify areas that 
need improvement. For example, Cummins tracks warranty costs as a 
measure of product quality, while Siemens Medical Solutions measures 
manufacturing process yields for its ultrasound systems. 

Different Environments Create Different Incentives to Improve Quality: 

The quality problems in our case studies and the practices that relate 
to them--whether systems engineering, manufacturing, or supplier 
quality practices--are strongly influenced and often the result of 
larger environmental factors. DOD's acquisition environment is not 
wholly conducive to incentivizing prime contractors to efficiently 
build high-quality weapon systems--ones that perform as expected, can 
be depended on to perform when needed, and are delivered on time and 
within cost estimates. During systems development, DOD usually pays for 
a contractor's best efforts, which can include efforts to achieve 
overly optimistic requirements. In such an environment, seeking to 
achieve overly optimistic requirements along with a lack of oversight 
over the development process contributes to quality problems. In 
contrast, commercial companies we visited operate in an environment 
that requires their own investment of significant funds to develop new 
products before they are able to sell them and recoup that investment. 
This high-cost environment creates incentives for reasonable 
requirements and best practices, as well as continuous improvement in 
systems engineering, manufacturing, and supplier quality. 

DOD's Environment: 

DOD uses cost reimbursement contracts with prime contractors for the 
development of its weapon systems. In this type of contract 
arrangement, DOD accepts most of the financial risks associated with 
development because of technical uncertainties. Because DOD often sets 
overly optimistic requirements for new weapon systems that require new 
and unproven technologies, development cycles can take up to 15 years. 
The financial risk tied to achieving these requirements during 
development is not borne by the contractor in this environment, but by 
the government. This environment provides little incentive for 
contractors to utilize the best systems engineering, manufacturing, and 
supplier quality practices discussed earlier in this report to ensure 
manageable requirements, stable designs, and controlled manufacturing 
processes to hold costs down. Finally, DOD's quality organizations, 
which collect information about prime contractors' quality systems and 
problems, provide limited oversight of prime contractor activities and 
do not aggregate quality data in a manner that helps decision makers 
assess or identify systemic quality problems. 

Overly Optimistic Requirements Hamper Good Quality Outcomes: 

DOD's ability to obtain a high-quality weapon system is adversely 
impacted by an environment where it both (1) assumes most of the 
financial risks associated with technical or cost uncertainties for the 
systems development and (2) sets requirements without adequate systems 
engineering knowledge.[Footnote 8] Without requirements that have been 
thoroughly analyzed for feasibility, development costs are impossible 
to estimate and are likely to grow out of control. 

DOD typically assumes most of the financial risk associated with a new 
weapon system's development by establishing cost reimbursement 
contracts with prime contractors. In essence, this means that prime 
contractors are asked to give their best effort to complete the 
contract and DOD pays for allowable costs, which often includes fixing 
quality problems experienced as part of the effort. As stated earlier, 
these problems can cost millions of dollars to fix. For example, DOD as 
the customer for the Expeditionary Fighting Vehicle signed a cost 
reimbursement contract with the prime contractor, General Dynamics, to 
develop a new weapon system that would meet performance and reliability 
requirements that had not yet been adequately informed by systems 
engineering analysis. Once General Dynamics performed a detailed 
requirements analysis, it informed DOD that more resources would be 
needed to meet the key reliability requirement established earlier. DOD 
decided not to invest the additional money at that time. However, when 
the vehicle was unable to meet its reliability goal prior to moving 
into production, DOD eventually decided to invest an additional $750 
million into its development program to meet the reliability goal. 

Often DOD enters into contracts with prime contractors before 
requirements for the weapon systems have been properly analyzed. For 
example, in March 2007 we reported that only 16 percent of the 62 DOD 
weapon system programs we reviewed had mature technologies to meet 
requirements at the start of development.[Footnote 9] The prime 
contractors on these programs ignored best systems engineering 
practices and relied on immature technologies that carry significant 
unknowns about whether they are ready for integration into a product. 
The situation is exacerbated when DOD adds or changes requirements to 
reflect evolving threats. Prime contractors must then spend time and 
resources redesigning the weapon system, flowing down the design 
changes to its suppliers, and developing new manufacturing plans. In 
some cases, special manufacturing tools the prime contractor thought it 
was going to use might have to be scrapped and new tooling procured. 

Lack of detailed requirements analysis, for example, caused significant 
problems for the Advanced Threat Infrared Countermeasure/Common Missile 
Warning System program. Prior to 1995, the services managed portions of 
the program separately. Then, in 1995, DOD combined the efforts and 
quickly put a developer on contract. This decision resulted in 
significant requirements growth and presented major design and 
manufacturing difficulties for the prime contractor. It took over a 
year to determine that the tactical fixed-wing aircraft requirements 
were incorrect. The extent of the shortfall, however, did not become 
evident until the critical design review and numerous changes were 
required in the contract statement of work. More than 4 years after the 
system's critical design review, the sensor units were built in 
prototype shops, with engineers only then trying to identify critical 
manufacturing processes. Further, sensor manufacturing was slowed by 
significant rework, and at one point was halted while the contractor 
addressed configuration control problems. The Navy and Air Force, which 
required the system for fixed-wing aircraft, dropped out of the program 
in 2000 and 2001, respectively. 

Ultimately, quality is defined in large part by reliability. But, in 
DOD's environment, reliability is not usually emphasized when a program 
begins, which forces the department to fund more costly redesign or 
retrofit activities when reliability problems surface later in 
development or after a system is fielded.[Footnote 10] The F-22A 
program illustrates this point. Because DOD as the customer assumed 
most of the financial risk on the program, it made the decision that 
system development resources primarily should be focused on 
requirements other than reliability, leading to costly quality 
problems. After 7 years in production, the Air Force had to budget an 
additional unplanned $400 million for the F-22A to address numerous 
quality problems and help the system achieve its baseline reliability 
requirements. 

Oversight of Development Programs Could Be Strengthened: 

DOD oversight of prime contractor activities varies and has decreased 
as its quality assurance workforce has decreased. Weapon system 
progress reviews at key decision points are a primary means for DOD to 
oversee prime contractor performance in building high-quality systems, 
but they are not used consistently across programs. The purpose of the 
reviews is to determine if the program has demonstrated sufficient 
progress to advance to the next stage of product development or to 
enter production. The department has developed decision criteria for 
moving through each phase of development and production; and DOD's 
acquisition executive has the authority to prevent programs from 
progressing to later stages of development if requisite knowledge has 
not been attained. Unfortunately most programs are allowed to advance 
without demonstrating sufficient knowledge. For example, in our recent 
review of 62 DOD weapon systems, we found that only 27 percent of the 
programs demonstrated that they had attained a stable design at the 
completion of the design phase.[Footnote 11] 

In addition, as a result of downsizing efforts over the past 15 years, 
DOD's oversight of prime contractor and major supplier manufacturing 
processes varies from system to system. DOD quality officials stated 
that they have had to scale back on the amount of oversight they can 
provide, focusing only on the specific areas that the weapon system 
program managers ask them to review. It is unclear what impact the 
reduction in quality assurance specialists and the reduction of 
oversight has had on the department's ability to influence quality 
outcomes. However, in the case of the Advanced SEAL Delivery System, a 
lapse in effective management oversight exercised by both the 
government and contractor contributed to very late discovery of costly 
quality problems. DOD quality organizations such as the Defense 
Contract Management Agency do capture a significant amount of 
information electronically about the quality of DOD weapon systems 
through audits and corrective action reports. They collect quality data 
on a program by program basis and share information about certain types 
of deficiencies and nonconforming parts they found. While the 
organizations are looking for additional opportunities to share 
information, they do not currently aggregate and consolidate the 
information in a manner that would allow the department to determine 
the overall quality of products it receives from prime contractors or 
to identify quality related systemic problems or trends with its prime 
contractors. 

Commercial Environment: 

Commercial companies must develop and deliver high-quality, highly 
capable products to markets on-time or suffer financial loss. The 
companies face competition and, therefore, their customers can choose 
someone else's products when they are not satisfied. It is this 
environment that incentivizes manufacturers to implement and use best 
practices to improve quality and reduce cost while delivering on-time. 
Commercial customers must set achievable product requirements for their 
manufacturers that they know will result in a reliable, high-quality, 
and desirable product that can be delivered on-time. Manufacturers then 
get their key manufacturing processes in control to reduce 
inconsistencies in the product. Commercial customers understand the 
need to monitor and track manufacturer and supplier quality performance 
over time to determine which companies they want to do business with in 
the future or to identify problem areas that need to be corrected. 

Commercial customers we visited--American Airlines and Intelsat-- 
expect to operate their products for 30 and 15 years, respectively. The 
companies focus a great deal of attention on setting performance and 
reliability goals that manufacturers like Boeing Commercial Airplanes 
and Space Systems/Loral must meet in order for them to purchase their 
products. This provides a strong, direct incentive for manufacturers 
and their customers to ensure that requirements are clear and 
achievable with available resources, including mature technologies, 
before the manufacturer will invest in a product's development. For 
example, Intelsat expects its satellites to be available at least 
99.995 percent of the time. To meet this goal, Intelsat expects its 
manufacturers to use mature technologies and parts where the 
reliability is already known. There are several reasons that drive this 
approach. The most obvious one is that there is no way to fix 
mechanical problems once a satellite has been launched. Another reason 
is that the company must credit television networks, telephone 
companies, or cable companies for any loss of service. The company also 
insures their satellite for launch plus the first year of in-orbit 
service. Having a proven record of in-orbit performance and using 
reliable and flight proven technology are two important factors that 
help the company get favorable terms from the insurance underwriters. 
And, the company does not want to spend a large sum of money for a 
replacement satellite prior to its design life since it will negatively 
impact the company's financial performance. 

In the commercial environment, manufacturers are motivated to develop 
and provide high-quality products because their profit is tied to 
customer expectations and satisfaction. For example, American Airlines 
makes an initial payment to Boeing Commercial Airplanes when it places 
an order for new aircraft, but will not make final payment until it is 
satisfied that their requirements have been met. 

In an another example, Cummins officials discussed how they were 
motivated to adopt more disciplined product development processes 
following the development effort for one of its highest selling 
engines, in the late 1990s. According to company officials, the design 
requirements were unstable from the start of development. They were 
changed and added to as development progressed, often without the 
benefit of timely and disciplined requirements analysis to ensure they 
could be met for the estimated investment cost. There were conflicting 
requirements (weight, size, performance, and fuel economy) that made 
development difficult. In addition, Cummins did not pay enough 
attention to reliability, focusing instead on weight and power 
considerations. As a result, development costs were higher than 
expected and, once the engine was sold, customers experienced less than 
expected. A Cummins official reported that the company found itself in 
an "intolerable" position with customers who were becoming increasingly 
dissatisfied. 

This significant event, in which Cummins lost customer confidence, 
caused the company to examine its product development processes. The 
result of this examination was an improved product development process 
that requires a more cross functional and data-based approach to new 
development programs. The improvements resulted in better analysis and 
understanding of customer requirements leading to resource allocations 
before beginning new programs. Cummins invested in both customer 
satisfaction and the development and support of its products. This 
investment provided the motivation to adopt a more disciplined product 
development approach for the production of high-quality products for 
its customers. 

Intelsat officials told us it makes progress payments to its 
manufacturers throughout development and production. However, the 
company holds about 10 to 20 percent of the contract value to award to 
the manufacturer after a satellite is successfully launched. According 
to company officials, the 10 to 20 percent is paid to the manufacturer 
over the expected life of the satellite, which is typically 15 years, 
when the satellite performs as expected. 

The commercial companies also all capture information about their 
manufacturing processes and key suppliers' quality. However, unlike 
DOD, they use the information when making purchasing decisions and 
determining how best to structure contracts to incentivize good quality 
outcomes. For example, in some cases Intelsat does not allow 
manufacturers to use certain suppliers whose parts do not meet 
specified reliability goals. In addition, Intelsat may include clauses 
in its contracts that require a manufacturer to conduct periodic 
inspections of particular suppliers. 

DOD Efforts to Improve Acquisition Outcomes: 

DOD has long recognized its acquisition problems and has initiated 
numerous improvement efforts over the years to address them. A recent 
set of initiatives are highlighted by the Under Secretary of Defense 
for Acquisition, Technology and Logistics in DOD's Defense Acquisition 
Transformation and Program Manager Empowerment and Accountability 
reports to Congress.[Footnote 12] Our analysis indicates that while 
none of the initiatives is aimed solely at improving the quality of DOD 
weapon systems or improving prime contractor quality practices, they 
could address some of the problems identified in this report, 
particularly the ones that improve the DOD requirements-setting process 
and limit requirements growth during development. A brief description 
of the initiatives is included below. 

* Concept Decision Reviews: DOD is pilot-testing a concept decision 
reviews program to provide a better framework for strategic investment 
decisions. A Concept Decision Committee composed of senior DOD 
officials is applying the reviews to four pilot programs--the Joint 
Lightweight Tactical Mobility program, the Integrated Air and Missile 
Defense program, the Global Strike Raid Scenario, and the Joint Rapid 
Scenario Generation program. A key aspect of the pilot programs is the 
early involvement and participation of systems engineering prior to 
concept decision. DOD expects this to provide decision makers better 
insight for setting firm requirements early, assessing technology 
options, considering alternative acquisition strategies, ensuring that 
new technology will mature in time to meet development and delivery 
schedules, and delivering systems with predictable performance to the 
warfighter. 

* Time-Defined Acquisition: Under the time-defined acquisition 
initiative, DOD plans to use such criteria as technology maturity, time 
to delivery, and requirement certainty to select the appropriate 
acquisition approach to provide a needed capability. The department 
envisions using a different acquisition approach, depending on whether 
a capability can be fielded in 2 years or less, more than 2 years to 
less than 4 years, or more than 4 years. In September 2006, the Under 
Secretary of Defense for Acquisition, Technology and Logistics stated 
that he anticipated the time-defined acquisition approach would 
facilitate better overall cost control and more effective use of total 
available resources. 

* Configuration Steering Boards: In July 2007, the Under Secretary of 
Defense for Acquisition, Technology and Logistics directed the 
establishment of Configuration Steering Boards for every current and 
future acquisition category I program in development.[Footnote 13] The 
boards, chaired by the service acquisition executive within each of the 
military services, are expected to review all requirements changes and 
significant technical configuration changes that have the potential to 
adversely affect program cost and schedule. Requirement changes are not 
to be approved unless funds are identified and schedule impacts are 
mitigated. However, the Under Secretary stated in his announcement of 
this initiative that such requirements changes would usually be 
rejected. 

* Key Performance Parameters/Key System Attributes:[Footnote 14] DOD 
has added new guidelines and procedures for establishing weapon system 
requirements in its Joint Capabilities Integration and Development 
System manual. The manual now requires that materiel availability be 
included as a key performance parameter for new weapon system 
development and that materiel reliability and ownership costs be 
included as key system attributes. Together, these requirements are 
aimed at ensuring weapon system sustainment considerations are fully 
assessed and addressed as part of the systems engineering process. 

* Award and Incentive Fees: DOD recently issued policy memorandums that 
reflect a change in policy related to the proper use of award and 
incentive fees. The memorandums emphasize the need to structure award 
fee contracts in ways that focus DOD and contractor efforts on meeting 
or exceeding cost, schedule, and performance requirements. The policy 
memorandums state that award fees should be linked to desired outcomes 
and payments should be commensurate to contractor performance. It also 
provides guidelines for how much contractors will be paid for excellent 
performance, satisfactory performance, and less than satisfactory 
performance. 

While these initiatives are not directly linked together, they have the 
potential to help DOD implement some of the leading commercial 
practices we have highlighted in the past. In particular, they could 
help the Under Secretary of Defense for Acquisition, Technology and 
Logistics ensure that DOD has a better match between warfighter needs 
and funding at the start of weapon system development and that 
technology, engineering, and production knowledge is properly 
considered at that time. They can also help control requirements 
changes and requirements growth, which can adversely affect system 
quality during development. The initiatives are still new and, in the 
case of concept decision reviews, small in scope; therefore, their 
effectiveness may not be known for some time. 

Conclusions: 

DOD has developed policies that address the need for setting achievable 
requirements, adopting commercial quality standards, using good systems 
engineering practices, and overseeing supplier quality. However, DOD 
still has difficulty acquiring high-quality weapon systems in a cost- 
efficient and timely manner. While many problems are caused by poor 
prime contractor practices related to systems engineering, 
manufacturing, and supplier quality, an underlying cause lies in the 
environment. DOD typically assumes most of the financial risk 
associated with development of complex systems. However, risks 
associated with this situation are exacerbated because DOD generally 
enters into development contracts without demonstrated knowledge or 
firm assurance that requirements are achievable, which too often result 
in inefficient programs and quality problems. 

DOD can learn from leading commercial companies in the way they deal 
with risk and ensure quality in their products. Because commercial 
companies invest their own money in product development and recoup that 
investment when their customers buy the finished good, they put a new 
product's requirements to the test with disciplined systems engineering 
practices before they commit to a large investment to develop it. If a 
highly valued requirement cannot be demonstrated as achievable through 
systems engineering, it is deferred to a subsequent product variation 
or to another program. Moreover, and very importantly, companies do not 
shortcut essential quality practices that ensure process controls and 
high supplier quality, including collecting and analyzing quality data. 
Like commercial companies, DOD must demand appropriate knowledge about 
requirements and make hard decisions about program risk before it 
initiates costly investments. 

Improvements in the way DOD uses existing tools to analyze requirements 
during development, along with potential results of some of the 
initiatives it has underway, can help reduce quality risks, and address 
some of the long-standing acquisition problems it faces. Although the 
initiatives are new and in the case of the concept decision reviews, 
small in scope, they are a good first step toward the department 
setting more realistic requirements and time frames for weapon system 
development. Additional oversight could help ensure that prime 
contractors can meet requirements with given resources, such as funding 
and technologies, prior to DOD entering into a development contract. In 
addition, continued leadership from the Under Secretary of Defense for 
Acquisition, Technology and Logistics and a combination of actions from 
both DOD and prime contractors are needed to make these improvements 
and get the most from its planned $1.5 trillion investment in new 
weapons programs. 

Recommendations for Executive Action: 

To ensure that the department is taking steps to improve the quality of 
weapon systems, we recommend that the Secretary of Defense take the 
following actions related to recent initiatives highlighted in DOD's 
Defense Acquisition Transformation and Program Manager Empowerment and 
Accountability reports to Congress to improve its focus on setting 
achievable requirements and oversight: 

* As a part of the concept decision review initiative, have contractors 
perform more detailed systems engineering analysis to develop sound 
requirements before DOD selects a prime contractor for the systems 
development contract, which would help ensure that weapon system 
requirements, including those for reliability, are achievable with 
given resources. 

* Establish measures to gauge the success of the concept decision 
reviews, time-defined acquisition, and configuration steering board 
initiatives and properly support and expand these initiatives where 
appropriate. 

To better assess the quality of weapon system programs and prime 
contractor performance, DOD needs to obtain and analyze more 
comprehensive data regarding prime contractors and their key suppliers. 
Therefore, we also recommend that the Secretary of Defense direct the 
Defense Contract Management Agency and the military services to: 

* Identify and collect data that provides metrics about the 
effectiveness of prime contractors' quality management system and 
processes by weapon system and business area over time and: 

* Develop evaluation criteria that would allow DOD to score the 
performance of prime contractors' quality management systems based on 
actual past performance, which could be used to improve quality and 
better inform DOD acquisition decision makers. 

Agency Comments and Our Evaluation: 

DOD provided us with written comments on a draft of this report. DOD 
partially concurred with each of the recommendations. DOD's comments 
appear in appendix III. 

In its comments, DOD partially concurred with the draft recommendation 
that, as part of its concept decision review initiative, prime 
contractors should complete systems engineering analysis prior to 
entering a development contract. The department stated that the 
recommendation was vague. DOD noted that it conducts systems 
engineering planning prior to entering into a development contract and 
that prime contractors conduct more detailed systems engineering 
analysis afterwards. Moreover, DOD noted that systems engineering is a 
continuous government-performed activity at the heart of any structured 
development process that proceeds from concept to production. The 
concept decision review initiative, in particular, considers 
fundamental systems engineering issues such as technology and 
integration and manufacturing risk before the concept decision review. 

To address DOD's concern that our recommendation was too vague, we 
modified it to add more detail. Specifically, as part of the concept 
decision review initiative, we recommend that contractors that are 
competing for the systems development contract provide DOD more 
detailed systems engineering requirements analysis to be completed 
before a systems development contract is awarded. This would help 
ensure that requirements are clear and reasonable before DOD enters 
into a development contract. We understand that currently DOD conducts 
systems engineering planning prior to entering a development contract 
with prime contractors and that prime contractors conduct a more 
thorough systems engineering analysis afterwards. However, because our 
work has found that many DOD systems development efforts have been 
hampered by poorly defined or poorly understood requirements, we 
believe that DOD should test, through the concept decision initiative, 
paying contractors to complete a more thorough systems engineering 
analysis prior to entering into a development contract. This would give 
the department the benefit of more knowledge when finalizing 
requirements and provide an opportunity for DOD to set requirements 
that can be met in a well-defined time frame, which could reduce the 
department's risk exposure in cost reimbursement contracts used for 
development. In addition, it would better position DOD to place more 
accountability on the winning contractor to meet the desired 
requirements within cost and schedule estimates. 

DOD also partially concurred with the recommendation to establish 
measures to gauge the success of the concept decision reviews, time- 
defined acquisition, and configuration steering board initiatives and 
properly support and expand these initiatives where appropriate. In its 
response, DOD stated that changes to the concept decision review and 
time-defined acquisition initiatives are being considered and any 
changes would be reflected in an update to DOD Instruction 5000.2. DOD 
also stated that the configuration steering board initiative is being 
implemented consistent with its policy. 

We are encouraged by the potential changes that could result from 
successful implementation of the concept decision reviews, time-defined 
acquisition, and configuration steering board initiatives. We believe 
that these three initiatives are aimed at addressing several of DOD's 
systemic problems that impact weapon system quality and that the 
department should not lose sight of these initiatives. While the 
initiatives are new and untested in practice, acquisition history tells 
us that these policy changes alone will not be sufficient to change 
outcomes. We have found that measures to gauge success can help 
facilitate senior-level oversight that is needed to bring about 
significant change within an organization. We, therefore, believe this 
recommendation remains valid. 

DOD partially concurred with the recommendation for the Defense 
Contract Management Agency and military services to identify and 
collect data that provides metrics about the effectiveness of prime 
contractors' quality management systems and processes by weapon system 
and business area over time. In its response, DOD stated that the 
Defense Contract Management Agency is in the process of identifying and 
will eventually collect data that could be used to determine the 
effectiveness of prime contractors' quality management systems. 
However, DOD stated that the added expense of capturing data by weapon 
system and business area does not seem warranted at this time. Further 
it commented that there is no need for the military services to engage 
in a similar effort to the Defense Contract Management Agency, since 
the agency is working in cooperation with the military services. 

We are encouraged by the Defense Contract Management Agency's efforts 
to identify and collect data on prime contractor quality management 
activities on a broad scale. As we noted in the report, this is a 
practice used by leading commercial companies we visited. During our 
review, the agency could only provide data on a weapon system by weapon 
system basis. We believe that data should be captured on both a weapon 
system and prime contractor basis and that the added expense of 
including data by weapon system is likely minimal, given that it is 
already being collected that way. Considering that DOD plans to invest 
about $1.5 trillion (in 2007 dollars) in its current portfolio of major 
weapon systems, we believe it would be valuable for DOD to know how the 
companies and business units responsible for delivering its high- 
quality weapon systems are performing as well as the quality associated 
with individual weapon systems. In addition, we believe the military 
services, particularly the Navy's Supervisor of Shipbuilding, which is 
responsible for overseeing contractor activities for shipbuilding, 
should identify and collect similar data so that information collected 
is consistent and can be used for comparison purposes. We, therefore, 
believe this recommendation remains valid. 

Finally, DOD partially concurred with the recommendation for the 
Defense Contract Management Agency and military services to develop 
evaluation criteria that would allow DOD to score the performance of 
prime contractors' quality management systems based on actual past 
performance. DOD stated that it plans to develop evaluation criteria 
based on data the Defense Contract Management Agency plans to collect 
in the future. DOD does not think the military services need to develop 
a parallel effort because Defense Contract Management Agency data will 
be shared with the military services. 

It was not our intent for the military services, the Defense Contract 
Management Agency, and the Navy's Supervisor of Shipbuilding to have 
parallel efforts. Rather, we expected that they would work 
collaboratively on this effort. Moreover, not only do we believe DOD 
should know how well the prime contractors and their respective 
programs are performing as noted above, we also believe that DOD should 
know how well the prime contractors' quality management systems are 
working. Again, this is a practice used by leading commercial companies 
we visited. We are encouraged that the Defense Contract Management 
Agency plans to develop evaluation criteria that would be used to score 
prime contractor quality management systems but believe the department 
should have a consistent methodology to be used across DOD. We, 
therefore, believe this recommendation remains valid. 

We are sending copies of this report to the Secretary of Defense and 
interested congressional committees. We will also make copies available 
at no charge on the GAO Web site at http://www.gao.gov. 

If you have any questions about this report or need additional 
information, please contact me at (202) 512-4841 or [email protected]. 
Contact points for our Offices of Congressional Relations and Public 
Affairs may be found on the last page of this report. Key contributors 
to this report are listed in appendix IV. 

Signed by: 

Michael Sullivan: 
Director:
Acquisition and Sourcing Management: 

List of Congressional Committees: 

The Honorable Carl Levin:
Chairman:
The Honorable John McCain:
Ranking Member:
Committee on Armed Services:
United States Senate: 

The Honorable Daniel K. Inouye:
Chairman:
The Honorable Ted Stevens:
Ranking Member:
Subcommittee on Defense:
Committee on Appropriations:
United States Senate: 

The Honorable Ike Skelton:
Chairman:
The Honorable Duncan Hunter:
Ranking Member:
Committee on Armed Services:
House of Representatives: 

The Honorable John P. Murtha:
Chairman:
The Honorable C.W. Bill Young:
Ranking Member:
Subcommittee on Defense:
Committee on Appropriations:
House of Representatives: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

This report compares Department of Defense (DOD) and its large prime 
contractors' quality management policies and practices with those of 
leading commercial companies--with a focus on improving the quality of 
DOD weapon systems. Specifically, we (1) determined the impact of 
quality problems on selected DOD weapon systems and defense 
contractors' practices that contributed to the problems, (2) identified 
practices used by leading commercial companies that can be used to 
improve the quality of DOD weapon systems, (3) identified problems DOD 
faces in terms of improving quality, and (4) identified recent DOD 
initiatives that could improve quality. 

To determine the impact of quality problems on selected DOD weapon 
systems and defense contractors' practices that contribute to the 
problems, we selected and reviewed 11 DOD weapon systems with known 
deficiencies from each of the military services and identified the 
quality problems associated with each deficiency. The 11 were chosen to 
demonstrate the types of problems DOD weapon systems experience and to 
help focus our discussions with leading commercial companies on aspects 
of development that caused DOD major quality problems. The prime 
contractors in charge of developing these systems include six of DOD's 
largest contractors; together, they are involved with a little over $1 
trillion, or about 76 percent, of the $1.5 trillion (in 2006 dollars) 
DOD plans to spend on weapon systems in its current portfolio. Systems 
we reviewed, along with the prime contractors responsible for 
developing the systems, are: 

* Advanced SEAL Delivery System, a battery-powered submarine funded by 
the Special Operations Command and developed by Northrop Grumman; 

* Advanced Threat Infrared Countermeasure/Common Missile Warning 
System, a defense countermeasure system for protection against infrared 
guided missiles in flight funded primarily by the Army and developed by 
BAE Systems; 

* Expeditionary Fighting Vehicle, an amphibious and armored tracked 
vehicle funded by the Navy for the Marine Corps and developed by 
General Dynamics; 

* F-22A, an air superiority fighter with an air-to-ground attack 
capability funded by the Air Force and developed by Lockheed Martin; 

* Global Hawk, a high-altitude, long endurance unmanned aircraft funded 
by the Air Force and developed by Northrop Grumman; 

* Joint Air-to-Surface Standoff Missile, an air-to-surface missile 
funded by the Air Force and developed by Lockheed Martin; 

* LPD 17, an amphibious transport ship funded by the Navy and developed 
by Northrop Grumman; 

* MH-60S, a fleet combat support helicopter funded by the Navy and 
developed by Sikorsky Aircraft; 

* Patriot Advanced Capability-3, a long-range high-to-medium altitude 
missile system funded by the Army and developed by Lockheed Martin; 

* V-22, a tilt rotor, vertical/short take-off and landing aircraft 
funded primarily by the Navy for the Marine Corps and developed jointly 
by Bell Helicopter Textron and Boeing Integrated Defense Systems; and: 

* Wideband Global SATCOM, a communications satellite funded by the Air 
Force and developed by Boeing Integrated Defense Systems. 

To evaluate each of the 11 DOD weapon systems, we examined program 
documentation, such as deficiency reports and corrective action 
reports, and held discussions with quality officials from DOD program 
offices, the prime contractor program office, and either the Defense 
Contract Management Agency or the Supervisor of Shipbuilding office 
where appropriate. Based on information gathered through documentation 
and discussions, we grouped the problems into three general categories: 
systems engineering, manufacturing, and supplier quality. When 
possible, we identified the impact that quality problems had on system 
cost, schedule, performance, reliability, availability, or safety. 
After completing our weapon systems reviews, we held meetings with 
senior quality leaders at selected prime contractors included in our 
review to discuss the quality problems we found and to obtain their 
views on why the problems occurred. 

To identify practices used by leading commercial companies that can be 
used to improve the quality of DOD weapon systems, we selected and 
visited five companies based on several criteria: companies that make 
products similar to DOD weapon systems in terms of complexity; 
companies that have been recognized in quality management literature or 
by quality-related associations/research centers for their high- 
quality products; companies that have won quality-related awards; and/ 
or companies that have close relationships with customers when 
developing and producing products. We met with these companies to 
discuss their product development and manufacturing processes, supplier 
quality activities, and the quality of selected products made by these 
companies. Much of the information we obtained from these companies is 
anecdotal, due to the proprietary nature of the data that could affect 
their competitive standing. Several of the companies provided data on 
specific products, which they agreed to let us include in this report. 
The companies we visited and the products we discussed include: 

* Boeing Commercial Airplanes, a leading aerospace company and a 
manufacturer of commercial jetliners. We met with quality officials in 
Seattle, Washington, and discussed the quality practices associated 
with the company's short-to-medium range 737 and extended range 777 
aircraft, as well as its new 787 aircraft. 

* Cummins Inc., a manufacturer of diesel and natural gas-powered 
engines for on-highway and off-highway use. We met with quality 
officials at their company's headquarters location in Columbus, 
Indiana, and discussed the development and quality of the ISX, a heavy- 
duty engine. 

* Kenworth Truck Company, a division of PACCAR Inc. and a leading 
manufacturer of heavy-and medium-duty trucks. We met with quality 
officials at its manufacturing plant in Chillicothe, Ohio, which was 
named Quality Magazine's 2006 Large Plant of the Year, to discuss the 
development and quality of various large trucks. 

* Siemens Medical Solutions, a business area within Siemens AG, which 
is a global producer of numerous products, including electronics, 
electrical equipment, and medical devices. We met with quality 
officials at a company facility located in Mountain View, California, 
and discussed the division's quality practices for developing and 
manufacturing ultrasound systems such as the Sequoia ultrasound system. 

* Space Systems/Loral, one of the world's premier designers, 
manufacturers, and integrators of geostationary satellites and 
satellite systems. We met with quality officials at the company's 
headquarters in Palo Alto, California, and discussed the company's 
quality practices for developing satellites such as the Intelsat IX 
series and iPSTAR satellites. 

To identify problems that DOD must overcome to improve the quality of 
weapon systems, we reviewed processes and tools DOD can use to 
influence weapon system quality. These include setting requirements, 
participating in key decisions during weapon system development and 
production, using contracts to incentivize good quality, and overseeing 
weapon system quality and prime contractor performance. We examined 
these processes and tools for the 11 weapons programs we reviewed and 
discussed the use of these processes and tools with acquisition and 
quality officials from the Office of the Secretary of Defense, military 
services, prime contractors, Defense Contract Management Agency, and 
Supervisor of Shipbuilding. We also relied on previous GAO best 
practices and weapon system reports to identify DOD actions that 
contributed to poor quality outcomes. A comprehensive list of reports 
we considered throughout our review can be found in the related 
products section at the end of this report. 

We met with officials at two commercial companies that purchase 
products manufactured by two of the leading commercial manufacturers we 
included in this review. These companies included: 

* American Airlines, the largest scheduled passenger airline in the 
world, which has purchased aircraft from Boeing Commercial Airplanes. 
We met with quality officials at a major maintenance facility located 
in Tulsa, Oklahoma. 

* Intelsat, a leading provider of fixed satellite services for 
telecommunications, Internet, and media broadcast companies, which 
purchases satellites from all major satellite manufacturers in the 
United States and Europe. We met with officials in space systems 
acquisition and planning at the company's headquarters located in 
Washington, D.C. 

Our discussions focused on (1) the companies' roles in establishing 
requirements; (2) the types of contracts they award to manufacturers 
and the specificity included in the contracts in terms of quality, 
reliability, and penalties; and (3) the amount of oversight they 
exercise over their suppliers' development and manufacturing 
activities. 

To identify recent DOD initiatives that could improve weapon system 
quality, we reviewed DOD's formal response to Sections 804 and 853 of 
the John Warner National Defense Authorization Act for Fiscal Year 
2007. This act requires DOD to report to the congressional defense 
committees on acquisition reform and program management initiatives. We 
also met with senior defense leaders to discuss the implementation 
status of the acquisition reform initiatives identified in DOD's 
February 2007 and September 2007 reports to the committees and relied 
on a previous GAO report for the implementation status of planned 
program management improvements.[Footnote 15] 

We conducted this performance audit from September 2006 to December 
2007 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

[End of section] 

Appendix II: Quality Problems for 11 DOD Weapon Systems: 

This appendix summarizes the quality problems experienced by the 11 DOD 
weapon systems we reviewed. The problems are categorized as systems 
engineering, manufacturing, and/or supplier quality problems. Most of 
the programs had problems in more than one of these categories. These 
summaries do not address all quality problems experienced on the 
programs; rather they emphasize major problems we discussed with 
officials from the military services, prime contractors, and the 
Defense Contract Management Agency. When possible, we include the 
direct impact the quality problems had on the program, the corrective 
actions the prime contractor or DOD took to address the problems, and 
the change in cost estimates and quantities from the start of program 
development to the present. 

The cost estimates were taken from DOD Selected Acquisition Reports or 
were program office estimates and include DOD's research, development, 
test and evaluation (RDT&E) and procurement expenditures on a 
particular program. We did not break out the portion of these funds 
that were paid to prime contractors versus the amount paid to 
suppliers. In addition, the change in cost estimates can be the result 
of a number of factors, including the amount paid to fix quality 
problems, a decision to procure more weapons, and increased labor rates 
or material prices. 

Figure 2: Advanced SEAL Delivery System: 

[See PDF for image] 

Photograph of the Advanced SEAL Delivery System, with the following 
accompanying data: 

DOD sponsor: Special Operations Command; 
Prime contractor: Northrop Grumman; 
Program start: 1994. 

RDT&E cost: 
Development estimate (in millions of dollars): $157; 
Current estimate[A] (in millions of dollars): $584; 
Change: +272%. 

Procurement cost: 
Development estimate (in millions of dollars): $139; 
Current estimate[A] (in millions of dollars): $173; 
Change: +24%. 

Total quantity: 
Development estimate (in millions of dollars): 3; 
Current estimate[A] (in millions of dollars): 1; 
Change: -2. 

Source: GAO analysis; US Navy (photo). 

[A] Estimated costs through fiscal year 2008. 

The Advanced SEAL Delivery System has experienced a number of problems 
that have degraded the boatï¿½s reliability and performance. Since 
accepting the boat in 2003, the Navy has issued contracts to Northrop 
Grumman valued at approximately $87 million, and much of this was to 
address design and reliability issues. However, continuing reliability 
and performance problems led to a decision in 2006 to cancel purchases 
of additional boats. The Navy subsequently issued another contract to 
Northrop Grumman for an estimated cost of $18 million to perform 
critical systems reviews and failure reviews, among other things, to 
improve the reliability of the first boat. DOD has also directed the 
Navy and the Special Operations Command to assess alternate material 
solutions to fulfill remaining operational requirements. In July 2007, 
the system was reinstated for System Fielding and Deployment Release. 
Examples of quality problems related to systems engineering and 
supplier quality are highlighted below. 

Systems engineering: 
Quality Problems: Ineffective program management by the contractor, 
including systems engineering deficiencies, was a key contributor to 
the systemï¿½s quality problems. Navy and Special Operations Command 
reviews found that the contractor had considerable difficulty 
interpreting the underwater shock portion of the performance 
requirements, in part due to the contractorï¿½s lack of experience in 
submarine design. Further, the contractor used substandard design 
methodologies, resulting in an unacceptable system design. The Advanced 
SEAL Delivery Systemï¿½s tail is an example of the systemï¿½s design 
problems. The systemï¿½s aluminum tail was redesigned due to fatigue 
stresses that were revealed during mated operations with the host 
submarine. The aluminum tail was not structurally adequate to meet the 
30-year tail life requirement. In 2005, the Navy awarded Northrop 
Grumman an $8 million contract to redesign the tail and upgrade to 
titanium. The replacement tail still has not resolved all the tail 
assembly design deficiencies, and the Navy has imposed operating 
restrictions that limit the speed of the host submarine while 
transporting the boat. 

Supplier quality: 
Quality Problems: In 2004, during testing of tail repairs, the 
propeller stator, which is part of the tail section, broke off, causing 
damage to the propeller. The resulting investigation attributed this 
failure to improper manufacturing of the propeller stator by a 
supplier, as it was not done in accordance with the stated design. 

[End of figure] 

Figure 3: Advanced Threat Infrared Countermeasure/Common Missile 
Warning System: 

[See PDF for image] 

Photograph of the Advanced Threat Infrared Countermeasure/Common 
Missile Warning System, with the following accompanying data: 

DOD sponsor: Army; 
Prime contractor: BAE Systems; 
Program start: 1995. 

RDT&E cost: 
Development estimate (in millions of dollars): $637; 
Current estimate[A] (in millions of dollars): $798; 
Change: +25%. 

Procurement cost: 
Development estimate (in millions of dollars): $2,605; 
Current estimate[A] (in millions of dollars): $4,515; 
Change: +73%. 

Total quantity: 
Development estimate (in millions of dollars): 3,094; 
Current estimate[A] (in millions of dollars): 3,589; 
Change: +495. 

Source: GAO analysis; BAE Systems (photo). 

The Advanced Threat Infrared Countermeasure/Common Missile Warning 
System is comprised of two systems initially managed by different 
military services. The Air Force and Navy initially managed the missile 
warning system designed to detect incoming missiles; and the Army 
managed the infrared countermeasure portion of the system, which 
employs laser energy to decoy or jam seekers on incoming missiles. In 
1995, DOD combined the two systems into a joint Army, Navy, and Air 
Force program. According to program officials, the services rushed to 
put a developer on contract, a move that resulted in significant 
requirements growth and presented major difficulties in designing the 
Common Missile Warning System sensor for use on both rotary wing (i.e., 
helicopter) and fixed-wing aircraft. Subsequently, the Navy and Air 
Force dropped out of the program in 2000 and 2001, respectively. 

Systems engineering: 
Quality problems: Reliability problems related to the Advanced Threat 
Infrared Countermeasure jam head and pointing accuracy forced the Army 
to halt laser testing in fiscal year 2005. As a result of these 
problems, fielding of the subsystem has been delayed for 5 years until 
fiscal year 2010 to develop a more reliable jam head. The program 
office estimated the cost of developing the new jam head at $117.3 
million. Reliability problems were caused, at least in part, by an 
early lack of focus on reliability. Specifically, according to a prime 
contractor official, neither the prime contractor nor the government 
had sufficient funding for a reliability testing program. 

Manufacturing: 
Quality problems: More than 4 years after the systemï¿½s critical design 
review, Common Missile Warning System sensor units were built in 
prototype shops, with engineers only then trying to identify critical 
manufacturing processes. [Footnote 18] Sensor manufacturing was slowed 
by significant rework, and at one point was halted while the contractor 
addressed configuration control problems. 

[End of figure] 

Figure 4: Expeditionary Fighting Vehicle: 

[See PDF for image] 

Photograph of the Expeditionary Fighting Vehicle, with the following 
accompanying data: 

DOD sponsor: Marine Corps; 
Prime contractor: General Dynamics; 
Program start: 2000. 

RDT&E cost: 
Development estimate (in millions of dollars): $1,569; 
Current estimate[A] (in millions of dollars): $3,565; 
Change: +127%. 

Procurement cost: 
Development estimate (in millions of dollars): $7,037; 
Current estimate[A] (in millions of dollars): $9,847; 
Change: +40%. 

Total quantity: 
Development estimate (in millions of dollars): 1,025; 
Current estimate[A] (in millions of dollars): 593; 
Change: -432. 

Source: GAO analysis; EFV Program Office (photo). 

The Expeditionary Fighting Vehicle has experienced significant 
reliability problems in development, despite reliability being one of 
the vehicleï¿½s seven key performance parameters. The prime contractor 
must meet a 43.5-hours mean time between operational mission failures 
requirement by fielding. The vehicle achieved only 7.7 hours between 
mission failures in pre-production testing, short of the 17 hours 
needed to be on an acceptable path for reliability growth. According to 
prime contractor officials, although reliability was a key performance 
parameter in development, DOD decided to focus its resources during 
this phase on meeting requirements related to water speed, 
survivability, and lethality. While this emphasis did not relieve the 
prime contractor of the reliability requirement, which was to be met at 
full-rate production, activities aimed at meeting the reliability 
requirement were to take place through a reliability growth program in 
which problems identified during testing would be fixed as they 
occurred. However, as a result of reliability problems, DOD extended 
the System Development and Demonstration phase, which program officials 
anticipate will last an additional 4 years at an estimated cost of $750 
million. The primary focus of the extension is to redesign the system 
for improved reliability. The extended development phase will focus on 
reliability improvements for several subsystems including the turret, 
hydraulics, drive train, software, and electrical/electronics. 

Systems engineering: 
Quality problems: According to program officials, design problemsï¿½ 
manifested as part and subsystem interferences at integration and 
assembly pointsï¿½were the primary cause of nonconformances noted during 
vehicle assembly. These interferences resulted from design and 
engineering changes that were not always passed to suppliers; this 
resulted in supplier parts not fitting into assemblies because they 
were produced using earlier designs. Interferences caused assembly 
schedule delays and, more generally, Defense Contract Management Agency 
officials said the high number of nonconformances experienced during 
assembly made every development vehicle late for testing. Prime 
contractor officials identified specific root causes for interference 
problems as: 
* tight engineering model release schedules; 
* design engineers lacked experience and did not comply with 
engineering standards; 
* computer model checks were inconsistently performed; and; 
* space claim checks were not performed/completed between subassembly 
teams, resulting in different components inadvertently claiming use of 
the same space. 

[End of figure] 

Figure 5: F-22A: 

[See PDF for image] 

Photograph of the F-22A, with the following accompanying data: 

DOD sponsor: Air Force; 
Prime contractor: Lockheed Martin; 
Program start: 1991. 

RDT&E cost: 
Development estimate (in millions of dollars): $23,820; 
Current estimate[A] (in millions of dollars): $36,723; 
Change: +54%. 

Procurement cost: 
Development estimate (in millions of dollars): $62,586; 
Current estimate[A] (in millions of dollars): $35,845; 
Change: +43%. 

Total quantity: 
Development estimate (in millions of dollars): 648; 
Current estimate[A] (in millions of dollars): 184; 
Change: -464. 

Source: GAO analysis; F-22A System Program Office (photo). 

In 2002, we reported that the F-22A program initially had taken steps 
to use commercial best practices to design and produce the aircraft. 
[Footnote 19] For example, the program planned to design in reliability 
and get critical manufacturing processes in statistical control by the 
full-rate production decision. In 2000, citing budgetary constraints 
and specific hardware quality problems that demanded attention, the Air 
Force decided to trade off producibility, reliability, and 
maintainability activities for performance in the systemï¿½s design and 
abandoned its efforts to get manufacturing processes in control. Less 
than 50 percent of its critical manufacturing processes were in control 
when the program entered production. Currently, the program is using 
post-assembly inspections to identify and fix defects rather than 
statistical process control techniques to prevent them. 

The program has not yet reached its reliability requirement of 3-hours 
mean time between maintenance actions. The Air Force estimated, when 
production began in 2001, Lockheed Martin should have been able to 
demonstrate that the F-22A could achieve almost 2 flying hours between 
maintenance actions. However, it could only demonstrate 0.66 hours, or 
about 40 minutes between maintenance actions. As of October 2007, the 
mean time between maintenance is 1.61 hours, or about 97 minutes. For 
fiscal year 2008, the program has over $400 million budgeted to improve 
the reliability and maintainability of the aircraft. Following are some 
of the more significant quality problems experienced on the system. 

Systems engineering: 
Quality problems: The program was structured to provide the aircraftï¿½s 
full capability with the first production unit off the line. This was 
an extreme design challenge and required the product design to include 
many new and unproven technologies, designs, and manufacturing 
processes. For example, the design of the transparency was a first for 
canopy systems. The program also included new low observable (stealth) 
materials, integrated avionics, and propulsion technology that were not 
mature at the start of the acquisition program. The program declared 
the design stable and ready to begin initial manufacturing, even though 
only 26 percent of the eventual design drawings were completed at the 
critical design review. The F-22A program did not achieve design 
stability, where 90 percent of drawings were complete, until almost 3 
years after the critical design review. By this time, the first two 
development aircraft had been delivered. 

Manufacturing: 
Quality problems: aircraftï¿½s cockpit. According to program officials, 
the program includes a first of a kind canopy, with an external 
metallic stealth layer, which makes it difficult to manufacture. 
Program officials acknowledge the program has addressed and corrected a 
number of manufacturing issues with the canopy including cracks 
emanating from mounting holes in the transparency in 2000, and they 
maintain they continue to incorporate improvements to the coating 
system in order to meet durability requirements. Another canopy 
problem, unrelated to the transparency, involved a defective activator 
that prevented a pilot from exiting the aircraft for 5 hours, resulting 
in a cost of over $100,000 in order to release the pilot and retrofit 
other existing aircraft. 

Supplier quality: 
Quality problems: The Air Force identified a potential major structural 
problem with heat treatment of titanium forgings near the aircraftï¿½s 
engine. According to program officials, the problem with the titanium 
was a material defect from a subcontractor. The program office, in 
conjunction with Air Force Research Laboratories and the contractor, 
performed a thorough review of the entire manufacturing process for the 
frames and determined that the root cause was that the frames had not 
spent enough time at the proper temperature during the heat treating 
process. An extensive structural test program was conducted to 
determine the service life impact of aircraft produced with incorrectly 
heat treated frames. The results of the testing showed that the frames 
met full durability life. The original heat treat vendor is no longer 
producing parts for the F-22A program, and the program office in 
conjunction with Air Force Research Laboratories and Lockheed Martin 
have implemented rigorous process controls at the new vendor to ensure 
that all frames for future production are properly heat treated. 

[End of figure] 

Figure 6: Global Hawk Unmanned Aircraft System: 

[See PDF for image] 

Photograph of the Global Hawk Unmanned Aircraft System, with the 
following accompanying data: 

DOD sponsor: Air Force; 
Prime contractor: Northrop Grumman; 
Program start: 2001. 

RDT&E cost: 
Development estimate (in millions of dollars): $989; 
Current estimate[A] (in millions of dollars): $3,682; 
Change: +272%. 

Procurement cost: 
Development estimate (in millions of dollars): $4,102; 
Current estimate[A] (in millions of dollars): $5,774; 
Change: +41%. 

Total quantity: 
Development estimate (in millions of dollars): 63; 
Current estimate[A] (in millions of dollars): 54; 
Change: -9. 

Source: GAO analysis; Northrop Grumman Corporation (photo). 

The Global Hawk program began as an Advanced Concept Technology 
Demonstration effort in 1994. Following a successful technology 
demonstration in 2001, DOD transitioned the program directly to a 
simultaneous development and production effort. In 2002, the program 
was restructured to include a more advanced Global Hawk model. 
Collectively, these decisions created several challenges for the 
program, including quality problems, as described below. 

Systems engineering: 
Quality problems: The advanced Global Hawk modelï¿½s design had been 
expected to be very similar to the previous Global Hawk model. However, 
as the design for the advanced model matured and production was about 
to start, the differences between the initial model and the advanced 
model were more extensive, complex, and costly than anticipated. Within 
a year, there were more than 2,000 engineering drawing changes to the 
baseline of 1,400 drawings. More than half of the changes were 
considered major. Design deficiencies, engineering changes, and work 
delays contributed to a $209 million overrun in the development 
contract. 

Supplier quality: 
Quality problems: The supplier producing the Integrated Sensor Suite, 
which is the primary air vehicle payload, encountered production 
problems as the program transitioned from an Advanced Concept 
Technology Demonstration effort to a development and production effort. 
During the demonstration effort, the supplier built the sensor suite in 
a laboratory using a labor-intensive process. However, this process was 
not efficient for a longer production effort, and the supplier 
struggled to meet the increased production rates for later deliveries. 
Defense Contract Management Agency officials stated that the 
specialized test equipment development delays resulted in approximately 
a four-month schedule slip in production of the integrated sensor 
suite. To address this issue, the Air Force had to invest $30 million 
in specialized testing equipment to help the contractor implement 
efficient production processes. However, this supplier continued to 
experience quality problems related to workmanship and was delivering 
sensors late. Starting in October 2006, the prime contractor placed an 
assistance team at the supplierï¿½s facility and addressed many of these 
problems. A 2006 review of lessons learned from the Global Hawk program 
by prime contractor and Air Force program personnel noted that the 
program had done little production planning as it began the development 
and production effort. In addition, the review found that the Air Force 
should have included funding in the first production estimates for 
specialized test equipment, which was needed to implement efficient 
production processes. 

[End of figure] 

Figure 7: Joint Air-to-Surface Standoff Missile: 

[See PDF for image] 

Photograph of the Joint Air-to-Surface Standoff Missile, with the 
following accompanying data: 

DOD sponsor: Air Force; 
Prime contractor: Lockheed Martin; 
Program start: 1998. 

RDT&E cost: 
Development estimate (in millions of dollars): $970; 
Current estimate[A] (in millions of dollars): $1,407; 
Change: +45%. 

Procurement cost: 
Development estimate (in millions of dollars): $1,208; 
Current estimate[A] (in millions of dollars): $3,998; 
Change: +231%. 

Total quantity: 
Development estimate (in millions of dollars): 2,469; 
Current estimate[A] (in millions of dollars): 5,006; 
Change: +2,537. 

Source: GAO analysis; Joint Air-to-Surface Standoff Missile (JASSM) 
Program Office (JASSM-Extended Range IT-2) (photo). 

The Joint Air-to-Surface Standoff Missile program entered production in 
December 2001; as of June 2007, more than 600 missiles had been 
delivered to DOD. However, following two flight test failures in the 
spring of 2005, quantities for one missile production lot were reduced 
and, in response to congressional concerns, DOD is focusing on 
increasing missile reliability. As of September 2007, the program 
office has spent $39.4 million on reliability improvements. Due to 
increased costs and schedule delays associated with reliability 
problems and development of an extended range version of the missile, 
the program reported a Nunn-McCurdy breach in 2006. [Footnote 20] 
According to a program official, in May 2007, DOD deferred Nunn-McCurdy 
certification to continue the program, pending improvements to system 
reliability. Overall, DODï¿½s Director, Operational Test and Evaluation 
office recorded 25 flight test failures, 12 of which were attributed to 
quality and hardware design issues. Problems related to manufacturing 
and supplier quality were responsible for some test failures. 

Systems engineering: 
Quality problems: Wing retention devicesï¿½piston-like parts that hold 
the wings in the stowed position inside the missileï¿½failed to deploy 
during two flight tests, causing test failures. The parts were designed 
to snap in an exact location when an electronic charge is fired, 
allowing wing deployment. Following the test failures the design was 
shown to be adequate but the manufacturing process could not guarantee 
the devices would snap in the precise spot every time. Manufacturing 
tolerances were subsequently changed to ensure an exact break, and this 
remedy was retrofitted to some missile production lots. 

Supplier quality: 
Quality problems: Malfunction of a mechanical fuse provided by a 
supplier was responsible for another flight test failure. Prime 
contractor officials said the failure exhibited a repeat of a fuse 
problem experienced early in manufacturing, which resulted in a 2004 
flight test failure. That earlier failure was attributed to foreign 
object damage and a corrective action was applied to missiles in 
production at the time. Additionally, the prime contractor has agreed 
to fund replacement of affected fuses from two production lots. Another 
supplier problem resulted in an April 2005 test failure involving a 
part needed to move the tail and wings into flight position. This 
problem, which originally surfaced prior to a planned mission the 
previous year, resulted from a supplier employee not following work 
instructions. The missile fleet was inspected at that time; but, 
following a review of the flight failure, the program office and prime 
contractor determined the criteria used for the inspection was 
inadequate. As a result, the prime contractor instituted a more robust 
inspection process at the supplier for future production. 

[End of figure] 

Figure 8: LPD 17 Amphibious Transport Dock: 

[See PDF for image] 

Photograph of the LPD 17 Amphibious Transport Dock, with the following 
accompanying data: 

DOD sponsor: Navy; 
Prime contractor: Northrop Grumman; 
Program start: 1993. 

RDT&E cost: 
Development estimate (in millions of dollars): $97; 
Current estimate[A] (in millions of dollars): $137; 
Change: +41%. 

Procurement cost: 
Development estimate (in millions of dollars): $11,025; 
Current estimate[A] (in millions of dollars): $13,557; 
Change: +23%. 

Total quantity: 
Development estimate (in millions of dollars): 12; 
Current estimate[A] (in millions of dollars): 9; 
Change: -3. 

Source: GAO analysis; U.S. Navy (PMS 317, PEO Ships). 

According to the program office the LPD 17 Amphibious Transport Dock, 
which was delivered to the Navy in July 2005, has experienced numerous 
quality problems of varying degrees that significantly impacted the 
shipï¿½s mission. These problems contributed to a delay of 3 years in the 
delivery of the ship and a cost increase of $846 million. According to 
Navy program officials, some of the problems are typical of those of a 
first ship of class production. Many of the problems can be attributed 
to systems engineering, manufacturing, and supplier issues as noted 
below. In June 2007, the Secretary of the Navy sent a letter to the 
Chairman of the Board of Northrop Grumman expressing his concerns for 
the contractorï¿½s ability to construct and deliver ships that conform to 
the quality standards maintained by the Navy and that adhere to the 
cost and schedule commitments agreed upon. Northrop Grummanï¿½s Chairman 
acknowledged that the company was aware of the problems and is working 
on improving its processes. 

Systems engineering: 
Quality problems: Many of the system engineering problems on the LPD 17 
can be attributed to the software-based design tool used by the 
contractors. The contractor selected a 3-D model to fulfill Navy 
requirements, the Integraph software package, which had been used in 
large construction efforts but not fully adapted for shipbuilding. It 
was intended for workers to design systems and extract drawings from 
this 3-D model. The modification of this design tool, at the same time 
the ship was under design, caused delays in the release of production 
drawings. According to the program office, Northrop Grumman experienced 
some difficulty in acquiring and training qualified personnel to use 
the system. Consequently, the program experienced higher than expected 
engineering hours due to a large number of design drawings that 
required rework. Design rework also affected the sequencing of work 
being done on the ship as well as the accuracy of that work. Northrop 
Grumman Ship Systems officials stated that completing design work after 
beginning ship construction affects both the work schedule and the 
quality of work. The LPD 17 also encountered a problem with the 
isolators on titanium piping. The isolators are used to separate 
different types of metals to keep them from corroding. The problem was 
discovered in 2006, about a year after the launch of the first ship. 
According to DOD program officials, the titanium piping is used 
throughout the ship because it is lighter than the traditional copper-
nickel piping and has a longer service life. However, it has not been 
used much in naval surface ships or by the American shipbuilding 
industry, and therefore required new manufacturing and installation 
processes. According to the program office, these processes were being 
developed as Northrop Grumman Ship Systems was building the ship. In 
addition, designs for the piping hangers, which hold the piping in 
place, as well as tests of the isolators were subsequently delayed. 
When the titanium piping on the ship was changed, the hanger design had 
to be modified as well. The final hanger design was not completed until 
about 90 percent of the titanium piping was already on the ship, which 
resulted in additional rework and schedule delays. The LPD 17 Class has 
had problems associated with its steering system as well. Hydraulic 
fluid contamination occurred during system flushing. System flushing is 
completed in order to clean out a system and involves running fluid 
throughout the piping. Additionally, there were problems in keeping air 
out of the system. After investigation, several steps were taken to 
mitigate these issues including installing additional filters, 
modifying the flushing procedures, and modifying the system design. 

Manufacturing: 
Quality problems: The ship encountered problems with faulty welds on P-
1 piping systems, a designation used in high-temperature, high-
pressure, and other critical systems. This class of piping is used 
primarily in hydraulic applications in engineering and machinery 
spaces. P-1 piping systems require more extensive weld documentation 
than other pipes as they are part of critical systems and could cause 
significant damage to the ship and crew if they failed. Welds of this 
nature must be documented to ensure they were completed by qualified 
personnel and inspected for structural integrity. Further investigation 
revealed that weld inspection documentation was incomplete. As a 
result, increased rework levels were necessary to correct deficiencies 
and to re-inspect all the welds. Failure to complete this work would 
have increased the risk of weld failure and potentially presented a 
hazard to the ship and crew. According to the program office, a 
contributing factor was turnover in production personnel and their lack 
of knowledge on how to complete the proper documentation. The program 
is also experiencing problems with non-skid applications, a type of 
coating used on the ship. The non-skid application is different from 
traditional surface coatings in that it creates a rough surface when it 
has dried. This is particularly important on a ship because it provides 
increased traction when wet as opposed to traditional surface coatings. 
One problem the program encountered with this particular type of 
coating was in preparation. When applying non-skid application, it is 
important to have a clean surface free of dirt and debris. 
Additionally, high humidity levels found along the Gulf Coast, where 
the ship was built interfere with the bonding process and require 
dehumidification. These conditions have been difficult to consistently 
achieve in a construction environment. As a result, the non-skid would 
not adhere properly and began to peel away. As of November 2007, no 
change in process has occurred. 

Supplier quality: 
Quality problems: The ship program also experienced numerous supplier 
quality problems. For example, an inspection completed in March of 2007 
identified the reverse osmosis units, which provide drinkable water to 
the crew, as one of the most troubled systems onboard the ship. At the 
time of the inspection, one of the three units was out of commission, 
one was unable to produce to capacity, and one was operational but 
unreliable. In this condition, the ship would not be able to support 
embarked troops for extended periods at sea and, as a result, the 
mission of the ship would be limited. During the design phase, it was 
determined that currently available reverse osmosis units could not 
meet the shipï¿½s output requirements. Therefore, a new design was 
developed specifically for the LPD 17 Class. Problems with the reverse 
osmosis units were caused by premature failures of some mechanical and 
electrical components. According to the Navy program office, the 
supplier of the shipï¿½s reverse osmosis units did not use parts rugged 
enough for the shipï¿½s needs. This supplier is providing reverse osmosis 
units for all ships in the LPD 17 Class. Consequently, the LPD 18 and 
LPD 19 will need to have their units reworked as well. According to the 
program office, the vendor is now using more rugged parts and will 
provide properly working units for the LPD 20, the fourth ship to be 
delivered in this class, and all subsequent ships. 

[End of figure] 

Figure 9: MH-60S Fleet Combat Support Helicopter: 

[See PDF for image] 

Photograph of the MH-60S Fleet Combat Support Helicopter, with the 
following accompanying data: 

DOD sponsor: Navy; 
Prime contractor: Sikorsky Aircraft; 
Program start: 1998. 

RDT&E cost: 
Development estimate (in millions of dollars): $85; 
Current estimate[A] (in millions of dollars): $654; 
Change: +666%. 

Procurement cost: 
Development estimate (in millions of dollars): $3,246; 
Current estimate[A] (in millions of dollars): $7,212; 
Change: +122%. 

Total quantity: 
Development estimate (in millions of dollars): 166; 
Current estimate[A] (in millions of dollars): 267; 
Change: +101. 

Source: GAO analysis; H-60 program office. 

The MH-60S is currently in production. In recent years, Sikorsky began 
to outsource some of its production work, and subsequently experienced 
problems related to the outsourcing of more complex assemblies to 
suppliers. An example of a problem related to the outsourcing of MH-60S 
cabin production is below. 

Manufacturing: 
Quality problems: The supplier building the cabin assemblies had to 
scrap two major assemblies for the helicopter cabins, and the program 
experienced approximately a 6-month delay in the delivery of the 
cabins. According to program office officials, these problems occurred 
because of inadequate work instructions for producing the cabins. 
Sikorsky sent the subcontractor its drawing packages and work 
instructions for the cabins, but due to the poor quality of this 
information, the subcontractor was not able to build the cabins 
properly. Initially, Sikorsky had planned to have a parallel production 
line with the subcontractor. However, Defense Contract Management 
Agency officials stated that Sikorsky discontinued the parallel 
production line after the supplier had built only a few cabins. 
Sikorsky officials said that the company did not provide enough overlap 
in the production line to ensure a smooth transition. 

[End of figure] 

Figure 10: Patriot Advanced Capability-3: 

[See PDF for image] 

Photograph of the Patriot Advanced Capability-3, with the following 
accompanying data: 

DOD sponsor: Army; 
Prime contractor: Lockheed Martin; 
Program start: 1994. 

RDT&E cost: 
Development estimate (in millions of dollars): $2,593; 
Current estimate[A] (in millions of dollars): $3,932; 
Change: +52%. 

Procurement cost: 
Development estimate (in millions of dollars): $2,357; 
Current estimate[A] (in millions of dollars): $5,665; 
Change: +140%. 

Total quantity: 
Development estimate (in millions of dollars): 1,200; 
Current estimate[A] (in millions of dollars): 969; 
Change: -231. 

Source: GAO analysis; PAC-3 Product Office, Lower Tier Project Office. 

When the Patriot Advanced Capability-3 program was initiated, the 
development contract did not require the prime contractor to have 
commercial quality standard certification. According to program 
officials, the contractï¿½s reference to quality standards was for 
guidance only. Since the program went into production in 1999, it has 
experienced a number of problems with the seeker (target finding) 
portion of the missile. Below are some examples of these problems. 

Systems engineering: 
Quality problems: During flight testing in November 2005, two missile 
seekers reset shortly after launch causing the missiles to fail to 
intercept their targets. These failures resulted from a design issue 
involving requirements passed from the prime contractor to the seeker 
manufacturer that did not contain a sufficient design margin. The Army 
paid for a failure analysis review as well as a short-term software fix 
to mitigate the effects of a potential future seeker reset. Related 
hardware improvements will cost up to $2.1 million. 

Manufacturing: 
Quality problems: We reported that in low-rate production only 25 
percent of the missileï¿½s seekers were being manufactured correctly the 
first time, with the rest being reworked on average four times before 
being acceptable. [Footnote 21] Additionally, prior to entering 
production only 40 percent of the missileï¿½s manufacturing processes 
were in control. In an effort to boost seeker first-pass yield rates, 
the Army agreed to pay an estimated $24 million for equipment to test 
subcomponents before integration into the seeker. This equipment has 
helped identify problems earlier in the manufacturing process and 
improve first-pass yield rates to 90 percent. 

Supplier quality: 
Quality problems: In the spring of 2006 a supplier producing a seeker 
component acted without authority in acceptance of non-conforming 
hardware. One program official attributed this in part to the supplier 
having formerly operated in a development environment in which 
procedures for material acceptance differ from those followed during 
production. The supplier involved has also experienced other problems, 
some involving manufacturing and poor workmanship issues. As a result 
of problems experienced by this supplier, its production facility was 
temporarily shut down, causing a 6-month schedule slip and delaying 
delivery of about 100 missiles. 

[End of figure] 

Figure 11: V-22 Joint Services Advanced Vertical Lift Aircraft: 

[See PDF for image] 

Photograph of the V-22 Joint Services Advanced Vertical Lift Aircraft, 
with the following accompanying data: 

DOD sponsor: Navy; 
Prime contractor: Bell Helicopter Textron and Boeing Integrated Defense 
Systems; 
Program start: 1986. 

RDT&E cost: 
Development estimate (in millions of dollars): $4,033; 
Current estimate[A] (in millions of dollars): $12,474; 
Change: +209%. 

Procurement cost: 
Development estimate (in millions of dollars): $33,823; 
Current estimate[A] (in millions of dollars): $42,088; 
Change: +24%. 

Total quantity: 
Development estimate (in millions of dollars): 913; 
Current estimate[A] (in millions of dollars): 458; 
Change: -455. 

Source: GAO analysis; V-22 Joint Program Office. 

The V-22 is currently in production. The program has experienced four 
crashes throughout its development and production, three of which 
resulted in casualties. Quality issues related to design were a 
contributor in one of these fatal crashes, as described below. 

Systems engineering: 
Quality problems: In 2000, a low-rate initial production aircraft 
crashed during a training mission. This crash killed the four Marines 
aboard the flight, and, as a result, the Navy and the Marine Corps 
suspended program flight operations from December 2000 to May 2002. An 
investigation into the crash attributed the accident to a combination 
of a hydraulic line failure and a flight control software problem. 
While neither the hydraulic line failure nor the software problem alone 
would have caused the accident, the combination of the two problems 
resulted in a loss of flight control. The hydraulic line failure was 
due to chafing of the line on a wire harness in the nacelle, which is 
the portion of the aircraft that tilts or rotates in order to convert 
from helicopter to aircraft operations. The accident investigation also 
noted that hydraulic line chafing was a repeated problem among 
aircraft, citing various Airframe Bulletins, Hazardous Material Reports 
and Quality Deficiency Reports from June 1999 through February 2001 
that described a chafing problem of wire bundles and hydraulic lines in 
the aircraft nacelles. The Navy program office established Integrated 
Product Teams to identify the hydraulic system challenges facing the V-
22. They concluded that chafing due to insufficient clearances among 
components, installation flaws, and variances among aircraft were all 
problems affecting the hydraulic lines in the nacelles. The program 
subsequently completed a redesign to address system separation, 
including the hydraulic system design, hydraulic tubing, wire 
harnesses, and fuel system. These design changes were incorporated into 
what are known as ï¿½Block Aï¿½ aircraft. The program office estimated that 
the recurring costs of the engineering change proposals to the aircraft 
design for the Block A aircraft was approximately $165 million. 

[End of figure] 

Figure 12: Wideband Global SATCOM: 

[See PDF for image] 

Illustration of the Wideband Global SATCOM, with the following 
accompanying data: 

DOD sponsor: Air Force and Army; 
Prime contractor: Boeing Integrated Defense Systems; 
Program start: 2000. 

RDT&E cost: 
Development estimate (in millions of dollars): $203; 
Current estimate[A] (in millions of dollars): $322; 
Change: +63%. 

Procurement cost: 
Development estimate (in millions of dollars): $929; 
Current estimate[A] (in millions of dollars): $1,699; 
Change: +83%. 

Total quantity: 
Development estimate (in millions of dollars): 3; 
Current estimate[A] (in millions of dollars): 5; 
Change: +2. 

Source: GAO analysis; WGS Program Office. 

The Wideband Global SATCOM acquisition is commercial in nature, with 
the satellite design based on an existing satellite manufactured by the 
prime contractor. As such, the program entered production in November 
2000 with mature technologies; however, supplier problems have delayed 
initial operational capability by 18 months. The first of the five 
satellites was launched in October 2007, 4 months later than expected. 

Manufacturing: 
Quality problems: Power dividers are located in the transmit phased 
array and in the receive phased array. The dividers, which split power 
between the separate elements that make up the arrays, failed hot and 
cold electrical performance tests resulting in a 6-month schedule slip 
for the program. An in-orbit power divider failure could result in the 
loss of one of its eight shapeable beams. This problem was caused by 
circuit traces that cracked in the manufacturing process. The problem 
was resolved by the prime contractor changing suppliers and improving 
processes for manufacturing the power dividers. 

Supplier quality: 
Quality problems: During replacement of a subcomponent on the first 
satellite, the prime contractor discovered that certain fasteners were 
installed incorrectly. As a result, 1,500 fasteners on each of the 
first three satellites had to be inspected or tested and 148 fasteners 
on the first satellite had to be reworked. The DOD program office 
reported that the resulting 15-month schedule slip would add rework and 
workforce costs (to be borne by the contractor) to the program and 
delay initial operating capability by 18 months. A prime contractor 
official estimated the impact to the program was at least $10 million. 
The problem resulted from a supplier not testing installed fasteners as 
required. 

[End of figure] 

[End of section] 

Appendix III: Comments from the Department of Defense: 

Office Of The Under Secretary Of Defense: 
Acquisition, Technology And Logistics: 
3000 Defense Pentagon: 
Washington, DC 20301-3000: 

January 18, 2008: 

Mr. Michael Sullivan: 
Director, Acquisition and Sourcing Management: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Mr. Sullivan: 

This is the Department of Defense (DOD) response to the GAO draft 
report GAO-08-294, "Best Practices: Increased Focus on Requirements and 
Oversight Needed to Improve DoD's Acquisition Environment and Weapon 
System Quality," dated December 21, 2007 (GAO Code 120642). Detailed 
comments on the report recommendations are enclosed. 

Sincerely, 

Signed by: 

Mark D. Schaeffer: 
Director: 
Systems and Software Engineering: 

Enclosure: As stated: 

Enclosure: 

GAO Draft Report Dated December 21, 2007: 
GAO-08-294 (GAO CODE 120642): 

"Best Practices: Increased Focus On Requirements And Oversight Needed 
To Improve Dod's Acquisition Environment And Weapon System Quality" 

Department Of Defense Comments To The Gao Recommendations: 

Recommendation 1: The GAO recommends that the Secretary of Defense 
include as a part of the concept decision review initiative, a 
requirement that systems engineering analysis be completed by the prime 
contractor prior to entering into a development contract. (p. 27/GAO 
Draft Report) 

DOD Response: Partially concur. The concept decision (CD) initiative 
reflects the requirement to consider such fundamental systems 
engineering issues as technology maturity and integration and 
manufacturing risk before the CD review. Currently the CD initiative is 
under consideration by the Under Secretary of Defense (Acquisition, 
Technology and Logistics) (USD (AT&L)). The recommendation to require 
that systems engineering analysis be completed by the prime contractor 
prior to entering into a development contract is vague. A prime 
contractor generally cannot perform a systems engineering analysis as 
part of the concept decision review because prime contractors are 
generally not selected until after Milestone B when a development 
contract is awarded for the Systems Development and Demonstration 
Phase. However, by its very nature, systems engineering is a continuous 
government-performed activity at the heart of any structured 
development process that proceeds from concept to production. Formal 
systems engineering planning is consistent with current DoD policy and 
is required to support program decisions prior to award of any 
development contract. 

Recommendation 2: The GAO recommends that the Secretary of Defense 
establish measures to gauge the success of the concept decision 
reviews, time-defined acquisition, and configuration steering board 
initiatives and properly support and expand these initiatives where 
appropriate. (p. 27/GAO Draft Report) 

DOD Response: Partially concur. Configuration steering boards are being 
implemented consistent with the USD (AT&L) policy of July 30, 2007, 
Configuration Steering Boards. Changes to DoD Instruction 5000.2, 
"Operation of the Defense Acquisition System," on concept decision 
reviews and time-defined acquisition are under consideration by the USD 
(AT&L). 

Recommendation 3: The GAO recommends that the Secretary of Defense 
direct the Defense Contract Management Agency and the Military Services 
to identify and collect data that provides metrics about the 
effectiveness of prime contractors' quality management system and 
processes by weapon system and business area over time. (p. 27/GAO 
Draft Report) 

DOD Response: Partially concur. The Defense Contract Management Agency 
is already in the process of identifying and eventually collecting data 
as a source of metrics on the effectiveness of prime contractors' 
quality management system and processes over time. The added expense of 
refining the collection process to capture data by weapon system and 
business area does not seem warranted at this time. Greater levels of 
detail may be considered once the data are used to score performance. 
(See the DoD response to Recommendation 4.) The Defense Contract 
Management Agency is doing this work in cooperation with the Military 
Services; therefore there is no need for them to engage in a similar 
effort. 

Recommendation 4: The GAO recommends that the Secretary of Defense 
direct the Defense Contract Management Agency and the Military Services 
to develop evaluation criteria that would allow DoD to score the 
performance of prime contractors' quality management systems based on 
actual past performance. (p. 27/GAO Draft Report) 

DOD Response: Partially concur. The plant-level data discussed in the 
DoD response to Recommendation 3 will be used to develop evaluation 
criteria that would allow DoD to score the performance of prime 
contractors' quality management systems based on actual performance in 
the past. Such scoring will be helpful in source selection and to 
reduce risk in contracts. These results will be made available to the 
Military Services so there is no need for them to conduct a parallel 
effort. 

[End of enclosure] 

[End of section] 

Appendix IV: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Michael Sullivan (202) 512-4841 or [email protected]. 

Acknowledgments: 

Key contributors to this report were Jim Fuquay, Assistant Director; 
Cheryl Andrew; Lily Chin; Julie Hadley; Lauren Heft; Laura Jezewski; 
Andrew Redd; Charlie Shivers, and Alyssa Weir. 

[End of section] 

Related GAO Products: 

Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-07- 
406SP. Washington, D.C.: March 30, 2007. 

Best Practices: Stronger Practices Needed to Improve DOD Technology 
Transition Processes. GAO-06-883. Washington, D.C.: September 14, 2006. 

Best Practices: Better Support of Weapon System Program Managers Needed 
to Improve Outcomes. GAO-06-110. Washington, D.C.: November 1, 2005. 

Defense Acquisitions: Major Weapon Systems Continue to Experience Cost 
and Schedule Problems under DOD's Revised Policy. GAO-06-368. 
Washington, D.C.: April 13, 2006. 

DOD Acquisition Outcomes: A Case for Change. GAO-06-257T. Washington, 
D.C.: November 15, 2005. 

Defense Acquisitions: Stronger Management Practices Are Needed to 
Improve DOD's Software-Intensive Weapon Acquisitions. GAO-04-393. 
Washington, D.C.: March 1, 2004. 

Best Practices: Setting Requirements Differently Could Reduce Weapon 
Systems' Total Ownership Costs. GAO-03-57. Washington, D.C.: February 
11, 2003. 

Defense Acquisitions: Factors Affecting Outcomes of Advanced Concept 
Technology Demonstration. GAO-03-52. Washington, D.C.: December 2, 
2002. 

Best Practices: Capturing Design and Manufacturing Knowledge Early 
Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 
2002. 

Defense Acquisitions: DOD Faces Challenges in Implementing Best 
Practices. GAO-02-469T. Washington, D.C.: February 27, 2002. 

Best Practices: Better Matching of Needs and Resources Will Lead to 
Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 
2001. 

Best Practices: A More Constructive Test Approach Is Key to Better 
Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 
2000. 

Defense Acquisition: Employing Best Practices Can Shape Better Weapon 
System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000. 

Best Practices: DOD Training Can Do More to Help Weapon System Programs 
Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August 
16, 1999. 

Best Practices: Better Management of Technology Development Can Improve 
Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 
1999. 

Defense Acquisitions: Best Commercial Practices Can Improve Program 
Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999. 

Defense Acquisitions: Improved Program Outcomes Are Possible. GAO/T- 
NSIAD-98-123. Washington, D.C.: March 17, 1998. 

Best Practices: Successful Application to Weapon Acquisition Requires 
Changes in DOD's Environment. GAO/NSIAD-98-56. Washington, D.C.: 
February 24, 1998. 

Best Practices: Commercial Quality Assurance Practices Offer 
Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 
1996. 

[End of section] 

Footnotes: 

[1] Boeing has two primary businesses: Boeing Commercial Airplanes and 
Boeing Integrated Defense systems. We held discussions with officials 
from both business areas and therefore refer to them separately 
throughout the report. 

[2] The ISO 9001 standard provides a framework for managing an 
organization's processes so that it consistently produces products that 
meet customer expectations. An ISO certification means that an 
independent external body has audited an organization's quality 
management system and verified that it conforms to the requirements 
specified in the standard. 

[3] For example, AS9100 is a set of quality standards for the aerospace 
industry; ISO/TS 16949 is a set of standards for the automotive 
industry. 

[4] GAO, Best Practices: Commercial Quality Assurance Practices Offer 
Improvements for DOD, GAO/NSIAD-96-162 (Washington, D.C.: Aug. 26, 
1996). 

[5] GAO, Best Practices: Better Matching of Needs and Resources Will 
Lead to Better Weapon System Outcomes, GAO-01-288 (Washington, D.C.: 
Mar. 8, 2001). 

[6] GAO, Best Practices: Capturing Design and Manufacturing Knowledge 
Early Improves Acquisition Outcomes, GAO-02-701 (Washington, D.C.: July 
15, 2002). 

[7] Six Sigma is a tool for measuring defects and improving quality. 
Over time, it has evolved into a business improvement methodology that 
focuses an organization on understanding customer requirements, 
aligning key business processes to achieve those requirements, 
utilizing rigorous data analysis to minimize variation in business 
processes, and rapid and sustainable improvement to business processes. 

[8] The Federal Acquisition Regulation (FAR), indicates that complex 
requirements, particularly those unique to the government, usually 
result in greater risk assumption by the government. This is especially 
true for complex research and development contracts when performance 
uncertainties or the likelihood of changes makes it difficult to 
estimate performance costs in advance. Cost-reimbursable contracts are 
suitable for use only when uncertainties involved in contract 
performance do not permit costs to be estimated with sufficient 
accuracy to use any type of fixed-price contract. FAR 16.104 and 16.301-
2. 

[9] GAO, Defense Acquisitions: Assessments of Selected Weapon Programs, 
GAO-07-406SP (Washington, D.C.: Mar. 30, 2007). 

[10] GAO, Best Practices: Setting Requirements Differently Could Reduce 
Weapon Systems' Total Ownership Costs, GAO-03-57 (Washington, D.C.: 
Feb. 11, 2003). 

[11] GAO-07-406SP. 

[12] Section 804 of the John Warner National Defense Authorization Act 
for Fiscal Year 2007, Pub. L. No. 109-364 (2006), requires DOD to 
submit to Congress on a biannual basis an update of its implementation 
plans to reform the acquisition system. Many of the initiatives 
highlighted in the report were initiated in response to other reform 
efforts, such as the Defense Acquisition Performance Assessment Project 
Report (DAPA, January 2006); Defense Science Board 2005 Summer Study on 
Transformation: "A Progress Assessment" (February 2006); The Center for 
Strategic and International Studies Report, "Beyond Goldwater Nichols: 
U.S. Government and Defense Reform for a New Strategic Era" (July 
2005); and The 2006 Quadrennial Defense Review Report (February 2006). 
Section 853 of the act requires the Secretary of Defense to develop a 
comprehensive strategy for enhancing the role of DOD program managers 
in developing and carrying out defense acquisition programs. 

[13] Acquisition category I programs are major defense acquisition 
programs that are estimated by the Under Secretary of Defense for 
Acquisition, Technology and Logistics to require eventual expenditure 
for research, development, test, and evaluation of more than $365 
million (FY 2000 constant dollars) or procurement of more than $2.190 
billion (FY 2000 constant dollars), or those designated by the 
Milestone Decision Authority to be acquisition category I programs. 10 
U.S.C. ï¿½ 2430. 

[14] Key performance parameters are defined as those attributes or 
characteristics of a system that are considered critical or essential 
to the development of an effective military capability and those 
attributes that make a significant contribution to the characteristics 
of the future joint force as defined in the Capstone Concept for Joint 
Operations. Key system attributes are attributes or characteristics 
that are considered crucial in support of achieving a balanced 
solution/approach deemed necessary by the program sponsor. Key 
performance parameters must be met before a weapon system can go into 
production. Key system attributes, on the other hand, can be traded in 
favor of other system attributes. 

[15] GAO, Defense Acquisitions: Department of Defense Actions on 
Program Manager Empowerment and Accountability, GAO-08-62R (Washington, 
D.C.: Nov. 9, 2007). 

[16] GAO-02-701. 

[17] 10 U.S.C. ï¿½ 2433 establishes the requirement for unit cost 
reports. If certain cost thresholds are exceeded (known as unit cost or 
Nunn-McCurdy breaches), DOD is required to report to Congress and, in 
certain circumstances, certify the program to Congress. 

[18] At this point, most design drawings should be released, prototype 
hardware developed, and units ready to build. 

[19] GAO-02-701. 

[20] 10 U.S.C. ï¿½ 2433 establishes the requirement for unit cost 
reports. If certain cost thresholds are exceeded (known as unit cost or 
Nunn-McCurdy breaches), DOD is required to report to Congress and, in 
certain circumstances, certify the program to Congress. 

[21] GAO-02-701. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "Subscribe to Updates." 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office: 
441 G Street NW, Room LM: 
Washington, D.C. 20548: 

To order by Phone: 
Voice: (202) 512-6000: 
TDD: (202) 512-2537: 
Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: [email protected]: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Gloria Jarmon, Managing Director, [email protected]: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, [email protected]: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: 		

*** End of document. ***