Best Practices: Better Support of Weapon System Program Managers 
Needed to Improve Outcomes (30-NOV-05, GAO-06-110).		 
                                                                 
The Department of Defense (DOD) relies on a relatively small	 
cadre of officials to develop and deliver weapon systems. In view
of the importance of DOD's investment in weapon systems, we have 
undertaken an extensive body of work that examines DOD's	 
acquisition issues from a perspective that draws lessons learned 
from the best commercial product development efforts to see if	 
they apply to weapon system acquisitions. In response to a	 
request from the Chairman and Ranking Minority Member of the	 
Subcommittee on Readiness and Management Support, Senate	 
Committee on Armed Services, this report assesses (1) how	 
successful commercial companies position their program managers, 
(2) how DOD positions its program managers, and (3) underlying	 
reasons for the differences. In compiling this report, GAO	 
conducted a survey of program managers. See GAO-06-112SP.	 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-06-110 					        
    ACCNO:   A42358						        
  TITLE:     Best Practices: Better Support of Weapon System Program  
Managers Needed to Improve Outcomes				 
     DATE:   11/30/2005 
  SUBJECT:   Best practices					 
	     Best practices reviews				 
	     Defense capabilities				 
	     Defense procurement				 
	     Lessons learned					 
	     Policy evaluation					 
	     Private sector					 
	     Private sector practices				 
	     Procurement practices				 
	     Program management 				 
	     Strategic planning 				 
	     Weapons research and development			 
	     Weapons systems					 
	     DOD Airborne Laser Program 			 
	     F-22 Raptor Aircraft				 
	     F/A-22 Aircraft					 
	     Joint Strike Fighter				 
	     Space-Based Infrared System			 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-06-110

Report to the Subcommittee on Readiness and Management Support, Committee
on Armed Services, U.S. Senate

United States Government Accountability Office

GAO

November 2005

BEST PRACTICES

Better Support of Weapon System Program Managers Needed to Improve
Outcomes

Best Practices Best Practices Best Practices Best Practices

GAO-06-110

Contents

Letter 1

Executive Summary 3
Purpose 3
Background 3
Results in Brief 4
Best Practice: Corporate Leadership and Disciplined, Knowledge-Based
Processes Are Critical to Program Manager Success 6
DOD: Critical Support Factors Are Missing 8
Differences in Incentives Contribute to Differences in Support for Program
Managers 10
Chapter 1 Introduction 12
Long-Standing Problems Hamper Weapons Systems Acquisitions 13
DOD Program Managers Are Central Executors of the Acquisition Process 14
Legislation to Improve Program Manager Proficiency 16
Objectives, Scope, and Methodology 17
Chapter 2 Senior Leader Support and Disciplined Knowledge- Based Processes
Are Critical Enablers for Program Managers 22
Senior Leadership Provides Program Managers with a Strong Foundation for
Success 23
Knowledge-Based Process Followed to Execute Programs 30
Continued Senior Leadership during Product Development Further Enabled
Success 35
Chapter 3 DOD Is Not Supporting Its Program Managers Effectively 37
Senior Leadership Does Not Provide a Strong Foundation for Success 38
Execution in DOD Does Not Provide Adequate Support and Accountability 45
Senior Leader Support during Execution 52
Chapter 4 Basic Incentives Drive Differences in How Program Managers Are
Supported and Held Accountable 54
Definition of Success 55
Means for Success 56
Other Differences Put Additional Pressures on DOD Program Managers 58
Chapter 5 Conclusions and Recommendations 61
Recommendations for Executive Action 62
Agency Comments and Our Evaluation 63
Appendix I Comments from the Department of Defense 65
Appendix II GAO Staff Acknowledgments 69
Related GAO Products 70

Tables

Table 1: Acquisition Categories 15
Table 2: Are Best Practices Present in DOD? 39
Table 3: Technology Maturity and Program Outcomes 42
Table 4: Are Best Practices Present in DOD? 47
Table 5: Program Manager Views on Formal vs. Informal Authority 50

Figures

Figure 1: Critical Support and Accountability Factors 23
Figure 2: 2005 Toyota Avalon 28
Figure 3: Siemens Bi-Plane AXIOM Artis 30
Figure 4: Best Practice Roles, Responsibilities, and Behaviors of Senior
Managers 36
Figure 5: Breakdowns in Support and Accountability Factors 38
Figure 6: Highlights of Program Manager Comments Regarding Competition for
Funding 40
Figure 7: To What Extent Were the Parameters of Your Program Reasonable at
Program Start? 43
Figure 8: How Program Managers Responded to an Open-ended Question on What
Were the Biggest Obstacles They Faced 44
Figure 9: Highlights of Program Manager Comments on What Types of
Authority They Need 51
Figure 10: Key Differences in Definition of Success and Resulting
Behaviors 55
Figure 11: Commercial vs. DOD Oversight Environments 59

Abbreviations

DAWIA Defense Acquisition Workforce Improvement Act

DOD Department of Defense

OSD Office of the Secretary of Defense

PEO program executive officer

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office

Washington, DC 20548

November 30, 2005

The Honorable John Ensign

Chairman

The Honorable Daniel K. Akaka

Ranking Minority Member

Subcommittee on Readiness and Management Support

Committee on Armed Services

United States Senate

As you requested, this report examines how program managers in the
Department of Defense are supported and how they are held accountable for
program outcomes. It compares department polices and practices to those of
leading commercial companies we visited and discusses actions DOD could
take to improve the accountability of program managers and provide them
with timely support as they manage the development of complex systems. We
make recommendations to the Secretary of Defense to (1) develop an
investment strategy to prioritize needed capabilities, (2) require, for
each new program, that senior level stakeholders formally commit to a
business case for program approval at the start of a new program, and (3)
implement a process to instill and sustain accountability for successful
program outcomes.

We are sending copies of this report to the Secretary of Defense; the
Secretary of the Army; the Secretary of the Navy; the Secretary of the Air
Force; the Director, Missile Defense Agency; the Director of the Office of
Management and Budget; and interested congressional committees. We will
also make copies available to others upon request. In addition, the report
will be available at no charge on the GAO Web site at http://www.gao.gov.

If you have any questions regarding this report, please call me at (202)
512-4841. Staff acknowledgements are listed in appendix II.

Michael J. Sullivan

Director, Acquisition and Sourcing Management

Exe  Executive Summary

                                    Purpose

The Department of Defense (DOD) plans to increase its investment in the
research, development, and procurement of new weapon systems from $144
billion in fiscal year 2005 to $185 billion in fiscal year 2009. U.S.
weapons are among the best in the world, but the programs to acquire them
often take significantly longer and cost more money than promised and
often deliver fewer quantities and other capabilities than planned. It is
not unusual for estimates of time and money to be off by 20 to 50 percent.
When costs and schedules increase, quantities are cut, and the value for
the warfighter-as well as the value of the investment dollar-is reduced.

In view of the importance of DOD's investment in weapon systems, we have
undertaken an extensive body of work that examines DOD's acquisition
issues from a different, more cross-cutting perspective-one that draws
lessons learned from the best commercial product development efforts to
see if they apply to weapon system acquisitions. In response to a request
from the Chairman and Ranking Minority Member of the Subcommittee on
Readiness and Management Support, Senate Committee on Armed Services, this
report assesses (1) how successful commercial companies position their
program managers, (2) how DOD positions its program managers, and (3)
underlying reasons for the differences.

                                   Background

DOD relies on a relatively small cadre of military and civilian
officials-known as program managers-to lead the development and delivery
of its weapon systems. The responsibility placed on this group is
enormous. The systems that program managers are responsible for range from
highly complex and sophisticated aircraft, missile interceptors,
submarines, and space-based sensors, to new communication and ground
control systems that support and interconnect this equipment, to smaller,
less complex systems that support the warfighter. In these times of
asymmetric threats and netcentricity, individual weapon system investments
are getting larger and more complex. The development process itself is
very challenging as many systems require successful management and
coordination of a broad array of military service and DOD officials,
outside suppliers, internal and external oversight entities, as well as
technical, business, contracting, and management expertise. Moreover, in
many cases, weapon systems are also expected to incorporate technologies
that push the state-of-the-art while operating in harsh and even untested
environments-adding daunting technical challenges to the already existing
business, management, and logistical challenges. Lastly, GAO has reported
many of the business processes that support weapons development-strategic
planning and budgeting, human capital management, infrastructure,
financial management, information technology, and contracting-are beset
with pervasive, decades-old management problems, which include outdated
organizational structures, systems, and processes.1

Weapon system program managers are the central executors of the
acquisition process. They are responsible for all aspects of development
and delivery of a new system and for assuring that systems are high
quality, affordable, supportable, and effective. In carrying out this
responsibility, they are also responsible for balancing factors that
influence cost, schedule, and performance. DOD employs about 729 program
managers to run its weapons programs. Both military officers and civilians
serve as program managers, but the majority is from the military. DOD's
program managers typically report to program executive officers (PEO) who
are charged with overseeing the execution of a portfolio of related
systems. PEOs, in turn, typically report to a military service acquisition
executive, who reports to a service secretary, or for some programs, the
PEO reports to the Defense Acquisition Executive.

                                Results in Brief

Program managers from the leading companies we spoke with believed that
two critical enablers-(1) support from top leadership and (2) disciplined,
knowledge-based processes for product development execution-empowered them
to succeed in delivering new products when needed within cost, quality,
and performance targets originally set by the company. Long before the
initiation of a new product development, senior company leaders make
critical strategic investment decisions about the firm's mix of products
and the return on investment they may yield. Once high-level investment
decisions were made, senior leaders ensured that programs did not begin
unless they had a business case that demonstrated the program was aligned
with the company's goals and that resources were in-hand to execute the
program-that is, time, technology, money, and people. Once a business case
was established, senior leaders tasked program managers with executing
that business case for each new product from initiation to delivery, but
required their program managers to use a knowledge-based product
development process that demanded appropriate demonstrations of
technology, designs, and processes at critical junctures. The program
manager was empowered to execute the business case, but also held
accountable for delivering the right product at the right time for the
right cost. Throughout execution, company senior leaders supported their
program managers by encouraging open and honest communication and
continually assured that the right levels of resources and management
attention were available for the project.

1 Defense Management: Key Elements Needed to Successfully Transform DOD
Business Operations, GAO-05-629T (Washington, D.C.: April 28, 2005) and
High-Risk Series: An Update, GAO-05-207 (Washington, D.C.: January 2005).

While DOD has taken action in recent years to better position programs for
success, it puts its program managers in a very different situation.
Program managers themselves believe that rather than making strategic
investment decisions, DOD starts more programs than it can afford and
rarely prioritizes them for funding purposes. The result is a competition
for funds that creates pressures to produce optimistic cost and schedule
estimates and to overpromise capability. Our own work has shown that many
programs begin without a business case, that is, without adequate
knowledge about technology, time, and cost and without demonstrating that
the program itself is the optimal approach for achieving a needed
capability. Moreover, once programs begin, the program manager is not
empowered to execute the program. In particular, program managers cannot
veto new requirements, control funding, or control staff. In fact, program
managers personally consider requirements and funding instability to be
their biggest obstacles to success. Program managers also believe that
they are not sufficiently supported once programs begin. In fact, they
must continually advocate for their programs in order to sustain support.
Our past reports also show that programs are incentivized to suppress bad
news and to continually produce optimistic estimates-largely due to
continual funding competition.

Many of these differences can be attributed to how success is defined
within the commercial and DOD environment. Success for the commercial
world is straightforward and simple: maximize profit. In turn, this means
selling products to customers at the right price, the right time, and the
right cost. With this imperative in hand, companies have no choice but to
adopt processes and cultures that emphasize basing decisions on knowledge,
reducing risks prior to undertaking new efforts, producing realistic cost
and schedule estimates, and assuring consistency and quality pervade all
efforts. At first glance, DOD's definition of success is very similar:
deliver capability to the warfighter at the right price, the right time,
and the right cost. But, for various reasons, it is clear that the implied
definition for success is to attract funds for new programs and to keep
funds for ongoing programs. While the annual appropriations process and
the wide variety of mission demands placed on DOD contribute to this
condition, DOD has made matters worse by not making hard tradeoff
decisions to ensure it does not pursue more programs than it can afford.
Once attracting funds becomes "success," harmful practices emerge. For
example, it is not in a program manager's interest to develop accurate
estimates of cost, schedule, and technology readiness, because honest
assessments could result in lost funding. Delayed testing becomes
preferred over early testing because that will keep "bad news" at bay. In
turn, knowing data being reported to them may not be reliable, senior
leaders believe they cannot trust it and must instill multiple oversight
mechanisms. Any attempts to improve policy and processes eventually
succumb to funding competition because no one wants to risk loss of
support.

We are making recommendations to DOD to better position program managers
for success. These recommendations focus on what is needed to be done to
provide the strategic leadership needed to provide the right foundation
for starting programs, ensure an executable business case is delivered to
program managers, and to hold program managers accountable for successful
outcomes. It is important to note that the success of all of our
recommendations hinge on DOD's ability to instill more discipline and
leadership over the investment process. After a review of a draft of this
report, DOD concurred with our recommendations and provided some
additional comments. The full text of DOD's comments may be found in
appendix I.

 Best Practice: Corporate Leadership and Disciplined, Knowledge-Based Processes
                    Are Critical to Program Manager Success

At all of the companies we visited, support for program managers began
well before they were assigned to a new product development effort-with
high-level strategic planning and investment decisions and concerted
efforts to make sure that any new initiative the company undertook was
achievable within the time and money and other resources the company had
available. Technology development and program advocacy were also generally
kept out of a program manager's domain. Once new efforts got off the
ground, program managers were empowered to manage resources, encouraged to
bring up problems and propose solutions, and consult with senior leaders
without fear of losing their support. At the same time, however, they were
expected to base their decisions on hard data and to assure the right
knowledge was in-hand before proceeding into the next phases of
development. They were also held accountable for their choices, though
companies generally found that with good pre-program decisions, a good
launch, a sound, disciplined process for execution, and continued support,
there was little need to punish or remove their program managers.
Ultimately, as long as a program manager could deliver the right product
at the right time for the right cost, he was incentivized to do so without
interference from above.

According to commercial program managers we spoke with, the most critical
support factors included the following:

           o  Investment strategies. Each of the companies we visited
           followed a rigorous process to forecast market needs against
           company resources, economic trends, available technologies, and
           its own strategic vision. These exercises culminated in short- and
           long-term investment strategies that provided program managers
           with confidence that the company was committed to their particular
           program and showed them where the project fit within overall
           corporate goals.

           o  Evolutionary development. All of the companies followed an
           incremental path toward meeting market needs rather than
           attempting to satisfy all needs in a single step. This provided
           program managers with more achievable requirements, which, in
           turn, facilitated shorter cycle times. With shorter cycle times,
           the companies could ensure both program managers and senior
           leaders stayed with programs throughout the duration.

           o  Matching requirements to resources. Once specific product
           concepts were identified, the companies worked vigorously to close
           gaps between requirements/customer needs, and resources-time,
           money, and technology. In effect, this took the investment
           strategy down to a project level, assuring that the program
           manager would be well positioned to execute within cost and
           schedule.

           o  Matching the right people to the program. All of the companies
           we visited took steps to ensure that they assigned the right
           people to the right programs. These included long term efforts to
           train and groom technical staff into program managers, mentoring
           on the part of senior leaders with program management experience,
           handpicking program managers based on their expertise and
           experience, and supporting program managers with teams of highly
           qualified functional and technical experts.

           o  Knowledge-driven development decisions. Once a new product
           development began, program managers and senior leaders used
           quantifiable data and demonstrable knowledge to make go/no-go
           decisions. These covered critical program facets such as cost,
           schedule, technology readiness, design readiness, production
           readiness, relationships with suppliers, etc. Development was not
           allowed to proceed until certain thresholds were met, for example,
           a high percentage of engineering drawings completed or production
           processes under statistical control. Development processes were
           also continually tailored based on lessons learned. Program
           managers themselves placed high value on these requirements, as
           they ensured programs were well positioned to move into subsequent
           phases and were less likely to encounter disruptive problems.

           o  Empowerment. At all the companies we visited, program managers
           were empowered to make decisions as to whether programs were ready
           to move forward and to resolve problems and implement solutions.
           They could redirect available funding, if needed. They could
           change team members. Prior to development, they often had a say in
           what requirements they would be handed.

           o  Accountability. With authority, came accountability. Program
           managers at all of the companies we visited were held accountable
           for their choices. To assure accountability, senior leaders set
           goals that were clear to the entire project team and provided
           incentives for program managers and others to meet those goals.

           o  Tenure. All of the companies we visited required that program
           managers stay on until the end of the program. This was a primary
           means of assuring accountability.

           o  Continued senior leadership. In addition to empowering them,
           program managers credited senior leaders with other vital levels
           of support. Namely, senior leaders' commitment to their programs
           were unwavering, they trusted their program managers, they
           encouraged them to share bad news, and they encouraged
           collaboration and communication. At the end of the day, it was the
           senior leaders' job to anticipate and remove obstacles and provide
           the right levels of support so that the path was cleared for the
           program manager to execute the program.

           At DOD, program managers are not put in a position to deliver a
           product within estimates, nor are they held accountable when there
           are failures to deliver products within estimates. While senior
           leaders work hard to develop a short- and long-term vision for the
           defense of the United States, these visions are rarely translated
           into realistic investment strategies that assure the right mix of
           programs is being pursued. Moreover, while recognized in policy as
           a best practice, DOD does not always make sure that there is a
           business case for new initiatives. Lastly, program managers are
           not empowered to execute programs once they begin or held
           accountable when programs get off track.

           The primary problem, according to many program managers and
           verified by GAO's work, is that DOD starts more programs than it
           can afford and does not prioritize programs for funding. This
           creates an environment where programs must continually compete for
           funding. Before programs are even started, advocates are
           incentivized to underestimate both cost and schedule and
           overpromise capability.

           A second problem is that gaps between resources and requirements
           are not closed before or even during program development. For
           example, we have reported that DOD allows many programs to go
           forward without knowing whether critical technologies-such as
           satellite's main sensor, a fighter aircraft's stealth technology,
           a new tank's networking capability-can work as intended.
           Invariably, when programs start with such unknowns, they spend a
           great deal of time and money later on fixing technical glitches
           while simultaneously trying to get other program aspects on track.
           One reason programs begin with immature technologies is that
           program advocates are rushed to start the acquisition program
           because it assures at least an initial commitment of funding.
           Compounding this problem is the fact that acquisition programs
           tend to attract funds over other activities, including science and
           technology efforts that ultimately support acquisition. As a
           result, program managers are incentivized to take on tasks that
           really should be accomplished within a laboratory environment,
           where it is easier and cheaper to discover and address technical
           problems.

           A third problem is that program managers themselves are not
           empowered to execute their programs. First, they have little
           control over funding and they cannot count on funding to be
           stable. When funding is taken away, program managers often find
           themselves in a negative spiral of funding-related
           problems-particularly because they've already made commitments to
           contractors based on certain anticipated levels of funding.
           Second, they cannot veto new requirements. Faced with long
           development life cycles and promising technology advances, users
           often ask for new or better capabilities as a program proceeds
           forward. Program managers themselves are not always empowered to
           say "no" to demands that may overly stretch their programs, and
           few senior leaders above them have been willing to. In addition,
           program managers have little authority over staffing and the
           ability to shift funds within the program. With so much outside
           their span of control, program managers say that DOD is unable to
           hold them accountable when programs get off track. Another reason
           that it is difficult to hold program managers accountable is that
           their tenure is relatively short. The problems being encountered
           today may well be the result of a poor decision made years ago by
           another program manager.

           DOD has tried to improve its processes and policies to better
           position programs for success. For example, policies embrace the
           concept of closing gaps between requirements and resources before
           launching new programs, and DOD is making changes to requirements
           setting and funding processes in an attempt to strengthen
           investment decisions. At this point, however, program managers do
           not see trade-offs being made in the front-end of product
           development that would ensure DOD could fully commit to their
           programs and allow program managers themselves to focus solely on
           executing their programs. The level of trust, collaboration and
           communication is low, while the level of oversight and second
           guessing is high.

           Differences between how program managers are supported and held
           accountable are rooted in differences in incentives and resulting
           behaviors. This begins with the definition of success. The
           commercial firms we studied concluded their survival hinged on
           their ability to increase their market share, which, in turn,
           meant developing higher quality products, at the lowest possible
           price, and delivering them in a timely fashion-preferably before
           their competitors could do the same. This imperative meant that
           they had no choice but to narrow the gap between requirements and
           resources in a manner that not only ensured they met their market
           targets, but did so in a manner that consumed resources fairly
           efficiently. It also meant that they had no choice but to fully
           support the development effort, instill strategic planning and
           prioritization, work collaboratively, follow a knowledge-based
           process that makes product development manageable, and,
           ultimately, make everyone accountable for success. Ultimately, the
           companies developed processes that embodied these tenets for
           success. At the strategic level, these include accurate, strategic
           planning and prioritization to ensure the right mix of products
           are pursued and strong systems engineering to help them establish
           a realistic business case. At the tactical level, companies
           developed development processes that required certain thresholds
           of knowledge to be gained before a decision to proceed forward is
           made.

           In theory, DOD's success likewise hinges on its ability to deliver
           high-quality weapons to the warfighter in a timely fashion. But in
           practice, success is defined as the ability of a program to win
           support and attract funds. Of course, there are reasons for this
           disconnect. Corporate revenue is generated by customer sales while
           DOD's funding is dependent on annual appropriations. Corporations
           go out of business when their product development efforts do not
           succeed; DOD does not. Selling products to customers is the single
           focus of a private-sector company while DOD is charged with a
           myriad of important missions-each of which also competes for
           budget share. Nevertheless, these conditions create a vastly
           different set of processes and behaviors affecting program
           managers. Program managers are incentivized, for example, to be
           optimistic and suppress bad news because doing otherwise could
           result in a loss of support and funding and further damage their
           program. In short, unknowns become acceptable and desirable rather
           than unacceptable as they are in the corporate environment. And
           accountability becomes much more difficult to define.

           DOD plans to spend about $1.3 trillion for its major programs
           between 2005 and 2009 and increase its investment in research and
           development during that period by about 28 percent-from $144
           billion to $185 billion. Although DOD's weapons are widely
           regarded as unrivaled in superiority, DOD has not received a
           predictable return on investment in major weapon systems
           acquisitions. For decades, many of DOD's weapon systems
           acquisitions have experienced large cost increases and extended
           schedules, which, in turn, have jeopardized performance and, more
           broadly, undermined DOD's buying power.

           To help better position DOD to successfully field weapons, we have
           undertaken a body of work over the past decade that has examined
           lessons learned from the best commercial product development
           efforts to see if they can be applied to DOD weapon system
           development. Leading commercial firms have developed increasingly
           sophisticated products in significantly less time and at lower
           costs. Our previous best practices reports1 have examined such
           topics as matching resources with requirements, controlling total
           ownership costs, effective use of testing, and product
           development. This report examines the program manager's role and
           the mechanisms that DOD and leading commercial companies use to
           position program managers for success and hold them accountable.
           As the central executor of the acquisition process, DOD depends on
           its program managers to efficiently and effectively run its large
           range of complex weapon systems acquisitions.

           The challenge that program managers now face is massive. Weapon
           systems themselves are becoming increasingly sophisticated and
           interdependent and, therefore, more complicated and difficult to
           develop. At the same time, however, DOD is faced with threats that
           are constantly evolving, requiring quicker development cycles and
           more flexibility within weapons programs. Moreover, many of the
           business processes that support weapons development-strategic
           planning and budgeting, human capital management, infrastructure,
           financial management, information technology, and contracting-are
           beset with pervasive, decades-old management problems, including
           outdated organizational structures, systems, and processes. In
           fact, these areas-along with weapons system acquisitions-are on
           GAO's high-risk list of major government programs and operations.
           Lastly, while DOD plans to considerably ramp up weapons system
           spending in the next 5 years in an effort to dramatically
           transform how it carries out its military operations, it is likely
           to face considerable pressure to reduce its investment in new
           weapons as the nation addresses long-term fiscal imbalances.

           While DOD's acquisition process has produced weapons that are
           among the best in the world, it also consistently yields
           undesirable consequences-such as cost increases, late deliveries
           to the warfighter, and performance shortfalls. Such problems have
           been highlighted, for example, in our past reviews of DOD's F/A-22
           Raptor, Space-Based Infrared System, Airborne Laser, the Joint
           Strike Fighter, and other programs. Our past work has found that
           problems occur because DOD's weapon programs do not capture early
           on the requisite knowledge that is needed to efficiently and
           effectively manage program risks. For example, programs move
           forward with unrealistic cost and schedule estimates, lack clearly
           defined and stable requirements, use immature technologies to
           launch the product development, and fail to solidify design and
           manufacturing processes at appropriate junctures in development.

           When costs and schedules increase, quantities are cut and the
           value for the warfighter, as well as the value of the investment
           dollar, is reduced. Moreover, in these times of asymmetric threats
           and netcentricity, individual weapon system investments are
           getting larger and more complex. Just 4 years ago, the top five
           weapon systems cost about $281 billion; today, in the same base
           year dollars, the top five weapon systems cost about $521
           billion.2 If these megasystems are managed with traditional
           margins of error, the financial consequences-particularly the
           ripple effects on other programs-can be dire.

           DOD has long recognized such problems and initiated numerous
           improvement efforts. In fact, between 1949 and 1986 five
           commissions studied issues such as cycle time and cost increases
           as well as the acquisition workforce. DOD has also undertaken a
           number of acquisition reforms. Specifically, DOD has restructured
           its acquisition policy to incorporate attributes of a
           knowledge-based acquisition model and has reemphasized the
           discipline of systems engineering. In addition, DOD recently
           introduced new policies to strengthen its budgeting and
           requirements determination processes in order to plan and manage
           systems based on joint warfighting capabilities. While these
           policy changes are positive steps, we recently testified that
           implementation in individual programs has not occurred because of
           inherent funding, management, and cultural factors that lead
           managers to develop business cases for new programs that
           overpromise on cost, delivery, and performance of weapon systems.

           DOD relies on a cadre of military and civilian officials-known as
           program managers-to lead the development and delivery of hundreds
           of weapon systems and subsystems. The services report a combined
           total of 729 program managers currently executing programs at all
           acquisition category levels. The systems that program managers are
           responsible for range from highly sophisticated air, land, sea,
           and space-based systems to smaller, less complex communications or
           support equipment that interconnects or supports larger systems.
           Program managers are responsible for assuring that these systems
           are reliable, affordable, supportable, and effective. They carry
           out multiple roles and responsibilities and are expected to have a
           working knowledge in such diverse areas as contracting, budgeting,
           systems engineering, and testing.

           DOD classifies its acquisition programs into categories based upon
           a number of factors such as their size, cost, complexity and
           importance. The largest, most complex and expensive programs
           generally fall under the responsibility of the Under Secretary of
           Defense (Acquisition, Technology and Logistics) while less complex
           and risky programs are overseen by the service or component
           acquisition executive. Table 1 provides more details.

           Table 1: Acquisition Categories

           Source: GAO.

           Note: Category I systems are referred to as "programs" and smaller
           related subsystems are called "projects" or "products." For
           example, the Air Force's B-1 aircraft system-a category IC
           program-includes category II and III projects that may have a
           designated manager. Category 1D and 1C programs are distinguished
           by their milestone decision authority.

           Program managers typically supervise a large staff of engineers,
           contracting personnel, logisticians, business, financial, and
           administrative personnel. The number of people assigned to program
           offices varies widely and depends on factors such as the
           complexity of the system, the category level, and the availability
           of staff. For example, the Joint Strike Fighter, a category ID
           program, is managing the development of three configurations of a
           new aircraft for the Navy, Marines and Air Force, and currently
           has about 200 government and international personnel assigned. By
           contrast the Light Utility Helicopter, a category II project
           relying largely on commercial off-the-shelf components, has a
           staff of 34.

           To successfully deliver a weapon system to the user, program
           managers must also work with a range of individuals outside their
           sphere of influence such as those charged with independent cost
           estimating, testing, funding, writing requirements, security, and
           ensuring interoperability. Simultaneously, the program manager is
           responsible for overseeing, integrating, and evaluating the
           defense contractor's work as the development progresses. Moreover,
           some program managers lead international teams. For example, the
           Joint Strike Fighter Program Office, in addition to the military,
           civilian, and contract team members, has eight international
           partners and approximately 40 international team members.

           The majority of DOD program managers for category I programs are
           military officers at the rank of colonel or (Navy) captain.
           Subsystem program managers are usually lower in rank and report
           directly to the system program manager. DOD also employs civilian
           program managers, usually GS-15s for its category I programs. As a
           rule, program managers report to a Program Executive Officer-a
           civilian at the senior executive level or military officer at the
           general officer rank-who typically manages a portfolio of related
           weapon systems. However, some program executive officers are
           responsible for a single large program, such as the Joint Strike
           Fighter or the F-22 aircraft. One level up from the program
           executive officer is the Service Acquisition Executive, a civilian
           (often a political appointee) who reports to the service
           Secretary. Programs classified as a category ID report through the
           defense acquisition executive, Undersecretary of Defense
           (Acquisition, Technology and Logistics), as their milestone
           decision authority.

           Program manager training and tenure is now governed by legislation
           known as the Defense Acquisition Workforce Improvement Act
           (DAWIA),3 enacted in 1990 after studies showed that a key problem
           affecting acquisitions was that program managers did not stay in
           their positions long enough to be accountable for outcomes and
           that many simply lacked the training and experience needed to
           assume their leadership roles. Congress amended the law in the
           fiscal year 2004 and 2005 defense authorization acts to allow the
           Secretary of Defense more flexibility to tailor tenure,
           experience, and education qualifications for program managers.

           The act specifically created a formal acquisition corps and
           defined educational, experience, and tenure criteria needed for
           key positions, including program managers as well as contracting
           officers and others involved the acquisition process. The act also
           provided for the establishment of a defense acquisition university
           to provide educational development and training for acquisition
           personnel. Under DOD regulations program managers are required to
           attend training and meet course requirements through the
           university in order to meet certification requirements for the
           program management track.

           There are three progressive certification levels: basic,
           intermediate, and advanced. Program managers of major defense
           acquisition programs are required to have Level 3 certification,
           which requires four years of acquisition experience and an
           advanced level Defense Acquisition University course in program
           management. DOD prefers that individuals with Level 3
           certification have a Master's degree in engineering, systems
           acquisition management, or business administration, and complete
           additional external coursework in relevant fields.

           The Chairman and the Ranking Member, Subcommittee on Readiness and
           Management Support, Senate Committee on Armed Services, requested
           that we examine best practices and DOD procedures for factors that
           affect program manager effectiveness. Our overall objectives for
           this report were to (1) identify best practices that have enabled
           organizations to successfully position their program managers for
           success, (2) identify DOD practices for supporting program
           managers and holding them accountable, and (3) compare and
           contrast DOD and commercial practices in order to identify
           possible improvements to DOD practices.

           To identify the best practices and processes that commercial
           companies employ to position their program managers for success,
           we used a case study methodology. We selected companies that, like
           DOD, research, develop, and field products, using program managers
           as the central executors of the programs. Selection of the
           companies was also based upon recognition by the American
           Productivity and Quality Center and the Project Management
           Institute and the recommendations of experts. Below are
           descriptions of the three companies that are specifically featured
           in this report.

           o  Toyota Motor Manufacturing of North America, Inc.

           Toyota Motor Manufacturing of North America, Inc., the third
           largest automobile producer in the world and the fifth largest
           industrial company in the world, designs, manufactures, and
           markets cars, trucks, and buses worldwide. In 2005, the company
           reported total net sales of $172.7 billion. We met with
           individuals involved with the development of the 2005 Toyota
           Avalon, a full-size sedan, at Toyota Motor Manufacturing in
           Erlanger, Kentucky.

           o  Siemens Medical Solutions USA, Inc.

           Siemens Medical Solutions is one of the world's largest suppliers
           in the healthcare industry. Siemens Medical manufactures and
           markets a wide range of medical equipment, including magnetic
           resonance imaging systems, radiation therapy equipment, ultrasound
           equipment, and patient monitoring systems. We met with individuals
           from the Angiography, Cardiology, and Neurology business unit,
           located in Hoffman Estates, Illinois.

           o  Motorola, Inc.

           Motorola is a Fortune 100 global communications leader that
           provides seamless mobility products and solutions across
           broadband, embedded systems and wireless networks. Seamless
           mobility harnesses the power of technology convergence and enables
           smarter, faster, cost-effective, flexible communication in homes,
           autos, workplaces and all spaces in between. Motorola had sales of
           $31.3 billion in 2004. We visited its offices in Arlington
           Heights, Illinois, and discussed program management practices and
           processes with representatives from the Networks sector.

           In addition to the three companies featured in this report, we
           visited two additional successful firms to assess whether they
           employed similar processes and practices for program management.
           These include Molson Coors Brewing Company and Wells Fargo. Both
           companies have undertaken projects that reflect some of the
           complexity and challenges that a DOD weapon systems program would
           face. For example, we met with managers of a Molson project
           intended to automate day-to-day marketing operations for digital
           assets. We also met with Wells Fargo officials who developed an
           electronic imaging process for paperless check clearance. At both
           companies, we also discussed broader corporate investment
           processes that supported these particular internal projects as
           well as the companies' main service lines.

           For each of the five companies, we interviewed senior management
           officials and program managers to gather consistent information
           about processes, practices, and metrics the companies use to
           support program managers and hold them accountable. In addition to
           the case studies, we synthesized information from GAO's past best
           practices work about product development.

           We also examined key best practices studies related to program
           management, including studies from organizations such as the
           Project Management Institute and the American Productivity and
           Quality Center. Moreover, we relied on our previous best practice
           studies, which have examined incentives and pressures affecting
           weapon system programs, the optimal levels of knowledge needed to
           successfully execute programs, and complementary management
           practices and processes that have helped commercial and DOD
           programs to reduce costs and cycle time.

           In order to determine DOD practices for supporting program
           managers and holding them accountable, we conducted five separate
           focus groups between July and October 2004. Each group was
           composed of project managers from one of the services or the
           Missile Defense Agency. A total of 28 acquisition category I
           program managers representing a range of DOD programs were
           identified by their respective services for the meetings held in
           separate locations in Huntsville, Ala.; El Segundo, Calif.;
           Dayton, Ohio; Arlington, Va.; and Ft. Belvoir, Va. For each focus
           group, the facilitators introduced discussion topics to discover
           how program managers define success, as well as what they are
           accountable for and how they are held accountable. In addition,
           participants were asked to discuss how program managers are
           supported and what obstacles they encounter in performing their
           duties.

           We analyzed the content of focus group transcripts and used the
           themes we identified to design a survey to gather information
           about acquisition category I and II program managers' perceptions
           about factors that assist or block their success and to more
           clearly define other issues in the DOD acquisition process that
           affect program manager effectiveness. We elicited input from
           several experts-retired program managers, active-duty members with
           program management experience, and senior acquisition officials
           who reviewed the questions and provided feedback on the draft
           survey.

           We pretested the survey with five program managers. During the
           pretest we asked the program managers questions to determine
           whether (1) the survey questions were clear, (2) the terms used
           were precise, (3) the questionnaire placed an undue burden on the
           respondents, and (4) the questions were unbiased. We then
           incorporated their comments into the survey, finalized the
           questions, and sent the web-based survey to acquisition category I
           and II program managers. We selected the category I and II program
           managers because they manage the more complex and expensive
           programs. We identified the program managers through consultation
           with each of the services. The survey consisted of open- ended and
           close-ended questions concerning support for program managers and
           how they are held accountable for program outcomes. Originally we
           e-mailed 237 program managers but later determined that 52 should
           not be included because they managed programs other than
           acquisition category I and II. Of the 185 remaining program
           managers, we received completed surveys from 69 percent.

           The surveys were conducted using self-administered electronic
           questionnaires posted on the World Wide Web. We sent e-mail
           notifications to all acquisition category I and II program
           managers on April 12, 2005. We then sent each potential respondent
           a unique password and username by e-mail to ensure that only
           members of the target population could participate in the survey.
           To encourage respondents to complete the questionnaire, we began
           sending e-mail messages to prompt each nonrespondent between April
           26, 2005 and May 19, 2005. Additionally, the team contacted
           nonrespondents through telephone calls between May 31, 2005 and
           July 12, 2005. We closed the survey on July 19, 2005.

           In this report we discuss some of the results obtained from the
           survey. A more complete tabulation of survey questions together
           with tables indicating the levels of response can be found on our
           Web site at GAO-06-112SP. The survey contained close-ended
           questions and open-ended questions. We conducted a content
           analysis of the open-ended questions and constructed tables
           showing the results of the analysis arranged into broad
           categories. Some of the respondents to our survey provided more
           than one answer to the open-ended questions. All responses that
           indicated equally important factors were tabulated in the
           appropriate categories. However, because some respondents provided
           more than one answer, the percentages may add up to more than 100
           percent of respondents. The web-based report does not contain all
           the results from the survey. For example, we do not report
           responses for questions about demographics, some open-ended
           questions, or questions with high item nonresponse rates.

           In addition to the focus groups and survey, we conducted in-depth
           interviews with individual program managers, program executive
           officers from across the services, as well as program managers
           from Boeing and Lockheed Martin for two major weapon systems. To
           further assess the conditions and environment program managers
           were operating in, we relied on previous GAO reports. For example,
           we relied on a recent study of space acquisition problems that
           incorporated interviews of more than 40 individuals, including
           experienced program managers, program executive officials,
           officials responsible for science and technology activities, and
           former and current officials within the Office of Secretary of
           Defense who have specific responsibility for space system
           oversight or more general weapon system oversight.

           To further determine relevant DOD policies and practices, we
           analyzed documents describing the roles and responsibilities of
           program managers, acquisition force career management, promotion
           rates, performance reporting, and training requirements. Moreover,
           we analyzed relevant legislation and the DOD 5000 series of
           directives and instructions. We also interviewed career
           acquisition service officials, Defense Acquisition University
           course managers, and the Director of Training. We reviewed studies
           from the Rand Corporation, the Center for Strategic and
           International Studies, and the Defense Science Board, among
           others, on weapons system program management and acquisition
           issues as well as studies performed by past commissions focused on
           acquisition reform.

           We conducted our review between April 2004 and November 2005 in
           accordance with generally accepted government auditing standards.

           Program managers from the leading companies we spoke with believed
           that two critical enablers-(1) support from top leadership and (2)
           disciplined, knowledge-based processes for strategic investment,
           program selection, and product development execution-empowered
           them to succeed in delivering new products when needed within
           cost, quality, and performance targets originally set by the
           company. At all of the companies we visited, corporate leadership
           began at a strategic level, long before the initiation of a new
           product development, with senior company leaders making critical
           strategic investment decisions about the firm's mix of products
           and the return on investment they may yield. Once high-level
           investment decisions were made, senior leaders assured that new
           programs did not begin until there was a business case for
           them-meaning there was assurance that the program fit in with the
           corporation's goals and investment strategy and that there were
           resources available to execute the program. Once a business case
           had been made, senior leaders selected and tasked program managers
           with executing the program. They also required the program
           managers to use a knowledge-based product development process that
           demanded appropriate demonstrations of technology, designs, and
           processes at critical junctures. They also empowered program
           managers as appropriate to execute the program and held them
           accountable for delivering the program within estimates. While
           they were empowered to execute the program, program managers were
           still supported by senior leaders, who encouraged open and honest
           communication and continually assured that the right levels of
           resources and management attention were available for the project.
           Figure 1 maps critical support and accountability factors.

                   DOD: Critical Support Factors Are Missing

Differences in Incentives Contribute to Differences in Support for Program
                                    Managers

Chapter 1: Introduction  Chapter 1: Introduction

1 A complete list of best practices reports is at the end of this report.

           Long-Standing Problems Hamper Weapons Systems Acquisitions

2 These figures represent the costs for the top five weapon systems in
2001 and the top five in 2005. For 2001, these systems were F/A-22 Raptor,
DDG-51 Guided Missile Destroyer, Virginia Class Submarine, C-17
Globemaster III, and the F/A 18 E/F, Naval Strike Fighter. The 2005
systems include the Joint Strike Fighter, Future Combat System, F/A- 22
Raptor, DDG-51 Guided Missile Destroyer, and the Virginia Class Submarine.

     DOD Program Managers Are Central Executors of the Acquisition Process

Acquisition                                                                
category     Definition                            Program examples
Category I   Research, development, test, and      Future Combat System    
                evaluation > $365M Procurement >      DD(X) Destroyer B-1     
                $2.19B                                Aircraft                
                                                      
                Milestone decision authority: 1D:     
                Under Secretary of Defense            
                (Acquisition, Technology and          
                Logistics) 1C: Service Acquisition    
                Executive                             
Category II  Research, development, test, and      All Source Analysis     
                evaluation > $140M Procurement >      System KC-130J Aircraft 
                $660M                                 Joint Surveillance and  
                                                      Target Attack Radar     
                Milestone decision authority: Service System                  
                or Component Acquisition Executive    
Category III No fiscal criteria                    10k W Auxiliary Power   
                                                      Unit Assault Breaching  
                                                      Vehicle C-5 Avionics    
Category IV  No fiscal criteria (Navy and Marine   C-130 Night Vision      
                Corps only)                           Lighting Advanced       
                                                      Recovery Control System 

               Legislation to Improve Program Manager Proficiency

3 10 U.S.C.S: 1701 et seq. (P.L. 101-510. Div A. Title XII (November 5,
1990)).

                       Objectives, Scope, and Methodology

Chapter 2: Senior Leader Support and Disciplined Knowledge-Based Processes
Are Critical Enablers for Program Managers  Chapter 2: Senior Leader
Support and Disciplined Knowledge-Based Processes Are Critical Enablers
for Program Managers

Figure 1: Critical Support and Accountability Factors

Senior Leadership Provides Program Managers with a Strong Foundation for Success

At each of the companies we visited, senior leaders invested a great deal
of time and effort positioning new development efforts for success. Before
even considering initiating a new project, senior leaders made high level
trade-off decisions between their long-term corporate goals, projected
resources, market needs, and alternative ways of meeting those needs.
These efforts culminated in investment strategies that assured that the
company could fully commit to any product development effort it pursued.
With a broad strategy in place, senior leaders would then begin concept
development for potential new products, analyzing proposed products in
terms of what requirements could be achieved today versus future versions
of the product and what resources would be needed-not just in terms of
cost, but in terms of technologies, time, and people. Once a specific
concept was selected, senior leaders would follow rigorous systems
engineering processes to narrow the gap between requirements and resources
to a point where they were assured that they were pursuing a product that
would meet market needs and could be developed within cost and schedule
goals. The end point of this process was a sound business case that senior
leaders could then hand off to a program manager-who was then empowered to
deliver the product on time and within cost. Program managers themselves
highly valued this support because it ensured the companies were committed
to their particular efforts, reduced the level of unknowns that they were
facing, and kept them focused solely on executing their programs. Put more
simply, they believed senior leaders consistently provided a sound
foundation on which they could launch their programs.

The most critical characteristics of the strategic leadership provided
include the following:

           o  Investment strategies. Because there are more product ideas
           than there is funding to pursue them, the commercial companies we
           visited used a knowledge-based process to make decisions about
           which product development efforts to invest in. They began by
           developing an investment strategy that supports a corporate
           vision. For the most profitable mix of new products, companies
           analyzed factors such as customer needs, available technology, and
           available resources. Companies ensured that decisions to start new
           product developments fit within the investment strategy. The
           investment strategy determined project priority as well as
           providing a basis for trade-off decisions against competing
           projects. Program managers found their company's use of investment
           strategies helpful because it gave them confidence that their
           project had commitment from their organization and from their top
           leaders and managers and clearly identified where their project
           stood within the company's overall investment portfolio and
           funding priorities.

           o  Evolutionary development. All of the companies generally
           followed an evolutionary path toward meeting market needs rather
           than attempting to satisfy all needs in a single step. In effect,
           the companies evolved products, continuously improving their
           performance as new technologies and methods allow. These
           evolutionary improvements to products eventually result in full
           desired capability, but in multiple steps, delivering enhanced
           capability to the customer more quickly through a series of
           interim products. For example, the 2005 Avalon involved redesign
           of about 60 per cent of the vehicle, but component sections such
           as the electronics and such features as the keyless ignition
           system and the reclining rear seat were either developed by
           suppliers or had been used on the Lexus. By using this method, the
           company changed the Avalon's overall design and functionality by
           increments. In more strategic investment planning, Toyota
           maintains an ongoing research into such technology areas as
           alternative fueled automobiles and environmental implications of
           automotive developments that will feed into its long-term
           planning. Our previous work has found that this approach reduces
           the amount of risk in the development of each increment,
           facilitating greater success in meeting cost, schedule, and
           performance requirements. The approach permits program managers to
           focus more on design and manufacturing with a limited array of new
           content and technologies in a program. It also ensures that the
           company has the requisite knowledge for a product's design before
           investing in the development of manufacturing processes and
           facilities. Conversely, our past work has found that organizations
           that set exceedingly high technology advancement goals invariably
           spend more time and money than anticipated trying to address
           technology-related challenges amid other product development
           activities, including design and production stabilization.

           o  Matching of Requirements and Resources. The companies we
           visited were able to achieve their overall investment goals by
           matching requirements to resources-that is time, money,
           technology, and people-before undertaking a new development
           effort. Any gaps that existed were relatively small, and it was
           the program manager's job to quickly close them as development
           began. More specifically:

                        o  The companies had already extensively researched
                        and defined requirements to ensure that they are
                        achievable given available resources before
                        initiating new efforts.

                        o  Technologies were mature at the start of a
                        program, that is, they had been proven to work as
                        intended. More ambitious technology development
                        efforts were assigned to corporate research
                        departments until they were ready to be added to
                        future generations (increments) of the product. In
                        rare instances when less mature technologies were
                        being pursued, the company accepted the additional
                        risk and planned for it.

                        o  Companies committed to fully fund projects before
                        they began. Not one of the program managers we spoke
                        with mentioned funding as a problem at the beginning
                        of a development effort and throughout. Funding was a
                        given once senior leaders had committed to their
                        project.

                        o  Systems engineering was typically used to close
                        gaps between resources and requirements before
                        launching the development process. As our previous
                        work has shown, requirements analysis, the first
                        phase of any robust systems engineering regimen, is a
                        process that enables the product developer to
                        translate customer wants into specific product
                        features for which requisite technological, software,
                        engineering, and production capabilities can be
                        identified. Once these are identified, a developer
                        can assess its own capabilities to determine if gaps
                        exist, and then analyze and resolve them through
                        investments, alternate designs, and, ultimately,
                        trade-offs. The companies we visited allowed their
                        engineers to analyze and weigh-in on the customers
                        needs as determined by its marketers.

           Our previous best practice work has consistently found the
           practice of matching requirements and resources prior to
           initiating a new program to be a hallmark for successful
           companies. Simply put, we have found that when wants and resources
           are matched before a product development is started, the
           development is more likely to meet performance, cost, and schedule
           objectives. When this match does not take place at the start of a
           program, programs typically encounter problems such as increased
           costs, schedule delays, and performance shortfalls as they try to
           meet requirements during product development. Program managers we
           spoke with for this review specifically cited this process as an
           enabler for their own success because it ensured they were in a
           good position to commit to cost and schedule estimates that were
           attainable, and it did not require them to perform "heroic"
           efforts to overcome problems resulting from large gaps between
           wants and resources, such as technology challenges or funding
           shortages.

           In addition to these critical strategic enablers, program managers
           at the companies also stated that senior leaders made concerted
           efforts to match program manager skills and experience to
           appropriate projects and to train and mentor program managers. In
           selecting program managers themselves, the companies placed high
           value on strong leadership qualities, including decision making
           skills, diplomacy, communication skills, ability to motivate
           others, and integrity, as well as how individual personalities fit
           with the job or team. Most of the program managers we spoke with
           had been groomed for their positions through formal training on
           budgeting, scheduling, and regulatory compliance and other aspects
           of program management; informal mentoring by senior executives or
           experienced program managers; and by being placed in positions
           that gradually increased their management responsibilities. In
           addition, many of the program managers we spoke with also
           possessed considerable technical experience. In fact, they often
           started at the company as engineers. The companies we visited were
           similarly deliberate in developing and deploying teams of
           functional experts to support a program manager. In some cases,
           the teams reported directly to the program manager. In others,
           they reported to their respective home units and worked
           collaboratively with the program managers. In either case, the
           program managers themselves valued the support they were getting
           from these teams-particularly because they enabled the program
           manager to employ a broad array of expertise from day-one of the
           development effort and to facilitate an exchange of ideas. The
           program managers we spoke with believed that their functional
           teams were also highly skilled-to the point where they could
           easily delegate major tasks.

           Strategic leadership for the development of Toyota's Avalon luxury
           sedan ties back to conscious decisions made by senior leaders in
           Japan when they built a Toyota facility in the United States 25
           years ago. To assure that the vehicles could be made to the same
           levels of quality as those in Japan, Toyota replicated its
           manufacturing facilities, used Japanese suppliers, and sent its
           managers to the United States to supervise development. As U.S.
           employees gained experience and demonstrated their capability, the
           reliance on Japanese suppliers and personnel gradually decreased.
           A second step Toyota took was to replicate its training and
           mentoring of program managers-pairing them with more experienced
           chief engineers, who oversee long-term planning across projects,
           and even bringing them to Japan to study how Toyota approached
           development.

           To support all of its new development efforts, senior leaders have
           developed an overall strategic plan-which takes a long- and
           short-term investment perspective. Over the long run, the plan
           envisions the company achieving significant advancements in
           capabilities, such as alternatively fueled engines, through
           incremental improvements to technologies. Over the short run, a
           specific vehicle development program uses a marketing analysis
           about features customers desire in new models; and the staff
           determines whether a market exists for a certain type of product
           at a certain price. In establishing a business case for the
           Avalon, Toyota embarked on a formal concept development effort,
           which was led by a chief engineer. The chief engineer, a
           high-level executive, was largely responsible for setting the
           vision for the new Avalon, securing resources needed for
           development effort prior to initiating the development program,
           and working with representatives from its sales division to make
           sure that the design and technologies being pursued still fit
           within market needs-not just in terms of cost, but in terms of
           vehicle features. A variety of functional experts were consulted
           during this phase, though the chief engineer had the most formal
           authority over concept development. At the conclusion of this
           effort, Toyota decided to take on a very extensive redesign of the
           Avalon but also set a goal bringing the vehicle to market in only
           18 months. Redesign features included a reinforced body,
           improvements to the engine and to the braking system, as well as
           features customers desired such as a keyless ignition system and
           reclining rear seats. Toyota leadership also decided to include
           mature technologies, often borrowed from other vehicle lines, or
           purchased from outsider suppliers. Once the design was approved,
           day to day project management shifted to the Chief Production
           Engineer, whose responsibility it was to see the vehicle through
           production to distribution.

           Figure 2: 2005 Toyota Avalon

           Corporate leadership at Siemens Medical took a similar shape in
           the development of new medical equipment. For example, senior
           leaders developed an overall investment strategy, based largely on
           researching their customers' technology needs as well as their own
           technology readiness, the direction their competitors were going
           in, economic trends, and projected manpower resources. From these
           assessments, a team within Siemens developed a portfolio of
           potential new projects to pursue, which upper management then
           prioritized based on their potential profit, how they fit in with
           corporate goals and projected resources. Ultimately, senior
           leaders produce a short-term (1 year) investment plan as well as a
           longer-term (3 to 5 year) plan. Once a specific project is
           selected, Siemens employs systems engineering practices to narrow
           down the gap between customer requirements and resources-working
           with both business and technical managers. A "product manager" is
           charged with making trade-offs between requirements, schedule, and
           cost prior to initiating product development and is held
           accountable for systems engineering decisions made to level
           requirements with resources for the business case. This person
           sits at a relatively high level within the company and possesses
           marketing and business expertise. A "project manager" who reports
           to the product manager is ultimately assigned to execute the
           business case, but he or she plays a role in the concept
           development by participating in trade-off decisions and raising
           concerns about how decisions can be executed.

           At Siemens Medical, many project managers begin by serving as the
           technical leader working with three to five people in systems
           engineering or another technical area of a project. As the
           technical team lead, they gain experience with scheduling,
           communicating, and managing people. Over time the individual is
           given more responsibilities such as becoming a subsystem project
           leader; as the manager gains experience, he or she transitions to
           handling cross-functional areas including business, budgeting,
           staffing, technology, and testing.

           Siemens Medical project managers are also given formal training,
           including courses on regulatory and quality requirements as well
           as courses that help program managers learn about their management
           styles. In addition, Siemens ensures that project managers are
           well-trained on risk management so that they can identify and
           mitigate potential risks at the beginning of the project. Also,
           since project managers function within a centralized project
           management department, they are mentored both by the head of the
           department and by their peers.

           Figure 3: Siemens Bi-Plane AXIOM Artis

           Once a new development effort began, program managers were
           empowered to execute the business case and were held accountable
           for doing so. At all of the companies we visited, program managers
           believed that following a disciplined, knowledge-based development
           process and continued support from senior leaders were essential
           to their success. The process itself was typically characterized
           by a series of gates or milestone decisions, which demanded
           programs assess readiness and remaining risk within key sectors of
           the program as well as overall cost and schedule issues, and it
           required go/no-go decisions to be made fairly quickly. The most
           important aspect of the process, in the view of the program
           managers, was that it empowered them to make decisions about
           design and manufacturing trade-offs, supplier base, staffing on
           the program team, etc.-as long as they were within the parameters
           of the original business case. At the same time, the process held
           program managers accountable and set clear goals and incentives.

           Common critical characteristics of the knowledge-based process
           followed to execute programs include the following:

           o  Knowledge-driven development decisions. Once a new product
           development began, program managers and senior leaders used
           quantifiable data and demonstrable knowledge to make go/no-go
           decisions. These covered critical facets of the program such as
           cost, schedule, technology readiness, design readiness, production
           readiness, and relationships with suppliers. Development was not
           allowed to proceed until certain thresholds were met, for example,
           a high proportion of engineering drawings completed or production
           processes under statistical control. Program managers themselves
           placed high value on these requirements, as it ensured they were
           well positioned to move into subsequent phases and were less
           likely to encounter disruptive problems.

           o  Empowerment for program managers to make decisions. At all the
           companies we visited, program managers were empowered to make
           decisions on the direction of the program and to resolve problems
           and implement solutions. They could make trade-offs among
           schedule, cost, and performance features, as long as they stayed
           within the confines of the original business case. When the
           business case changed, senior leaders were brought in for
           consultation-at this point, they could become responsible for
           trade-off decisions.

           o  Accountability. Program managers at all the companies we
           visited were held accountable for their choices. Sometimes this
           accountability was shared with the program team and/or senior
           leaders. Sometimes, it resided solely with the program manager on
           the belief that company had provided the necessary levels of
           support. In all cases, the process itself clearly spelled out what
           the program manager was accountable for-the specific cost,
           performance, schedule, and other goals that needed to be achieved.

           o  Tenure. To further ensure accountability, program managers were
           also required to stay with a project to its end. Sometimes senior
           leaders were also required to stay. At the same time, program
           managers were incentivized to succeed. If they met or exceeded
           their goals, they received substantial bonuses and/or salary
           increases. Awards could also be obtained if the company as a whole
           met larger objectives. In all cases, companies refrained from
           removing a program manager in the midst of a program. Instead,
           they chose first to assess whether more support was needed in
           terms of resources for the program or support and training for the
           program manager.

           Other important aspects within the development process included
           the following:

           o  Common templates and tools to support data gathering and
           analysis. These tools included databases of demonstrated,
           historical cost, schedule, quality, test, and performance data
           that helped program managers produce metrics as well as standard
           forms and guidance for conducting the meetings. Program managers
           valued these tools because they greatly reduced the time needed to
           prepare for milestone meetings. In all cases, program managers did
           not believe they were spending time collecting data that was
           valuable to senior management but not to them.

           o  Common processes that supported product development. The
           companies generally found that requiring program managers to
           employ similar risk management, project management, requirements
           approval, testing, quality management, problem resolution, and
           other processes enabled them to add additional discipline and
           consistency to product development. Some companies were certified
           by professional organizations as achieving the highest level of
           proficiency within supporting development processes. For example,
           Motorola was certified as a level 5 software development
           organization by Carnegie Mellon's Software Engineering Institute.

           o  Lessons learned. All of the companies we visited continually
           refined and enhanced their development process via some sort of
           lessons-learned process. The program managers themselves placed a
           great deal of value on these processes-as they were seen as the
           primary means for learning how to tailor the process to better fit
           a project and to prevent the same mistakes from recurring.

           Program managers also cited flexibility as an enabling quality of
           their processes. All of the companies allowed their processes to
           be tailored as needed. Milestones that were deemed unnecessary
           could be dropped. More often, however, additional meetings were
           added to gain consensus on how to address particular problems.
           Another enabling factor was that their processes ensured
           decisionmakers were not flooded with data. Often, program teams
           boiled down data into one or two pages, using simple metrics
           depicting status and risk on critical facets of the program such
           as cost, schedule, technology readiness, design readiness, and
           production readiness. Program managers valued the process of
           translating detailed data into higher level metrics because it
           required them to think about their programs in more strategic
           terms and focus on the highest problem areas.

           Motorola's development process is comprised of 16 milestones or
           "gates"-the first five of which pertain to processes employed to
           develop a product concept and the business case. Eleven gates
           comprise the execution of the business case, from project
           initiation, to systems requirements definition, design readiness,
           testing, controlled introduction, and then full deployment. Each
           gate demands an array of indicators on status and progress,
           including resources, cost, scope, risk, and schedule. A
           centralized database helps program managers produce this data and
           allows users to obtain data at any time and at any level of detail
           that they need. For meetings themselves, program managers are
           required to produce a set of "vital few" performance measures
           relating to cost, quality, program status, and customer
           satisfaction. At the gates themselves, program managers discuss
           the status of the program with senior leaders, but they are
           ultimately responsible for making decisions on whether to proceed
           to the next phase. In the past, program managers did not have this
           responsibility and acted more as an administrator than a leader,
           according to senior executives. With less responsibility and
           accountability, programs were not managed as well-often employing
           disjointed management processes with less attention to efficiency
           and effectiveness. By increasing program manager's ownership and
           accountability over the project, senior leaders found that they
           were more incentivized to meet and exceed cost, schedule, and
           performance goals. To support this change, the company also
           adopted common supporting processes, including configuration
           management, design, training, testing, defect prevention, quality
           management, supplier management, and system upgrades. The common
           processes assured program managers employed the same set of
           quality controls and that deployed tools and guidance enabled
           program managers to reduce cycle times as well as to produce
           better and more consistent management data.

           Toyota's process is comprised of eight key milestones-starting
           with a lessons-learned gate. At this point, senior leaders and
           project teams formally review what worked well and not so-well in
           the prior development effort and assess whether the process needs
           to be tailored as a result. The Avalon program manager told us
           that these "reflections" are not taken lightly; they are developed
           through a very detailed and soul-searching process during which
           people have to openly admit errors and inadequacies so that better
           processes and procedures can be devised. The next gate, "image"
           represents the process by which the chief engineer derives the
           business case. Once he is done, direct supervision of the project
           is transferred to a "chief production engineer," who is charged
           with its execution of the business case although the chief
           engineer continues to be involved in the development. The next few
           milestones come as the car is designed, prototyped, tested, and
           put through quality assurance. The last milestone, the production
           stage also contains a customer feedback phase, which is used to
           refine the next development effort.

           Within the business case itself, Toyota places highest importance
           on schedule because a number other vehicle development efforts are
           dependent on the same resources and staff being used by the
           current effort. As a result, the chief production engineer is more
           inclined to make trade-offs that favor schedule over other
           factors. At each milestone meeting, the chief production engineer
           reviews the status of the program with senior leaders, focusing
           first on what problems are occurring and what his solutions are
           for overcoming them. The meeting itself employs streamlined
           reporting with simple indicators of remaining risk on critical
           facets of the program-specifically, a circle, meaning low
           remaining risk and okay to proceed; a triangle, meaning there are
           problems but they can be fixed, or an "x," meaning there is a
           problem without a solution. The chief production engineer is
           responsible for making decisions as to how to proceed at these
           milestones, unless there is a problem that significantly affects
           the business case. If so, senior leaders become more involved in
           the decision-making rather than simply advising the chief
           production engineer.

           While the Toyota process only employs eight formal milestones, the
           chief production engineer actually involves functional experts,
           senior executives, and other stakeholders in frequent meetings to
           make tactical decisions about the program. For example, the Avalon
           chief production engineer told us that he held "obeya" (literally
           "big room," signifying that all inputs are desired) meetings twice
           a week, which involved all functional areas as well as "namewashi"
           (literally binding the roots together, signifying gathering facts
           and moving toward a decision) meetings before a formal milestone
           meeting-at which functional officials consulted with each other to
           identify problems and develop potential solutions that would be
           presented to senior leaders at the milestone. Overall, the
           accountability for meeting the Avalon program's goals was shared
           between the chief production engineer, the functional team, and
           senior executives. At Toyota, senior leaders assume that the
           processes they have in place will work, and if the process is not
           delivering a suitable quality outcome, then it was the shared
           responsibility of managers and staff to resolve the issue. If
           performance issues arose, senior leaders attempted to address them
           first through training, mentoring, and additional support, rather
           than removing the program manager.

           Empowering program managers to make decisions in executing the
           business case was seen as the most significant type of support
           provided by senior leaders. But program managers themselves
           pointed to other types of support that made it easier for them to
           succeed. Primarily, senior leaders did the following:

           o  Provided unwavering commitment to the development effort. At
           all the firms we visited, senior leaders were champions of the
           project throughout its life and fully committed to supporting it.
           When significant problems arose that jeopardized the business
           case, they found ways to address those problems, rather than
           rejecting the program in favor of another one.

           o  Trusted their program managers. Senior leaders trusted the
           information being provided by the program manager as well as his
           or her expertise. This reduced the need for instilling additional
           layers of oversight that could slow down the program. At the same
           time, however, senior leaders took personal responsibility for
           assuring their program managers had the knowledge and capability
           needed to succeed-in some cases, by personally mentoring them for
           a long period of time.

           o  Encouraged program managers to share bad news. Senior leaders
           went out of their way to encourage program managers to share
           problems. In fact, program managers were often expected to discuss
           problems before anything else at key milestones. And, in some
           cases, program managers were evaluated based on their ability to
           identify and share problems. At the same time, senior leaders
           expected their program managers to come up with solutions-to take
           ownership over their efforts.

           o  Encouraged collaboration and communication. Senior leaders
           spent a great deal of time breaking down stovepipes and other
           barriers to sharing information. The Avalon chief production
           engineer, in fact, told us that Toyota's development processes
           alone were much like other automobile manufacturers he had worked
           for. What separated Toyota from the others was its emphasis on
           open information exchange, cooperation, and collaboration. He
           believed that this was the key enabler for Toyota's superior
           systems integration.

           Figure 4: Best Practice Roles, Responsibilities, and Behaviors of
           Senior Managers

           While DOD's leadership has taken action in recent years it hopes
           will better position programs and improve planning and budgeting,
           it is still not effectively positioning or supporting program
           managers for success. For example, rather than making strategic
           investment decisions, DOD starts more programs than it can afford
           and rarely prioritizes them for funding purposes. The result is a
           competition for funds that creates pressures to produce optimistic
           cost and schedule estimates and to overpromise capability.
           Moreover, our work has shown that DOD often starts programs
           without establishing a business case. Specifically, technologies
           are not always mature at start, requirements are not fully
           defined, and cost and schedule estimates are not always realistic.
           In addition, program managers are not empowered to execute
           programs. They cannot veto requirements and they do not control
           funding or other resources. In fact, program managers who
           responded to our survey personally consider requirements and
           funding instability to be their biggest obstacles to success.
           Program managers also believe that they are not sufficiently
           supported once programs begin. In particular, they believe that
           program decisions are based on funding needs of other programs
           rather than demonstrable knowledge; they lack tools needed to
           enable them to provide leadership consistent with cost, schedule
           and performance information; they are not trusted; they are not
           encouraged to share bad news; and they must continually advocate
           for their programs in order to sustain commitment.

Strategic Leadership at Toyota and Siemens Medical

              Knowledge-Based Process Followed to Execute Programs

Knowledge-Based Development at Motorola and Toyota

 Continued Senior Leadership during Product Development Further Enabled Success

Chapter 3: DOD Is Not Supporting Its Program Managers Effectively  Chapter
3: DOD Is Not Supporting Its Program Managers Effectively

Figure 5: Breakdowns in Support and Accountability Factors

       Senior Leadership Does Not Provide a Strong Foundation for Success

According to program managers we interviewed as well as comments to our
survey and our past reviews, senior leadership within DOD does not provide
a strong foundation for success. While DOD is adept at developing
long-term visions and strategic plans, it does not develop realistic,
integrated investment strategies for weapons acquisitions to carry out
these plans. Instead, more programs are started than can be funded and too
many programs must compete for funding, which, in turn, creates incentives
to produce overly optimistic estimates and to overpromise capability.
Moreover, when faced with a lower budget, program managers believe that
senior executives within the Office of the Secretary of Defense (OSD) and
the services would rather make across-the-board cuts to a span of programs
rather than hard decisions as to which ones to keep and which ones to
cancel or cut back. Our work continues to show that, while DOD has adopted
evolutionary development in its policies, programs are being encouraged to
pursue significant leaps in capability. In addition, DOD's policy now
encourages programs to match resources to requirements before program
initiation, but program managers reported in our survey that requirements
and funding are not stabilized and were the biggest obstacles to their
success. Further, while program managers believe their training has been
adequate, they also believe that mentoring has been uneven and that they
could benefit with tours of duty inside the Pentagon, for example, in
offices that oversee budget or financial management. Table 2 highlights
differences between strategic senior leadership support within the
commercial companies we visited and DOD.

Table 2: Are Best Practices Present in DOD?

Best practices            DOD                                       
Develop long-term vision  DOD has long-term vision, but not an      
and investment strategy   investment strategy. Lack of investment   
                             strategy has created competition for      
                             funding and spurred low cost-estimating,  
                             optimistic schedules, and suppression of  
                             bad news.                                 
Adopt evolutionary path   DOD has adopted evolutionary development  
toward meeting customer   in policy but not in practice.            
needs                     
Match requirements and    DOD has encouraged achieving match in     
resources before starting policy but not in practice. Requirements  
new product development   are not stable; funding commitments are   
                             not enforced; key technologies are not    
                             matured before development. Requirements  
                             and funding are biggest obstacles in view 
                             of program managers.                      

Source: GAO.

Investment Strategy and Evolutionary Development

DOD is attempting to address some of the problems identified, but it is
too early to determine how effective its solutions are. For example, it is
implementing a new requirements setting processes-known as the Joint
Capabilities Integration and Development System-in an attempt to bring
more discipline to investment decisions. The system is organized around
key functional concepts and areas, such as command and control, force
application, battlespace awareness, and focused logistics. For each area,
boards of high-ranking military and civilian officials identify long-term
joint needs and make high-level trade-offs on how those needs should be
met. Once specific programs are proposed, the process is designed to
encourage a more evolutionary approach by allowing requirements setters
the flexibility to define requirements in terms of capabilities as well as
to defer final requirements formulation to later in the development
process. DOD has also been attempting to implement complementary planning
and budgeting processes-for example, by asking the military services to
plan budgets around guidance that takes a joint perspective and by taking
a portfolio planning approach. However, there is no evidence to date that
shows these enhancements are providing DOD with a sound investment
strategy as well as the right controls for enforcing that strategy.

While some program managers we spoke with believed the process' focus on
capabilities versus requirements promised more flexibility, program
managers comments to our survey show that they also still widely believed
that they were operating in an environment where there was unfair
competition for funding. Figure 6 highlights specific views.

Figure 6: Highlights of Program Manager Comments Regarding Competition for
Funding

DOD has also adopted policies that encourage evolutionary development.1
However, our reviews continue to find that programs are still pursuing
significant leaps in capabilities. For example, we reported this year2
that the Joint Strike Fighter acquisition strategy was striving to achieve
the ultimate fighter capability within a single product development
increment, and that it had bypassed early opportunities to trade or defer
to later increments those features and capabilities that could not be
readily met. We also testified3 that while DOD's space acquisition policy
states its preference for evolutionary development, programs still attempt
to achieve significant leaps in one step.

1 DOD Directive 5000.1, the Defense Acquisition System (May 2003) and DOD
Instruction 5000.2 Operation of the Defense Acquisition System (May 2003).
The directive establishes evolutionary acquisition strategies as the
preferred approach to satisfying DOD's operational needs. The directive
also requires program managers to provide knowledge about key aspects of a
system at key points in the acquisition process. The instruction
implements the policy and establishes detailed policy for evolutionary
acquisition.

Matching Resources to Requirements

In recent years, DOD has changed its acquisition policy to encourage
decisionmakers to match requirements to resources before starting a new
program. For example, the policy specifically encourages that technologies
be demonstrated in a relevant environment before being included in a
program; that a full funding commitment be made to a program before it is
started and that requirements be informed by the systems engineering
process. Concurrently, DOD's new requirements process is designed to
instill more discipline during initial requirements development and
postpone final determination of requirements to assure that requirements
being set are achievable.

In practice, however, our work has shown that there are still significant
gaps between requirements and technology resources when programs begin.
Our most recent annual assessment of major weapon systems programs,4 for
example, showed that only 15 percent of the programs we reviewed began
development having demonstrated that all of their technologies were
mature. More often than not, programs had to worry about maturing
technologies well into system development, when they should have focused
on maturing system design and preparing for production. These assessments
also show that programs that started development with mature technologies
experienced lower development and unit cost increases than those programs
that started with immature technologies. Table 3 provides some examples.

2 Tactical Aircraft: Opportunity to Reduce Risks in the Joint Strike
Fighter Program with Different Acquisition Strategy,
GAO-05-271(Washington, D.C.: March 15, 2005).

3 Space Acquisitions: Stronger Development Practices and Investment
Planning Needed to Address Continuing Problems, GAO-05-891T, (Washington,
D.C.: July 12, 2005).

4 Defense Acquisitions: Assessments of Selected Major Weapon Programs,
GAO-05-301 (Washington, D.C.: March 31, 2005).

Table 3: Technology Maturity and Program Outcomes

                                                    Percent of critical 
                                  Percent increase     technologies and 
                                     in R&D (first  associated maturity 
                                  full estimate to level at development 
Program                        latest estimate)                start 
Advanced Threat Infrared                                             
Countermeasures/Common Missile                    50 % (3 of 6) at 6 
Warning System                              5.6            or higher
C-5 Reliability Enhancement                      100 % (11 of 11) at 
and Reengining Program                      2.1          6 or higher 
DD(X) Destroyer                                  25 % (3 of 12) at 6 
                                             417.3            or higher 
Future Combat System                             32 % (17of 52) at 6 
                                              50.8            or higher 
Joint Strike Fighter                             25 % (2 of 8) are 6 
                                              30.1            or higher 

Source: GAO.

Note: Technology readiness level of 7 or higher at program launch is
considered best practice; a technology readiness level of 6 or higher is
DOD standard.

Although the majority of respondents to our survey believed that the
initial baselines of their programs were reasonable, a significant group,
about 24 percent of program managers, responded that their program
parameters were not reasonable at the start and 45 program managers
responded that their program had been re-baselined one or more times for
cost and schedule increases; 18 percent said one or more key technologies
fell below a readiness level of 7, which is proven to work in an
operational environment. They also noted that the most frequently missing
critical skill was systems engineering-a key function for matching
requirements to the technologies needed and for providing reasonable
baselines at the beginning of development. In addition, in written
comments and individual interviews, program managers noted pressure to
agree to cost commitments that could be attained only if programs enjoyed
higher-level support. They also noted that requirements were often not
fully defined at the onset of a program, and many also pointed out that
users and stakeholders often did not stick to the agreements they made
when programs were launched, especially if technologies did not mature as
planned.

Figure 7: To What Extent Were the Parameters of Your Program Reasonable at
Program Start?

Figure 8: How Program Managers Responded to an Open-ended Question on What
Were the Biggest Obstacles They Faced

Program managers' views were mixed when it came to whether human capital
resources were well matched to new programs. They cited major improvements
in DOD's training programs and credited cross-functional teams as a
valuable resource. They also generally believed they personally had the
right mix of experience and training to do their jobs well. Ninety-four
percent of the program managers responding to our survey reported that
they had been certified at the highest level for program management by
DOD's Defense Acquisition University. More than 80 percent also believed
they had adequate training in the areas of systems engineering, business
processes, contracting, management, program representation, cost control,
and planning and budgeting. Slightly less, about 76 percent, believed they
had enough leadership training. In addition, about 92 percent said that
they believed that their service consistently assigned people with the
skills and experience to be effective program managers.

At the same time, however, program managers comments and interviews with
program executive officers pointed to critical skill shortages for staff
that support them-including program management, systems engineering, cost
estimating, and software development. Some of these officials attributed
the shortages to shifts in emphasis from oversight to insight of
contractor operations. Lastly, in their written comments, about 18 percent
of program managers who provided written comments cited shortcomings in
their career path, such as lack of opportunities at the general officer
level and requirements to move often as a disincentive; 13 percent cited
the lack of financial incentives. Some program managers also noted that
DOD loses opportunities to retain valuable experience, merely because
there are no formal incentives for military officers to stay on as program
managers after they are eligible for retirement. Civilians in program
management also cited a lack of career opportunities; one problem cited
was having to find their next job in contrast to military program
managers, whose subsequent job is presented to them.

     Execution in DOD Does Not Provide Adequate Support and Accountability

According to program managers and our past reviews, the execution process
does not provide adequate support and accountability. In particular,
knowledge-based development processes are not employed, program managers
are not empowered to execute, and they are not held accountable for
delivering programs within targets.

More specifically, DOD has encouraged following knowledge-based
development processes in its acquisition policy but not always in
practice. The acquisition process itself mirrors many aspects of the
commercial companies. For example, it requires a variety of senior,
functional, and program-level personnel to come together, assess progress,
identify problems, and make go/no-go decisions at key points in
development. It encourages oversight personnel to base these decisions on
quantifiable data and demonstrated knowledge. To enhance product
development, DOD has also been attempting to adopt and improve policies in
areas such as software development and systems engineering.

However, program managers who responded to our survey believe that the
acquisition process does not enable them to succeed because it does not
empower them to make decisions on whether the program is ready to proceed
forward or even to make relatively small trade-offs between resources and
requirements as unexpected problems are encountered. Program managers
assert that they are also not able to make shifts within personnel to
respond to changes affecting the program. At the same time, program
managers commented that requirements continue to be added as the program
progresses and funding instability continues throughout. These two factors
alone cause the greatest disruption to programs, according to program
managers. Compounding this problem is the fact that because acquisition
programs tend to attract funds over other activities, including science
and technology efforts that ultimately support acquisition, program
managers are incentivized to take on tasks that really should be
accomplished within a laboratory environment, where it is easier and
cheaper to discover and address technical problems.

With many factors out of their span of control, program managers in our
focus groups also commented that it was difficult to hold them accountable
for mistakes. In addition, in their written comments to the survey, many
program managers expressed frustration with the time required of them to
answer continual demands for information from oversight officials-many of
which did not seem to add value. Some program managers in fact estimated
that they spent more then 50 percent of their time producing and tailoring
and explaining status information to others.

More broadly, in interviews and written comments, many program managers
and program executive officials said that did not believe that DOD's
acquisition process really supported or enabled them. Instead, they viewed
the process as cumbersome and the information produced as non-strategic.
When strategic plans or useful analyses were produced, they were done so
apart from the acquisition process.

Our own reviews have pointed to a number of structural problems with the
acquisition process.5 In particular, while DOD's acquisition policy has
embraced best practice criteria for making decisions, it does not yet have
the necessary controls to ensure knowledge is used for decision-making
purposes. As a result, programs can move forward into design, integration,
and production phases without demonstrating that they are ready to.
Without a means to ensure programs and senior managers are adhering to the
process, the process itself can become an empty exercise-and, in the view
of program managers, a time-consuming one.

Table 4 highlights differences between DOD and commercial knowledge-based
development support and accountability factors-collectively from the
perspective of program managers, our past reports, and observations we
made during the course of the review.

5 Defense Acquisitions: DOD's Revised Policy Emphasizes Best Practices but
More Controls Are Needed, GAO-04-53 (Washington, D.C.: Nov. 17, 2003).

Table 4: Are Best Practices Present in DOD?

Best practices           DOD                                         
Base decisions on        DOD policy encourages decisions to be based 
quantifiable data and    on quantifiable data and demonstrated       
demonstrated knowledge   knowledge, but not happening in practice.   
Empower program managers Program managers say they are not empowered 
to make decisions        in the same way as commercial companies.    
                            They do not control resources. They do not  
                            have authority to move programs to next     
                            phases.                                     
Hold program managers    Difficult to enforce accountability.        
accountable              
Program managers stay    Tenure has been lengthened, but program     
through execution        managers generally do not stay after 3 to 4 
                            years.                                      

Source: GAO.

Data Supporting Oversight and Management Decisions

We reported that while DOD's acquisition policy has embraced best practice
criteria for making decisions, it does not yet have the necessary controls
to ensure demonstrable data is used for decision-making purposes. We
recommended that DOD assure that program launch decisions capture
knowledge about cost and schedule estimates based on analysis from a
preliminary design using systems engineering tools. In transitioning from
system integration to system demonstration, we recommended that DOD ensure
the capture of knowledge about the completion of engineering drawings;
completion of subsystem and system design reviews; agreement from all
stakeholders that the drawings are complete; and identification of
critical manufacturing processes, among other indicators. And in
transition to production, we recommended that DOD capture knowledge about
production and availability of representative prototypes along with
statistical process control data.

We recommended adopting these controls because, in our view, they would
help set program managers up for success by (1) empowering them with
demonstrated knowledge as they move toward production and (2) bringing
accountability to their positions and making the business case more
understandable. Without these types of controls, the process can become an
empty and time-consuming exercise in the view of program managers. At
present, our reports continue to show that programs are allowed to proceed
without really showing that they are ready to. In our most recent annual
assessment of major weapon systems, for example, only 42 percent of
programs had achieved design stability at design review and almost none of
the programs in production or nearing production planned to assure
production reliability through statistical control of key processes.

Our survey also indicated that a relatively small percentage of programs
used knowledge indicators that successful commercial companies use. For
example, in responding to our survey, only 32 percent of program managers
said they used design drawing completion extensively to measure design
maturity; only 26 percent said they used production process controls to a
great extent. Even fewer program managers reported that their immediate
supervisor used these measures extensively to evaluate progress.

In our survey and interviews, program managers and program executive
officers also frequently commented that they spend too much time preparing
data for oversight purposes that is not strategic or very useful to them.
In fact, more than 90 percent of survey respondents said that they spent
either a moderate, great, or very great extent of their time representing
their program to outsiders and developing and generating information about
program progress. In addition, program managers told us that they do not
have standard tools for preparing program-status data. Instead, they must
hand-tailor data to the requester's particular demands for format and
level of detail. The Air Force was cited by some program managers as
taking initiative in developing a database (known as the System Management
Analysis Reporting Tool) that could save time in answering internal
oversight demands for data, but they also wanted to be able to use such a
tool to answer outside demands. While program managers said they were
spending a great deal of time reporting on program status to outsiders,
some program executive officers and program managers also commented that
they had to separately produce data, analyses, and strategic plans for
their own purposes in order to keep their programs on track-the types of
plans and analyses that they used were simply not called for by the
process itself. One program executive officer said that he used three
documents, the approved program baseline, the acquisition strategy, and
the test plan to evaluate the program manager's plans-all of these
documents and many more are required under current acquisition
planning-but these were of most significance. In addition the executive
officer held a one-day review per quarter with each program manager and
reviewed metrics such as earned value, use of award fee, contract growth,
and schedule variation.

Program Manager Authority

In several key areas, program managers said that they do not have the
necessary authority to overcome obstacles and make trade-offs to achieve
program goals. About 60 percent of the program managers that responded to
our survey said that program managers should have more authority to manage
their programs-particularly when it comes to funding, deciding when
programs are ready to proceed to the next phase, and shifting staff. In
interviews and written comments, some program managers commented that they
were seeking the ability to make relatively small trade-offs-for example,
moving a staff member from one section of a program to another and
shifting a small amount of funds from procurement accounts to research and
development accounts, while others advocated for greater authority, as
long as their program stayed on track. In addition, program managers often
commented that they should have a larger role in requirements decisions
that are made after a program is started-specifically, the ability to veto
new requirements that would put too much strain on the program. A few
program managers we interviewed, however, believed that they did have
sufficient authority and that many program managers have not learned how
to exercise it or are risk averse. Others commented that program managers
were simply not allowed by senior managers to exercise their authority. At
the same time, program executive officers, who manage a set of programs,
commented in interviews that they also lacked authority over simple
matters such as moving staff or shifting small amounts of funds. Lastly,
in our focus groups and in written comments, program managers who
specifically worked for the Missile Defense Agency indicated that they did
have authority to make trade-offs among cost, schedule, and performance
and to set requirements for the business case. They found that this
authority alone greatly separated their current positions from past
program manager positions and consistently cited it as a major enabler.

Table 5 shows how program managers answered survey questions regarding the
types of formal and information authority they have. Figure 9 highlights
comments that were provided by program managers.

Table 5: Program Manager Views on Formal vs. Informal Authority

In percent                                              
                             I have formal I have informal              
Type of authority            authoritya       authority No authority
Developing program                                                   
requirements                         10              82            7
Changes in program                                                   
requirements                         13              85            2
Flexibility within                                                   
program to reallocate                                   
funding                              81              15            5
Developing technology                42              45            9 
Setting testing                                                      
requirements                         48              49            2
Selecting contractor                                                 
sources                              48              33           11
Administering contracts              60              37            3 
Addressing difficulties                                              
to meet requirements                 66              31            2

Source: GAO.

aNote: Numbers may not total 100 percent due to rounding.

Figure 9: Highlights of Program Manager Comments on What Types of
Authority They Need

Accountability

Program manager views with regard to accountability are mixed. In our
interviews and our focus groups, many program managers stated they
personally held themselves accountable; however, many also commented that
it is difficult to be accountable when so much is outside their span of
control. During our focus groups, program managers cited sporadic
instances when program managers were removed from their positions or
forced to retire. They also cited instances when a program manager was
promoted, even though the program was experiencing difficulties. In their
written comments for our survey, program managers often commented that it
was a disincentive that senior leaders who were impacting their program
negatively were not being held accountable.

We observed some key differences between the commercial companies we
visited and DOD when it comes to accountability.

           o  Commercial companies make it very clear who is accountable on a
           program and for what. Goals that must be achieved are clearly
           spelled out and understood by the entire program team. In DOD, it
           is not always clear who is responsible. Moreover, the expectations
           set for program managers by their supervisors may not necessarily
           match up with the goals of their program-particularly when the
           program manager is a military officer who reports to both a PEO
           and another commanding official.

           o  Program managers and senior managers in the commercial sector
           are required to stay with programs until they are done; at DOD
           they are not.

           o  Program managers in the commercial sector are incentivized to
           stay with programs and be accountable for them-principally through
           empowerment and financial incentives, but also through their
           desire to help the company achieve its goals. At DOD, program
           managers strongly asserted that they are incentivized to help the
           warfighter, but few said they were incentivized by financial or
           promotion incentives or by empowerment.

           In commenting on senior leader support during program execution,
           program managers had mixed views on whether they received
           sustained commitment from their program executive officers, but
           widely believed that they did not receive sustained commitment
           from other senior leaders and stakeholders-unless their programs
           enjoyed priority and support from very high level officials,
           Congress, or the President. More often than not, programs
           struggled to compete for funding and were continually beset by
           changing demands from users. Others noted that while DOD is
           emphasizing jointness in programs more and more, collaboration
           among senior leaders needed to achieve jointness is not always
           happening. Some program managers lamented that they felt they were
           not respected in DOD, while others believed their service was
           taking some positive actions to put program managers on a par with
           military officers in operational positions.

           Program managers were also troubled by constant demands for
           information for oversight purposes as well as interruptions from
           stakeholders (for example, in department-wide budget or testing
           offices) that seemed to be non value-added. As we noted earlier,
           over 90 percent of the survey respondents said that they spent
           either a moderate, great, or very great extent of their time
           representing their program to outsiders and developing and
           generating information about program progress.

           Several program managers also cited reluctance on the part of
           senior managers to hear bad news. Our past reviews have similarly
           noted that the overall competition for funding in DOD spurs
           program managers to suppress bad news because it can result in
           funding cuts.

           Differences between DOD and leading companies in how program
           managers are supported and held accountable are rooted in
           differences in incentives and resulting behaviors. This begins
           with the definition of success. The commercial firms we studied
           concluded their survival hinged on their ability to increase their
           market share, which, in turn, meant developing higher-quality
           products, at the lowest possible price, and delivering them in a
           timely fashion-preferably before their competitors could do the
           same. This imperative meant that they had no choice but to narrow
           the gap between requirements and resources in a manner that not
           only ensured they met their market targets, but did so in a manner
           that consumed resources efficiently. It also meant that they had
           no choice but to fully support the development effort, to instill
           strategic planning and prioritization, to work collaboratively, to
           follow a knowledge-based process that makes product development
           manageable, and ultimately, assign accountability to all involved
           for success or failure. In theory, DOD's success likewise hinges
           on its ability to deliver high quality weapons to the warfighter
           in a timely fashion. But in practice, the implied definition of
           success is the ability of a program to win support and attract
           funds. Of course, there are reasons for this disconnect. Corporate
           revenue is generated by customer sales while DOD's funding is
           dependent on annual appropriations. Corporations go out of
           business when their product development efforts do not succeed;
           DOD does not. Selling products to customers is the single focus of
           a private-sector company while DOD is charged with a myriad of
           important missions-each of which also competes for budget share.
           As a result, program managers are incentivized to overpromise on
           performance because it makes their program stand out from others.
           They are incentivized to underestimate cost and schedule and to
           suppress bad news because doing otherwise could result in a loss
           of support and funding and further damage their program. In short,
           unknowns become acceptable and desirable rather than unacceptable
           as they are in the corporate environment. And accountability
           becomes much more difficult to define.

           Figure 10: Key Differences in Definition of Success and Resulting
           Behaviors

           Success for the commercial world is straightforward and simple:
           maximize profit. In turn, this means selling products to customers
           at the right price, right time, and right cost. Each of the
           commercial companies we visited enjoyed success to this end, but
           at some point in time, as competitors made gains, markets
           tightened, and the pace of technology changes grew faster, they
           realized they needed to do more to be successful. Toyota decided
           it needed to expand its role in the world market place and this
           need persisted as competition grew stronger over the years. For
           Siemens this realization came in the 1990s-when Siemens Medical
           Division took a hard look at its profitability for its medical
           devices and for Motorola in the 1980's when it began losing market
           share for its communication devices. To turn themselves around,
           all three companies chose not to depend on technology
           breakthroughs or exotic marketing, but rather to improve their
           position by looking inward at how they approached development.
           Each found that there was room for improvement, starting with
           corporate cultures and ending with processes and controls. In
           Toyota's case, emphasis was largely placed on collaboration and
           consistency. In Siemens case, emphasis was placed on quality,
           particularly because its medical products come under extensive
           Food and Drug Administration regulations. For Motorola, emphasis
           was placed on empowerment and commonality, particularly in the
           processes that support product development like software
           development.

           At DOD, success is often formally defined in similar terms as the
           commercial world: deliver high quality products to customers (the
           warfighter) at the right time and the right cost. Virtually all
           program managers we spoke with first defined success in terms of
           enabling warfighters and doing so in a timely and cost-efficient
           manner. But when the point was pursued further, it became clear
           that the implied definition for success in DOD is attracting funds
           for new programs, and keeping funds for ongoing programs. Program
           managers themselves say they spend enormous amounts of time
           retaining support for their efforts and that their focus is
           largely on keeping funds stable. They also observe that DOD starts
           more programs than it can afford to begin with, which merely sets
           the stage for competition and resulting behaviors. As noted
           earlier, there are factors that contribute to how success is
           defined in practice, including the fact that DOD depends on annual
           appropriations and it must fund a wide variety of missions beyond
           weapon systems development. However, according to program
           managers, the willingness to make trade-off decisions alone, would
           help DOD mitigate these circumstances.

           Regardless of where they placed greatest emphasis, each company we
           studied adopted processes and support mechanisms that emphasized
           the following:

           o  risk reduction,
           o  knowledge-based decisionmaking,
           o  discipline,
           o  collaboration,
           o  trust,
           o  commitment,
           o  consistency,
           o  realism, and
           o  accountability.

           Such characteristics were seen as absolutely essential to gaining
           strength in the market place. With limited opportunities to invest
           in new product development efforts, companies understand it is
           essential, for example, that they know they are pursuing efforts
           that will optimize profits. Therefore, estimates of costs and
           technology maturity must be accurate and they must be used for
           making decisions. Consistency and discipline are integral to
           assuring that successful efforts can be repeated. Ultimately,
           these characteristics translate into processes that help companies
           develop products quicker, cheaper, and better. At the strategic
           level, processes include accurate, strategic planning and
           prioritization to ensure the right mix of products are pursued;
           investment strategies that prioritize projects for funding; and
           strong systems engineering to help them establish a realistic
           business case that levels market needs with available resources
           prior to beginning a product development. At the tactical level,
           this includes knowledge-based developments that center on
           designing and manufacturing products that will sell well enough to
           make an acceptable profit. This combination of focused leadership
           and disciplined processes promotes positive behaviors, such as an
           insistence that technology development take place separately from
           product development programs and trade-offs between requirements
           and resources be made before beginning a program; it promotes an
           atmosphere of early candor and openness from everyone as to
           potential program risks; and underscores the need for realistic,
           knowledge-based cost and schedule estimates to support full
           funding decisions; and the ability to test early, allowing "red
           lights" for problems that must be proven solved before they can be
           changed to "green lights."

           Once attracting and sustaining funds becomes a part of the
           definition of success, as it is at DOD, different values and
           behaviors emerge. For example, it is not necessarily in a program
           manager's interest to develop accurate estimates of cost,
           schedule, and technology readiness, because honest assessments
           could result in lost funding. Delayed testing becomes preferred
           over early testing since that will keep "bad news" at bay.

           Ultimately, no matter how well-intentioned or what improvements
           are made, DOD's processes and support mechanisms eventually play
           into funding competition. On paper, the requirements process may
           emphasize realism and the importance of incremental development,
           but in practice, it consistently encourages programs to promise
           performance features that significantly distinguish them from
           other systems. Likewise, changes may be made to make the funding
           process more strategic, but because there are still many programs
           competing for funds, it encourages cost and schedule estimates to
           be comparatively soft with little benefit from systems engineering
           tradeoffs. By favoring acquisition programs over science and
           technology efforts, the funding process also encourages programs
           to take on technology development that should be carried out in
           research labs. Lastly, the acquisition process may adopt
           world-class criteria for making decisions, but because it is much
           easier to attract funds for a formal weapons program than funds
           for the exercise of gaining knowledge about technologies, the
           process encourages programs to move forward despite risks with the
           assumption that programs can resolve technical, design, or
           production "glitches" later on. Significant unknowns are accepted
           in this environment. Delivering a product late and over cost does
           not necessarily threaten program success. The cumulative effect of
           these pressures is unpredictable cost and schedule estimates at
           the outset of a program that are breached, sometimes very
           significantly, by the time the weapon system is fielded.

           There are other environmental differences that put additional
           pressures on program managers within DOD. They include layers of
           internal and external oversight that come with DOD's stewardship
           responsibilities, personnel rules that make it more difficult to
           manage human capital and hold people accountable, laws and
           regulations that place additional constraints on an acquisition,
           and the mere size and scope of DOD, which adds significant
           challenges to communicating and collaborating effectively.

           For example, as shown below, commercial companies we visited
           tended to have fairly streamlined oversight. No matter what level
           the program manager resided, they had access to top executives who
           were empowered to help them make go/no-go decisions. In addition
           to this structure, the companies were governed by a degree of
           oversight from shareholders, but this was not actualized in
           day-to-day management of program development activities. At DOD,
           by contrast, program managers operate under many layers of
           oversight-both internally and externally. These include Congress,
           the President, the Secretary of Defense, a myriad of functional
           agencies as well as the military services-all of whom have a say
           in DOD's overall budget as well as funding for specific programs.
           Moreover, within these confines, leaders at all levels shift
           frequently. Much of this oversight is necessary for carrying out
           stewardship responsibilities for public money, but studies
           conducted by a variety of commissions assessing acquisition
           problems through the years have consistently found that there are
           opportunities to reduce oversight layers and streamline oversight
           processes and protect programs from frequent leadership changes.
           Program managers themselves understood the need for oversight, but
           found that responding to oversight demands was taking too much of
           their time. They also identified opportunities to make it easier
           to work within this environment, including development of
           databases to support internal and external oversight requests,
           empowering program managers for more day-to-day decisions, and
           making stakeholders more accountable.

                     Senior Leader Support during Execution

Chapter 4: Basic Incentives Drive Differences in How Program Managers Are
Supported and Held Accountable  Chapter 4: Basic Incentives Drive
Differences in How Program Managers Are Supported and Held Accountable

                             Definition of Success

                               Means for Success

       Other Differences Put Additional Pressures on DOD Program Managers

Figure 11: Commercial vs. DOD Oversight Environments

Program managers also cited several trends that have increased pressures
that they face. These include DOD's movement toward developing more
technical complex families of weapon systems as one package (system of
systems), which they believe vastly increases management challenges and
makes it more difficult to oversee contractors and DOD's reduction in
acquisition workforces over the past decade, which has made it more
difficult to carry out day-to-day responsibilities and retain technical
and business expertise. Overall, however, program managers themselves
consistently attribute their problems to competition for funding over
these other factors.

Chapter 5: Conclusions and Recommendations  Chapter 5: Conclusions and
Recommendations

Like the commercial world, DOD has a mandate to deliver high-quality
products to its customers, at the right time, and the right price. Quality
and time are especially critical to maintain DOD's superiority over
others, to counter quickly changing threats, and to better protect and
enable the warfighter. Cost is critical given DOD's stewardship over
taxpayer money, long-term budget forecasts which indicate that the nation
will not be able to sustain its current level of investment in weapon
systems, and DOD's desire to dramatically transform the way it conducts
military operations. At this time, however, DOD is not positioned to
deliver high quality products in a timely and cost-efficient fashion. It
is not unusual to see cost increases that add up to tens of millions of
dollars, schedule delays that add up to years, and large and expensive
programs to be continually rebaselined. Recognizing this dilemma, DOD has
tried to embrace best practices in its policies, instill more discipline
in requirements setting, strengthen training for program managers and has
lengthened program manager tenures. It has also reorganized offices that
support and oversee programs, required programs to use independent cost
estimates and systems engineering, and it has alternately relaxed and
strengthened oversight over contractors in an effort to extract better
performance from them. Yet despite these and many other actions, programs
are still running over cost and over schedule and, in some cases, changes
have merely added tasks for program managers while adding no value.

Our work shows that this condition will likely persist until DOD provides
a better foundation on which program managers can launch programs and more
consistent and steadfast support once it commits to programs. At the core
of this solution is developing and enforcing an investment strategy that
prioritizes programs based on customer needs and DOD's long term vision
and reduces the burden of advocacy on the part of the program manager. DOD
will always be facing funding uncertainties due to the environment it
operates in. But it has an opportunity to greatly mitigate the risks that
come with this environment by separating long-term wants from needs,
matching them up against what technologies are available today, tomorrow,
and decades from now, as well as being realistic in determining what
resources can be counted on. Without an investment strategy, all other
improvements will likely succumb to the negative incentives and behaviors
that come with continual competition for funding. With an investment
strategy, senior leaders will be better positioned to formally commit to a
business case that assures new programs fit in with priorities, that they
begin with adequate knowledge about technology, time, and cost, and that
they will follow a knowledge-based approach as they move into design and
production. Another core enabler for improving program managers' chances
for success lies in leadership's ability to implement evolutionary,
incremental acquisition programs that have limited cycle times from
beginning to delivery of the weapon system. This allows DOD to align
program managers' tenures to delivery dates, thereby substantially
increasing accountability for successful outcomes.

Once senior leaders do their part-by providing program managers with an
executable business case and committing their full support to a
program-they can begin to empower program managers more and hold them
accountable. By embracing a model for supporting program managers that
incorporates all these elements, DOD can achieve the same outcomes for its
weapons programs as other high-performing organizations.

                      Recommendations for Executive Action

We recommend that the Secretary of Defense take the following actions to
ensure program managers are well positioned to successfully execute and be
held accountable for weapon acquisitions:

           o  DOD should develop an investment strategy that, at a minimum,

                        o  determines the priority order of needed
                        capabilities with a corollary assessment of the
                        resources, that is dollars, technologies, time and
                        people needed to achieve these capabilities. The
                        remaining capabilities should be set out separately
                        as desirable, resources permitting.
                        o  lays out incremental product development programs
                        for achieving desired capabilities, and
                        o  establishes controls to ensure that requirements,
                        funding, and acquisition processes will work together
                        so that DOD will sustain its commitment to its
                        priority programs.

           As DOD works to develop the strategy, it should take an interim
           step by identifying priorities for programs that are already past
           milestone B (the formal start of development). Once the strategy
           is complete, it should be used by the Office of the Secretary of
           Defense to prepare and assess annual budget proposals as well as
           to balance funding between science and technology efforts and
           acquisition efforts to ensure that robust technology development
           efforts are conducted, but outside the acquisition program until
           reaching maturity.

           o  For each new major weapons program, require that senior-level
           officials from the requirements, science and technology, program
           management, testing communities as well as the Office of the
           Comptroller formally commit to a business case prior to approving
           a program at milestone B. At a minimum, the business case should
           demonstrate that

                        o  a requirement exists that warrants a materiel
                        solution consistent with national military strategy,
                        o  an independent analysis of alternatives has been
                        conducted
                        o  the developer has the requisite technical
                        knowledge to meet the requirement,
                        o  the developer has a knowledge-based product
                        development and production plan that will attain high
                        levels of design and production maturity,
                        o  reasonable estimates have been developed to
                        execute the product development and production plan,
                        and
                        o  funding is available to execute the plan.

           o  Develop and implement a process to instill and sustain
           accountability for successful program outcomes. At a minimum, this
           should consider

           In commenting on a draft of our report, DOD's Acting Director for
           Procurement and Acquisition Policy concurred with our
           recommendations. In doing so, DOD asserted it was already taking
           actions to address our recommendations, notably by reviewing its
           overall approach to acquisition governance and investment
           decisionmaking as part of its Quadrennial Defense Review due in
           February 2006 and identifying ways to more effectively implement
           existing policies. DOD also stated it intended to develop a plan
           to address challenges in acquisition manpower including program
           manager tenure and career path and it intends to enhance its
           information management systems to facilitate oversight and
           management decisions. As underscored in our report, DOD has
           attempted similar efforts in the past-that is, reviewed its
           approach to governance and investment decisions and
           policies-without achieving significant improvements. This is
           because DOD has not assured such actions were executed in tandem
           with (1) instilling more leadership and discipline in investment
           decisionmaking, in both the short and long term and (2) instilling
           accountability-by requiring key senior officials to sign a
           business case, based on systems engineering knowledge, prior to
           every new acquisition as well as by matching program managers'
           tenure to cycle time. Therefore, in pursuing the actions it
           identifies in its response to our report, we believe that DOD
           should address the important questions of who should be held
           accountable for acquisition decisions; how much deviation from the
           original business case is allowed before it is no longer
           considered valid and the investment reconsidered; and what is the
           penalty when investments do not result in meeting promised
           warfighter needs.

           The full text of the department's response is in appendix I.

           Michael J. Sullivan (202) 512-4841 or [email protected]

           Greg Campbell, Cristina Chaplain, Ron La Due Lake, Sigrid McGinty,
           Jean McEwen, Carol Mebane, Guisseli Reyes, Lesley Rinner, Lisa
           Simon, Bradley Trainor, and Michele Williamson.

           DOD Acquisition Outcomes: A Case for Change. GAO-06-257T.
           Washington, D.C.: November 15, 2005.

           Defense Acquisitions: Stronger Management Practices Are Needed to
           Improve DOD's Software-Intensive Weapon Acquisitions. GAO-04-393.
           Washington, D.C.: March 1, 2004.

           Best Practices: Setting Requirements Differently Could Reduce
           Weapon Systems' Total Ownership Costs. GAO-03-57. Washington,
           D.C.: February 11, 2003

           Best Practices: Capturing Design and Manufacturing Knowledge Early
           Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July
           15, 2002.

           Defense Acquisitions: DOD Faces Challenges in Implementing Best
           Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.

           Best Practices: Better Matching of Needs and Resources Will Lead
           to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.:
           March 8, 2001.

           Best Practices: A More Constructive Test Approach Is Key to Better
           Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July
           31, 2000.

           Defense Acquisition: Employing Best Practices Can Shape Better
           Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.:
           April 26, 2000.

           Best Practices: DOD Training Can Do More to Help Weapon System
           Programs Implement Best Practices. GAO/NSIAD-99-206. Washington,
           D.C.: August16, 1999.

           Best Practices: Better Management of Technology Development Can
           Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington,
           D.C.: July 30, 1999.

           Defense Acquisitions: Best Commercial Practices Can Improve
           Program Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17,
           1999.

           Defense Acquisition: Improved Program Outcomes Are Possible.
           GAO/T-NSIAD-98-123. Washington, D.C.: March 17, 1998.

           Best Practices: DOD Can Help Suppliers Contribute More to Weapon
           System Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17,
           1998.

           Best Practices: Successful Application to Weapon Acquisition
           Requires Changes in DOD's Environment. GAO/NSIAD-98-56.
           Washington, D.C.: February 24, 1998.

           Best Practices: Commercial Quality Assurance Practices Offer
           Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August
           26, 1996.

           The Government Accountability Office, the audit, evaluation and
           investigative arm of Congress, exists to support Congress in
           meeting its constitutional responsibilities and to help improve
           the performance and accountability of the federal government for
           the American people. GAO examines the use of public funds;
           evaluates federal programs and policies; and provides analyses,
           recommendations, and other assistance to help Congress make
           informed oversight, policy, and funding decisions. GAO's
           commitment to good government is reflected in its core values of
           accountability, integrity, and reliability.

           The fastest and easiest way to obtain copies of GAO documents at
           no cost is through GAO's Web site (www.gao.gov). Each weekday, GAO
           posts newly released reports, testimony, and correspondence on its
           Web site. To have GAO e-mail you a list of newly posted products
           every afternoon, go to www.gao.gov and select "Subscribe to
           Updates."

           The first copy of each printed report is free. Additional copies
           are $2 each. A check or money order should be made out to the
           Superintendent of Documents. GAO also accepts VISA and Mastercard.
           Orders for 100 or more copies mailed to a single address are
           discounted 25 percent. Orders should be sent to:

           U.S. Government Accountability Office 441 G Street NW, Room LM
           Washington, D.C. 20548

           To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
           (202) 512-6061

           Contact:

           Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
           [email protected] Automated answering system: (800) 424-5454 or
           (202) 512-7470

           Gloria Jarmon, Managing Director, [email protected] (202) 512-4400
           U.S. Government Accountability Office, 441 G Street NW, Room 7125
           Washington, D.C. 20548

           Paul Anderson, Managing Director, [email protected] (202)
           512-4800 U.S. Government Accountability Office, 441 G Street NW,
           Room 7149 Washington, D.C. 20548

                        o  matching program manager tenure with delivery of a
                        product or for system design and demonstration,
                        o  tailoring career paths and performance management
                        systems to incentivize longer tenures,
                        o  empowering program managers to execute their
                        programs, including an examination of whether and how
                        much additional authority can be provided over
                        funding, staffing, and approving requirements
                        proposed after milestone B,
                        o  developing and providing automated tools to
                        enhance management and oversight as well as to reduce
                        time required to prepare status information.

                       Agency Comments and Our Evaluation

Appendix I: Comments from the Department of Defense  Appendix I: Comments
from the Department of Defense

A  Appendix II: GAO Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

Related GAO Products  Related GAO Products

(120320)

GAO's Mission

Obtaining Copies of GAO Reports and Testimony

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Congressional Relations

Public Affairs

www.gao.gov/cgi-bin/getrpt?GAO-06-110
www.gao.gov/cgi-bin/getrpt?GAO-06-112SP

To view the full product, including the scope

and methodology, click on the links above.

For more information, contact Michael J. Sullivan at (202) 512-4841 or
[email protected].

Highlights of GAO-06-110, a report to the Subcommittee on Readiness and
Management Support, Committee on Armed Services, U.S. Senate

November 2005

BEST PRACTICES

Better Support of Weapon System Program Managers Needed to Improve
Outcomes

The Department of Defense (DOD) relies on a relatively small cadre of
officials to develop and deliver weapon systems. In view of the importance
of DOD's investment in weapon systems, we have undertaken an extensive
body of work that examines DOD's acquisition issues from a perspective
that draws lessons learned from the best commercial product development
efforts to see if they apply to weapon system acquisitions. In response to
a request from the Chairman and Ranking Minority Member of the
Subcommittee on Readiness and Management Support, Senate Committee on
Armed Services, this report assesses (1) how successful commercial
companies position their program managers, (2) how DOD positions its
program managers, and (3) underlying reasons for the differences.

In compiling this report, GAO conducted a survey of program managers. See
GAO-06-112SP.

What GAO Recommends

GAO recommends the Secretary of Defense develop an investment strategy to
prioritize needed capabilities; require senior stakeholders to formally
commit to business cases for new weapon system developments; and develop a
process to instill and sustain accountability for successful program
outcomes. DOD agreed with our recommendations.

U.S. weapons are among the best in the world, but the programs to acquire
them often take significantly longer and cost more money than promised and
often deliver fewer quantities and capabilities than planned. It is not
unusual for estimates of time and money to be off by 20 to 50 percent.
When costs and schedules increase, quantities are cut, and the value for
the warfighter-as well as the value of the investment dollar-is reduced.

When we examined private sector companies that developed complex and
technical products similar to DOD, we found that their success hinged on
the tone set by leadership and disciplined, knowledge-based processes for
product development and execution. More specifically, long before the
initiation of a new program, senior company leaders made critical
investment decisions about the firm's mix of products so that they could
commit to programs they determined best fit within their overall goals.
These decisions considered long-term needs versus wants as well as
affordability and sustainability. Once high level investment decisions
were made, senior leaders ensured that programs did not begin unless they
had a business case that made sure resources were in-hand to execute the
program-that is, time, technology, money, and people. Once a business case
was established, senior leaders tasked program managers with executing
that business case for each new product from initiation to delivery, but
required their program managers to use a knowledge-based product
development process that demanded appropriate demonstrations of
technology, designs, and processes at critical junctures. The program
manager was empowered to execute the business case, but also held
accountable for delivering the right product at the right time for the
right cost. Requiring the program manager to stay throughout the length of
a project was a principal means of enforcing accountability. Overall, by
providing the right foundation and support for program managers, the
companies we visited were able to consistently deliver quality products
within targets, and in turn, transform themselves into highly competitive
organizations.

DOD program managers are put in a very different situation. DOD leadership
rarely separates long-term wants from needs based on credible, future
threats. As a result, DOD starts many more programs than it can
afford--creating a competition for funds that pressures program managers
to produce optimistic cost estimates and to overpromise capabilities.
Moreover, our work has shown that DOD allows programs to begin without
establishing a formal business case. And once they begin, requirements and
funding change over time. In fact, program managers personally consider
requirements and funding instability-which occur throughout the program-to
be their biggest obstacles to success. Program managers also believe that
they are not sufficiently empowered to execute their programs, and that
because much remains outside of their span of control, they cannot be held
accountable.
*** End of document. ***