    Best Practices: Better Support of Weapon System Program Managers
    Needed to Improve Outcomes (30-NOV-05, GAO-06-110).

    The Department of Defense (DOD) relies on a relatively small
    cadre of officials to develop and deliver weapon systems. In view
    of the importance of DOD's investment in weapon systems, we have
    undertaken an extensive body of work that examines DOD's
    acquisition issues from a perspective that draws lessons learned
    from the best commercial product development efforts to see if
    they apply to weapon system acquisitions. In response to a
    request from the Chairman and Ranking Minority Member of the
    Subcommittee on Readiness and Management Support, Senate
    Committee on Armed Services, this report assesses (1) how
    successful commercial companies position their program managers,
    (2) how DOD positions its program managers, and (3) underlying
    reasons for the differences. In compiling this report, GAO
    conducted a survey of program managers. See GAO-06-112SP.
    -------------------------Indexing Terms-------------------------
    REPORTNUM:   GAO-06-110
        ACCNO:   A42358
      TITLE:     Best Practices: Better Support of Weapon System Program
    Managers Needed to Improve Outcomes
         DATE:   11/30/2005
      SUBJECT:   Best practices
    	     Best practices reviews
    	     Defense capabilities
    	     Defense procurement
    	     Lessons learned
    	     Policy evaluation
    	     Private sector
    	     Private sector practices
    	     Procurement practices
    	     Program management
    	     Strategic planning
    	     Weapons research and development
    	     Weapons systems
    	     DOD Airborne Laser Program
    	     F-22 Raptor Aircraft
    	     F/A-22 Aircraft
    	     Joint Strike Fighter
    	     Space-Based Infrared System

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO Product.                                                 **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ******************************************************************
    GAO-06-110

    Report to the Subcommittee on Readiness and Management Support, Committee
    on Armed Services, U.S. Senate

    United States Government Accountability Office

    GAO

    November 2005

    BEST PRACTICES

    Better Support of Weapon System Program Managers Needed to Improve
    Outcomes

    Best Practices Best Practices Best Practices Best Practices

    GAO-06-110

    Contents

    Letter 1

    Executive Summary 3
    Purpose 3
    Background 3
    Results in Brief 4
    Best Practice: Corporate Leadership and Disciplined, Knowledge-Based
    Processes Are Critical to Program Manager Success 6
    DOD: Critical Support Factors Are Missing 8
    Differences in Incentives Contribute to Differences in Support for Program
    Managers 10
    Chapter 1 Introduction 12
    Long-Standing Problems Hamper Weapons Systems Acquisitions 13
    DOD Program Managers Are Central Executors of the Acquisition Process 14
    Legislation to Improve Program Manager Proficiency 16
    Objectives, Scope, and Methodology 17
    Chapter 2 Senior Leader Support and Disciplined Knowledge- Based Processes
    Are Critical Enablers for Program Managers 22
    Senior Leadership Provides Program Managers with a Strong Foundation for
    Success 23
    Knowledge-Based Process Followed to Execute Programs 30
    Continued Senior Leadership during Product Development Further Enabled
    Success 35
    Chapter 3 DOD Is Not Supporting Its Program Managers Effectively 37
    Senior Leadership Does Not Provide a Strong Foundation for Success 38
    Execution in DOD Does Not Provide Adequate Support and Accountability 45
    Senior Leader Support during Execution 52
    Chapter 4 Basic Incentives Drive Differences in How Program Managers Are
    Supported and Held Accountable 54
    Definition of Success 55
    Means for Success 56
    Other Differences Put Additional Pressures on DOD Program Managers 58
    Chapter 5 Conclusions and Recommendations 61
    Recommendations for Executive Action 62
    Agency Comments and Our Evaluation 63
    Appendix I Comments from the Department of Defense 65
    Appendix II GAO Staff Acknowledgments 69
    Related GAO Products 70

    Tables

    Table 1: Acquisition Categories 15
    Table 2: Are Best Practices Present in DOD? 39
    Table 3: Technology Maturity and Program Outcomes 42
    Table 4: Are Best Practices Present in DOD? 47
    Table 5: Program Manager Views on Formal vs. Informal Authority 50

    Figures

    Figure 1: Critical Support and Accountability Factors 23
    Figure 2: 2005 Toyota Avalon 28
    Figure 3: Siemens Bi-Plane AXIOM Artis 30
    Figure 4: Best Practice Roles, Responsibilities, and Behaviors of Senior
    Managers 36
    Figure 5: Breakdowns in Support and Accountability Factors 38
    Figure 6: Highlights of Program Manager Comments Regarding Competition for
    Funding 40
    Figure 7: To What Extent Were the Parameters of Your Program Reasonable at
    Program Start? 43
    Figure 8: How Program Managers Responded to an Open-ended Question on What
    Were the Biggest Obstacles They Faced 44
    Figure 9: Highlights of Program Manager Comments on What Types of
    Authority They Need 51
    Figure 10: Key Differences in Definition of Success and Resulting
    Behaviors 55
    Figure 11: Commercial vs. DOD Oversight Environments 59

    Abbreviations

    DAWIA Defense Acquisition Workforce Improvement Act

    DOD Department of Defense

    OSD Office of the Secretary of Defense

    PEO program executive officer

    This is a work of the U.S. government and is not subject to copyright
    protection in the United States. It may be reproduced and distributed in
    its entirety without further permission from GAO. However, because this
    work may contain copyrighted images or other material, permission from the
    copyright holder may be necessary if you wish to reproduce this material
    separately.

    United States Government Accountability Office

    Washington, DC 20548

    November 30, 2005

    The Honorable John Ensign

    Chairman

    The Honorable Daniel K. Akaka

    Ranking Minority Member

    Subcommittee on Readiness and Management Support

    Committee on Armed Services

    United States Senate

    As you requested, this report examines how program managers in the
    Department of Defense are supported and how they are held accountable for
    program outcomes. It compares department polices and practices to those of
    leading commercial companies we visited and discusses actions DOD could
    take to improve the accountability of program managers and provide them
    with timely support as they manage the development of complex systems. We
    make recommendations to the Secretary of Defense to (1) develop an
    investment strategy to prioritize needed capabilities, (2) require, for
    each new program, that senior level stakeholders formally commit to a
    business case for program approval at the start of a new program, and (3)
    implement a process to instill and sustain accountability for successful
    program outcomes.

    We are sending copies of this report to the Secretary of Defense; the
    Secretary of the Army; the Secretary of the Navy; the Secretary of the Air
    Force; the Director, Missile Defense Agency; the Director of the Office of
    Management and Budget; and interested congressional committees. We will
    also make copies available to others upon request. In addition, the report
    will be available at no charge on the GAO Web site at http://www.gao.gov.

    If you have any questions regarding this report, please call me at (202)
    512-4841. Staff acknowledgements are listed in appendix II.

    Michael J. Sullivan

    Director, Acquisition and Sourcing Management

    Exe  Executive Summary

                                        Purpose

    The Department of Defense (DOD) plans to increase its investment in the
    research, development, and procurement of new weapon systems from $144
    billion in fiscal year 2005 to $185 billion in fiscal year 2009. U.S.
    weapons are among the best in the world, but the programs to acquire them
    often take significantly longer and cost more money than promised and
    often deliver fewer quantities and other capabilities than planned. It is
    not unusual for estimates of time and money to be off by 20 to 50 percent.
    When costs and schedules increase, quantities are cut, and the value for
    the warfighter-as well as the value of the investment dollar-is reduced.

    In view of the importance of DOD's investment in weapon systems, we have
    undertaken an extensive body of work that examines DOD's acquisition
    issues from a different, more cross-cutting perspective-one that draws
    lessons learned from the best commercial product development efforts to
    see if they apply to weapon system acquisitions. In response to a request
    from the Chairman and Ranking Minority Member of the Subcommittee on
    Readiness and Management Support, Senate Committee on Armed Services, this
    report assesses (1) how successful commercial companies position their
    program managers, (2) how DOD positions its program managers, and (3)
    underlying reasons for the differences.

                                       Background

    DOD relies on a relatively small cadre of military and civilian
    officials-known as program managers-to lead the development and delivery
    of its weapon systems. The responsibility placed on this group is
    enormous. The systems that program managers are responsible for range from
    highly complex and sophisticated aircraft, missile interceptors,
    submarines, and space-based sensors, to new communication and ground
    control systems that support and interconnect this equipment, to smaller,
    less complex systems that support the warfighter. In these times of
    asymmetric threats and netcentricity, individual weapon system investments
    are getting larger and more complex. The development process itself is
    very challenging as many systems require successful management and
    coordination of a broad array of military service and DOD officials,
    outside suppliers, internal and external oversight entities, as well as
    technical, business, contracting, and management expertise. Moreover, in
    many cases, weapon systems are also expected to incorporate technologies
    that push the state-of-the-art while operating in harsh and even untested
    environments-adding daunting technical challenges to the already existing
    business, management, and logistical challenges. Lastly, GAO has reported
    many of the business processes that support weapons development-strategic
    planning and budgeting, human capital management, infrastructure,
    financial management, information technology, and contracting-are beset
    with pervasive, decades-old management problems, which include outdated
    organizational structures, systems, and processes.1

    Weapon system program managers are the central executors of the
    acquisition process. They are responsible for all aspects of development
    and delivery of a new system and for assuring that systems are high
    quality, affordable, supportable, and effective. In carrying out this
    responsibility, they are also responsible for balancing factors that
    influence cost, schedule, and performance. DOD employs about 729 program
    managers to run its weapons programs. Both military officers and civilians
    serve as program managers, but the majority is from the military. DOD's
    program managers typically report to program executive officers (PEO) who
    are charged with overseeing the execution of a portfolio of related
    systems. PEOs, in turn, typically report to a military service acquisition
    executive, who reports to a service secretary, or for some programs, the
    PEO reports to the Defense Acquisition Executive.

                                    Results in Brief

    Program managers from the leading companies we spoke with believed that
    two critical enablers-(1) support from top leadership and (2) disciplined,
    knowledge-based processes for product development execution-empowered them
    to succeed in delivering new products when needed within cost, quality,
    and performance targets originally set by the company. Long before the
    initiation of a new product development, senior company leaders make
    critical strategic investment decisions about the firm's mix of products
    and the return on investment they may yield. Once high-level investment
    decisions were made, senior leaders ensured that programs did not begin
    unless they had a business case that demonstrated the program was aligned
    with the company's goals and that resources were in-hand to execute the
    program-that is, time, technology, money, and people. Once a business case
    was established, senior leaders tasked program managers with executing
    that business case for each new product from initiation to delivery, but
    required their program managers to use a knowledge-based product
    development process that demanded appropriate demonstrations of
    technology, designs, and processes at critical junctures. The program
    manager was empowered to execute the business case, but also held
    accountable for delivering the right product at the right time for the
    right cost. Throughout execution, company senior leaders supported their
    program managers by encouraging open and honest communication and
    continually assured that the right levels of resources and management
    attention were available for the project.

    1 Defense Management: Key Elements Needed to Successfully Transform DOD
    Business Operations, GAO-05-629T (Washington, D.C.: April 28, 2005) and
    High-Risk Series: An Update, GAO-05-207 (Washington, D.C.: January 2005).

    While DOD has taken action in recent years to better position programs for
    success, it puts its program managers in a very different situation.
    Program managers themselves believe that rather than making strategic
    investment decisions, DOD starts more programs than it can afford and
    rarely prioritizes them for funding purposes. The result is a competition
    for funds that creates pressures to produce optimistic cost and schedule
    estimates and to overpromise capability. Our own work has shown that many
    programs begin without a business case, that is, without adequate
    knowledge about technology, time, and cost and without demonstrating that
    the program itself is the optimal approach for achieving a needed
    capability. Moreover, once programs begin, the program manager is not
    empowered to execute the program. In particular, program managers cannot
    veto new requirements, control funding, or control staff. In fact, program
    managers personally consider requirements and funding instability to be
    their biggest obstacles to success. Program managers also believe that
    they are not sufficiently supported once programs begin. In fact, they
    must continually advocate for their programs in order to sustain support.
    Our past reports also show that programs are incentivized to suppress bad
    news and to continually produce optimistic estimates-largely due to
    continual funding competition.

    Many of these differences can be attributed to how success is defined
    within the commercial and DOD environment. Success for the commercial
    world is straightforward and simple: maximize profit. In turn, this means
    selling products to customers at the right price, the right time, and the
    right cost. With this imperative in hand, companies have no choice but to
    adopt processes and cultures that emphasize basing decisions on knowledge,
    reducing risks prior to undertaking new efforts, producing realistic cost
    and schedule estimates, and assuring consistency and quality pervade all
    efforts. At first glance, DOD's definition of success is very similar:
    deliver capability to the warfighter at the right price, the right time,
    and the right cost. But, for various reasons, it is clear that the implied
    definition for success is to attract funds for new programs and to keep
    funds for ongoing programs. While the annual appropriations process and
    the wide variety of mission demands placed on DOD contribute to this
    condition, DOD has made matters worse by not making hard tradeoff
    decisions to ensure it does not pursue more programs than it can afford.
    Once attracting funds becomes "success," harmful practices emerge. For
    example, it is not in a program manager's interest to develop accurate
    estimates of cost, schedule, and technology readiness, because honest
    assessments could result in lost funding. Delayed testing becomes
    preferred over early testing because that will keep "bad news" at bay. In
    turn, knowing data being reported to them may not be reliable, senior
    leaders believe they cannot trust it and must instill multiple oversight
    mechanisms. Any attempts to improve policy and processes eventually
    succumb to funding competition because no one wants to risk loss of
    support.

    We are making recommendations to DOD to better position program managers
    for success. These recommendations focus on what is needed to be done to
    provide the strategic leadership needed to provide the right foundation
    for starting programs, ensure an executable business case is delivered to
    program managers, and to hold program managers accountable for successful
    outcomes. It is important to note that the success of all of our
    recommendations hinge on DOD's ability to instill more discipline and
    leadership over the investment process. After a review of a draft of this
    report, DOD concurred with our recommendations and provided some
    additional comments. The full text of DOD's comments may be found in
    appendix I.

     Best Practice: Corporate Leadership and Disciplined, Knowledge-Based Processes
                        Are Critical to Program Manager Success

    At all of the companies we visited, support for program managers began
    well before they were assigned to a new product development effort-with
    high-level strategic planning and investment decisions and concerted
    efforts to make sure that any new initiative the company undertook was
    achievable within the time and money and other resources the company had
    available. Technology development and program advocacy were also generally
    kept out of a program manager's domain. Once new efforts got off the
    ground, program managers were empowered to manage resources, encouraged to
    bring up problems and propose solutions, and consult with senior leaders
    without fear of losing their support. At the same time, however, they were
    expected to base their decisions on hard data and to assure the right
    knowledge was in-hand before proceeding into the next phases of
    development. They were also held accountable for their choices, though
    companies generally found that with good pre-program decisions, a good
    launch, a sound, disciplined process for execution, and continued support,
    there was little need to punish or remove their program managers.
    Ultimately, as long as a program manager could deliver the right product
    at the right time for the right cost, he was incentivized to do so without
    interference from above.

    According to commercial program managers we spoke with, the most critical
    support factors included the following:

               o  Investment strategies. Each of the companies we visited
               followed a rigorous process to forecast market needs against
               company resources, economic trends, available technologies, and
               its own strategic vision. These exercises culminated in short- and
               long-term investment strategies that provided program managers
               with confidence that the company was committed to their particular
               program and showed them where the project fit within overall
               corporate goals.

               o  Evolutionary development. All of the companies followed an
               incremental path toward meeting market needs rather than
               attempting to satisfy all needs in a single step. This provided
               program managers with more achievable requirements, which, in
               turn, facilitated shorter cycle times. With shorter cycle times,
               the companies could ensure both program managers and senior
               leaders stayed with programs throughout the duration.

               o  Matching requirements to resources. Once specific product
               concepts were identified, the companies worked vigorously to close
               gaps between requirements/customer needs, and resources-time,
               money, and technology. In effect, this took the investment
               strategy down to a project level, assuring that the program
               manager would be well positioned to execute within cost and
               schedule.

               o  Matching the right people to the program. All of the companies
               we visited took steps to ensure that they assigned the right
               people to the right programs. These included long term efforts to
               train and groom technical staff into program managers, mentoring
               on the part of senior leaders with program management experience,
               handpicking program managers based on their expertise and
               experience, and supporting program managers with teams of highly
               qualified functional and technical experts.

               o  Knowledge-driven development decisions. Once a new product
               development began, program managers and senior leaders used
               quantifiable data and demonstrable knowledge to make go/no-go
               decisions. These covered critical program facets such as cost,
               schedule, technology readiness, design readiness, production
               readiness, relationships with suppliers, etc. Development was not
               allowed to proceed until certain thresholds were met, for example,
               a high percentage of engineering drawings completed or production
               processes under statistical control. Development processes were
               also continually tailored based on lessons learned. Program
               managers themselves placed high value on these requirements, as
               they ensured programs were well positioned to move into subsequent
               phases and were less likely to encounter disruptive problems.

               o  Empowerment. At all the companies we visited, program managers
               were empowered to make decisions as to whether programs were ready
               to move forward and to resolve problems and implement solutions.
               They could redirect available funding, if needed. They could
               change team members. Prior to development, they often had a say in
               what requirements they would be handed.

               o  Accountability. With authority, came accountability. Program
               managers at all of the companies we visited were held accountable
               for their choices. To assure accountability, senior leaders set
               goals that were clear to the entire project team and provided
               incentives for program managers and others to meet those goals.

               o  Tenure. All of the companies we visited required that program
               managers stay on until the end of the program. This was a primary
               means of assuring accountability.

               o  Continued senior leadership. In addition to empowering them,
               program managers credited senior leaders with other vital levels
               of support. Namely, senior leaders' commitment to their programs
               were unwavering, they trusted their program managers, they
               encouraged them to share bad news, and they encouraged
               collaboration and communication. At the end of the day, it was the
               senior leaders' job to anticipate and remove obstacles and provide
               the right levels of support so that the path was cleared for the
               program manager to execute the program.

               At DOD, program managers are not put in a position to deliver a
               product within estimates, nor are they held accountable when there
               are failures to deliver products within estimates. While senior
               leaders work hard to develop a short- and long-term vision for the
               defense of the United States, these visions are rarely translated
               into realistic investment strategies that assure the right mix of
               programs is being pursued. Moreover, while recognized in policy as
               a best practice, DOD does not always make sure that there is a
               business case for new initiatives. Lastly, program managers are
               not empowered to execute programs once they begin or held
               accountable when programs get off track.

               The primary problem, according to many program managers and
               verified by GAO's work, is that DOD starts more programs than it
               can afford and does not prioritize programs for funding. This
               creates an environment where programs must continually compete for
               funding. Before programs are even started, advocates are
               incentivized to underestimate both cost and schedule and
               overpromise capability.

               A second problem is that gaps between resources and requirements
               are not closed before or even during program development. For
               example, we have reported that DOD allows many programs to go
               forward without knowing whether critical technologies-such as
               satellite's main sensor, a fighter aircraft's stealth technology,
               a new tank's networking capability-can work as intended.
               Invariably, when programs start with such unknowns, they spend a
               great deal of time and money later on fixing technical glitches
               while simultaneously trying to get other program aspects on track.
               One reason programs begin with immature technologies is that
               program advocates are rushed to start the acquisition program
               because it assures at least an initial commitment of funding.
               Compounding this problem is the fact that acquisition programs
               tend to attract funds over other activities, including science and
               technology efforts that ultimately support acquisition. As a
               result, program managers are incentivized to take on tasks that
               really should be accomplished within a laboratory environment,
               where it is easier and cheaper to discover and address technical
               problems.

               A third problem is that program managers themselves are not
               empowered to execute their programs. First, they have little
               control over funding and they cannot count on funding to be
               stable. When funding is taken away, program managers often find
               themselves in a negative spiral of funding-related
               problems-particularly because they've already made commitments to
               contractors based on certain anticipated levels of funding.
               Second, they cannot veto new requirements. Faced with long
               development life cycles and promising technology advances, users
               often ask for new or better capabilities as a program proceeds
               forward. Program managers themselves are not always empowered to
               say "no" to demands that may overly stretch their programs, and
               few senior leaders above them have been willing to. In addition,
               program managers have little authority over staffing and the
               ability to shift funds within the program. With so much outside
               their span of control, program managers say that DOD is unable to
               hold them accountable when programs get off track. Another reason
               that it is difficult to hold program managers accountable is that
               their tenure is relatively short. The problems being encountered
               today may well be the result of a poor decision made years ago by
               another program manager.

               DOD has tried to improve its processes and policies to better
               position programs for success. For example, policies embrace the
               concept of closing gaps between requirements and resources before
               launching new programs, and DOD is making changes to requirements
               setting and funding processes in an attempt to strengthen
               investment decisions. At this point, however, program managers do
               not see trade-offs being made in the front-end of product
               development that would ensure DOD could fully commit to their
               programs and allow program managers themselves to focus solely on
               executing their programs. The level of trust, collaboration and
               communication is low, while the level of oversight and second
               guessing is high.

               Differences between how program managers are supported and held
               accountable are rooted in differences in incentives and resulting
               behaviors. This begins with the definition of success. The
               commercial firms we studied concluded their survival hinged on
               their ability to increase their market share, which, in turn,
               meant developing higher quality products, at the lowest possible
               price, and delivering them in a timely fashion-preferably before
               their competitors could do the same. This imperative meant that
               they had no choice but to narrow the gap between requirements and
               resources in a manner that not only ensured they met their market
               targets, but did so in a manner that consumed resources fairly
               efficiently. It also meant that they had no choice but to fully
               support the development effort, instill strategic planning and
               prioritization, work collaboratively, follow a knowledge-based
               process that makes product development manageable, and,
               ultimately, make everyone accountable for success. Ultimately, the
               companies developed processes that embodied these tenets for
               success. At the strategic level, these include accurate, strategic
               planning and prioritization to ensure the right mix of products
               are pursued and strong systems engineering to help them establish
               a realistic business case. At the tactical level, companies
               developed development processes that required certain thresholds
               of knowledge to be gained before a decision to proceed forward is
               made.

               In theory, DOD's success likewise hinges on its ability to deliver
               high-quality weapons to the warfighter in a timely fashion. But in
               practice, success is defined as the ability of a program to win
               support and attract funds. Of course, there are reasons for this
               disconnect. Corporate revenue is generated by customer sales while
               DOD's funding is dependent on annual appropriations. Corporations
               go out of business when their product development efforts do not
               succeed; DOD does not. Selling products to customers is the single
               focus of a private-sector company while DOD is charged with a
               myriad of important missions-each of which also competes for
               budget share. Nevertheless, these conditions create a vastly
               different set of processes and behaviors affecting program
               managers. Program managers are incentivized, for example, to be
               optimistic and suppress bad news because doing otherwise could
               result in a loss of support and funding and further damage their
               program. In short, unknowns become acceptable and desirable rather
               than unacceptable as they are in the corporate environment. And
               accountability becomes much more difficult to define.

               DOD plans to spend about $1.3 trillion for its major programs
               between 2005 and 2009 and increase its investment in research and
               development during that period by about 28 percent-from $144
               billion to $185 billion. Although DOD's weapons are widely
               regarded as unrivaled in superiority, DOD has not received a
               predictable return on investment in major weapon systems
               acquisitions. For decades, many of DOD's weapon systems
               acquisitions have experienced large cost increases and extended
               schedules, which, in turn, have jeopardized performance and, more
               broadly, undermined DOD's buying power.

               To help better position DOD to successfully field weapons, we have
               undertaken a body of work over the past decade that has examined
               lessons learned from the best commercial product development
               efforts to see if they can be applied to DOD weapon system
               development. Leading commercial firms have developed increasingly
               sophisticated products in significantly less time and at lower
               costs. Our previous best practices reports1 have examined such
               topics as matching resources with requirements, controlling total
               ownership costs, effective use of testing, and product
               development. This report examines the program manager's role and
               the mechanisms that DOD and leading commercial companies use to
               position program managers for success and hold them accountable.
               As the central executor of the acquisition process, DOD depends on
               its program managers to efficiently and effectively run its large
               range of complex weapon systems acquisitions.

               The challenge that program managers now face is massive. Weapon
               systems themselves are becoming increasingly sophisticated and
               interdependent and, therefore, more complicated and difficult to
               develop. At the same time, however, DOD is faced with threats that
               are constantly evolving, requiring quicker development cycles and
               more flexibility within weapons programs. Moreover, many of the
               business processes that support weapons development-strategic
               planning and budgeting, human capital management, infrastructure,
               financial management, information technology, and contracting-are
               beset with pervasive, decades-old management problems, including
               outdated organizational structures, systems, and processes. In
               fact, these areas-along with weapons system acquisitions-are on
               GAO's high-risk list of major government programs and operations.
               Lastly, while DOD plans to considerably ramp up weapons system
               spending in the next 5 years in an effort to dramatically
               transform how it carries out its military operations, it is likely
               to face considerable pressure to reduce its investment in new
               weapons as the nation addresses long-term fiscal imbalances.

               While DOD's acquisition process has produced weapons that are
               among the best in the world, it also consistently yields
               undesirable consequences-such as cost increases, late deliveries
               to the warfighter, and performance shortfalls. Such problems have
               been highlighted, for example, in our past reviews of DOD's F/A-22
               Raptor, Space-Based Infrared System, Airborne Laser, the Joint
               Strike Fighter, and other programs. Our past work has found that
               problems occur because DOD's weapon programs do not capture early
               on the requisite knowledge that is needed to efficiently and
               effectively manage program risks. For example, programs move
               forward with unrealistic cost and schedule estimates, lack clearly
               defined and stable requirements, use immature technologies to
               launch the product development, and fail to solidify design and
               manufacturing processes at appropriate junctures in development.

               When costs and schedules increase, quantities are cut and the
               value for the warfighter, as well as the value of the investment
               dollar, is reduced. Moreover, in these times of asymmetric threats
               and netcentricity, individual weapon system investments are
               getting larger and more complex. Just 4 years ago, the top five
               weapon systems cost about $281 billion; today, in the same base
               year dollars, the top five weapon systems cost about $521
               billion.2 If these megasystems are managed with traditional
               margins of error, the financial consequences-particularly the
               ripple effects on other programs-can be dire.

               DOD has long recognized such problems and initiated numerous
               improvement efforts. In fact, between 1949 and 1986 five
               commissions studied issues such as cycle time and cost increases
               as well as the acquisition workforce. DOD has also undertaken a
               number of acquisition reforms. Specifically, DOD has restructured
               its acquisition policy to incorporate attributes of a
               knowledge-based acquisition model and has reemphasized the
               discipline of systems engineering. In addition, DOD recently
               introduced new policies to strengthen its budgeting and
               requirements determination processes in order to plan and manage
               systems based on joint warfighting capabilities. While these
               policy changes are positive steps, we recently testified that
               implementation in individual programs has not occurred because of
               inherent funding, management, and cultural factors that lead
               managers to develop business cases for new programs that
               overpromise on cost, delivery, and performance of weapon systems.

               DOD relies on a cadre of military and civilian officials-known as
               program managers-to lead the development and delivery of hundreds
               of weapon systems and subsystems. The services report a combined
               total of 729 program managers currently executing programs at all
               acquisition category levels. The systems that program managers are
               responsible for range from highly sophisticated air, land, sea,
               and space-based systems to smaller, less complex communications or
               support equipment that interconnects or supports larger systems.
               Program managers are responsible for assuring that these systems
               are reliable, affordable, supportable, and effective. They carry
               out multiple roles and responsibilities and are expected to have a
               working knowledge in such diverse areas as contracting, budgeting,
               systems engineering, and testing.

               DOD classifies its acquisition programs into categories based upon
               a number of factors such as their size, cost, complexity and
               importance. The largest, most complex and expensive programs
               generally fall under the responsibility of the Under Secretary of
               Defense (Acquisition, Technology and Logistics) while less complex
               and risky programs are overseen by the service or component
               acquisition executive. Table 1 provides more details.

               Table 1: Acquisition Categories

               Source: GAO.

               Note: Category I systems are referred to as "programs" and smaller
               related subsystems are called "projects" or "products." For
               example, the Air Force's B-1 aircraft system-a category IC
               program-includes category II and III projects that may have a
               designated manager. Category 1D and 1C programs are distinguished
               by their milestone decision authority.

               Program managers typically supervise a large staff of engineers,
               contracting personnel, logisticians, business, financial, and
               administrative personnel. The number of people assigned to program
               offices varies widely and depends on factors such as the
               complexity of the system, the category level, and the availability
               of staff. For example, the Joint Strike Fighter, a category ID
               program, is managing the development of three configurations of a
               new aircraft for the Navy, Marines and Air Force, and currently
               has about 200 government and international personnel assigned. By
               contrast the Light Utility Helicopter, a category II project
               relying largely on commercial off-the-shelf components, has a
               staff of 34.

               To successfully deliver a weapon system to the user, program
               managers must also work with a range of individuals outside their
               sphere of influence such as those charged with independent cost
               estimating, testing, funding, writing requirements, security, and
               ensuring interoperability. Simultaneously, the program manager is
               responsible for overseeing, integrating, and evaluating the
               defense contractor's work as the development progresses. Moreover,
               some program managers lead international teams. For example, the
               Joint Strike Fighter Program Office, in addition to the military,
               civilian, and contract team members, has eight international
               partners and approximately 40 international team members.

               The majority of DOD program managers for category I programs are
               military officers at the rank of colonel or (Navy) captain.
               Subsystem program managers are usually lower in rank and report
               directly to the system program manager. DOD also employs civilian
               program managers, usually GS-15s for its category I programs. As a
               rule, program managers report to a Program Executive Officer-a
               civilian at the senior executive level or military officer at the
               general officer rank-who typically manages a portfolio of related
               weapon systems. However, some program executive officers are
               responsible for a single large program, such as the Joint Strike
               Fighter or the F-22 aircraft. One level up from the program
               executive officer is the Service Acquisition Executive, a civilian
               (often a political appointee) who reports to the service
               Secretary. Programs classified as a category ID report through the
               defense acquisition executive, Undersecretary of Defense
               (Acquisition, Technology and Logistics), as their milestone
               decision authority.

               Program manager training and tenure is now governed by legislation
               known as the Defense Acquisition Workforce Improvement Act
               (DAWIA),3 enacted in 1990 after studies showed that a key problem
               affecting acquisitions was that program managers did not stay in
               their positions long enough to be accountable for outcomes and
               that many simply lacked the training and experience needed to
               assume their leadership roles. Congress amended the law in the
               fiscal year 2004 and 2005 defense authorization acts to allow the
               Secretary of Defense more flexibility to tailor tenure,
               experience, and education qualifications for program managers.

               The act specifically created a formal acquisition corps and
               defined educational, experience, and tenure criteria needed for
               key positions, including program managers as well as contracting
               officers and others involved the acquisition process. The act also
               provided for the establishment of a defense acquisition university
               to provide educational development and training for acquisition
               personnel. Under DOD regulations program managers are required to
               attend training and meet course requirements through the
               university in order to meet certification requirements for the
               program management track.

               There are three progressive certification levels: basic,
               intermediate, and advanced. Program managers of major defense
               acquisition programs are required to have Level 3 certification,
               which requires four years of acquisition experience and an
               advanced level Defense Acquisition University course in program
               management. DOD prefers that individuals with Level 3
               certification have a Master's degree in engineering, systems
               acquisition management, or business administration, and complete
               additional external coursework in relevant fields.

               The Chairman and the Ranking Member, Subcommittee on Readiness and
               Management Support, Senate Committee on Armed Services, requested
               that we examine best practices and DOD procedures for factors that
               affect program manager effectiveness. Our overall objectives for
               this report were to (1) identify best practices that have enabled
               organizations to successfully position their program managers for
               success, (2) identify DOD practices for supporting program
               managers and holding them accountable, and (3) compare and
               contrast DOD and commercial practices in order to identify
               possible improvements to DOD practices.

               To identify the best practices and processes that commercial
               companies employ to position their program managers for success,
               we used a case study methodology. We selected companies that, like
               DOD, research, develop, and field products, using program managers
               as the central executors of the programs. Selection of the
               companies was also based upon recognition by the American
               Productivity and Quality Center and the Project Management
               Institute and the recommendations of experts. Below are
               descriptions of the three companies that are specifically featured
               in this report.

               o  Toyota Motor Manufacturing of North America, Inc.

               Toyota Motor Manufacturing of North America, Inc., the third
               largest automobile producer in the world and the fifth largest
               industrial company in the world, designs, manufactures, and
               markets cars, trucks, and buses worldwide. In 2005, the company
               reported total net sales of $172.7 billion. We met with
               individuals involved with the development of the 2005 Toyota
               Avalon, a full-size sedan, at Toyota Motor Manufacturing in
               Erlanger, Kentucky.

               o  Siemens Medical Solutions USA, Inc.

               Siemens Medical Solutions is one of the world's largest suppliers
               in the healthcare industry. Siemens Medical manufactures and
               markets a wide range of medical equipment, including magnetic
               resonance imaging systems, radiation therapy equipment, ultrasound
               equipment, and patient monitoring systems. We met with individuals
               from the Angiography, Cardiology, and Neurology business unit,
               located in Hoffman Estates, Illinois.

               o  Motorola, Inc.

               Motorola is a Fortune 100 global communications leader that
               provides seamless mobility products and solutions across
               broadband, embedded systems and wireless networks. Seamless
               mobility harnesses the power of technology convergence and enables
               smarter, faster, cost-effective, flexible communication in homes,
               autos, workplaces and all spaces in between. Motorola had sales of
               $31.3 billion in 2004. We visited its offices in Arlington
               Heights, Illinois, and discussed program management practices and
               processes with representatives from the Networks sector.

               In addition to the three companies featured in this report, we
               visited two additional successful firms to assess whether they
               employed similar processes and practices for program management.
               These include Molson Coors Brewing Company and Wells Fargo. Both
               companies have undertaken projects that reflect some of the
               complexity and challenges that a DOD weapon systems program would
               face. For example, we met with managers of a Molson project
               intended to automate day-to-day marketing operations for digital
               assets. We also met with Wells Fargo officials who developed an
               electronic imaging process for paperless check clearance. At both
               companies, we also discussed broader corporate investment
               processes that supported these particular internal projects as
               well as the companies' main service lines.

               For each of the five companies, we interviewed senior management
               officials and program managers to gather consistent information
               about processes, practices, and metrics the companies use to
               support program managers and hold them accountable. In addition to
               the case studies, we synthesized information from GAO's past best
               practices work about product development.

               We also examined key best practices studies related to program
               management, including studies from organizations such as the
               Project Management Institute and the American Productivity and
               Quality Center. Moreover, we relied on our previous best practice
               studies, which have examined incentives and pressures affecting
               weapon system programs, the optimal levels of knowledge needed to
               successfully execute programs, and complementary management
               practices and processes that have helped commercial and DOD
               programs to reduce costs and cycle time.

               In order to determine DOD practices for supporting program
               managers and holding them accountable, we conducted five separate
               focus groups between July and October 2004. Each group was
               composed of project managers from one of the services or the
               Missile Defense Agency. A total of 28 acquisition category I
               program managers representing a range of DOD programs were
               identified by their respective services for the meetings held in
               separate locations in Huntsville, Ala.; El Segundo, Calif.;
               Dayton, Ohio; Arlington, Va.; and Ft. Belvoir, Va. For each focus
               group, the facilitators introduced discussion topics to discover
               how program managers define success, as well as what they are
               accountable for and how they are held accountable. In addition,
               participants were asked to discuss how program managers are
               supported and what obstacles they encounter in performing their
               duties.

               We analyzed the content of focus group transcripts and used the
               themes we identified to design a survey to gather information
               about acquisition category I and II program managers' perceptions
               about factors that assist or block their success and to more
               clearly define other issues in the DOD acquisition process that
               affect program manager effectiveness. We elicited input from
               several experts-retired program managers, active-duty members with
               program management experience, and senior acquisition officials
               who reviewed the questions and provided feedback on the draft
               survey.

               We pretested the survey with five program managers. During the
               pretest we asked the program managers questions to determine
               whether (1) the survey questions were clear, (2) the terms used
               were precise, (3) the questionnaire placed an undue burden on the
               respondents, and (4) the questions were unbiased. We then
               incorporated their comments into the survey, finalized the
               questions, and sent the web-based survey to acquisition category I
               and II program managers. We selected the category I and II program
               managers because they manage the more complex and expensive
               programs. We identified the program managers through consultation
               with each of the services. The survey consisted of open- ended and
               close-ended questions concerning support for program managers and
               how they are held accountable for program outcomes. Originally we
               e-mailed 237 program managers but later determined that 52 should
               not be included because they managed programs other than
               acquisition category I and II. Of the 185 remaining program
               managers, we received completed surveys from 69 percent.

               The surveys were conducted using self-administered electronic
               questionnaires posted on the World Wide Web. We sent e-mail
               notifications to all acquisition category I and II program
               managers on April 12, 2005. We then sent each potential respondent
               a unique password and username by e-mail to ensure that only
               members of the target population could participate in the survey.
               To encourage respondents to complete the questionnaire, we began
               sending e-mail messages to prompt each nonrespondent between April
               26, 2005 and May 19, 2005. Additionally, the team contacted
               nonrespondents through telephone calls between May 31, 2005 and
               July 12, 2005. We closed the survey on July 19, 2005.

               In this report we discuss some of the results obtained from the
               survey. A more complete tabulation of survey questions together
               with tables indicating the levels of response can be found on our
               Web site at GAO-06-112SP. The survey contained close-ended
               questions and open-ended questions. We conducted a content
               analysis of the open-ended questions and constructed tables
               showing the results of the analysis arranged into broad
               categories. Some of the respondents to our survey provided more
               than one answer to the open-ended questions. All responses that
               indicated equally important factors were tabulated in the
               appropriate categories. However, because some respondents provided
               more than one answer, the percentages may add up to more than 100
               percent of respondents. The web-based report does not contain all
               the results from the survey. For example, we do not report
               responses for questions about demographics, some open-ended
               questions, or questions with high item nonresponse rates.

               In addition to the focus groups and survey, we conducted in-depth
               interviews with individual program managers, program executive
               officers from across the services, as well as program managers
               from Boeing and Lockheed Martin for two major weapon systems. To
               further assess the conditions and environment program managers
               were operating in, we relied on previous GAO reports. For example,
               we relied on a recent study of space acquisition problems that
               incorporated interviews of more than 40 individuals, including
               experienced program managers, program executive officials,
               officials responsible for science and technology activities, and
               former and current officials within the Office of Secretary of
               Defense who have specific responsibility for space system
               oversight or more general weapon system oversight.

               To further determine relevant DOD policies and practices, we
               analyzed documents describing the roles and responsibilities of
               program managers, acquisition force career management, promotion
               rates, performance reporting, and training requirements. Moreover,
               we analyzed relevant legislation and the DOD 5000 series of
               directives and instructions. We also interviewed career
               acquisition service officials, Defense Acquisition University
               course managers, and the Director of Training. We reviewed studies
               from the Rand Corporation, the Center for Strategic and
               International Studies, and the Defense Science Board, among
               others, on weapons system program management and acquisition
               issues as well as studies performed by past commissions focused on
               acquisition reform.

               We conducted our review between April 2004 and November 2005 in
               accordance with generally accepted government auditing standards.

               Program managers from the leading companies we spoke with believed
               that two critical enablers-(1) support from top leadership and (2)
               disciplined, knowledge-based processes for strategic investment,
               program selection, and product development execution-empowered
               them to succeed in delivering new products when needed within
               cost, quality, and performance targets originally set by the
               company. At all of the companies we visited, corporate leadership
               began at a strategic level, long before the initiation of a new
               product development, with senior company leaders making critical
               strategic investment decisions about the firm's mix of products
               and the return on investment they may yield. Once high-level
               investment decisions were made, senior leaders assured that new
               programs did not begin until there was a business case for
               them-meaning there was assurance that the program fit in with the
               corporation's goals and investment strategy and that there were
               resources available to execute the program. Once a business case
               had been made, senior leaders selected and tasked program managers
               with executing the program. They also required the program
               managers to use a knowledge-based product development process that
               demanded appropriate demonstrations of technology, designs, and
               processes at critical junctures. They also empowered program
               managers as appropriate to execute the program and held them
               accountable for delivering the program within estimates. While
               they were empowered to execute the program, program managers were
               still supported by senior leaders, who encouraged open and honest
               communication and continually assured that the right levels of
               resources and management attention were available for the project.
               Figure 1 maps critical support and accountability factors.

                       DOD: Critical Support Factors Are Missing

    Differences in Incentives Contribute to Differences in Support for Program
                                        Managers

    Chapter 1: Introduction  Chapter 1: Introduction

    1 A complete list of best practices reports is at the end of this report.

               Long-Standing Problems Hamper Weapons Systems Acquisitions

    2 These figures represent the costs for the top five weapon systems in
    2001 and the top five in 2005. For 2001, these systems were F/A-22 Raptor,
    DDG-51 Guided Missile Destroyer, Virginia Class Submarine, C-17
    Globemaster III, and the F/A 18 E/F, Naval Strike Fighter. The 2005
    systems include the Joint Strike Fighter, Future Combat System, F/A- 22
    Raptor, DDG-51 Guided Missile Destroyer, and the Virginia Class Submarine.

         DOD Program Managers Are Central Executors of the Acquisition Process

    Acquisition
    category     Definition                            Program examples
    Category I   Research, development, test, and      Future Combat System
                    evaluation > $365M Procurement >      DD(X) Destroyer B-1
                    $2.19B                                Aircraft

                    Milestone decision authority: 1D:
                    Under Secretary of Defense
                    (Acquisition, Technology and
                    Logistics) 1C: Service Acquisition
                    Executive
    Category II  Research, development, test, and      All Source Analysis
                    evaluation > $140M Procurement >      System KC-130J Aircraft
                    $660M                                 Joint Surveillance and
                                                          Target Attack Radar
                    Milestone decision authority: Service System
                    or Component Acquisition Executive
    Category III No fiscal criteria                    10k W Auxiliary Power
                                                          Unit Assault Breaching
                                                          Vehicle C-5 Avionics
    Category IV  No fiscal criteria (Navy and Marine   C-130 Night Vision
                    Corps only)                           Lighting Advanced
                                                          Recovery Control System

                   Legislation to Improve Program Manager Proficiency

    3 10 U.S.C.S: 1701 et seq. (P.L. 101-510. Div A. Title XII (November 5,
    1990)).

                           Objectives, Scope, and Methodology

    Chapter 2: Senior Leader Support and Disciplined Knowledge-Based Processes
    Are Critical Enablers for Program Managers  Chapter 2: Senior Leader
    Support and Disciplined Knowledge-Based Processes Are Critical Enablers
    for Program Managers

    Figure 1: Critical Support and Accountability Factors

    Senior Leadership Provides Program Managers with a Strong Foundation for Success

    At each of the companies we visited, senior leaders invested a great deal
    of time and effort positioning new development efforts for success. Before
    even considering initiating a new project, senior leaders made high level
    trade-off decisions between their long-term corporate goals, projected
    resources, market needs, and alternative ways of meeting those needs.
    These efforts culminated in investment strategies that assured that the
    company could fully commit to any product development effort it pursued.
    With a broad strategy in place, senior leaders would then begin concept
    development for potential new products, analyzing proposed products in
    terms of what requirements could be achieved today versus future versions
    of the product and what resources would be needed-not just in terms of
    cost, but in terms of technologies, time, and people. Once a specific
    concept was selected, senior leaders would follow rigorous systems
    engineering processes to narrow the gap between requirements and resources
    to a point where they were assured that they were pursuing a product that
    would meet market needs and could be developed within cost and schedule
    goals. The end point of this process was a sound business case that senior
    leaders could then hand off to a program manager-who was then empowered to
    deliver the product on time and within cost. Program managers themselves
    highly valued this support because it ensured the companies were committed
    to their particular efforts, reduced the level of unknowns that they were
    facing, and kept them focused solely on executing their programs. Put more
    simply, they believed senior leaders consistently provided a sound
    foundation on which they could launch their programs.

    The most critical characteristics of the strategic leadership provided
    include the following:

               o  Investment strategies. Because there are more product ideas
               than there is funding to pursue them, the commercial companies we
               visited used a knowledge-based process to make decisions about
               which product development efforts to invest in. They began by
               developing an investment strategy that supports a corporate
               vision. For the most profitable mix of new products, companies
               analyzed factors such as customer needs, available technology, and
               available resources. Companies ensured that decisions to start new
               product developments fit within the investment strategy. The
               investment strategy determined project priority as well as
               providing a basis for trade-off decisions against competing
               projects. Program managers found their company's use of investment
               strategies helpful because it gave them confidence that their
               project had commitment from their organization and from their top
               leaders and managers and clearly identified where their project
               stood within the company's overall investment portfolio and
               funding priorities.

               o  Evolutionary development. All of the companies generally
               followed an evolutionary path toward meeting market needs rather
               than attempting to satisfy all needs in a single step. In effect,
               the companies evolved products, continuously improving their
               performance as new technologies and methods allow. These
               evolutionary improvements to products eventually result in full
               desired capability, but in multiple steps, delivering enhanced
               capability to the customer more quickly through a series of
               interim products. For example, the 2005 Avalon involved redesign
               of about 60 per cent of the vehicle, but component sections such
               as the electronics and such features as the keyless ignition
               system and the reclining rear seat were either developed by
               suppliers or had been used on the Lexus. By using this method, the
               company changed the Avalon's overall design and functionality by
               increments. In more strategic investment planning, Toyota
               maintains an ongoing research into such technology areas as
               alternative fueled automobiles and environmental implications of
               automotive developments that will feed into its long-term
               planning. Our previous work has found that this approach reduces
               the amount of risk in the development of each increment,
               facilitating greater success in meeting cost, schedule, and
               performance requirements. The approach permits program managers to
               focus more on design and manufacturing with a limited array of new
               content and technologies in a program. It also ensures that the
               company has the requisite knowledge for a product's design before
               investing in the development of manufacturing processes and
               facilities. Conversely, our past work has found that organizations
               that set exceedingly high technology advancement goals invariably
               spend more time and money than anticipated trying to address
               technology-related challenges amid other product development
               activities, including design and production stabilization.

               o  Matching of Requirements and Resources. The companies we
               visited were able to achieve their overall investment goals by
               matching requirements to resources-that is time, money,
               technology, and people-before undertaking a new development
               effort. Any gaps that existed were relatively small, and it was
               the program manager's job to quickly close them as development
               began. More specifically:

                            o  The companies had already extensively researched
                            and defined requirements to ensure that they are
                            achievable given available resources before
                            initiating new efforts.

                            o  Technologies were mature at the start of a
                            program, that is, they had been proven to work as
                            intended. More ambitious technology development
                            efforts were assigned to corporate research
                            departments until they were ready to be added to
                            future generations (increments) of the product. In
                            rare instances when less mature technologies were
                            being pursued, the company accepted the additional
                            risk and planned for it.

                            o  Companies committed to fully fund projects before
                            they began. Not one of the program managers we spoke
                            with mentioned funding as a problem at the beginning
                            of a development effort and throughout. Funding was a
                            given once senior leaders had committed to their
                            project.

                            o  Systems engineering was typically used to close
                            gaps between resources and requirements before
                            launching the development process. As our previous
                            work has shown, requirements analysis, the first
                            phase of any robust systems engineering regimen, is a
                            process that enables the product developer to
                            translate customer wants into specific product
                            features for which requisite technological, software,
                            engineering, and production capabilities can be
                            identified. Once these are identified, a developer
                            can assess its own capabilities to determine if gaps
                            exist, and then analyze and resolve them through
                            investments, alternate designs, and, ultimately,
                            trade-offs. The companies we visited allowed their
                            engineers to analyze and weigh-in on the customers
                            needs as determined by its marketers.

               Our previous best practice work has consistently found the
               practice of matching requirements and resources prior to
               initiating a new program to be a hallmark for successful
               companies. Simply put, we have found that when wants and resources
               are matched before a product development is started, the
               development is more likely to meet performance, cost, and schedule
               objectives. When this match does not take place at the start of a
               program, programs typically encounter problems such as increased
               costs, schedule delays, and performance shortfalls as they try to
               meet requirements during product development. Program managers we
               spoke with for this review specifically cited this process as an
               enabler for their own success because it ensured they were in a
               good position to commit to cost and schedule estimates that were
               attainable, and it did not require them to perform "heroic"
               efforts to overcome problems resulting from large gaps between
               wants and resources, such as technology challenges or funding
               shortages.

               In addition to these critical strategic enablers, program managers
               at the companies also stated that senior leaders made concerted
               efforts to match program manager skills and experience to
               appropriate projects and to train and mentor program managers. In
               selecting program managers themselves, the companies placed high
               value on strong leadership qualities, including decision making
               skills, diplomacy, communication skills, ability to motivate
               others, and integrity, as well as how individual personalities fit
               with the job or team. Most of the program managers we spoke with
               had been groomed for their positions through formal training on
               budgeting, scheduling, and regulatory compliance and other aspects
               of program management; informal mentoring by senior executives or
               experienced program managers; and by being placed in positions
               that gradually increased their management responsibilities. In
               addition, many of the program managers we spoke with also
               possessed considerable technical experience. In fact, they often
               started at the company as engineers. The companies we visited were
               similarly deliberate in developing and deploying teams of
               functional experts to support a program manager. In some cases,
               the teams reported directly to the program manager. In others,
               they reported to their respective home units and worked
               collaboratively with the program managers. In either case, the
               program managers themselves valued the support they were getting
               from these teams-particularly because they enabled the program
               manager to employ a broad array of expertise from day-one of the
               development effort and to facilitate an exchange of ideas. The
               program managers we spoke with believed that their functional
               teams were also highly skilled-to the point where they could
               easily delegate major tasks.

               Strategic leadership for the development of Toyota's Avalon luxury
               sedan ties back to conscious decisions made by senior leaders in
               Japan when they built a Toyota facility in the United States 25
               years ago. To assure that the vehicles could be made to the same
               levels of quality as those in Japan, Toyota replicated its
               manufacturing facilities, used Japanese suppliers, and sent its
               managers to the United States to supervise development. As U.S.
               employees gained experience and demonstrated their capability, the
               reliance on Japanese suppliers and personnel gradually decreased.
               A second step Toyota took was to replicate its training and
               mentoring of program managers-pairing them with more experienced
               chief engineers, who oversee long-term planning across projects,
               and even bringing them to Japan to study how Toyota approached
               development.

               To support all of its new development efforts, senior leaders have
               developed an overall strategic plan-which takes a long- and
               short-term investment perspective. Over the long run, the plan
               envisions the company achieving significant advancements in
               capabilities, such as alternatively fueled engines, through
               incremental improvements to technologies. Over the short run, a
               specific vehicle development program uses a marketing analysis
               about features customers desire in new models; and the staff
               determines whether a market exists for a certain type of product
               at a certain price. In establishing a business case for the
               Avalon, Toyota embarked on a formal concept development effort,
               which was led by a chief engineer. The chief engineer, a
               high-level executive, was largely responsible for setting the
               vision for the new Avalon, securing resources needed for
               development effort prior to initiating the development program,
               and working with representatives from its sales division to make
               sure that the design and technologies being pursued still fit
               within market needs-not just in terms of cost, but in terms of
               vehicle features. A variety of functional experts were consulted
               during this phase, though the chief engineer had the most formal
               authority over concept development. At the conclusion of this
               effort, Toyota decided to take on a very extensive redesign of the
               Avalon but also set a goal bringing the vehicle to market in only
               18 months. Redesign features included a reinforced body,
               improvements to the engine and to the braking system, as well as
               features customers desired such as a keyless ignition system and
               reclining rear seats. Toyota leadership also decided to include
               mature technologies, often borrowed from other vehicle lines, or
               purchased from outsider suppliers. Once the design was approved,
               day to day project management shifted to the Chief Production
               Engineer, whose responsibility it was to see the vehicle through
               production to distribution.

               Figure 2: 2005 Toyota Avalon

               Corporate leadership at Siemens Medical took a similar shape in
               the development of new medical equipment. For example, senior
               leaders developed an overall investment strategy, based largely on
               researching their customers' technology needs as well as their own
               technology readiness, the direction their competitors were going
               in, economic trends, and projected manpower resources. From these
               assessments, a team within Siemens developed a portfolio of
               potential new projects to pursue, which upper management then
               prioritized based on their potential profit, how they fit in with
               corporate goals and projected resources. Ultimately, senior
               leaders produce a short-term (1 year) investment plan as well as a
               longer-term (3 to 5 year) plan. Once a specific project is
               selected, Siemens employs systems engineering practices to narrow
               down the gap between customer requirements and resources-working
               with both business and technical managers. A "product manager" is
               charged with making trade-offs between requirements, schedule, and
               cost prior to initiating product development and is held
               accountable for systems engineering decisions made to level
               requirements with resources for the business case. This person
               sits at a relatively high level within the company and possesses
               marketing and business expertise. A "project manager" who reports
               to the product manager is ultimately assigned to execute the
               business case, but he or she plays a role in the concept
               development by participating in trade-off decisions and raising
               concerns about how decisions can be executed.

               At Siemens Medical, many project managers begin by serving as the
               technical leader working with three to five people in systems
               engineering or another technical area of a project. As the
               technical team lead, they gain experience with scheduling,
               communicating, and managing people. Over time the individual is
               given more responsibilities such as becoming a subsystem project
               leader; as the manager gains experience, he or she transitions to
               handling cross-functional areas including business, budgeting,
               staffing, technology, and testing.

               Siemens Medical project managers are also given formal training,
               including courses on regulatory and quality requirements as well
               as courses that help program managers learn about their management
               styles. In addition, Siemens ensures that project managers are
               well-trained on risk management so that they can identify and
               mitigate potential risks at the beginning of the project. Also,
               since project managers function within a centralized project
               management department, they are mentored both by the head of the
               department and by their peers.

               Figure 3: Siemens Bi-Plane AXIOM Artis

               Once a new development effort began, program managers were
               empowered to execute the business case and were held accountable
               for doing so. At all of the companies we visited, program managers
               believed that following a disciplined, knowledge-based development
               process and continued support from senior leaders were essential
               to their success. The process itself was typically characterized
               by a series of gates or milestone decisions, which demanded
               programs assess readiness and remaining risk within key sectors of
               the program as well as overall cost and schedule issues, and it
               required go/no-go decisions to be made fairly quickly. The most
               important aspect of the process, in the view of the program
               managers, was that it empowered them to make decisions about
               design and manufacturing trade-offs, supplier base, staffing on
               the program team, etc.-as long as they were within the parameters
               of the original business case. At the same time, the process held
               program managers accountable and set clear goals and incentives.

               Common critical characteristics of the knowledge-based process
               followed to execute programs include the following:

               o  Knowledge-driven development decisions. Once a new product
               development began, program managers and senior leaders used
               quantifiable data and demonstrable knowledge to make go/no-go
               decisions. These covered critical facets of the program such as
               cost, schedule, technology readiness, design readiness, production
               readiness, and relationships with suppliers. Development was not
               allowed to proceed until certain thresholds were met, for example,
               a high proportion of engineering drawings completed or production
               processes under statistical control. Program managers themselves
               placed high value on these requirements, as it ensured they were
               well positioned to move into subsequent phases and were less
               likely to encounter disruptive problems.

               o  Empowerment for program managers to make decisions. At all the
               companies we visited, program managers were empowered to make
               decisions on the direction of the program and to resolve problems
               and implement solutions. They could make trade-offs among
               schedule, cost, and performance features, as long as they stayed
               within the confines of the original business case. When the
               business case changed, senior leaders were brought in for
               consultation-at this point, they could become responsible for
               trade-off decisions.

               o  Accountability. Program managers at all the companies we
               visited were held accountable for their choices. Sometimes this
               accountability was shared with the program team and/or senior
               leaders. Sometimes, it resided solely with the program manager on
               the belief that company had provided the necessary levels of
               support. In all cases, the process itself clearly spelled out what
               the program manager was accountable for-the specific cost,
               performance, schedule, and other goals that needed to be achieved.

               o  Tenure. To further ensure accountability, program managers were
               also required to stay with a project to its end. Sometimes senior
               leaders were also required to stay. At the same time, program
               managers were incentivized to succeed. If they met or exceeded
               their goals, they received substantial bonuses and/or salary
               increases. Awards could also be obtained if the company as a whole
               met larger objectives. In all cases, companies refrained from
               removing a program manager in the midst of a program. Instead,
               they chose first to assess whether more support was needed in
               terms of resources for the program or support and training for the
               program manager.

               Other important aspects within the development process included
               the following:

               o  Common templates and tools to support data gathering and
               analysis. These tools included databases of demonstrated,
               historical cost, schedule, quality, test, and performance data
               that helped program managers produce metrics as well as standard
               forms and guidance for conducting the meetings. Program managers
               valued these tools because they greatly reduced the time needed to
               prepare for milestone meetings. In all cases, program managers did
               not believe they were spending time collecting data that was
               valuable to senior management but not to them.

               o  Common processes that supported product development. The
               companies generally found that requiring program managers to
               employ similar risk management, project management, requirements
               approval, testing, quality management, problem resolution, and
               other processes enabled them to add additional discipline and
               consistency to product development. Some companies were certified
               by professional organizations as achieving the highest level of
               proficiency within supporting development processes. For example,
               Motorola was certified as a level 5 software development
               organization by Carnegie Mellon's Software Engineering Institute.

               o  Lessons learned. All of the companies we visited continually
               refined and enhanced their development process via some sort of
               lessons-learned process. The program managers themselves placed a
               great deal of value on these processes-as they were seen as the
               primary means for learning how to tailor the process to better fit
               a project and to prevent the same mistakes from recurring.

               Program managers also cited flexibility as an enabling quality of
               their processes. All of the companies allowed their processes to
               be tailored as needed. Milestones that were deemed unnecessary
               could be dropped. More often, however, additional meetings were
               added to gain consensus on how to address particular problems.
               Another enabling factor was that their processes ensured
               decisionmakers were not flooded with data. Often, program teams
               boiled down data into one or two pages, using simple metrics
               depicting status and risk on critical facets of the program such
               as cost, schedule, technology readiness, design readiness, and
               production readiness. Program managers valued the process of
               translating detailed data into higher level metrics because it
               required them to think about their programs in more strategic
               terms and focus on the highest problem areas.

               Motorola's development process is comprised of 16 milestones or
               "gates"-the first five of which pertain to processes employed to
               develop a product concept and the business case. Eleven gates
               comprise the execution of the business case, from project
               initiation, to systems requirements definition, design readiness,
               testing, controlled introduction, and then full deployment. Each
               gate demands an array of indicators on status and progress,
               including resources, cost, scope, risk, and schedule. A
               centralized database helps program managers produce this data and
               allows users to obtain data at any time and at any level of detail
               that they need. For meetings themselves, program managers are
               required to produce a set of "vital few" performance measures
               relating to cost, quality, program status, and customer
               satisfaction. At the gates themselves, program managers discuss
               the status of the program with senior leaders, but they are
               ultimately responsible for making decisions on whether to proceed
               to the next phase. In the past, program managers did not have this
               responsibility and acted more as an administrator than a leader,
               according to senior executives. With less responsibility and
               accountability, programs were not managed as well-often employing
               disjointed management processes with less attention to efficiency
               and effectiveness. By increasing program manager's ownership and
               accountability over the project, senior leaders found that they
               were more incentivized to meet and exceed cost, schedule, and
               performance goals. To support this change, the company also
               adopted common supporting processes, including configuration
               management, design, training, testing, defect prevention, quality
               management, supplier management, and system upgrades. The common
               processes assured program managers employed the same set of
               quality controls and that deployed tools and guidance enabled
               program managers to reduce cycle times as well as to produce
               better and more consistent management data.

               Toyota's process is comprised of eight key milestones-starting
               with a lessons-learned gate. At this point, senior leaders and
               project teams formally review what worked well and not so-well in
               the prior development effort and assess whether the process needs
               to be tailored as a result. The Avalon program manager told us
               that these "reflections" are not taken lightly; they are developed
               through a very detailed and soul-searching process during which
               people have to openly admit errors and inadequacies so that better
               processes and procedures can be devised. The next gate, "image"
               represents the process by which the chief engineer derives the
               business case. Once he is done, direct supervision of the project
               is transferred to a "chief production engineer," who is charged
               with its execution of the business case although the chief
               engineer continues to be involved in the development. The next few
               milestones come as the car is designed, prototyped, tested, and
               put through quality assurance. The last milestone, the production
               stage also contains a customer feedback phase, which is used to
               refine the next development effort.

               Within the business case itself, Toyota places highest importance
               on schedule because a number other vehicle development efforts are
               dependent on the same resources and staff being used by the
               current effort. As a result, the chief production engineer is more
               inclined to make trade-offs that favor schedule over other
               factors. At each milestone meeting, the chief production engineer
               reviews the status of the program with senior leaders, focusing
               first on what problems are occurring and what his solutions are
               for overcoming them. The meeting itself employs streamlined
               reporting with simple indicators of remaining risk on critical
               facets of the program-specifically, a circle, meaning low
               remaining risk and okay to proceed; a triangle, meaning there are
               problems but they can be fixed, or an "x," meaning there is a
               problem without a solution. The chief production engineer is
               responsible for making decisions as to how to proceed at these
               milestones, unless there is a problem that significantly affects
               the business case. If so, senior leaders become more involved in
               the decision-making rather than simply advising the chief
               production engineer.

               While the Toyota process only employs eight formal milestones, the
               chief production engineer actually involves functional experts,
               senior executives, and other stakeholders in frequent meetings to
               make tactical decisions about the program. For example, the Avalon
               chief production engineer told us that he held "obeya" (literally
               "big room," signifying that all inputs are desired) meetings twice
               a week, which involved all functional areas as well as "namewashi"
               (literally binding the roots together, signifying gathering facts
               and moving toward a decision) meetings before a formal milestone
               meeting-at which functional officials consulted with each other to
               identify problems and develop potential solutions that would be
               presented to senior leaders at the milestone. Overall, the
               accountability for meeting the Avalon program's goals was shared
               between the chief production engineer, the functional team, and
               senior executives. At Toyota, senior leaders assume that the
               processes they have in place will work, and if the process is not
               delivering a suitable quality outcome, then it was the shared
               responsibility of managers and staff to resolve the issue. If
               performance issues arose, senior leaders attempted to address them
               first through training, mentoring, and additional support, rather
               than removing the program manager.

               Empowering program managers to make decisions in executing the
               business case was seen as the most significant type of support
               provided by senior leaders. But program managers themselves
               pointed to other types of support that made it easier for them to
               succeed. Primarily, senior leaders did the following:

               o  Provided unwavering commitment to the development effort. At
               all the firms we visited, senior leaders were champions of the
               project throughout its life and fully committed to supporting it.
               When significant problems arose that jeopardized the business
               case, they found ways to address those problems, rather than
               rejecting the program in favor of another one.

               o  Trusted their program managers. Senior leaders trusted the
               information being provided by the program manager as well as his
               or her expertise. This reduced the need for instilling additional
               layers of oversight that could slow down the program. At the same
               time, however, senior leaders took personal responsibility for
               assuring their program managers had the knowledge and capability
               needed to succeed-in some cases, by personally mentoring them for
               a long period of time.

               o  Encouraged program managers to share bad news. Senior leaders
               went out of their way to encourage program managers to share
               problems. In fact, program managers were often expected to discuss
               problems before anything else at key milestones. And, in some
               cases, program managers were evaluated based on their ability to
               identify and share problems. At the same time, senior leaders
               expected their program managers to come up with solutions-to take
               ownership over their efforts.

               o  Encouraged collaboration and communication. Senior leaders
               spent a great deal of time breaking down stovepipes and other
               barriers to sharing information. The Avalon chief production
               engineer, in fact, told us that Toyota's development processes
               alone were much like other automobile manufacturers he had worked
               for. What separated Toyota from the others was its emphasis on
               open information exchange, cooperation, and collaboration. He
               believed that this was the key enabler for Toyota's superior
               systems integration.

               Figure 4: Best Practice Roles, Responsibilities, and Behaviors of
               Senior Managers

               While DOD's leadership has taken action in recent years it hopes
               will better position programs and improve planning and budgeting,
               it is still not effectively positioning or supporting program
               managers for success. For example, rather than making strategic
               investment decisions, DOD starts more programs than it can afford
               and rarely prioritizes them for funding purposes. The result is a
               competition for funds that creates pressures to produce optimistic
               cost and schedule estimates and to overpromise capability.
               Moreover, our work has shown that DOD often starts programs
               without establishing a business case. Specifically, technologies
               are not always mature at start, requirements are not fully
               defined, and cost and schedule estimates are not always realistic.
               In addition, program managers are not empowered to execute
               programs. They cannot veto requirements and they do not control
               funding or other resources. In fact, program managers who
               responded to our survey personally consider requirements and
               funding instability to be their biggest obstacles to success.
               Program managers also believe that they are not sufficiently
               supported once programs begin. In particular, they believe that
               program decisions are based on funding needs of other programs
               rather than demonstrable knowledge; they lack tools needed to
               enable them to provide leadership consistent with cost, schedule
               and performance information; they are not trusted; they are not
               encouraged to share bad news; and they must continually advocate
               for their programs in order to sustain commitment.

    Strategic Leadership at Toyota and Siemens Medical

                  Knowledge-Based Process Followed to Execute Programs

    Knowledge-Based Development at Motorola and Toyota

     Continued Senior Leadership during Product Development Further Enabled Success

    Chapter 3: DOD Is Not Supporting Its Program Managers Effectively  Chapter
    3: DOD Is Not Supporting Its Program Managers Effectively

    Figure 5: Breakdowns in Support and Accountability Factors

           Senior Leadership Does Not Provide a Strong Foundation for Success

    According to program managers we interviewed as well as comments to our
    survey and our past reviews, senior leadership within DOD does not provide
    a strong foundation for success. While DOD is adept at developing
    long-term visions and strategic plans, it does not develop realistic,
    integrated investment strategies for weapons acquisitions to carry out
    these plans. Instead, more programs are started than can be funded and too
    many programs must compete for funding, which, in turn, creates incentives
    to produce overly optimistic estimates and to overpromise capability.
    Moreover, when faced with a lower budget, program managers believe that
    senior executives within the Office of the Secretary of Defense (OSD) and
    the services would rather make across-the-board cuts to a span of programs
    rather than hard decisions as to which ones to keep and which ones to
    cancel or cut back. Our work continues to show that, while DOD has adopted
    evolutionary development in its policies, programs are being encouraged to
    pursue significant leaps in capability. In addition, DOD's policy now
    encourages programs to match resources to requirements before program
    initiation, but program managers reported in our survey that requirements
    and funding are not stabilized and were the biggest obstacles to their
    success. Further, while program managers believe their training has been
    adequate, they also believe that mentoring has been uneven and that they
    could benefit with tours of duty inside the Pentagon, for example, in
    offices that oversee budget or financial management. Table 2 highlights
    differences between strategic senior leadership support within the
    commercial companies we visited and DOD.

    Table 2: Are Best Practices Present in DOD?

    Best practices            DOD
    Develop long-term vision  DOD has long-term vision, but not an
    and investment strategy   investment strategy. Lack of investment
                                 strategy has created competition for
                                 funding and spurred low cost-estimating,
                                 optimistic schedules, and suppression of
                                 bad news.
    Adopt evolutionary path   DOD has adopted evolutionary development
    toward meeting customer   in policy but not in practice.
    needs
    Match requirements and    DOD has encouraged achieving match in
    resources before starting policy but not in practice. Requirements
    new product development   are not stable; funding commitments are
                                 not enforced; key technologies are not
                                 matured before development. Requirements
                                 and funding are biggest obstacles in view
                                 of program managers.

    Source: GAO.

    Investment Strategy and Evolutionary Development

    DOD is attempting to address some of the problems identified, but it is
    too early to determine how effective its solutions are. For example, it is
    implementing a new requirements setting processes-known as the Joint
    Capabilities Integration and Development System-in an attempt to bring
    more discipline to investment decisions. The system is organized around
    key functional concepts and areas, such as command and control, force
    application, battlespace awareness, and focused logistics. For each area,
    boards of high-ranking military and civilian officials identify long-term
    joint needs and make high-level trade-offs on how those needs should be
    met. Once specific programs are proposed, the process is designed to
    encourage a more evolutionary approach by allowing requirements setters
    the flexibility to define requirements in terms of capabilities as well as
    to defer final requirements formulation to later in the development
    process. DOD has also been attempting to implement complementary planning
    and budgeting processes-for example, by asking the military services to
    plan budgets around guidance that takes a joint perspective and by taking
    a portfolio planning approach. However, there is no evidence to date that
    shows these enhancements are providing DOD with a sound investment
    strategy as well as the right controls for enforcing that strategy.

    While some program managers we spoke with believed the process' focus on
    capabilities versus requirements promised more flexibility, program
    managers comments to our survey show that they also still widely believed
    that they were operating in an environment where there was unfair
    competition for funding. Figure 6 highlights specific views.

    Figure 6: Highlights of Program Manager Comments Regarding Competition for
    Funding

    DOD has also adopted policies that encourage evolutionary development.1
    However, our reviews continue to find that programs are still pursuing
    significant leaps in capabilities. For example, we reported this year2
    that the Joint Strike Fighter acquisition strategy was striving to achieve
    the ultimate fighter capability within a single product development
    increment, and that it had bypassed early opportunities to trade or defer
    to later increments those features and capabilities that could not be
    readily met. We also testified3 that while DOD's space acquisition policy
    states its preference for evolutionary development, programs still attempt
    to achieve significant leaps in one step.

    1 DOD Directive 5000.1, the Defense Acquisition System (May 2003) and DOD
    Instruction 5000.2 Operation of the Defense Acquisition System (May 2003).
    The directive establishes evolutionary acquisition strategies as the
    preferred approach to satisfying DOD's operational needs. The directive
    also requires program managers to provide knowledge about key aspects of a
    system at key points in the acquisition process. The instruction
    implements the policy and establishes detailed policy for evolutionary
    acquisition.

    Matching Resources to Requirements

    In recent years, DOD has changed its acquisition policy to encourage
    decisionmakers to match requirements to resources before starting a new
    program. For example, the policy specifically encourages that technologies
    be demonstrated in a relevant environment before being included in a
    program; that a full funding commitment be made to a program before it is
    started and that requirements be informed by the systems engineering
    process. Concurrently, DOD's new requirements process is designed to
    instill more discipline during initial requirements development and
    postpone final determination of requirements to assure that requirements
    being set are achievable.

    In practice, however, our work has shown that there are still significant
    gaps between requirements and technology resources when programs begin.
    Our most recent annual assessment of major weapon systems programs,4 for
    example, showed that only 15 percent of the programs we reviewed began
    development having demonstrated that all of their technologies were
    mature. More often than not, programs had to worry about maturing
    technologies well into system development, when they should have focused
    on maturing system design and preparing for production. These assessments
    also show that programs that started development with mature technologies
    experienced lower development and unit cost increases than those programs
    that started with immature technologies. Table 3 provides some examples.

    2 Tactical Aircraft: Opportunity to Reduce Risks in the Joint Strike
    Fighter Program with Different Acquisition Strategy,
    GAO-05-271(Washington, D.C.: March 15, 2005).

    3 Space Acquisitions: Stronger Development Practices and Investment
    Planning Needed to Address Continuing Problems, GAO-05-891T, (Washington,
    D.C.: July 12, 2005).

    4 Defense Acquisitions: Assessments of Selected Major Weapon Programs,
    GAO-05-301 (Washington, D.C.: March 31, 2005).

    Table 3: Technology Maturity and Program Outcomes

                                                        Percent of critical
                                      Percent increase     technologies and
                                         in R&D (first  associated maturity
                                      full estimate to level at development
    Program                        latest estimate)                start
    Advanced Threat Infrared
    Countermeasures/Common Missile                    50 % (3 of 6) at 6
    Warning System                              5.6            or higher
    C-5 Reliability Enhancement                      100 % (11 of 11) at
    and Reengining Program                      2.1          6 or higher
    DD(X) Destroyer                                  25 % (3 of 12) at 6
                                                 417.3            or higher
    Future Combat System                             32 % (17of 52) at 6
                                                  50.8            or higher
    Joint Strike Fighter                             25 % (2 of 8) are 6
                                                  30.1            or higher

    Source: GAO.

    Note: Technology readiness level of 7 or higher at program launch is
    considered best practice; a technology readiness level of 6 or higher is
    DOD standard.

    Although the majority of respondents to our survey believed that the
    initial baselines of their programs were reasonable, a significant group,
    about 24 percent of program managers, responded that their program
    parameters were not reasonable at the start and 45 program managers
    responded that their program had been re-baselined one or more times for
    cost and schedule increases; 18 percent said one or more key technologies
    fell below a readiness level of 7, which is proven to work in an
    operational environment. They also noted that the most frequently missing
    critical skill was systems engineering-a key function for matching
    requirements to the technologies needed and for providing reasonable
    baselines at the beginning of development. In addition, in written
    comments and individual interviews, program managers noted pressure to
    agree to cost commitments that could be attained only if programs enjoyed
    higher-level support. They also noted that requirements were often not
    fully defined at the onset of a program, and many also pointed out that
    users and stakeholders often did not stick to the agreements they made
    when programs were launched, especially if technologies did not mature as
    planned.

    Figure 7: To What Extent Were the Parameters of Your Program Reasonable at
    Program Start?

    Figure 8: How Program Managers Responded to an Open-ended Question on What
    Were the Biggest Obstacles They Faced

    Program managers' views were mixed when it came to whether human capital
    resources were well matched to new programs. They cited major improvements
    in DOD's training programs and credited cross-functional teams as a
    valuable resource. They also generally believed they personally had the
    right mix of experience and training to do their jobs well. Ninety-four
    percent of the program managers responding to our survey reported that
    they had been certified at the highest level for program management by
    DOD's Defense Acquisition University. More than 80 percent also believed
    they had adequate training in the areas of systems engineering, business
    processes, contracting, management, program representation, cost control,
    and planning and budgeting. Slightly less, about 76 percent, believed they
    had enough leadership training. In addition, about 92 percent said that
    they believed that their service consistently assigned people with the
    skills and experience to be effective program managers.

    At the same time, however, program managers comments and interviews with
    program executive officers pointed to critical skill shortages for staff
    that support them-including program management, systems engineering, cost
    estimating, and software development. Some of these officials attributed
    the shortages to shifts in emphasis from oversight to insight of
    contractor operations. Lastly, in their written comments, about 18 percent
    of program managers who provided written comments cited shortcomings in
    their career path, such as lack of opportunities at the general officer
    level and requirements to move often as a disincentive; 13 percent cited
    the lack of financial incentives. Some program managers also noted that
    DOD loses opportunities to retain valuable experience, merely because
    there are no formal incentives for military officers to stay on as program
    managers after they are eligible for retirement. Civilians in program
    management also cited a lack of career opportunities; one problem cited
    was having to find their next job in contrast to military program
    managers, whose subsequent job is presented to them.

         Execution in DOD Does Not Provide Adequate Support and Accountability

    According to program managers and our past reviews, the execution process
    does not provide adequate support and accountability. In particular,
    knowledge-based development processes are not employed, program managers
    are not empowered to execute, and they are not held accountable for
    delivering programs within targets.

    More specifically, DOD has encouraged following knowledge-based
    development processes in its acquisition policy but not always in
    practice. The acquisition process itself mirrors many aspects of the
    commercial companies. For example, it requires a variety of senior,
    functional, and program-level personnel to come together, assess progress,
    identify problems, and make go/no-go decisions at key points in
    development. It encourages oversight personnel to base these decisions on
    quantifiable data and demonstrated knowledge. To enhance product
    development, DOD has also been attempting to adopt and improve policies in
    areas such as software development and systems engineering.

    However, program managers who responded to our survey believe that the
    acquisition process does not enable them to succeed because it does not
    empower them to make decisions on whether the program is ready to proceed
    forward or even to make relatively small trade-offs between resources and
    requirements as unexpected problems are encountered. Program managers
    assert that they are also not able to make shifts within personnel to
    respond to changes affecting the program. At the same time, program
    managers commented that requirements continue to be added as the program
    progresses and funding instability continues throughout. These two factors
    alone cause the greatest disruption to programs, according to program
    managers. Compounding this problem is the fact that because acquisition
    programs tend to attract funds over other activities, including science
    and technology efforts that ultimately support acquisition, program
    managers are incentivized to take on tasks that really should be
    accomplished within a laboratory environment, where it is easier and
    cheaper to discover and address technical problems.

    With many factors out of their span of control, program managers in our
    focus groups also commented that it was difficult to hold them accountable
    for mistakes. In addition, in their written comments to the survey, many
    program managers expressed frustration with the time required of them to
    answer continual demands for information from oversight officials-many of
    which did not seem to add value. Some program managers in fact estimated
    that they spent more then 50 percent of their time producing and tailoring
    and explaining status information to others.

    More broadly, in interviews and written comments, many program managers
    and program executive officials said that did not believe that DOD's
    acquisition process really supported or enabled them. Instead, they viewed
    the process as cumbersome and the information produced as non-strategic.
    When strategic plans or useful analyses were produced, they were done so
    apart from the acquisition process.

    Our own reviews have pointed to a number of structural problems with the
    acquisition process.5 In particular, while DOD's acquisition policy has
    embraced best practice criteria for making decisions, it does not yet have
    the necessary controls to ensure knowledge is used for decision-making
    purposes. As a result, programs can move forward into design, integration,
    and production phases without demonstrating that they are ready to.
    Without a means to ensure programs and senior managers are adhering to the
    process, the process itself can become an empty exercise-and, in the view
    of program managers, a time-consuming one.

    Table 4 highlights differences between DOD and commercial knowledge-based
    development support and accountability factors-collectively from the
    perspective of program managers, our past reports, and observations we
    made during the course of the review.

    5 Defense Acquisitions: DOD's Revised Policy Emphasizes Best Practices but
    More Controls Are Needed, GAO-04-53 (Washington, D.C.: Nov. 17, 2003).

    Table 4: Are Best Practices Present in DOD?

    Best practices           DOD
    Base decisions on        DOD policy encourages decisions to be based
    quantifiable data and    on quantifiable data and demonstrated
    demonstrated knowledge   knowledge, but not happening in practice.
    Empower program managers Program managers say they are not empowered
    to make decisions        in the same way as commercial companies.
                                They do not control resources. They do not
                                have authority to move programs to next
                                phases.
    Hold program managers    Difficult to enforce accountability.
    accountable
    Program managers stay    Tenure has been lengthened, but program
    through execution        managers generally do not stay after 3 to 4
                                years.

    Source: GAO.

    Data Supporting Oversight and Management Decisions

    We reported that while DOD's acquisition policy has embraced best practice
    criteria for making decisions, it does not yet have the necessary controls
    to ensure demonstrable data is used for decision-making purposes. We
    recommended that DOD assure that program launch decisions capture
    knowledge about cost and schedule estimates based on analysis from a
    preliminary design using systems engineering tools. In transitioning from
    system integration to system demonstration, we recommended that DOD ensure
    the capture of knowledge about the completion of engineering drawings;
    completion of subsystem and system design reviews; agreement from all
    stakeholders that the drawings are complete; and identification of
    critical manufacturing processes, among other indicators. And in
    transition to production, we recommended that DOD capture knowledge about
    production and availability of representative prototypes along with
    statistical process control data.

    We recommended adopting these controls because, in our view, they would
    help set program managers up for success by (1) empowering them with
    demonstrated knowledge as they move toward production and (2) bringing
    accountability to their positions and making the business case more
    understandable. Without these types of controls, the process can become an
    empty and time-consuming exercise in the view of program managers. At
    present, our reports continue to show that programs are allowed to proceed
    without really showing that they are ready to. In our most recent annual
    assessment of major weapon systems, for example, only 42 percent of
    programs had achieved design stability at design review and almost none of
    the programs in production or nearing production planned to assure
    production reliability through statistical control of key processes.

    Our survey also indicated that a relatively small percentage of programs
    used knowledge indicators that successful commercial companies use. For
    example, in responding to our survey, only 32 percent of program managers
    said they used design drawing completion extensively to measure design
    maturity; only 26 percent said they used production process controls to a
    great extent. Even fewer program managers reported that their immediate
    supervisor used these measures extensively to evaluate progress.

    In our survey and interviews, program managers and program executive
    officers also frequently commented that they spend too much time preparing
    data for oversight purposes that is not strategic or very useful to them.
    In fact, more than 90 percent of survey respondents said that they spent
    either a moderate, great, or very great extent of their time representing
    their program to outsiders and developing and generating information about
    program progress. In addition, program managers told us that they do not
    have standard tools for preparing program-status data. Instead, they must
    hand-tailor data to the requester's particular demands for format and
    level of detail. The Air Force was cited by some program managers as
    taking initiative in developing a database (known as the System Management
    Analysis Reporting Tool) that could save time in answering internal
    oversight demands for data, but they also wanted to be able to use such a
    tool to answer outside demands. While program managers said they were
    spending a great deal of time reporting on program status to outsiders,
    some program executive officers and program managers also commented that
    they had to separately produce data, analyses, and strategic plans for
    their own purposes in order to keep their programs on track-the types of
    plans and analyses that they used were simply not called for by the
    process itself. One program executive officer said that he used three
    documents, the approved program baseline, the acquisition strategy, and
    the test plan to evaluate the program manager's plans-all of these
    documents and many more are required under current acquisition
    planning-but these were of most significance. In addition the executive
    officer held a one-day review per quarter with each program manager and
    reviewed metrics such as earned value, use of award fee, contract growth,
    and schedule variation.

    Program Manager Authority

    In several key areas, program managers said that they do not have the
    necessary authority to overcome obstacles and make trade-offs to achieve
    program goals. About 60 percent of the program managers that responded to
    our survey said that program managers should have more authority to manage
    their programs-particularly when it comes to funding, deciding when
    programs are ready to proceed to the next phase, and shifting staff. In
    interviews and written comments, some program managers commented that they
    were seeking the ability to make relatively small trade-offs-for example,
    moving a staff member from one section of a program to another and
    shifting a small amount of funds from procurement accounts to research and
    development accounts, while others advocated for greater authority, as
    long as their program stayed on track. In addition, program managers often
    commented that they should have a larger role in requirements decisions
    that are made after a program is started-specifically, the ability to veto
    new requirements that would put too much strain on the program. A few
    program managers we interviewed, however, believed that they did have
    sufficient authority and that many program managers have not learned how
    to exercise it or are risk averse. Others commented that program managers
    were simply not allowed by senior managers to exercise their authority. At
    the same time, program executive officers, who manage a set of programs,
    commented in interviews that they also lacked authority over simple
    matters such as moving staff or shifting small amounts of funds. Lastly,
    in our focus groups and in written comments, program managers who
    specifically worked for the Missile Defense Agency indicated that they did
    have authority to make trade-offs among cost, schedule, and performance
    and to set requirements for the business case. They found that this
    authority alone greatly separated their current positions from past
    program manager positions and consistently cited it as a major enabler.

    Table 5 shows how program managers answered survey questions regarding the
    types of formal and information authority they have. Figure 9 highlights
    comments that were provided by program managers.

    Table 5: Program Manager Views on Formal vs. Informal Authority

    In percent
                                 I have formal I have informal
    Type of authority            authoritya       authority No authority
    Developing program
    requirements                         10              82            7
    Changes in program
    requirements                         13              85            2
    Flexibility within
    program to reallocate
    funding                              81              15            5
    Developing technology                42              45            9
    Setting testing
    requirements                         48              49            2
    Selecting contractor
    sources                              48              33           11
    Administering contracts              60              37            3
    Addressing difficulties
    to meet requirements                 66              31            2

    Source: GAO.

    aNote: Numbers may not total 100 percent due to rounding.

    Figure 9: Highlights of Program Manager Comments on What Types of
    Authority They Need

    Accountability

    Program manager views with regard to accountability are mixed. In our
    interviews and our focus groups, many program managers stated they
    personally held themselves accountable; however, many also commented that
    it is difficult to be accountable when so much is outside their span of
    control. During our focus groups, program managers cited sporadic
    instances when program managers were removed from their positions or
    forced to retire. They also cited instances when a program manager was
    promoted, even though the program was experiencing difficulties. In their
    written comments for our survey, program managers often commented that it
    was a disincentive that senior leaders who were impacting their program
    negatively were not being held accountable.

    We observed some key differences between the commercial companies we
    visited and DOD when it comes to accountability.

               o  Commercial companies make it very clear who is accountable on a
               program and for what. Goals that must be achieved are clearly
               spelled out and understood by the entire program team. In DOD, it
               is not always clear who is responsible. Moreover, the expectations
               set for program managers by their supervisors may not necessarily
               match up with the goals of their program-particularly when the
               program manager is a military officer who reports to both a PEO
               and another commanding official.

               o  Program managers and senior managers in the commercial sector
               are required to stay with programs until they are done; at DOD
               they are not.

               o  Program managers in the commercial sector are incentivized to
               stay with programs and be accountable for them-principally through
               empowerment and financial incentives, but also through their
               desire to help the company achieve its goals. At DOD, program
               managers strongly asserted that they are incentivized to help the
               warfighter, but few said they were incentivized by financial or
               promotion incentives or by empowerment.

               In commenting on senior leader support during program execution,
               program managers had mixed views on whether they received
               sustained commitment from their program executive officers, but
               widely believed that they did not receive sustained commitment
               from other senior leaders and stakeholders-unless their programs
               enjoyed priority and support from very high level officials,
               Congress, or the President. More often than not, programs
               struggled to compete for funding and were continually beset by
               changing demands from users. Others noted that while DOD is
               emphasizing jointness in programs more and more, collaboration
               among senior leaders needed to achieve jointness is not always
               happening. Some program managers lamented that they felt they were
               not respected in DOD, while others believed their service was
               taking some positive actions to put program managers on a par with
               military officers in operational positions.

               Program managers were also troubled by constant demands for
               information for oversight purposes as well as interruptions from
               stakeholders (for example, in department-wide budget or testing
               offices) that seemed to be non value-added. As we noted earlier,
               over 90 percent of the survey respondents said that they spent
               either a moderate, great, or very great extent of their time
               representing their program to outsiders and developing and
               generating information about program progress.

               Several program managers also cited reluctance on the part of
               senior managers to hear bad news. Our past reviews have similarly
               noted that the overall competition for funding in DOD spurs
               program managers to suppress bad news because it can result in
               funding cuts.

               Differences between DOD and leading companies in how program
               managers are supported and held accountable are rooted in
               differences in incentives and resulting behaviors. This begins
               with the definition of success. The commercial firms we studied
               concluded their survival hinged on their ability to increase their
               market share, which, in turn, meant developing higher-quality
               products, at the lowest possible price, and delivering them in a
               timely fashion-preferably before their competitors could do the
               same. This imperative meant that they had no choice but to narrow
               the gap between requirements and resources in a manner that not
               only ensured they met their market targets, but did so in a manner
               that consumed resources efficiently. It also meant that they had
               no choice but to fully support the development effort, to instill
               strategic planning and prioritization, to work collaboratively, to
               follow a knowledge-based process that makes product development
               manageable, and ultimately, assign accountability to all involved
               for success or failure. In theory, DOD's success likewise hinges
               on its ability to deliver high quality weapons to the warfighter
               in a timely fashion. But in practice, the implied definition of
               success is the ability of a program to win support and attract
               funds. Of course, there are reasons for this disconnect. Corporate
               revenue is generated by customer sales while DOD's funding is
               dependent on annual appropriations. Corporations go out of
               business when their product development efforts do not succeed;
               DOD does not. Selling products to customers is the single focus of
               a private-sector company while DOD is charged with a myriad of
               important missions-each of which also competes for budget share.
               As a result, program managers are incentivized to overpromise on
               performance because it makes their program stand out from others.
               They are incentivized to underestimate cost and schedule and to
               suppress bad news because doing otherwise could result in a loss
               of support and funding and further damage their program. In short,
               unknowns become acceptable and desirable rather than unacceptable
               as they are in the corporate environment. And accountability
               becomes much more difficult to define.

               Figure 10: Key Differences in Definition of Success and Resulting
               Behaviors

               Success for the commercial world is straightforward and simple:
               maximize profit. In turn, this means selling products to customers
               at the right price, right time, and right cost. Each of the
               commercial companies we visited enjoyed success to this end, but
               at some point in time, as competitors made gains, markets
               tightened, and the pace of technology changes grew faster, they
               realized they needed to do more to be successful. Toyota decided
               it needed to expand its role in the world market place and this
               need persisted as competition grew stronger over the years. For
               Siemens this realization came in the 1990s-when Siemens Medical
               Division took a hard look at its profitability for its medical
               devices and for Motorola in the 1980's when it began losing market
               share for its communication devices. To turn themselves around,
               all three companies chose not to depend on technology
               breakthroughs or exotic marketing, but rather to improve their
               position by looking inward at how they approached development.
               Each found that there was room for improvement, starting with
               corporate cultures and ending with processes and controls. In
               Toyota's case, emphasis was largely placed on collaboration and
               consistency. In Siemens case, emphasis was placed on quality,
               particularly because its medical products come under extensive
               Food and Drug Administration regulations. For Motorola, emphasis
               was placed on empowerment and commonality, particularly in the
               processes that support product development like software
               development.

               At DOD, success is often formally defined in similar terms as the
               commercial world: deliver high quality products to customers (the
               warfighter) at the right time and the right cost. Virtually all
               program managers we spoke with first defined success in terms of
               enabling warfighters and doing so in a timely and cost-efficient
               manner. But when the point was pursued further, it became clear
               that the implied definition for success in DOD is attracting funds
               for new programs, and keeping funds for ongoing programs. Program
               managers themselves say they spend enormous amounts of time
               retaining support for their efforts and that their focus is
               largely on keeping funds stable. They also observe that DOD starts
               more programs than it can afford to begin with, which merely sets
               the stage for competition and resulting behaviors. As noted
               earlier, there are factors that contribute to how success is
               defined in practice, including the fact that DOD depends on annual
               appropriations and it must fund a wide variety of missions beyond
               weapon systems development. However, according to program
               managers, the willingness to make trade-off decisions alone, would
               help DOD mitigate these circumstances.

               Regardless of where they placed greatest emphasis, each company we
               studied adopted processes and support mechanisms that emphasized
               the following:

               o  risk reduction,
               o  knowledge-based decisionmaking,
               o  discipline,
               o  collaboration,
               o  trust,
               o  commitment,
               o  consistency,
               o  realism, and
               o  accountability.

               Such characteristics were seen as absolutely essential to gaining
               strength in the market place. With limited opportunities to invest
               in new product development efforts, companies understand it is
               essential, for example, that they know they are pursuing efforts
               that will optimize profits. Therefore, estimates of costs and
               technology maturity must be accurate and they must be used for
               making decisions. Consistency and discipline are integral to
               assuring that successful efforts can be repeated. Ultimately,
               these characteristics translate into processes that help companies
               develop products quicker, cheaper, and better. At the strategic
               level, processes include accurate, strategic planning and
               prioritization to ensure the right mix of products are pursued;
               investment strategies that prioritize projects for funding; and
               strong systems engineering to help them establish a realistic
               business case that levels market needs with available resources
               prior to beginning a product development. At the tactical level,
               this includes knowledge-based developments that center on
               designing and manufacturing products that will sell well enough to
               make an acceptable profit. This combination of focused leadership
               and disciplined processes promotes positive behaviors, such as an
               insistence that technology development take place separately from
               product development programs and trade-offs between requirements
               and resources be made before beginning a program; it promotes an
               atmosphere of early candor and openness from everyone as to
               potential program risks; and underscores the need for realistic,
               knowledge-based cost and schedule estimates to support full
               funding decisions; and the ability to test early, allowing "red
               lights" for problems that must be proven solved before they can be
               changed to "green lights."

               Once attracting and sustaining funds becomes a part of the
               definition of success, as it is at DOD, different values and
               behaviors emerge. For example, it is not necessarily in a program
               manager's interest to develop accurate estimates of cost,
               schedule, and technology readiness, because honest assessments
               could result in lost funding. Delayed testing becomes preferred
               over early testing since that will keep "bad news" at bay.

               Ultimately, no matter how well-intentioned or what improvements
               are made, DOD's processes and support mechanisms eventually play
               into funding competition. On paper, the requirements process may
               emphasize realism and the importance of incremental development,
               but in practice, it consistently encourages programs to promise
               performance features that significantly distinguish them from
               other systems. Likewise, changes may be made to make the funding
               process more strategic, but because there are still many programs
               competing for funds, it encourages cost and schedule estimates to
               be comparatively soft with little benefit from systems engineering
               tradeoffs. By favoring acquisition programs over science and
               technology efforts, the funding process also encourages programs
               to take on technology development that should be carried out in
               research labs. Lastly, the acquisition process may adopt
               world-class criteria for making decisions, but because it is much
               easier to attract funds for a formal weapons program than funds
               for the exercise of gaining knowledge about technologies, the
               process encourages programs to move forward despite risks with the
               assumption that programs can resolve technical, design, or
               production "glitches" later on. Significant unknowns are accepted
               in this environment. Delivering a product late and over cost does
               not necessarily threaten program success. The cumulative effect of
               these pressures is unpredictable cost and schedule estimates at
               the outset of a program that are breached, sometimes very
               significantly, by the time the weapon system is fielded.

               There are other environmental differences that put additional
               pressures on program managers within DOD. They include layers of
               internal and external oversight that come with DOD's stewardship
               responsibilities, personnel rules that make it more difficult to
               manage human capital and hold people accountable, laws and
               regulations that place additional constraints on an acquisition,
               and the mere size and scope of DOD, which adds significant
               challenges to communicating and collaborating effectively.

               For example, as shown below, commercial companies we visited
               tended to have fairly streamlined oversight. No matter what level
               the program manager resided, they had access to top executives who
               were empowered to help them make go/no-go decisions. In addition
               to this structure, the companies were governed by a degree of
               oversight from shareholders, but this was not actualized in
               day-to-day management of program development activities. At DOD,
               by contrast, program managers operate under many layers of
               oversight-both internally and externally. These include Congress,
               the President, the Secretary of Defense, a myriad of functional
               agencies as well as the military services-all of whom have a say
               in DOD's overall budget as well as funding for specific programs.
               Moreover, within these confines, leaders at all levels shift
               frequently. Much of this oversight is necessary for carrying out
               stewardship responsibilities for public money, but studies
               conducted by a variety of commissions assessing acquisition
               problems through the years have consistently found that there are
               opportunities to reduce oversight layers and streamline oversight
               processes and protect programs from frequent leadership changes.
               Program managers themselves understood the need for oversight, but
               found that responding to oversight demands was taking too much of
               their time. They also identified opportunities to make it easier
               to work within this environment, including development of
               databases to support internal and external oversight requests,
               empowering program managers for more day-to-day decisions, and
               making stakeholders more accountable.

                         Senior Leader Support during Execution

    Chapter 4: Basic Incentives Drive Differences in How Program Managers Are
    Supported and Held Accountable  Chapter 4: Basic Incentives Drive
    Differences in How Program Managers Are Supported and Held Accountable

                                 Definition of Success

                                   Means for Success

           Other Differences Put Additional Pressures on DOD Program Managers

    Figure 11: Commercial vs. DOD Oversight Environments

    Program managers also cited several trends that have increased pressures
    that they face. These include DOD's movement toward developing more
    technical complex families of weapon systems as one package (system of
    systems), which they believe vastly increases management challenges and
    makes it more difficult to oversee contractors and DOD's reduction in
    acquisition workforces over the past decade, which has made it more
    difficult to carry out day-to-day responsibilities and retain technical
    and business expertise. Overall, however, program managers themselves
    consistently attribute their problems to competition for funding over
    these other factors.

    Chapter 5: Conclusions and Recommendations  Chapter 5: Conclusions and
    Recommendations

    Like the commercial world, DOD has a mandate to deliver high-quality
    products to its customers, at the right time, and the right price. Quality
    and time are especially critical to maintain DOD's superiority over
    others, to counter quickly changing threats, and to better protect and
    enable the warfighter. Cost is critical given DOD's stewardship over
    taxpayer money, long-term budget forecasts which indicate that the nation
    will not be able to sustain its current level of investment in weapon
    systems, and DOD's desire to dramatically transform the way it conducts
    military operations. At this time, however, DOD is not positioned to
    deliver high quality products in a timely and cost-efficient fashion. It
    is not unusual to see cost increases that add up to tens of millions of
    dollars, schedule delays that add up to years, and large and expensive
    programs to be continually rebaselined. Recognizing this dilemma, DOD has
    tried to embrace best practices in its policies, instill more discipline
    in requirements setting, strengthen training for program managers and has
    lengthened program manager tenures. It has also reorganized offices that
    support and oversee programs, required programs to use independent cost
    estimates and systems engineering, and it has alternately relaxed and
    strengthened oversight over contractors in an effort to extract better
    performance from them. Yet despite these and many other actions, programs
    are still running over cost and over schedule and, in some cases, changes
    have merely added tasks for program managers while adding no value.

    Our work shows that this condition will likely persist until DOD provides
    a better foundation on which program managers can launch programs and more
    consistent and steadfast support once it commits to programs. At the core
    of this solution is developing and enforcing an investment strategy that
    prioritizes programs based on customer needs and DOD's long term vision
    and reduces the burden of advocacy on the part of the program manager. DOD
    will always be facing funding uncertainties due to the environment it
    operates in. But it has an opportunity to greatly mitigate the risks that
    come with this environment by separating long-term wants from needs,
    matching them up against what technologies are available today, tomorrow,
    and decades from now, as well as being realistic in determining what
    resources can be counted on. Without an investment strategy, all other
    improvements will likely succumb to the negative incentives and behaviors
    that come with continual competition for funding. With an investment
    strategy, senior leaders will be better positioned to formally commit to a
    business case that assures new programs fit in with priorities, that they
    begin with adequate knowledge about technology, time, and cost, and that
    they will follow a knowledge-based approach as they move into design and
    production. Another core enabler for improving program managers' chances
    for success lies in leadership's ability to implement evolutionary,
    incremental acquisition programs that have limited cycle times from
    beginning to delivery of the weapon system. This allows DOD to align
    program managers' tenures to delivery dates, thereby substantially
    increasing accountability for successful outcomes.

    Once senior leaders do their part-by providing program managers with an
    executable business case and committing their full support to a
    program-they can begin to empower program managers more and hold them
    accountable. By embracing a model for supporting program managers that
    incorporates all these elements, DOD can achieve the same outcomes for its
    weapons programs as other high-performing organizations.

                          Recommendations for Executive Action

    We recommend that the Secretary of Defense take the following actions to
    ensure program managers are well positioned to successfully execute and be
    held accountable for weapon acquisitions:

               o  DOD should develop an investment strategy that, at a minimum,

                            o  determines the priority order of needed
                            capabilities with a corollary assessment of the
                            resources, that is dollars, technologies, time and
                            people needed to achieve these capabilities. The
                            remaining capabilities should be set out separately
                            as desirable, resources permitting.
                            o  lays out incremental product development programs
                            for achieving desired capabilities, and
                            o  establishes controls to ensure that requirements,
                            funding, and acquisition processes will work together
                            so that DOD will sustain its commitment to its
                            priority programs.

               As DOD works to develop the strategy, it should take an interim
               step by identifying priorities for programs that are already past
               milestone B (the formal start of development). Once the strategy
               is complete, it should be used by the Office of the Secretary of
               Defense to prepare and assess annual budget proposals as well as
               to balance funding between science and technology efforts and
               acquisition efforts to ensure that robust technology development
               efforts are conducted, but outside the acquisition program until
               reaching maturity.

               o  For each new major weapons program, require that senior-level
               officials from the requirements, science and technology, program
               management, testing communities as well as the Office of the
               Comptroller formally commit to a business case prior to approving
               a program at milestone B. At a minimum, the business case should
               demonstrate that

                            o  a requirement exists that warrants a materiel
                            solution consistent with national military strategy,
                            o  an independent analysis of alternatives has been
                            conducted
                            o  the developer has the requisite technical
                            knowledge to meet the requirement,
                            o  the developer has a knowledge-based product
                            development and production plan that will attain high
                            levels of design and production maturity,
                            o  reasonable estimates have been developed to
                            execute the product development and production plan,
                            and
                            o  funding is available to execute the plan.

               o  Develop and implement a process to instill and sustain
               accountability for successful program outcomes. At a minimum, this
               should consider

               In commenting on a draft of our report, DOD's Acting Director for
               Procurement and Acquisition Policy concurred with our
               recommendations. In doing so, DOD asserted it was already taking
               actions to address our recommendations, notably by reviewing its
               overall approach to acquisition governance and investment
               decisionmaking as part of its Quadrennial Defense Review due in
               February 2006 and identifying ways to more effectively implement
               existing policies. DOD also stated it intended to develop a plan
               to address challenges in acquisition manpower including program
               manager tenure and career path and it intends to enhance its
               information management systems to facilitate oversight and
               management decisions. As underscored in our report, DOD has
               attempted similar efforts in the past-that is, reviewed its
               approach to governance and investment decisions and
               policies-without achieving significant improvements. This is
               because DOD has not assured such actions were executed in tandem
               with (1) instilling more leadership and discipline in investment
               decisionmaking, in both the short and long term and (2) instilling
               accountability-by requiring key senior officials to sign a
               business case, based on systems engineering knowledge, prior to
               every new acquisition as well as by matching program managers'
               tenure to cycle time. Therefore, in pursuing the actions it
               identifies in its response to our report, we believe that DOD
               should address the important questions of who should be held
               accountable for acquisition decisions; how much deviation from the
               original business case is allowed before it is no longer
               considered valid and the investment reconsidered; and what is the
               penalty when investments do not result in meeting promised
               warfighter needs.

               The full text of the department's response is in appendix I.

               Michael J. Sullivan (202) 512-4841 or sullivanm@gao.gov

               Greg Campbell, Cristina Chaplain, Ron La Due Lake, Sigrid McGinty,
               Jean McEwen, Carol Mebane, Guisseli Reyes, Lesley Rinner, Lisa
               Simon, Bradley Trainor, and Michele Williamson.

               DOD Acquisition Outcomes: A Case for Change. GAO-06-257T.
               Washington, D.C.: November 15, 2005.

               Defense Acquisitions: Stronger Management Practices Are Needed to
               Improve DOD's Software-Intensive Weapon Acquisitions. GAO-04-393.
               Washington, D.C.: March 1, 2004.

               Best Practices: Setting Requirements Differently Could Reduce
               Weapon Systems' Total Ownership Costs. GAO-03-57. Washington,
               D.C.: February 11, 2003

               Best Practices: Capturing Design and Manufacturing Knowledge Early
               Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July
               15, 2002.

               Defense Acquisitions: DOD Faces Challenges in Implementing Best
               Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.

               Best Practices: Better Matching of Needs and Resources Will Lead
               to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.:
               March 8, 2001.

               Best Practices: A More Constructive Test Approach Is Key to Better
               Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July
               31, 2000.

               Defense Acquisition: Employing Best Practices Can Shape Better
               Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.:
               April 26, 2000.

               Best Practices: DOD Training Can Do More to Help Weapon System
               Programs Implement Best Practices. GAO/NSIAD-99-206. Washington,
               D.C.: August16, 1999.

               Best Practices: Better Management of Technology Development Can
               Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington,
               D.C.: July 30, 1999.

               Defense Acquisitions: Best Commercial Practices Can Improve
               Program Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17,
               1999.

               Defense Acquisition: Improved Program Outcomes Are Possible.
               GAO/T-NSIAD-98-123. Washington, D.C.: March 17, 1998.

               Best Practices: DOD Can Help Suppliers Contribute More to Weapon
               System Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17,
               1998.

               Best Practices: Successful Application to Weapon Acquisition
               Requires Changes in DOD's Environment. GAO/NSIAD-98-56.
               Washington, D.C.: February 24, 1998.

               Best Practices: Commercial Quality Assurance Practices Offer
               Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August
               26, 1996.

               The Government Accountability Office, the audit, evaluation and
               investigative arm of Congress, exists to support Congress in
               meeting its constitutional responsibilities and to help improve
               the performance and accountability of the federal government for
               the American people. GAO examines the use of public funds;
               evaluates federal programs and policies; and provides analyses,
               recommendations, and other assistance to help Congress make
               informed oversight, policy, and funding decisions. GAO's
               commitment to good government is reflected in its core values of
               accountability, integrity, and reliability.

               The fastest and easiest way to obtain copies of GAO documents at
               no cost is through GAO's Web site (www.gao.gov). Each weekday, GAO
               posts newly released reports, testimony, and correspondence on its
               Web site. To have GAO e-mail you a list of newly posted products
               every afternoon, go to www.gao.gov and select "Subscribe to
               Updates."

               The first copy of each printed report is free. Additional copies
               are $2 each. A check or money order should be made out to the
               Superintendent of Documents. GAO also accepts VISA and Mastercard.
               Orders for 100 or more copies mailed to a single address are
               discounted 25 percent. Orders should be sent to:

               U.S. Government Accountability Office 441 G Street NW, Room LM
               Washington, D.C. 20548

               To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
               (202) 512-6061

               Contact:

               Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
               fraudnet@gao.gov Automated answering system: (800) 424-5454 or
               (202) 512-7470

               Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400
               U.S. Government Accountability Office, 441 G Street NW, Room 7125
               Washington, D.C. 20548

               Paul Anderson, Managing Director, AndersonP1@gao.gov (202)
               512-4800 U.S. Government Accountability Office, 441 G Street NW,
               Room 7149 Washington, D.C. 20548

                            o  matching program manager tenure with delivery of a
                            product or for system design and demonstration,
                            o  tailoring career paths and performance management
                            systems to incentivize longer tenures,
                            o  empowering program managers to execute their
                            programs, including an examination of whether and how
                            much additional authority can be provided over
                            funding, staffing, and approving requirements
                            proposed after milestone B,
                            o  developing and providing automated tools to
                            enhance management and oversight as well as to reduce
                            time required to prepare status information.

                           Agency Comments and Our Evaluation

    Appendix I: Comments from the Department of Defense  Appendix I: Comments
    from the Department of Defense

    A  Appendix II: GAO Staff Acknowledgments

    GAO Contacts

    Staff Acknowledgments

    Related GAO Products  Related GAO Products

    (120320)

    GAO's Mission

    Obtaining Copies of GAO Reports and Testimony

    Order by Mail or Phone

    To Report Fraud, Waste, and Abuse in Federal Programs

    Congressional Relations

    Public Affairs

    www.gao.gov/cgi-bin/getrpt?GAO-06-110
    www.gao.gov/cgi-bin/getrpt?GAO-06-112SP

    To view the full product, including the scope

    and methodology, click on the links above.

    For more information, contact Michael J. Sullivan at (202) 512-4841 or
    sullivanm@gao.gov.

    Highlights of GAO-06-110, a report to the Subcommittee on Readiness and
    Management Support, Committee on Armed Services, U.S. Senate

    November 2005

    BEST PRACTICES

    Better Support of Weapon System Program Managers Needed to Improve
    Outcomes

    The Department of Defense (DOD) relies on a relatively small cadre of
    officials to develop and deliver weapon systems. In view of the importance
    of DOD's investment in weapon systems, we have undertaken an extensive
    body of work that examines DOD's acquisition issues from a perspective
    that draws lessons learned from the best commercial product development
    efforts to see if they apply to weapon system acquisitions. In response to
    a request from the Chairman and Ranking Minority Member of the
    Subcommittee on Readiness and Management Support, Senate Committee on
    Armed Services, this report assesses (1) how successful commercial
    companies position their program managers, (2) how DOD positions its
    program managers, and (3) underlying reasons for the differences.

    In compiling this report, GAO conducted a survey of program managers. See
    GAO-06-112SP.

    What GAO Recommends

    GAO recommends the Secretary of Defense develop an investment strategy to
    prioritize needed capabilities; require senior stakeholders to formally
    commit to business cases for new weapon system developments; and develop a
    process to instill and sustain accountability for successful program
    outcomes. DOD agreed with our recommendations.

    U.S. weapons are among the best in the world, but the programs to acquire
    them often take significantly longer and cost more money than promised and
    often deliver fewer quantities and capabilities than planned. It is not
    unusual for estimates of time and money to be off by 20 to 50 percent.
    When costs and schedules increase, quantities are cut, and the value for
    the warfighter-as well as the value of the investment dollar-is reduced.

    When we examined private sector companies that developed complex and
    technical products similar to DOD, we found that their success hinged on
    the tone set by leadership and disciplined, knowledge-based processes for
    product development and execution. More specifically, long before the
    initiation of a new program, senior company leaders made critical
    investment decisions about the firm's mix of products so that they could
    commit to programs they determined best fit within their overall goals.
    These decisions considered long-term needs versus wants as well as
    affordability and sustainability. Once high level investment decisions
    were made, senior leaders ensured that programs did not begin unless they
    had a business case that made sure resources were in-hand to execute the
    program-that is, time, technology, money, and people. Once a business case
    was established, senior leaders tasked program managers with executing
    that business case for each new product from initiation to delivery, but
    required their program managers to use a knowledge-based product
    development process that demanded appropriate demonstrations of
    technology, designs, and processes at critical junctures. The program
    manager was empowered to execute the business case, but also held
    accountable for delivering the right product at the right time for the
    right cost. Requiring the program manager to stay throughout the length of
    a project was a principal means of enforcing accountability. Overall, by
    providing the right foundation and support for program managers, the
    companies we visited were able to consistently deliver quality products
    within targets, and in turn, transform themselves into highly competitive
    organizations.

    DOD program managers are put in a very different situation. DOD leadership
    rarely separates long-term wants from needs based on credible, future
    threats. As a result, DOD starts many more programs than it can
    afford--creating a competition for funds that pressures program managers
    to produce optimistic cost estimates and to overpromise capabilities.
    Moreover, our work has shown that DOD allows programs to begin without
    establishing a formal business case. And once they begin, requirements and
    funding change over time. In fact, program managers personally consider
    requirements and funding instability-which occur throughout the program-to
    be their biggest obstacles to success. Program managers also believe that
    they are not sufficiently empowered to execute their programs, and that
    because much remains outside of their span of control, they cannot be held
    accountable.
    *** End of document. ***