Best Practices: DOD Training Can Do More to Help Weapon System Programs
Implement Best Practices (Chapter Report, 08/16/1999, GAO/NSIAD-99-206).

The Pentagon plans to boost its annual procurement investment to about
$60 billion by fiscal year 2001. The military has high expectations for
this investment: that new weapons will be better, yet less expensive,
than their predecessors and will be developed in half of the time.
Essential to getting these kinds of results will be the adaptation of
best commercial practices that have enabled leading private-sector firms
to develop new products faster, cheaper, and better. The Defense
Department (DOD) has begun several acquisition reform initiatives based
on commercial practices to help foster these outcomes. Their success
depends greatly on the extent to which program offices apply the
practices to individual weapon system acquisitions. Training provided to
the program offices both creates a culture that is receptive to new
practices and provides the knowledge needed to implement them in the
workplace. This report evaluates the role that DOD training is playing
in implementing best practices in weapon system programs. GAO discusses
(1) the contribution DOD training makes to program offices' ability to
apply best practices, (2) the different methods used by DOD and leading
commercial firms in training on best practices, and (3) the strategic
approaches that underlie DOD's and leading commercial firms' training
methods for best practices.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  NSIAD-99-206
     TITLE:  Best Practices: DOD Training Can Do More to Help Weapon
	     System Programs Implement Best Practices
      DATE:  08/16/1999
   SUBJECT:  Personnel management
	     Weapons systems
	     Private sector practices
	     Defense procurement
	     Training utilization
	     Human resources training
	     Human resources utilization
IDENTIFIER:  DOD Advanced Amphibious Assault Vehicle Program
	     Advanced Medium Range Air-to-Air Missile
	     Army Battlefield Combat Identification System
	     Joint Air-to-Surface Standoff Missile
	     Joint Strike Fighter
	     Joint Surveillance Target Attack Radar System-Joint
	     Tactical Terminal

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  This text was extracted from a PDF file.        **
** Delineations within the text indicating chapter titles,      **
** headings, and bullets have not been preserved, and in some   **
** cases heading text has been incorrectly merged into          **
** body text in the adjacent column.  Graphic images have       **
** not been reproduced, but figure captions are included.       **
** Tables are included, but column deliniations have not been   **
** preserved.                                                   **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************

    United States General Accounting Office GAO                 Report
    to the Chairman and Ranking Minority Member, Subcommittee on
    Readiness and Management Support, Committee on Armed Services,
    U.S. Senate August 1999         BEST PRACTICES DOD Training Can Do
    More to Help Weapon System Programs Implement Best Practices
    GAO/NSIAD-99-206 United States General Accounting Office
    National Security and Washington, D.C. 20548
    International Affairs Division B-280234
    Letter August 16, 1999 The Honorable James Inhofe Chairman The
    Honorable Charles Robb Ranking Minority Member Subcommittee on
    Readiness and Management Support Committee on Armed Services
    United States Senate As you requested, this report assesses the
    extent the Department of Defense's (DOD) training helps weapon
    system program offices apply best practices and whether such
    training can be of more help. We make recommendations to the
    Secretary of Defense on how DOD training can better support
    program offices in the application of best practices. We are
    sending copies of this report to the Honorable William S. Cohen,
    Secretary of Defense; the Honorable Louis Caldera, Secretary of
    the Army; the Honorable Richard Danzig, Secretary of the Navy; the
    Honorable F. Whitten Peters, Acting Secretary of the Air Force;
    the Honorable Jacob J. Lew, Director, Office of Management and
    Budget; and to interested congressional committees.  We will also
    make copies available to others upon request. If you or your staff
    have any questions regarding this report, please call me at (202)
    512-4841. Contacts and key contributors to this report are listed
    in appendix III. Katherine V. Schinasi Associate Director Defense
    Acquisitions Issues Letter Executive Summary Purpose
    The Department of Defense (DOD) plans to increase its annual
    procurement investment to about $60 billion by fiscal year 2001.
    DOD has high expectations from this investment: that new weapons
    will be better yet less expensive than their predecessors and will
    be developed in half the time. Essential to getting these kinds of
    outcomes will be the adaptation of best commercial practices that
    have enabled leading commercial firms to develop new products
    faster, cheaper, and better. DOD has begun a number of acquisition
    reform initiatives based on commercial practices to help foster
    these outcomes. Their success depends greatly on the extent to
    which the program offices responsible for managing weapon
    acquisitions can implement the practices on individual programs.
    Training provided to the program offices serves as a key agent in
    both creating a culture that is receptive to new practices and in
    providing the knowledge needed to implement new practices at the
    workplace.  At the request of the Chairman and the Ranking
    Minority Member, Subcommittee on Readiness and Management Support,
    Senate Committee on Armed Services, GAO evaluated the role DOD
    training is playing in implementing best practices in weapon
    system programs. This report addresses (1) the contribution DOD
    training makes to program offices' ability to apply best
    practices, (2) the different methods used by DOD and leading
    commercial firms in training on best practices, and (3) the
    strategic approaches that underlie DOD's and leading commercial
    firms' training methods for best practices. Background
    GAO's review focused on weapon system program offices because they
    comprise a key component of DOD's acquisition workforce.  In
    planning, managing, and executing acquisition programs, these
    program offices are responsible for managing about $80 billion of
    DOD's annual research, development, and procurement funds.  As an
    entry point for DOD acquisitions, program offices greatly
    influence the work of the rest of the acquisition workforce.  The
    primary responsibility for training the acquisition workforce
    falls within the Office of the Under Secretary of Defense for
    Acquisition and Technology.  In 1992, the Defense Acquisition
    University, a consortium of 13 schools, was created to develop and
    provide training for the acquisition workforce.  Each service also
    has an acquisition reform office that provides the workforce with
    the latest information about practices and initiatives that apply
    to acquisitions. Based on personnel reductions mandated by the
    National Defense Authorization Acts for Fiscal Years 1996 and
    1997, DOD expects the acquisition workforce of fiscal year 2000 to
    be 25 percent smaller than that of fiscal year 1995. Thus,
    training Letter    Page 2
    GAO/NSIAD-99-206 Best Practices Executive Summary will become even
    more important as new authority and responsibility is granted to
    those who remain in the workforce. To determine the extent to
    which DOD training and other sources helped program offices obtain
    the knowledge needed for implementing best practices, GAO focused
    on five specific practices: cost as an independent variable,
    integrated product teams, evaluation of contractors' past
    performance, setting performance specifications, and managing
    supplier relationships. The first four are formal DOD initiatives
    that are based on best commercial practices, while supplier
    relationships is a best practice GAO has observed in leading
    commercial firms. GAO selected six program offices considered by
    DOD as leaders in implementing one or more of the practices.  As
    such, they represented best case examples of marshaling training
    and other resources needed to implement new acquisition practices.
    The term "standard training" is used in this report to describe
    the training provided by the Defense Acquisition University and
    the services' acquisition reform offices, as distinct from
    training that program offices provide on their own. Results in
    Brief        DOD's standard training did not make a major
    contribution to the leading program offices' ability to implement
    best practices. In evaluating their key sources of knowledge, none
    of the key officials from programs at the forefront of
    implementing best practices ranked standard DOD training first,
    with many ranking it last.  DOD training either did not reach the
    right people when it was needed or did not reach them at all.
    When training on best practices was received, it did not contain
    the depth or practical insights program office people needed to
    implement the practices.  It was primarily through their own
    efforts-learning on the job, finding external training, or
    developing their own training program-that they attained the
    knowledge needed to apply best practices.  Thus, success depended
    on their having the foresight to see what was needed, the
    ingenuity to find good sources of knowledge, and the resources
    needed to attain that knowledge.  Replicating this approach
    broadly on other programs is problematic.  Managers may not see
    the relevance of a practice to their programs and thus may not
    realize what training is needed. Others may not be able to afford
    the needed training. Leading commercial firms and DOD use
    different training methods to implement key practices.  Commercial
    firms use targeted, hands-on methods to ensure that program
    offices are trained on key practices.  Their training
    organizations conduct front-end analyses to determine the Letter
    Page 3                                            GAO/NSIAD-99-206
    Best Practices Executive Summary programs' training requirements
    and involve the program offices in designing the training.
    Training is customized to meet the specific needs of those
    implementing the practice.  Company officials believe the targeted
    method results in more useful training, which helps to improve
    outcomes of the final product.  DOD does not have a counterpart to
    this method. DOD relies primarily on its standard training,
    including classroom courses, videos, internet-based training,
    satellite broadcasts, and roadshows, to inform staff on best
    practices. These methods were designed for functional training,
    such as for engineers, and for increasing the awareness of new
    practices.  As such, they do not provide the depth or reach enough
    of the right people at the right time to be of great help in
    implementing best practices at program offices.  Also, DOD does
    not systematically involve program offices in the design of
    training. The intensive training methods leading commercial firms
    employ on new practices are the result of a strategic,
    institutionally driven approach to implementation.  These firms
    commit their resources and attention to a few well-defined
    practices and make a significant front-end investment in the
    training to be provided to the workforce.  Also, the firms strive
    to create an environment to put those responsible for implementing
    the practices in a good position to succeed.  DOD's training
    methods for best practices do not stem from such a strategic
    approach.  DOD has promulgated as many as 40 acquisition
    management initiatives in the past few years without communicating
    their relative priority to trainers or implementers. Often, the
    initiatives have not been accompanied by clear guidance or by the
    initial training needed for implementation. While DOD commits
    significant resources to training, it does not make a uniform
    front-end investment to ensure that program offices will succeed
    with the new practices.  Since 1997, two studies commissioned by
    DOD have recommended ways to make training organizations more
    effective in providing training of best practices. These
    recommendations were not adopted in favor of a more traditional
    training role.  In June 1999, another DOD study proposed that DOD
    training organizations become change agents and be modeled after
    their corporate counterparts.  If the latest study's
    recommendations are adopted, DOD may be in a much better position
    to provide the type of help program offices need to successfully
    implement best practices. GAO makes recommendations to the
    Secretary of Defense on how DOD's training on best practices can
    better support the needs of program offices. Page 4
    GAO/NSIAD-99-206 Best Practices Executive Summary Principal
    Findings DOD Training Not a Major        Training often did not
    exist or was not provided when program officials Catalyst for Best
    Practices     began to implement an individual practice.  For
    example, when the Joint Strike Fighter program office started to
    implement cost as an independent variable, there were no guiding
    documents, and no one, including the training community, knew what
    the initiative really meant.  Some people involved with
    implementing best practices were missed altogether by DOD training
    offerings because they had not been required to take training
    since before the initiatives began.  Others, such as requirements
    authors, fell outside the definition-and training curriculum-of
    the acquisition workforce.  In one case, program officials reduced
    contractor reporting requirements to 2 items, in line with
    acquisition reform, only to have 40 more added by another office
    with approval authority over the contract. Contractors are also
    essential to the application of best practices, but they are not
    part of the DOD defined acquisition workforce or the training
    offered.  Consequently, they do not necessarily understand or know
    how to implement new practices.  Program officials noted that
    DOD's standard training typically provided a general awareness of
    the practices but not the "how-to" knowledge needed for
    implementation.  Training was not tailored in such a way that
    individuals could see how the practices could be applied to their
    program.  Program officials also observed that in some cases
    training suffered because instructors did not use up-to-date case
    studies and were not current on new practices themselves.  They
    also believed they had little opportunity to influence the
    training they received from DOD. Program officials used a
    combination of ways-generally outside of standard DOD training
    offerings-to get the knowledge they needed to apply best
    practices.  Several program officials relied on the cumulative job
    expertise of the staff and personal research to teach themselves
    how to implement new initiatives.  For example, on the Advanced
    Medium-Range Air-to-Air Missile program, officials relied on their
    personal experiences to know how to set contract specifications at
    a performance, rather than a detailed, level.  Officials also used
    their program funds to send staff to nongovernment sources or to
    bring experts in.  One program manager sends people to outside
    training, such as university leadership courses, to develop
    creative thinking skills.  One of his managers said some of his
    best training was from off-site sessions sponsored by the program
    office that dealt with the people issues critical to making
    integrated product teams work.  For the Advanced Amphibious
    Assault Vehicle program, the prime Page 5
    GAO/NSIAD-99-206 Best Practices Executive Summary contractor was
    responsible for providing the training on integrated product teams
    and cost as an independent variable.  The contractor hired a third
    party to develop a training program tailored to the Advanced
    Amphibious Assault Vehicle program, and both program office and
    contractor staff were taught together, on-site as a team. DOD's
    Training Methods for  For routine training, such as skill
    building, leading commercial firms have Best Practices Do Not Go
    as  standard training offerings, including functional area courses
    and Far as Leading Commercial  instruction on corporatewide
    issues, such as communications or ethics. Firms'
    However, when implementing key practices-such as those that change
    how product development and production are conducted-leading firms
    go beyond standard training offerings.  Commercial firms use a
    targeted, hands-on approach to ensure program teams are put in a
    good position to implement a new practice.  The elements common to
    how leading firms provide training on a key practice include (1)
    front-end analysis of program teams' needs and training
    requirements, (2) involvement of program teams in key training
    decisions, (3) customized training to meet program team specific
    needs, and (4) targeted training for the implementation of
    specific practices.  Program staff participate in and often
    influence a wide range of training decisions, including the amount
    of training provided for certain job descriptions, course topics,
    depth of course coverage, and identification of the appropriate
    course recipients. The involvement of the program staff has
    improved course depth, timeliness, and coverage of personnel in
    the commercial firms. In the Boeing Company, training
    representatives develop a partnership with the program staff when
    a team is formulated to design and manufacture a new airplane.
    The training organization forms "drop teams" to colocate with the
    program staff to learn as much as possible about the business
    process and the staff's concerns and to determine what training is
    needed to help the program staff implement a practice.  Boeing
    officials stated that training was instrumental to the
    implementation of key practices, such as design build teams, on
    the 777 aircraft program.  They noted that such teams were at odds
    with the company's culture because employees were not accustomed
    to working in a team environment and sharing information across
    functional areas. Training officials worked side by side with the
    program staff to create a training program that provided team
    building and conflict resolution techniques as well as new
    technical skills training. To ensure all program staff were
    equally trained, employees were required to complete training
    before they reported for duty on the program.  The professional
    employees-engineers and drafters-were required to Page 6
    GAO/NSIAD-99-206 Best Practices Executive Summary complete 120
    hours of start-up training on several key 777 practices before
    they were allowed to report for duty.  Teams were often trained
    together at the work location.  Ford followed a similar approach
    when it implemented the Ford Product Development System-a lean
    engineering process. Responsibility for training on best practices
    is diffused among several DOD organizations, including the Defense
    Acquisition University and the service acquisition reform offices.
    However, GAO did not find an organization that was able to tailor
    and help deliver training on best practices to the program offices
    visited.  Training provided by the university is designed
    primarily to enable people in individual career fields or
    functions, such as engineering and cost estimating, to meet
    professional certification requirements.  The university
    incorporates best practices topics into these functionally
    oriented courses as drop-in modules that provide a survey of the
    topic, but not in-depth coverage.  Although program offices see a
    greater need for training that cuts across functions to implement
    new practices and to manage in a team environment, it is difficult
    for a person in one career field to obtain training in another
    field.  The usefulness of these courses for best practices is
    further hampered by limited availability; according to an official
    from the university, the member schools get about 10 percent of
    the workforce into training each year. DOD's Acquisition Reform
    Communication Center and the acquisition reform offices in the
    services communicate best practice information through videos,
    periodic satellite broadcasts, roadshows, and Acquisition Reform
    Week.  These methods can reach more people than Defense
    Acquisition University courses and are designed around practices-
    versus functions-but are not tailored to specific program offices
    and are not necessarily delivered at the time those implementing
    new practices or initiatives need them.  For example, roadshows,
    traveling multiday training workshops provided to staff at a
    number of locations, typically provide awareness training on the
    practices.  DOD officials estimate that only 10 to 15 percent of
    the acquisition workforce attend the second day of workshops,
    where more detailed training is provided. The annual Acquisition
    Reform Weeks, which are a combination of satellite broadcasts and
    local presentations, mainly provide awareness-level training.
    Neither individual attendance nor the level of learning attained
    by attendees is tracked by these methods. Page 7
    GAO/NSIAD-99-206 Best Practices Executive Summary Differences
    Between DOD         Leading commercial firms shared a common
    strategy for adopting and and Commercial Training
    deploying key new practices.  First, the firms' corporate
    management Reflect Different Strategic     committed to and
    adopted few key practices-seven or less-at any given Approaches
    time.  In doing so, the companies were able to concentrate their
    attention and target resources to implementing the practices.  It
    also signaled the importance of the practices to trainers and
    implementers.  Second, the firms assessed which staff should be
    included in the implementation.  For example, Ford's training unit
    determined which engineering teams working on vehicle lines could
    benefit from the new production development system.  Third,
    company leaders made implementing the practice mandatory for the
    target population.  Lastly, companies developed well-defined
    learning objectives to better ensure that the target population
    consistently understood how to apply a new practice to improve
    production outcomes-the ultimate goal.  According to officials
    from these firms, it was a corporate responsibility to ensure that
    those implementing the practice received the necessary training
    and other assistance to succeed.  It was for this reason that the
    companies made a significant front-end investment to support the
    needs of program offices that would implement key practices.
    Company officials also pointed out that training is just one of
    the several components necessary for adopting new practices. They
    stressed that providing strong leadership and the right
    environment were key to ensuring the implementation of new
    practices and to developing quality training. DOD's approach to
    implementing best practices is less structured and more reliant on
    individual program offices to make the necessary training
    investment.  DOD policies on individual reform initiatives are
    typically promulgated without indicating what components of the
    acquisition workforce or which programs are expected to implement
    the practice.  In addition, the policies themselves are not always
    clear.  For example, although the initiative on cost as an
    independent variable was promulgated in 1995, Advanced Amphibious
    Assault Vehicle program officials developed training in 1997 to
    define the initiative for the program. In November 1998, an Air
    Force workshop on the cost initiative reported that it was still
    not well understood or widely implemented.  DOD has proposed over
    40 reform initiatives since 1994, without an indication of
    relative priority, leaving educators and implementers to decide
    what is important. Program offices are not necessarily in a good
    position to sort through the initiatives to focus on those that
    are the most important to the job at hand. A service acquisition
    reform official observed that the combination of many reform
    initiatives and unclear priorities causes the office to guess what
    is Page 8                                           GAO/NSIAD-99-
    206 Best Practices Executive Summary the most important, which
    leads to emphasizing what is perceived to be popular. DOD is aware
    of the need to improve the means by which the acquisition
    workforce receives and implements new initiatives.  A 1997 study
    by a DOD team and a 1998 study by the Logistics Management
    Institute pointed out several weaknesses in the focus and delivery
    of DOD training.  Weaknesses included the need for the Defense
    Acquisition University to be more active in implementing best
    practices and reforms, the tendency for the training curriculum to
    be functionally stove-piped, and the need for the university to
    have more involvement with the workforce-the recipients of
    training. The studies made recommendations for significant
    organizational and operational changes in the university,
    including that it should follow the corporate university model of
    becoming a change agent and a proponent of best practices and put
    more emphasis on cross-functional training.  In September 1998,
    the university proposed a revised structure, which has not been
    approved.  While the proposal offers some improvements, it stays
    close to its roots of providing functional training.  It does not
    reflect the corporate university model, a broadened role as a
    change agent, or a closer relationship with the acquisition
    workforce.  In short, it does not discernibly address key
    weaknesses in providing training of best practices. In June 1999,
    a DOD study team chartered to identify training on commercial
    business practices for the acquisition workforce concluded that
    adopting the most effective commercial practices requires a
    cultural and organizational transformation within DOD.  The team
    proposed a cross-functional plan for managing acquisitions that
    embraces best practices and calls for "learning organizations that
    seek out and adopt best practices that improve individual and
    organizational performance."  The plan proposes new roles for
    several organizations in fostering the adoption of best practices.
    Among these is the Defense Acquisition University.  For example,
    it recommends that the university be broadly recast to adopt the
    corporate university model and become a change agent.  This plan,
    while not specific about the help that program offices would
    receive, does call for a strategic approach that would make it
    more likely that DOD could provide its program offices tailored
    training-more help-in implementing best practices.  However, the
    fact that the September 1998 and the June 1999 proposals are still
    vying for approval indicates that DOD has not yet decided what
    role it wants acquisition training to play on best practices. Page
    9                                             GAO/NSIAD-99-206
    Best Practices Executive Summary Recommendations    GAO makes
    several recommendations to the Secretary of Defense that are
    intended to ensure that DOD's approach to the training of new
    practices better supports the needs of program offices by (1)
    developing a strategy for a structured approach to training on new
    practices; (2) providing tailored training assistance to program
    offices; and (3) improving the standard training curriculum so
    that it is more timely, relevant, and accessible.  These
    recommendations appear in full in chapter 5. Agency Comments
    DOD concurred with the views expressed in the report and all of
    the recommendations.  A discussion of DOD's actions appears in
    chapter 5. DOD's comments appear in appendix I. Page 10
    GAO/NSIAD-99-206 Best Practices Executive Summary Page 11
    GAO/NSIAD-99-206 Best Practices Contents Letter
    1 Executive Summary
    2 Chapter 1                       Responsibilities for Training
    the Acquisition Workforce Within Introduction
    DOD 16 Objectives, Scope, and Methodology
    21 Chapter 2                       DOD Training Did Not Reach the
    Right People at the Right Time              26 DOD Training Is Not
    a  DOD Training Offerings Did Not Provide the Depth Needed to
    Implement Best Practices
    31 Major Catalyst for Best  Limitations of Standard Training Led
    Program Offices to Develop Practices                         Their
    Own Training Solutions
    34 Chapter 3                       Commercial Firms Use Targeted,
    Hands-on Methods to Improve DOD Training Methods
    Training Usefulness
    39 DOD Does Not Target Training on Key Practices to Program
    Offices           45 for Best Practices Do Not Go as Far as
    Leading Commercial Firms Chapter 4                       Leading
    Firms' Strategic Approach Better Ensures Implementation DOD and
    Commercial                of Key Practices
    50 DOD's Training on Best Practices Does Not Stem From a Strategic
    Training Methods                  Approach 52 Reflect Different
    DOD Proposals to Alter Training Reflect Conflicting Approaches
    54 Strategic Approaches to New Practices Chapter 5
    Conclusions 59 Conclusions and                 Recommendations 60
    Agency Comments and Our Evaluation
    62 Recommendations Page 12
    GAO/NSIAD-99-206 Best Practices Contents Appendixes
    Appendix I: Comments From the Department of Defense
    64 Appendix II: Description of Program Offices Visited
    67 Appendix III: GAO Contacts and Staff Acknowledgments
    69 Related GAO Products
    72 Tables                  Table 1.1:  The Defense Acquisition and
    Technology Workforce as of April 1999
    18 Table 1.2:  Best Practices Evaluated at Program Offices Visited
    23 Figures                 Figure 1.1: Organizations Responsible
    for Training Development 20 Figure 2.1:  JSF
    28 Figure 2.2:  JSTARS-JTT
    30 Figure 2.3:  AMRAAM
    32 Figure 2.4:  AAAV
    35 Figure 2.5:  JASSM
    37 Figure 3.1:  777 Airplane
    42 Figure 3.2:  Ford Focus
    43 Page 13                                         GAO/NSIAD-99-
    206 Best Practices Contents Abbreviations AAAV         Advanced
    Amphibious Assault Vehicle AMRAAM Advanced Medium-Range Air-to-Air
    Missile ARCC         Acquisition Reform Communications Center BCIS
    Battlefield Combat Identification System CAIV         cost as an
    independent variable DAU Defense Acquisition University DAWIA
    Defense Acquisition Workforce Improvement Act DOD
    Department of Defense FPDS         Ford Production Development
    System IBM          International Business Machines IPT
    integrated product teams JASSM Joint Air-to-Surface Standoff
    Missile JSF          Joint Strike Fighter JSTARS-JTT Joint
    Surveillance Target Attack Radar System-Joint Tactical Terminal
    Page 14                                         GAO/NSIAD-99-206
    Best Practices Contents Page 15     GAO/NSIAD-99-206 Best
    Practices Chapter 1 Introduction
    Chapter 1 The Department of Defense (DOD) plans to increase its
    annual procurement investment to $60 billion by fiscal year 2001.
    DOD has high expectations from this investment: that new weapons
    will be better and less expensive than their predecessors and will
    be developed in half the time. Essential to getting these kinds of
    outcomes will be the adaptation of best commercial practices that
    have enabled leading commercial firms to develop new products
    faster, cheaper, and better. To help foster these outcomes, DOD
    has begun a number of acquisition reform initiatives based on
    commercial practices. Success depends greatly on the extent to
    which the program offices responsible for managing weapon
    acquisitions can implement the practices on individual programs.
    The training DOD provides program office staff to help them
    implement best practices should play a central role in getting the
    desired outcomes. While first-hand experience and "learning by
    doing" are instrumental in adopting new practices, training serves
    as a key agent in creating a culture that is receptive to new
    practices and providing the knowledge to implement the new
    practices at the workplace.  The relationship between training and
    implementing new practices was highlighted in a 1994 study of 300
    "improvement-driven" organizations conducted by Coopers & Lybrand-
    with the American Society of Quality Control, Rutgers University
    Center for Public Productivity, and the National Institute of
    Canada.  The study found that training played a "critical,
    integrative role as driver of cultural change, process alignment,
    job redesign, and continuous improvement."  Organizations included
    in the study used training as the vehicle for implementing and
    sustaining the changes at the level where the work was done.  The
    leading commercial firms we reviewed committed substantial
    investments to the training on key practices, underscoring its
    importance to getting the outcomes desired by the firms.   The
    significant numbers of reform initiatives that DOD has introduced,
    which involve adopting a number of new practices in the
    acquisition of weapons, present implementation challenges that
    training can help meet. Responsibilities for           For nearly
    50 years, the importance of an educated professional DOD Training
    the                   acquisition workforce has been emphasized by
    government leaders and reflected in the work of key studies and
    reform commissions.  The First Acquisition Workforce  and Second
    Hoover Commissions (1949 and 1955), the Fitzhugh Within DOD
    Commission (1970), and the Commission on Government Procurement
    (1972) all recognized the importance of high quality, well-
    educated acquisition professionals to the successful operation of
    DOD.  The Packard Commission, which undertook a broad examination
    of DOD management Page 16
    GAO/NSIAD-99-206 Best Practices Chapter 1 Introduction practices
    and procedures, reported in its June 1986 report that the DOD
    acquisition workforce was undertrained and inexperienced.  One of
    its recommendations was to improve the education and training of
    the acquisition workforce for the purpose of enhancing the defense
    acquisition process. Based in part by the recommendations of the
    Packard Commission, the Defense Acquisition Workforce Improvement
    Act (DAWIA) was passed in 1990 as part of the National Defense
    Authorization Act for Fiscal Year 1991.1 Its primary objective was
    to improve the DOD acquisition system by enhancing the education,
    training, and career development of members of the acquisition
    workforce.  Accordingly, DAWIA established the Defense Acquisition
    University (DAU) to provide for the professional educational
    development and training of the DOD acquisition workforce.  The
    act also charged DOD officials with the responsibility to
    designate certain positions as acquisition positions, to set
    qualification requirements, and to establish policies and
    procedures for training the acquisition workforce. The program
    offices that manage weapon system acquisitions are a key component
    of DOD's acquisition workforce. This workforce is generally
    defined as those people who are responsible for managing the wide
    array of DOD acquisitions, including contracting professionals,
    program managers, engineers, scientists, logisticians, and other
    occupational fields, from the earliest phases of basic research to
    the logistical support and disposal of old systems. However, there
    have been several definitions of what comprises the DOD
    acquisition workforce, which have led to varying estimates of the
    workforce size, ranging from about 106,000 to 355,000 people.  At
    the end of fiscal year 1997, DOD estimated the acquisition
    workforce covered by DAWIA included approximately 106,000
    positions, of which about 90,000 were civilian. Its most recent
    definition places the workforce, now called the acquisition and
    technology workforce, at about 150,000 people2 and includes people
    from science and technology organizations. Table 1.1 lists the
    different acquisition and technology workforce occupations and the
    number in each occupation based on this definition. 1 P. L. 101-
    510, Nov. 5, 1990. 2 This definition was based on the study,
    "Identification of the Department of Defense Key Acquisition and
    Technology Workforce," April 1999, Jefferson Solutions,
    Washington, D. C. Page 17
    GAO/NSIAD-99-206 Best Practices Chapter 1 Introduction Table 1.1:
    The Defense Acquisition and Technology Workforce as of April 1999
    Acquisition and technology workforce occupations
    Total persons Engineers (electronics, aerospace, mechanical,
    civil, and general)                        44,117 Contracting
    19,387 Management
    15,509 Business and industry
    12,989 Communications and computers
    9,370 Administration and programs
    5,116 Scientists
    4,476 Auditing
    3,692 Financial management
    3,618 Procurement assistants
    2,650 Mathematics and statistics
    2,400 Purchasing
    2,158 Supply program management
    1,753 Inventory management
    944 Equipment specialists
    858 General supply
    326 Miscellaneous
    3,698 Military
    16,378 Total
    149,439 Source: DOD. Definitions aside, the DOD acquisition
    workforce has been undergoing restructuring and downsizing.  The
    National Defense Authorization Acts for Fiscal Years 1996 and 1997
    mandated reductions in the number of civilian and military
    employees in acquisition organizations.  DOD estimates that, as a
    result, the acquisition workforce will be 25 percent smaller at
    the end of fiscal year 2000 compared to fiscal year 1995.  DOD
    understands that effective training will become even more
    important as the workforce is reduced and new authority and
    responsibility are granted to those who remain.  We have
    previously reported that decisions to restructure or reduce this
    workforce should be linked to getting better outcomes from the
    acquisition process; doing otherwise would miss an opportunity to
    address the deep-seated causes of acquisition problems.3 3 Defense
    Acquisition Organizations: Linking Workforce Reductions With
    Better Program Outcomes (GAO/T-NSIAD-97-140, Apr. 8, 1997). Page
    18                                                      GAO/NSIAD-
    99-206 Best Practices Chapter 1 Introduction Although weapon
    system program offices comprise a subset of this workforce, they
    are a highly leveraged subset.  In planning, managing, and
    executing acquisition programs, program management offices are
    responsible for about $80 billion of DOD's annual research,
    development, and procurement funds-about 30 percent of the fiscal
    year 1999 DOD budget.  They influence much of the work of the rest
    of the acquisition workforce because they are bringing new
    equipment into the inventory, which must be managed, budgeted for,
    maintained, and supplied.  In a sense, they are a point of entry
    for DOD acquisitions.  In addition, these offices are made up of a
    cross section of people that draw from most of the acquisition and
    technology workforce occupations cited in table 1.1.  As such,
    they are a medium in which the training of new initiatives in
    different career fields converge. The primary responsibility for
    training the acquisition workforce in general-and new initiatives
    in particular-falls within the Office of the Under Secretary of
    Defense for Acquisition and Technology.  That office is
    responsible for setting the standards that the workforce must
    meet. DAU, which is responsible for designing and conducting the
    training to meet those standards, also reports to that office.
    These and other organizations responsible for setting training
    standards and providing training are shown in figure 1.1. Page 19
    GAO/NSIAD-99-206 Best Practices Chapter 1 Introduction Figure 1.1:
    Organizations Responsible for Training Development Under Secretary
    of Defense for Acquisition and Technology Director for Acquisition
    Education, Training, and Career Development
    Defense Acquisition University Functional Boards
    Consortium of Schools * Acquisition management
* Defense Systems Management * Contracting
    College * Technical management (e.g.,
* Industrial College of the Armed engineering and testing
    Forces *Business, cost estimating, and
* Service Schools financial management The Under Secretary of
    Defense for Acquisition and Technology delegated responsibility
    for developing career paths and establishing educational standards
    to the Office of the Director of Acquisition Education, Training,
    and Career Development. That office is supported by four
    functional boards that have established experience, education, and
    training standards for the acquisition workforce and for
    professional certification levels in each of the acquisition
    career fields.  A DOD manual, DOD 5000.52M, "Acquisition Career
    Development Program," implements and prescribes procedures for
    career development of the acquisition workforce.  The Page 20
    GAO/NSIAD-99-206 Best Practices Chapter 1 Introduction manual
    establishes experience, education, and training standards for 3
    certification levels in each of the 11 acquisition career fields.
    Level I is basic or entry level, level II is the intermediate or
    journeyman level, and level III represents the advanced or senior
    level.  The specific training to increase competency or attain
    higher certification levels within a career field is often
    referred to as functional training.  There are also standards for
    specific acquisition workforce positions (such as program
    managers), position categories, and membership in the acquisition
    corps. DAU was created by DAWIA to develop the training curriculum
    to meet these standards and to coordinate the efforts of its
    consortium of 13 DOD-wide and service schools that conduct the
    training courses.  The Defense Systems Management College, a DOD-
    wide school, is dedicated to providing acquisition-related
    training. Other member schools, including those run by individual
    military services, provide a variety of training, of which
    acquisition training is only a part. While the service schools
    provide information on new initiatives, they do not have primary
    responsibility for educating the acquisition workforce. Each
    service also has an acquisition reform office that helps make
    people aware of the latest practices and initiatives that apply to
    acquisitions, although this office does not play a significant
    role in designing or providing the training.  The Acquisition
    Reform Communications Center is an organization related to DAU
    that has the mission of sharing knowledge about acquisition reform
    by providing and disseminating information on how DOD is changing
    the way it acquires goods and services. Objectives, Scope, and
    The Chairman and the Ranking Minority Member, Senate Committee on
    Methodology                    Armed Services, Subcommittee on
    Readiness and Management Support, asked us to review how DOD is
    training its acquisition workforce to implement best practices for
    acquiring weapon systems. The Subcommittee's request is part of a
    broader interest in seeing that best practices are incorporated
    into DOD's acquisition process as a way of saving money for
    modernization, increasing efficiency, and improving quality.  The
    objectives of this report are to assess (1) the contribution DOD
    training makes to program offices that are applying best
    practices, (2) the different training methods DOD and leading
    commercial firms use in providing training on practices, and (3)
    the different strategic approaches that underlie the training
    methods DOD and leading commercial firms use in implementing
    practices. Page 21
    GAO/NSIAD-99-206 Best Practices Chapter 1 Introduction To meet
    these objectives, we focused on five specific practices and
    identified program offices recognized as being leaders in applying
    them. We chose the five practices based on discussions with DOD
    and on our previous work in the application of best practices to
    weapon acquisitions. The first four are formal DOD initiatives
    that are based on best commercial practices, while supplier
    relationships is a best practice we observed in leading firms and
    in some leading weapon system programs.4 The selected practices
    are the following: * Cost as an independent variable (CAIV): An
    acquisition management practice in which aggressive life-cycle
    cost goals are achieved through trade-offs between requirements
    and performance. * Integrated product teams (IPT): Teams composed
    of members from functional disciplines such as engineering, test,
    and contract management.  All members contribute their particular
    expertise to team decisions and to resolve issues. * Performance
    specifications: States requirements in terms of required results
    without stating the methods to achieve those results.  They define
    the functional requirement for operation, interface, and
    interchange characteristics, and have criteria for verifying
    performance compliance. * Past performance: Information on a
    contractor's past performance on relevant prior work is used as a
    factor in source selection.  The information is used to evaluate
    risk and the potential for future contractor success. * Supplier
    relationships: A commercial practice in which maximum
    participation of suppliers and their suppliers is encouraged to
    promote product excellence.  The best suppliers are selected and
    supported in a number of ways that ensure a mutually beneficial
    partnership. We focused on weapon system program offices because
    of the significant role they play in implementing best practices.
    Initially, we considered gathering information from a cross
    section of program offices that had a range of experience in
    implementing best practices.  However, we found that a program
    office that had little or no experience with a new practice was
    not in a good position to comment on the training needed to
    implement the practice.  Consequently, we selected program offices
    that DOD considered to be leaders in one or more of the five
    practices. 4 See Best Practices: DOD Can Help Suppliers Contribute
    More to Weapon System Programs (GAO/NSIAD-98-87, Mar. 1998). Page
    22                                                      GAO/NSIAD-
    99-206 Best Practices Chapter 1 Introduction We based our
    selections on extensive consultation with DOD officials to ensure
    that the program offices had experience in implementing the
    practices and thus were in a good position to comment on the
    training resources that enabled them to implement the practices.
    (See app. II for a description of the programs.)  These program
    offices represented best case examples in DOD for specific
    practices, although none of the programs was considered a leader
    in all five practices.  The program offices selected were the
    Advanced Amphibious Assault Vehicle (AAAV), Advanced Medium-Range
    Air-to-Air Missile (AMRAAM), Battlefield Combat Identification
    System (BCIS), Joint Air-to-Surface Standoff Missile (JASSM),
    Joint Strike Fighter (JSF) and Joint Surveillance Target Attack
    Radar System-Joint Tactical Terminal (JSTARS-JTT). Table 1.2 shows
    which of the program offices were considered leaders for the
    different practices. Table 1.2:  Best Practices Evaluated at
    Program Offices Performance          Past       Supplier CAIV
    IPT specifications performance relationships AAAV
    XX                          X AMRAAM                          XX
    BCIS                            XX JASSM
    XX X JSF                             XX JSTARS-JTT
    XX Source: GAO analysis of DOD information. At the program
    offices, we used structured questions to interview the key people
    responsible for implementing an individual practice.  Through the
    interviews, we determined the various sources they used to develop
    the knowledge needed to implement best practices and the extent to
    which the training DOD provided to the program office contributed
    to this knowledge. To assess strategy and the methods DOD uses to
    train program office staff on the use of best practices, we
    concentrated on the DAU training organizations and resources
    established by DAWIA to provide for the professional educational
    development and training of the DOD-wide acquisition workforce.
    We also assessed how these resources met program office needs in
    implementing best practices.  In addition, we assessed other Page
    23                                                    GAO/NSIAD-
    99-206 Best Practices Chapter 1 Introduction acquisition training
    sources, including service and agency specific schools and the
    services' acquisition reform offices. However, our report focuses
    on the training approach for best practices and is not intended to
    describe all DOD training practices.  We met with the Director of
    Acquisition, Education, Training, and Career Development; the
    President of the DAU; officials of the Defense Systems Management
    College; and representatives from the Office of the Under
    Secretary of Defense for Acquisition and Technology.   We also met
    with and collected data from officials at the Air Force, the Army,
    and the Navy.  We reviewed the organizational structure and
    responsibilities for DAU consortium members and service
    organizations, the process for establishing the DAU training
    curriculum, and survey data on DOD training recipients. We
    evaluated the strategies and methods of commercial firms
    recognized for their training excellence by examining how these
    leading companies used training to implement new practices at
    their program management organizations. To identify these
    companies, we conducted literature searches, consulted with and
    collected data from professional associations, and spoke with
    university faculty specializing in corporate organizations. We
    visited the following four companies recognized as being leaders
    in the area of training: * The Boeing Company is the largest
    manufacturer of commercial jetliners and military aircraft with
    234,000 employees worldwide. Boeing's Employee Training and
    Development organization is responsible for training all
    Commercial Airplane Group employees, approximately 97,000.  There
    are 500 to 600 employees in the office, with roughly 300 serving
    as trainers. * Ford Motor Company is one of the largest U.S.
    manufacturers of automobiles, trucks and provider of automotive
    services with 345,000 employees worldwide.  Ford's Product
    Development Process Leadership organization was created to provide
    assistance, including training, to engineers implementing the Ford
    Production Development System (FPDS). * International Business
    Machines (IBM) Corporation is one of the world's top providers of
    computer hardware and software with 290,000 employees worldwide.
    Different internal organizations provide training that serves
    employees worldwide, including Learning Services and the Center
    for Excellence. * Motorola is one of the world's leading providers
    of wireless communications, semiconductors, advanced electronic
    systems, and services with over 150,000 employees worldwide.
    Motorola University Page 24
    GAO/NSIAD-99-206 Best Practices Chapter 1 Introduction has a staff
    of over 600 operating through account managers that are assigned
    to each business unit to provide a "one-face" education and
    training contact. These companies, recognized as industry leaders,
    place strong emphasis on training.  At these companies, we
    reviewed company documents and training data and met with
    individuals responsible for designing and developing programs to
    educate and train employees on major new practices.   We also met
    with representatives from major program offices who were involved
    in key training decisions and were recipients of the training.
    Our report highlights the best commercial training approaches for
    implementing key new practices.  As such, they are not intended to
    describe all commercial training practices or suggest that
    commercial firms are without flaws. Finally, we reviewed several
    studies on DOD's training organizations and methods.  We used
    these, as well as the previous information and analysis, to
    determine the extent to which DOD's proposals to reshape DAU and
    continuous learning policy held potential for better delivery of
    training to foster implementation of best practices by program
    offices. We conducted our review between April 1998 and June 1999
    in accordance with generally accepted government auditing
    standards. Page 25
    GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a
    Major Catalyst for Best Practices
    Chapter 2 Key officials from weapon system programs at the
    forefront of implementing best practices did not find that
    standard DOD training offerings provided the information they
    needed to apply the practices to their programs.  In evaluating
    their key sources of knowledge for implementing best practices,
    none of the program officials ranked required DOD training first,
    with many ranking it last.  DOD training either did not reach the
    right people when it was needed or did not reach them at all. When
    training on best practices was received, it did not contain the
    depth or practical insights program office staff needed to
    implement the practices.  It was primarily through their own
    efforts-learning on the job, finding external training, or
    developing their own training program-that they attained the
    knowledge needed to apply best practices. Programs that became
    leaders in applying best practices did so primarily because their
    managers realized that the practices were key to the programs'
    success.  In so doing, the managers were able to identify what
    knowledge they needed to apply the practices.  Thus, their success
    depended on having the foresight to see what was needed, the
    ingenuity to find good sources of knowledge, including training,
    and the resources needed to attain that knowledge.  Replicating
    this approach broadly on other programs is problematic.  Other
    managers may not realize the significance of a practice to the
    success of their programs and the need for additional training.
    Also, they may be uncertain about testing new initiatives on their
    programs.  Some may recognize a practice's importance but be
    unable to fund their own training efforts and be left relying on
    standard DOD training. DOD Training Did Not  For training to
    facilitate the adoption of a new practice, it must be received
    Reach the Right People  by those responsible when they are ready
    to begin implementing the practice.  Program officials stressed
    that on a new initiative or practice, at the Right Time
    training must begin when a practice is to be introduced.  If
    training is provided too late, people will be forced to devise
    their own means of getting the knowledge needed to begin
    implementation or risk improper implementation.  If training is
    provided too soon, knowledge could fade before it is applied at
    the workplace.  For the programs we reviewed, those responsible
    for managing or implementing best practices noted that in some
    cases, training did not exist when they began implementing the
    practices; in other cases, the training was received too late for
    the job at hand.  Some were missed altogether by DOD training
    offerings because they were too senior to be required to take
    courses, training was not Page 26
    GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a
    Major Catalyst for Best Practices provided locally, or they fell
    outside the definition-and training curriculum-of the acquisition
    workforce. Training Came at the Wrong  Program office officials
    believed it is essential that training on new Time for Some People
    initiatives accompany implementation to explain what the
    initiatives are and to come up with a common understanding and way
    to tackle program-specific issues.  For the program offices
    leading the way in best practices, officials reported relevant DOD
    training was not offered at the time implementation began.  For
    example, the BCIS program applied performance-based specifications
    soon after the 1994 DOD initiative was announced.  However,
    specific training on performance specifications had not been
    developed.  The AAAV program had already implemented performance
    specifications without DOD provided training.  A top manager for
    JSF similarly reported that there was no training for program
    office staff when they started to implement CAIV.  There were no
    guiding documents and no one, including the training community,
    knew what CAIV was. Similarly, on the JSTARS-JTT program, CAIV
    training was not available when the program office began applying
    the practice in 1995. Now, some training is available, but it was
    not when it was needed for the program.  Nor was training
    available when the JASSM program office began assessing contractor
    past performance. Training could have helped avoid a protest of
    the prime contract award, according to an official. Page 27
    GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a
    Major Catalyst for Best Practices Figure 2.1:  JSF Implementation
    of CAIV on the JSF program began with a clean slate-no advance
    training. Source: DOD, artist rendition. Training can also come
    too early.  A program official questioned the staff's ability to
    retain the information when training is not provided at a
    practical time for the assignment.  For example, he noted that
    symposiums are good Page 28
    GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a
    Major Catalyst for Best Practices ideas, but people may not have
    an opportunity to apply the ideas at the time and may not remember
    when they need to. Key People Were Missed by  Applying best
    practices on a weapon system program involves reaching not
    Training on Best Practices        only program office staff but
    also other members of the acquisition workforce, DOD people
    outside the acquisition workforce, and contractors.  Experience on
    the programs we reviewed shows that it is hard to reach all of
    these people with best practices information through standard DOD
    training. The promotion of many best practices took place after
    the implementation of DAWIA course requirements so that senior
    people have not taken formal acquisition training that includes
    exposure to best practices. Several of the senior program
    officials we spoke with did not receive training on new practices
    because they had not taken courses that incorporated best
    practices.  These officials had been acquisition officials for
    many years before DAWIA and were grandfathered into their
    certification level.  Their job experiences were applied to meet
    professional certification requirements and thus they did not have
    to take any DAWIA training. For example, an official who helped
    establish JSF performance specifications noted that he had his
    last class 3 years earlier, a contracting refresher course.  The
    course may have had a short section on performance specifications,
    but he could not remember it. Training misses some program office
    people because it is not offered locally.  According to program
    officials, training needs to be conducted at the local level by
    subject matter when it is needed.  This is especially true for
    small programs, for which it is difficult to spare people from the
    programs for long periods.  One official from a small program said
    that although senior management stresses the importance of
    training, it is difficult to attend classes that run over 14
    weeks.  He added that these classes have not been offered locally.
    Another program official asked why DOD training organizations do
    not bring some of the harder to get courses to the field, as is
    done by private organizations. Page 29
    GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a
    Major Catalyst for Best Practices Figure 2.2:  JSTARS-JTT JSTARS-
    JTT officials believe training could be improved by being provided
    on-site. Source: DOD. Training is not reaching people outside
    program offices that also play key roles in successfully
    implementing best practices.  For example, program offices are
    typically supplemented with people from separate functional
    organizations, such as engineering directorates.  These people may
    not receive training on new initiatives in their home
    organizations and can be unfamiliar with the initiatives when they
    come to a program.  People that set requirements early in the
    process and those in logistics must also be knowledgeable and
    committed for practices to be successful.  Exercising flexibility
    in requirements, for instance, is critical to the success of CAIV
    and performance specifications.  The need for training was
    extended further to those that play a role in a weapon system's
    approval.  Officials from one program reported that they had
    worked with a prime contractor to streamline contract reporting
    requirements down to four items, in line with acquisition reform.
    However, based on a review by a separate office with approval
    authority over the contract, 40 contract requirements were Page 30
    GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a
    Major Catalyst for Best Practices added, returning the contract to
    traditional reporting methods.  The program official said that the
    reviewing official did not know what the program office was trying
    to accomplish and did not care. Prime contractors and
    subcontractors are also essential to the application of best
    practices but are not part of the DOD defined acquisition
    workforce or otherwise targeted for training on best practices.
    Several program officials we met with believed that some means of
    informing contractors is needed to make them aware of important
    DOD initiatives.   Contractors do not necessarily understand or
    know how to implement new DOD initiatives; for example, they may
    not know how an integrating IPT or overarching IPT works. While
    DOD is not responsible for training contractors, it does have to
    ensure that contractors understand best practices, as well as give
    them an opportunity to help shape how these practices are applied
    on programs. The AAAV program management ensured their prime
    contractor was knowledgeable of best practices by requiring
    training for contractor and program office staff in the prime
    contract. DOD Training                     Program officials
    reported that standard DOD training did not prepare Offerings Did
    Not                them well for implementing the practices at the
    workplace.  DOD's training typically provided only an awareness of
    the practices, not the knowledge Provide the Depth
    that is needed for actual implementation.  Also, the training was
    not Needed to Implement  tailored to allow program offices to see
    how individual practices applied to Best Practices
    their specific programs.  Further, program officials noted that
    trainers did not have the practical experience to share and were
    not up to date with the most recent examples of programs that had
    implementation experience. Program officials stated that they need
    to go beyond the theoretical concepts covered in most training
    courses.  They believed that the "how to" is missing on all the
    initiatives and that they need to know how to move from
    traditional practices to the new practices.  DOD courses were
    described in general as too esoteric and not relevant to the tasks
    at hand. For example, a JASSM official noted that a contractor's
    past performance is often equated with DOD's Contractor
    Performance Assessment Rating System, but it involves much more.
    The JASSM program office evaluated contractors' past products, and
    in doing so, learned about the quality of their design,
    management, and production processes.  Program officials said they
    went well beyond the performance assessment rating system covered
    in standard training courses.  Nonetheless, they said that they
    could not avoid a protest by the losing contractor.  They believed
    they Page 31                                          GAO/NSIAD-
    99-206 Best Practices Chapter 2 DOD Training Is Not a Major
    Catalyst for Best Practices needed guidance on how to collect past
    performance information, which was not covered by training.
    Program officials believed, overall, that training should be
    designed more for the customer because acquisitions are unique and
    that programs may have different implementation issues based on
    program size, stage of the acquisition, or type of weapon.  A
    senior AAAV official said general courses can provide information
    about individual practices, but to implement a practice, the
    training must be tailored to the program.  JSF and BCIS officials
    concurred. Some program officials have observed that training
    courses overemphasize the application of new practices for larger
    and newer programs and believe that not enough coverage is given
    to teaching how the new practices apply to programs that are
    smaller or older. For example, officials noted that newer programs
    have an advantage in starting with a clean slate and that training
    should also show how the practices apply to older programs and the
    benefits to be obtained. Figure 2.3:  AMRAAM AMRAAM officials
    believe training should help program managers in deciding how
    initiatives apply to their particular circumstances. Source: DOD.
    Page 32
    GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a
    Major Catalyst for Best Practices Program officials also found the
    presentation methods for training on best practices were not
    helpful in applying the practices in the workplace.  They believed
    that instructors lacked practical experience and current
    information.  For example, a JSF manager reported that the
    executive course he took in 1997 on CAIV was not valuable because
    the instructors had no experience with CAIV and did not know how
    to explain it.  Another program official thought that instructors
    were left in place too long and had only old experiences to share.
    Officials noted that course material was frequently out of date
    and sometimes incorrect.  For example, one JASSM program official
    attended an engineering course in which the instructor's answer on
    CAIV was wrong.  Another said the acquisition reform example the
    instructor used-from the official's own program-was about 2 years
    behind what was being applied on the program.  The official
    brought up more current examples of practices being used on the
    program, but the instructor did not want to discuss them. Training
    also suffered from limited use of case studies that would allow
    students to see how an initiative in the abstract might apply to
    their own programs.  Officials from several programs added that
    current case histories should show the application of acquisition
    principles in a program context.  Case studies would include the
    successes, as well as the pitfalls and solutions.  The case
    studies should be designed for the customer, another said, and be
    applied at multiple levels.  For example, programs could benefit
    if training allowed students to play off the risks and benefits of
    how new initiatives apply to their particular program.  Real case
    experiences help others visualize how practices could apply.  BCIS
    officials reported that during the presentation of their case
    study for the 1998 Acquisition Reform Week, they realized, as they
    explained what they did to implement initiatives, that the
    audience gained better insight on the issues. Another official
    said he shared his program's experiences with those in his class,
    but those in other classes would not have the benefit of his
    experiences. Program officials did not believe that there was an
    effective means for providing feedback on the quality and
    usefulness of courses, such as the need to update course
    materials, or course relevance.  They would like to see the
    students-the practitioners-have more impact on what adjustments
    are needed to courses.  The only means of feedback program office
    people were aware of was the end of class survey form.  However,
    these surveys did not allow students to give in-depth feedback or
    ask the questions that got to the larger issue of whether
    students' training needs were met. Page 33
    GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a
    Major Catalyst for Best Practices Limitations of
    In evaluating their key sources of knowledge for implementing best
    Standard Training Led  practices, none of the program officials
    ranked required DOD training first, with many ranking it last.  To
    obtain the knowledge needed to implement Program Offices to
    key practices, officials in leading programs developed their own
    solutions. Develop Their Own                 They had a vision of
    what they wanted to accomplish and they devised a Training
    Solutions                variety of methods, such as funding
    training beyond standard DOD offerings, creating their own
    internal training programs, and learning on the job.  Strong
    program managers, supported by the executives above them and
    strong working relationships with their contractors, were key to
    the implementation of the practices.  Program managers of the
    leading programs cautioned, however, that not all officials have
    the vision or the resources to mirror this approach.
    Implementation Success            All the leading programs had at
    least one element in common-strong Tied to Vision and Support
    leadership committed to implementing practices that would help
    their at All Levels                     programs succeed.  Leaders
    were described as having vision and knowing what they had to do to
    realize that vision.  This included accepting the risks for trying
    new approaches.  For example, the original AAAV program manager
    conceived of the unique approach of collocating the program office
    and the contractor because he thought it was essential to making
    IPTs work.  IPTs became the backbone of the AAAV program.
    Consequently, team training was viewed as so important that it was
    incorporated into the prime contract as a requirement.  Support
    for collocation and the team approach came from top management of
    both the contractor and the program office.  According to program
    officials, it does not make any difference how good the training
    is without management support; junior people can come back from
    training with new ideas and have them go nowhere if they do not
    get management support. Page 34
    GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a
    Major Catalyst for Best Practices Figure 2.4:  AAAV Training for
    AAAV was tailored to support contractor and program office
    collocation. Source: DOD. In turn, managers point to the support
    that they receive from their senior management as an important
    factor in their ability to be successful.  One official described
    his manager as aggressive in his efforts to pave the way for
    trying new practices.  To do that, the senior manager said that he
    is willing to go up against the established bureaucracy and
    provide cover for his program managers to try new things. One
    program manager cited a personal commitment to a reform-minded DOD
    official as part of his motivation for making acquisition reform
    work.  He said that with his bosses' support, he can do what he
    thinks is right on his program, such as making past performance
    central in selecting a prime contractor.  However, he noted that
    other managers have not had that ability because they are not
    supported when they propose doing things differently. Program
    managers believe that a key element for adopting best practices
    was developing trust or partnership with the contractor.  An
    AMRAAM manager believes the ability to make radical program
    changes has Page 35
    GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a
    Major Catalyst for Best Practices depended on working closely with
    the prime contractor, and seeing that the same relationship exists
    between the prime contractor and subcontractors. The JASSM office
    organized its IPTs to mirror the structure of the contractor's
    IPTs, rather than along functional lines like most offices.  A top
    JASSM official credited the high level of trust between the
    program office and the contractor, along with the program office's
    creativity and innovation, for enabling their use of new
    practices. This perspective is essential to managing programs in
    today's environment and to make teams work, another official
    noted, but it is not taught in standard DOD training offerings.
    More managers would attempt to apply best practices if training
    encouraged it, a program official said, particularly the training
    provided for junior program managers.  More general training is
    needed to support a new program environment for adopting new
    practices that goes beyond the current training emphasis on the
    mechanics of management and theory. Program officials reported
    that new program manager training as recent as 1997 did not
    reflect the new approach they have practiced on their programs.
    One believed that it was a lost opportunity for DOD to not impress
    upon new managers their role in adopting new practices and the
    potential benefits. Leading Program Officials     With the support
    of senior management, program officials used a mixture Innovated
    to Get Needed       of strategies-generally outside of standard
    DOD training offerings-to get Training                      the
    knowledge they needed to apply best practices.  A number of
    programs used their own resources, including cumulative job
    expertise of the staff and personal research to learn how to
    implement new initiatives.  For example, JSF officials stated they
    learned how to apply CAIV by organizing warfighters, engineers,
    and analysts together in a team.  They learned as they went,
    developing materials on their own.  The AMRAAM program had a young
    staff member gather information on CAIV from various sources such
    as the internet and conferences.  From that starting point, they
    learned as they went along.  In setting performance
    specifications, it was the personal experience of AMRAAM
    officials, coupled with commitment, that allowed the
    specifications to be set at a performance, rather than detailed,
    level.  Despite their own experiences, program officials did not
    recommend the learn-as-you-go approach; everyone should not have
    to reinvent the wheel. Programs officials used their own program
    funds to go beyond standard DOD training by sending staff to
    nongovernment training or to bring Page 36
    GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a
    Major Catalyst for Best Practices experts in.  For example,
    frustrated with DOD training, the JASSM program manager sends
    people to outside training, such as  Harvard leadership courses,
    to have them learn and grow beyond the basic training and develop
    creative and innovative thinking.  He has his staff take 80 hours
    of general, nonfunctional training.  Another manager said some of
    his best training was from off-site sessions sponsored by JASSM
    that dealt with people issues that were critical to making IPTs
    work.  JSF officials took a private sector course on performance-
    based specifications and used a model developed by their
    contractor.  BCIS officials hired an outside firm to teach program
    officials on two occasions, both of which included a CAIV
    component. Figure 2.5:  JASSM JASSM officials sponsor training to
    develop staff leadership skills. Source: DOD. AAAV officials
    developed their own training program tailored to the program's
    work environment and training needs.  A key characteristic of the
    program is the collocation of government and contractor program
    Page 37
    GAO/NSIAD-99-206 Best Practices Chapter 2 DOD Training Is Not a
    Major Catalyst for Best Practices officials, making team dynamics
    an important factor.  The prime contract specified that teams must
    be used, and IPT training was charged to the contract.  The
    contractor hired a third party to develop a training program
    tailored to the AAAV program, and program office, contractor, and
    subcontractor staff were taught together, on-site as a team.
    Joint training was used to establish a common culture for
    participants.  Training was given one time to the team initially
    and then to every new person.  The team training started with
    mandatory 10 hours of IPT training, with sessions covering the
    AAAV program, trust, communications, and other team dynamics.
    CAIV training was added as part of the training for implementation
    within the team context. The team approach to training was
    supported by other leading programs as well.  On the JASSM
    program, the manager reported that he just had his entire office
    (about 30 staff) take 1 week of leadership training, which
    included topics such as stress management, critical thinking, and
    decision-making.  JSF officials took the "train the trainer"
    approach and developed a team of experts within JSF and then the
    experts trained everyone that would be involved in implementing
    performance specifications.  The experts used multiple sources to
    train themselves, such as published guidance and talking with
    their peers.  They then developed basic guidance for the staff,
    such as engineers, who needed a common understanding on how to
    write requirements at a performance level. Other Programs May Lack
    Program officials reported that they were fortunate to have staff
    that could Leading Programs' Ability to  use their collective
    experiences to work through problems in implementing Innovate
    initiatives.  However, they noted that not all programs will have
    the same advantages.  JASSM officials said top service officials
    handpicked program officials for the program team because their
    personal characteristics supported flexibility and creativity.
    Similarly, JSF officials reported that the program office was
    staffed with multifaceted people, as well as strong, senior
    management support and upfront money for training.  They noted
    that smaller programs may be staffed by junior officials with less
    experience to draw upon or be unable to devote staff to research
    information on how practices might apply to their situation.
    Limited funding may also be an issue for some programs.  As one
    program official said, his program office could pay for external
    training, but for many programs, the cost would be an impediment
    since training is one of the first items cut in a program budget.
    Consequently, smaller programs might need to rely more heavily on
    standard DOD training as their main source of information. Page 38
    GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for
    Best Practices Do Not Go as Far as Leading Commercial Firms
    Chapter 3 Leading commercial firms and DOD apply different
    training methods to implement key practices.  Commercial firms use
    targeted, hands-on methods that include conducting a front-end
    analysis to determine the teams' training requirements and
    regularly involve those implementing the practices in making
    important training decisions to ensure program teams are trained
    on key practices.  Thus, the training is customized to meet the
    specific needs of the teams.  These methods also involve providing
    many hours of training-beyond standard skill-based or functional
    training- focused on the implementation of a single practice.
    Company officials believe the targeted approach results in more
    useful training-improving the personnel coverage, course depth,
    and timing-which helps to improve outcomes of the final product.
    In contrast, DOD training on best practices is delivered through
    traditional DAWIA certification courses and vehicles such as
    videos and computer-based training, which are limited in reaching
    the right people at the right time and in providing the needed
    depth to implement best practices. DOD does not have a counterpart
    to the commercial method of providing customized, hands-on
    assistance to support program office staff and other implementers
    of key practices. Although exceptions exist, there is no
    systematic effort-or responsible organizations-within DOD to
    directly assist key implementers to use new practices.  Further,
    DOD does not have a comprehensive means for allowing program staff
    and others to influence training decisions in a way that could
    improve the relevance of training. Commercial Firms Use  For
    routine training, such as skill-building, leading commercial firms
    Targeted, Hands-on               provide standard training
    offerings, including functional area courses and instruction on
    corporatewide issues, such as communications or ethics. Methods to
    Improve               However, when implementing key new
    practices-such as those that Training Usefulness
    change product development and production-the firms go beyond the
    standard training offerings.  Commercial firms use a targeted,
    hands-on training approach to ensure program teams are in a good
    position to implement a new practice.  They provide numerous hours
    of training, typically through a single company organization,
    targeted to the implementation of a key practice.  The practice-
    specific training hours are targeted to the program teams most
    likely to implement the new practice. Page 39
    GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for
    Best Practices Do Not Go as Far as Leading Commercial Firms The
    elements common to how the four leading firms provide training on
    a key initiative or practice include * a front-end analysis of
    program teams' needs and training requirements; * involvement of
    program teams in key training decisions; * customized training to
    meet program teams' specific needs; * targeted training for the
    implementation of specific practices; and * improved training
    outcomes, including better course depth, timeliness, and reach.
    The training organizations of leading commercial firms conduct a
    front-end analysis to determine the needs and training
    requirements of program offices implementing new practices.  The
    analysis is also used to identify and address barriers each
    program office faces when implementing new practices.  According
    to the Director of the Benchmarking Forum for the American Society
    for Training and Development, this type of analysis is crucial for
    an organization to be able to institute performance-improving
    measures.  Using information from the front-end analysis, the
    training organizations customize the training to ensure that it
    directly assists program teams in implementing new practices.  One
    company official told us that training is costly and when it
    misses the mark, the company pays a big price.  Given the
    importance of training when implementing a key practice, company
    officials believe that it is crucial to ensure that the training
    is beneficial to the key implementers of the practice. To ensure
    that the training will address the needs of the program teams, the
    training organizations involve the staff in making important
    training decisions. Program staff help decide the amount of
    training to be provided for certain job positions, course
    objectives, and depth of course coverage.  Company officials
    believe their training approach, which includes program staff, has
    resulted in the right amount of course depth, timeliness, and
    coverage of personnel in the commercial firms.  Following are
    descriptions of the training methods employed by companies on key
    initiatives or new practices. The Boeing Company    Officials from
    Boeing's Employee Training and Development organization state that
    their primary goal is to support their customers-employees
    assigned to the Commercial Airplane Group.  The training
    representatives develop a partnership with the staff from the
    beginning of the program to design and manufacture a new airplane.
    The training representatives form "drop teams" to collocate with
    the program to conduct a front-end analysis and learn as much as
    possible about the business process and the staff's Page 40
    GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for
    Best Practices Do Not Go as Far as Leading Commercial Firms
    concerns.  The analysis allows the drop team to determine what
    training is needed to support the staff implementing new
    practices. Boeing training officials said they worked side by side
    with the program staff to create a training program that provided
    team building and conflict resolution techniques and technical
    skills training that specifically focused on improving work
    competencies that would change as a result of the 777's new
    digital environment.  To ensure all 777 staff were equally
    trained, employees were required to complete training before they
    reported to the program.  For example, the professional employees-
    engineers and drafters-were required to complete 120 hours of
    start-up training on several key 777 practices, including design
    build teams and computer-aided three-dimensional interactive
    application1 software.  Teams were often trained together at the
    work location. Boeing officials stated that training was
    instrumental to the implementation of key practices on the 777
    program, such as design build teams-essentially IPTs.  The
    officials stated that design build teams were at odds with the
    company's culture because employees were not accustomed to working
    in a team environment and sharing information across functional
    areas. 1 This application is a computer-based design tool that
    allows designers the opportunity to view design drawings and the
    interface of the millions of airplane parts as three dimensional.
    Page 41
    GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for
    Best Practices Do Not Go as Far as Leading Commercial Firms Figure
    3.1:  777 Airplane Boeing's use of design build teams created a
    major culture change for the staff assigned to the 777 program.
    Source: The Boeing Company. Boeing officials believe their
    partnership approach improved their training. Training
    representatives stated that the partnership resulted in program
    management support, which ultimately led to acceptance from the
    program staff.  The representative stated that since Boeing has
    involved program staff in decisions regarding training, course
    "no-show" rates have decreased.  A senior manager for program
    operations for the 777 program stated that because he and other
    senior program leaders drove key training decisions, the training
    was tailored to the staff's needs and provided the necessary
    skills and orientation to work in the new environment. Page 42
    GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for
    Best Practices Do Not Go as Far as Leading Commercial Firms Ford
    Motor Company    The Ford Motor Company created an organization,
    Product Development Process Leadership, with the singular purpose
    of supporting its 100 program offices in designing new or
    modifying vehicle lines in the implementation of the FPDS--a lean
    engineering process.  The organization provides internal
    communication regarding FPDS to the teams, and its training
    representatives work with the teams to conduct an analysis to
    learn first hand if impediments to FPDS implementation exist.
    Other support, such as team coaching, is provided to facilitate
    the engineering team's implementation of FPDS in the workplace.
    Figure 3.2:  Ford Focus Ford uses training to improve the
    timeliness and quality of new vehicle launches. Source: Ford Motor
    Company. Ford officials told us that their training focus is to
    provide practical skills just in time-that is, when it coincides
    with the need to apply the skills on Page 43
    GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for
    Best Practices Do Not Go as Far as Leading Commercial Firms the
    job.  Ford provides over 50 hours of training to instruct
    engineers on how to do their jobs using the FPDS process.  To that
    end, Ford uses internal subject matter experts-engineers who have
    been on teams designing new or modifying existing vehicle lines-to
    help tailor FPDS course topics to make them relevant to the work
    environment.  Ford officials stated that the subject matter
    experts understand the details of the FPDS and are needed to
    ensure that the training developed is practical at the working
    level.  According to the manager responsible for training
    employees to launch new vehicles at Ford, his office pulls
    together the training that is necessary to get the job done.  To
    further improve practicality and timeliness, the manager stated
    that on-the-job-training at the worksite is used to the extent
    possible. IBM                    IBM's Center for Excellence
    provides in-house education consultants to personally understand
    the business situation and product.  The consultants interview
    business unit staff to assess the staff's training needs and
    identify inhibitors to implementing new practices.  The
    assessments can take 2 weeks to 4 months, depending on the size of
    the business unit.  An IBM trainer for Object Orientation Project-
    a companywide software development practice-explained that
    providing practical training is one of the Center's guiding
    principles.  To achieve this goal, the company first stresses the
    importance of using experienced instructors who have practical
    knowledge in "doing" what they are teaching.  Second, employees
    are assigned to an Object Orientation Project before they take the
    5- to 7-week immersion training course.  According to the IBM
    trainer, this requirement has improved practicality because the
    students have better knowledge retention, as opposed to having to
    wait 6 months to apply the information.  Lastly, to further
    enhance practicality and relevance, IBM integrates case studies
    with real examples related to the student's next assignment into
    the Object Orientation Project training.  According to an official
    from IBM's Center of Excellence, most training is provided and
    tailored to entire work teams or, at a minimum, to individuals
    with common responsibilities. Motorola University    Motorola
    University focuses on providing training and education solutions
    for its business units. The university recently began to assign an
    account management team to consult and advise senior leadership
    for each business sector in order to anticipate and provide
    appropriate training. The management team partners with the
    business unit staff to identify their training needs for
    implementing new practices, such as Five Nines-an effort to
    improve product reliability to the level of 99.999 percent.  The
    account team works with the business units to assess their
    training needs Page 44
    GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for
    Best Practices Do Not Go as Far as Leading Commercial Firms and
    develop a plan to meet those needs.   This partnership enables the
    university to customize its training to the specific needs of the
    various types of engineers, such as software engineers, within the
    business units. A software engineer stated that the university
    provides separate training courses tailored to meet the different
    and often distinct needs within the engineering community
    implementing Five Nines.  For example, a software engineer could
    receive up to 120 hours of training targeted to implementing Five
    Nines.  Although the account management structure is relatively
    new, the engineer observed that most business unit staff are
    reacting favorably to the effort because the coordinated approach
    provides one-stop shopping. DOD Does Not Target  DOD does not have
    a counterpart to the commercial hands-on approach for Training on
    Key                directly assisting key implementers of the best
    practices. DOD relies primarily on its standard training,
    including DAWIA courses, augmented by Practices to Program
    videos, internet-based training, satellite broadcasts, and
    roadshows, to Offices                        inform staff on best
    practices. These venues were designed to focus on functional
    training and to increase the awareness of new practices.  As such,
    they do not provide the necessary depth or reach enough of the
    right people at the right time to be of help in implementing best
    practices at program offices.  Responsibility for training on best
    practices is diffused among several DOD organizations, including
    DAU and the service acquisition reform offices. We did not find an
    organization that was able to tailor and help deliver training on
    best practices to the program offices we visited.  Furthermore,
    DOD does not systematically involve program office staff and other
    implementers in key decisions regarding best practices training.
    Currently, no feedback mechanism exists to determine the effect of
    DOD's training on the implementation of new practices at the
    program office level. DAWIA Training Not Well        DAU training
    is designed primarily for employees seeking career level Suited as
    a Primary Means      certification in the acquisition workforce,
    as required by the DAWIA for Conveying Best             standards.
    DAU incorporates best practice topics into the DAWIA courses
    Practices                      as drop-in modules that provide a
    survey of the topic.  The information conveyed is enough to
    provide a general awareness of the concept but not enough to
    implement the practice at the workplace.  While this approach may
    provide sufficient information for the target audience-newer
    acquisition employees-it has inherent limitations when it comes to
    providing best practices' implementation training to the entire
    acquisition workforce.  For example, DAU courses have been
    developed by functional Page 45
    GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for
    Best Practices Do Not Go as Far as Leading Commercial Firms boards
    that teach skill-based competencies for functional career fields
    versus best practices.  As a result, DAU courses are primarily
    aligned along career fields, such as engineering and cost
    estimating. Because personnel assigned to a particular functional
    area are given priority for training in that function, program
    officials told us that it is difficult for personnel outside of
    their areas to take courses.  On one program, an official noted
    that because of limited space, only 5 to 10 percent of the program
    staff can take training outside of their functional areas each
    year. One program manager believed that the functional training,
    while important to career fields, was no longer as relevant to the
    role of today's program office.  He noted that the combination of
    best practices, delegation of key responsibilities to contractors,
    and fewer staff has altered the program office's role.  In his
    opinion, the program office used to be closely involved with
    managing the design of a weapon system and double-checking the
    prime contractor's work.  In this role, the program office was
    organized and operated along functional lines, and functional
    training was relevant to how a program office operated.  He
    observed that today the program office is not as involved with the
    designing of the weapon system, nor is it able to mirror the
    contractor's functions.  Rather, the program office must be expert
    at what the government can control- which the program manager
    referred to as key leverage points.  These leverage points include
    the requirement trade-off process, the selection of a prime
    contractor, and the establishment of key relationships that enable
    the program office staff to have insight into the contractor's
    actual progress.  Program office staff, working in an IPT
    environment, must have knowledge of multiple career fields and
    work in a cross-functional setting. As a result, he believes that
    functional training alone no longer covers the things most
    critical to a program manager. The kind of training that is
    needed-such as on the leverage points-must be obtained elsewhere
    by the program office. The usefulness of the DAU courses is
    further hampered by limited availability, which restricts program
    offices from receiving training when needed.  According to a DAU
    school representative, the consortia of schools can train about 10
    percent of the workforce each year. Furthermore, more senior staff
    may have limited exposure to best practices because the majority
    of them have already met training requirements and are not taking
    the certification courses that introduce best practices.  While
    these staff are not prohibited from taking certification courses
    as part of their continuing education requirements, availability
    is limited and priority is given to individuals seeking Page 46
    GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for
    Best Practices Do Not Go as Far as Leading Commercial Firms
    certification.  These inherent limitations are consistent with the
    training shortfalls noted by program office staff in chapter 2.
    Other Training Methods Are  DAU's Acquisition Reform Communication
    Center (ARCC) is a key avenue More Dedicated to Best
    for disseminating information on the best practices to the
    acquisition Practices but Are                  workforce. ARCC
    provides training through videos and periodic satellite Awareness-
    oriented                 broadcasts on a variety of best practice
    topics. However, ARCC does not track attendees and has no
    assurance that the workforce adequately receives the training.
    For those that do attend, the introductory nature of the training
    may not provide the depth or specificity to implement the practice
    at the workplace or in a time frame that is helpful. The
    acquisition reform offices in the services communicate best
    practice information through acquisition reform courses, periodic
    satellite broadcasts, and informational videos, which are
    sufficient for broad exposure on best practices but again are of
    limited depth for practical application. Roadshows, traveling
    multiday training workshops provided to staff at a number of
    locations, and Acquisition Reform Week, designated by DOD as an
    opportunity for all service organizations to cease their normal
    operations and focus on acquisition and logistics reform
    initiatives, are also used to provide best practices training to a
    wide range of the acquisition workforce.  These methods also have
    limitations in depth and workforce coverage. For example,
    roadshows typically provide awareness training on the practices
    and do not provide in-depth information needed for implementation
    at the workplace.  A program official believes that only 10 to 15
    percent of the acquisition staff attend the second day of roadshow
    workshops, where more detailed training is provided.  The annual
    Acquisition Reform Weeks also provide awareness level training.
    Neither method is tailored to specific program offices or provides
    assurance that it is delivered at the time most needed by the
    workforce. DOD Training Organizations  DOD does not have
    organizations that are comparable to those in Not Set Up to Help
    Design          commercial firms and that work with program
    offices in identifying best or Deliver Tailored Training
    practices applicable to a particular program, designing an
    approach to to Program Offices                 training the
    program office staff, or delivering the training to the program
    offices.   DOD training organizations-those that deliver DAWIA and
    acquisition reform training-are not set up or have the resources
    to deliver best practices quickly and easily to program staff, and
    then ensure implementation at the working level. Page 47
    GAO/NSIAD-99-206 Best Practices Chapter 3 DOD Training Methods for
    Best Practices Do Not Go as Far as Leading Commercial Firms The
    DAU consortium integrates best practices topics into DAWIA
    certification or acquisition reform training but does not develop
    training specifically for implementing the practices.  Although
    DAU has a substantial full-time faculty, the faculty members' main
    priority has been to teach in the classroom.  One course director
    informed us that faculty members are evaluated on the basis of
    hours of training provided in the classroom, which implicitly
    discourages work outside the classroom, such as consulting with
    program offices.  Also, little or no consultation occurs between
    the course designers and the implementers on how to best implement
    a practice at the program office level.  The Director of DOD's
    International and Commercial Systems Acquisition unit stated that
    DOD's current structure for defining and developing training
    courses does not have clear accountability to ensure that training
    on best practices is provided at the program level.  Several
    program office staff informed us that they have not been given the
    opportunity to provide input to influence the content of the
    courses.  Instructors for the courses, who do have frequent
    contact with program office staff, also believed that their
    ability to influence the content of the courses was limited. DOD
    does use internet site surveys to obtain staff feedback on
    acquisition reform training, which could help tailor the courses
    more to the needs of the program offices. However, program office
    staff believed the survey forms were inadequate for addressing the
    specific training needs of individual program offices.
    Organizations with more direct responsibility for providing
    training on best practices are staffed to the level needed to
    design distance learning courses, such as web-based training,
    satellite broadcasts, and instructional videos.  For example, the
    services' acquisition reform offices each employ from 8 to 20
    people, not nearly enough to provide on-site training or in-depth
    consulting to individual program offices. Some DOD organizations
    have recognized the value of providing training tailored to
    specific program offices and are attempting to go beyond
    traditional training roles. However, these are largely ad hoc
    efforts that do not go as far as commercial methods.  For example,
    the Navy is creating Total Ownership Cost teams, that are to
    advise program offices on the use of CAIV. Navy officials stated
    that due to limited resources, the teams will only reach about 15
    of the 300 program offices per year. The Air Force plans to begin
    using training support teams to address best practices issues from
    the program office perspective.  However, the teams have not yet
    begun and full operation is not scheduled to begin until fiscal
    year 2001. Also, an Air Force official stated that another team,
    created to address acquisition reform training issues, will not
    involve program managers, which raises the question of whether
    program office needs will be adequately considered. Page 48
    GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial
    Training Methods Reflect Different Strategic Approaches to New
    Practices
    Chapter 4 The intensive, hands-on training methods leading
    commercial firms employ on new practices are the result of a
    systematic, institutionally driven approach to implementation.
    These firms commit to and concentrate their resources and
    attention on a few well-defined practices and make a significant
    front-end investment in providing the training to the primary
    implementers.  Additionally, the firms strive to create a
    supportive environment to put the implementers of the practice in
    a good position to succeed.  Their objective is to have successful
    implementation at the work level; training methods are shaped to
    meet that objective.  DOD's training methods for best practices do
    not benefit from a similarly strategic approach for deploying new
    best practices and provide little assurance that the practices are
    consistently implemented in the workplace. DOD has promulgated
    numerous initiatives in the past few years without communicating
    their relative priority to trainers and implementers. Often, the
    initiatives were not accompanied by clear guidance or by the
    initial training needed for implementation. While DOD commits
    significant resources to training, it does not make a uniform
    front-end investment to ensure the implementers of new practices
    will succeed.  This approach depends more on the individual
    program offices to recognize the need for and to make this
    investment. Recent DOD actions reflect its recognition that
    training improvements are needed.  Two plans proposing divergent
    improvements to the training of the acquisition workforce are now
    being considered.  One plan does not focus on best practices
    training specifically, offering instead an incremental approach to
    changing the DAU structure.  The other targets implementation of
    best practices directly, with a broad scope of efforts that would
    require substantial changes from the current approach to training.
    The latter plan is much more aligned with the commercial approach
    to training described in this report.  However, if it is adopted,
    it faces many challenges to providing better training for program
    offices.  Another recent DOD action-creation of a continuous
    learning policy-may provide more training, especially for more
    senior members of the acquisition workforce, but it does not
    ensure that best practices will be included. Page 49
    GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial
    Training Methods Reflect Different Strategic Approaches to New
    Practices Leading Firms'            Although the four companies we
    contacted initiated very different types of Strategic Approach
    practices, they exhibited a similar strategic approach to ensuring
    that the key practices were implemented.  The elements of this
    approach can be Better Ensures            summarized as
    Implementation of Key Practices                 * clear vision, *
    adoption of few practices at any one time, * assessment and
    identification of target population for implementation, * well-
    defined goals, * significant investment in training and other
    vehicles to aid implementation, and * supportive environment to
    facilitate implementation. In general, we found that the firms
    shared a common strategy for implementing key practices that were
    intended to change company culture. First, the firms' corporate
    management committed to and adopted seven or less key practices at
    any given time.  One firm we contacted adopted only one or two key
    new practices, which enabled the company to concentrate its
    attention and target resources, including training, to the
    implementation of the practice, and signaled the importance of the
    practices to trainers and implementers.  Second, for practices
    that were not companywide, the firms assessed which staff should
    be included in the implementation.  For example, Ford's training
    unit assessed each existing program team working on vehicle lines
    beyond the initial development phase to determine if the team
    could benefit from adopting portions of FPDS or if some older
    teams should be excluded.  An IBM official stated that while the
    company currently promotes six practices, not all will apply to
    every segment of the employee population.  Third, once the target
    population was identified, company leaders made the implementation
    of the practice mandatory.  For example, the Ford Motor Company
    required that all new vehicle lines built for the year 2000 and
    beyond use FPDS.  Lastly, to further assist in the implementation,
    the companies developed well-defined goals to better ensure that
    the target population consistently understood how to apply a new
    practice to improve product production-the ultimate goal. The
    firms we contacted ensured that the implementers of the practice
    received the assistance necessary to succeed.  Consequently, these
    companies made a significant investment, including a comprehensive
    front-end effort, to support the needs of program offices that
    would implement the practice.  According to a Ford training
    official, if the initiative is important enough to the
    organization, then the investment is Page 50
    GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial
    Training Methods Reflect Different Strategic Approaches to New
    Practices justified.  Other companies agreed with that way of
    thinking. Boeing's Learning Program Development Director
    summarized the corporate training strategy for implementing new
    practices as one that includes a clearly stated vision or mission
    statement, well-defined goals, and enablers, such as training and
    good processes, to support the implementers.  This philosophy
    enabled Boeing to take a year to develop the training program
    tailored to the 777 program-which was intended to change the
    corporate culture and encourage employees to rethink how they did
    their jobs.  Both Boeing training and program officials believe
    that the training investment resulted in the successful
    implementation of the key 777 program practices. While the company
    officials acknowledged that training was instrumental in the
    implementation of the key practices, nearly every official we
    spoke with stated that training is just one of the necessary
    components. They noted that creating the right environment is key
    to the successful implementation of new practices and that the
    quality of training was dependent on this environment. Company
    officials stressed that strong leadership is often another key.
    An IBM official stated that, at inception, top leaders need to
    provide sufficient funding for training, well-defined
    expectations, clear direction, oversight, continued interest, and
    incentives to ensure that the new practices are possible to
    implement.  The manager for the 777 program stated that Boeing's
    management works in teams-a key practice. He believed that it was
    management's ability to lead by example that helped to prevent a
    return to the former functional way of operating.  These companies
    believe that other factors,  such as an accommodating
    organizational structure, good internal communication, consistent
    application, and supportive technology, are needed to foster the
    implementation of key new practices. DOD's Training on Best  DOD
    does not have a uniform or defined strategic approach to Practices
    Does Not              implementing best practices. Accordingly,
    training methods are not shaped by the same drive for
    implementation found in leading commercial firms. Stem From a
    Strategic  DOD's approach commonly begins with policy memorandums
    widely Approach                        distributed to the entire
    acquisition workforce.  However, the policy provides little
    specific guidance on how or which program offices should implement
    the practice.  Since 1994, DOD has proposed close to 40
    acquisition reform initiatives without an indication of relative
    priority.  The absence of priorities makes it difficult to not
    only develop training courses but also to determine which courses
    are most appropriate for the needs of program offices. Page 51
    GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial
    Training Methods Reflect Different Strategic Approaches to New
    Practices DOD's Policy on Best         DOD's policy on
    implementing the best practices is promulgated without a Practices
    Is Not Coupled     strategy or specificity as to whom within the
    acquisition workforce is With a Strategy for          expected to
    implement the practices or which program offices are in the
    Implementation               best position, in a program life
    cycle, to implement the practices.  Typically, the policy is not
    promulgated in conjunction with a detailed plan that outlines the
    support and training required by the program offices tasked with
    implementing the practice.  Similarly, a 1997 Coopers & Lybrand
    study on acquisition reform implementation found inconsistencies
    in DOD's implementation of new practices.  It stated that these
    inconsistencies may have resulted from a lack of an integrated,
    cohesive DOD strategic plan to put them in context to each other
    and to larger strategic goals.  While DOD has made some progress
    in establishing such goals through its Government Performance
    Reporting Act plans, organizations responsible for training and
    implementing the practices have not experienced the effects. In
    addition, the policies themselves are not always clear.  For
    example, although the CAIV initiative was promulgated in 1995,
    training officials reported they did not have a clear
    understanding of what CAIV means. AAAV program officials said they
    developed training for 1997 to clarify the CAIV concept for the
    program. A November 1998 Air Force workshop on CAIV reported that
    the concept is still not well understood or widely implemented.
    Similarly, the IPT policy states that IPTs should be implemented
    "when it makes sense."  The policy for the use of past performance
    data states that the practice should be implemented to the "degree
    possible."  While a fair degree of latitude is good for
    implementers, some DOD officials believe that this level of
    generality is insufficient to ensure implementation of the
    practice.  For example, a training official observed that policies
    are not well defined because of the desire to bring new practices
    quickly to the workforce and so some are deployed when they are
    not ready for implementation.  According to a DAU official,
    unclear and general policies can contribute to ineffective
    training because it makes it difficult to develop a course of
    instruction that teaches a unified, cohesive way to implement a
    policy.  A program official noted that general policy statements
    provide little guidance or assistance in the program offices'
    implementation of the practice, even though program offices are
    often the primary implementers of best practices. Page 52
    GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial
    Training Methods Reflect Different Strategic Approaches to New
    Practices DOD Attempts to              In addition to IPTs,
    performance specifications, CAIV, and past Implement Many
    Practices     performance, DOD has proposed numerous reform
    initiatives since 1994. Without Prioritization       In 1997, the
    Secretary of Defense proposed several new reform initiatives under
    the umbrella of the Defense Reform Initiative.  These initiatives
    have been promulgated without an indication of relative priority,
    leaving educators and implementers to decide on what is important.
    The lack of prioritization makes it difficult to determine
    training needs, especially when the number of initiatives
    continues to increase.  While the prolific offering of new ideas
    can stimulate innovation, the absence of priorities makes it
    difficult to focus training on the specific initiatives that are
    the most important to implement.  Moreover, program management
    offices are not necessarily in a good position to sort through the
    initiatives to focus on those that are the most important to the
    job at hand. Several DOD officials expressed concerns regarding
    the number of initiatives without prioritization.  For example, a
    DAU training official noted that it is impractical to expect to
    train the entire workforce on all of the initiatives and that the
    Office of the Secretary of Defense should set the priorities for
    implementation of initiatives. A DAU consultant echoed the view
    that there are too many new reform initiatives and that DOD needs
    to set priorities on which initiatives to address.  A service
    acquisition reform official observed that the large numbers of
    best practices promoted by DOD overwhelms the services.  This
    official told us that the three services' acquisition reform
    offices meet quarterly to share progress on acquisition reform
    initiatives.  He believes these meetings could serve as an
    opportunity to prioritize the practices.  However, the current
    focus of the meetings is unclear and the offices' charters are
    under review.  Another acquisition reform office representative
    stated that the combination of a large number of reform
    initiatives and unclear priorities puts the office in the position
    of having to guess at what is the most important.  This leads to
    emphasizing what is perceived to be popular, he added. Other
    officials explained that due to the frequency of new initiatives
    being introduced, training courses should be regularly revisited.
    However, training personnel have not been able to consistently
    keep up with the pace of change.  For example, a service
    acquisition reform official stated that it requires at least a
    full year to determine the elements of a new initiative and then
    the challenge of how to train the initiative still exists.  A 1997
    DOD study team that reviewed DOD's acquisition education and
    training structure and process found that curriculum development
    often lagged behind the rapidly changing policy requirements.  The
    study also noted that instructors were not up to date on current
    acquisition policies and that Page 53
    GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial
    Training Methods Reflect Different Strategic Approaches to New
    Practices because they were out of touch with DOD policy makers,
    the instructors found it difficult to develop up-to-date training
    courses. DOD Proposals to Alter  DOD is aware of the need to
    improve the means by which the acquisition Training Reflect
    workforce receives and implements new initiatives.  Two proposals
    being considered by the Under Secretary of Defense for Acquisition
    and Conflicting                     Technology that are intended
    to improve workforce training, and a new Approaches
    continuing education policy has been put in place.  The first
    proposal is a DAU transition plan for improving DAU's structure.
    The plan is based on a 1997 study by a DOD team and a 1998 study
    by the Logistics Management Institute.  Both studies confirmed
    strengths in the DAU approach, particularly relating to functional
    or skill-based training and the weaknesses in some of the same
    areas that surfaced in our work.  The studies made recommendations
    for significant organizational and operational changes in DAU to
    better meet the educational needs of the acquisition workforce.
    However, a more moderate approach was adopted in preparing the
    transition plan for DAU.  While the proposed structure offers
    improvements, it does not discernibly address key weaknesses in
    the training of best practices.  The plan has yet to be approved.
    The second proposal, a report from the DOD Section 912(c)
    Commercial Business Environment Group directly advocates
    incorporating best practices and is much broader in recommending
    changes.  It proposes that DOD's acquisition structure be
    transformed into a team type of organization with members drawn
    from across the DOD procurement enterprise.  A new learning
    organization would be created specifically to support accelerated
    change. DOD has also instituted a continuous learning policy,
    which should help people obtain current training, even if they are
    already fully certified. However, the required levels of
    continuous learning can be met in many ways and do not place any
    particular emphasis on training in best practices.   These DOD
    efforts are a step in the right direction for seeing that program
    offices have the information they need, but none focus on the
    program offices' needs.  Given the pivotal role they play in
    weapons acquisition, a more direct approach may be required. Page
    54                                           GAO/NSIAD-99-206 Best
    Practices Chapter 4 DOD and Commercial Training Methods Reflect
    Different Strategic Approaches to New Practices DAU Transition
    Plan Is     The DOD transition plan for restructuring DAU was
    based on Largely Silent on          recommendations of a 1997
    process action team that DOD chartered to Improvements to Best
    create a clear vision and a structure for future workforce
    training and Practices Training         education.  Several
    findings of the study related to training for new initiatives and
    practices, including (1) the existing curriculum design process
    was functionally driven, did not address the needs of people
    outside the functional career fields, and did not lend itself to
    successful development of cross-functional curricula; (2) a
    mechanism was needed to validate and prioritize requirements
    before they are submitted; (3) faculty members did not incorporate
    up-to-date case studies; (4) DAU schools used procedures for
    measuring performance that focused on student satisfaction with a
    course, not whether the user community's acquisition education
    requirements were being met; and (5) DAU's use of technology-
    based learning was insufficient. The team recommended that DAU be
    replaced with a unified Defense Acquisition Institute that would
    have responsibility for development and delivery of acquisition
    training, the consortium be streamlined to reduce duplication, and
    lines of authority between DAU and the members be strengthened.
    The team envisioned the institute as, among other things, (1)
    fostering innovation and facilitating reform and continuous
    improvements, (2) having a clear message of support from DOD
    acquisition leadership that education is the key to meeting DOD
    goals, (3) identifying acquisition training needs that are cross-
    functional areas or that are multifunctional in nature, (4)
    focusing on the needs of the learner, and (5) employing a process
    to validate the effectiveness of the training received.  However,
    a dissenting opinion offered by one of the team members to evolve
    the current DAU organization structure through a series of near-
    term and long-term actions was accepted by the Under Secretary of
    Defense. Subsequently, the Logistic Management Institute conducted
    a review for DOD and developed a structural reorganization for DAU
    that was to be used as the starting point for the transition plan.
    The Institute noted that DAU training was generally skill and
    specialty oriented, with limited overlap among "stove-piped"
    career fields.  It recommended that the new structure be designed
    along the lines of a corporate university.  As such, its
    curriculum would include cross-functional training and business
    executive development and act as an agent of change, being more
    involved with the workforce in determining how changes would be
    carried out. The DAU president submitted the draft transition plan
    for restructuring in September 1998.  As of June 1999, DOD had not
    approved the plan.  The Page 55
    GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial
    Training Methods Reflect Different Strategic Approaches to New
    Practices plan has several features that offer promise for more
    effective training overall.  These include (1) a unified structure
    in which DAU has full responsibility for ensuring that the overall
    course structure meets the needs of the acquisition workforce, (2)
    consolidation of the consortium schools into four main campuses
    under the direct authority of DAU, (3) an increased emphasis on
    distance learning techniques,  (4) steps to improve the
    qualifications of faculty, and (5) replacement of functional
    boards with functional IPTs that have representation from both
    functional experts and instructors and that will design curricula
    together. Other features of the plan raise doubts about its
    ability to provide a strategic approach to shape training that
    will help implementation of best practices at program offices.
    The transition team that prepared the plan concluded that DAU was
    unique and could not be matched with other institutions, like
    corporate universities.  Thus, the corporate university model
    recommended by the Institute was not adopted. While the curriculum
    may benefit from the functional IPTs, the plan does not mention
    the need for cross-functional or best practices training or for
    DAU to accept the role of a change agent.  It is silent on
    prioritizing initiatives or improving feedback.  While it is
    possible that under the proposed plan DAU could play a more active
    role in the design and delivery of tailored training to the
    workplace, the plan does not that suggest that, other than
    increased distance learning, training will be offered in a manner
    substantially different from what has been traditionally offered
    or that the relationship between DAU and the acquisition community
    will be any closer. Commercial Business               The National
    Defense Authorization Act for Fiscal Year 1998, section 912,
    Environment Plan Proposes  directed the Secretary of Defense to
    submit to Congress an implementation Major Changes to Hasten
    plan to streamline the acquisition organizations, workforce, and
    Reform                            infrastructure.  As part of that
    mandate, the Deputy Under Secretary of Defense for Acquisition and
    Technology formed a team to develop a plan for ensuring that new
    practices would be incorporated by DOD.  The draft plan, dated
    June 1999, concludes that adopting the most effective commercial
    practices requires a cultural and organizational transformation
    within DOD.  The draft proposes a very broad approach to making
    this change, which includes a training regime to implement key
    commercial business practices and a change model to accelerate the
    implementation of new practices.  The model calls for establishing
    goals and a scorecard to measure whether they are met.  To inform
    everyone of the goals, a tiered approach is envisioned, starting
    with senior leaders, rolling down to teams with specific reform
    target goals.  The teams are to report on their Page 56
    GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial
    Training Methods Reflect Different Strategic Approaches to New
    Practices outcomes within 60 days.  A mission support office will
    support the teams as well as capture results across DOD and make
    them available to others. This office is to be led by a senior DOD
    official and staffed by management and facilitation experts, and
    possibly outside contractors. The plan proposes a team approach
    for managing acquisitions that embraces best practices and
    maintains "learning organizations that seek out and adopt best
    practices that improve individual and organizational performance."
    The plan also calls for new organizational roles in adopting of
    best practices.  Among these is DAU.  The plan recommends that DAU
    be broadly recast to adopt the corporate university model and
    become a change agent.  This plan, while not specific about the
    help that program offices would receive, calls for a strategic
    approach that would make it more likely that DOD could provide its
    program offices tailored training- more help-in implementing best
    practices.  Nonetheless, translating this plan into the type of
    training and other help program offices need to implement best
    practices is challenging.  Key elements, such as the cascading of
    goals from senior levels on down and the revision of what
    constitutes teams, are major undertakings and challenge long
    established patterns of interaction.  How the mission support
    office would reach specific program offices is unclear, as is how
    it would interface with DAU. Also, whether initiatives are to be
    prioritized and how DAU would be restructured to provide
    functional training and become a change agent for new practices
    remains to be delineated. Continuous Learning Policy  The
    objective of DOD's recently issued continuous learning policy is
    to Does Not Ensure Best               augment the acquisition
    training standards for career field certification of Practices
    Training                 acquisition professionals.  The policy
    requires all civilian and military members of the acquisition
    workforce to complete 80 hours of continued education within a 2-
    year period.  As such, it is a good step for reaching more senior
    people in the acquisition workforce who would normally not receive
    this kind of training because they are fully certified.  The
    policy also places increased emphasis on distance learning, which
    may address some of the timeliness and availability issues raised
    by program office officials in obtaining useful best practice
    information.  One distance learning course that deals directly
    with a best practice was jointly developed last year by the
    National Contract Management Association and the National
    Association of Purchasing Management at the request of DOD.  It is
    an internet-based course to integrate best commercial practices
    for managing suppliers into DOD acquisitions.  It entitles
    participants to 24 hours of Page 57
    GAO/NSIAD-99-206 Best Practices Chapter 4 DOD and Commercial
    Training Methods Reflect Different Strategic Approaches to New
    Practices continuous education credit and participants are
    encouraged to take the course as teams. While the policy does not
    preclude staff from receiving training on new practices, given the
    array of options for meeting requirements, it does little to
    ensure that best practices training will be received.  An earlier
    version of the policy recommended 16 units of acquisition reform
    training that could have included best practices, but the
    requirement was deleted.   The policy encourages acquisition
    reform training but allows for requirements to be met in many
    ways-including courses, professional activities, and on the job
    experience-which may or may not include best practices
    information. According to a training official, crediting
    acquisition workforce members for almost anything dilutes the
    policy. Officials of one program noted that no one advises them on
    what courses to take.  They can take courses or attend conferences
    that they believe would benefit them, but they believed this may
    be inadequate for ensuring that senior officials are aware of new
    ideas.   Another program official said that her program office
    could meet continuing education requirements by mandatory
    attendance of Acquisition Reform Week.  While this exposure would
    increase awareness of new practices, such training was
    characterized as providing just a "flavor" for the initiative-not
    providing enough information to implement the practice. Page 58
    GAO/NSIAD-99-206 Best Practices Chapter 5 Conclusions and
    Recommendations
    Chapter 5 Conclusions    Leading commercial firms and weapon
    system program offices have successfully implemented best
    practices or new initiatives by taking a tailored, on-site
    training approach.  Such an approach is a concentrated, up-front
    effort that creates an environment for accepting the practice
    within a program and provides the impetus and know-how to apply
    the practice at the workplace.  It is warranted by the investment
    and importance that large programs represent.  The difference is
    that within DOD, the approach is allowed to happen, while leading
    commercial firms ensure it happens.  These firms ensure that
    initiatives are successfully implemented at the program level by
    customizing the initiatives to fit the programs, working with
    program office staff to develop training that meets the needs of
    the program, and providing on-site assistance to the program. This
    approach would not necessarily be used every time by commercial
    firms to train their entire workforce or to impart every skill; it
    is instead an intensive approach they reserve for key initiatives
    and individual programs. It stems from an overall training
    strategy that restricts the number of initiatives that are brought
    to the workforce, ensures that the initiatives have well-defined
    objectives, and makes training organizations responsible for
    seeing that policy is turned into practice at the program level.
    Little regarding implementation is left to chance. Within DOD,
    this approach can be taken if a program's management has the
    insight to recognize the importance of an initiative, the
    necessary resources needed to implement the initiative at the work
    site, and support from top management.  Even though DOD has drawn
    its reform initiatives from the practices of leading commercial
    firms, it has not adopted the accompanying training strategy to
    ensure a concerted effort is made to implement key initiatives at
    the program level.  As a result, more is left to chance.  In the
    past 4 years, over 40 initiatives have been introduced to the DOD
    acquisition workforce, without delineating which are the most
    important for weapon system programs.  Several organizations are
    responsible for training and each service has an office that
    promotes acquisition reform initiatives.  Although these
    organizations are doing valuable work and are contributing to
    workforce training, it is unclear how, collectively, they can
    drive key initiatives to the program office level. DAU's main tool
    for training the acquisition workforce-functional training-is
    aimed at increasing the expertise of people in their career
    fields, but it does not have the throughput and ability to reach
    the program office as a work unit, nor the practical depth needed
    to imbed an initiative at the program office.  Other tools, such
    as satellite broadcasts, distance Page 59
    GAO/NSIAD-99-206 Best Practices Chapter 5 Conclusions and
    Recommendations learning, and Acquisition Reform Week, make
    program office staff aware of new initiatives, but they do not
    provide the depth or environment to implement them because DOD's
    training organizations and methods have not been designed to
    provide this kind of support. DOD recognizes the importance of
    training as a tool that can help infuse best practices into weapon
    acquisitions and has several draft plans to reorganize how
    training is delivered.  However, it is not evident that the
    planning and methods called for by the DAU transition plan will
    change the way such training is conceived and delivered to weapon
    system program offices.  The Commercial Business Environment Plan
    does call for a strategic approach that would make it more likely
    that DOD could provide its program offices tailored training-more
    help-in implementing best practices.  The fact that neither plan
    has been approved indicates that DOD has not decided what role it
    wants acquisition training to play on best practices. Whether
    through existing organizations or through new ones, DOD needs to
    increase its capability to provide tailored training of specific
    initiatives at the program office level.  Developing this
    additional capability will require a strategy for implementation
    and collaboration with program offices on the design of training.
    Without a concerted approach to foster the implementation of best
    practices by providing for customized training at the program
    offices, better outcomes in weapon programs will be more difficult
    to achieve.  The recommendations that follow address ways DOD can
    provide tailored training on selected practices to program
    offices.  The recommendations are made without prejudice toward
    the functional and other training DOD provides to the acquisition
    workforce. Recommendations    We recommend that the Secretary of
    Defense develop a strategy for delivering targeted training on
    selected new practices to program offices to ensure the practices
    will be implemented.   This strategy should accomplish the
    following. * Identify those initiatives most worthy of a targeted
    training investment by screening the numerous initiatives to
    separate those for which a general awareness is sufficient from
    those that warrant a targeted approach.  Those to be targeted
    should be relatively few in number at any given point in time.
    The process of setting priorities would be of a continuing nature
    and would benefit from the input of current program office
    members. Page 60
    GAO/NSIAD-99-206 Best Practices Chapter 5 Conclusions and
    Recommendations * Decide whether DAU is to play a more traditional
    role, as called for by the DAU Transition Plan, or a more
    proactive role, as called for by the Commercial Business
    Environment Plan and clearly communicate to DAU and other
    providers of DOD training their responsibilities in supporting a
    targeted approach to training. * Identify the key organizations
    and people that are critical to the program offices' ability to
    implement best practices, including those not currently defined as
    part of the acquisition workforce and contractors, as the
    potential audience of targeted training. We also recommend that
    the Secretary make the Under Secretary of Defense for Acquisition
    and Technology responsible for taking steps to institutionalize
    the methods for tailoring training on key initiatives that have
    been shown to be successful by leading program offices and
    commercial firms.  This involves having proactive organizations
    and the tools to inform, prepare, and assist program offices to
    implement the initiatives most applicable to their programs.  This
    approach should include the following. * Making new program
    managers aware of the initiatives that could significantly affect
    the outcomes of their programs and the role the managers play in
    having their staffs trained to implement those initiatives.  This
    could be done through existing program managers' courses. *
    Working with individual program managers to tailor the initiatives
    to their programs. * Developing an approach, in conjunction with
    the program managers, to create a culture-and necessary
    incentives-within the program office to make it receptive for
    adopting the initiatives. * Helping the program managers determine
    the best methods for making initial and sustaining resource
    investments in training the program office staff.  This help could
    consist of providing advice, assisting in the identification of
    experts in the needed areas, and culling lessons learned from
    other programs that have implemented the initiative. To introduce
    and reinforce new practices, we recommend that the Secretary
    require the Under Secretary of Defense for Acquisition and
    Technology more effectively use existing training vehicles to *
    incorporate new practices more quickly, * better reflect the
    changing role of program managers, Page 61
    GAO/NSIAD-99-206 Best Practices Chapter 5 Conclusions and
    Recommendations * ensure that instructors' knowledge keeps pace
    with the latest practices, and * provide more case study material
    that gives current implementation examples for a variety of
    situations. Agency Comments and  DOD concurred with a draft of
    this report and all of its recommendations Our Evaluation
    (see app. I).  DOD noted that it was taking steps to develop a
    strategy for delivering targeted training on selected new
    practices to program offices, consistent with our report.  It
    cited the Commercial Business Environment study team's vision for
    accelerating cultural change within the acquisition community, in
    conjunction with a strategy of delivering team training, to
    implement best practices.  DOD stated that it anticipates adopting
    and launching many of the study's recommendations without delay.
    DOD also recognized that DAU should augment its capability,
    consistent with the best practices of the private sector corporate
    universities, by embracing the role of "change agent" and by
    designating "performance consultants" who focus on developing
    tailored training to meet program team needs. DOD noted that it
    has also begun the restructuring of DAU to provide a more
    centralized and integrated education and training program that
    will also be integrated with the initiatives identified by the
    Commercial Business Environment report.  We agree that the DOD
    study frames a strategic approach that would make it more likely
    that DOD could provide its weapon system program offices tailored
    training to help them implement new practices.  We are also
    encouraged by the speed with which DOD is implementing the study's
    recommendations.  We do note that the study is not specific about
    the help that program offices would receive and that translating
    the study into such specifics will be a significant challenge. As
    DOD proceeds with implementation, we reiterate the importance of
    screening the numerous initiatives to identify those most worthy
    of targeted investments and to involve the staff of current
    program offices in setting these priorities. In agreeing on the
    need for targeted training, DOD noted that DAU's communications
    arm will work to provide for immediate delivery of tailored
    training on key initiatives to meet the needs of program offices.
    DAU's current communications arm-the ARCC-provides training
    primarily by distributing videos and periodic satellite
    broadcasts.  As such, ARCC may not be well-poised to take on the
    responsibilities of a proactive organization with the tools needed
    to inform, prepare, and assist program offices to help them
    implement the initiatives most applicable to their Page 62
    GAO/NSIAD-99-206 Best Practices Chapter 5 Conclusions and
    Recommendations programs, as we have recommended.  Regardless of
    which organization is assigned the responsibility, it is important
    that the organization go beyond the traditional approach of making
    standard training available and instead work with program offices
    to (1) tailor the initiatives to their programs; (2) develop an
    approach to help make their staffs receptive to adopting new
    practices; and (3) help the program managers-on site, if
    necessary- determine the best methods for making the investments
    in training the program office staff. Regarding making more
    effective use of existing training vehicles, DOD stated that
    during a review of core curriculum requirements, it will be
    completely reviewing the principal program management training
    toward implementing a fully integrated strategy.   It noted that
    the strategic plan will address the development of tailored case
    study materials, practical exercises, and assessment criteria for
    the adoption of new practices.   We believe that DOD's proposed
    review has the potential to more effectively promote best
    practices training.  In addition to these actions, as DOD conducts
    its review, it needs to have existing training vehicles
    incorporate new practices more quickly, better reflect the
    changing role of program managers, and ensure that instructors'
    knowledge keeps pace with the latest practices.  Greater
    involvement of weapon system program office staff in the design
    and content of these training vehicles could link the training
    more closely with the job at hand.  While DOD plans to focus its
    continuous learning activities on key acquisition reforms,
    currently, members of the acquisition workforce do not necessarily
    have to take training on acquisition reform to meet continuous
    learning requirements. Page 63
    GAO/NSIAD-99-206 Best Practices Appendix I Comments From the
    Department of Defense Appendix I Page 64             GAO/NSIAD-99-
    206 Best Practices Appendix I Comments From the Department of
    Defense Page 65                                    GAO/NSIAD-99-
    206 Best Practices Appendix I Comments From the Department of
    Defense Page 66                                    GAO/NSIAD-99-
    206 Best Practices Appendix II Description of Program Offices
    Visited
    Appendix II Following is a description of each of the program
    offices we visited. Advanced Amphibious  The Advanced Amphibious
    Assault Vehicle (AAAV) is to be used by the Assault Vehicle
    Marine Corps to provide high-speed transportation of troops from
    ships located beyond the horizon to the beaches.  It will also
    provide armor protection, land mobility, and fire support during
    combat operations on the shore.  It is a category I1 major
    acquisition program for the Marine Corps. The program office is
    staffed by about 75 people. Advanced                       The
    Advanced Medium-Range Air-to-Air Missile (AMRAAM) is a new Medium-
    Range                   generation radar-guided missile that
    fighter aircraft are to use against enemy aircraft.  It can be
    launched beyond visual range, day or night and in Air-to-Air
    Missile             all weather.  The missile is used on the Air
    Force F-15 and F-16 and the Navy F-14 and F/A-18.  It is a
    category I acquisition program for the Air Force.  The AMRAAM
    program started over 20 years ago.  About 70 people staff the
    program office. Battlefield Combat             The Battlefield
    Combat Identification System (BCIS) is a millimeter wave
    Identification System          electronic, question, and answer
    combat identification system capable of identifying friendly
    ground combat vehicles.  The BCIS interrogation is triggered
    automatically by activation of the shooter, which sends an
    encrypted query message to the targeted vehicle.  If the targeted
    vehicle is friendly and equipped with BCIS, its transponder
    answers with an encrypted friend message that is illuminated in
    the shooter's sights. It is a category II2 acquisition managed by
    the Army.  The program office is staffed by 16 core people and 30
    additional people who are assigned from other government
    organizations or contractor support. 1 Category I programs are
    defined as major defense acquisition programs estimated to cost
    over $355 million for research, development, test, and evaluation,
    or have procurement costs of more than $2.135 billion. 2 Category
    II programs are defined as acquisition programs estimated to cost
    over $75 million for research, development, test and evaluation,
    or have procurement costs of more than $300 million. Page 67
    GAO/NSIAD-99-206 Best Practices Appendix II Description of Program
    Offices Visited Joint Air-to-Surface              The Joint Air-
    to-Surface Standoff Missile (JASSM) is an autonomous, Standoff
    Missile                  long-range cruise missile to be capable
    of launch from outside area defenses to hit ground targets.  It is
    to be launched from a wide range of bomber, attack, and fighter
    fixed-wing aircraft.  It is a joint Air Force and Navy category I
    major acquisition program. Joint Strike Fighter              The
    Joint Strike Fighter (JSF) program's objective is to develop and
    field an affordable, highly common family of next generation
    multirole strike fighter aircraft for the Navy, the Air Force, the
    Marine Corps, and U.S. allies.  The focus of the program is
    affordability.  It is a joint service category I major acquisition
    program with Navy as lead.  About 100 people staff  the program
    office. Joint Surveillance and  The Joint Surveillance and Target
    Attack Radar System Joint Tactical Target Attack Radar
    Terminal (JSTARS-JTT) is to provide warfighters with near real-
    time tactical intelligence and targeting information.  It is a
    terminal that supplies SystemJoint Tactical  the critical data
    link from various intelligence sources to battle managers Terminal
    across all services.  The terminal is integrated into other weapon
    systems and is to be mounted in fixed- and rotary-wing aircraft,
    surface ships, and fixed or mobile ground platforms and vehicles.
    It is managed by the Army and is a category III3 program.  About
    10 people staff the program office. 3 Category III programs are
    defined as those acquisitions programs that do not meet the
    criteria for categories I or II. Page 68
    GAO/NSIAD-99-206 Best Practices Appendix III GAO Contacts and
    Staff Acknowledgments
    Appendix I II GAO Contacts       Louis J. Rodrigues (202) 512-4841
    Paul L. Francis (202) 512-2811 Acknowledgments    In addition to
    those named above, Charles Cannon, Monica L. Kelly, Gordon W.
    Lusby, Elisabeth G. Ryan, Sally Shipman, and Yelena K. Thompson
    made key contributions to this report. Page 69
    GAO/NSIAD-99-206 Best Practices Page 70    GAO/NSIAD-99-206 Best
    Practices Page 71    GAO/NSIAD-99-206 Best Practices Related GAO
    Products Best Practices: Commercial Quality Assurance Practices
    Offer Improvements for DOD (GAO/NSIAD-96-162, Aug. 26, 1996).
    Major Acquisitions: Significant Changes Underway in DOD's Earned
    Value Management Process (GAO/NSIAD-97-108, May 5, 1997). Best
    Practices: Successful Application to Weapon Acquisitions Requires
    Changes in DOD's Environment (GAO/NSIAD-98-56, Feb. 24, 1998).
    Best Practices: DOD Can Help Suppliers Contribute More to Weapon
    System Programs (GAO/NSIAD-98-87, Mar. 17, 1998). Defense
    Acquisition: Improved Program Outcomes Are Possible (GAO/T-NSIAD-
    98-123, Mar. 18, 1998). Defense Acquisition: Best Commercial
    Practices Can Improve Program Outcomes (GAO/T-NSIAD-99-116, Mar.
    17, 1999). Best Practices: Better Management of Technology
    Development Can Improve Weapon System Outcomes (GAO/NSIAD-99-162,
    July 30, 1999). (707343)    Letter    Page 72
    GAO/NSIAD-99-206 Best Practices Ordering Information The first
    copy of each GAO report and testimony is free. Additional copies
    are $2 each.  Orders should be sent to the following address,
    accompanied by a check or money order made out to the
    Superintendent of Documents, when necessary, VISA and MasterCard
    credit cards are accepted, also. Orders for 100 or more copies to
    be mailed to a single address are discounted 25 percent. Orders by
    mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC
    20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts.
    NW) U.S. General Accounting Office Washington, DC Orders may also
    be placed by calling (202) 512-6000 or by using fax number (202)
    512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of
    newly available reports and testimony.  To receive facsimile
    copies of the daily list or any list from the past 30 days, please
    call (202) 512-6000 using a touchtone phone.  A recorded menu will
    provide information on how to obtain these lists. For information
    on how to access GAO reports on the INTERNET, send an e-mail
    message with "info" in the body to: [email protected] or visit
    GAO's World Wide Web Home Page at: http://www.gao.gov United
    States                       Bulk Rate General Accounting Office
    Postage & Fees Paid Washington, D.C. 20548-0001            GAO
    Permit No. GI00 Official Business Penalty for Private Use $300
    Address Correction Requested

*** End of document. ***