    [Senate Hearing 111-149]
    [From the U.S. Government Publishing Office]

                                                            S. Hrg. 111-149

     ACQUISITION OF MAJOR WEAPONS SYSTEMS BY THE DEPARTMENT OF DEFENSE AND
           S. 454, THE WEAPON SYSTEMS ACQUISITION REFORM ACT OF 2009

    =======================================================================

                                    HEARING

                                   before the

                          COMMITTEE ON ARMED SERVICES
                              UNITED STATES SENATE

                         ONE HUNDRED ELEVENTH CONGRESS

                                 FIRST SESSION

                                   __________

                                 MARCH 3, 2009

                                   __________

             Printed for the use of the Committee on Armed Services

                      U.S. GOVERNMENT PRINTING OFFICE
    53-267                    WASHINGTON : 2009
    -----------------------------------------------------------------------
    For Sale by the Superintendent of Documents, U.S. Government Printing Office
    Internet: bookstore.gpo.gov  Phone: toll free (866) 512-1800; (202) 512�091800
    Fax: (202) 512�092104 Mail: Stop IDCC, Washington, DC 20402�090001

                          COMMITTEE ON ARMED SERVICES

                         CARL LEVIN, Michigan, Chairman

    EDWARD M. KENNEDY, Massachusetts     JOHN McCAIN, Arizona
    ROBERT C. BYRD, West Virginia        JAMES M. INHOFE, Oklahoma
    JOSEPH I. LIEBERMAN, Connecticut     JEFF SESSIONS, Alabama
    JACK REED, Rhode Island              SAXBY CHAMBLISS, Georgia
    DANIEL K. AKAKA, Hawaii              LINDSEY GRAHAM, South Carolina
    BILL NELSON, Florida                 JOHN THUNE, South Dakota
    E. BENJAMIN NELSON, Nebraska         MEL MARTINEZ, Florida
    EVAN BAYH, Indiana                   ROGER F. WICKER, Mississippi
    JIM WEBB, Virginia                   RICHARD BURR, North Carolina
    CLAIRE McCASKILL, Missouri           DAVID VITTER, Louisiana
    MARK UDALL, Colorado                 SUSAN M. COLLINS, Maine
    KAY R. HAGAN, North Carolina
    MARK BEGICH, Alaska
    ROLAND W. BURRIS, Illinois

                       Richard D. DeBobes, Staff Director

                   Joseph W. Bowab, Republican Staff Director

                                      (ii)

    ?

                                C O N T E N T S

                                   __________

                        CHRONOLOGICAL LIST OF WITNESSES

     Acquisition of Major Weapons Systems by the Department of Defense and
           S. 454, the Weapon Systems Acquisition Reform Act of 2009

                                 march 3, 2009

                                                                       Page

    Sullivan, Michael J., Director, Acquisition and Sourcing
      Management, Government Accountability Office...................     7
    Gansler, Hon. Jacques S., Chairman, Defense Science Board Task
      Force on Industrial Structure for Transformation...............    19
    Kaminski, Hon. Paul G., Chair, Committee on Pre-Milestone A
      Systems Engineering, Air Force Studies Board, National Research
      Council........................................................    25
    Adolph, Charles E. (Pete), Chairman, Defense Science Board Task
      Force on Developmental Test and Evaluation.....................    40

                                     (iii)

    ACQUISITION OF MAJOR WEAPONS SYSTEMS BY  THE  DEPARTMENT  OF  DEFENSE
         AND  S. 454, THE WEAPON SYSTEMS ACQUISITION REFORM ACT OF 2009

                                  ----------

                             TUESDAY, MARCH 3, 2009

                                           U.S. Senate,
                                   Committee on Armed Services,
                                                        Washington, DC.
        The committee met, pursuant to notice, at 9:36 a.m. in room
    SH-216, Hart Senate Office Building, Senator Carl Levin
    (chairman) presiding.
        Committee members present: Senators Levin, Lieberman,
    Akaka, Bill Nelson, Webb, McCaskill, Udall, Hagan, Begich,
    Burris, McCain, Chambliss, Thune, Martinez, Burr, and Collins.
        Committee staff members present: Richard D. DeBobes, Staff
    Director, and Leah C. Brewer, Nominations and Hearings Clerk.
        Majority staff members present: Creighton Greene,
    professional staff member; Peter K. Levine, general counsel;
    John H. Quirk V, professional staff member; Arun A. Seraphin,
    professional staff member; and William K. Sutey, professional
    staff member.
        Minority staff members present: Joseph W. Bowab, Republican
    staff director; Daniel A. Lerner, professional staff member;
    David M. Morriss, minority counsel; Lucian L. Niemeyer,
    professional staff member; and Christopher J. Paul,
    professional staff member.
        Staff assistants present: Jessica L. Kingston, Brian F.
    Sebold, and Breon N. Wells.
        Committee members' assistants present: Jay Maroney,
    assistant to Senator Kennedy; Bonni Berge, assistant to Senator
    Akaka; Christopher Caple, assistant to Senator Bill Nelson; Jon
    Davey, assistant to Senator Bayh; Gordon I. Peterson, assistant
    to Senator Webb; Stephen C. Hedger, assistant to Senator
    McCaskill; Jennifer Barrett, assistant to Senator Udall;
    Michael Harney, assistant to Senator Hagan; David Ramseur,
    assistant to Senator Begich; Brady King, assistant to Senator
    Burris; Sandra Luff, assistant to Senator Sessions; Clyde A.
    Taylor IV, assistant to Senator Chambliss; Jason Van Beek,
    assistant to Senator Thune; Erskine W. Wells III, assistant to
    Senator Wicker; and Kevin Kane, assistant to Senator Burr.

           OPENING STATEMENT OF SENATOR CARL LEVIN, CHAIRMAN

        Chairman Levin. Good morning, everybody. The committee
    meets today to consider the performance of the Department of
    Defense's (DOD) acquisition programs at a time when cost growth
    on these programs has reached levels that we cannot afford,
    including consideration of our bill, S. 454, the Weapon Systems
    Acquisition Reform Act of 2009, which Senator McCain and I
    recently introduced.
        Since the beginning of 2006, nearly half of the DOD's
    largest acquisition programs have exceeded the so-called Nunn-
    McCurdy cost-growth standards established by Congress to
    identify seriously troubled programs.
        As Secretary Gates pointed out in his testimony before our
    committee last month, the list of big-ticket weapon systems
    that have experienced contract or program performance problems
    spans the Services and includes the Air Force tanker, the CSAR-
    X, the VH-71, the Osprey, the Future Combat Systems (FCS), the
    Armed Reconnaissance Helicopter, the Littoral Combat Ship
    (LCS), the Joint Strike Fighter (JSF), and so on.
        Overall, DOD's 95 defense acquisition programs, known as
    Major Defense Acquisition Programs (MDAPs), have exceeded their
    research and develop budgets by an average of 40 percent, seen
    their acquisition costs grow by an average of 26 percent, and
    experienced an average schedule delay of almost 2 years. Last
    summer, the Government Accountability Office (GAO) reported
    that cost overruns on DOD's MDAPs now total $295 billion over
    the original program estimates. That's true, even though we've
    cut unit quantities and reduced performance expectations on
    many programs, in an effort to hold costs down.
        These cost overruns happen because of fundamental flaws
    that are endemic to our acquisition system. We have a pretty
    good idea of what those flaws are. Major acquisition programs
    fail because DOD: one, continues to rely on unreasonable cost
    and schedule estimates; two, establishes unrealistic
    performance expectations; three, insists on the use of immature
    technologies; and four, adopts costly changes to program
    requirements, production quantities, and funding levels in the
    middle of ongoing programs.
        Earlier this year, Under Secretary of Defense for
    Acquisition John T. Young, Jr., wrote a memo in which he sought
    to explain the cost growth on some of DOD's largest programs.
    This is what his memorandum said: ``A number of programs had a
    poor foundation and Milestone B, the starting point for major
    development and manufacturing design. . . Fundamentally, these
    programs moved past Milestone B with inadequate foundations
    built on artificially low cost estimates, optimistic schedules
    and assumptions, immature design or technology, fluid
    requirements, and other issues.''
        Mr. Young then went on to list the flaws of each MDAP. The
    JSF: too little understanding of the design; the FCS: fluid
    program strategy; the V-22: immature technology and Congress
    reversed DOD termination; the C-17: development issues and
    underfunding; the Army's Family of Medium Tactical Vehicles:
    design flaws; the CH-47F: low estimates and invalid
    remanufacture assumptions; the advanced EHF Satellite:
    optimistic schedule; the LPD-17: flawed lead ship design
    process and knowledge base; and the F-22A: immature, exquisite
    technology.
        Now, the first two of these programs, JSF and FCS, account
    for almost $80 billion in cost overruns, with average unit
    costs that have already increased by roughly 40 percent each
    over original program estimates, and are likely to rise
    further. According to GAO, both programs were initiated with
    insufficiently mature technologies and overly optimistic
    assumptions about system performance.
        With regard to the JSF, GAO reports that initial estimates
    assumed that commonality between the three variants of the
    aircraft could cut development costs by about 40 percent;
    however, this level of commonality has proven impossible to
    achieve. Twelve years after the program started, three of the
    JSF's eight critical technologies are still not mature, its
    production processes are not mature, and its designs are still
    not fully proven and tested.
        With regard to FCS, GAO reports that the estimated lines of
    code needed to support FCS's software and development are
    almost three times the original assumptions, leading to an
    increase in software development costs that now approaches $8
    billion. Eight years after the program started, only 3 of the
    FCS's 44 critical technologies are fully mature. GAO tells us
    that the Army has not advanced the maturity of 11 critical
    technologies since 2003, and that 2 other technologies, which
    are central to the Army's plans, are now rated less mature than
    when the program began.
        This is the price that we have paid for our failure to
    complete needed systems engineering tasks, perform appropriate
    developmental testing, and build prototypes. Particularly at
    this time, when the Federal budget is under immense strain as a
    result of the economic crisis, we cannot continue this kind of
    waste and inefficiency.
        That is why Senator McCain and I have introduced the Weapon
    Systems Acquisition Reform Act of 2009. This bill is designed
    to help put MDAPs on a sound footing from the outset by
    addressing program shortcomings in the early phases of the
    acquisition process.
        In particular, our bill would address unreasonable
    performance requirements by requiring DOD to rebuild its
    systems engineering capability, reestablish the position of
    director of developmental testing, and use the Joint
    Requirements Oversight Council (JROC) to make early tradeoffs
    between cost schedule and performance requirements.
        Our bill will address unreasonable cost and schedule
    estimates by establishing a new director of independent cost
    assessment to ensure that cost estimates for MDAPs are fair,
    reliable, and unbiased.
        Our bill will reduce the use of immature technologies by
    requiring DOD to make greater use of prototypes, including
    competitive prototypes, and requiring the Director of Defense
    Research and Engineering to periodically review and provide
    independent assessments of the maturity of critical
    technologies on major weapon systems.
        Our bill, finally, addresses costly changes in the middle
    of programs by ensuring, through preliminary design review,
    that requirements are well understood before a program receives
    Milestone B approval, by providing an incentive for contractors
    to improve performance in ongoing programs by developing
    mechanisms to maintain competitive pressure through the program
    cycle, and by tightening the so-called Nunn-McCurdy
    requirements for underperforming programs by providing for the
    termination of any program that cannot be justified after a
    complete reexamination and revalidation.
        Today, we will hear from two distinguished former Under
    Secretaries of Defense for Acquisition, Paul Kaminski and
    Jacques Gansler. We will also hear from Pete Adolph, a former
    DOD Director of Developmental Testing, and Mike Sullivan, the
    GAO Director of Acquisition and Sourcing Management. Each of
    our witnesses has great experience in the area of weapon
    systems acquisition; and, in the course of the last year, each
    has completed a major report recommending significant
    improvements and reforms. We all look forward to their
    testimony on these issues.
        I now call on Senator McCain.

                    STATEMENT OF SENATOR JOHN McCAIN

        Senator McCain. Thank you, Mr. Chairman, for today's
    hearing, and more importantly, your leadership on the bill that
    is the subject of today's hearing, the Weapons Acquisition
    Reform Act of 2009. I join you in welcoming our expert
    witnesses today.
        Let me set the overall context of today's discussion, and
    I'll do so very simply. A train wreck is coming. Look at the
    President's 10-year budget and you'll see an overall decrease
    in defense spending. Unless difficult decisions are made and
    serious reform measures are undertaken, our ability to provide
    for our national security will be, over time, fundamentally
    compromised.
        Clearly, the endless cycle of runaway costs, prolonged
    delivery schedules, and poor performance in the acquisition of
    major weapons has, in my view, us mired in a form of unilateral
    disarmament. Since scrutinizing the tanker lease scandal years
    ago, I'm not sure that things are any better. For example, how
    could DOD award a multibillion-dollar contract based on a
    proposal it later found was fundamentally unexecutable? That's
    exactly what happened on the Navy's VH-71 program, the program
    to replace the President's own helicopters.
        Just over the last few years, the VH-71 program has
    doubled, with an additional cost of $6 billion for 28 aircraft
    that will likely cost taxpayers well over $400 million each.
    How could DOD laden a multibillion-dollar shipbuilding program
    with so many requirements that the program more than doubled in
    cost, with DOD basically asleep at the switch? That happened
    with the Navy's LCS program. At times, the program saw change
    orders averaging 75 per week.
        How could a multibillion-dollar program for next-generation
    fighter jets produce planes that are operating below
    satisfactory readiness rates and could end up being too
    expensive to operate? That happened with the Air Force's F-22
    Raptor program.
        How could DOD spend billions for the Army's biggest
    transformational program, valued at almost $200 billion, only
    for it to be, in many respects, closer to the beginning of
    development than it is to the end? That's the FCS program. At
    this point, it's not been clear when, or even if, the
    information network at the heart of the FCS concept can be
    built.
        On our military satellite program, how could a design flaw
    recently emerge that will take at least 1 year, and up to $1
    billion to fix? That's the Air Force's Space-Based Infrared
    System (SBIRS), high satellite program. More cost and schedule
    increases are likely there.
        But, to understand the depth and breadth of our acquisition
    problems, one needs to go no further than to look at the status
    of particular programs. Across all the Services, the top 75
    programs have unfunded cost overruns of at least $295 billion.
    From 2000, the number of MDAPs has increased from 75 to 95, and
    the cost of those programs has doubled from $790 billion to
    $1.6 trillion, leaving unfunded acquisition commitments equal
    to more than 10 years' worth of major weapons procurement
    funding.
        In other words, in the current fiscal environment we find
    ourselves, the DOD acquisition plan is unaffordable. In my
    view, meaningful reform is only going to happen if DOD itself
    decides to change, develops an overarching management
    philosophy, sets up clear lines of authority and
    accountability, brings discipline and control over the
    requirement process, shuts the revolving door, and restores the
    corps of qualified and experienced acquisition and contracting
    professionals. That's what this legislation helps to do.
        In this bill, the chairman and I built on previous reform
    initiatives by focusing on costs and risk. The bill reflects
    that a key to managing defense procurement programs effectively
    is starting them right by requiring key program reviews up
    front to catch costly design flaws and technology risks before
    we actually buy them.
        Probably the most aggressive feature of the bill gives DOD
    a big stick, bigger than anything available under current law,
    to wield against the very worst-performing programs. It does so
    by giving DOD additional tools to enforce fair, reliable, and
    unbiased independent cost estimates with the creation of a new
    director. Unlike merely promulgated DOD instructions, which
    apply only to new programs, that provision will capture
    chronically-poor performers that are in the development
    pipeline now.
        Mr. Chairman, I don't want to go all over the features of
    the bill; I want to hear from our witnesses. But, for truly
    meaningful reform to endure, the commitment to reform must
    begin with the fiduciaries of the taxpayers dollars within the
    DOD itself.
        I thank you, Mr. Chairman. I welcome the witnesses.
        [The prepared statement of Senator McCain follows:]

                   Prepared Statement by Senator John McCain

        Thank you, Mr. Chairman, for today's hearing and, more importantly,
    your leadership on the bill that is the subject of today's hearing,
    S.454, ``The Weapons Systems Acquisition Reform Act of 2009.''
        I appreciate the opportunity to serve as an original co-sponsor on
    it with you, and I too welcome our expert witnesses today.
        Before I turn to the bill, let me briefly set the overall context
    of today's discussion, and I'll do so simply.
        A train wreck is coming, and unless hard decisions are made and
    serious reform measures undertaken, our ability to see to our national
    security interests will be over time fundamentally compromised.
        Let me be clear. The endless cycle of runaway costs, prolonged
    delivery schedules and poor performance we have seen in major weapons
    has us, in my view, mired in a form of unilateral disarmament.
        Since we closely scrutinized the tanker lease scandal years ago,
    I'm not sure that things have gotten much better. For example:

             How could the DOD have awarded a multibillion contract
            based on a proposal it later found was ``fundamentally
            unexecutable.'' That's exactly what happened on the Navy's VH-
            71 program--the program to replace the President's own
            helicopters. Just over the last few years, the program has
            increased by about $6 billion--for aircraft that will likely
            cost taxpayers well over $400 million each.
        How could the DOD have loaded up a multibillion shipbuilding
    program with so many requirements that the program doubled in cost (by
    about $400 million) without the DOD really noticing until it was too
    late? That happened--on the Navy's Littoral Combat Ship (LCS) program.
    At times, that program saw change orders averaging 75 per week.
             How could a multibillion dollar program for next
            generation fighter jets have produced planes that may end up
            being too expensive to operate? That too happened--on the F-22
            Raptor program. While being the Nation's most expensive fighter
            aircraft, those jets continue to operate below satisfactory
            reliability rates.
             How could the DOD have spent billions for the Army's
            biggest transformational program, valued at about $200 billion,
            only for it to be (in many respects) closer to the beginning of
            development than it is to the end? That's the Future Combat
            Systems (FCS) program. At this point, it's not even clear when
            (or even if) the information network at the heart of the FCS
            concept can be built.
             On a military satellite program, how could a design
            flaw have recently emerged that will take at least 1 year and
            up to $1 billion to fix? That's the Air Force's Space-Based
            Infrared System High satellite program. More cost and schedule
            increases are likely there.

        We're supposed to have laws in the books that are supposed to
    prevent these sorts of things from happening. Why didn't they work? To
    the person--who is responsible?
        The fact that we're asking those questions (with billions of
    dollars of taxpayer money at stake) and that we have no answers, lays
    out what we're dealing with far better than I can possibly describe
    using facts and figures.
        So, I don't have to mention that the top 75 programs across all of
    the Services have unfunded cost overruns of at least $295 billion. Or,
    that (since fiscal year 2000) the number of major defense acquisition
    programs has increased from 75 to 95. Or, that within that period the
    cost of those programs doubled from $790 billion to $1.6 trillion. Or,
    that this left unfunded acquisition commitments equal to more than 10
    years worth of major weapons procurement funding.
        I don't have to talk about how risky developing most of those
    programs are; or the likelihood that they too will balloon in costs; or
    how much other government-wide priorities will constrain defense
    spending going forward. I don't have to go into all that to make the
    point that the DOD's acquisition plan--as it currently stands--is
    itself likely unaffordable.
        However one looks at it, the honeymoon is over.
        In my view, meaningful reform is only going to happen if DOD itself
    decides to change. DOD has to:

             Develop an overarching management philosophy that
            dictates an overall approach to ensuring the timely delivery of
            major weapons that satisfy the needs of the joint warfighter at
            the most reasonable cost to the taxpayer.
             Set up clear lines of authority and accountability for
            managing procurement programs.
             Bring discipline and control over the requirements
            process and get out of the business of gold-plating programs.
             Shut the revolving door. While a leavening of
            experienced DOD procurement officials working for defense
            contractors (and vice-versa) is healthy, the lack of meaningful
            controls on this revolving door is creating an unhealthy
            tolerance of conflict-of-interest.
             Restore the corps of qualified and experienced
            acquisition and contracting professionals that DOD had in the
            1980s before it gave its functions over to contractor/lead
            systems integrators, thereby letting the fox guard the hen
            house.

        Until and unless administration and DOD leadership do these sorts
    of things--things that set, if you will, a ``command climate'' that's
    conducive to investing the taxpayers' money responsibly--I fear that
    reform efforts may amount to only rearranging the deck chairs on the
    Titanic. That's something that the chairman and I have no interest in
    doing.
        At the end of the day, we in Congress can only give DOD tools that
    it can use to pursue truly lasting solutions that ensure desirable
    cost/scheduling and performance outcomes in our most complicated, most
    expensive weapons systems. That's what this bill helps do.
        In this bill, the chairman and I build on previous reform
    initiatives by focusing on costs and risk. The bill reflects that a key
    to managing defense procurement programs effectively is starting them
    right--by requiring key program review upfront to catch costly design
    flaws and technology risk before we actually buy the system.
        In so doing, we continue our efforts to move DOD closer towards
    fixed price-type procurement contracting by requiring that technology
    and integration risk can be meted out of a program early. So, by the
    time a program heads into procurement, if a contractor isn't ready to
    sign a good fixed price-type contract, the government shouldn't be
    signing a contract with that company to buy that system at all. There's
    still too much risk.
        Probably the most aggressive feature of the bill gives DOD a big
    stick--bigger than anything available under current law--to wield
    against the very worst performing programs. With that provision we
    intend DOD to, if you will, ``enforce'' fair, reliable, and unbiased
    cost estimates verified by a new director of independent cost
    estimates, also created by this bill. Unlike rules that DOD recently
    put in place, that provision doesn't apply to just new programs. It
    will capture chronically poor performers that are in the development
    pipeline now.
        Another important provision requires DOD to consider a broad range
    of cost-effective measures to help maximize competition throughout the
    life of a weapons program. The industry consolidation that occurred
    after the end of the Cold War went too far. Some 50 prime contractors
    merged into only 6. That's far too few to support a competitive base
    for our current and future needs. It's resulted in a serious decline in
    innovation.
        Other provisions in the bill elaborate in the ``starting programs
    right'' theme by:

             Renewing focus on systems engineering early;
             Requiring the completion of preliminary design reviews
            before a program can move into the development phase; and
             Strengthening DOD's developmental testing and
            evaluation capability.

        Other helpful provisions include those that:

             Require DOD budget, requirements, and acquisition
            officials to consult each other and make trade-offs between
            cost, schedule, and performance early in the process; and
             Crack down on the frequent changes to programs, which
            tend to cause many cost increases.

        While this bill is not intended as a panacea to cure all that ails
    the defense procurement process, it is an important next step in
    Congress' continuing efforts to help DOD culturally reform the system.
        As I said a moment ago, for meaningful reform to truly endure, the
    commitment to reform must begin with the fiduciaries of the taxpayers'
    dollars within the Department itself.
        Thank you, Mr. Chairman. I look forward to hearing from our
    witnesses today.

        Chairman Levin. Senator McCain, we thank you.
        Now we'll call on our witnesses. First, we'll call on
    Michael Sullivan from the GAO.
        Would you please proceed?

      STATEMENT OF MICHAEL J. SULLIVAN, DIRECTOR, ACQUISITION AND
         SOURCING MANAGEMENT, GOVERNMENT ACCOUNTABILITY OFFICE

        Mr. Sullivan. Chairman Levin, Senator McCain, members of
    the committee, I'm pleased to be here today to discuss DOD's
    acquisition outcomes and the legislation proposed by this
    committee to improve them. I'll make a brief oral statement and
    ask that my written testimony be placed in the record.
        Chairman Levin. It will be.
        Mr. Sullivan. We've been reporting for years on poor cost
    and schedule outcomes on DOD's major weapon system
    acquisitions. As the chairman noted, most recently we reported
    that 95 programs in DOD's current portfolio have grown in cost
    by $295 billion and are, on average, delivered about 21 months
    late. We believe there are problems at the strategic and at the
    program levels that cause these outcomes.
        At the strategic level, DOD's three systemic processes for
    building its investment strategy are fragmented and broken. The
    requirement-setting process, known as the Joint Capabilities
    and Integration Development System (JCIDS) is stovepiped, it
    does not consider resources, and it approves nearly every
    proposal that it reviews. The funding process accepts programs
    with unrealistic cost estimates, and does not fully fund their
    development costs. These two processes are poorly integrated,
    and this poor communication leads to unhealthy competition,
    where too many programs are chasing too few dollars.
        Finally, at the program level, the acquisition process
    initiates programs with unreliable cost estimates and without
    knowledge from proper systems engineering analysis to
    understand each weapon system's requirements and the resources
    that will be needed to achieve them. These programs move
    forward with too much technology, design, and manufacturing
    risk as a result.
        DOD understands this and has recently revised its policies
    to address some of these problems. Its new acquisition policy,
    for instance, encourages more systems engineering activity
    earlier in programs, competitive prototyping to gain knowledge
    more quickly and to maintain competition, earlier milestone
    reviews, and steering boards to protect programs against the
    desire to add more requirements once they've started.
        Recent decisions by DOD on some programs have been
    encouraging, and some of the newer programs appear to have
    undergone more disciplined reviews.
        For many years, there's been a broad consensus that weapon
    system acquisition problems are serious and their resolution is
    overdue. With the Federal budget under increasing strain from
    the economic crisis facing our Nation, the time to change is
    now.
        In testimony before this committee last month, Secretary of
    Defense Gates identified many of the systemic problems
    associated with acquisitions and indicated that efforts are
    underway to address them.
        We believe that the legislation this committee has proposed
    will help address the toughest problems, and we
    enthusiastically support it. We believe it precisely targets
    key problem areas, provides much-needed oversight, and provides
    increased authority and independence to the critical functions
    of cost estimating and development testing by requiring them to
    report to the Secretary and to Congress.
        Among other things, its provision to require a full
    inventory of DOD's current systems engineering skills is an
    excellent beginning to rebuilding that sorely-needed
    capability. Its addition of a termination criterion for Nunn-
    McCurdy breaches sends a strong signal to programs to have
    realistic cost estimates when they start.
        It is important to state that there is also a need for
    changes to the overall acquisition culture and the incentives
    it provides. The culture should begin to change by resisting
    the urge to achieve the revolutionary, but unachievable,
    capability in one step by allowing technologies to mature in
    the technology base rather than forcing them on the acquisition
    programs too early, by ensuring that urgent requirements are
    well-defined and quickly achievable, and by instituting
    shorter, more predictable development cycles.
        These changes will not be easy to make. Tough decisions
    must be made about DOD's overall portfolio of weapon programs
    and about specific programs; and stakeholders from DOD, the
    military services industry, and Congress will have to play a
    constructive role in this decisionmaking. We see the proposed
    legislation discussed here today as a very healthy step in that
    direction.
        Mr. Chairman, this concludes my statement. I'll wait to
    answer questions.
        [The prepared statement of Mr. Sullivan follows:]

                   Prepared Statement by Michael J. Sullivan

        Mr. Chairman and members of the committee: I am pleased to be here
    today to discuss the Department of Defense's (DOD) management of its
    major weapon system acquisitions--an area that has been on the
    Government Accountability Office's (GAO) high-risk list since 1990.
    Prior to and since that time, Congress and DOD have continually
    explored ways to improve acquisition outcomes without significant
    results. While the technological sophistication of DOD weapon systems
    is unparalleled, major weapon programs continue to cost more, take
    longer to complete, and deliver fewer quantities and capabilities than
    originally planned. Last year we reported that the cumulative cost
    growth in DOD's portfolio of 95 major defense acquisition programs was
    $295 billion from first estimates and the average delay in delivering
    promised capabilities to the warfighter was 21 months. Clearly, some
    problems are to be expected in developing weapon systems given the
    technical risks and complexities involved. However, all too often we
    have found that cost and schedule problems are rooted in poor planning,
    execution, and oversight.
        Investment in weapon systems is now at its highest level in two
    decades, and DOD plans to spend more than $357 billion over the next 5
    years on major defense acquisition programs. Effective management of
    this substantial investment is critical as competition for funding has
    increased dramatically within the department and across the government.
    DOD faces a number of fiscal pressures: the ongoing military campaigns
    in Afghanistan and Iraq, rising personnel costs, and the rebuilding and
    modernization of the force. In addition, the economic and fiscal crises
    now facing the Nation have required unprecedented spending by the
    Federal Government, and budget deficits are projected to remain high
    for many years to come. At a time when the Federal budget is strained
    by spending needs for a growing number of national priorities, it is
    imperative that DOD get the best value for every dollar it invests in
    weapon system programs. Every dollar wasted during the development and
    acquisition of weapon systems is money not available for other
    priorities within DOD and elsewhere in the government.
        Today, I will discuss: (1) the systemic problems that have
    contributed to cost, schedule, and performance problems in DOD's
    acquisition of major weapon systems; (2) recent actions the department
    has taken to address these problems; (3) our observations on the
    committee's proposed acquisition reform legislation; and (4) steps that
    Congress and the department need to take to improve the future
    performance of acquisition programs. The statement includes findings
    from our July 2008 report on a knowledge-based funding approach and
    February 2009 report on potential changes to DOD's acquisition
    management framework.\1\ It also draws from our extensive body of work
    on DOD's acquisition of weapon systems. A list of our key products is
    provided at the end of this statement. This work was conducted in
    accordance with generally accepted government auditing standards. Those
    standards require that we plan and perform the audit to obtain
    sufficient, appropriate evidence to provide a reasonable basis for our
    findings and conclusions based on our audit objectives. We believe that
    the evidence obtained provides a reasonable basis for our findings and
    conclusions based on our audit objectives.
    ---------------------------------------------------------------------------
        \1\ GAO, Defense Acquisitions: A Knowledge-Based Funding Approach
    Could Improve Major Weapon System Program Outcomes, GAO-08-619
    (Washington, DC: July 2, 2008), and Defense Acquisitions: Perspectives
    on Potential Changes to DOD's Acquisition Management Framework, GAO-09-
    295R (Washington, DC: Feb. 27, 2009).
    ---------------------------------------------------------------------------
     FRAGMENTED INVESTMENT DECISIONMAKING, UNEXECUTABLE PROGRAMS, AND LACK
              OF ACCOUNTABILITY UNDERLIE POOR ACQUISITION OUTCOMES

        Over the past several years our work has highlighted a number of
    underlying systemic causes for cost growth and schedule delays at both
    the strategic and program levels. At the strategic level, DOD's
    processes for identifying warfighter needs, allocating resources, and
    developing and procuring weapon systems--which together define DOD's
    overall weapon system investment strategy--are fragmented. As a result,
    DOD fails to effectively address joint warfighting needs and commits to
    more programs than it has resources for, thus creating unhealthy
    competition for funding. At the program level, a military Service
    typically establishes and DOD approves a business case containing
    requirements that are not fully understood and cost and schedule
    estimates that are based on overly optimistic assumptions rather than
    on sufficient knowledge. Once a program begins, it too often moves
    forward with inadequate technology, design, testing, and manufacturing
    knowledge, making it impossible to successfully execute the program
    within established cost, schedule, and performance targets.
    Furthermore, DOD officials are rarely held accountable for poor
    decisions or poor program outcomes.

    DOD Lacks an Integrated Approach to Balance Weapon System Investments
        At the strategic level, DOD largely continues to define warfighting
    needs and make investment decisions on a Service-by-Service and
    individual platform basis, using fragmented decisionmaking processes.
    This approach makes it difficult for the department to achieve a
    balanced mix of weapon systems that are affordable and feasible and
    that provide the best military value to the joint warfighter. In
    contrast, we have found that successful commercial enterprises use an
    integrated portfolio management approach to focus early investment
    decisions on products collectively at the enterprise level and ensure
    that there is a sound basis to justify the commitment of resources.\2\
    By following a disciplined, integrated process--during which the
    relative pros and cons of competing product proposals are assessed
    based on strategic objectives, customer needs, and available resources,
    and where tough decisions about which investments to pursue and not to
    pursue are made--companies minimize duplication between business units,
    move away from organizational stovepipes, and effectively support each
    new development program they commit to. To be effective, integrated
    portfolio management must have strong, committed leadership; empowered
    portfolio managers; and accountability at all levels of the
    organization.
    ---------------------------------------------------------------------------
        \2\ GAO, Best Practices: An Integrated Portfolio Management
    Approach to Weapon System Investments Could Improve DOD's Acquisition
    Outcomes, GAO-07-388 (Washington, DC: Mar. 30, 2007).
    ---------------------------------------------------------------------------
        DOD determines its capability needs through the Joint Capabilities
    and Integration Development System (JCIDS). While JCIDS provides a
    framework for reviewing and validating needs, it does not adequately
    prioritize those needs from a joint, departmentwide perspective; lacks
    the agility to meet changing warfighter demands; and validates almost
    all of the capability proposals that are submitted. We recently
    reviewed JCIDS documentation related to new capability proposals and
    found that most--almost 70 percent--were sponsored by the military
    Services with little involvement from the joint community, including
    the combatant commands, which are responsible for planning and carrying
    out military operations.\3\ Because DOD also lacks an analytic approach
    to determining the relative importance of the capabilities needed for
    joint warfighting, all proposals appear to be treated as equal
    priorities within the JCIDS process. By continuing to rely on
    capability needs defined primarily by the Services, DOD may be losing
    opportunities for improving joint warfighting capabilities and reducing
    the duplication of capabilities in some areas. The JCIDS process has
    also proven to be lengthy and cumbersome--taking on average up to 10
    months to validate a need--thus undermining the department's efforts to
    effectively respond to the needs of the warfighter, especially those
    needs that are near term. Furthermore, the vast majority of capability
    proposals that enter the JCIDS process are validated or approved
    without accounting for the resources or technologies that will be
    needed to acquire the desired capabilities. Ultimately, the process
    produces more demand for new weapon system programs than available
    resources can support.
    ---------------------------------------------------------------------------
        \3\ GAO, Defense Acquisitions: DOD's Requirements Determination
    Process Has Not Been Effective in Prioritizing Joint Capabilities, GAO-
    08-1060 (Washington, DC: Sept. 25, 2008).
    ---------------------------------------------------------------------------
        The funding of proposed programs takes place through a separate
    process, the department's Planning, Programming, Budgeting, and
    Execution (PPBE) system, which is not synchronized with JCIDS. While
    JCIDS is a continuous, need-driven process that unfolds in response to
    capability proposals as they are submitted by sponsors, PPBE is a
    calendar-driven process comprising phases occurring over a 2-year
    cycle, which can lead to resource decisions for proposed programs that
    may occur several years later. In addition, because PPBE is structured
    by military Service and defense programs and not by the joint
    capability areas being used in JCIDS, it is difficult to link resources
    to capabilities. The PPBE process also largely allocates resources
    based on historical trends rather than on a strategic basis. Service
    shares of the overall budget have remained relatively static for
    decades, even though DOD's strategic environment and warfighting needs
    have changed dramatically in recent years. Because DOD's programming
    and budgeting reviews occur at the back end of the PPBE process--after
    the Services have developed their budgets--it is difficult and
    disruptive to make changes, such as terminating programs to pay for
    new, higher-priority programs.
        We recently reviewed the impact of the PPBE process on major
    defense acquisition programs and found that the process does not
    produce an accurate picture of the department's resource needs for
    weapon system programs, in large part because it allows too many
    programs to go forward with unreliable cost estimates and without a
    commitment to fully fund them.\4\ The cost of many of the programs we
    reviewed exceeded the funding levels planned for and reflected in the
    Future Years Defense Program (FYDP)--the department's long-term
    investment strategy (see fig. 1). DOD's failure to balance its needs
    with available resources promotes an unhealthy competition for funding
    that encourages sponsors of weapon system programs to pursue overly
    ambitious capabilities and underestimate costs to appear affordable.
    Rather than limit the number and size of programs or adjust
    requirements, DOD opts to push the real costs of programs to the
    future. With too many programs underway for the available resources and
    high cost growth occurring in many programs, the department must make
    up for funding shortfalls by shifting funds from one program to pay for
    another, reducing system capabilities, cutting procurement quantities,
    or in rare cases terminating programs. Such actions not only create
    instability in DOD's weapon system portfolio, they further obscure the
    true future costs of current commitments, making it difficult to make
    informed investment decisions.
    ---------------------------------------------------------------------------
        \4\ GAO-08-619.
    ---------------------------------------------------------------------------

    Initiating Programs with Inadequate Knowledge of Requirements and
            Resources Often Results in Poor Outcomes
        At the program level, the key cause of poor outcomes is the
    approval of programs with business cases that contain inadequate
    knowledge about requirements and the resources--funding, time,
    technologies, and people--needed to execute them. Our work in best
    practices has found that an executable business case for a program
    demonstrated evidence that: (1) the identified needs are real and
    necessary and that they can best be met with the chosen concept; and
    (2) the chosen concept can be developed and produced within existing
    resources. Over the past several years, we have found no evidence of
    the widespread adoption of such an approach for major acquisition
    programs in the department. Our annual assessments of major weapon
    systems have consistently found that the vast majority of programs
    began system development without mature technologies and moved into
    system demonstration without design stability.
        The chief reason for these problems is the encouragement within the
    acquisition environment of overly ambitious and lengthy product
    developments--sometimes referred to as revolutionary or big bang
    acquisition programs--that embody too many technical unknowns and not
    enough knowledge about the performance and production risks they
    entail. The knowledge gaps are largely the result of a lack of early
    and disciplined systems engineering analysis of a weapon system's
    requirements prior to beginning system development. Systems engineering
    translates customer needs into specific product requirements for which
    requisite technological, software, engineering, and production
    capabilities can be identified through requirements analysis, design,
    and testing. Early systems engineering provides the knowledge a product
    developer needs to identify and resolve performance and resource gaps
    before product development begins by either reducing requirements,
    deferring them to the future, or increasing the estimated cost for the
    weapon system's development. Because the government often does not
    perform the proper upfront requirements analysis to determine whether
    the program will meet its needs, significant contract cost increases
    can and do occur as the scope of the requirements changes or becomes
    better understood by the government and contractor. Not only does DOD
    not conduct disciplined systems engineering prior to the beginning of
    system development, it has allowed new requirements to be added well
    into the acquisition cycle. We have reported on the negative impact
    that poor systems engineering practices have had  on  several
    programs,  such  as  the  Global  Hawk  Unmanned  Aircraft  System,  F-
    22A, Expeditionary Fighting Vehicle, and Joint Air-to-Surface Standoff
    Missile.\5\
    ---------------------------------------------------------------------------
        \5\ GAO, Best Practices: Increased Focus on Requirements and
    Oversight Needed to Improve DOD's Acquisition Environment and Weapon
    System Quality, GAO-08-294 (Washington, DC: Feb. 1, 2008).
    ---------------------------------------------------------------------------
        With high levels of uncertainty about requirements, technologies,
    and design, program cost estimates and related funding needs are often
    understated, effectively setting programs up for cost and schedule
    growth. We recently assessed the service and independent cost estimates
    for 20 major weapon system programs and found that while the
    independent estimates were somewhat higher, both estimates were too low
    in most cases.\6\ In some of the programs we reviewed, cost estimates
    have been off by billions of dollars. For example, the initial service
    estimate for the development of the Marines' Expeditionary Fighting
    Vehicle was about $1.1 billion. The department's Cost Analysis and
    Improvement Group (CAIG) estimated the development cost of the program
    to be $1.4 billion, but development costs for the program are now
    expected to be close to $3.6 billion. In the case of the Future Combat
    System (FCS), the Army's initial estimate for the development cost was
    about $20 billion, while CAIG's estimate was $27 billion. The
    department began the program using the program office's estimate of $20
    billion, but development costs for the FCS are now estimated to be $28
    billion and the program is still dealing with significant technical
    risk. Estimates this far off the mark do not provide the necessary
    foundation for sufficient funding commitments and realistic long-term
    planning.
    ---------------------------------------------------------------------------
        \6\ GAO-08-619.
    ---------------------------------------------------------------------------
        The programs we reviewed frequently lacked the knowledge needed to
    develop realistic cost estimates. For example, program Cost Analysis
    Requirements Description documents--used to build the program cost
    estimate--often lack sufficient detail about planned program content
    for developing sound cost estimates. Without this knowledge, cost
    estimators must rely heavily on parametric analysis and assumptions
    about system requirements, technologies, design maturity, and the time
    and funding needed. A cost estimate is then usually presented to
    decisionmakers as a single, or point, estimate that is expected to
    represent the most likely cost of the program but provides no
    information about the range of risk and uncertainty or level of
    confidence associated with the estimate.

    Lack of Accountability for Making Weapon System Decisions Hinders
            Achieving Successful Outcomes
        DOD's requirements, resource allocation, and acquisition processes
    are led by different organizations, thus making it difficult to hold
    any one person or organization accountable for saying no to a proposed
    program or for ensuring that the department's portfolio of programs is
    balanced. DOD's 2006 Defense Acquisition Performance Assessment study
    observed that these processes are not connected organizationally at any
    level below the Deputy Secretary of Defense and concluded that this
    weak structure induces instability and inhibits accountability.
    Furthermore, a former Under Secretary of Defense for Acquisitions,
    Technology, and Logistics has stated that weapon system investment
    decisions are a shared responsibility in the department and, therefore,
    no one individual is accountable for these decisions. Frequent turnover
    in leadership positions in the department exacerbates the problem. The
    average tenure, for example, of the Under Secretary of Defense for
    Acquisition, Technology, and Logistics over the past 22 years has been
    only about 20 months.\7\
    ---------------------------------------------------------------------------
        \7\ The position of Under Secretary of Defense for Acquisition was
    established in 1986 and the title was subsequently changed to the Under
    Secretary of Defense for Acquisition, Technology, and Logistics in
    2000. Since 1986, there have been 11 Under Secretaries.
    ---------------------------------------------------------------------------
        When DOD's strategic processes fail to balance needs with resources
    and allow unsound, unexecutable programs to move forward, program
    managers cannot be held accountable when the programs they are handed
    already have a low probability of success. Program managers are also
    not empowered to make go or no-go decisions, have little control over
    funding, cannot veto new requirements, and have little authority over
    staffing. At the same time, program managers frequently change during a
    program's development, making it difficult to hold them accountable for
    the business cases that they are entrusted to manage and deliver.
        The government's lack of control over and accountability for
    decisionmaking is further complicated by DOD's growing reliance on
    technical, business, and procurement expertise supplied by contractors.
    This reliance may reach the point where the foundation upon which
    decisions are based may be largely crafted by individuals who are not
    employed by the government, who are not bound by the same rules
    governing their conduct, and who are not required to disclose any
    financial or other personal interests they may have that conflict with
    the responsibilities they have performing contract tasks for DOD. For
    example, while the total planned commitments to major acquisition
    programs have doubled over recent years, the size of the department's
    systems engineering workforce has remained relatively stable, leading
    program offices to rely more on contractors for systems engineering
    support. Further, in systems development, DOD typically uses cost-
    reimbursement contracts in which it generally pays the reasonable,
    allocable, and allowable costs incurred for the contractor's best
    efforts, to the extent provided by the contract. The use of these
    contracts may contribute to the perpetuation of an acquisition
    environment that lacks incentives for contractors to follow best
    practices and keep costs and schedules in check.

     RECENT DOD POLICY CHANGES COULD IMPROVE FUTURE PERFORMANCE OF WEAPON
                                SYSTEM PROGRAMS

        The department understands many of the problems that affect
    acquisition programs and has recently taken steps to remedy them. It
    has revised its acquisition policy and introduced several initiatives
    based in part on direction from Congress and recommendations from GAO
    that could provide a foundation for establishing sound, knowledge-based
    business cases for individual acquisition programs. However, to improve
    outcomes, DOD must ensure that its policy changes are consistently
    implemented and reflected in decisions on individual programs--not only
    new program starts but also ongoing programs as well. In the past,
    inconsistent implementation of existing policy has hindered DOD's
    efforts to execute acquisition programs effectively. Moreover, while
    policy improvements are necessary, they may be insufficient unless the
    broader strategic issues associated with the department's fragmented
    approach to managing its portfolio of weapon system investments are
    also addressed.
        In December 2008, DOD revised its policy governing major defense
    acquisition programs in ways intended to provide key department leaders
    with the knowledge needed to make informed decisions before a program
    starts and to maintain disciplined development once it begins. The
    revised policy recommends the completion of key systems engineering
    activities before the start of development, includes a requirement for
    early prototyping, establishes review boards to identify and mitigate
    technical risks and evaluate the impact of potential requirements
    changes on ongoing programs, and incorporates program manager
    agreements to increase leadership stability and management
    accountability. The policy also establishes early milestone reviews for
    programs going through the pre--systems acquisition phase. In the past,
    DOD's acquisition policy may have encouraged programs to rush into
    systems development without sufficient knowledge, in part, because no
    formal milestone reviews were required before system development. If
    implemented, these policy changes could help programs replace risk with
    knowledge, thereby increasing the chances of developing weapon systems
    within cost and schedule targets while meeting user needs. Some aspects
    of the policy were first pilot-tested on selected programs, such as the
    Joint Light Tactical Vehicle program, and indications are that these
    programs are in the process of acquiring the requisite knowledge before
    the start of systems development. Some key elements of the department's
    new acquisition policy include:

             a new materiel development decision as a starting
            point for all programs regardless of where they are intended to
            enter the acquisition process,
             a more robust Analysis of Alternatives (AOA) to assess
            potential materiel solutions that address a capability need
            validated through JCIDS,
             a cost estimate for the proposed solution identified
            by the AOA,
             early program support reviews by systems engineering
            teams,
             competitive prototyping of the proposed system or key
            system elements as part of the technology development phase,
             certifications for entry into the technology
            development and system development phases (as required by
            congressional legislation),
             preliminary design review (PDR) that may be conducted
            before the start of systems development, and
             configuration steering boards to review all
            requirements and technical changes that have potential to
            affect cost and schedule.

        As part of its strategy for enhancing the roles of program managers
    in major weapon system acquisitions, the department has established a
    policy that requires formal agreements among program managers, their
    acquisition executives, and the user community setting forth common
    program goals. These agreements are intended to be binding and to
    detail the progress the program is expected to make during the year and
    the resources the program will be provided to reach these goals. DOD
    also requires program managers to sign tenure agreements so that their
    tenure will correspond to the next major milestone review closest to 4
    years. The department acknowledges that any actions taken to improve
    accountability must be based on a foundation whereby program managers
    can launch and manage programs toward successful performance, rather
    than focusing on maintaining support and funding for individual
    programs. DOD acquisition leaders have also stated that any
    improvements to program managers' performance depend on the
    department's ability to promote requirements and resource stability
    over weapon system investments.
        Over the past few years, DOD has also been testing portfolio
    management approaches in selected capability areas--command and
    control, net-centric operations, battlespace awareness, and logistics--
    to facilitate more strategic choices for resource allocation across
    programs. The department recently formalized the concept of capability
    portfolio management, issuing a directive in 2008 that established
    policy and assigned responsibilities for portfolio management. The
    directive established nine joint capability area portfolios, each to be
    managed by civilian and military co-leads. While the portfolios have no
    independent decisionmaking authority over requirements determination
    and resource allocation, according to some DOD officials, they provided
    key input and recommendations in this year's budget process. However,
    without portfolios in which managers have authority and control over
    resources, the department is at risk of continuing to develop and
    acquire systems in a stovepiped manner and of not knowing if its
    systems are being developed within available resources.

            OBSERVATIONS ON PROPOSED ACQUISITION REFORM LEGISLATION

        Overall, we believe that the legislative initiatives being proposed
    by the committee have the potential, if implemented, to lead to
    significant improvements in DOD's management of weapon system programs.
    Several of the initiatives--including the increased emphasis on systems
    engineering and developmental testing, the requirement for earlier
    PDRs, and the strengthening of independent cost estimates and
    technology readiness assessments--could instill more discipline into
    the front end of the acquisition process when it is critical for
    programs to gain knowledge. Establishing a termination criterion for
    Nunn-McCurdy cost breaches could help prevent the acceptance of
    unrealistic cost estimates as a foundation for starting programs.\8\
    Having greater involvement by the combatant commands in determining
    requirements and requiring greater consultation between the
    requirements, budget, and acquisition processes could help improve the
    department's efforts to balance its portfolio of weapon system
    programs. In addition, several of the proposals as currently drafted
    will codify what DOD policy already calls for, but are not being
    implemented consistently in weapon programs.
    ---------------------------------------------------------------------------
        \8\ 10 U.S.C. Sec. 2433(a)(5) requires the Secretary of Defense to
    report to Congress when a program's acquisition unit cost increases by
    at least 25 percent over the current baseline estimate or increases by
    at least 50 percent over the original baseline estimate.
    ---------------------------------------------------------------------------
    Section 101: Systems Engineering Capabilities
        Requires DOD to: (1) assess the extent to which the department has
    in place the systems engineering capabilities needed to ensure that key
    acquisition decisions are supported by a rigorous systems analysis and
    systems engineering process; and (2) establish organizations and
    develop skilled employees to fill any gaps in such capabilities.
        The lack of disciplined systems engineering analysis conducted
    prior to starting system development has been a key factor contributing
    to poor acquisition outcomes. Systems engineering activities--
    requirements analysis, design, and testing--are needed to ensure that a
    weapon system program's requirements are achievable and designable
    given available resources, such as technologies. In recent years, DOD
    has taken steps to improve its systems engineering capabilities by
    establishing a Systems and Software Engineering Center of Excellence
    and publishing guidance to assist the acquisition workforce in the
    development of systems engineering plans, education, and training.
    However, as the National Research Council recently reported, DOD's
    systems engineering capabilities have declined over time and shifted
    increasingly to outside contractors.\9\ A comprehensive assessment to
    determine what systems engineering capabilities are in place and what
    capabilities are needed, as recommended in the proposed legislation, is
    a critical first step in enhancing the function of systems engineering
    in DOD acquisitions. At the same time, it will be important for DOD to
    implement steps to ensure systems engineering is applied in the right
    way and at the right time.
    ---------------------------------------------------------------------------
        \9\ National Research Council, Pre-Milestone A and Early-Phase
    Systems Engineering: A Retrospective Review and Benefits for Future Air
    Force Systems Acquisition (Washington, DC: February 2008).
    ---------------------------------------------------------------------------
    Section 102: Developmental Testing
        Requires DOD to reestablish the position of Director of
    Developmental Test and Evaluation and requires the Services to assess
    and address any shortcomings in their developmental testing
    organizations and personnel.
        Robust developmental testing efforts are an integral part of the
    systems development process. They help to identify, evaluate, and
    reduce technical risks, and indicate whether the design solution is on
    track to satisfy the desired capabilities. As the Defense Science Board
    reported in 2008, developmental testing in weapon system programs needs
    to be improved.\10\ We believe that developmental testing would be
    strengthened by a formal elevation of its role in the acquisition
    process and the reestablishment of a Director of Developmental Test and
    Evaluation position. Furthermore, requiring the Director to prepare an
    annual report for Congress summarizing DOD's developmental test and
    evaluation activities would provide more accountability. We also agree
    that the military Services should be required to assess their
    respective developmental testing entities and address any shortcomings.
    This action would help ensure that the Services have the knowledge and
    capacity for effective developmental test efforts.
    ---------------------------------------------------------------------------
        \10\ Defense Science Board, Report on Developmental Test &
    Evaluation (Washington, DC: May 2008).
    ---------------------------------------------------------------------------
    Section 103: Technical Maturity Assessments
        Makes it the responsibility of the Director of Defense Research and
    Engineering (DDR&E) to periodically review and assess the technological
    maturity of critical technologies used in major defense acquisition
    programs.
        Ensuring that programs have mature technology before starting
    systems development is critical to avoiding cost and schedule problems,
    yet for many years we have reported that a majority of programs go
    forward with immature technologies and experience significant cost
    growth. Legislation enacted by Congress in 2006, requiring DOD to
    certify that the technology in a program has been demonstrated in a
    relevant environment before it receives approval to start system
    development, has begun to help address this problem. Since the
    legislation was enacted, DOD has asked the DDR&E to conduct independent
    reviews of technology readiness assessments for system development
    milestone decisions. Although DDR&E reviews are advisory in nature, we
    have seen reviews that have pushed programs to do more to demonstrate
    technology maturity. The improvements that this proposed legislation,
    as currently written, is intended to bring about may already be
    occurring in DOD. Congress, however, may wish to consider requiring the
    DDR&E to conduct technology readiness reviews not just periodically,
    but for all major defense acquisition programs, and whether or not
    DDR&E has the capacity and resources to effectively conduct technology
    assessments.

    Section 104: Independent Cost Assessment
        Establish a Director of Independent Cost Assessment to ensure that
    cost estimates for major defense acquisition programs are fair,
    reliable, and unbiased.
        Within DOD, the CAIG is the organization responsible for conducting
    independent costs estimates for major defense acquisition programs. The
    CAIG reports to the department's Director of Program Analysis and
    Evaluation, but its principal customer is the Under Secretary of
    Defense for Acquisition, Technology, and Logistics. We believe that
    establishing an independent assessment office that reports directly to
    the Secretary or Deputy Secretary of Defense and to Congress--similar
    to the Office of the Director of Operation Test and Evaluation--would
    more fully integrate cost estimating with the acquisition management
    framework and provide an increased level of accountability. We see no
    reason why CAIG should not form the basis of the proposed organization.
    Congress may also wish to consider appointing the Director for a time-
    certain term and making the Director responsible for prescribing cost-
    estimating policy and guidance and for preparing an annual report
    summarizing cost estimates for major acquisition programs. Ultimately,
    however, improved cost estimating will only occur if there is a better
    foundation for planning and acquiring weapon system programs--one that
    promotes well-defined requirements, is knowledge-based and informed by
    disciplined systems engineering, requires mature technology, and
    adheres to shorter development cycle times.

    Section 105: Role of Combatant Commanders
        Requires the Joint Requirements Oversight Council (JROC) to seek
    and consider input from the commanders of the combatant commands in
    identifying joint military requirements.
        Requirements determination in DOD, particularly for major weapon
    system programs, continues to be driven largely by the military
    Services. Studies by the Defense Science Board, Center for Strategic
    and International Studies, and others have revealed that although the
    combatant commands--which are responsible for planning and executing
    military missions--are the principal joint warfighting customer in DOD,
    they have played a limited role in determining requirements. Currently,
    the JROC is doing more to seek out and consider input from the
    combatant commands through regular trips and meetings to discuss
    capability needs and resourcing issues. However, many of the combatant
    commands do not believe that their needs, which are reflected through
    the Integrated Priority List process, are sufficiently addressed
    through the department's JCIDS process. For the combatant commands to
    meet this proposed legislative mandate and have more influence in
    establishing requirements, DOD should consider providing the combatant
    commands with additional resources to establish robust analytical
    capabilities for identifying and assessing their capability needs.
    Ultimately, the department must better prioritize and balance the needs
    of the military Services, combatant commands, and other defense
    components, and be more agile in responding to near-term capability
    needs.

    Section 201: Trade-offs of Cost, Schedule, and Performance
        Requires consultation between the budget, requirements, and
    acquisition processes to ensure the consideration of trade-offs between
    cost, schedule, and performance early in the process of developing
    major weapon systems.
        As currently structured, DOD's budget, requirements, and
    acquisition processes do not operate in an integrated manner. The
    function and timing of the processes are not sufficiently synchronized,
    and the decisionmakers for each process are motivated by different
    incentives. These weaknesses have contributed to the development of a
    portfolio with more programs than available resources can support and
    programs that launch into system development without executable
    business cases. We have recommended that the department establish an
    enterprisewide portfolio management approach to weapon system
    investment decisions that integrates the determination of joint
    warfighting needs with the allocation of resources, and cuts across the
    Services by functional or capability area.\11\ To ensure the success of
    such an approach, we believe that the department should establish a
    single point of accountability with the authority, responsibility, and
    tools to implement portfolio management effectively.
    ---------------------------------------------------------------------------
        \11\ GAO-07-388.
    ---------------------------------------------------------------------------
    Section 202: Preliminary Design Review
        Require the completion of a PDR and a formal post-PDR assessment
    before a major defense acquisition program receives approval to start
    system development.
        We have found that a key deliverable in a knowledge-based
    acquisition process is the preliminary design of the proposed solution
    based on a robust systems engineering assessment prior to making a
    large financial commitment to system development. Early systems
    engineering provides the knowledge needed by a developer to identify
    and resolve gaps, such as overly optimistic requirements that cannot be
    met with current resources, before product development begins.
    Consequently, DOD would have more confidence that a particular system
    could successfully proceed into a detailed system development phase and
    meet stated performance requirements within cost, schedule, risk, and
    other relevant constraints. The recently revised DOD acquisition policy
    places an increased emphasis on programs planning for PDR prior to the
    start of system development but does not go as far as making it a
    requirement to do so. We support any effort to add controls to the
    acquisition process to ensure that timely and robust systems
    engineering is conducted before major investment decisions, such as the
    approval to start system development, are made.

    Section 203: Life-Cycle Competition
        Require DOD to adopt measures recommended by the 2008 Defense
    Science Board Task Force on Defense Industrial Structure for
    Transformation--such as competitive prototyping, dual sourcing, open
    architectures, periodic competitions for subsystem upgrades, and
    licensing of additional suppliers--to maximize competition throughout
    the life of a program.
        We have reported in the past on the problem of diminishing
    competition and the potential benefits of more competition.\12\ In
    discussing the environment that leads to poor acquisition outcomes, we
    have noted that changes within the defense supplier base have added
    pressure to this environment. We noted that in 2006, a DOD-commissioned
    study found that the number of fully competent prime contractors
    competing for programs had fallen from more than 20 in 1985 to only 6,
    and that this has limited DOD's ability to maximize competition in
    order to reduce costs and encourage innovation. However, avenues exist
    for reducing costs through competition. For example, we reported that
    although continuing an alternate engine program for the Joint Strike
    Fighter would cost significantly more in development costs than a sole-
    source program, it could, in the long run, reduce overall life-cycle
    costs and bring other benefits.
    ---------------------------------------------------------------------------
        \12\ GAO, Defense Acquisitions: Better Weapon Program Outcomes
    Require Discipline, Accountability, and Fundamental Changes in the
    Acquisition Environment, GAO-08-782T (Washington, DC: June 3, 2008).
    ---------------------------------------------------------------------------
    Section 204: Nunn-McCurdy Breaches
        Requires that a major defense acquisition program that experiences
    a critical Nunn-McCurdy cost breach be terminated unless the Secretary
    of Defense certifies that: (1) continuing the program is essential to
    national security and the program can be modified to proceed in a cost-
    effective manner; and (2) the program receives a new milestone approval
    prior to the award of any new or modified contract extending the scope
    of the program.
        In order for DOD to improve its program outcomes, realistic cost
    estimates must be required when programs are approved for development
    initiation. DOD often underestimates costs in large part because of a
    lack of knowledge and overly optimistic assumptions about requirements
    and critical technologies. This underestimation is also influenced by
    DOD's continuing failure to balance its needs with available resources,
    which promotes unhealthy competition among programs and encourages
    programs to overpromise on performance capabilities and underestimate
    cost. This false optimism is reinforced by an acquisition environment
    in which there are few ramifications for cost growth and delays. Only
    in very rare instances have programs been terminated for poor
    performance. When DOD consistently allows unsound, unexecutable
    programs to begin with few negative ramifications for poor outcomes,
    accountability suffers. As section 204 proposes, the strengthening of
    the Nunn-McCurdy provision--by including the potential termination of
    programs that experience critical cost growth--could facilitate a
    change in DOD's behavior by preventing the acceptance of unrealistic
    cost estimates as a foundation for program initiation and placing more
    accountability on senior DOD leadership for justifying program
    continuation. Programs may thus be forced to be more candid and upfront
    about potential costs, risks, and funding needs, and the likelihood of
    delivering a successful capability to the warfighter at the cost and in
    the time promised may grow.

    Section 205: Organizational Conflicts of Interest
        Prohibits systems engineering contractors from participating in the
    development or construction of major weapon systems on which they are
    advising DOD, and requires tightened oversight of organizational
    conflicts of interest by contractors in the acquisition of major weapon
    systems.
        The defense industry has undergone significant consolidation in
    recent years which has resulted in a few large, vertically integrated
    prime contractors. This consolidation creates the potential for
    organizational conflicts of interest where, for example, one business
    unit of a large company may be asked to provide systems engineering
    work on a system being produced by another unit of the same company. As
    the Defense Science Board has recognized, these conflicts of interest
    may lead to impaired objectivity, which may not be mitigated
    effectively through techniques such as erecting a firewall between the
    employees of the two units. While the Federal Acquisition Regulation
    currently covers some cases of potential organizational conflicts of
    interest involving the systems engineering function, there may be a
    need for additional coverage in this area. In general, we would support
    efforts to enhance the oversight of potential organizational conflicts
    of interest, particularly in the current environment of a heavily
    consolidated defense industry.

    Section 206: Acquisition Excellence
        Establishes an annual awards program to recognize individuals and
    teams that make significant contributions to the improved cost,
    schedule, and performance of defense acquisition programs.
        We support the creation of an annual awards program to recognize
    individuals and teams for improving the cost, schedule, and performance
    of defense acquisition programs. We have reported that meaningful and
    lasting reform will not be achieved until the right incentives are
    established and accountability is bolstered at all levels of the
    acquisition process. The need for incentives emerged as a significant
    issue in our recent discussions with acquisition experts examining
    potential changes to the acquisition processes enumerated in last
    year's defense authorization act. The discussions revealed that those
    changes may not achieve the desired improvement in acquisition outcomes
    unless they are accompanied by changes in the overall acquisition
    environment and culture, and the incentives they provide for success.
               concluding observations on what remains to be done
        A broad consensus exists that weapon system problems are serious
    and that their resolution is overdue. With the Federal budget under
    increasing strain from the Nation's economic crisis, the time for
    change is now. DOD is off to a good start with the recent revisions to
    its acquisition policy, which, if implemented properly, should provide
    a foundation for establishing sound, knowledge-based business cases
    before launching into development and for maintaining discipline after
    initiation. The new policy will not work effectively, however, without
    changes to the overall acquisition environment. Resisting the urge to
    achieve the revolutionary but unachievable capability, allowing
    technologies to mature in the science and technology base before
    bringing them onto programs, ensuring that requirements are well-
    defined and doable, and instituting shorter development cycles would
    all make it easier to estimate costs accurately, and then predict
    funding needs and allocate resources effectively. But these measures
    will succeed only if the department uses an incremental approach.
    Constraining development cycle times to 5 or 6 years will force more
    manageable commitments, make costs and schedules more predictable, and
    facilitate the delivery of capabilities in a timely manner.
        Acquisition problems are also likely to continue until DOD's
    approach to managing its weapon system portfolio: (1) prioritizes needs
    with available resources, thus eliminating unhealthy competition for
    funding and the incentives for making programs look affordable when
    they are not; (2) facilitates better decisions about which programs to
    pursue and which not to pursue given existing and expected funding; and
    (3) balances the near-term needs of the joint warfighter with the long-
    term need to modernize the force. Achieving this affordable portfolio
    will require strong leadership and accountability. Establishing a
    single point of accountability could help the department align
    competing needs with available resources.
        The department has tough decisions to make about its weapon systems
    and portfolio, and stakeholders, including military Services, industry,
    and Congress, have to play a constructive role in the process toward
    change. Reform will not be achieved until DOD changes its acquisition
    environment and the incentives that drive the behavior of its
    decisionmakers, the military Services, program managers, and the
    defense industry.
        Mr. Chairman, this concludes my prepared statement. I would be
    happy to answer any questions you may have at this time.
                         contacts and acknowledgements
        For further information about this statement, please contact
    Michael J. Sullivan (202) 512-4841 or sullivanm@gao.gov. Contact points
    for our Offices of Congressional Relations and Public Affairs may be
    found on the last page of this statement. Individuals who made key
    contributions to this statement include John Oppenheim, Charlie
    Shivers, Dayna Foster, Matt Lea, Susan Neill, Ron Schwenn, and Bruce
    Thomas.

        Chairman Levin. Thank you so much, Mr. Sullivan.
        Dr. Gansler?

    STATEMENT OF HON. JACQUES S. GANSLER, CHAIRMAN, DEFENSE SCIENCE
      BOARD TASK FORCE ON INDUSTRIAL STRUCTURE FOR TRANSFORMATION

        Dr. Gansler. Mr. Chairman, members of the committee, thank
    you very much for this honor of appearing before you at what I
    think is a critical period and on such an important topic.
        I don't have to tell this committee of the incredible
    national security challenges that the United States is facing
    in the 21st century, brought on by the rather dramatic world
    changes that I believe require a new, holistic view of
    security--DOD, Department of State, Department of Homeland
    Security, Director of National Intelligence, and so forth--and
    utilizing both hard and soft power, and addressing a very broad
    spectrum of the security missions with great unpredictability
    and covering the full spectrum, from terrorism all the way
    through nuclear deterrence.
        I would also emphasize that we need to take full advantage
    of globalization of the technology of industry, not restricting
    or gaming the benefits from globalization through restrictive
    legislation.
        In recognizing the long-term national security implications
    of the global financial crisis, the need for energy security,
    worldwide pandemics, the impact of climate change, growing
    anti-globalization backlash, and the challenging U.S.
    demographics. We have to do all of this, as Senator McCain
    emphasized, in a likely fiscally-constrained budget
    environment.
        Now, to address these challenges, I believe four highly
    interrelated acquisition issues must be addressed, and they
    have to be addressed by both DOD and Congress. First, what
    goods and services to buy; that's the requirements process.
    Second, how to buy them; that's the acquisition reform. Third,
    who does the acquiring; we have major issues in the acquisition
    workforce. Fourth, from whom it is acquired; namely, the
    industrial base.
        Now, I wish I could tell you that there is some silver
    bullet to address all of these needed changes, but it truly
    requires a very broad set of initiatives in each of the four
    areas if the Nation is to achieve the required 21st century
    national security posture.
        In my prepared testimony, which, Mr. Chairman, I would
    appreciate being put into the record----
        Chairman Levin. It will be.
        Dr. Gansler.----I listed the required actions in each of
    these four areas, and I'd be pleased to discuss any of these
    with you at any time. However, for now let me summarize.
        I believe this is a very critical period, perhaps somewhat
    similar to the period following the launch of Sputnik or the
    fall of the Berlin Wall. Today, the security world is changing
    dramatically, especially since September 11--geopolitically,
    technologically, threats, missions, warfighting, commercially,
    et cetera--and this holistic perspective that I mentioned is
    required. Moreover, a decade of solid budget growth, which I
    believe will almost certainly change, has deferred the
    difficult choices--for example, between more 20th century
    equipment versus 21st century equipment--and the controlling
    acquisition policies, practices, laws, and so forth, as well as
    the Services' budgets and requirements priorities have not been
    transformed sufficiently to match the needs of this new world.
    In fact, there's still an emphasis on resetting versus
    modernization.
        Now, leadership is required to achieve the needed changes.
    You look at the literature on culture change, which I think
    this clearly is, two things are required to successfully bring
    about the needed changes. First is the recognition of the need,
    a crisis. In this case, I believe it is a combination of the
    economic--the budget, if you will, crisis--and the changing
    security needs, along with the shortage of the senior
    acquisition experienced personnel to address these needs.
    Second, leadership, with a vision, a strategy, and an action
    plan. I honestly believe that President Obama, Congress, and
    Secretary Gates support the needed changes; however, it's
    pretty clear that the changes can be expected to be severely
    resisted. Significant change always is.
        I would start, as my highest priority, with the important
    role of the service chiefs and secretaries in recognizing and
    promoting senior acquisition personnel, military and civilian.
    Over the last decade, the DOD acquisition workforce has been
    greatly undervalued. DOD leadership now must demonstrate their
    personal recognition of the critical nature of senior
    experienced acquisition personnel and of the smart acquisition
    practices that they would bring to America's military posture
    in the 21st century.
        As my second priority, I would emphasize the importance of
    weapons costs as a true military requirement, to achieve
    adequate numbers of weapons in a resource-constrained
    environment. This will require enhanced systems engineering,
    including cost-performance tradeoffs, throughout both the
    government and industry, and incentives to industry for
    achieving lower cost.
        By the way, this has been done before; for example, with
    the Joint Direct Attack Munition missile, where the Air Force
    Chief of Staff said it should hit the target and cost under
    $40,000 each. It now sells for under $20,000 and precisely hits
    the targets.
        Finally, as my third priority, I would emphasize the value
    of rapid acquisition, from both its military and its economic
    benefits, which will require the full use of spiral
    development, with each block based on proven, tested technology
    and continuous user and logistician feedback for the subsequent
    block improvements, and with the option of continuous effective
    competition, at the prime or at the sublevel. If they're not
    continuously achieving improved performance at lower costs,
    then they should be competed.
        Achieving these required changes will take political
    courage and sustained, strong leadership by both the executive
    and legislative branches, working together. I hope, and firmly
    believe, that it can be achieved. The American public, and
    particularly our fighting men and women, deserve it, and the
    Nation's future security depends upon it.
        Thank you.
        [The prepared statement of Dr. Gansler follows:]
               Prepared Statement by Hon. Jacques S. Gansler \1\
    ---------------------------------------------------------------------------
        \1\ Dr. Gansler is Professor and Roger C. Lipitz Chair, as well as
    Director of the Center for Public Policy and Private Enterprise at the
    School of Public Policy, University of Maryland. He served as Under
    Secretary of Defense (Acquisition, Technology, and Logistics) from
    1997-2001.
    ---------------------------------------------------------------------------
        Mr. Chairman, members of the committee, thank you for the honor of
    appearing before you at this critical period, and on such an important
    topic.
        I need not tell you that the U.S., in the 21st century, faces
    incredible national security challenges--brought on by dramatic world
    changes that require:

             A new, Holistic View of Security (e.g., DOD, State,
            DHS, DNI, etc.)--utilizing both ``hard'' and ``soft'' power
             Addressing a Broad Spectrum of Security Missions--with
            great unpredictability (from Terrorism to Nuclear Deterrence)
             Taking full advantage of Globalization (of Technology,
            Industry, etc.)
             Recognizing the long-term national security
            implications of:

                     The global financial crisis
                     The need for energy security
                     Worldwide pandemics
                     The impact of climate change
                     The growing anti-globalization backlash
                     The challenging U.S. demographics

             To do all of this in a likely fiscally-constrained
            budget environment

        To address these challenges, four, highly-interrelated acquisition
    issues must be addressed (by the DOD and Congress):

             What goods and services to buy (the ``requirements''
            process)
             How to buy them (``acquisition reform'')
             Who does the acquiring (the acquisition workforce)
             From whom is it acquired (the industrial base)

        I wish I could tell you that there was a ``silver bullet'' to
    address the needed changes; but this truly requires a broad set of
    initiatives in each of the four areas--if the Nation is to achieve the
    required 21st century national security posture.
        This need, for the four sets of broad changes, was emphasized in a
    recent Defense Science Board report; where they found:

             ``DOD policies, processes, and management of the
            Defense Acquisition Enterprise (broadly defined) impede the
            transition to an effective, agile, and affordable overall,
            joint military force for the 21st century.''
             ``U.S. Government policies, practices, and processes
            do not facilitate the development, deployment, and support of
            the innovative, affordable, and rapidly acquired weapons,
            systems, and services needed for the 21st century forces.''
             ``The absence of many of the needed skills, (e.g.,
            experienced program management, systems engineering, biotech,
            advanced IT) in DOD's acquisition workforce, (particularly at
            the senior military and civilian levels), combined with the
            coming retirement and prior, large acquisition workforce
            reductions, significantly impedes the development, production,
            support, and oversight of the military capabilities needed for
            the 21st century.''
             ``Government acquisition policies and industry trends
            (e.g., further horizontal and vertical consolidations) will not
            produce the required competitive, responsive, efficient, and
            innovative National Security Industrial Base.''

        So let me (very briefly) summarize the changes required in each of
    the four, critical (and interrelated) areas: [in priority order within
    each category]

    What is acquired:

        To meet the wide range of challenges, within a resource-constrained
    environment, the Nation must focus on:

              1. Lower cost systems and services
              2. Optimized, net-centric systems-of-systems (vs. individual
            ``platforms'')
              3. A ``reserve'' of resources to rapidly respond to combat
            commanders' urgent needs
              4. More ``balanced'' allocation of resources (to address
            ``irregular'' operations): C\3\ISR, unmanned systems, Special
            Forces, ``Land Warriors,'' cyberdefense, etc; [and these
            resources must be moved from the Supplementals into the base
            budget]
              5. Interoperability of ``joint'' systems; and coalition
            systems
              6. Planning, equipping, and exercising ``as we'll fight'':
            with allies, multi-agencies, and ``contractors on the
            battlefield''

    How goods and services are acquired:

        To achieve higher performance at lower costs and faster:

              1. Require ``cost'' as a design/military ``requirement''
            (because cost, in a resource-constrained environment, is
            numbers)
              2. Provide viable, continuous ``competition options'' (as the
            incentive for higher performance at lower costs) e.g.
            competitive prototypes, competitive split-buys, etc.
              3. Fully utilize ``spiral development,'' with demonstrated
            technologies (because it is lower cost, lower risk, faster to
            field; maintains the option of competition; avoids
            obsolescence; can respond rapidly to combat needs)
              4. Make maximum use of commercial products and services (at
            all levels--utilizing Other Transactions Authority; especially
            at lower tiers)
              5. Institutionalize a ``Rapid acquisition,'' parallel process
            (to respond to COCOM urgent needs)
              6. Create incentives for contractors to achieve desired
            results (in cost, schedule, and performance)
              7. Implement modern, integrated, enterprise-wide IT systems
            (logistics, business, personnel, etc.)--including linking
            Government and Industry
              8. Address Conflict of Interest concerns (from
            LSIMake/BuySETA); but don't reduce the value
            of relevant experience

    Who does the acquiring:

        A flexible, responsive, efficient, and effective acquisition
    program (for sophisticated, hi-tech goods and services) requires
    ``smart buyers.''
        This depends on both quantity and quality of senior and experienced
    military and civilian personnel (especially for expeditionary
    operations). In the last decade-plus, this ``requirement'' has not been
    met! In fact, the acquisition workforce declined on seniority and
    quantity even as procurement appropriations increased.

        Therefore, one of the Nation's highest priorities (not just in the
    DOD) must be to address the acquisition workforce.

             The DOD, especially, has an acquisition workforce
            problem:

                     Greatly reduced senior officers and SESs

                             In 1990 the Army had five general
                            officers with contract background; in 2007 they
                            had zero.
                             In 1995, the Air Force had 40 General
                            Officers in Acquisition, today 24; and in 1995,
                            87 SESs and today 49
                             The Defense Contract Management Agency
                            (25,000 people in 1990 down to 10,000 today;
                            and 4 General Officers to 0)

        These reductions (due to the under valuing of the importance of the
    acquisition workforce) introduce ``opportunities'' for ``waste, fraud
    and abuse'' (e.g., 90 fraud cases under review from war zone; examples
    of poor acquisition process results, such as the Air Force Tanker, the
    Presidential Helicopter, etc.). These Government acquisition workforce
    issues must be addressed. I believe that President Obama, Congress, and
    Secretary Gates all agree on this (but it will take the priority
    attention of the Service Chiefs and Secretaries to make it a priority).

    From whom goods and services are acquired:

        To quote, again, from the recent Defense Science Board study (on
    the desired 21st century defense industry):

            ``The last two decades have seen a consolidation of the Defense
            Industry around 20th Century Needs--The next step is DOD
            leadership in transforming to a 21st Century National Security
            Industrial Structure.''

        The ``vision'' for this 21st Century National Security Industrial
    Base (which appropriate government actions, i.e. acquisitions,
    policies, practices, and laws, must incentivize and facilitate) are:

              1. Efficient, responsive, technologically advanced, highly-
            competitive (at all levels, including public and private
            sectors)
              2. Globalized (utilizing ``best in class'')--requires
            significant changes to U.S. export controls (i.e., changes to
            ITAR, EAR, etc.)
              3. Healthy (profitable); and investing in IR&D and capital
            equipment (rules should separate IR&D and B&P)
              4. Includes commercial firms and equipment, and maximizes
            dual-use facilities and workforce (barriers must be removed)
              5. ``Independent'' systems-of-systems architecture and
            systems engineering firms (to support the Government--as the
            integrator)
              6. Merger and Acquisition policy guidelines to be based on
            this vision
              7. Strong Government-Industry Communications encouraged
              8. All non-inherently-governmental work to be done
            competitively (public vs. private, for current government work)
              9. Structural changes to eliminate appearance, or reality, of
            conflict of interest (regarding ``vertical integration'')--but
            great care to assure relevant--experienced firms and people
            involved

        In summary, I believe this is a critical period, perhaps similar to
    the period following the launch of Sputnik or the fall of the Berlin
    Wall. Today the security world is changing dramatically--especially
    since September 11, 2001 (geopolitically, technologically, threats,
    missions, warfighting, commercially, etc.)--and a holistic perspective
    is required (including State, DHS, and DNI, as well as coalition
    operations). Moreover, a decade of solid budget growth--which will
    almost certainly change--has deferred difficult choices (between more
    20th century equipment vs. 21st century equipment). The controlling
    acquisition policies, practices, laws, etc. and the Services' budgets
    and ``requirements'' priorities have not been transformed sufficiently
    to match the needs of this new world (in fact, there is still an
    emphasis on ``resetting'' vs. ``modernization'').
        Leadership is required to achieve the needed changes! All of the
    literature on ``cultural change'' (which this clearly must be) state
    that two things are required to successfully bring about the needed
    changes:

             Recognition of the need (a ``crisis'')

        In this case, I believe it is the combination of economic/budget
    ``crisis,'' the changing security needs, and the shortage of the senior
    acquisition-experienced personnel to address the needs; and

             Leadership--with a ``vision,'' a ``strategy,'' and an
            ``action plan.'' I believe that President Obama, Congress, and
            Secretary Gates support the needed changes. However, the
            changes can be expected to be severely resisted--significant
            change always is!

        I would start with the important role of the Service Chiefs and
    Secretaries in recognizing, and promoting senior acquisition personnel
    (military and civilian) in order to demonstrate their personal
    recognition of the critical nature of smart acquisition practices to
    American's military posture in the 21st century. As my second priority,
    I would emphasize the importance of weapons costs as a military
    requirement (to achieve adequate members of weapons in a resource-
    constrained environment)--which will require enhanced systems
    engineering (throughout both government and industry) and incentives to
    industry for achieving lower cost systems. Finally, as my third
    priority, I would emphasize the value of ``rapid acquisition'', for
    both its military and economic benefits--which will require the full
    use of ``spiral development'' (with each ``block'' based on proven/
    tested technology, and continuous user and logistician feedback, for
    subsequent ``block'' improvement--and with the option of effective
    competition (at the prime and/or sub-level, if they are not
    continuously achieving improved performances at lower and lower costs).
        Achieving these required changes will take political courage and
    sustained, strong leadership--by both the executive and legislative
    branches (working together). I hope, and firmly believe, it can be
    achieved. The American public, and particularly, our fighting men and
    women, deserve it--and the Nation's future security depends upon it.
        Thank you.

        Chairman Levin. Dr. Gansler, thank you so much.
        Dr. Kaminski?

      STATEMENT OF HON. PAUL G. KAMINSKI, CHAIR, COMMITTEE ON PRE-
       MILESTONE A SYSTEMS ENGINEERING, AIR FORCE STUDIES BOARD,
                       NATIONAL RESEARCH COUNCIL

        Dr. Kaminski. Mr. Chairman, Senator McCain, and members of
    the committee, first of all, I want to thank you for your
    leadership on these critical acquisition issues and for the
    invitation to testify.
        Since you've asked me to testify, first, in my role as
    Chairman of the National Research Council's Study on Pre-
    Milestone A Systems Engineering, with your permission I would
    ask that my statement, which includes a full summary, be put in
    the record, and then I will proceed to provide a short verbal
    summary of the summary.
        Chairman Levin. All the statements will be made part of the
    record. Thank you.
        Dr. Kaminski. Thank you, sir.
        Recent years have seen a serious erosion in our ability to
    field new weapon systems quickly in response to changing
    threats, as well as a large increase in the cost of these
    weapon systems. Our programs today for developing weapon
    systems take two to three times longer than they did 30 years
    ago. I note that time is money in this process, and time also
    leaves room for disruptions, uncertainty, and changes in
    commercial technology. In a 15- or 20-year period, we're seeing
    commercial technology turnover three, four, or five times. When
    a weapon system takes 15 to 20 years to develop, the technology
    that you start with isn't going to be supported when it's
    fielded. So, we have to vigorously attack this time issue.
        Our committee also noted the importance of systems
    engineering in reducing this acquisition time, when combined
    with development planning. We further underscored the
    importance of an early systems engineering effort, in that, the
    decisions made prior to and the key Milestone A and B decisions
    impact somewhere between 75 percent and 85 percent of the total
    life-cycle cost. So, the time to address those issues is up
    front, before those decisions are made.
        Our committee also noted that many of the conclusions that
    we reached have been reached in several previous studies. So,
    the issue isn't disagreement on what the recommendations are,
    the issue is implementing those recommendations. So, once
    again, we thank you for your leadership in creating a forum for
    that kind of implementation.
        Let me address now the issues that you asked me to address.
        First of all, just one overall comment on systems
    engineering. I agree with Secretary Gates, who, when asked
    about acquisition, said, ``There is no one silver bullet that
    is going to correct all the problems.'' But, I do believe that
    good systems engineering, coupled with effective development
    planning early on, are two of the most important contributors
    to successful acquisition.
        Our report provided some formal definitions of ``systems
    engineering,'' but they tend to be arcane, so I thought I might
    start with a couple of examples. I'll briefly describe some
    examples of good systems engineering in the work we've done,
    and also where have we seen poor systems engineering.
        One of the really good examples is the Apollo program. That
    program, from a dead start, put man on the moon in about 8
    years. When that program was started, we didn't have mature
    technology. What we did was good upfront systems engineering
    and development planning, so we could proceed in a sequential
    way, step by step, with each new step building on the previous
    step. In building hardware, we were also building the
    experience of our acquisition workforce and our industry, so we
    could, step by step, increase our capabilities, eventually
    going to the moon.
        Another really good example is the Air Force
    Intercontinental Ballistic Missile (ICBM) programs that were
    done in the 1970s and 1980s. What we saw there is that we would
    never start a full-scale development contract for a new ICBM
    until we had done the upfront systems engineering and
    development planning. The development planning produced an
    inertial guidance system for the ICBM, as well as critical
    propulsion components, and a reentry vehicle. That not only
    reduced the risk of the hardware development and integration in
    the future, it also gave domain experience to our key people in
    government and industry, so that when we threw the switch and
    started full-scale development, we could typically expect a
    first flight in 3 to 4 years, as a result of that experience
    base. That's what we need to restore.
        You asked: ``What were systemic contributors to acquisition
    problems?'' I listed five.
        The first of these is the lack of this early and continuing
    systems engineering, coupled to a development planning program
    early on, right upfront.
        The second key impediment is the lack of alignment of
    responsibility, authority, and accountability of the program
    manager. A program manager needs to be able to exercise his or
    her judgment. Much of the program manager's authority has been
    taken away by one-size-fits-all approaches to acquisition and
    by the oversight process, which has some onerous elements that
    are nonvalue-added.
        A third major impediment is the lack of stability in
    program funding. Many contribute to that.
        A fourth is the lack of early attention to test and
    evaluation, with insufficient planning and investment in the
    tools, such as modeling and simulation, test equipment,
    facilities, and personnel, to provide us with the timely and
    meaningful results needed by program management and for
    continuous systems engineering to refine our performance
    objectives and development plans.
        Finally, the root fundamental issue here is this excessive
    time to acquire that I had spoken about. Time is money. As this
    time increases from a few years in the past to 15 years today,
    it undermines our entire process, causing the key participants
    to lose what I call the ``recipe'' for how we move forward and
    also to lose a sense of accountability. When we see new
    capabilities that are developed and fielded in 5 years, the
    engineers, the managers, the testers, the cost analysts are all
    able to benefit and apply their experience from previous
    programs, and they can also be held accountable, since they can
    be in place managing the programs deliverables during one
    assignment. That all changes when we move to 15-year
    acquisitions and we have five rollovers of management,
    engineers, and cost analysts, and five rollovers of the
    technology in the process.
        So, attacking acquisition time is fundamental. I would say,
    a testament to our failure today is the fact that we have to
    discard our current acquisition approaches to deal with our
    urgent needs and field systems, such as the Mine Resistant
    Ambush Protected vehicle and counter-improvised explosive
    devices (IEDs) by forming and using rapid-reaction
    organizations, because our existing ones don't work. They can't
    respond to the cycle time that we need.
        So, what do we do about this? Again, I've listed in my
    statement five steps.
        The first is to ensure that we not only restore, but
    enhance, this early and continuing systems engineering work,
    coupled with development planning. This means restoring funding
    upfront in the programs, and using independent estimates to
    ensure we have enough funding upfront. It also means attracting
    best and brightest to the critical systems engineering work,
    and providing a path to career advancement, career tracking,
    and leadership for the key people that we need to rebuild in
    cadre.
        Second issue is the alignment of the responsibility,
    authority, and accountability of the program manager. I've
    listed several steps in my statement about what's needed to be
    done to do that.
        Third issue is improving funding stability. We pay a great
    deal for the instability we cause by making funding adjustments
    to program. My experience shows that every time we make a cut
    in a program, for financial or other reasons unrelated to
    performance, we end up eventually putting in three times what
    we cut to restore the program later and get it back to a base.
        Fourth item is giving early and serious attention to the
    test and evaluation issues that I noted earlier, so they can be
    part of a rapid process. When we wait for test and evaluation
    results because we haven't done a good job planning and
    preparing, what we have is hundreds of people sitting on their
    hands, waiting for results, and we're paying all those people
    while they wait for results.
        Finally, the last item is fundamentally attacking this
    problem of long development times by the combination of the
    previous four items.
        I believe action on these five issues will have a
    significant and demonstrable impact on our serious acquisition
    problems. I believe that we need to move now with the same
    urgency and priority that we expect in combat operations to
    permit the timely and effective development and fielding of new
    capabilities and services with what I expect will be more
    limited future defense dollars.
        Thank you, Mr. Chairman.
        [The prepared statement of Dr. Kaminski follows:]

                   Prepared Statement by Dr. Paul G. Kaminski

        Chairman Levin, and Ranking Member McCain:
        Thank you for your leadership on the Department of Defense (DOD)
    acquisition, and for the invitation to testify on these important
    acquisition issues. Since you have asked me to testify in my role as
    Chair, Committee on Pre-Milestone A Systems Engineering, Air Force
    Studies Board, National Research Council, I will begin by providing a
    summary of our report, which was approved by the Governing Board of the
    National Research Council and published in 2008. The report is
    available to the public at http://www.nap.edu. After the report
    summary, I will provide my personal views on systems engineering and
    respond to the key issues you asked that I address.

                                    SUMMARY

        Recent years have seen serious erosion in the ability of U.S.
    forces to field new weapons systems quickly in response to changing
    threats, as well as a large increase in the cost of these weapons
    systems. Today the military's programs for developing weapons systems
    take two to three times longer to move from program initiation to
    system deployment than they did 30 years ago. This slowdown has
    occurred during a period in which threats have been changing more
    rapidly than ever and when technology advances and accumulated
    experience should have been accelerating rather than slowing the
    development process.
        Many causes for this trend have been suggested, including the
    increased complexity of the tasks and the systems involved from both
    technological and human/organizational perspectives; funding
    instability; loss of ``mission urgency'' after the end of the Cold War;
    bureaucracy, which increases cost and schedule but not value; and the
    need to satisfy the demands of an increasingly diverse user community.
    The difficulty of focusing on a specific, homogeneous, post-Cold War
    threat made problems even worse. Yet although the suggested causal
    factors have merit, a common view is that better systems engineering
    (SE) and development planning could help shorten the time required for
    development, making it more like what it was 30 years ago.
        Simply stated, SE is the translation of a user's needs into a
    definition of a system and its architecture through an iterative
    process that results in an effective system design. SE applies over the
    entire program life cycle, from concept development to final disposal.
        The Committee on Pre-Milestone A Systems Engineering was tasked by
    the U.S. Air Force to examine the role that SE can play during the
    defense acquisition life cycle in addressing the root causes of program
    failure, especially during the pre-Milestone A and early phases of a
    program. Currently, few formal SE processes are applied to Air Force
    development programs before the Milestone A review.\1\
    ---------------------------------------------------------------------------
        \1\ This is a result of the elimination in the 1990s of the
    development planning function that had existed in the Air Force Systems
    Command.
    ---------------------------------------------------------------------------
        The committee devoted considerable time and space in its report to
    trying to define a minimum set of systems engineering processes. The
    most important of these processes are summarized in the checklist in
    Box S-1 below. A few of the things that need to be taken care of before
    Milestone A and just after it are the following: the consideration of
    alternative concepts (solutions) up front; the setting of clear,
    comprehensive key performance parameters (KPPs) and system
    requirements; and early attention to interfaces and interface
    complexity, to the concept of operations, and to the system
    verification approach. It is these early-stage processes that are
    covered in this report. The importance of stable requirements and
    funding between Milestone B and the achievement of initial operational
    capability (IOC) is stressed, as are processes including good
    configuration management and change control. The committee further
    stresses in the report what it regards as six of the most important
    process areas in its discussion of six ``seeds of failure''.

        Figure S-1 DOD life-cycle acquisition process. Points A, B, and C
    at the top of the figure represent Milestones A, B, and C. LCC, life-
    cycle cost. Source: Richard Andrews, 2003, An Overview of Acquisition
    Logistics. Fort Belvoir, VA; Defense Acquisition University. Available
    at http://www.afcea.org/events/pastevents/documents/
    Track4Session4AMCEmphasisonCustomerFocusedITInitiatives.ppt#364,12,Slide
     12. Last accessed on November 20, 2007.

             SYSTEMS ENGINEERING AND THE DOD ACQUISITION LIFE CYCLE

        The use of formal systems engineering practices throughout the life
    cycle of an acquisition program is critical to fielding the required
    system on time and within budget. Across the top of Figure S-1 are the
    points at which important management decisions are made: Milestones A,
    B, and C. Concept development and refinement occur before Milestone A,
    and further technology development, to reduce system design and
    development (SDD) risk, occurs before Milestone B. Only after Milestone
    B does a program become an enterprise with dedicated funding.
    Importantly, Figure S-1 shows that about three-quarters of total system
    life-cycle costs are influenced by decisions made before the end of the
    concept refinement phase at Milestone A, while about three-quarters of
    life-cycle funds are not actually spent until after Milestone C. This
    means that although high-quality SE is necessary during the entire
    acquisition cycle, the application of SE to decisions made in the pre-
    Milestone A period is critical to avoiding (or at least minimizing)
    cost and schedule overruns later in a program. Much of the value of
    early, high-quality SE will be manifested as success in fulfilling
    Milestone B requirements.

                       MAIN FINDINGS AND RECOMMENDATIONS

        The committee's main findings and recommendations are given below.
    Finding
        Attention to a few critical systems engineering processes and
    functions particularly during preparation for Milestones A and B is
    essential to ensuring that Air Force acquisition programs deliver
    products on time and on budget.
        Today's weapons systems provide unprecedented capabilities but also
    involve complex interfaces with external command, control, and
    communications systems and rely on a greater volume of software than
    ever before. Early decisions on the weapons system requirements and
    capabilities have a disproportionately large impact on program cost and
    schedule. The committee also recognizes that a lack of flexibility (a
    result of overly rigid processes or a lack of trust among program
    participants or stakeholders) can limit the ability of a program
    manager to change early decisions that warrant changing.
        The committee found many gaps and inconsistencies in the way that
    the Air Force manages pre-Milestone A activities. The committee heard
    from presenters of some cases for which required documents were
    completed pro forma and filed away, never to be seen again, or for
    which required steps were skipped completely. The current practice of
    initiating programs at Milestone B denies the acquisition review
    authority the earlier opportunity (at Milestone A) to make judgments
    about the maturity of the technologies on which the program is based
    and to decide whether technologies need to be further developed prior
    to making a Milestone B commitment to system development and
    demonstration.
    Recommendation
        The Air Force leadership should require that Milestones A and B be
    treated as critical milestones in every acquisition program and that a
    checklist such as the ``Pre-Milestone A/B Checklist'' suggested by the
    committee (see Box S-1 in this Summary) be used to judge successful
    completion.
        A rigorous, standard checklist of systems engineering issues should
    be addressed by each program through both the pre-Milestone A and pre-
    Milestone B phases. The committee's recommended 20-item checklist is
    shown in Box S-1.

        While the committee considers that each item on the checklist is
    important, it calls attention to several items that warrant further
    discussion. Item 2 recognizes that the world changes too fast to be
    friendly to long development cycles. The committee believes that the
    Air Force should strive to structure major development programs so that
    initial deployment is achieved within, say, 3 to 7 years. Thirty years
    ago, this was a typical accomplishment--for example, nearly 40 years
    ago, the Apollo program put the first man on the Moon in fewer than 8
    years.
        The development time issue is addressable by applying systems
    engineering to Items 3, 4, and 13 through 15 before Milestones A and B.
    The definition of clear KPPs by Milestone A and clear requirements by
    Milestone B that can remain stable through IOC can be essential to an
    efficient development phase. It is also important that critical
    technologies be sufficiently mature prior to starting SDD. The
    committee observed that although today's systems are not necessarily
    more complex internally than those of 30 years ago, their ``external
    complexity'' often is greater, because today's systems are more likely
    to try to meet many diverse and sometimes contradictory requirements
    from multiple users. This kind of complexity can often lead to
    requirements being changed between Milestone B and IOC, and it can lead
    to relying on immature technology.
        Item 19 of the checklist stresses the importance of placing
    experienced, domain-knowledgeable managers in key program positions.
    The committee has observed that many of the truly extraordinary
    development programs of the past, such as Apollo, the Manhattan
    Project, the early imaging satellite programs, the U-2, the fleet
    ballistic missile system, and nuclear submarines, were managed by
    relatively small (and often immature) agencies with few established
    processes and controls. In that environment, dedicated managers driven
    by urgent missions accomplished feats that often seem incredible today.
        The committee believes that the accumulation of processes and
    controls over the years--well meant, of course--has stifled domain-
    based judgment that is necessary for timely success. Formal SE
    processes should be tailored to the application. But they cannot
    replace domain expertise. In connection with item 19, the committee
    recommends that the Air Force place great emphasis on putting seasoned,
    domain-knowledgeable personnel in key positions--particularly the
    program manager, the chief system engineer, and the person in charge of
    ``requirements''--and then empower them to tailor standardized
    processes and procedures as they feel is necessary.
        One key pre-Milestone A task is the analysis of alternatives (AoA),
    which entails evaluating alternative concepts and comparing them in
    terms of capabilities, costs, risks, and so on. Checklist items 1
    through 4, 12, and 13 should be completed before the AoA, while items 5
    through 11 and 14 through 20 may be addressed after the AoA.

    Finding
        The creation of a robust systems engineering process is critically
    dependent on having experienced systems engineers with adequate
    knowledge of the domain relevant to a contemplated program.
        While the systems engineering process has broad use, effective
    application depends on having domain experts who are aware of what has
    gone wrong (and right) in the past, recognize the potential to repeat
    the successes under new circumstances, and avoid repeating the errors.
        Ideally, a person or persons with domain knowledge would have had
    experience working on exactly the same problem, or at least a problem
    related to the one at hand. If that is not so (and it might not be if
    the problem has never been addressed before, as was the case for Apollo
    and nuclear submarines), the term could be taken to refer to academic
    training in the relevant field of engineering or science. It could also
    refer to the practice of critical thinking and problem solving that
    comes with learning to be a systems engineer and then building on that
    foundation to gain the experiential knowledge and understanding of
    engineering in the context of an entire system. Systems engineering is
    enabled by tools that have been developed to assist in the management
    of systems engineering (not to be confused with the practice of systems
    engineering).
        Both industry and Air Force presenters told the committee that
    there are not enough domain-knowledgeable and experienced systems
    engineers to support all of the programs that need them.

    Recommendation
        The Air Force should assess its needs for officers and civilians in
    the systems engineering field and evaluate whether either its internal
    training programs, which include assignments on Air Force programs that
    provide mentoring by experienced people and hands-on experience in the
    application of systems engineering principles, or external
    organizations are able to produce the required quality and quantity of
    systems engineers and systems engineering skills. Based on this
    assessment, the Air Force first should determine how and where students
    should be trained, in what numbers, and at what cost, and then
    implement a program that meets its needs.
        The Air Force needs to attract, develop, reward, and retain systems
    engineers across the full spectrum of relevant domains, engage them in
    the early (pre-Milestone A) phase of new programs (or modification
    programs), and sustain their participation throughout the life of the
    programs. One important step in this process would be to create an Air
    Force occupational code for systems engineering so that engineers'
    experience and education can be tracked and managed more effectively.
    The Air Force should support an internal systems engineering career
    track that rewards the mentoring of junior systems engineering
    personnel, provides engineers with broad systems engineering
    experience, provides appropriate financial compensation to senior
    systems engineers, and enables an engineering career path into program
    management and operations.

    Finding
        The Government, federally Funded Research and Development Centers
    (FFRDCs), and industry all have important roles to play throughout the
    acquisition life cycle of modern weapons systems.
        Since the need for a new or upgraded weapons system is most often
    first recognized by the military user, it is appropriate for the
    military to codify its requirements and, with support from FFRDC and
    independent systems engineering and technical assistance contractors,
    to explore materiel and nonmateriel solutions (such as doctrinal,
    organizational, or procedural changes) as well as to assess the
    potential for new technology to provide enhanced capabilities. While it
    is appropriate and usually desirable to engage development contractors
    in the pre-Milestone B process using competitive study contracts, the
    source selection for system development and demonstration should not be
    made until after the work associated with Milestones A and B is
    complete.

    Recommendation
        Decisions made prior to Milestone A should be supported by a
    rigorous systems analysis and systems engineering process involving
    teams of users, acquirers, and industry representatives.
        Working together, government and industry can develop and explore
    solutions using systems engineering methodology to arrive at an optimal
    systems solution.

    Finding
        The Air Force used to have a development planning organization that
    applied pre-Milestone A systems engineering processes to a number of
    successful programs, but that organization was allowed to lapse.
        The role of the Air Force development planning organization, which
    was within the Air Force Systems Command, was to provide standard
    evaluation tools and perform pre-Milestone A systems engineering
    functions across acquisition programs. The early 1990s saw an erosion
    of this front-end planning organization along with its funding as the
    Air Force Systems Command (now the Air Force Materiel Command) began to
    play a decreasing role in program execution. In the opinion of several
    speakers who met with the committee, one main reason for the erosion of
    funding was a lack of congressional support for the planning function.

    Recommendation
        A development planning function should be established in the
    military departments to coordinate the concept development and
    refinement phase of all acquisition programs to ensure that the
    capabilities required by the country as a whole are considered and that
    unifying strategies such as network-centric operations and
    interoperability are addressed.
        The Air Force and the other military services should establish a
    development planning organization like that which existed in the early
    1990s.
        The roles and functions of the various organizations involved in
    acquiring major weapons systems need to be clearly defined. The
    responsibility for executing systems engineering and program management
    in the pre-Milestone A and B phases should be vested in the military
    departments that do the actual development planning functions. This
    should not be the responsibility of the Office of the Secretary of
    Defense (OSD) or of the Joint Staff. Instead, those offices need to
    enable the creation and functioning of military department development
    planning organizations with policy measures and, where appropriate,
    resources. The Joint Staff, under the auspices of the Joint
    Requirements Oversight Council, may help to define the requirements for
    major programs in the course of the development planning process, but
    it should not run the process itself.
        The existence of ``joint'' programs or a program such as Missile
    Defense, which has several related systems being developed by different
    military services, requires clear guidance from both OSD and the Joint
    Staff about who is in charge. These programs need to be harmonized and
    integrated by the responsible integrating agency. However, development
    planning activities should still take place in the military departments
    where the expertise resides. Consequently, the development planning
    should be managed by that agency.
        While this committee cannot predict how Congress will view the
    revival of a good planning process to support pre-Milestone A program
    efforts, it is still important for the Air Force and DOD to make the
    case for the critical importance of this process before Congress and
    others. A development planning process is important not to start new
    programs, but rather to ensure that any new program (or a new start of
    any kind) is initiated with the foundation needed for success. Funding
    for this planning function needs to be determined by the military
    services, including both the acquisition communities and those (the
    warfighters) who generate the operational requirements.

                              CONCLUDING THOUGHTS

        Many of the conclusions reached and recommendations made by the
    committee are similar to those of previous reviews. Most of the past
    recommendations were never implemented, so one of this committee's most
    critical thoughts relates to the importance of implementation. A
    sampling of key findings and recommendations from previous studies
    follows:

             Government Accountability Office (GAO) 2,3
    ---------------------------------------------------------------------------
        \2\ GAO, 2003, Defense Acquisitions: Improvements Needed in Space
    Systems Acquisition Management Policy, September. Available at http://
    www.gao.gov/new.items/d031073.pdf. Last accessed April 2, 2007.
        \3\ GAO, 2005, Space Acquisitions: Stronger Development Practices
    and Investment Planning Needed to Address Continuing Problems, July.
    Available at http://www.gao.gov/new.items/d05891t.pdf. Last accessed
    April 2, 2007.

                     Separate technology development from systems
                    acquisition. Commit to a program only if the technology
                    is sufficiently mature. Set the minimum Technology
                    Readiness Level (TRL).
                     Stabilize the requirements early.
                     Employ systems engineering techniques before
                    committing to product development.
                     Employ evolutionary approaches that pursue
                    incremental increases in capability.
                     Address shortfalls in science, engineering,
                    and program management staff.

             National Defense Industrial Association (NDIA) \4\
    ---------------------------------------------------------------------------
        \4\ NDIA Systems Engineering Division, 2003, Task Report: Top Five
    Systems Engineering Issues in Defense Industry, January, Arlington, VA:
    NDIA.

                     Increase SE awareness and recognize SE
                    authority in the program formulation and decision
                    process.
                     Incentivize career SE positions within the
                    government.

             Defense Science Board (DSB) \5\
    ---------------------------------------------------------------------------
        \5\ DSB/Air Force Scientific Advisory Board (AFSAB) Joint Task
    Force, 2003, Acquisition of National Security Space Programs, May,
    Washington, DC.
    ---------------------------------------------------------------------------
                     Overhaul the requirements process.
                     Stabilize acquisition tours.
                     Establish a robust SE capability.

             Defense Acquisition Performance Assessment (DAPA) \6\
    ---------------------------------------------------------------------------
        \6\ Ronald Kadish, Gerald Abbott, Frank Cappuccio, Richard Hawley,
    Paul Kern, and Donald Kozlowski. 2006. Defense Acquisition Performance
    Assessment. Available at http://www.acq.osd.mil/dapaproject/documents/
    DAPA-Report-web/DAPA-Report-web-feb21.pdf. Last accessed on April 2,
    2007.

                     Strategic technology exploitation is a key
                    U.S. advantage. Opportunities need to be identified
                    early.
                     The U.S. economic and security environments
                    have changed--for example, there are fewer prime
                    contractors, smaller production runs, reduced plant
                    capacity, fewer programs, and unpredictable threats.
                     The acquisition system must deal with
                    instability of external funding.
                     The DOD management model is based on a lack of
                    trust. Quantity of review has replaced quality. There
                    is no clear line of responsibility, authority, or
                    accountability.
                     Oversight is preferred to accountability.
                     Oversight is complex, not process- or program-
                    focused (as it should be).
                     The complexity of the acquisition process
                    increases costs and draws out the schedule.
                     Incremental improvement applied solely to the
                    ``little a'' acquisition process \7\ requires all
                    processes to be stable--but they are not.
    ---------------------------------------------------------------------------
        \7\ The Acquisition--``Big A''--system is often believed to be a
    simple construct that efficiently integrates three independent
    processes: requirements, budgeting, and acquisition. ``Little a'' on
    the other hand, refers to the acquisition process that focuses on ``how
    to buy'' in an effort to balance cost, schedule, and performance; it
    does not include requirements and budgeting.

        The committee notes that successful implementation of these
    recommendations requires the ``zipper concept''--making connections at
    all levels, from the senior leadership of the Air Force and DOD down to
    the working levels within key program management offices and
    supervisory staffs.
       committee on pre-milestone a systems engineering: a retrospective
          review and benefits for future air force systems acquisition
        Paul Kaminski, Chair, Technovation, Inc.
        Lester Lyles, Vice Chair, U.S. Air Force, (Ret.)
        Dev Banerjee, The Boeing Company
        Thomas Blakely, Lockheed Martin Aeronautics Company
        Natalie Crawford, RAND Corporation
        Stephen Cross, Georgia Tech Research Institute
        Gilbert Decker, Independent Consultant
        Llewelln Dougherty, Raytheon
        John Farr, Stevens Institute of Technology
        James Frey, Frey Associates
        Robert Fuhrman, Lockheed Martin (Ret.)
        David Gorney, The Aerospace Corporation
        John Griffin, Griffin Consulting
        Wesley Harris, Massachusetts Institute of Technology
        Ronald Kadish, Booz Allen Hamilton
        Robert Latiff, Science Applications International Corporation
        Alden Munson, Independent Consultant
        Mark Wilson, Mark Wilson Consulting
    Staff
        Michael A. Clarke, Board Director
        James C. Garcia, Study Director
        Greg Eyring, Senior Program Officer
        Carter W. Ford, Associate Program Officer
        William E. Campbell, Senior Program Associate
        Lanita Jones, Program Associate

        Having summarized the findings and recommendations of the Committee
    on Pre-Milestone A Systems Engineering, let me now add my personal
    views on systems engineering, and the two additional questions that you
    asked me to address: 1) the systemic issues that have contributed to
    cost, schedule, and performance problems in the acquisition of major
    weapon systems; and 2) the steps that Congress and the Department need
    to take to improve performance of the Department's acquisition
    programs.

                              SYSTEMS ENGINEERING

        I agree with Secretary Gates who said that there is no one silver
    bullet that will correct all of the DOD acquisition problems. But I
    believe that good systems engineering coupled with effective
    development planning are the two most important contributors to
    successful acquisition. Our report provided formal definitions of
    systems engineering and development planning that are somewhat arcane.
    So rather than provide further definition, I find it easier to
    illustrate by choosing examples of good and bad systems engineering and
    development planning. Examples of good work include the Apollo Program
    and the U.S. Air Force intercontinental ballistic missile (ICBM)
    programs (e.g., Minuteman, MX) in the 1970s-1980s. Apollo succeeded in
    putting men on the moon in about 8 years. At the start of the program,
    almost all of the key technologies were immature. But good systems
    engineering and development planning were applied to develop a
    systematic approach, reducing risk by taking a series of limited steps,
    and applying the learning and domain experience gained from each step
    to the subsequent step. The U.S. Air Force ICBM programs used a similar
    approach, beginning with conceptual studies and technology development,
    and holding initiation of full scale development (FSD) contracts until
    key guidance system, re-entry system and propulsion technologies had
    been demonstrated. As a result, the time from initiation of FSD until
    first flight was typically 3-4 years. An example of poor systems
    engineering is the SBIRS program, in which a lack of domain experience
    and analysis led to a failure to anticipate the possibility of severe
    radio frequency interference between two key payloads--discovering this
    problem years after program initiation. Inadequate systems engineering
    and development are therefore the first of five items listed below as
    systemic contributors to acquisition problems.

                 SYSTEMIC CONTRIBUTORS TO ACQUISITION PROBLEMS

              1. The lack of early and continuing systems engineering and
            the absence of a closely-coupled development planning program
            are a fundamental contributor, as identified in our report. The
            root causes include: (a) lack of sufficient personnel (in both
            government and industry) with adequate education, training and
            domain experience (this includes personnel in requirements
            development as well as in acquisition); and (b) lack of
            sufficient front end investment necessary to understand the key
            tradeoffs in cost/schedule/performance, and to identify and
            address the key risks in a systematic manner.
              2. Lack of alignment of responsibility, authority, and
            accountability of the program manager. In many cases the
            program manager's authority is diffused by many levels of
            oversight in both the Department and in Congress, and the
            financial and performance constraints imposed do not allow
            sufficient freedom of action to apply informed judgment in a
            timely manner. Flexibility is further limited by application of
            a ``one-size-fits-all'' approach imposed by the DOD 5000
            system, and the oversight practiced by the DOD and Congress. A
            program manager needs the freedom to tailor the acquisition
            approach to the problem, to ensure that the program response
            time will fall within the response time of the threat, and to
            apply a variety of tools and techniques (such as the use of
            prototypes, competitive prototypes, modeling and simulation,
            critical subsystem and component demostration). For this to
            work, we need program managers with the education, training,
            and domain experience needed to enable timely responses and
            excellent judgment relevant to the domain.
              3. Lack of stability in program funding.
              4. Lack of early attention to test and evaluation, with
            insufficient planning and investment in the tools (e.g.,
            modeling and simulation, test equipment, facilities, and
            personnel) to provide the timely and meaningful results needed
            by program management and systems engineering to continually
            refine performance objectives and development plans.
              5. Excessive (and growing) time from program initiation to
            fielding. As this time increases from a few years to 15 years
            or more, it undermines the entire acquisition process by
            causing key participants to ``lose the recipe'', and lose a
            sense of accountability as well as a sense of being able to
            make a difference. When new capabilities are developed and
            fielded in 5 years, engineers, managers, testers, cost
            analysts, etc. are able to benefit from and apply the
            experience gained from a previous program or program phase.
            They can also see the results of their decisions and be held
            accountable. We can also meaningfully employ past performance
            of the contractor as a factor in the award of future programs--
            an important factor in incentivizing contractor performance.
            This all changes dramatically when the time extends to 15
            years, and we have five roll-overs of management, engineers,
            cost analysts, and commercial technology during this time
            period. This long and growing time period is a result of the
            inflexibility inherent in our entire system of requirements
            development, budgeting and acquisition, and it creates a
            vicious cycle in which it further exacerbates the contributors
            above, and they in turn further increase the time and cost
            growth. We see the result when we must discard our current
            acquisition system in order to deal with urgent needs and field
            systems such as MRAP and jammers to counter IED's by forming
            and using rapid reaction organizations. This cycle must be
            broken by attacking the root causes.
              steps that congress and the department need to take
              1. The first step is to insure that we not only restore, but
            enhance early and continuing systems engineering coupled with
            effective development planning. This will require commitment of
            more significant investment dollars earlier in our acquisition
            programs, and a commitment to build a cadre of systems
            engineers and development planners with the education, training
            and domain experience needed to be effective. Attracting ``best
            and brightest'' to this work--and keeping them--will require a
            personnel system that will identify and track these important
            human resources and establish a career path to allow those who
            are successful to advance to senior program management and
            leadership positions. Their domain experience will be enhanced
            by managing the building of critical subsystems during the
            development planning program, reducing risk and building skills
            and experience at the same time. Congress and the Department
            can assist by providing incentives for attracting and keeping
            key personnel (not only financial incentives, but educational,
            training, recognition, and most important--the opportunity to
            take on challenging developments and see that they can make a
            difference). We will need metrics to assess how well we are
            doing in building and applying this cadre, and we must
            recognize that this will not be accomplished in 4, or even 8
            years. But we must begin in earnest and begin now. Finally, we
            need a means to insure that we have adequate funding upfront
            for new programs; one approach would require a report at
            program initiation from an independent cost analyst working
            with system engineers and development planners who have
            developed their skills on previous programs.
              2. Alignment of the responsibility, authority, and
            accountability of the program manager requires that a degree of
            trust be established between the program manager and those
            responsible for our oversight mechanisms. We must be prepared
            to delegate authority to the program manager, and provide him
            or her with some flexibility to manage--to adjust levels and
            allocation of funding, to adjust the allocation of performance
            parameters, to adjust schedule, and to tailor the acquisition
            approach to be responsive to the need. Clearly, there must be
            bounds established beyond which the program manager must seek
            approval from oversight authorities. But I believe these bounds
            are too narrow and inflexible today. One size does not fit all
            programs. Congress and the Department should be willing to
            consider and tailor many of the restrictions which
            unnecessarily limit and delay program managers today. I have
            seen many of our successful classified special programs benefit
            from greater management flexibility than that afforded to their
            in conventional program counterparts. The good managers of
            these special programs have used that flexibility to the
            benefit of the program and the Department by operating with
            transparency and maintaining trust. I realize that it seems
            counterintuitive to recommend greater flexibility and trust in
            an environment rife with acquisition problems, but I believe we
            need to break the current cycle. One way to begin is with a
            limited number of pilot programs, with first priority to those
            programs addressing urgent needs, and assignment of our most
            experienced program managers to meet those urgent needs. Since
            these programs will be moving with dispatch, they offer the
            best opportunity to produce early indications of whether this
            is a sound approach which should be extended to other programs.
              3. Improving funding stability will require that the
            Department and Congress be willing to give up some of their
            flexibility in making annual (or more frequent) adjustments in
            funding. Doing so will require tradeoffs of the costs and
            benefits, and I believe it is time to make explicit
            consideration of these tradeoffs. I have seen the projected
            benefits of stable funding by looking at theoretical Monte-
            Carlo simulations (which show efficiency improvements of
            perhaps 8-10 percent as a result of holding a small capital
            Reserve of less than 10 percent). We can also see the benefits
            of multiyear procurements saving similar or greater amounts. I
            have also seen many examples in which funding cuts of x dollars
            today result in later additions of 3x dollars to catch up.
              4. Giving early and serious attention to test and evaluation
            will require strengthening our test and evaluation
            organizations and personnel. Test and evaluation is often an
            afterthought, and contracts are often written without any
            mention of how we will test the product. We spend large amounts
            of money when a large development team waits for test results.
            The alternative is to spend less money and time by considering
            testing and investment in test resources as part of our systems
            engineering and development planning efforts. The actions
            recommended in paragraph 1 above are the same actions required
            to address these critical test and evaluation needs.
              5. Reducing the time from program initiation to fielding will
            require the combination of all actions suggested above. Further
            benefits will be derived by placing more emphasis on time-
            certain acquisition. This will be helped by better development
            planning and alignment of incentives. With good development
            planning, we can assign managers to develop prototypes,
            critical systems or components needed to better understand
            cost/performance trades and reduce risk. It is reasonable to
            expect that many of these developments can be completed in 2-4
            years, so one manager will be in place from start to delivery.
            This will help align authority and accountability in both
            government and industry. As these critical subsystems are
            delivered and tested, the risk reduction and domain experience
            gained in both government and industry will allow us to reduce
            the time required to develop, integrate and test the full
            system. We can also apply meaningful incentive programs to link
            profits to demonstrated performance, and use that performance
            as a factor in making future competitive awards. We can rely on
            the experience gained during development planning to apply
            informed judgment to adjust requirements to improve value,
            reduce time, and better estimate and manage costs. The
            Department and Congress can assist by placing more emphasis on
            time-certain acquisition, with the opportunity for milestone
            reviews at the completion of major development planning
            activities.

        I believe action on these five issues will have a significant and
    demonstrable impact on our serious acquisition problems. We need to
    move now with the same urgency and priority that we expect in combat
    operations to permit the timely and effective development and fielding
    of new capabilities and services with what I expect will be more
    limited future defense dollars.

        Chairman Levin. Thank you so much, Dr. Kaminski.
        Mr. Adolph?

       STATEMENT OF CHARLES E. (PETE) ADOLPH, CHAIRMAN, DEFENSE
     SCIENCE BOARD TASK FORCE ON DEVELOPMENTAL TEST AND EVALUATION

        Mr. Adolph. Chairman Levin, Senator McCain, members of the
    committee, I'd like to thank you for inviting me today.
        I chaired a recent Defense Science Board (DSB) study of
    developmental test and evaluation, and during my opening
    remarks, I'll summarize the key points from the study. I ask
    that my written testimony, which addresses the major findings
    and recommendations in more detail, be put into the record.
        Chairman Levin. It will be.
        Mr. Adolph. The task force was originally convened in 2007
    to investigate the causal factors for the high percentage of
    programs completing initial operational test and evaluation in
    recent years, which have been evaluated as not operationally
    effective and/or suitable.
        The task force was asked to assess roles and
    responsibilities for test and evaluation oversight in the
    Office of the Secretary of Defense (OSD). We were also tasked
    to recommend changes to facilitate the discovery of suitability
    problems earlier, and thus, improve the likelihood of
    operational suitability during initial operational test and
    evaluation.
        Very early in the study, it became obvious that the high
    suitability failure rates were the result of systemic changes
    that had been made to the acquisition process, and that changes
    in test and evaluation alone could not remedy poor program
    formulation and execution.
        A number of major changes in the last 15 years have had a
    significant impact on the acquisition process. First,
    congressional direction from 1996 through 1999 reduced the
    acquisition workforce, which, of course, includes developmental
    test and evaluation. In many instances, services acquisition
    organizations went well beyond the mandated cuts, some making
    up to 60 percent reductions in organizations providing
    acquisition support.
        Concurrent with acquisition reform, the general practice of
    reliability growth during development was deemphasized and, in
    most cases, eliminated. This departure from the widely
    recognized best practice may not have been a direct result of
    acquisition reform, but may instead be related to the loss of
    key personnel and experience, as well as shortsighted attempts
    to save acquisition funds at the expense of increased
    sustainment and life-cycle costs. Numerous studies have
    conclusively demonstrated that investing in system reliability
    during development will yield a substantial reduction in
    support costs.
        Our study reached the conclusion that the single most
    important step necessary to correct high suitability failure
    rates is to ensure that programs are formulated to execute a
    viable systems engineering strategy, including a robust
    reliability, availability, and maintainability program, as an
    integral part of design and development.
        Moving on to government test organizations, in the last 15
    years, with some exceptions, there's been a significant
    decrease in government involvement in test planning, conduct,
    and execution. One of our task force members observed that, in
    many instances, the government has gone from oversight to
    insight to out-of-sight. Our task force recommends that
    government test organizations reconstitute and retain a cadre
    of experienced test and evaluation personnel to perform the
    test oversight function.
        Regarding OSD roles and responsibilities for test
    oversight, the study team found that the developmental test
    office, which had existed for decades, was disestablished in
    the late 1990s. Currently there is no OSD organization with
    comprehensive developmental test oversight, responsibility,
    authority, or staff. We recommend that the office be
    reestablished as a direct report to the Deputy Under Secretary
    for Acquisition and Technology, as outlined in the proposed
    legislation.
        I'd like to make a few additional observations about the
    systemic issues that have contributed to the current problems.
        First, during a time of increased programmatic and
    technical complexity, there has been a loss of a large number
    of the most experienced management and technical personnel
    without an adequate replacement pipeline. Solutions to
    acquisition problems must begin with reconstituting a trained
    and experienced government acquisition workforce, which
    includes program managers, subject-matter experts, as well as
    systems engineers, contracts personnel, testers, and
    evaluators.
        Second, more attention must be paid to technology
    readiness, to include prototyping and testing crucial
    technologies.
        Finally, I believe that the major recommendations in the
    recent study chaired by Dr. Kaminski on pre-Milestone A systems
    engineer would, if implemented and combined with a revitalized
    acquisition workforce, go a long way towards correcting many of
    the current acquisition problems.
        Thank you.
        [The prepared statement of Mr. Adolph follows:]

                 Prepared Statement by Charles ``Pete'' Adolph

        Mr. Chairman, members of the committee, I am Pete Adolph, the
    chairman of a recent Defense Science Board (DSB) Task Force study of
    Developmental Test and Evaluation (DT&E). I am pleased to present a
    summary of the study results. The findings and recommendations I will
    discuss represent a consensus of the Task Force members and do not
    reflect an official position of the Department of Defense (DOD).
        A DSB Task Force on DT&E was convened in the summer of 2007 to
    investigate the causal factors for the high percentage of programs
    entering Initial Operational Test and Evaluation (IOT&E) in recent
    years which have been evaluated as both not operationally effective and
    not operationally suitable. The following are the specific issues which
    the Task Force was asked to assess:

             Office of the Secretary of Defense (OSD) organization,
            roles, and responsibilities for Test and Evaluation (T&E)
            oversight. Compare organization, roles, and responsibilities in
            both DT&E and OT&E. Recommend changes that may contribute to
            improved DT&E oversight, and facilitate integrated T&E.
             Changes required to establish statutory authority for
            OSD DT&E oversight. Title 10, U.S.C. has an OT&E focus, and
            does not address OSD authority in oversight of DT&E. Recommend
            changes to title 10 or other U.S. statutes that may improve OSD
            authority in DT&E oversight.
             Many IOT&E failures have been due to lack of
            operational suitability. Specific problems have been in the
            materiel readiness sustainment areas of reliability,
            maintainability, and availability. Recommend improvements in
            DT&E process to discover suitability problems earlier, and thus
            improve likelihood of operational suitability in IOT&E.

                               PROBLEM DEFINITION

       In recent years, there has been a dramatic increase in the number
    of systems not meeting suitability requirements during IOT&E.
    Reliability, Availability and Maintainability (RAM) deficiencies
    comprise the primary shortfall areas. DOD IOT&E results from 2001 to
    2006 are summarized in Figures 1 through 3. These charts graphically
    depict the high suitability failure rates during IOT&E resulting from
    RAM deficiencies.

        Early in the DSB study, it became obvious that the high suitability
    failure rates were the result of systemic changes that had been made to
    the acquisition process; and that changes in DT&E could not remedy poor
    program formulation and execution. Accordingly, the Task Force study
    was expanded to address the broader programmatic issues, as well as the
    issues previously identified.
        A number of major changes in the last 15 years have had a
    significant impact on the acquisition process. First, congressional
    direction in National Defense Authorization Acts for Fiscal Year 1996,
    1997, 1998, and 1999 reduced the acquisition workforce (which includes
    DT&E). Several changes resulted from the implementation of Acquisition
    Reform in the late 1990s. The use of commercial specifications and
    standards was encouraged, unless there was justification for the use of
    military specifications. Industry was encouraged to use commercial
    practices. Numerous military specifications and standards were
    eliminated in some Service acquisition organizations. The requirement
    for a reliability growth program during development was also
    deemphasized, and in most cases, eliminated. At the same time, systems
    became more complex, and systems-of-systems integration became more
    common. Finally, there was a loss of a large number of the most
    experienced management and technical personnel in government and
    industry without an adequate replacement pipeline because of the
    personnel cuts. The loss of personnel was compounded in many cases by
    the lack of up-to-date standards and handbooks, which had been allowed
    to atrophy, or in some cases, eliminated. It should be noted that
    Acquisition Reform included numerous beneficial initiatives. There have
    been many programs involving application of poor judgment in the last
    15 years that can be attributed to acquisition/test workforce
    inexperience and funding reductions. It is probable that these problems
    would have occurred independently of most Acquisition Reform
    initiatives.
        All Service acquisition and test organizations experienced
    significant personnel cuts, the magnitude varying from organization to
    organization. Over time, in-house DOD offices of subject matter experts
    (who specialized in multiple areas, such as promoting the use of proven
    reliability development methods) were drastically reduced, and in some
    cases, disestablished. A summary of reductions in developmental test
    personnel follows. The Army essentially eliminated their military
    Developmental Testing (DT) component and declared the conduct of DT by
    the government to be discretionary in each program. The Navy reduced
    their DT workforce by 10 percent but no shift of ``hands-on''
    government DT to industry DT occurred. The trend within the Air Force
    gave DT conduct and control to the contractor. Air Force test personnel
    have been reduced by approximately 15 percent and engineering personnel
    supporting program offices have been reduced by as much as 60 percent
    in some organizations. The reduction of Acquisition Program Office and
    Test personnel in the Services occurred during a time when programs
    have become increasingly complex (e.g., significant increases in
    software lines of code, off-board sensor data integration, and systems
    of systems testing).

                     PRINCIPAL FINDINGS AND RECOMMENDATIONS

    Reliability, Availability, and Maintainability
        As a result of industry recommendations in the early 1970s, the
    Services began a concerted effort to implement reliability growth
    testing as an integral part of the development process. This
    implementation consisted of a reliability growth process wherein a
    system is continually tested from the beginning of development,
    reliability problems are uncovered, and corrective actions are taken as
    soon as possible. The Services captured this practice in their
    reliability regulations, and DOD issued a new military standard on
    reliability, which included reliability growth and development testing
    as a best practice task. The goal of this process from 1980 until the
    mid-1990s was to achieve good reliability by focusing on reliability
    fundamentals during design and manufacturing rather than merely setting
    numerical requirements and testing for compliance towards the end of
    development.
        The general practice of reliability growth was discontinued in the
    mid- to late 1990s. This discontinuance may not be a direct result of
    Acquisition Reform, but may be related instead to the loss of key
    personnel and experience, as well as short-sighted attempts to save
    acquisition funds at the expense of increased life-cycle costs. With
    the current DOD policy, most development contracts do not include a
    robust reliability growth program. The lack of failure prevention
    during design, and the resulting low initial Mean Time Between Failure
    and low growth potential are the most significant reasons that systems
    are failing to meet their operational reliability requirements.
        According to Army studies, almost 90 percent of the sustainment
    costs are directly correlated with the reliability of the system. Given
    the amount of resources consumed during sustainment, investments in
    reliability enhancements can provide a very large return on that
    investment. A case study conducted by the Logistics Management
    Institute, provided data that indicated an investment in total program
    reliability would yield a substantial reduction in support costs.

        Findings
             Acquisition personnel reductions combined with
            acquisition system changes in the last 15 years had a
            detrimental impact on RAM practices

                     With some exceptions, the practice of
                    reliability growth methodologies was discontinued
                    during System Design and Development
                     Relevant military specifications, standards,
                    and other guidance were not used
                     Suitability criteria, including RAM, were de-
                    emphasized

             Improved RAM decreases life-cycle costs and reduces
            demand on the logistics system
             The Deficiency Report can be a valuable tool for early
            identification of RAM-related suitability problems, when used
            in conjunction with an adequately resourced deficiency
            correction system

        Recommendations
        The single most important step necessary to correct high
    suitability failure rates is to ensure programs are formulated to
    execute a viable systems engineering strategy from the beginning,
    including a robust RAM program, as an integral part of design and
    development. No amount of testing will compensate for deficiencies in
    RAM program formulation. To this end, the following RAM-related actions
    are required as a minimum:

             Identify and define RAM requirements during the Joint
            Capabilities Integration Development System, and incorporate
            them in the Request for Proposal (RFP) as a mandatory
            contractual requirement
             During source selection, evaluate the bidders'
            approaches to satisfying RAM requirements
             Ensure flow-down of RAM requirements to subcontractors
             Require development of leading indicators to ensure
            RAM requirements are met
             Make RAM, to include a robust reliability growth
            program, a mandatory contractual requirement and document
            progress as part of every major program review
             Ensure that a credible reliability assessment is
            conducted during the various stages of the technical review
            process and that reliability criteria are achievable in an
            operational environment
             Strengthen program manager accountability for RAM-
            related achievements
             Develop a military standard for RAM development and
            testing that can be readily referenced in future DOD contracts
             Ensure a adequate cadre of experienced RAM personnel
            are part of the Service acquisition and engineering office
            staffs

    Roles and Responsibilities of Government Test and Evaluation
            Organizations
        The role of the government in the DT process has evolved over the
    past 50 years. With some exceptions, there has been a significant
    decrease in government involvement in test planning, conduct and
    execution, in the last 15 years.
        The traditional role of the government during the DT planning phase
    included the identification of the test resource requirements and
    government test facilities, the development of the test strategy and
    detailed T&E plans, as well as the actual conduct of T&E. When a
    program moved from the planning phase to the test execution phase, the
    government traditionally participated in test conduct and analysis;
    performing an evaluation of the test results for the program office.
    With some exceptions, this is no longer the case. Until recently, it
    was recognized that there should be some level of government
    involvement and oversight even when the contractor has the primary
    responsibility regarding planning and execution of the DT program.
        In addition to the reduction in the number of government
    acquisition and test personnel, the experience level of both government
    and industry personnel has steadily diminished in recent years. A
    significant percentage of the workforce became eligible to retire since
    2000, and due to prior downsizing, there has not been a steady pipeline
    of younger technical personnel to replace them.

        Findings
        The changes in the last 15 years, when aggregated, have had a
    significant negative impact on DOD's ability to successfully execute
    increasingly complex acquisition programs. Major contributors include
    massive workforce reductions in acquisition and test personnel, a lack
    of up-to-date process guidance in some acquisition organizations,
    acquisition process changes, as well as the high retirement rate of the
    most experienced technical and managerial personnel in government and
    industry without an adequate replacement pipeline.

             Major personnel reductions have strained the pool of
            experienced government test personnel
             A significant amount of DT is currently performed
            without a needed degree of government involvement or oversight
            and in some cases, with limited government access to contractor
            data
        Recommendations
             As a minimum, government test organizations should
            develop and retain a cadre of experienced T&E personnel to
            perform the following functions:

                     Participate in the translation of operational
                    requirements into contract specifications, and in the
                    source selection process, including RFP preparation
                     Participate in DT&E planning including Test
                    and Evaluation Master Plan (TEMP) preparation and
                    approval
                     Participate in technical review processes
                     Participate in test conduct, data analysis,
                    and evaluation and reporting; with emphasis on analysis
                    and reporting

             Utilize red teams, where appropriate, to compensate
            for shortages in skilled, experienced T&E domain and process
            experts
             Develop programs to attract and retain government
            personnel in T&E career fields so that the government can
            properly perform its role as a contract administrator and as a
            ``smart buyer''

    Integrated Test and Evaluation
        Integrated testing is not a new concept within the DOD, but its
    importance in recent years has been highlighted, due in part to the
    growth of asymmetric threats and the adoption of net-centric warfare. A
    December 2007 OSD Test and Evaluation Policy Revisions memorandum
    reinforces the need for integrated testing. Implementation of
    integrated test concepts has been allowed to evolve on an ad-hoc basis.
    The time has come to pursue more consistency in integrated test
    planning and execution.
        Collaboration between developmental and operational testers to
    build a robust integrated test program will increase the amount of
    operationally relevant data that can be used by both communities. DT
    and Operational Test (OT) planning is separate and this inhibits
    efforts by the Services to streamline test schedules, thereby
    increasing the acquisition timeline and program test costs.
        Additionally, there is a widely held assumption by many in the OT
    community that only data from independent OT is acceptable for
    operational evaluation purposes. While not all information from DT may
    be useable by the Operational Test Agency to support IOT&E, a
    significant amount of developmental test data can be used to partially
    satisfy OT requirements. More importantly, an operational perspective
    earlier in the developmental process has often proven to be a catalyst
    to early identification and correction of problems.
        DOD policy should mandate integrated test planning and execution on
    all programs to the extent possible. To accomplish this, programs must
    establish a team made up of all relevant organizations (including
    contractors, developmental and OT&E communities) to create and manage
    the approach to incorporate integrated testing into the T&E Strategy
    and the TEMP.
        Findings
             Service acquisition programs are incorporating
            integrated testing to a limited degree through varying
            approaches
             Additional emphasis on integrated testing will result
            in greater T&E process efficiency and program cost reductions
        Recommendations
             Implement OSD and Service policy mandating integrated
            DT&E/OT&E planning and execution throughout the program

                     Require sharing and access to all appropriate
                    system-level and selected component-level test and
                    model data by government DT and OT organizations, as
                    well as the prime contractor, where appropriate
                     Integrate test events, where practical, to
                    satisfy OT and DT requirements

    Operational Test Readiness Review
        Each Service has an Operational Test Readiness Review (OTRR)
    process. Although it varies from Service to Service, the process
    generally results in in-depth reviews of readiness to undergo an IOT&E
    event.
        Findings
             A DOD Instruction requires that ``the Service
            Acquisition Executive shall evaluate and determine materiel
            system readiness for IOT&E''

                     Decision authority is frequently delegated to
                    the appropriate Program Executive Officer
                     Materiel developer is also required to furnish
                    DT&E report to the Under Secretary of Defense for
                    Acquisition, Technology and Logistics (USD[AT&L]) and
                    Director, OT&E

             Shortcomings in system performance, suitability, and
            RAM are usually identified during the OTRR
             In most cases, the operational test readiness
            certifying authority is well aware of the risk of not meeting
            OT criteria when major shortcomings exist
             Because of funding constraints, the low priority given
            to sustainment, as well as the urgency in recent years to get
            new capabilities to the Warfighter, major suitability
            shortcomings have rarely delayed the commencement of dedicated
            IOT&E
        Recommendations
             Conduct periodic operational assessments to evaluate
            progress and the potential for achieving pre-determined
            entrance criteria for operational test events
             Conduct an independent Assessment of Operational Test
            Readiness prior to the OTRR
             Include a detailed RAM template in preparation for the
            OTRR
             Require the Command Acquisition Executive to submit a
            report to OSD that provides the rationale for the readiness
            decision

    OSD Test and Evaluation Organization
        The Task Force was asked to assess OSD roles and responsibilities
    for T&E oversight. T&E has been a visible part of OSD since the early
    1970s, reporting to the Research and Engineering command section when
    it was in charge of acquisition oversight and subsequently to the Under
    Secretary of Defense for Acquisition (now AT&L). The early T&E office
    was responsible for all T&E, ranges, resources oversight, and policy.
    In 1983, Congress established an independent Director, OT&E
    organization, reporting directly to the Secretary of Defense (SECDEF),
    responsible for OT&E policy, budget review, and assessments of
    operational effectiveness and suitability. The Live Fire Test (LFT)
    oversight function was created and added to the DT&E office
    responsibilities in the mid-1980s. Later, the LFT oversight function
    was moved to the DOT&E organization.
        In 1999, the DT&E organization was disestablished. Many functions
    were moved to DOT&E, including test ranges and resources, and joint T&E
    oversight. Some of the remaining T&E personnel billets were eliminated
    to comply with a congressionally mandated (AT&L) acquisition staff
    reduction. The residual DT&E policy and oversight functions were
    separated and moved lower in the AT&L organization.
        A 2000 DSB Task Force Study on Test and Evaluation Capabilities
    recommended that DOD create a T&E resource enterprise within the office
    of the DOT&E to provide more centralized management of T&E facilities.
    This recommendation ultimately led to removing the test ranges and
    resources oversight from DOT&E, abandoning the notion of centralized
    management, and the establishment of the Test Resource Management
    Center (TRMC) in AT&L (as directed by the National Defense
    Authorization Act for Fiscal Year 2003).
        Findings
        Current policy as of December 2007 mandates that developmental and
    operational test activities be integrated and seamless throughout the
    system life cycle. There must be enough experts in OSD with the ability
    to understand and articulate lessons learned in early testing and the
    ability to execute the new T&E policy. That policy is to ``take into
    account all available and relevant data and information from
    contractors and government sources'' in order to ``maximize the
    efficiency of the T&E process and effectively integrate developmental
    and operational T&E.''

             Currently there is not an OSD organization with
            comprehensive DT oversight responsibility, authority or staff
            to coordinate with the operational test office

                     The historic DT organization has been broken
                    up and residual DT functions were moved lower in
                    organization in 1999, and lower yet in 2002
                     Programmatic DT oversight is limited by staff
                    size and often performed by generalists vice T&E
                    experts
                     Recruitment of senior field test personnel is
                    hampered by DT's organizational status
                     Existing residual organizations are fragmented
                    and lack clout to provide DT guidance
                     System performance information and DT lessons
                    learned across DOD has been lost
                     DT is not viewed as a key element in AT&L
                    system acquisition oversight
                     Documentation of DT results by OSD is minimal

             Access to models, data, and analysis results is
            restricted by current practice in acquisition contracting, and
            the lack of expertise in the DT organization
             TRMC has minimal input to program-specific questions
            or interaction with oversight organizations on specific
            programs

                     Organizational separation is an impediment
        Recommendations
             Implementation of integrated and seamless DT and OT
            will require, at a minimum, greater coordination and
            cooperation between all testing organizations
             Consolidate DT-related functions in AT&L to help
            reestablish a focused, integrated, and robust organization

                     Program oversight and policy, and Foreign
                    Comparative Test (FCT)
                     Have Director, DT&E directly report to Deputy
                    Under Secretary of Defense, Acquisition and Technology
                    (DUSD[A&T])
                     Restore TEMP approval authority to Director,
                    DT&E

             Integrate TRMC activities early into DT program
            planning

                     Make TRMC responsible for reviewing the
                    resources portion of the TEMP

             If such an organization is established and proves
            itself effective, consider as part of a future consolidation
            moving LFT back to its original DT location (this would require
            congressional action and DOT&E concurrence)

        Most of the organizational changes recommended above are within the
    purview of AT&L. The LFT change requires the concurrence of DOT&E and a
    legislative change to Title 10 because of the change in reporting
    official. All the other recommendations can be implemented within
    current DOD authority.

        Chairman Levin. Thank you so much.
        I think we'll try a 7-minute round. I'm not sure there will
    be time for a second round; I think we have a vote scheduled
    around 11:30 a.m., if I'm not mistaken.
        Let me start with this question. Mr. Sullivan, you
    commented on the reform bill, the Weapon Systems Acquisition
    Reform Act of 2009, which I've introduced with Senator McCain.
    Each of you, in a way, has commented on it. Let me start with
    you, Dr. Gansler. Would you comment on the bill, any parts you
    like, any parts you don't like, any additions you might be able
    to recommend at this time?
        Dr. Gansler. Overall, Mr. Chairman, I thought it was in the
    right direction and important. I think that we have to
    recognize that simply writing a memo, passing a law, doesn't
    change the system. Each of the areas that you highlighted, I
    think are in the right direction. What we have to be careful of
    is not going too far in one direction. For example, in the
    conflict of interest, clearly we have to avoid conflict of
    interest, but we don't have to go so far that we have only
    people who have no experience in those positions. That's the
    danger of going too far, in terms of the legislation of it.
        In each of the areas, I think that there's some clarity
    that could be added, but, in general, I think you've gone in
    the right direction. For example, you emphasize systems
    engineering. I think you need to define that as including costs
    so that there's no ambiguity as to whether that's a pure
    engineering problem or a cost issue, from a design perspective.
        I think the importance of test and evaluation that you
    highlight is clearly something that's very important. It could
    be emphasized without really changing the title of the office.
    But, nonetheless, when I was Under Secretary, I felt it was a
    very critical piece. But, it's important, when you talk about
    test and evaluation, that it be viewed as a part of a
    development process. We learn from that testing. We do it
    early, as Mr. Adolph indicated, but we also don't view it as a
    pass-fail final exam because we're doing spiral development,
    and we're continuing to learn from the testing as it goes
    along, and some people have tended to think of the test process
    as a final exam.
        In the area of independent cost analysis, it is absolutely
    essential. On the other hand, we have the Cost Analysis
    Improvement Group (CAIG) office, which I used all the time, and
    I felt it was critically important. The problem is, people
    don't want to use their numbers. If they say it's realistically
    going to cost more, and government and industry want to put in
    the low bid in order to get a program into the overall budget,
    that's a management question; it's not a matter of what
    organization you set up. But, it's a very important function,
    as you highlighted.
        In terms of getting the combatant commanders involved in
    the requirements process, that was exactly what we intended
    with the Packard Commission and Goldwater-Nichols intended with
    the establishment of the Vice Chairman of the Joint Chiefs.
    That was the purpose of that statement. Somehow that hasn't
    gotten the intended strength, it has been much more the
    suppliers than the users, if you will, the warfighters. I think
    it is important to get the combatant commanders much more into
    the loop on the requirements process, and I think you've gotten
    that properly emphasized.
        I think, in the acquisition area I would emphasize that
    projected unit costs are a military requirement because that's
    one of the things we've lost in some of the programs. Global
    Hawk started out that way, got off track. You mentioned the
    JSF. That started off as a unit cost being one of the
    requirements for globalization of that program. It got out off
    track because it lost sight of that unit cost as being one of
    its principal considerations in design.
        As I mentioned, in terms of competition, I think it's
    really important that we view competition as an option
    throughout the programs, at the prime and at the lower tiers,
    but not as a law. You don't compete for its own sake; you
    compete when the current contractors, prime or sub, aren't
    getting better performance at lower cost. That's their
    incentive for doing it. If you tell them we're going to compete
    it anyhow, they have no incentive. If you tell them that if
    they can get higher performance and lower cost, then, in fact,
    they will try to achieve that so that they don't have to
    compete in the next round, and it's doing exactly what the
    government wants them to do.
        In the same way, if they achieve the objectives, we ought
    to use, in effect, the same way that the commercial world does:
    through price elasticity. If you get a lower cost, we'll buy
    more of them. We don't take the money away and put it in the
    general treasury, so there will be a need to create incentives
    for industry, as well as government, for doing a better job of
    achieving higher performance at lower cost, continuously.
        In terms of conflict of interest, I think we need to focus
    on some structural ways to address conflict of interest. We do
    this, in terms of foreign ownership, through limited liability
    corporations, in effect, the special boards are set up. Maybe
    there's some ways we can do that in order to address conflict
    of interest without the sort of blanket requirement that
    someone in an engineering job shouldn't know anything about
    that job. That's wrong.
        Those are the suggestions that I would have. But, overall,
    I think you are definitely going in the right direction with
    the bill.
        Chairman Levin. Thank you. We would welcome any specific
    language changes you would recommend.
        Now, Mr. Sullivan, you already commented. Do you have any
    additional comments?
        Mr. Sullivan. Yes, just briefly. I would say that, given
    the package that we've looked at, we support everything in it.
    We thought it was very well-targeted to the key problems on
    acquisition programs, and that the idea to give more authority
    to the combatant commanders, in terms of getting urgent needs
    met, was a good provision.
        I think the most important thing on an acquisition program
    is, at the outset--we've all discussed it here--to have more
    knowledge about the requirements that you're going to build to,
    before you begin, than they have now.
        So, the two provisions that we think are most important--
    first is the cost-estimating provision. We wrote that we don't
    see any reason why the CAIG actually couldn't fulfill that
    position. The important thing there is to probably provide the
    CAIG with more resources so they can do their jobs on a more
    regular basis and rather than just periodically, at every
    milestone. The other critical thing there is that if the
    director was moved up, out of the bureaucracy a little bit
    more, the estimates that the CAIG make might be looked at as
    less personality-driven, if you will, depending upon who's in
    office.
        As Under Secretary of Defense, Dr. Gansler used the CAIG,
    others may not use it as much. If they're reporting to a higher
    level, and they owe Congress a report, I think that'll really
    improve their authority and their visibility and their
    independence. But, the key is always going to go back to the
    requirement-setting process, and jelling that with cost
    estimates. If you begin with not enough information about what
    it is you're going to, what you want to build, you are not
    going to get a good cost estimate. So, it really has to be
    based on knowledge.
        So, the cost-estimating provision, I think, is very good.
    The systems-engineering provisions that you have in there go a
    long way to providing the knowledge that the cost estimators
    would need up front.
        Chairman Levin. Thank you.
        Dr. Kaminski, do you have any comment?
        Dr. Kaminski. Yes, sir, I do. I made some specific comments
    in my statement, but let me make a couple of big-picture
    comments, if I may.
        If there's a direction in which I would try to move
    language in the proposed legislation, it would be to focus, not
    only on process, on oversight mechanisms, but a focus on
    people; people that make this system work. I don't care how
    good a process you put in place, if you don't have people who
    are experienced and know what they're doing, I think you're
    going to end up with problems.
        When I say ``people,'' I am referring very broadly to our
    acquisition workforce. Think about this for a minute. These
    people and requirements development skills are also key for
    those who are doing development of requirements. They have to
    participate in this tradeoff process to consider the cost of
    what they're asking for and how those tradeoff with the
    performance capabilities that are desired.
        Testing is a critical piece of this process. If you think
    about it--I know my experience is, there isn't any program I've
    ever worked on in which I didn't know a heck of a lot more
    about the program 6 months or a year into it than I did when I
    started. So this is a continuing learning process as we find
    out things from testing--what's hard to do, what's easier to
    do. We need to have a continuing dialogue in this requirements
    tradeoff loop.
        So, requirements developers and acquisition personnel have
    to have training and experience in these systems engineering
    tools and techniques. We would never let a fighter pilot get
    into an aircraft without a very extensive training program to
    prepare him or her for that operation. It's one of the reasons
    we do so well with our forces. When I compare our requirements
    for operational requirements with the training and education
    requirements of people going into acquisition requirements
    generation, they pale in comparison. We have to be able to
    develop the training, education, and the domain experience that
    go with this to make it work.
        I also believe it is worth it for us to look very hard at:
    How can we systematically reduce this long development time?
    That's just killing us. There are programs in which we can do
    time-certain acquisition. Time is money, here. By doing good
    development planning to be able to reduce the risk, following
    the example of Apollo, following the example of our ICBM
    programs, I believe we can compress that time. But, we need
    some targets, and some incentives to do that.
        We also need to make better and more extensive use of
    prototypes, in a sensible way. Competitive prototypes for some
    programs, perhaps single-thread programs for others. We also
    need to realize that one size does not fit all, here. For
    example, in dealing with IEDs, we find that we need to have an
    acquisition system whose cycle time is measured in weeks.
    That's a different acquisition system than you need for a
    strategic bomber. The acquisition system has to fit the cycle
    of the threat that we're dealing with, so we have to tailor it
    in that way.
        The last comment I would make is that, with respect to
    implementation, what better place could we find to start to
    implement some of these processes, procedures, and people
    development than in these urgent programs that are doing rapid
    acquisition. Why start there? One, it's urgent. Two, we'll be
    able to see the impact of changes that we make more quickly in
    programs that are operating in cycle times of weeks or months,
    and see what's benefiting us and what is not. I'd like to see
    that commitment to implementation because it'll happen much
    more quickly than if we simply write new processes or new 5000
    series in the DOD.
        Chairman Levin. Thank you.
        My time's way up, but, Mr. Adolph, do you have a brief
    comment on the bill? Any changes?
        Mr. Adolph. Very brief comment. In general, I believe that
    the proposed legislation will go a long way toward alleviating
    the major acquisition problems which have occurred in recent
    years.
        I also have a quick comment regarding the legislation
    pertaining to the director of the developmental test function.
    I would add one responsibility, that the director participate
    in the acquisition program reviews conducted by the Under
    Secretary for Acquisition and submit a status of the
    developmental testing for the programs under review. That was
    the norm in the past, for many years, and I think that's
    important.
        Chairman Levin. Thank you so much.
        Senator McCain.
        Senator McCain. Thank you, Mr. Chairman. Again, thanks, to
    the witnesses.
        I think there's general agreement on the part of the
    witnesses that there's been a dramatic erosion in acquisition
    workforce and test and evaluation personnel. Maybe I could ask
    you, Dr. Kaminski, what happened?
        Dr. Kaminski. I think a number of things happened. Just as
    there's no one silver bullet, there's no single entity or
    action to blame. But, let me list some of the things that
    happened. I think, as we looked at major programs of
    acquisition reform, we tried to do more with less. We also had
    some pretty strong direction from some portions of Congress on
    this. I can recall the chairman of a key committee who publicly
    made statements that said we had too many ``shoppers'' in the
    Department, and we really needed to reduce this acquisition
    workforce. So, over a period of 3 or 4 years, nearly 50 percent
    of the acquisition workforce was taken out. Now, the Department
    was a partner in that, agreeing to those reductions.
        Senator McCain. We removed incentives for people to remain
    in the acquisition workforce, in the form of lack of promotion
    or career enhancement?
        Dr. Kaminski. Exactly. If we are able to attract and train
    the right system engineers--if that system engineer doesn't see
    some path for advancement in the DOD, he or she is going to go
    find a place where they can make a difference and have an
    opportunity for advancement.
        Senator McCain. Which means a revolving door evolves?
        Dr. Kaminski. Yes, sir. But, that revolving door actually
    may be the commercial industry. Again, I return to the comment
    I was making about the importance of cycle time in the
    acquisition system. If you have someone who's really worth
    their salt and able to make contributions, and they get into a
    DOD acquisition system which is going to produce something in
    15 or 20 years, it won't take them long to realize that their
    knowledge base is going to erode so that they will no longer be
    valuable to commercial industry, which is producing things in
    2- or 3-year timeframes.
        So, to be able to have some revolving door from commercial
    industry back to DOD will benefit from getting these cycle
    times down.
        Senator McCain. Dr. Gansler?
        Dr. Gansler. Yes, Senator McCain. In my prepared testimony,
    I actually showed a graph that came out of the commission that
    I ran. I was shocked at how much the acquisition workforce has
    been allowed to deteriorate. In fact, as we came out of the
    Cold War, the procurement budgets dropped, the number of
    acquisition people came out with it. Then, as Dr. Kaminski
    said, there was a legislative mandate in 1996 to take another
    25 percent out. So we've ended up, now, dramatically, where we
    had about 500,000 people in 1990, we now have about 200,000
    people. But, the acquisition dollars have gone up dramatically,
    so you have this huge gap between the dollars and the people.
        But, much more important is the point that you just made
    about the officers and the senior people. In 1990, the Army had
    five general officers; in 2007, when we did the study, there
    were zero general officers with a contracting background. In
    the contract management organization, the Defense Contract
    Management Agency (DCMA), they had 25,000 people in 1990, they
    have 10,000 today. There's basically an undervaluing. They used
    to have four general officers; they have zero. So, as you
    suggest, if you're a young major, you're not getting into that
    career field, and, as a result, it's been just totally
    undervalued. Without those experienced senior people, both
    civilian and military, they don't know what questions to ask,
    and they are not going to be able to make the right judgments.
        Senator McCain. Obviously, we need to have some personnel
    policy changes, as well. Mr. Sullivan, the issue of Nunn-
    McCurdy--when it was first passed, we thought it was really
    important and effective. For a while, it was. I think a breach
    of Nunn-McCurdy was a big deal. Now it seems to be a routine
    kind of event that the notification comes over, we see it, and
    ho-hum. Are we in danger of experiencing the same thing with
    this measure?
        Mr. Sullivan. We have noticed that the Nunn-McCurdy
    breaches we see take a lot longer to resolve and come out with
    a new program, and the funding is continuing on that program as
    they do that.
        As I read the proposal that the committee has now, I think
    it's really good to have a termination criteria like that. I
    think GAO thinks that's a good thing. In other words, a program
    termination would not just have someone do a review, but also
    probably look at what happens, what the triggers are for that.
    I think, right now, it states that the program cannot change
    the scope of work, cannot start new contracts, and there might
    be one other thing that the program can do. I don't know if
    legislation can basically say that that program can no longer
    obligate money, but that would, to me, be a much more direct
    way to get the point across. If a program is automatically
    terminated when it passes a threshold, and cannot obligate
    money, that might get people's attention.
        Senator McCain. I can imagine the blowback when some vital
    program is shut down because of our failure to act, but I think
    we're in agreement that there has to be a more robust oversight
    and ability to exercise that oversight as we see these costs
    spiral completely out of control.
        Mr. Sullivan. Yes, sir.
        Senator McCain. Mr. Adolph, on the issue of Nunn-McCurdy,
    do you think it's sufficient measure to impose more discipline
    on the cost overruns?
        Mr. Adolph. I believe it's sufficient. But, the issue that
    concerns me, and the remedy has already been discussed, is to
    get the technology readiness right, at the outset; and second,
    to get a realistic cost estimate. There's too much concurrency,
    there are actions that are taken in the program to kick the can
    down the road on Nunn-McCurdy, which, in the long run, in my
    opinion, adversely impact the program. The example from the F-
    35 is getting rid of two of the test articles.
        Again, I believe Nunn-McCurdy is fundamentally sound. The
    problems in recent years are a combination of issues which I
    believe the proposed legislation, if it's really implemented in
    the Services--and, again, back to Dr. Kaminski and Dr.
    Gansler's point--in order to implement it, we have to
    reconstitute the acquisition workforce. That's the first step.
        Senator McCain. Also I think it would be important for us
    to have the combatant commanders more involved in the
    requirements process, as well. I think sometimes we have
    neglected that aspect of the equation.
        I thank you, Mr. Chairman. I thank the witnesses.
        Chairman Levin. Thank you, Senator McCain.
        Senator Akaka.
        Senator Akaka. Thank you very much, Mr. Chairman.
        I want to add my welcome to the witnesses before us today,
    and to point out that I'm glad that, Mr. Chairman, you are
    holding this hearing on acquisition. For me over the years, I'm
    beginning to feel that we need to change, or try to change, the
    heart and soul of DOD, and, really, the culture of DOD, and to
    get to acquisition.
        Gentlemen, as you well know, the reform of DOD acquisition
    process is an extraordinarily complex undertaking involving
    many, many moving parts. However, I believe the first step to
    tackling any problem is to prioritize. So, let me ask this
    question to each of you. In order to most effectively reform
    the DOD acquisition process, what do we need to focus on first?
        First, Dr. Gansler.
        Dr. Gansler. I believe it's the people. I think if we've
    undervalued the importance of this area, in terms of promotion,
    in terms of experience, in terms of numbers, all across the
    board, both civilians and military, that we're not going to get
    there, even if we pass all the laws in the world. We need the
    people who are going to be driving this process. That is my
    number-one priority. We have neglected it and, of course, in
    the last 8 years we've been living in a rich man's world, so
    money doesn't matter, and if people overrun or they don't
    perform, ``Let's spend more money.'' Now, that's not going to
    be the case, and we need people who are smart, experienced, and
    competent to run their programs--with flexibility, though. They
    have to make tradeoffs of cost and performance, systems
    engineering kinds of work, test and evaluation, so forth. That
    requires management judgment, and you can't just legislate
    that, and therefore, you need people with experience to be able
    to make those management judgments. That's my number one
    priority.
        Senator Akaka. Dr. Gansler, would you also say that another
    part of that would be inadequate staffing?
        Dr. Gansler. Yes.
        Senator Akaka. From what I gathered, there were positions
    that were not filled.
        Dr. Gansler. Absolutely. I was shocked to find that, in
    Iraq and Afghanistan, only 35 percent of the people that were
    in their jobs were qualified for those jobs, even with the
    minimal qualifications that Dr. Kaminski mentioned. Besides
    that, most of the positions weren't even filled, and they were
    almost all volunteer civilians in the warzone. We need to be
    able to get some senior military there, as well. So, there's a
    great lack of people, numbers, but you don't want just numbers,
    you also want qualified people. Numbers won't do it. It has to
    be qualified, experienced people. Some of those by the way, can
    come from industry. You can rotate people from industry,
    without conflict of interest, very easily. That's what we've
    had with many of the people with past experience. We do that in
    DARPA, we bring them in and out, and we've done it in other
    parts of the government. I think that people are out there with
    experience; we just have to make sure that we make it
    attractive to get them in these government acquisition
    positions.
        Senator Akaka. Mr. Sullivan.
        Mr. Sullivan. I think, at a strategic level, one of the
    biggest problems the DOD faces is that it is unable to
    prioritize what weapon system capabilities should be put into
    programs. They have 95 weapon system acquisition programs,
    major acquisitions, that are underway right now. That's up from
    about 75 or so in 2000.
        There is a tendency to have too many programs vying for the
    acquisition dollars that are available to DOD on an annual
    basis. When that happens, you get a very unhealthy competition,
    where many programs and few dollars drive the requirements-
    setting process, which is stovepiped in many ways by the
    Services.
        So, the Services are all vying for a solution, they want
    requirements that are very, very tough to make, so that their
    weapon system can do the most. They, as a result of that, tend
    to put in optimistic cost estimates. The funders, of course,
    are looking at those very optimistic cost estimates that are
    very heavy on assumption and very light on facts, data, and
    actual costs. The acquisition process begins with the lack of
    the systems engineering that we've talked about here today.
        So, you have too many programs chasing too few dollars,
    with business cases that are unexecutable. The whole system is
    segregated in such a way with process owners and stakeholders
    that, in a way, it works for everyone. That's the culture. It's
    a performance-driven culture, and we all understand that and
    accept that, but there's also a lot of players involved in the
    culture that create this kind of unhealthy competition at the
    outset. I think that's the culture change that has to take
    place in this DOD, and that is almost intractable, when you
    think about it: the difficulty of changing that. I think
    legislation can go a long way to requiring people to do certain
    things, but, in the end, I agree with Dr. Gansler, it falls on
    the people to have the right principles, if you will, to change
    what this system is really supposed to create.
        Senator Akaka. Dr. Kaminski, with your research and study
    background, do you care to comment?
        Dr. Kaminski. Yes. Let me not repeat because I agree with
    everything that's been said. Let me put another dimension on
    the people. I, too, would answer: most important is people.
    But, what you have to do is recognize the dynamics that involve
    people. If we're going to attract our very best and brightest
    people to an activity, I've found that the principal incentive
    usually isn't money. With the salaries we pay military officers
    and civilians in DOD, we are still able to attract very capable
    people to key jobs.
        What attracts them? What usually attracts them is the
    ability to make a difference, to see that they can have a
    personal impact on a major program, on the security of this
    Nation.
        So, I want to come back again to my issue about time. When
    programs are taking 15 or 20 years, many of those best and
    brightest people say, ``What's the difference? If I'm not going
    to see something happen for 15 or 20 years, why don't I go
    someplace where I can make something happen sooner?'' If they
    don't have any freedom to make decisions and influence things
    because of excessive oversight processes and complexity of the
    process, they'll go find another place to work.
        I want to share with you a perspective that I got from a
    different position. Rather than a perspective of a previous
    Under Secretary of Defense, I want to share the perspective
    that I gained when I served on Active Duty in the Air Force.
    I'd say I spent two-thirds of my career there working on
    special access programs, part of that career, in the early days
    of the National Reconnaissance Office, where I was a program
    manager for one of our National Reconnaissance spacecraft
    that's up and flying today, and for several years in the
    stealth  program.  Let  me  just  pick  one  example  there,
    the  F-117 program. I was heavily involved in that program.
    When we initiated that program, our plan was, from the
    beginning of full-scale development to field the aircraft in 3
    years. We missed that, sir. We missed it by a year. We fielded
    the aircraft in 4 years. Everybody who worked on that program
    could see that they were making a difference. They could see
    that capability coming along to be fielded. There was excellent
    interation and tradeoffs in that process, between the testers
    and the program managers and the users who were going to use
    that aircraft. We made continual adjustments. So, that
    motivation of people and reduced time go together. To attract
    the right kind of people, we have to work on this time issue.
        Senator Akaka. Mr. Adolph?
        Mr. Adolph. As I mentioned in my earlier remarks, people
    are certainly the highest priority, and that's been reiterated
    here by the previous comments.
        Another area for improvement is certainly overly ambitious
    requirements. We need to continue to push technology, without a
    doubt. But, at the same time, we need to ensure that the
    technology is sufficiently matured to incorporate in a weapon
    system. That means prototyping and testing, and testing the
    prototype item in an environment in which it's going to be
    placed in a combat environment. So, again, first, people.
    Second, getting the requirements right and making sure that
    they're not overly ambitious. Again, Dr. Kaminski's study
    really addresses the latter issue quite well.
        Senator Akaka. Thank you very much, Mr. Chairman.
        Chairman Levin. Thank you, Senator Akaka.
        Senator Collins.
        Senator Collins. Thank you. Thank you, Mr. Chairman.
        Mr. Chairman, I first want to salute you and Senator McCain
    for your initiative. I think you've brought forth a bill that
    will really make a difference. I also want to point out that
    I'm very pleased to hear the witnesses today all stress the
    importance of the acquisition workforce. This is an area that
    Senator Lieberman, Senator Akaka, and many of us, have said is
    the number-one problem, over and over again. When I brought
    this up at the White House Conference last week, however, some
    of my colleagues felt that it was a lesser problem. So, I was
    very pleased to see the panel of experts before us list this as
    perhaps the greatest problem that we're facing.
        Dr. Gansler, you and I also worked together, many, many
    years ago, on how to increase competition in Federal
    contracting, and it's very good to see you here again, as well.
        I want to ask the panel about some of the specific
    provisions in the Levin-McCain bill. In particular, this bill
    would require that costs be considered right up front, when the
    requirements are set. That is a pretty dramatic change from how
    military requirements are set now, when they're done in an
    environment that does not consider costs, but, rather, an
    idealized world, where costs would not be a factor.
        Dr. Gansler, you endorsed including cost as a design
    military requirement right up front, so I'm going to skip over
    you for this question and go to the rest of the panel and ask
    all of you: Should costs be considered up front when military
    requirements are first established?
        Mr. Sullivan?
        Mr. Sullivan. I think that is a good idea. Dr. Gansler said
    that should be one of the key performance parameters on any
    major weapon systems, in their business case, and I think we
    would agree with that.
        When you're trying to set requirements, and you're doing
    the requirements analysis that's needed, I think you begin, at
    first, in an unconstrained manner, and try to get from the user
    what the user would like, in an unconstrained environment. But
    then it's critical, at some point, to start bonding that with
    the realities of the time it's going to take to get that to the
    user, the amount of money, and the technologies you have
    available to do it. So, I don't think you could do that without
    precluding the ability to think, unconstrained. But the exit
    criteria would be something that is constrained, at least in a
    cost range. Then once it exits the requirements process, there
    is a stage before it would become a set business case and begin
    as an acquisition program, where that cost range could be
    further reduced to more of a point estimate by continuing to
    make trades.
        Senator Collins. Dr. Kaminski?
        Dr. Kaminski. Senator Collins, I also agree that cost
    should be an upfront factor. I'd add another factor to go with
    it, though.
        Senator Collins. Yes?
        Dr. Kaminski. Back to my comment about time. Time and money
    go together. When we're doing requirements tradeoffs, if we're
    going to have an acquisition cycle time that operates within
    the threat cycle time, those developing requirements have to
    look at the time they want something fielded, as well. So, that
    needs to be an important consideration in this process, and we
    need to manage the time.
        The one other comment I wanted to make with respect to cost
    estimates: we talked earlier about breaches in Nunn-McCurdy.
    The surveillance system that finds the breaches, I think, is
    fine. But, one of the things we want to do is look at root
    causes. What's causing us to get into Nunn-McCurdy? One of the
    constructive uses for the independent cost estimate that was in
    the bill might be to add another consideration. We've talked
    about the importance of the upfront work in systems engineering
    and development planning. One of the things that would be
    useful for us all to ensure is that there is adequate funding
    at the beginning of a program, between Milestone A and B, for
    us to apply our systems engineering and development planning
    capabilities to get a good handle on what those cost estimates
    are, and do a thorough job involving stakeholders, the
    requirements part of the equation, and the program manager in
    that process, along with the CAIG.
        I, like Dr. Gansler, used the CAIG very heavily. In fact, I
    advocated that we fund programs to the CAIG estimate, but we
    hold the program manager and the contractor to the estimates
    they developed, so we had some finance reserve between the two
    estimates.
        Senator Collins. Thank you.
        Mr. Adolph?
        Mr. Adolph. Senator Collins, certainly we need to consider
    the cost issue upfront. Another very important driver early on
    is technology readiness. In program after program after
    program, we're into full-scale development and discover that
    some of the critical technologies simply aren't mature enough.
    So the program is delayed, and that drives the schedule, and
    these slippages occur.
        A key to getting the costs right is to ensure that the
    technology is really sufficiently mature.
        Senator Collins. Thank you.
        Dr. Gansler, an issue that hasn't been discussed today is
    the impact of a lack of stability and predictability on a
    program's cost growth. That obviously can drive up the unit
    cost. When the military starts out with one plan for acquiring
    a weapon system, and then switches direction or reduces the
    number of units, doesn't that also drive up the cost?
        The reason I bring this up is, we talk a lot about the
    errors made by contractors, we talk about weapon systems that
    get gold-plated because additional requirements are added, but
    there's also an important issue, as far as the stability of
    funding and the lack of predictability driving up the unit
    cost. Could you address that issue?
        Dr. Gansler. Yes. But, let me just briefly comment on your
    first question, though, because frequently the military doesn't
    think that cost is a military requirement. What they neglect is
    the fact that numbers are a military requirement. If you're
    resource-constrained, the total dollars that you have divided
    by the unit cost gives you the numbers. Numbers really matter
    in military operations, whether it's by Lancaster's law of N-
    squared or by numbers. Either way, numbers really matter, and
    therefore, cost really matters. That's why it's so important to
    have the unit cost as part of the requirements.
        Now you get to your changes, and unless we estimate the
    cost of those changes and their impact on the ultimate cost of
    the equipment, we let things get out of hand. So, if cost is a
    requirement, then every time a change comes in, and as Senator
    McCain said earlier, it was 75 a week on the LCS, you have to
    price those each out to make sure it is not having a big cost
    impact on the program.
        Then, when programs, in general, become relatively stable,
    you don't get this ripple effect through the budget, which is
    the point that you're really making, Senator. If I want to pay
    for program A, I take it out of program B, not recognizing that
    program B now is in really bad shape because they don't have
    the stability of the funding. That stability of funding is a
    critical issue.
        Senator Collins. Thank you.
        Chairman Levin. Thank you, Senator Collins.
        Senator Burris.
        Senator Burris. Thank you, Mr. Chairman.
        This is an interesting experience for me, to hear the
    distinguished testimony, Mr. Chairman. I would rather listen
    than to talk. I yield my time.
        Chairman Levin. Thank you.
        Senator Burr is next on this side; he's not there.
        Senator Chambliss.
        Senator Chambliss. Thank you, Mr. Chairman.
        Gentlemen, it's good to see all of you, and thanks for your
    service over the years, and thanks for your being here on what
    I think is, if not the most important issue at the Pentagon
    today, certainly it ranks up pretty high. With these budget
    times that we're in, trying to figure out a way to buy the
    weapon systems that we need within the timeframe we need is
    virtually impossible. That's why your testimony is so critical.
        I want to go back to what Senator Collins was talking about
    there, and what all of you have alluded to in some part, and
    that's this issue of instability, whether it's requirements,
    whether it's personnel, whether it's funding, or whether, as
    you say, Mr. Adolph, it's the technology aspect of it, where
    we, too many times, tend to come up with a great idea, and by
    the time we get into the production phase, we've wasted, not
    only time, we've wasted money, but oftentimes, there's a new
    idea that has been developed in the meantime.
        I'm amazed, Dr. Kaminski, that you say, on the F-117, that
    we were in production in 4 years. Gosh, if we had done that
    with the F-22, we wouldn't be having the arguments we're having
    today, and we'd have a great airplane, and we'd be worrying
    about other issues.
        But, it doesn't make any difference whether you're talking
    about an aircraft carrier, whether you're talking about FCS or
    a tactical fighter, we come up with this idea, and we get into
    the R&D phase, and there is, all of a sudden, a great idea, but
    instability in all four of those areas runs that cost up
    tremendously. Then, you throw in what Senator Collins alluded
    to, about the number of these units that we're going to buy,
    and all of a sudden it explodes again and it becomes such a
    negative at the Pentagon, rather than the positive that it
    started out to be.
        My question for a comment from each of you is: How do we
    get back to this? How do we get back to the point to where we
    come up with this idea? If it's a tactical fighter, it's
    supposed to be air-to-air, or supposed to be air-to-ground,
    whatever it may be, how do we develop that and get it into
    production right away, without technology intervening and all
    of a sudden having to add this and add that? How do we get our
    arms around that issue?
        Jack, let's start with you.
        Dr. Gansler. I think one of the main opportunities we have
    is to accept the concept of spiral development, that for the
    block 1 system, we have a fixed set of requirements, we have a
    fixed price that we're trying to get a fixed schedule, as Dr.
    Kaminski said, and we go ahead with block 1 under the
    assumption that if we can then demonstrate new technology, if
    we find that the user needs something different, if we find
    that even the logistician has a problem with maintenance of
    that equipment or the reliability as we deploy it, that becomes
    block 2, block 3, block 4. But, block 1 has to use proven
    technology and get out there quickly, all with a set of
    constraints. So, it's a stable program, as you point out.
        The most successful acquisition that, in fact, Congressman
    Aspin used to always highlight, was the Navy's Polaris,
    Poseidon, Trident. When I was in industry, I always knew how
    much money I was going to get next year for that program. I
    could hire, I could plan my workforce, and so forth. That
    stability is very important for efficiency. I think, if we go
    to a spiral development model, whereby block 1 is stable, and
    block 2 is being developed while block 1 is being deployed, you
    have the concept of stability built in, and evolutionary
    systems are still stable. That's the way the real world, the
    commercial world, works. You constantly are upgrading the
    software, the hardware of computers, but you're constantly
    getting higher performance at lower cost. That has to the be
    objective of each of the blocks as we're going along in spiral
    development.
        Senator Chambliss. Mike?
        Mr. Sullivan. Goldwater-Nichols legislation, from years
    ago, tried to bring jointness into the DOD. I think it
    succeeded on the operations side. We now have the combatant
    command's matrix, the military forces that fight wars jointly
    very well. The same thing did not occur on the acquisition
    side. I think if you look at what's going on now, there's a
    kind of a stovepipe system for how you get programs started,
    and that creates this kind of competition for big requirements
    and cost estimates that are heavy on assumption.
        I would agree with everything that Dr. Gansler just said.
    If you can work on that and get a more joint requirement
    setting and funding system, and try to get the proper balance
    of weapon systems started, try to get rid of the stovepipes,
    you'd have an environment that could do what Dr. Gansler, I
    think, is describing, a little more easily.
        Another program I would throw out is the F-16 program. Back
    in the 1970s, it was a block program. It was the capabilities
    that the Air Force wanted for the F-16. They knew that they
    needed an aircraft faster than they could develop the
    technologies to get those capabilities, so they had blocks. If
    you look at the F-16s, and, for that matter, the F-15s,
    performance over the last 30 years, it's pretty impressive.
    They basically upgraded those aircraft pretty efficiently as
    they went because they started without that big revolutionary
    leap, that one-step, big-bang kind of a thing.
        So, we've done this before, and I think it's possible to
    get back to it, but this is where the culture comes in. I think
    there's some culture change that needs to take place.
        Dr. Gansler has other things that I read in his report that
    would help this significantly. Open systems, for example, on
    these weapon systems, when, if you can make interfaces on the
    weapon systems uniform, you can keep proprietary data that
    subcontractors have that supply subsystems to them, and all
    they have to do is have the proper interface. Then you can open
    up competition.
        One more point I would make is the difference between
    technology development and product development probably needs
    to be better understood. Technology is the kind of thing you
    should think of when you think of scientists and lab coats and
    trial and error, and it's done in a smaller-dollar environment,
    where you can test, use trial and error, and make mistakes. You
    have to keep that off of these acquisition programs. I think
    someone up here, I don't know if it was Dr. Kaminski or Mr.
    Adolph, said that when you have a technology that's not mature,
    and it's on an acquisition program that's driving towards
    production, you have an entire workforce--an entire supply
    chain, for that matter--that's waiting for that technology to
    mature. The burn rate is pretty big on that workforce that you
    have.
        Dr. Kaminski. I'd emphasize that point. While everybody's
    waiting, we're paying, and so what you want to do is decouple
    those two.
        In terms of approaches to do this, I agree completely with
    the spiral development approach. One problem I see is in the
    application of spiral. In the few spirals I've seen that we
    actually implemented, we had everything but the kitchen sink in
    the first spiral, rather than stretching this out over a period
    of time, like we did in Apollo. So, we need development
    planning. You have to have a plan with stability to do this.
    The program manager has to have the discipline and the
    experience to reject things that aren't in the plan or that
    aren't mature enough to be harvested.
        I have found that one of the characteristics of a good
    program manager is a big lower right-hand drawer in their desk,
    and what went into that drawer were all the ideas for
    improvement. The drawer was kept closed until such time as we
    fielded the first system; that is the time open the drawer and
    look to see what development plans we need to deal with
    shortfalls or upgrades for that system.
        This time-certain development is important because, as time
    goes by, the technology gets old, and new ideas are introduced
    that end up being disruptive to the process. So, time is a key
    factor here.
        This stability issue is really key. If I look back through
    my whole career, there is only one program I ever worked on
    where we actually produced the system at the rate we planned.
    That was the F-117. We built one a month. Every other program I
    can think of, by the time we were done with development, we
    couldn't afford the build rate that we planned.
        Senator Chambliss. Anything to add, Mr. Adolph?
        Mr. Adolph. Just one thing, since I certainly agree with
    the other panel members, but with my background in tests, I'll
    add that issue. In the case of the F-15 and F-16, which were
    mentioned earlier, I worked on both programs, and I was out in
    the field, test business, working for the Air Force.
        Chairman Levin. Please talk a little bit louder, if you
    would.
        Mr. Adolph. Yes. In the case of the F-15 and F-16, I was
    working for the Air Force in the field at the time, and the F-
    15 test program was structured well; we had 19 test articles
    and sufficient articles in the case of the F-16 as well. In the
    latter F-16 case, the propulsion system had been matured. In
    order to move a program along, you need an adequate number of
    test articles, and you need to be able to do as much testing as
    you can. Avionics is a good example, where considerable testing
    can be accomplished on a test bed platform, rather than the
    developmental platform.
        So, that's, I think, one key component of keeping a program
    moving. When a program stagnates, when you only have one or two
    test articles, and you have a `standing army' of test personnel
    waiting, and the fixed cost of those people is almost as great
    as the incremental cost of doing additional testing,
    particularly when you recover the article. That's not the case
    in a missile program, where the test article is destroyed on
    each test.
        Chairman Levin. Thank you, Senator Chambliss.
        Senator McCaskill.
        Senator McCaskill. Thank you, Mr. Chairman.
        Let me start with Mr. Sullivan, and there have been several
    references that sometimes we have excessive oversight with no
    value added. That's a frustrating comment to me, in that I have
    watched the Defense Contracting Audit Agency basically get
    taken to task by their peers for failure to even follow basic
    auditing standards. I've read, I can't tell how many of your
    reports that have, in fact, identified weapons acquisition as
    high risk, since 1990. It is not as though the oversight's not
    occurring, it's just that it's being ignored. It's not adding
    value because nobody's paying any attention to it.
        Let me ask you, in that regard, about JCIDS, Planning,
    Programming, Budgeting, and Execution (PPBE), and Future Years
    Defense Program (FYDP). Now, first of all, I think we ought to
    pass a law that they quit talking in initials because you all
    know what those things are I just said, but I guarantee you,
    nobody that I work for in Missouri has any idea what JCIDS is,
    PPBE is, or FYDP. What they are is: one, is all the Services
    getting together and basically giving each other what they
    want; two, is a 2-year calendar-driven process, where they're
    supposed to be figuring how they're going to spend the money;
    and three, it is the Secretary of Defense trying to low-ball
    what it's going to cost, long-term, in order to make sure that
    the other two go along with it. Is that an accurate summary of
    what those three are?
        Mr. Sullivan. The--that---- [Laughter.]
        I would say, number one, that the JCIDS is a requirement-
    setting process; it's where all of the Services tend to get
    together and figure out what it is that they require.
        Senator McCaskill. But, in your testimony, Mr. Sullivan,
    you pointed out, they never say no.
        Mr. Sullivan. That's true.
        Senator McCaskill. Have they ever said no that you're aware
    of since you've been looking at this?
        Mr. Sullivan. We issued a report for this committee, I
    think, about a year ago, where we looked at that, and it was--I
    think the JCIDS process then, if I'm not mistaken--this may not
    be exact, but I can get it for you--about 90 percent of the
    proposals that went in were granted.
        [Excerpt of GAO-08-1060 follows:]

        Senator McCaskill. Explain to me, so I understand, what is
    the makeup of the three groups of people. These three are
    really the stovepipes that cause a lot of the problem because
    you have, ``Here's what we want, here's how we pay for it, and
    here's how we figure out how much it's going to cost in the
    long run.''
        Mr. Sullivan. Yes, ma'am.
        Senator McCaskill. What I don't understand is: what is
    keeping them from making that one thing, so they all have to do
    all of that at once?
        Mr. Sullivan. The requirement-setting process, the JCIDS,
    is run by the JROC, and the JROC is made up of the military
    service chiefs, more or less. It's run by the Vice Chief of
    Staff. So, it, in a sense, is a matrix organization, but it
    receives most of the proposals for needs to be validated as
    weapon systems from the three Services, as stovepipes.
        Now, the JCIDS was established to have something called
    ``functional capability boards,'' which were supposed to be a
    matrixed organization based on looking at things like
    battlespace awareness or force protection or force projection,
    looking at it functionally instead of across the Services. What
    we found when we did that study was that the DOD has not
    staffed those functional capability boards properly, it hasn't
    resourced them properly, so they don't really do a lot of joint
    decisionmaking to send proposals forward to the JROC. So,
    mostly what they are receiving is proposals for capabilities
    that are coming from the Air Force, the Navy, and the Army, and
    they compete with each other.
        Senator McCaskill. The sense is, at JCIDS, that if you say
    no to what they want, then they're going to say no to what you
    want. Isn't that part of the problem, in terms of the way this
    is actually supposed to be oversight with no value added.
        Mr. Sullivan. I would say that the oversight is far from
    perfect. There's not a lot of value to it.
        Senator McCaskill. The idea of the Joint Staff is that
    they're supposed to be picking winners and losers. What is your
    sense of how effective the organization of the Joint Chiefs
    office has been, in terms of weapon acquisition and picking
    winners and losers?
        Mr. Sullivan. In terms of that organization that does that
    picking of the winners and losers, the JROC, there are a lot of
    redundancies right now in the weapon system portfolio because
    they can't make proper decisions, it seems to us.
        I'll give you an example. Right now, there are some
    unmanned aerial systems that are in development that we
    believe, and I think the Department actually believes, should
    have been joint programs, but the Services had unique-enough
    requirements and missions to be separate. Right now, there is
    the Predator, which has been very valuable in Iraq and
    Afghanistan and is an Air Force program. Air Force is currently
    making a bigger, more powerful Predator, called the Reaper.
    Another one of the Services, I believe it's the Army, has
    started what they call the Sky Warrior. Both drones are done by
    the same contractor and both with very similar requirements,
    but the Services have determined that they're different enough
    that they each have to have their own acquisition program.
    That's the sort of thing that the JROC is contending with; it's
    a very parochial kind of an attitude.
        Senator McCaskill. So, is this not fixable?
        Mr. Sullivan. This is the cultural aspects of this, I
    think, that we've all been addressing, where you can write
    legislation, you can have policies.
        Senator McCaskill. It doesn't do any good.
        Mr. Sullivan. But, unless you change the culture, and I
    guess that is the number one question: How do you change that
    culture, that has been in existence for so long, to try to turn
    it a little bit, to do things a little more efficiently? It's
    very much a culture issue.
        Senator McCaskill. I also think, Mr. Chairman, one of the
    things we have to do, in some instances, is take out a mirror
    because I think there are times that, when the military has
    tried, either by the way they've done the budget or by actually
    being so bold as to say we need to wind down a program, that
    Congress decides, because of our parochial interests, that it's
    important that we go to bat to augment the budget to take care
    of the weapon system that we think is important in our part of
    the world. So we contribute to this problem, and I think we
    shouldn't complete this hearing without at least acknowledging
    that sometimes Congress has their hand in this stew.
        Mr. Sullivan. Yes, ma'am. I would say, sometimes we should
    think of this as a system that is in equilibrium because there
    are very many stakeholders in this system that are getting
    specific things, even the GAO. It's a pretty good employment
    program for us. We report on cost schedule and performance
    problems, and we have been doing it 30 years. So, culturally
    speaking, if you examine it as a system that, maybe, is in
    equilibrium, in a sense--it's not necessarily broken for the
    people that are involved in it.
        Senator McCaskill. Thank you.
        Thank you, Mr. Chairman.
        Chairman Levin. Thank you very much.
        Just one quick comment on that. We do, in this bill,
    attempt to make the JROC system cover some of the issues that
    Senator McCaskill talked about by requiring it to make these
    early tradeoffs, by looking at cost and at schedule, by the
    way, as well as the requirements and the performance
    requirements.
        Mr. Sullivan. Yes, sir.
        Chairman Levin. There is an effort in this bill, Senator
    McCaskill. I just want to give you assurance that at least this
    bill attempts to do what we can, legislatively, to put those
    elements into the JROC process, which would, hopefully, cut
    down the parochialism by forcing consideration of cost and
    schedule, not just requirements. So, I'd just get that on the
    record.
        Thank you, Senator McCaskill.
        Senator Martinez is next.
        Senator Martinez. Mr. Chairman, thank you very much.
        Dr. Gansler, I wonder if I could ask of you to define for
    me a little more broadly what you mention as a ``holistic
    approach'' to defense needs.
        Dr. Gansler. Yes. In fact, when I even looked at the
    situation in Iraq and Afghanistan, I was surprised to find that
    the State Department, United Sates Agency for International
    Development, and DOD were all there, but not integrated, in
    terms of the buying, the contracting, and planning purposes.
    But, much more at a higher level than that, it seems to me that
    the world of the 21st century is going to require us to combine
    hard and soft power in the kinds of operations involving
    expeditionary operations or insurgent operations that we get
    involved with around the world. So, we're going to need a much
    closer tie between, in that case, State and Defense. But, I
    would go further and say, Homeland Security and Defense. It
    shocked me, in fact, that Saddam Hussein didn't try and pull
    some terrorist actions at the same time as we attacked him. I
    would expect that's going to happen in the future. So, it is an
    integrated, holistic perspective that is the meaning of
    ``national security'' in the 21st century, that will involve
    Homeland Security, that will involve State, will involve the
    intelligence community, will involve the DOD.
        I think that's basically what National Security Advisor
    General Jim Jones is now trying to do, some of the
    restructuring that's taking place at the national security
    level within the President's office. I think this combination
    of soft and hard power is going to be required, very clearly.
    That's what I was thinking of, in terms of the ``holistic
    perspective.'' But, also the types of threats. Think about it,
    the energy case, the pathogen spreading worldwide, the economic
    crisis that we're in, these are all national security issues
    for the 21st century that we have to start to incorporate into
    our thinking of national security. That's what I had in mind.
        Senator Martinez. Thank you. I think it is an intriguing
    future that we all are stepping into. I think, by the way,
    Saddam would have if he could have.
        But, the LCS program is one that I'm very much fond of, and
    one that I think is essential to the national security
    interests of our country. I wonder how you believe the Navy got
    so far off track on that particular program? Was it too many
    requirements being put on the platform by the Navy? Was it the
    length it's taken to develop it? We now have two hulls being
    developed. So, to the extent that any of you could speak to the
    LCS program and what you see, going forward, it would be
    helpful to me.
        Dr. Gansler. I just actually published, I think last week,
    a DSB report which looks specifically at the LCS and the
    presidential helicopter, a couple of programs of that sort,
    where the initial concept was: get something relatively fast,
    take something ``off-the-shelf'' that could be used, and
    addressing Dr. Kaminski's point about rapid acquisition--then
    maybe block 2, 3, and 4 would add some of the additional
    things.
        What happened on the LCS: the first thing they said was,
    ``Has to go through Sea State 8.'' Well, that's like going
    through a hurricane and it wasn't initially designed for that.
    Then the next thing they said was, ``Well, it has to have a new
    Navy sprinkler system.'' The sprinkler that was in the system
    in those two ships that you talked about, it wasn't going to be
    adequate, for some reason or other. So, each of these special
    requirements ended up basically changing the original block 1
    system and introduced the instability, cost growth, and
    schedule impact that we've talked about in all these other
    programs.
        Yes, we badly needed the LCS, but is it going to have to be
    a battleship? Does it have to do everything that a battleship
    does? How it's going to be used by the Navy was resisted, in
    terms of the nontraditional solutions. So, this is the culture
    change that we've been talking about, as well. If you kept the
    cost and the schedule, and got a block 1 system out there much
    faster--I like the idea that we did it competitively. I think
    that was a very important step. So, I would encourage that to
    be done in these earlier demonstration systems.
        Senator Martinez. Dr. Kaminski?
        Dr. Kaminski. Yes, sir. I think the concept initially was a
    good concept. What was missing was the upfront systems
    engineering and development planning that I spoke about. I can
    remember back to the time I was serving in the acquisition
    executive's job in DOD; I was a big proponent of commercial
    practice, and also in buying commercial systems where that made
    sense. But, if somebody came to me and said we were going to
    buy a commercial ship for this mission, my first question to
    him was, ``What are we going to change in the mission to be
    able to do it with a commercial ship?'' If the answer is
    ``nothing,'' then I have to ask a second question, ``Wait a
    minute, this commercial ship doesn't have the kind of military
    requirements you would have for fire-safe cables, for a fire
    sprinkler system, or a whole variety of things. What are you
    going to do about those?'' I don't think we started asking
    those questions about the LCS program until we were well into
    the program, so we missed this upfront set of tradeoffs. Those
    are tradeoffs that you have to make. They can be made sensibly
    if you approach them, understand them, look at the costs, the
    performance, and the schedule to make those tradeoffs. I don't
    believe we made those tradeoffs upfront. That, for me, does not
    necessarily damn the LCS program. There may still be value
    derived from looking at these tradeoffs and now making sensible
    decisions to go forward. I agree with Dr. Gansler about the
    advantage of having a competitive environment to be able to do
    that.
        Senator Martinez. Mr. Adolph?
        Mr. Adolph. I'm not really familiar with the LCS. I've had
    no involvement with the program.
        Senator Martinez. Okay, and you, Mr. Sullivan? I don't know
    whether you've had any experience.
        Mr. Sullivan. I don't have any specific experience with
    that, but we do have a team that looks at all of our Navy
    ships, and I'm sure we've had a report on that recently, which
    I could submit for the record.
        Senator Martinez. That would be great. I'd appreciate that.
        Mr. Sullivan. Okay.
        [Excerpt of GAO-05-255 follows:]

        Senator Martinez. I'm sure going to try to look up your
    article, as well, Dr. Gansler.
        You haven't written on DGD-1000 and the DGD-51 to date,
    have you?
        Dr. Gansler. Not lately.
        Senator Martinez. Okay. Do any of you share an opinion on
    the needs of the Navy as it relates to these two programs?
        Mr. Sullivan. I would just say that we're going to come out
    with our annual assessment of programs, and I think both the
    DGD-1000 and LCS are going to be programs that are covered in
    that, so you will get our take on them, probably in the next
    couple of weeks. I'll follow up, as well.
        Senator Martinez. Very good. Thank you.
        [Excerpts of GAO-09-326SP follows:]

        Senator Martinez. My time's up, thank you very much.
        Chairman Levin. Thank you, Senator Martinez.
        Senator Begich.
        Senator Begich. Thank you. Thank you very much, Mr.
    Chairman.
        It has been very interesting, listening to all the
    testimony. I will try to ask my questions very quickly, and I
    may not ask every single one of you to answer because I want to
    get some specifics here.
        Before I talk about the personnel issue, I'm a former
    mayor; I've just become a Senator. So, I believe I'm a mayor
    who just happens to be a Senator, so I like some of the
    comments you've made on personnel issues that I'll talk about
    in a second.
        I'm not familiar with the CAIG. Is that the CAIG? I looked
    through the bill as quickly as I could, and I see an annual
    report to Congress. Do you think it would be helpful for us to
    require more? I believe it's a combination of oversight; it's
    not just internal oversight, but this body needs to do more. I
    think this bill attempts to do that. But, do you think there
    should be a report--and maybe there is and I'm just not
    familiar with it--that comes to this committee on a regular
    basis, maybe quarterly or twice a year, that shows what the
    CAIG said it would cost before a system is started?
        Mr. Gansler, you made the comment that--I may get these
    words wrong--but it's almost like they ignore it. It comes out,
    it says, maybe, what it might really cost, but then they kind
    of push it down. I'm familiar with this, as a mayor. We call
    them HMS, Inc., studies, where the contractor wants to build
    it, as well the person inside the system who wants to build it,
    always seems to have a different price. When it's all done,
    it's pretty close to the one that HMS, Inc., did. Is that
    something we should have in this legislation, so we can see,
    before these systems start kicking off, here is what the real,
    or what another group said it could cost, so we at least have
    some understanding? I don't know who can answer that.
        Dr. Gansler. The idea of the independent cost analysis,
    right upfront, when we're doing the early systems engineering,
    is to be able to see the impact of the various requirements,
    and to be able to trade requirements, schedule, and cost as
    part of that early design requirements setting. Cost is
    actually an engineering challenge, just as schedule and
    technical performance are. So, if you, upfront, try to say,
    ``What's the cost impact of this?'' and the independent cost
    analysis group will say, ``Historically here's what it's
    been,'' and then people will come back and say, ``But, this
    time it's going to be different.''
        Senator Begich. Everything's different the next time.
        Dr. Gansler. Right, and so it's important to keep that in
    perspective, that the new technology comes along, and so forth.
    But, it's absolutely essential to get that independent
    estimate. Now, do you want to have Congress legislate what the
    price should be? I don't think so.
        Senator Begich. No, I'm not asking that. But here we are,
    complaining about all these cost overruns. I agree with all
    your comments. You can write all the legislation you want, but
    you do not change the culture and remove people who are not
    doing the job they should be doing, and putting people in there
    who should be doing the job, you don't change anything, we'll
    be back here in a couple of years. So, I'm not saying,
    legislate the price, but we become more aware, so we are
    putting the pressure where it should be. Let's be honest about
    the pricing, so when we do the budgeting authorization and
    appropriation, we don't go from 75 to 95, with two-thirds
    probably in the planning and design stage, and we always get
    the answer, ``We're this far, we have to do a little bit more,
    a little bit more, a little bit more.'' So what I'm asking is,
    should we have a more regular reporting period? Because once
    you're a year into a project, even though some are longer,
    they're already obligating more money. By the time this system
    here moves, you could be 2 years into having any commentary on
    it.
        That's my question. Should it be more regular reporting?
        Dr. Gansler. I think the thing you're missing is the fact
    that, in the development of a program, the program changes
    weekly. There's always technical issues that come up, there's
    always schedule problems, there's personnel problems, and so
    forth. You don't want Congress to be micromanaging the
    programs. You do want to make sure that the process is a good
    process. That's where I think the independent cost analysis is
    a very important thing and that your emphasis on it is the
    right thing to do. But, I don't think you want to get down to
    the point where you have a weekly report from the DOD.
        Senator Begich. I didn't ask that. I'm trying to get to the
    point. Is annual enough?
        Dr. Gansler. I think that--if you've convinced the DOD that
    you care about this, that you are going to be monitoring it,
    and that's what you're trying to do with the threshold numbers
    and the controls on that--I think that you're giving the
    message that the DOD needs to care about cost.
        Senator Begich. Okay. Let me make one comment. I forget who
    said it, about, ``stop the obligation of money at a certain
    point.'' I know, as a mayor, that's what you do. You turn the
    dollars off, and suddenly you get a response, and you get
    people paying more attention. So, I might be a little different
    than earlier comments that were made. I'm a little more direct
    on that. But, the personnel issue, to me, this is the
    challenge. If you don't change the culture, nothing changes.
    We'll be back here, and the numbers will grow, as they have
    over the years. It's not about adding more people, and I think
    Congress made a huge error by reducing down the amount of
    people. That was a huge mistake. We basically took the people
    who manage our programs out of the equation.
        So, besides putting more people into the system to make
    sure we have more folks out there, do you think--and I'm not
    sure I want to ask this question because I'm not sure you'll
    want to answer it--but, do you think, within the system that
    currently exists today, we have to change the deck? When I say
    ``change the deck,'' change personnel, people, and not just add
    more people and move people around so we satisfy their issues.
    This is a very hard, direct question. I've had to do this, as
    mayor. You might have half a dozen, a dozen, or 100 people; it
    doesn't really matter--they're in the wrong place. Anyone dare
    to want to answer that question?
        Mr. Sullivan. I would just say, just real quickly, that a
    lot of this is organizational. I think the people that work in
    the DOD now are great people, and really, really capable
    people, and good public servants and everything else.
        Senator Begich. Right.
        Mr. Sullivan. But, when we talk about culture, I guess,
    it's more the way things are organized. For example, I think,
    in our written statement, we have that the Under Secretary of
    Defense for Acquisition, Technology, and Logistics probably
    should have more ability to make the final decisions on things,
    acquisition, than he or she has now. One of the things that
    probably gets in the way of that is the fact that that position
    had an average turnover of 20 months since it started in 1986.
    I think that's part of the problem; there's an accountability
    issue, people change over too much, there's not a lot of direct
    communication. The three processes, I think, that this
    legislation does address, that you're trying to get the three
    big processes that we've talked about to communicate with each
    other more and to share in decisionmaking; right now, that's
    not there. So, I don't know if the people are good people. The
    structures and the way they're organized and the way they come
    and go is the big issue.
        Dr. Gansler. If I might comment that if we implemented what
    Goldwater-Nichols says to do, relative to promotion potential
    for the acquisition community, that you'd get a big step
    forward there. Instead of putting someone into a four-star
    position who has no acquisition background, but happens to be
    called an acquisition job, that's where we lose out. Each of
    the promotion-potential reviews and so forth need to really
    show that we value the acquisition workforce, civilians and
    military, and that's a critical point, I think, in order to
    keep people coming in. Dr. Kaminski said they're not doing it
    for the money, they're doing it so they can have an impact, and
    they need to have promotion potential.
        Senator Begich. I know my time's out, but you're about to
    jump out of your seat, Mr. Adolph.
        Chairman Levin. Can you do it real quickly?
        Mr. Adolph. Very quickly. There are three issues: numbers,
    training, and people. Particularly in the Air Force, they need
    to plus up because they drew down their acquisition workforce
    to a greater extent.
        I think the training that Defense Acquisition University
    provides is on the mark, for the most part. They were a part of
    our study. They ground in what we found into their training
    courses. I believe people need to be moved around a bit more,
    particularly in the civilian workforce, so that they get a
    variety of experiences rather than 1 or 2 years' experience 10
    times.
        Senator Begich. Thank you very much, Mr. Chairman.
        Chairman Levin. Thank you, Senator Begich.
        Senator Lieberman.
        Senator Lieberman. Thanks, Mr. Chairman. Thanks to the
    panel. It's probably not been an exciting hearing in content,
    but it's actually very important, an educational hearing, and
    there's a lot on the line in it, so I thank the four of you for
    the accumulated experience and wisdom you've brought to the
    table. I thank Senator Levin and Senator McCain for their
    legislation, S. 454, which certainly would take us forward in
    significant ways. I hope it passes.
        Earlier, Senator Collins referred to the fiscal
    responsibility summit that President Obama convened last week
    at the White House, and then a group of us on this committee
    happened to be in a breakout session on procurement reform. As
    Senator Collins indicated, most, though not all, but most, of
    us focused on the acquisition workforce, and the size of it. I
    want to talk to you about that in the time I have, hopefully,
    and at least one other subject we talked about.
        Dr. Gansler, you had a chart here in your testimony which
    has two lines; one is procurement by DOD, in dollars, from 1990
    to 2006, about as close as we can get to today, and then the
    other is the acquisition workforce. Obviously, the procurement
    dollars go up dramatically as the acquisition workforce goes
    down. But, beneath that, there are some numbers that are quite
    stunning, I think, and I know when I was here, Mr. Sullivan
    referred to them. The acquisition workforce, in 1990, was
    actually 500,000 people, and today it's dropped, but it's still
    200,000 people, which is an enormous number of people in
    acquisition. I note in your testimony that you focus in on the
    DCMA and say that it had 25,000 people in 1990, down to 10,000
    today. Then the other four general officers, and down to zero
    today. Give me some sense of the 200,000. Because my first
    reaction to it is, ``Wow, 200,000 people, isn't it enough to
    handle acquisitions by the DOD, even though acquisitions are so
    large?''
        Dr. Gansler. A large share of those are in the military
    depots that you have insisted do 50 percent of the maintenance
    work. A depot that has 20,000 people, that adds up pretty fast.
    To get to 200,000, you only need 10 of those.
        Senator Lieberman. So, acquisition wouldn't be what most of
    those do?
        Dr. Gansler. That's part of the acquisition workforce
    because logistics is part of the acquisition.
        Senator Lieberman. Got it. But in the conventional meaning,
    I think that's important, so I wanted to bring out.
        Dr. Gansler. Very few people are actually doing contract
    work, program management work, or things like that. As Mr.
    Adolph pointed out, the test and evaluation community is down
    significantly, but they're part of that community. So it's the
    total encompassing the research labs that the government runs
    all the way through the maintenance and logistics support.
        Senator Lieberman. Is there any way, now or maybe
    afterward, to submit for the record, you could give us a sense
    of what we would normally, in conversation, consider to be the
    acquisition workforce--how many people in the DOD are actually
    involved in acquisition, contracting, et cetera?
        Dr. Gansler. It's a small percentage of the people that are
    actually involved in that.
        In other words, you have the comptroller people, you have
    the personnel people, you have the policy people, and you have
    the acquisition workforce.
        Senator Lieberman. Right. I'm curious as to really what the
    size of the real acquisition workforce is, leaving out the
    depots and the rest.
        Mr. Sullivan. I don't have that answer right now. I know
    we're doing work on that.
        Senator Lieberman. Good. Get it for the record, please.
        Mr. Sullivan. Yes, sir.
        [The information referred to follows:]

        In our recent review of the Department of Defense's (DOD)
    acquisition workforce management (GAO-09-342), we found that, according
    to DOD data, the number of civilian and military personnel in the
    Department's acquisition workforce totaled about 126,000 at the end of
    fiscal year 2007 compared to about 129,000 personnel in 2001, a decline
    of about 2.5 percent. During this same time period, the number of
    contracting actions valued at over $100,000 increased by 62 percent and
    dollars obligated on contracts increased by 116 percent, according to
    DOD. Moreover, DOD has reported that the number of major defense
    acquisition programs has increased from 70 to 95. To augment its
    declining in-house acquisition workforce, DOD has relied more heavily
    on contractor personnel. However, DOD does not collect or track
    information on contractor personnel, despite their being a key segment
    of the total acquisition workforce.

        Senator Lieberman. There were two other things that there
    seemed to be an interesting consensus on, and they're quite
    different, about, you might say, principles for acquisition; a
    little different than anything we've talked about today.
    They're not unfamiliar. One was that our original position on
    acquisition should be to favor fixed-price contracts over cost-
    plus, and to favor competitive bidding, as opposed to
    negotiated contracts. I want to ask, to the extent that we have
    time--let's just focus on the fixed-price. Our sense, as we
    discussed at our breakout session with probably about 25
    people, was, generally speaking, private sector favors
    competition. So, why are we favoring cost-plus? Does the
    taxpayer really benefit from that?
        Mr. Adolph, you always get asked last because we're going
    left to right, so let's start from the right and ask you about
    that.
        Mr. Adolph. I think the other three panel members have more
    expertise in this issue. But, when you're in the basic research
    arena, it's very difficult to do on a fixed-price basis. It
    should be accomplished using cost-plus contracts.
        Once you get beyond research, then the next challenge in
    the development is system integration. That's not an
    insignificant task with the very complex systems we're
    developing today. But, once you get beyond that point then I
    think you reach a point where you can really consider going to
    fixed-price for downstream procurement.
        Senator Lieberman. Right. That's interesting.
        Dr. Kaminski, I think, just to clarify--if I could really
    make it too simple--what if we had a law that said, ``Defense
    contracts ought to be done on a fixed-price basis unless the
    Secretary certifies that there's a good reason not to''?
        Dr. Kaminski. I think that would end up requiring a lot of
    certification, Senator Lieberman, for the following reason.
        Senator Lieberman. Right.
        Dr. Kaminski. I believe fixed-price contracts are
    completely appropriate when we know precisely what it is we're
    going to buy. If there's uncertainty in what we're going to
    buy, and we know we're going to change, and we don't know yet
    quite how we're going to change, I think we end up on the wrong
    end of the bargain negotiating a fixed-price contract and then
    having to go back and renegotiate that effort for every change
    that occurs, especially when the contractor has already priced
    in some contingency in the fixed price.
        So, there's a time, for example, in the program, where
    perhaps we are working through this in development, and then we
    settle in on what we want to buy, and we're ready to enter a
    well-defined production program. That would be a fine time to
    do a fixed-price contract.
        So, I think you have to pass that criteria, knowing what it
    is you want to buy, and have it be predictable.
        Senator Lieberman. Okay. My time is running out, but, Mr.
    Sullivan, Dr. Gansler, if you'd give me a quick answer to a big
    question and follow up with writing.
        Mr. Sullivan. For major developments, I think it would be
    very difficult to go to a fixed-price contract for that because
    of the unknowns that are involved. But, I would say that if you
    work on requirements and try to do some of the things that
    we've been talking about here today, in terms of staying in
    what is doable, and having shorter cycle times to get these
    things done, you could have cost-plus development contracts
    that don't get so out of control. It really goes back to how
    much knowledge you have when you set out.
        Senator Lieberman. Okay.
        Dr. Gansler?
        Dr. Gansler. I think that it's a question of what's the
    meaning of a ``fixed-price contract''? As Senator McCain said,
    you get 75 changes every week. The contract continues to change
    hourly, in effect. I think it's very clear, when you have a
    stabilized and lower-risk program, that a fixed-price makes a
    lot of sense, it does give an incentive for the contractors. On
    the other hand, the cost-plus, I would say, we haven't been
    using the incentives that are available with the cost-plus-type
    contracts as well as we should, and I think, clearly, for
    research-and-development-type activities, cost-plus is an
    appropriate way to do it, but the ``plus'' part is an incentive
    rather than a fixed fee, I think. I would use the incentive
    more.
        Senator Lieberman. Those are very helpful answers. You
    encourage me to think that we ought to take a look, not at
    fixed prices on across-the-board answer, but to apply it by
    some selective means, and that, in doing so, we might benefit
    the taxpayers.
        Thanks very much.
        Chairman Levin. Thank you, Senator Lieberman.
        Senator Webb.
        Senator Webb. Thank you, Mr. Chairman. I'd like to add my
    appreciation for your taking on this issue. As has been so
    clear today, we're burning up a lot of money, and we're not
    getting a lot of product right now, particularly in the
    shipbuilding programs and aircraft programs.
        Dr. Gansler, when you're talking about people, which
    everyone seems to agree is the major issue, I was thinking
    about all the different years and different positions I've had
    on different sides of the table, here, working on this issue.
    It's so clear that what we need is disciplined management, not
    only on the people side, but in the system itself. ``People''
    include people in government, on this side of government, it
    includes people in business. We have challenges because there
    are not a lot of people in the military who use the business
    concepts, quite frankly, and they're asked to manage these
    programs. There are not a lot of businesspeople who are used to
    how product comes through a governmental system.
        I believe it was Mr. Sullivan who mentioned the creation of
    the Under Secretary of Defense for Acquisition in 1986. I was
    actually on Caspar Weinberger's staff when that position was
    created. We had a very talented individual who came into the
    position. He was bewildered with all the different steps that
    were required to get a system through the process. It's
    something you just don't see in the normal business world.
    There are lots of checks and balances. Some of them are
    appropriate, and some of them are less than appropriate.
        Dr. Gansler, I found your analogy with the LCS brought back
    a lot of different memories. They say that a camel is a horse
    created by a committee. I can go all the way back to the M-16,
    when they first developed the M-16, and they developed it
    directly toward a jungle environment, and then the different
    requirements were put on it to be able to be used in the desert
    and other different places. They put a different round in it
    that carboned up the chamber so people were dying in Vietnam
    because the weapon system requirements had changed as it
    evolved.
        I was in DOD when we were trying to do the Bradley fighting
    vehicle, and there were different requirements put on it here
    in Congress, so that it was very similar to what you were
    talking about with the LCS. They were saying it should perform
    different functions from the original design, and then there
    were all these press reports about the Bradley fighting vehicle
    falling over when it was going through a water obstacle because
    it got top heavy.
        Or the FG-7s, the USS Stark-class ships, which were
    designed to build to cost. So, we have fixed cost that we were
    going to build a ship toward, and then you go inside one of
    these FG-7s, you could plink the bulkheads on a FG-7, they were
    so thin. So when an Exocet missile hit the USS Stark in the
    Persian Gulf when I was Secretary of the Navy, it went all the
    way through the ship because they had had to make adjustments
    based on the cost rather than on some other areas.
        So it's a very complicated question. I think the key, when
    I look back, is if you can find the right leadership at the top
    on a program, negotiate and agree on general requirements--
    there are always going to be fixes--we'll get a program
    through. Probably the best example of that is when they put Al
    Gray, who later became Commandant of the Marine Corps, and he
    had the Development Center, they put him on the Light Armored
    Vehicle, and he got that program through in about a year. He
    just pushed it through, made all the negotiations, was very
    firm with people over here in Congress, as to what the
    requirements were for the Marine Corps, and it was a very
    successful program.
        I have one question I would like to put in front of you
    because I'm very concerned about it, and that is the state of
    all these programs in the United States Navy right now. You
    talk about the Polaris as having been probably the best
    analytical prototype of how to build a weapon system. One of
    the things about our submarine programs is that we built the
    frame, and then we added the technology onto the frame, similar
    to, say, the C-130, rather than continually building a new
    frame with all the costs and the time that goes into that. I've
    just been really struck over the last couple of years at how
    difficult the Navy procurements appear to be, and I'm trying to
    get my arms around why. I would be happy to hear from any of
    you who would like to begin answering that question.
        Dr. Kaminski. I'll make one comment. I think a key thing
    that impacts the programs are stability. So, if you see a
    program that we've had in production for some period of time,
    they have very good learning curves on those ships when we have
    a stable production program. Our issues sometimes are with
    first-ship cost. But, if you look at the cost of subsequent
    ships, what's happening there is very competitive with
    commercial-kind of production experience. Where we're producing
    ships regularly--one of the ship families in which we're doing
    that is the DDG-51. We have two yards. We have some competitive
    arrangements. Not quite head-to-head competitions, but there
    are some incentives in those programs. But, it is a well-
    planned, stable program that we've been producing. I think that
    approach would benefit us. It's the areas where we've had
    instability where we've more problems.
        Dr. Gansler. Yes, I guess I would approach it by thinking
    about, ``What is it the Navy really needs for the 21st
    century?'' and what types of ships they're going to want. There
    is a resistance to change. Take the Arsenal ship, for example,
    which was primarily to support the Marines and the Army
    onshore, it was resisted significantly in terms of it being a
    low-cost ship for few people. The highest costs in the Navy are
    for people and fuel if you look at the life-cycle costs of a
    ship. So trying to drive down the number of people on the ship
    and improve the fuel utilization are things we need to stress.
    Those are not the traditional things that are emphasized in the
    Navy construction of ships. I think it's a different look that
    we need to think about. The LCS has the same thing. Is that
    something the Navy really wanted or really resisted? So it's
    more the institutional inertia that has to be changed, I think,
    in terms of what the future Navy for the 21st century is going
    to need.
        Senator Webb. Do any of you see this as leadership failures
    in the Navy? [No response.]
        In terms of defining these objectives?
        Dr. Gansler. There have been some real successes. The F-
    18E/F on the Navy program was extremely well-managed, but that
    was because they had some really top people doing it, they had
    a clear objective, it was a incremental version of a prior
    demonstrated program, and it was well done.
        Another big success is the patrol frigate. It kept up
    competition throughout its life, and it had the steepest
    learning curves of any ship in the Navy. So, there are some
    success stories, but I think lessons learned haven't been
    widely applied.
        Senator Webb. Thank you, Mr. Chairman. My time's up.
        Chairman Levin. Thank you, Senator Webb.
        Any other questions of this panel before we excuse them?
    [No response.]
        Thank you all for your time. Some of you came some distance
    to get here, at some inconvenience. At least one of you had to
    give up a family commitment, and we won't identify who that was
    because the family is better off not knowing, maybe.
    [Laughter.]
        But we're very grateful to all of you for your testimony.
    It's very, very helpful.
        I will also submit for the record the text of the Weapon
    Systems Acquisition Reform Act of 2009; a statement from Ken
    Krieg, former Under Secretary of Defense for Acquisition,
    Technology, and Logistics; a statement from Moshe Schwartz from
    the Congressional Research Service; and the DOD Inspector
    General Acquisition Workforce Count Report.
        [Additional material included for the record.]

        Chairman Levin. We'll stand adjourned.
        [Questions for the record with answers supplied follow:]
               Questions Submitted by Senator Joseph I. Lieberman

                      STATUS OF CURRENT ACQUISITION SYSTEM

        1. Senator Lieberman. Mr. Sullivan and Dr. Gansler, I have several
    concerns with the acquisition system as it currently operates. Most
    importantly, it seems that the requirements validation process under
    the Joint Capabilities Integration and Development System (JCIDS) is
    not synchronized with the Planning, Programming, Budgeting, and
    Execution (PPBE) funding process. Can any effort at reform succeed
    without addressing this fundamental flaw in acquisition management?
        Mr. Sullivan. We do not believe that there can be success until
    there is better synchronization between the Department of Defense's
    (DOD) requirements, funding, and acquisition processes. The Government
    Accountability Office (GAO) has previously reported that the lack of
    integration between the JCIDS and PPBE processes is a key factor that
    contributes to the Department's inability to achieve a balanced
    portfolio of weapon system programs that meets the needs of the joint
    warfighter and matches needs with available resources. The JCIDS
    process has not been effective in prioritizing needs from a joint,
    Department-wide perspective, and it largely approves capability needs
    without accounting for the resources or technologies that will be
    needed to acquire the desired capabilities. Resource allocation
    decisions in DOD take place through the PPBE process, which is separate
    from JCIDS. PPBE largely allocates resources on a Service-by-Service
    basis and does not effectively link resources to capabilities. In
    addition, the process allows too many programs to start development
    with unreliable cost estimates and without a commitment to fully fund
    them. Until DOD establishes a more integrated approach to weapon system
    acquisition, it will continue to struggle to effectively prioritize
    warfighting needs, make informed trade-offs, and achieve a balanced mix
    of weapon systems that are affordable, feasible, and provide the best
    value to the warfighter.
        Dr. Gansler. I believe ``cost'' should be an essential part of the
    ``requirements process''--since, in a resource-constrained environment,
    it directly determines ``numbers'' (of systems procured) and that
    determines force effectiveness (even more than individual weapon's
    effectiveness).

        2. Senator Lieberman. Mr. Sullivan and Dr. Gansler, what are your
    recommendations for effectively balancing the requirement validation
    process with the calendar-driven funding process?
        Mr. Sullivan. To improve DOD's ability to deliver weapon systems at
    the right time and right cost, GAO has recommended that the Department
    implement an enterprise-wide portfolio management approach to making
    weapon system investments that integrates the determination of
    warfighting needs with available resources and cuts across the Services
    by functional or capability areas. Such an approach would focus on
    assessing weapon system investments collectively from an enterprise
    level, rather than as independent and unrelated initiatives. By
    following a disciplined, integrated process--during which the relative
    pros and cons of competing weapon system proposals are assessed based
    on strategic objectives, warfighter needs, and available resources, and
    where tough decisions about which investments to pursue and not to
    pursue are made--DOD could minimize duplication between Service
    components and more effectively support each new development program it
    commits to. Furthermore, to ensure effective weapon investment
    decisions are made, GAO has recommended that a single point of
    accountability be established at the Department level with the
    authority, responsibility, and tools to implement portfolio management.
        Dr. Gansler. My answer to question #1 (above) will greatly aid in
    this; but, in general, the ``requirements process'' must look well
    beyond the near-term budget (funding) process. Nonetheless, the
    frequent budget changes (particularly, by Congress) introduce great
    instability and inefficiency into the implementation of the weapons
    acquisition process.

                               JOINT REQUIREMENTS

        3. Senator Lieberman. Mr. Sullivan and Dr. Kaminski, I am concerned
    that DOD has historically done a poor job of procuring capabilities
    that address joint requirements, and more specifically, that the
    Services have little incentive to spend what they see as their limited
    budgets on capabilities that will principally benefit sister Services.
    One particular instance of this is the Multi-Platform Radar Technology
    Insertion Program, and the Air Force's apparent reluctance to procure a
    system that will principally serve the Army's battlefield management
    requirements. How can the acquisition process better prioritize and
    procure such crucial joint capabilities?
        Mr. Sullivan. GAO has recommended that DOD should determine and
    allocate appropriate resources for more effective joint capabilities
    development planning. The functional capabilities boards, which were
    established to manage the JCIDS process and facilitate the
    prioritization of needs, have not been staffed or resourced to
    effectively carry out these duties. Similarly, the combatant commands
    (COCOMs) also lack the analytic capacity and resources to become more
    fully engaged in JCIDS. GAO recently reviewed JCIDS documentation
    related to new warfighting capabilities and found that most--almost 70
    percent--were sponsored by the military Services with little
    involvement from the joint community, including the COCOMs, which are
    responsible for carrying out military missions. The Services drive the
    determination of capability needs, in part because they retain most of
    DOD's analytical capacity and resources for requirements development.
        Within the Department's PPBE process, the individual military
    Services are responsible for budgeting and allocating resources under
    authority that is commonly understood to be based on title 10 of the
    United States Code--to organize, train, and equip military forces. In
    this structure, the budget is based more on individual, service-focused
    needs than on joint warfighting needs. In the past, GAO has reported
    that this situation has contributed to interoperability problems among
    weapon systems and unnecessary duplication of capabilities in some
    areas. The Office of the Secretary of Defense (OSD) reviews and makes
    adjustments to the Services' budgets, but this takes place toward the
    end of the PPBE cycle when it can be difficult and disruptive to make
    changes, such as moving funds to higher-priority programs or to support
    joint needs. DOD has recently taken steps to establish capability
    portfolio management in selected areas in an attempt to overcome the
    service-centric nature of the resource allocation process. However,
    because the portfolios are largely advisory and lack authority and
    control over the allocation of resources, there may be limits to
    achieving joint capabilities.
        Dr. Kaminski. The priority must be set in the budget process, with
    key oversight from OSD and the Joint Chief of Staff. The procurement
    can be improved by first developing (a several year process) then
    assigning key program management personnel with the requisite training
    and domain experience. Alignment of the responsibility, authority, and
    accountability of the program manager will require that a degree of
    trust be established between the program manager and those responsible
    for our oversight mechanisms. We must be prepared to delegate authority
    to the program manager, and provide him or her with some flexibility to
    manage--to adjust levels and allocation of funding, to adjust the
    allocation of performance parameters, to adjust schedule, and to tailor
    the acquisition approach to be responsive to the need. Clearly, there
    must be bounds established beyond which the program manager must seek
    approval from oversight authorities. But I believe these bounds are too
    narrow and inflexible today. We also need to provide sufficient upfront
    funding, and maintain funding stability throughout program execution.

        4. Senator Lieberman. Mr. Sullivan and Dr. Kaminski, are either the
    JCIDS or PPBE as currently constructed able to overcome parochial
    service interests in this regard?
        Mr. Sullivan. No, I do not believe that the JCIDS and PPBE
    processes are constructed effectively to address the service-centric
    manner in which weapon systems are acquired in DOD. Past studies
    chartered by DOD and other organizations have reported similar
    conclusions and made a number of recommendations to improve the
    acquisition of joint warfighting capabilities.
        Dr. Kaminski. Both help, but I don't believe they are sufficient.

              DEPARTMENT OF DEFENSE SCIENCE AND TECHNOLOGY PROGRAM

        5. Senator Lieberman. Dr. Gansler and Dr. Kaminski, as we try to
    reduce risk in acquisition programs, what role does (or should) DOD
    science and technology (S&T) program play?
        Dr. Gansler. S&T are critical to both ``staying ahead'' (which is
    the basis of our national security strategy of ``technological
    superiority'') as well as to risk reduction--since we need to prove out
    the S&T first, before introducing it into weapons acquisition.
        Dr. Kaminski. With good S&T programs supporting development
    planning, we can assign managers to develop prototypes, critical
    systems or components needed to better understand cost/performance
    trades and reduce risk. With the appropriate technology base in place,
    it is reasonable to expect that many of these developments can be
    completed in 2-4 years, so one manager will be in place from start to
    delivery of those critical subsystems during the development planning
    phase. This should allow the full system development to proceed on a
    shorter schedule as a result of the risk reduction.

        6. Senator Lieberman. Dr. Gansler and Dr. Kaminski, do you think
    that the Department's S&T programs, and in particular the DOD
    laboratories, are sufficiently resourced and staffed to provide support
    to the acquisition programs and address the problems you are discussing
    today?
        Dr. Gansler. The DOD's S&T program must be adequately funded to
    ``stay ahead''; and, traditionally, whenever there is a shortage of DOD
    money it is one of the areas the Services tend to cut (since it is
    long-term). Thus, in the likelihood of a coming budget crunch, it must
    be protected--not just in the DOD labs, but particularly in the
    universities and in industry.
        Dr. Kaminski. No. We are short in both quantity and quality of
    staffing, and are not using the latest tools to support early modeling,
    simulation, and testing.

        7. Senator Lieberman. Dr. Gansler and Dr. Kaminski, what
    recommendations do you have for this committee to improve the quality
    of the DOD S&T program and the quality of the DOD laboratories?
        Dr. Gansler. Adequate funding, and allowance for foreign graduate
    students, scholars, and researchers (per Presidential Decision
    Directive-189) to work on DOD S&T, along with ``peer review'' of all
    proposals, are critical to the United States staying ahead. The SBIR
    and STTR programs are also a great benefit, and must be maintained.
        Dr. Kaminski. The first step is to ensure that we not only restore,
    but also enhance early and continuing systems engineering coupled with
    the effective development planning needed to drive our S&T program.
    This will require commitment of more significant investment dollars
    earlier in our acquisition programs, and a commitment to build a cadre
    of systems engineers and development planners with the education,
    training, and domain experience needed to be effective. Attracting
    ``best and brightest'' to this work--and keeping them--will require a
    personnel system that will identify and track these important human
    resources and establish a career path to allow those who are successful
    to advance to senior program management and leadership positions. Their
    domain experience will be enhanced by managing the building of critical
    technology demonstrators and subsystems during the development planning
    program, reducing risk, and building skills and experience at the same
    time.
                                     ______

                    Question Submitted by Senator Jack Reed

                               S. 454 LEGISLATION

        8. Senator Reed. Dr. Gansler, in the proposed S. 454 legislation,
    section 203 focuses on maximizing competition throughout the life cycle
    of major acquisition programs. One proposed provision is that modular,
    open architectures are used to enable competition for upgrades. Please
    comment on how the implementation of this recommendation would
    positively impact major weapons programs that have been sole sourced
    for decades, and how the Services can rapidly implement these changes
    to achieve maximum benefits. In addition, please comment on the
    feasibility of applying this provision to current programs, not just
    new programs.
        Dr. Gansler. Let me begin by noting that the objective should not
    be ``maximizing competition''; rather, it should be ``maximizing the
    effectiveness of competition.'' Competition for its own sake, can be
    detrimental. For example, there must always be the ``option'' of
    competition, but if a firm is constantly improving performance and
    lowering costs, they should be rewarded by not recompeting the program;
    but there must always be the option of introducing competition if costs
    are rising (so the ``threat'' of competition is what creates the
    incentive for lowering costs; and the follow-on is the reward).
        Open architectures are clearly one way to allow competition to be
    introduced at the subsystem level (which often is 70-80 percent of the
    total weapons cost); so it should be greatly encouraged--both on
    upgrades of current systems and on new ones.
                                     ______

                 Questions Submitted by Senator Daniel K. Akaka

                     WEAPON SYSTEMS ACQUISITION REFORM ACT

        9. Senator Akaka. Mr. Sullivan, section 105 of the Weapon Systems
    Acquisition Reform Act recommends the Joint Requirement Oversight
    Council (JROC) seeks input from the combatant commanders in identifying
    joint military requirements. Currently, the combatant commanders
    produce an annual Integrated Priority List (IPL) that attempts to
    satisfy this same goal. In your prepared testimony, you stated some of
    the combatant commanders do not feel their needs are sufficiently
    addressed. What must be done within DOD to fully implement the intent
    of this legislation to ensure combatant commander's influence
    throughout the acquisition system to decrease the Service-centric
    environment we've seen for years?
        Mr. Sullivan. GAO and several other studies have recommended that
    DOD increase joint analytic resources for a less stovepiped
    understanding of warfighting needs. While each COCOM submits an IPL to
    the Chairman of the Joint Chiefs of Staff each year, which defines the
    COCOM's highest-priority needs, it does not appear that the COCOM needs
    are well integrated with JCIDS. According to officials from several
    COCOMs that we have met with, needs identified through IPLs are not
    typically developed into JCIDS capability proposals. Some COCOM
    officials pointed out to us that because of their limited resources,
    they must pick and choose capability needs and persuade one or more of
    the military Services to sponsor a proposal that addresses the COCOM's
    need. The Department has been working to give more time and priority to
    the commanders' IPLs, but more attention is needed.

        10. Senator Akaka. Mr. Sullivan, in your prepared testimony, you
    mentioned the establishment of the DOD capability portfolio management
    process. DOD identified nine joint capability area portfolios to
    mitigate the tendency to develop Service-centric, stovepiped solutions.
    However, the capability portfolio managers lack the necessary authority
    and control required to be successful. How can DOD reform their
    capability portfolio management process to mitigate the tendency to
    develop Service-centric, stovepiped solutions?
        Mr. Sullivan. DOD's capability portfolios are intended to advise
    the Deputy Secretary of Defense on how to optimize investments within
    their respective capability areas. They have no independent
    decisionmaking authority and must operate within the Department's
    existing framework for acquiring weapon systems. That is, capability
    needs are still determined through JCIDS and resources are allocated
    through PPBE. Although it is too soon to assess the impact of
    capability portfolio management, according to some DOD officials, the
    portfolios provided key input and recommendations in the 2010 budget
    process. However, without portfolios in which managers have authority
    and control over resources, DOD is at risk of continuing to acquire
    weapon systems in a stovepiped manner. Ultimately, DOD needs to develop
    an integrated portfolio approach that brings the determination of
    requirements together with the allocation of resources. To be
    effective, such an approach must have committed leadership, empowered
    portfolio managers, and accountability at all levels of the Department.

        11. Senator Akaka. Dr. Gansler, Dr. Kaminski, and Mr. Adolph,
    changes made to a system after acceptance of review cost many times
    more than the same changes made during the requirements gathering
    phase. How can we improve the requirements gathering phases of the
    acquisition of major weapons systems to front-load these changes to
    achieve significant cost savings?
        Dr. Gansler. Please refer to my answer to question #1. If cost is a
    requirement, then changes that come in must not only have showed how
    they will improve performance but how they will live within the cost
    requirement.
        Clearly, the earlier this is done, the better; but technology
    changes, threat changes, et cetera, do come along, and as long as they
    are explicitly within the overall cost requirement they can be allowed
    in.
        Dr. Kaminski. Requirements adjustments during execution of
    acquisition programs are not necessarily bad if they are conducted
    within the framework of value propositions. Good program management
    requires continuing adjustment of the requirements flow-down and
    allocation to major subsystems based upon expected performance,
    schedule, and cost. Even top level requirements require continuing
    assessment and adjustment based upon similar value propositions. As we
    discover more about costs and performance during execution, good
    program managers should be interacting with requirements developers to
    adjust as needed to provide the best value for the user. This requires
    some flexibility in cost allocation, and it also places demands upon
    requirements developers to become involved in tradeoff decisions. To be
    effective, requirements developers as well as acquisition managers must
    have experience with systems engineering tools and techniques, and both
    must have sufficient domain experience to engage productively in cost-
    performance tradeoffs.
        Mr. Adolph. A disciplined analysis of alternatives process should
    be employed to support the JCIDS and acquisition processes, from
    capability needs identification to include system design and
    development as well as life-cycle improvement. Emphasis must be placed
    on improving the processes for relating mission effectiveness and cost
    to system design, system performance, and suitability. Effective
    feedback processes are of special importance for spiral developments to
    identify enhancements which will improve performance and suitability.
    Improving the quality and speed of this feedback is increasingly
    important in responding to rapid changes in threat environments.

        12. Senator Akaka. Dr. Gansler, Dr. Kaminski, and Mr. Adolph,
    significant cost overruns are incurred from the instability of mission
    and design requirements during testing and even, in some cases,
    production. To what extent do you think it would be advantageous to
    more rigorously enforce a freeze on fundamental requirements at the
    system requirements review gate?
        Dr. Gansler. The greatest causes of cost growth, after program
    initiation, are budget changes (often caused by Congress) and
    ``requirements'' changes--both of which can be controlled by
    discipline. (See question #11.)
        Dr. Kaminski. It is always beneficial to have a firm understanding
    of very top level requirements at the start of a program. But
    adjustments of subordinate requirements during execution of acquisition
    programs are not necessarily bad if they are conducted within the
    framework of value propositions. Even top level requirements need
    continuing assessment based upon evolving value propositions. As we
    discover more about costs and performance during execution, good
    program managers should be interacting with requirements developers to
    adjust as needed to provide best value for the user. This requires some
    flexibility in cost allocation, and it also places demands upon
    requirements developers to become involved in tradeoff decisions. To be
    effective, requirements developers as well as acquisition managers must
    have experience with systems engineering tools and techniques, and both
    must have sufficient domain experience to engage productively in cost-
    performance tradeoffs.
        Mr. Adolph. Rigorous enforcement of key requirements thresholds
    should be the norm when entering system development and demonstration.
    There needs to be more rigor and discipline in the entire requirements
    definition and acquisition process. Issues that need to be addressed in
    relation to requirements setting include technology readiness (see
    response to question 13 below), the translation of requirements into
    design criteria, with attention to testability at the subsystem and
    system levels, as well as defining thresholds for key performance
    parameters.

        13. Senator Akaka. Dr. Gansler, Dr. Kaminski, and Mr. Adolph, cost
    overruns are also caused by the inherent unpredictability of the
    development of new technologies in the course of fulfilling system
    contracts. Would you recommend restricting the proposed designs for
    weapons systems to proven technologies while adequately funding
    contractual vehicles for pure research?
        Dr. Gansler. I would restrict the first ``block'' of the system to
    proven technology; then--using ``spiral development''--evolve the
    system's future blocks as future technology is proven out; and as long
    as the future technology either improves performance (at the same cost)
    or lowers the cost. This future technology should be funded in parallel
    with the system, until it is proven.
        Dr. Kaminski. This will be helped by better development planning
    and alignment of incentives. With good development planning, we can
    assign managers to develop prototypes, critical systems or components
    needed to better understand cost/performance trades and reduce risk. It
    is reasonable to expect that many of these developments can be
    completed in 2-4 years, so we do not need to restrict ourselves to
    proven technology at the beginning of development planning. As these
    critical subsystems are delivered and tested, the risk reduction and
    domain experience gained in both government and industry will allow us
    to prove the critical technologies involved and reduce the time
    required to develop, integrate, and test the full system. We can rely
    on the experience gained during development planning to apply informed
    judgment to adjust requirements to improve value, reduce time, and
    better estimate and manage costs.
        Mr. Adolph. Weapon system design should be based on proven
    technologies. History has shown repeatedly that doing otherwise
    introduces unacceptable risk in to weapons system development which
    necessarily translates into cost growth and schedule delays. Assessing
    whether a technology is sufficiently mature is currently accomplished
    through the Technology Readiness Levels (TRL) construct which should
    employ government test and evaluation (T&E) as the key component in
    moving a technology from TRL 4 to TRL 6. Recent consideration has been
    given to strengthening early involvement of the test community by
    reinvigorating the government developmental test process and by
    involvement of OT&E in readiness assessments. These are sound
    initiatives and should be focused on providing rigor to the TRL
    assessments in terms of the test methodology as well as the test
    environment. Implementation of this more disciplined approach will
    require a reversal to the trend of cutting back on government test
    personnel as well as strengthening their role in supporting the
    acquisition process. In that regard, the Director of Developmental Test
    and Evaluation should make an assessment of technological maturity and
    integration risk, based on test results and report the findings to the
    Undersecretary of Defense for Acquisition, Technology and Logistics.
        There may be rare exceptions when a requirement is so pressing that
    risks must be taken to design a system which incorporates an unproven
    technology. In those instances, a high-priority, adequately funded, and
    closely monitored effort must be undertaken to mature the technology.
    This must include an early, rigorous assessment of the difficulties
    associated with the development effort as well as testing to fully
    assess the requisite maturity.
        Additionally, the TRL process has been focused on the technology of
    system components. Problems with manufacturing these components in
    production quantities have also led to significant cost growth for
    weapons systems. Previous legislation established the Manufacturing
    Technology Program to identify and develop initiatives to improve
    manufacturing quality, productivity and technology. Consideration
    should be given to expanding the TRL process to evaluate the maturity
    of the production methods by which parts are manufactured.
                                     ______

                   Questions Submitted by Senator Bill Nelson

                            MODELING AND SIMULATION

        14. Senator Bill Nelson. Dr. Gansler, Dr. Kaminski, and Mr. Adolph,
    computer modeling and simulation (M&S) tools can be very useful in
    discovering problems in major systems before we spend a lot of money or
    time on hardware development. DOD has some programs in M&S--which in my
    opinion are underfunded and disjointed. What is your assessment of M&S
    activities within DOD and their ability to address some of the
    acquisition problems we are discussing here today?
        Dr. Gansler. M&S can be a great aid in system design and
    performance evaluation, and should be an important element in the
    development of all systems.
        Dr. Kaminski. M&S can have a very significant positive impact on
    acquisition performance if there is sufficient emphasis on continuing
    validation of M&S tools and integration with T&E. Giving early and
    serious attention T&E will require strengthening our T&E organizations
    and personnel. T&E is often an afterthought, and contracts are often
    written without any mention of how we will test the product. We spend
    large amounts of money when a large development team waits for test
    results. The alternative is to spend less money and time by early
    development of M&S tools, continuously validating the models with
    testing at critical points, and considering the investment in M&S and
    test resources as a critical part of our systems engineering and
    development planning efforts.
        Mr. Adolph. With some exceptions, the capabilities and limitations
    of simulations, simulators and physics-based models of individual
    subsystems are well understood. These tools are generally used where
    appropriate. However the use of modeling to support requirements
    definition, analysis of alternatives, and test design needs to be
    pursued more vigorously. The expectations regarding the use of M&S to
    support elements of the acquisition process, including system
    integration, T&E, and manufacturing have increased along with advances
    in computer technology. Distributed T&E (and training) using multiple
    ranges, laboratories, and facilities have benefited from the use of
    live, virtual, and constructive simulations. However, the availability
    of adequate models and modeling tools involving human dynamics for use
    in T&E, mission planning, systems engineering, and risk assessment has
    not kept pace.
        M&S to support acquisition/test programs has been the norm for
    decades, and received additional impetus in the 1990s with advances in
    information processing technology. In many cases, the incremental
    improvements as related to acquisition were oversold in the late 1980s
    and 1990s. Additionally, the requirements for verification, validation,
    and accreditation have proven to be more costly and timeconsuming than
    anticipated. Many of the M&S activities initiated in the last 15 years
    were disjointed, with few successful attempts to coordinate
    requirements, build and share common models, correlate performance, and
    ensure that the models were adequate for the intended applications.
    Numerous recent M&S developments have been terminated after spending
    millions because of the failure to produce a useful product.
        Building models to meet every need is not feasible. An overall M&S
    vision is required to identify where efforts should be focused, and to
    ensure that new efforts are within that vision. If high priority M&S
    efforts within that vision could be identified, it would be possible to
    establish requirements, understand what is currently available, and
    then determine shortfalls in M&S capability. One approach is
    exemplified by the Testing in a Joint Environment Roadmap, which was
    recently developed by DOT&E and approved by the Deputy Secretary of
    Defense. While focused on T&E usage, this capability would meet a
    significant percentage of contractor development and testing M&S
    requirements. The T&E roadmap identifies a requirement for a
    distributed live, virtual, constructive T&E capability; largely
    supported by existing models of friendly and threat systems. Much of
    this capability could come from existing models of acquisition systems
    developed by program contractors, in conjunction with threat models
    available from the intelligence agencies. However, there has been
    virtually no effort to identify requirements for high priority joint
    missions, to determine what friendly and threat force representations
    are required, assess the availability of existing models that meet
    those requirements, and then use that information to define M&S
    shortfalls. Additionally, there are insufficient mechanisms and
    processes to feed back data from operational tests and field exercises
    to further validate and refine models. Archiving mechanisms that can
    store and locate data for future applications are needed.
        There are also supporting M&S requirements (engineering and
    physics-based models) which are required in the validation of higher-
    level engagement models. Higher level models, such as envisioned in the
    T&E roadmap, could be used to identify areas with the greatest risk and
    uncertainty to better define areas where engineering and physics-based
    models are required. The initial effort should be to review the past
    studies on M&S in the DOD and determine those requirements that could
    contribute the most to improving M&S capability. While most studies
    have identical or similar recommendations, the majority of those
    recommendations have not been implemented.

        15. Senator Akaka. Dr. Gansler, Dr. Kaminski, and Mr. Adolph, how
    would you recommend addressing shortfalls in DOD activities in M&S?
        Dr. Gansler. To address shortfalls, I would ask an independent
    group (such as the Defense Science Board) for an assessment and a set
    of recommendations (including appropriate funding).
        Dr. Kaminski. We need upfront investment in M&S tools to understand
    the value propositions and the tradeoffs among cost, schedule, and
    performance of the mission. We spend large amounts of money when a
    large development team waits for test results. The alternative is to
    spend less money and time by early development of M&S tools,
    continuously validating the models with testing at critical points, and
    considering the investment in M&S and test resources as a critical part
    of our systems engineering and development planning efforts.
        Mr. Adolph. Focused M&S business plans must be developed. Separate
    plans are needed for requirements definition, acquisition/test,
    training and force structure evaluations. As discussed above, the
    latter plans should be used to identify critical joint missions and
    provide a basis for prioritizing M&S requirements; to understand
    current capabilities and identify major shortfalls.
        An issue that needs to be addressed is the availability of existing
    models to meet current requirements. Many models were developed by
    system contractors to meet specific needs. In some cases they are
    proprietary and there are rarely provisions for models to be maintained
    current so they represent a system in the field with normal
    improvements and modifications, or changes to the treats. Often
    contractors are not funded to make their models generally available.
    The government must address issues relating to long-term configuration
    control, upgrades, ownership, and funding throughout the system
    lifecycle, to include training. In addition, addressing the
    requirements for a single joint mission doesn't capture requirements
    across several potential joint missions and scenarios.
        However, it would provide a starting point for future requirements
    and identify the most sensitive issues in implementation of the overall
    concept. Addressing one major joint mission scenario would identify a
    large percentage of requirements for other joint missions.
        Finally, an important consideration is the physical location of
    major simulation laboratories and related facilities. There are often
    valid reasons to locate a simulation capability at a contractor
    development facility to support initial development; however it places
    limitations on the future utility and accessibility of the simulation,
    as well as the need to replicate parts of the simulation capability at
    a government test facility or logistics center. Prior to making a
    decision regarding the location of simulation facilities for each major
    program, an assessment should be made as to the most cost-effective
    locations(s), taking into account follow-on and future programs. This
    decisionmaking process should include the appropriate Service MRTFB
    representatives, as well as the TRMC. (See also my response to question
    #17.)

                         TEST AND EVALUATION ENTERPRISE

        16. Senator Akaka. Mr. Adolph, a major step in developing weapons
    systems is ensuring that they are appropriately tested during their
    development and before their operational use. I believe that DOD is
    letting its testing enterprise decline precipitously in terms of
    testing equipment, infrastructure, and workforce. What role do you feel
    that the DOD testing enterprise plays in ensuring acquisition programs
    deliver their systems on schedule and on budget?
        Mr. Adolph. The role of the DOD testing enterprise is addressed in
    detail in the recent Defense Science Board Task Force Report on
    Developmental T&E. My written testimony covers the major findings and
    recommendations from that study, and while it emphasizes the impacts of
    reliability, availability, and maintainability on life-cycle costs, the
    implementation of some acquisition reform initiatives diminished the
    role of government T&E in the acquisition process.
        A brief summary response follows. The traditional role of the
    government during the test planning phase included the identification
    of test resource requirements and test facilities, the development of a
    test strategy and detailed T&E plans. When programs moved from the
    planning phase to the execution phase, the government traditionally
    performed operational assessments, participated in test conduct and
    analysis, and performed an evaluation of the test results for the
    government Program Office. With a few exceptions, this is no longer the
    case. In many instances the government testing enterprise has gone from
    vigorous test participation to oversight to out-of-sight.
        As a minimum, government test organizations should reconstitute and
    retain a cadre of experienced T&E personnel to perform the following
    functions:

             Participate in the translation of operational
            requirements into contract specifications and key performance
            parameters
             Participate in RFP preparation and source selection,
            including assurance that contractor developed models and
            simulations are available to the government
             Participate in technology maturity determination and
            prototype evaluations
             Perform operational assessments in conjunction with
            the systems engineering process
             Participate in developmental T&E planning and
            technical reviews
             Participate in test conduct, analysis, reporting
            capabilities, and limitations, with emphasis on government
            analysis and reporting

        17. Senator Akaka. Mr. Adolph, what is your assessment of the state
    of the DOD's testing facilities and their ability to perform their
    designated missions?
        Mr. Adolph. The focus of DOD's testing facilities is the Major
    Range and Test Facility Base (MRTFB) which is comprised of those ranges
    and facilities designated most critical to supporting the T&E needs of
    DOD development and acquisition programs. Since the 1970s, DOD's intent
    has been to ensure the capabilities and capacities of these MRTFB
    facilities/ranges meet the T&E requirements of emerging technologies
    and weapon system programs, taking into consideration the long-lead
    times necessary to acquire these capabilities, which often dictates
    that new test technologies be developed and matured. Likewise, a
    skilled T&E workforce must be in place to address the challenges of
    testing and evaluating new weapon systems with increasing complexities
    of technologies, integration, and missions. Given that this is the
    intention of the MRTFB, where do we stand today?
        As I noted in my written statement, several changes resulted from
    the implementation of acquisition reform initiatives in the late 1990s.
    Among them was a de-emphasis on the use of government test facilities,
    testers, and evaluators. When programs need to use government test
    facilities, the first inclination has sometimes been to use non-MRTFB
    capabilities to avoid reimbursement costs required by the MRTFB charge
    policies, often using training ranges having little or no T&E
    infrastructure. If the MRTFB facilities/ranges are used, programs
    typically prefer to request only the raw data to perform their own
    evaluations. Consequently, the Services have been reducing their
    budgets and T&E workforce for sustaining the MRTFB. Not only has the
    MRTFB experienced loss of a large number of the experienced subject
    matter engineering experts, scientific, and mathematical personnel,
    several MRTFB test facilities are being mothballed or closed. The
    government skills to restore or replace them will be lost over time.
        Investments in new test capabilities in the MRTFB for anticipated
    weapon system technologies and test programs have been increasingly
    awaiting the first program to need them for funding rather than using
    T&E institutional funds. This practice is counter to the intent of
    having an MRTFB; i.e., anticipating and having T&E capabilities in
    place when new development or acquisition programs require them. Some
    new T&E capabilities require long-lead times to plan, budget, and to
    develop to be ready when needed by a program, thus can cause the test
    program schedules to slip, inadequate capabilities to be accepted as
    work-arounds, or not to be tested at all. Furthermore, these facilities
    are normally specialized for the programs paying for them, and may have
    little or no residual use for other programs, particularly when they
    are co-located with the prime contractor's manufacturing facility.
    While the Test Resources Management Center (TRMC) ``Strategic Plan for
    DOD Test and Evaluation Resources'' serves as a good guide to the
    actions required to sustain a capable MRTFB, the TRMC has virtually no
    power to require the military departments to sustain adequate operating
    or investment funding beyond the funds managed through the Central Test
    and Evaluation Investment Program and T&E/S&T Program. As a result, my
    assessment is that in some instances, the MRTFB no longer meet its
    intended purpose of having adequate capabilities and capacity available
    for future acquisition programs.
                                     ______

               Questions Submitted by Senator E. Benjamin Nelson

                             FISCAL RESPONSIBILITY

        18. Senator Ben Nelson. Mr. Sullivan, Dr. Gansler, Dr. Kaminski,
    and Mr. Adolph, cost overruns and fiscal irresponsibility are great
    concerns for me. During his hearing in January, the Secretary of
    Defense stated that many programs that cost more than anticipated are
    built on an inadequate initial foundation, which in my opinion begins
    with defining requirements. It has been reported that many of the
    COCOMs do not believe their requirements are sufficiently represented
    by the current process, which seems to be centered on each Service, not
    the Armed Forces as a whole. This results in duplication of
    capabilities, non-alignment of requirements to capability, and
    interoperability problems, all of which lead to expensive requirements
    creep. Should COCOMs play a lead role in defining needed capabilities?
        Mr. Sullivan. Yes, we believe that the COCOMs should play a
    stronger role in defining capability needs. Requirements determination
    in DOD, particularly for major weapon system programs, continues to be
    driven largely by the military Services. Our work has revealed that
    although the COCOMs--which are responsible for planning and executing
    military missions--are the principal joint warfighting customer in DOD,
    they have played a limited role in determining requirements. In
    analyzing requirements documentation related to new capability
    proposals, we recently reported that most--almost 70 percent--were
    sponsored by the military Services with little involvement from the
    COCOMs. Other studies have also raised concerns that the Services and
    the COCOMS do not routinely collaborate to identify possible joint
    capability solutions.

             The Defense Acquisition Performance Assessment Panel
            concluded that JCIDS resulted in capabilities that did not meet
            warfighter needs in a timely manner and recommended that JCIDS
            be replaced with a COCOM-led requirements process in which the
            Services and defense agencies compete to provide solutions.
             The Defense Science Board similarly reported that
            JCIDS has not provided for increased warfighter influence, but
            instead actually suppresses joint needs in favor of military
            Service interests, and recommended an increase in the formal
            participation role of the COCOMs in the JCIDS process.
             The Center for Strategic and International Studies has
            also recommended that the Joint Forces Command take the lead in
            conducting capabilities development planning for the COCOMs and
            become a formal member of the JROC.

        By continuing to rely on capability needs defined primarily by the
    Services, DOD may be losing opportunities for improving joint
    warfighting capabilities and reducing the duplication of capabilities.
        Dr. Gansler. The Packard Commission and (subsequently) Goldwater-
    Nichols stated that all requirements were ``joint'' (since that is the
    way we fight). The JROC was set up to represent the COCOMs in the
    ``requirements process.'' Interoperability should always be a critical
    design and test requirement. The COCOMs should be the source of
    requirements for both ``urgent needs'' and for JCTDs.
        Dr. Kaminski. I believe that the COCOMs should play a strong role.
    But to do so, they need to develop the expertise needed to engage in
    cost, schedule, and performance tradeoffs. Even top level requirements
    need continuing assessment based upon evolving value propositions. As
    we discover more about costs and performance during execution, good
    program managers should be interacting with requirements developers to
    adjust as need to provide best value for the user. This requires some
    flexibility in cost allocation, and it also places demands upon
    requirements developers to become involved in tradeoff decisions. To be
    effective, requirements developers as well as acquisition managers must
    have experience with systems engineering tools and techniques, and both
    must have sufficient domain experience to engage productively in cost-
    performance tradeoffs.
        Mr. Adolph. I don't have the background or experience to respond to
    this question.

        19. Senator Ben Nelson. Mr. Sullivan, Dr. Gansler, Dr. Kaminski,
    and Mr. Adolph, how successful are DOD's processes for matching
    warfighter needs with resources?
        Mr. Sullivan. GAO has previously reported that DOD's inability to
    achieve a balanced portfolio of weapon system programs that matches
    joint warfighting needs with available resources is due in part to the
    fragmented structure of the Department's requirements, funding, and
    acquisition processes. The JCIDS process has not been effective in
    prioritizing needs from a joint, Department-wide perspective, and it
    largely approves capability needs without accounting for the resources
    or technologies that will be needed to acquire the desired
    capabilities. Resource allocation decisions in DOD take place through
    the PPBE process, which is separate from JCIDS. PPBE largely allocates
    resources on a Service-by-Service basis and does not effectively link
    resources to capabilities. In addition, the process allows too many
    programs to start development with unreliable cost estimates and
    without a commitment to fully fund them. Until DOD establishes a more
    integrated approach to weapon system acquisition, it will continue to
    struggle to effectively prioritize warfighting needs, make informed
    trade-offs, and achieve a balanced mix of weapon systems that are
    affordable, feasible, and provide the best value to the warfighter.
        While DOD recently reported that the JROC is doing more to seek out
    and consider input from the COCOMs through regular trips and meetings
    to discuss capability needs and resourcing issues, we found that many
    of the COCOMs still do not believe their needs--reflected through the
    IPL process--are sufficiently addressed through DOD's requirements
    process. In order to grant the COCOMs more influence in identifying
    requirements, DOD should consider providing the COCOMs with additional
    resources to establish robust analytical capabilities for identifying
    and assessing their capability needs. In addition, DOD and the Joint
    Staff should ensure the JROC gives equal consideration to COCOM needs.
        Dr. Gansler. As Secretary Gates has stressed, we need a better
    ``balance'' between likely warfighter needs and resource allocations.
    Today, the budgets are unbalanced in the direction of potential, future
    peer competitors.
        Dr. Kaminski. The success is mixed. Warfighters are often very able
    to address their immediate needs, but addressing future needs can be
    challenging. Warfighters also need to be open to new conops--new
    approaches to performing their mission that can be derived from new,
    enabling technologies. Similarly, developers and cost estimators are
    more able to address near-term estimates involving technologies that
    are mature and familiar, and less able to do so with future
    technologies that are less mature and less familiar. Matching needs and
    resources must be a continuing, iterative process. This process must
    have a solid foundation in systems engineering, and the systems
    engineering must in turn have some foundation in M&S which is validated
    by building and testing. Effective value proposition assessments will
    occur only with skilled, trained, and experienced personnel from
    requirements and acquisition communities working together within a
    systems engineering framework to refine needs and solutions.
        Mr. Adolph. I don't have the background or experience to respond to
    this question.

                    TECHNOLOGICAL CHANGE AND WEAPONS SYSTEMS

        20. Senator Ben Nelson. Mr. Sullivan, Dr. Gansler, Dr. Kaminski,
    and Mr. Adolph, one of the challenges we face with new technology and
    the rapid advancement of technology is the desire to outfit our
    military men and women with the latest and most advanced equipment.
    While we all want to provide our military with the latest advances in
    technology, I worry that as technology changes we are in an ever state
    of trying to upgrade a system while it is in development. How can we
    best handle constant technology changes for weapons systems that take
    years to develop and deliver?
        Mr. Sullivan. First, technology development must be placed on the
    critical path of any new acquisition program. Technology development
    requires trial and error, discovery, and invention, which should all be
    done in a S&T environment before technologies are brought onto
    programs. Second, GAO has found that successful commercial companies
    use an evolutionary or incremental approach that reduces development
    risks and enables quicker delivery of products to the customer. These
    commercial companies have implemented an evolutionary approach by
    establishing time-phased plans to develop a new product in increments
    (5 years or less) based on technologies and resources achievable now
    and later. They set requirements for products that require only
    technologies that they know will work, and wait to put more risky
    technologies on future increments once their risks have been
    diminished. An open architecture approach to development is also
    important so that product components, such as technologies, can be
    upgraded or added in subsequent product development increments, without
    having to do a major redesign of the product.
        Historically, DOD's approach to developing new weapon systems often
    attempts to satisfy the full capability in a single step, ``big-bang''
    approach regardless of the design complexity or the maturity of the
    technologies. Under this approach, the warfighter can wait over a
    decade to receive any new capabilities and by the time the capabilities
    are delivered, they may be out of date. Implementing knowledge-based
    evolutionary acquisition practices--where individual program increments
    are defined on the basis of mature technologies and a feasible design
    that incorporates an open architecture approach--would allow DOD to
    more effectively manage technology changes in acquisition programs.
    Shorter, more manageable development cycles would also provide the
    warfighter with useful technologies quicker and provide more confidence
    that technologies can be developed within program cost estimates.
        Dr. Gansler. See my answer to question #13.
        Dr. Kaminski. I have always found that the mark of a good program
    manager is a large lower right hand desk drawer. In this drawer are
    placed all the ideas for incorporation of new technology advancements
    until the first version of the system is fielded. At that time the
    drawer is opened, and representatives from the requirements and
    acquisition communities meet and apply systems engineering tools and
    techniques to assess shortfalls in the fielded system, evolving needs,
    and evolving technologies to examine value propositions and decide what
    to incorporate in the next block of that fielded system.
        Mr. Adolph. With platforms and major systems being procured less
    frequently and taking many years in development, there is often a
    tendency to over-reach since another opportunity to provide advanced
    technology to the warfighter may not occur for many years. The first,
    and most, beneficial action would be to work toward shortening the
    acquisition cycle. For various reasons, the acquisition cycle has
    become overly extended, due in part, as discussed in my response to
    question #13, to initiating acquisition programs based on immature
    technologies. This trend must be reversed. Recommendations have been
    made repeatedly in numerous studies that would result in a shortened
    cycle.
        One approach to a shortened cycle for large programs would be to
    incorporate new technologies in ``block upgrades.'' This approach has
    been successful for many defense systems, sometimes as a result of
    necessity and sometimes as preplanned improvements. The latter approach
    has been implemented on many aircraft programs where planned multi-
    staged improvement programs are executed at regular intervals.
    Strategies such as spiral acquisition and pre-planned product
    improvement have been pursued with various degrees of success. If too
    many new technologies are incorporated into a system, the risk is
    compounded by the complex task of integrating advanced technologies
    with sometimes insufficient technical maturity. These risks may be
    mitigated through disciplined processes to properly evaluate technology
    maturity, larger investments in technical domain expertise as well as
    in systems engineering. More emphasis should also be placed on analysis
    of alternatives, to include an OSD-level review. Designs that support
    ``plug and play,'' e.g. standard interfaces are also a partial
    solution. Producing the technology and integrating it into the
    logistics system as well as training maintenance personnel, et cetera,
    can still be challenges if the technology has never been deployed to
    the warfighter. (See also my response to question #13.)

                             ACQUISITION PERSONNEL

        21. Senator Ben Nelson. Mr. Sullivan, Dr. Gansler, Dr. Kaminski,
    and Mr. Adolph, a consistent theme throughout the hearing was the
    insufficiency of the current acquisition workforce. What would your
    recommendation be for increasing the number of Active Duty billets
    specifically for acquisition?
        Mr. Sullivan. Although we did not examine DOD's decisionmaking
    process for using military versus civilian personnel, we recently
    reported that DOD lacks critical Department-wide information to
    determine the sufficiency of its acquisition workforce to meet its
    mission needs. We recommended that DOD identify the number and skill
    sets of its total acquisition workforce--including civilian, military,
    and contractor personnel--and conduct analyses using this information
    to inform acquisition workforce decisions regarding the appropriate mix
    of civilian, military, and contractor personnel. We are encouraged by
    the Department's recent announcement that it plans to significantly
    increase the size of DOD's acquisition workforce by converting 11,000
    contractors and hiring an additional 9,000 government acquisition
    professionals by 2015--beginning with 4,100 in fiscal year 2010.
        Dr. Gansler. In the Commission I chaired (the ``Commission on Army
    Acquisition and Program Management in Expeditionary Operations'') we
    stressed that it is not just numbers that matter, but rank as well as
    experience. The senior military acquisition positions should be staffed
    with General Officers that have extensive acquisition experience. As to
    numbers, we should at least get back the 25 percent that Congress
    mandated be cut in fiscal year 1996 (which, across the DOD acquisition
    workforce, military and civilian, is approximately 50,000 additions--of
    which perhaps 20 percent are military).
        Dr. Kaminski. Numbers alone aren't the answer. We need an
    appropriate mix of quality and quantity. We need personnel able to
    provide early and continuing systems engineering and closely-coupled
    development planning. We need: a) sufficient personnel (in both
    government and industry) with adequate education, training and domain
    experience (this includes personnel in requirements development as well
    as in acquisition); and b) sufficient front end investment in the tools
    necessary to understand the key tradeoffs in cost/schedule/performance,
    and to identify and address the key risks in a systematic manner.
        Mr. Adolph. As a minimum, the government acquisition workforce,
    which includes military and civilian personnel, should be increased to
    the levels that existed in the mid-1990s prior to the congressional and
    Service reductions. The military/civilian acquisition position mix
    should be left to the individual Services as each is procuring
    different commodities, and has a different philosophy regarding a
    career military acquisition workforce. As I noted in my opening
    statement, during a time of increased programmatic and technical
    complexity, there has been a loss of a large number of the most
    experienced management and technical personnel, without an adequate
    replacement pipeline. Solutions to the acquisition problems must begin
    with reconstituting a trained and experienced government workforce,
    which includes programs managers, domain subject matter experts as well
    as systems engineers, contracts personnel, testers and evaluators.

        22. Senator Ben Nelson. Mr. Sullivan, Dr. Gansler, Dr. Kaminski,
    and Mr. Adolph, to ensure we get the quality and expertise we require,
    how do you recommend we recruit and retain this force?
        Mr. Sullivan. DOD has just announced plans to substantially
    increase its in-house acquisition workforce over the next few years.
    Our recent report on DOD's acquisition workforce included practices of
    leading companies that could provide insights for DOD as it moves
    forward to hire and retain these employees. We found, for example, that
    the companies employed a variety of recruitment and retention
    initiatives. More importantly, some companies assessed their efforts at
    filling workforce gaps by tracking data on specific recruiting and
    retention metrics. One company assessed the quality of their hiring
    sources by assessing the performance of new hires over their first 2
    years.
        Dr. Gansler. Promotion potential and acquisition-experienced
    leaders are the keys to recruiting, developing, and retaining top
    acquisition personnel.
        Dr. Kaminski. You are quite right in noting that we must address
    retention as well as recruitment to build the requisite work force. We
    want people who want to make a difference. When we recruit these
    people, we must expect that we won't keep them if they can't see that
    they in fact are making a difference. Excessive (and growing) time from
    program initiation to fielding is a big problem for young people who
    want to see the impact of their work. As this time increases from a few
    years to 15 years or more, it undermines the entire acquisition process
    by causing key participants to ``lose the recipe'', and lose a sense of
    accountability as well as a sense of being able to make a difference.
    When new capabilities are developed and fielded in 5 years, engineers,
    managers, testers, cost analysts, et cetera, are able to benefit from
    and apply the experience gained from a previous program or program
    phase. They can also see the results of their decisions and be held
    accountable. We can also meaningfully employ past performance of the
    contractor as a factor in the award of future programs--an important
    factor in incentivizing contractor performance. This all changes
    dramatically when the time extends to 15 years, and we have five roll-
    overs of management, engineers, cost analysts, and commercial
    technology during this time period. This long and growing time period
    is a result of the inflexibility inherent in our entire system of
    requirements development, budgeting, and acquisition, and it creates a
    vicious cycle in which it further exacerbates the contributors above,
    and they in turn further increase the time and cost growth. We see the
    result when we must discard our current acquisition system in order to
    deal with urgent needs and field systems such as MRAP and jammers to
    counter IEDs by forming and using rapid reaction organizations. This
    cycle must be broken by attacking the root causes.
        Mr. Adolph. When the acquisition workforce reductions were made in
    the late 1990s, they were implemented by encouraging early retirements
    as well as hiring freezes, which often lasted for years. As a
    consequence of the latter, the pipeline of new civilian employees was
    shut off, and today there are large gaps in experience as well as
    inadequate numbers of personnel. Some Service test organizations are
    aggressively addressing these issues and have more insight into what
    works in today's environment than I do. Based on my past experience,
    initial hiring, particularly for positions at remote test locations, is
    the biggest challenge. It requires aggressive recruiting at
    universities, along with co-op programs. The challenge of the work
    environment and the learning experiences, dealing with the latest
    technologies, combined with well-funded full-time advanced degree
    educational opportunities (with associated follow-on career
    commitments), are powerful positive retention forces after people are
    brought on board. On-the-job learning experiences have been diminished
    in some test organizations, where government personnel have been
    relegated to less challenging supporting functions.

        23. Senator Ben Nelson. Mr. Sullivan, Dr. Gansler, Dr. Kaminski,
    and Mr. Adolph, what can we do in the meantime to handle the personnel
    shortage since it will take many years to develop the level of
    expertise required for proper acquisition reform?
        Mr. Sullivan. As we recently reported in our review of DOD's
    acquisition workforce management, DOD is currently implementing
    initiatives aimed at improving the capacity of its acquisition
    workforce using the newly established Acquisition Workforce Development
    Fund. These efforts are focused in three key areas: (1) recruiting and
    retention; (2) training and development; and (3) retention and
    recognition. DOD has also recently announced plans for significantly
    increasing its in-house acquisition workforce over the next few years.
    While DOD integrates these additional employees into its acquisition
    workforce, the Department can also leverage the experience and
    knowledge of its current workforce. DOD can take steps to retain
    experienced personnel to aide in the development of newer staff and to
    ensure it maintains and transfers key institutional knowledge.
        Dr. Gansler. To fill the gap, while developing a new group, I would
    bring into the government some middle and senior people from industry;
    and I would use contractors (with experience, but without conflicts) to
    fill the rest of the slots.
        Dr. Kaminski. We can begin to hire people with education in
    engineering and systems engineering. We can send our current personnel
    back to universities to enhance their education. We can send our
    current personnel to industry to gain domain experience. We can bring
    back retirees on a part time basis to provide mentoring to our new
    hires and help them gain domain experience. We need to do all of these.
        Mr. Adolph. During the period when the normal pipeline is being
    replenished, Centers of Excellence must be created in selected test and
    acquisition engineering support organizations, with a focus on
    technical domain subject matter expertise. These cadres of expertise
    will provide the basic oversight functions such as those I outlined in
    my answer to question #16. The first step is to provide oversight in
    key technical disciplines at key programmatic times, e.g., RFP
    preparation, source selection, TEMP preparation, test reporting to
    support program reviews; transitioning to more active, continuous
    involvement as the workforce increases. The Services may be reluctant
    to take this approach. These Service cadres can also be augmented by
    subject matter experts from FFRDCs. The use of red teaming by outside
    experts (e.g., cross-Service, FFRDCs, and SETAs) should also be
    employed to provide the requisite expertise, as well as a measure of
    independence which is too often lacking in government program office-
    centric reviews.

          MANDATORY PROCEDURES FOR MAJOR DEFENSE ACQUISITION PROGRAMS

        24. Senator Ben Nelson. Mr. Sullivan, Dr. Gansler, Dr. Kaminski,
    and Mr. Adolph, what oversight and reporting mechanisms do you
    recommend we establish to ensure the Mandatory Procedures for Major
    Defense Acquisition Programs (MDAPs) meet warfighter requirements while
    eliminating redundancy and wasteful spending?
        Mr. Sullivan. DOD has recently revised its acquisition policy in
    ways intended to provide key Department leaders with the knowledge
    needed to make informed decisions before a program starts and to
    maintain disciplined development once it begins. The revised policy
    recommends the completion of key systems engineering activities before
    the start of development, requires early prototyping, establishes
    review boards to identify and mitigate technical risks, and implements
    early milestone reviews. If implemented, these policy changes could
    help programs replace risk with knowledge, thereby increasing the
    chances of developing weapon systems within cost and schedule targets
    while meeting user needs. To achieve improved outcomes in acquisition
    programs, DOD must ensure its policy changes are consistently
    implemented and reflected in decisions on individual programs. However,
    DOD could do more by requiring new programs to have manageable
    development cycles and establish knowledge-based cost and schedule
    estimates. Limiting the length of development cycles would make it
    easier to more accurately estimate costs, predict future funding needs,
    effectively allocate resources, and hold decision makers accountable.
        The acquisition reform legislation recently proposed by the Senate
    Armed Services Committee should also help achieve further improvements.
    Provisions increasing the emphasis on systems engineering, requiring
    early preliminary design reviews, and strengthening independent cost
    estimates and technology readiness assessments should make the critical
    front end of the acquisition process more disciplined. Establishing a
    termination criterion for critical cost breaches could help prevent the
    acceptance of unrealistic cost estimates at program initiation. In
    addition, having greater COCOM involvement in determining requirements
    and greater consultation between the requirements, budget, and
    acquisition processes could help improve DOD's efforts to balance its
    portfolio of weapon system programs.
        While policy revisions and legislation may lead to improvements,
    they will not be effective without changes to DOD's overall acquisition
    environment and the incentives that drive the behavior of its
    decisionmakers, the military Services, program managers, and the
    defense industry. Changing the environment will require strong
    leadership and accountability within the Department.
        Dr. Gansler. I believe we currently have enough oversight and
    reporting mechanisms for MDAPs (and more would only slow the system
    down even more). The critical need is to use these mechanisms more
    effectively--with experienced personnel (who know what to look for, and
    what questions to ask).
        Dr. Kaminski. We first need to realize that ``one size does not fit
    all.'' For example, programs dealing with countering IEDs need cycle
    times measured in weeks, while a next generation bomber program must
    plan for fielding and support cycles measured in years.
        In most cases today the program manager's authority is diffused by
    many levels of oversight in both the Department and in Congress, and
    the financial and performance constraints imposed do not allow
    sufficient freedom of action to apply informed judgment in a timely
    manner. Flexibility is further limited by application of a ``one-size-
    fits-all'' approach imposed by the DOD 5000 system, and the oversight
    practiced by the DOD and Congress. A program manager needs the freedom
    to tailor the acquisition approach to the problem, to ensure that the
    program response time will fall within the response time of the threat,
    and to apply a variety of tools and techniques (such as the use of
    prototypes, competitive prototypes, M&S, critical subsystem and
    component demonstration). For this to work, we need program managers
    with the education, training, and domain experience needed to enable
    timely responses and excellent judgment relevant to the domain. They
    need the flexibility to apply their expertise.
        Mr. Adolph. I believe the current oversight and reporting
    mechanisms, when combined with the changes in the proposed legislation,
    will provide adequate direction and guidance for the acquisition
    process. The key is putting more discipline in every step of the
    process by the Services and OSD. This was achieved in numerous past
    successful programs in the 1960s, 1970s, and 1980s without being overly
    bureaucratic. One impediment is the recent trend towards contractual
    vehicles which relieve development contractors of reporting
    requirements, including regular status reports on costs and technical
    performance. The government must have continued access to cost and
    performance data to provide effective oversight.
        Most of the basic policies and directives for the acquisition
    process represent best practices based on decades of experience. The
    fundamental problem is a lack of adequate discipline throughout the
    process. This begins with proper definition of operational requirements
    at the outset of a program and stating those requirements in terms that
    are reasonable and quantifiable for design and test purposes. Another
    major issue is the lack of adequate technology maturity (See my
    response to question #13.)
        There have been some problems with military specifications and
    standards in the past, in that they were not properly tailored to the
    system to be procured. However, these standards have evolved and were
    updated over decades and were, for the most part, excellent compendiums
    of best practices and lessons learned. If properly used, they provide
    guidance for system development to ensure a systematic and disciplined
    approach. Proper application of these guidance documents also ensures
    that problems, which are a normal part of any complex high technology
    development, are identified early. Many of these documents have been
    allowed to atrophy over the last several years or have been eliminated.
    Previously eliminated specifications and standards should be
    selectively updated and reinstated; while retaining the option to use
    commercial specifications and standards when available, provided the
    latter adequately address the requisite military performance and
    suitability requirements in the intended operating environment.
    Finally, the most important ingredient remains a capable and
    experienced government acquisition workforce with equally capable
    leaders, who remain on the job long enough to realize the consequences
    of their programmatic decisions.
                                     ______

                  Questions Submitted by Senator David Vitter

                        PERSONNEL IN ACQUISITION REFORM

        25. Senator Vitter. Dr. Gansler, a recurring message from the panel
    is the importance of improving the quality and experience of personnel
    in acquisition reform, and I'm glad that Chairman Levin and Ranking
    Member McCain included section 206 in their acquisition reform bill to
    reward excellence. This section requires the establishment of a program
    recognizing excellent performance and the award of cash bonuses.
    However, it was stated that cash isn't necessarily the greatest
    incentive for encouraging excellence, and making a difference is the
    best incentive. In your opinion, what can DOD implement in an
    excellence program to demonstrate improvements so that personnel can
    track and actually see their work reflected in tangible results to
    provide and maintain motivation?
        Dr. Gansler. Combining appropriate metrics with rewards for success
    (including recognition, and even token financial rewards) are
    desirable--but the managers must be allowed to have sufficient
    flexibility to exercise their judgment, if they are to succeed.

        26. Senator Vitter. Dr. Gansler, how much of the problem in
    attracting the right quantity of qualified acquisition personnel is a
    product of the overall Federal employee hiring rules established by
    Office of Personnel Management (OPM) or the specific rules established
    by DOD?
        Dr. Gansler. The DOD must be able to compete with industry for
    ``the best and brightest.'' The process must be flexible and fast--yet
    fair. Salary will not be the driver (government jobs offer the rewards)
    but the process can be neither bureaucratic nor political.

        27. Senator Vitter. Dr. Gansler, last year, Congress authorized
    $300 million to rebuild the acquisition workforce. Do you believe this
    is sufficient?
        Dr. Gansler. It is an important start, and will likely need to be
    continued for a few years.

        28. Senator Vitter. Dr. Gansler, cost overruns were endemic in the
    1980s and 1990s when the acquisition workforce was robust. Is it likely
    then that adding more people will solve the problem of cost overruns?
        Dr. Gansler. As I noted in question #21, numbers alone are not the
    answer, senior and experienced people with an improved acquisition
    process (as covered in my answers above--especially numbers 1, 8, and
    13).

        29. Senator Vitter. Dr. Gansler, it seems that there is a problem
    with keeping quality program managers in their roles for an extended
    period of time. How can we ensure continuity at these positions?
        Dr. Gansler. Goldwater-Nichols intended the program managers to
    remain at least through the next milestone. In the Packard Commission
    we emphasized that they be allowed to be promoted in place (so that the
    extended period is a reward, not a career-limiting assignment).

                       FUNDING FOR DEVELOPMENTAL TESTING

        30. Senator Vitter. Mr. Adolph, to what extent does a lack of
    funding for developmental testing (DT) early in a program's schedule
    result in cost increases and delays later in its schedule?
        Mr. Adolph. The lack of adequate funding for DT has a major adverse
    impact on developmental cost and schedule. There are numerous issues
    relating to DT funding. The weapons timelines for RDT&E continue to
    increase, driven by inadequate resourcing of the entire developmental
    process as well as the inclusion of immature technologies in systems.
    Inadequate funding of DT, which includes the number of test assets,
    results in delayed identification and correction of problems, many of
    which are a normal part of a high-technology developmental program.
    Late identification results in more difficulty and expense involved in
    fixes. Most recent programs have had an inadequate number of assets to
    execute a robust DT program in a timely manner. When there are an
    inadequate number of test assets, the DT program is drastically
    impacted, often for months, when a test vehicle must be laid up for a
    retrofit. By the time a program reaches the full-scale platform test
    phase, there is a huge cadre of test and test support personnel who
    cannot be efficiently used whenever a test program comes to a halt. The
    fixed cost of maintaining this cadre often exceeds the incremental
    variable costs of conducting test missions. Another factor in efficient
    testing is the availability of adequate test personnel and test
    facilities. The latter includes physical test support facilities and
    assets, instrumentation, and data processing (see also my response to
    question #17). Numerous developmental programs have been delayed
    because of the lack of adequate facilities, capacity, and people to
    accomplish the basic data processing tasks quickly, as well as
    inadequate number of domain subject matter experts to analyze test
    results. This is exacerbated when earlier involvement by DT&E and OT&E
    personnel is limited by a lack of sufficient experienced personnel.
    Commercial programs routinely resource facilities and personnel to
    conduct testing on a multiple shift basis during critical phases of the
    developmental process.

                            COMPETITIVE PROTOTYPING

        31. Senator Vitter. Dr. Kaminski, how much can competitive
    prototyping help in identifying the best product for warfighters?
        Dr. Kaminski. Competitive prototyping is one arrow in our quiver of
    tools and techniques in development planning. Not all prototyping needs
    to be competitive. Prototyping should not be limited to the full system
    level. There are cases where prototyping is best applied only to
    address major platform integration risk areas, or to critical
    subsystems. Again, I don't believe there is a ``one-size-fits-all''
    solution to prototyping. But I do believe that at a big picture level
    we are not doing enough prototyping in our development planning.
    Reducing risk and developing domain expertise require that we build
    things. M&S are of great value, and so are our computer based analysis
    and design tools, but all of my experience tells me that we need to
    build and test on a continuing basis to make mistakes, learn from those
    mistakes, and apply that learning in the development of new
    capabilities.

        32. Senator Vitter. Dr. Kaminski, the F-16 program, lauded by many
    as an example of how to acquire a capability, was competed to the point
    where the Air Force actually flew competing planes before deciding on
    which defense contractor won the award. Why have we gotten so far away
    from that?
        Dr. Kaminski. We haven't completely departed from that approach. We
    did something very similar in the JSF program. But I agree that we are
    not doing enough. We can do quite a bit more. I believe we can do so
    without increasing costs by spending money upfront to reduce or avoid
    the cost of significant overruns later.

                      JOINT REQUIREMENTS OVERSIGHT COUNCIL

        33. Senator Vitter. Mr. Sullivan, the Vice Chiefs have important
    jobs internally managing their respective Services. Do you believe that
    JROC should instead be staffed with independent personnel who are not
    advocates for a particular Service?
        Mr. Sullivan. GAO has found DOD's requirements process to be too
    Service-centric. The needs of the COCOMs should be given greater
    consideration. There are a number of ways the requirements setting
    process might be changed to address this issue. For example, the
    combatant commanders might supplant the Service Vice Chiefs role in the
    process or might be given an equal say in the process. Several outside
    reviews have suggested such solutions:

             The Beyond Goldwater-Nichols study team recommended a
            more ``joint'' JROC in which the Service Vices are replaced by
            COCOM Deputies and civilians responsible for requirements
            policy.
             The Defense Acquisition Performance Panel (DAPA)
            recommended replacing the current requirements setting process
            with a combatant commander-led requirements process in which
            the Services and defense agencies compete to provide solutions.
            Each of the combatant commanders would be tasked to prepare
            extended planning annexes to each of their operational and
            contingency plans (to be updated on a 2-year cycle) that would
            provide a 15-year forecast of both capability gaps and excesses
            relative to mission requirements.
             Some members of the Defense Science Board Summer Study
            on Transformation felt that the requirements process continues
            to be dominated by the force providers and the Joint Staff and
            that COCOM needs are under-represented. They proposed making
            the COCOMs more equal partners with the force providers from
            the beginning of the process, particularly when identifying
            capabilities needed to carry out the Department's operational
            missions.

        We believe that these proposals deserve further consideration as a
    means to help improve the Department's ability to balance joint
    warfighting needs.

        34. Senator Vitter. Mr. Sullivan, should we have a new, independent
    JROC that could hear all advocates for all programs and then recommend
    continuation or elimination of the program based on current and likely
    future threats to the United States?
        Mr. Sullivan. Because many major defense acquisition programs have
    ambitious and lengthy product developments--in some cases, delivery of
    a weapon system to the warfighter takes as long as 15 years--there can
    be a significant amount of time that elapses between JROC reviews and
    changes in the threat environment can occur. Before a major weapon
    program is approved to begin system development, the JROC must validate
    the capabilities to be developed, including the system's key
    performance parameters. The JROC also validates the production elements
    of a program before it is approved to start production. For programs
    that have lengthy development cycles, more frequent reviews may be
    necessary to assess whether programs are worth continuing in light of
    current and likely future threats. In prior work we have conducted on
    how successful companies manage their portfolios of product development
    efforts, we found that the companies revisit their investment decisions
    at multiple stages throughout product development to ensure products
    are still of high value.

        35. Senator Vitter. Mr. Sullivan, should the Services be required
    to budget their programs to the cost estimates provided by a new
    Director of Independent Cost Assessment?
        Mr. Sullivan. Budgeting programs to an independent cost estimate
    would be an improvement, however, for cost estimates to be effective
    they must be based on a high degree of knowledge about requirements,
    technology, design, and manufacturing.
        A reliable cost estimate helps ensure a program's projected funding
    needs are adequate to execute the program. Less than a quarter of the
    48 programs in GAO's annual assessment of weapon system programs that
    provided data used the estimate made by DOD's Cost Analysis Improvement
    Group (CAIG) as a basis for the program's baseline, while almost 70
    percent of the programs used the program office or Service cost
    estimate. While cost estimates from the CAIG can underestimate a
    program's costs by billions of dollars, GAO has previously found that
    these independent estimates generally underestimate costs by a smaller
    amount than program office and Service estimates.
        Cost estimates are inaccurate in part because they are based on
    limited knowledge about the requirements, technologies, design
    maturity, and the time and funding needed to execute a program. GAO has
    found, for example, that program Cost Analysis Requirements Documents--
    used to build the program cost estimate--often lack sufficient detail
    about planned program content for developing sound cost estimates.
    Without this knowledge, cost estimators must rely heavily on parametric
    analysis and assumptions about system requirements, technology, and
    design maturity, and the time and funding needed. The assumptions used
    in developing estimates also tend to be overly optimistic. Furthermore,
    cost estimates that lack knowledge and rely on assumptions have
    inherently high levels of risk and uncertainty that are not typically
    communicated to decision makers.

        36. Senator Vitter. Mr. Sullivan, with the annual Selected
    Acquisition Reports and current Nunn-McCurdy reporting requirements, is
    Congress suffering from a lack of oversight and reports from DOD?
        Mr. Sullivan. In terms of the reports it receives from DOD,
    Congress is suffering from a lack of insight into the risks on programs
    that could lead to cost, schedule, and performance shortfalls. The
    Selected Acquisition Reports and Nunn-McCurdy reporting that Congress
    gets from DOD on its major defense acquisition programs primarily
    report on program outcomes to date--including quantitative measures of
    cost, schedule, and performance over time. While these are useful
    indicators of the health of acquisition programs and whether they are
    meeting their intended goals, they have limited predictive value. DOD
    could improve the information it provides to Congress by reporting on
    knowledge metrics that evaluate whether programs have attained certain
    levels of technology, design, and manufacturing knowledge by key points
    in the acquisition process. This reporting would facilitate the
    identification of potential problems that could lead to cost, schedule,
    and performance shortfalls before they occur.

        37. Senator Vitter. Mr. Sullivan, is there any credence to the
    concern that program managers have ``too many'' reports to file and are
    not focusing on managing their programs?
        Mr. Sullivan. Oversight and reporting requirements need to be
    evaluated both from the perspective of the time they take and the value
    they provide. In the past, we have reported on concerns in both of
    those areas. In a November 2005 report, we reported that program
    managers and program executive officers commented that they spend too
    much time producing data that is not strategic or very useful to them.
    In fact, more than 90 percent of 126 respondents to our survey of
    program managers said that they spent either a moderate, great, or very
    great extent of their time representing their program to outsiders and
    developing and generating information about program progress.
        We have also found that the reporting that is done does not include
    key information that decisionmakers need to decide if a program is
    ready to proceed into the next phase of the acquisition process. For
    example, in transitioning from system integration to system
    demonstration, we have recommended that DOD ensure the capture of
    knowledge about the completion of engineering drawings; completion of
    subsystem and system design reviews; agreement from all stakeholders
    that the drawings are complete; and identification of critical
    manufacturing processes, among other indicators. In the transition to
    production, we recommended that DOD capture knowledge about production
    and availability of representative prototypes along with statistical
    process control data. Our 2005 report indicated that a relatively small
    percentage of programs used these knowledge indicators. For example,
    only 32 percent of 126 program managers who responded to our survey
    said they used design drawing completion extensively to measure design
    maturity; only 26 percent said they used production process controls to
    a great extent. Even fewer program managers reported that their
    immediate supervisor used these measures extensively to evaluate
    progress. Broader use and reporting of these types of metrics would
    help decisionmakers gauge progress and ensure that programs managers
    are: (1) conducting activities to capture relevant product development
    knowledge; (2) providing evidence that this knowledge has been
    captured; and (3) holding decision reviews to determine that the
    requisite knowledge has been captured before proceeding to the next
    phase of the acquisition process.

        [Whereupon, at 11:52 a.m., the committee adjourned.]