[House Hearing, 111 Congress]
[From the U.S. Government Publishing Office]



                         [H.A.S.C. No. 111-63]

                         MEASURING PERFORMANCE:

                            DEVELOPING GOOD

                          ACQUISITION METRICS

                               __________

                                HEARING

                               BEFORE THE

                  PANEL ON DEFENSE ACQUISITION REFORM

                                 OF THE

                      COMMITTEE ON ARMED SERVICES

                        HOUSE OF REPRESENTATIVES

                     ONE HUNDRED ELEVENTH CONGRESS

                             FIRST SESSION

                               __________

                              HEARING HELD

                              MAY 19, 2009

                                     
[GRAPHIC(S) NOT AVAILABLE IN TIFF FORMAT]





                  U.S. GOVERNMENT PRINTING OFFICE
50-297                    WASHINGTON : 2010
-----------------------------------------------------------------------
For sale by the Superintendent of Documents, U.S. Government Printing 
Office Internet: bookstore.gpo.gov Phone: toll free (866) 512-1800; DC 
area (202) 512-1800 Fax: (202) 512-2104  Mail: Stop IDCC, Washington, DC 
20402-0001





                  PANEL ON DEFENSE ACQUISITION REFORM

                  ROBERT ANDREWS, New Jersey, Chairman
JIM COOPER, Tennessee                K. MICHAEL CONAWAY, Texas
BRAD ELLSWORTH, Indiana              DUNCAN HUNTER, California
JOE SESTAK, Pennsylvania             MIKE COFFMAN, Colorado
                Andrew Hunter, Professional Staff Member
                 John Wason, Professional Staff Member
                     Alicia Haley, Staff Assistant














                            C O N T E N T S

                              ----------                              

                     CHRONOLOGICAL LIST OF HEARINGS
                                  2009

                                                                   Page

Hearing:

Tuesday, May 19, 2009, Measuring Performance: Developing Good 
  Acquisition Metrics............................................     1

Appendix:

Tuesday, May 19, 2009............................................    23
                              ----------                              

                         TUESDAY, MAY 19, 2009
       MEASURING PERFORMANCE: DEVELOPING GOOD ACQUISITION METRICS
              STATEMENTS PRESENTED BY MEMBERS OF CONGRESS

Andrews, Hon. Robert, a Representative from New Jersey, Chairman, 
  Panel on Defense Acquisition Reform............................     1
Coffman, Hon. Mike, a Representative from Colorado, Panel on 
  Defense Acquisition Reform.....................................     2

                               WITNESSES

Fitch, David P., USN (Retired), Director, AT&L Leadership 
  Learning Center of Excellence, Defense Acquisition University..     7
Nussbaum, Dr. Daniel A., Visiting Professor, Department of 
  Operations Research, Naval Postgraduate School.................     9
Patterson, J. David, USAF (Retired), Executive Director, National 
  Defense Business Institute, University of Tennessee............     5

                                APPENDIX

Prepared Statements:

    Andrews, Hon. Robert.........................................    27
    Coffman, Hon. Mike...........................................    28
    Dillard, John T., Senior Lecturer, Graduate School of 
      Business and Public Policy, Naval Postgraduate School......    77
    Fitch, David P...............................................    57
    Nussbaum, Dr. Daniel A.......................................    66
    Patterson, J. David..........................................    32

Documents Submitted for the Record:

    [There were no Documents submitted.]

Witness Responses to Questions Asked During the Hearing:

    Mr. Andrews..................................................    83

Questions Submitted by Members Post Hearing:

    [There were no Questions submitted post hearing.]
 
       MEASURING PERFORMANCE: DEVELOPING GOOD ACQUISITION METRICS

                              ----------                              

                  House of Representatives,
                       Committee on Armed Services,
                          Defense Acquisition Reform Panel,
                             Washington, DC, Tuesday, May 19, 2009.
    The panel met, pursuant to call, at 8:00 a.m., in room 
2212, Rayburn House Office Building, Hon. Robert Andrews 
(chairman of the panel) presiding.

OPENING STATEMENT OF HON. ROBERT ANDREWS, A REPRESENTATIVE FROM 
   NEW JERSEY, CHAIRMAN, PANEL ON DEFENSE ACQUISITION REFORM

    Mr. Andrews. Good morning, ladies and gentlemen. The panel 
will come to order. Appreciate your attendance. I especially 
want to thank the witnesses for their very diligent 
preparation. We have had the opportunity to review the 
testimony ahead of time. It looks like we are going to have a 
very engaging and meaningful discussion this morning.
    This is the last of our series of hearings looking at the 
first question that we are going to be looking at in our 
inquiry. The members will recall that we are proceeding on a 
series of questions, the first of which is, ``Can we design a 
series of metrics that accurately measure the difference, if 
any, between the price paid by the taxpayers and the value 
received by the taxpayers and the warfighters for the systems 
and services that we are buying?''
    We are going to proceed after today's hearing with our 
second category of questions, which is really hypotheses as to 
what has gone wrong. We are going to have a series of panels 
talk about their theories and analyses of why we have a 
difference between the price paid and the value received.
    The first of our two panels on this metrics question dealt 
with the sort of measuring the orthodox algorithms that are 
used. In the major weapons system side, we had a panel on that, 
and on the services side, we had a panel on that.
    The purpose of today's panel is to bring in some people who 
we think perhaps look at the whole question through a different 
prism. They are willing to give us some new perspectives 
through which we can analyze the difference between the price 
we pay and the value that we receive.
    We have three witnesses with a wealth and breadth of 
experience in the acquisition field, but also who I think 
benefit from what I would call a healthy distance from the 
daily responsibilities for that function, so they can give us a 
perspective that marries experience with a fresh perspective. 
And hopefully, we will be able to use the testimony of these 
witnesses to go forward, draw some conclusions about the best 
metrics to use in our work, and then proceed with our work in 
analyzing the various hypotheses given for why we have suffered 
these cost overruns.
    I also want to take a moment and thank Mr. Conaway, who 
can't be with us this morning but who is ably represented by 
Mr. Coffman, and all the members of the panel for their 
excellent contribution to the work on the Weapons Acquisition 
System Reform Through Enhancing Technical Knowledge and 
Oversight (WASTE TKO) bill, which has passed the House. We will 
be meeting with the Senate in formal conference at 4:30 this 
afternoon to, I believe, put the finishing touches on that 
conference report so we will be able to proceed to the floor in 
short order on that, get the bill on the President's desk. Each 
member of the panel has made a very significant and welcome 
contribution to that effort.
    Having said that, as the chairman and Mr. McHugh have said, 
we think that that effort covers about 20 percent of the 
landscape. It looks at major weapons acquisition, which 
certainly needed to be examined. But there are so many other 
areas that fall outside of that that the panel has to pursue, 
as well as, frankly, reviewing the early stage implementation 
of the law that we believe the President will sign this week.
    So we are by no means at the conclusion of our work. We are 
really at the outset of it. And one thing we would ask the 
witnesses to do this morning is to think about the fact that, 
although the Congress is about to pass major legislation 
dealing with major weapons systems, we have not yet addressed 
hardware that doesn't fall into the major weapons system 
category and the 55 to 60 percent of acquisition that is 
services that we have to look at on behalf of those who wear 
the uniform and on behalf of the taxpayers.
    We are glad that you are here. And at this time, I want to 
turn to Mr. Coffman for his opening statement, and we will 
proceed with the witnesses after that.
    [The prepared statement of Mr. Andrews can be found in the 
Appendix on page 27.]

STATEMENT OF HON. MIKE COFFMAN, A REPRESENTATIVE FROM COLORADO, 
              PANEL ON DEFENSE ACQUISITION REFORM

    Mr. Coffman. Good morning. Mr. Chairman, ladies and 
gentlemen, I would like to extend a welcome on behalf of 
Ranking Member Conaway, as well. He is sorry he could not be 
with us today. I would like to thank the chairman for allowing 
me to take a few introductory remarks in Representative 
Conaway's place.
    Today's hearing is an appropriate follow on to the panel's 
two hearings which focus on how the Department of Defense (DOD) 
and the Government Accountability Office (GAO) currently assess 
performance on major weapons systems and service contracts. The 
purpose of this hearing is to think outside the box on how we 
should measure performance and about what we should be 
measuring and less about the how we are doing it today.
    Our previous hearings revealed that current measures of 
performance tend to break down if the program baseline is 
unrealistic. We want to know if the program can be corrected--
if the problem can be corrected. Furthermore, are there metrics 
beyond cost and schedule performance that are of value, such as 
how closely does the delivery system meet actual warfighter 
needs. Does the time of delivery of operational capabilities 
satisfy warfighter needs? How do we determine whether optional 
or tradable capabilities requested by the warfighter are 
affordable?
    Mr. Chairman, we have a distinguished panel of witnesses in 
front of us today. I have looked at their written testimony, 
and I look forward to their testimony. There are a couple of 
points I would like to highlight. Although Mr. Dillard is not 
testifying today, he did submit a written statement that I 
would like to briefly comment on.
    [The prepared statement of Mr. Dillard can be found in the 
Appendix on page 77.]
    Mr. Coffman. He states that, ``The proliferation of 
autonomous fiefdoms within the department continues to 
increase, with each being a stovepipe of oversight expertise, 
imposing unique reporting requirements, assessments and 
reviews.''
    He goes on to state that, in regards to adding additional 
acquisition workforce professionals, that, ``These new people 
should not be housed in the Pentagon, but instead where the 
execution of programs occur.'' Mr. Chairman, I think this is an 
important point that I believe we need to follow closely.
    Another point of observation I would like to make is in 
regards to requirements in joint programs. We have many 
programs out there that have the word ``joint'' in front of 
them, but they are joint in name only. I understand that the 
current requirements generation process is called the Joint 
Capabilities Integration and Development System, or JCIDS.
    It seems to me that, in order to have a truly joint program 
that the requirements for that program must be borne jointly. 
Yet, I do not believe our current system, or as Mr. Dillard 
describes, current fiefdoms, foster such an environment. So I 
would be interested in hearing from our witnesses in this 
regard.
    I also encourage our witnesses to share their views on 
existing laws and regulations that are particularly helpful or 
not helpful to the Department's efforts to obtain the best 
value and capability for our warfighters. We have heard from 
the Department in two separate hearings about how they 
currently measure performance and value on contracts. What we 
haven't heard enough about is how they should be measuring 
value. Consequently, your input will be greatly appreciated.
    With that, I conclude. And again, thank you, fellow 
members, and thank you, Mr. Chairman. I look forward to the 
witnesses' testimony.
    Mr. Andrews. Mr. Coffman, thank you.
    Without objection, the statement of any other member of the 
panel will be included in the record. And without objection, 
the statement of each of the three witnesses, plus the 
supplemental floor statement, will be included in the record of 
the hearing.
    [The prepared statement of Mr. Coffman can be found in the 
Appendix on page 28.]
    Mr. Andrews. I think each of you are veterans at this 
process. You know that we generally have a five-minute rule 
where we ask for succinct summaries of your written testimony. 
We are going to be a little bit more liberal with that this 
morning, but then we are going to turn as quickly as we can to 
the questions from the panel and get into the give-and-take.
    I am going to read a brief biography of each witness, and 
then, Mr. Patterson, we are going to begin with you.
    Mr. Dave Patterson is the Executive Director of the 
National Defense Business Institute, which he is establishing 
at the University of Tennessee in the College of Business 
Administration. It is an institution inspiring business 
innovation for both government and industry by providing 
practical, sound assistance and creating economically efficient 
and effective defense business and acquisition programs.
    Prior to taking his current duties, he was the Principal 
Deputy Under Secretary of Defense Comptroller. As the Principal 
Deputy, he was directly responsible for advising and assisting 
the Under Secretary of Defense with development, execution and 
oversight of the DOD budget exceeding $515 billion, with annual 
supplemental requests of more than $160 billion.
    From August 2003 to June 2005, Mr. Patterson held duties as 
the Special Assistant to the Deputy Secretary of Defense. In 
that capacity, he was responsible for managing the Deputy 
Secretary of Defense's personal staff, as well as providing 
direction and advice to the Office of the Secretary of Defense 
staff on a wide range of national security operations and 
policy subjects.
    He served in the Air Force from 1970 to 1993, retiring in 
the rank of colonel. During that time, he held responsible 
leadership and management positions, with assignments at the 
air wing level as the C5-A aircraft Commander and Deputy 
Operations Group Commander at Major Command Headquarters, U.S. 
Air Force, the Office of the Chairman, Joint Chiefs of Staff, 
the Office of the Secretary of Defense and Inspector General. 
In 1986, he was the Air Force Fellow at the American Enterprise 
Institute and served in Vietnam, flying 02-As as forward air 
controller.
    Thank you for your service to our country, Mr. Patterson. 
Glad to have you with us.
    Mr. Fitch, Mr. David P. Fitch enlisted in the Navy in 1966. 
He was commissioned in 1968 after graduation from San Jose 
State College and Aviation Officer Candidate School in 
Pensacola, Florida.
    In 1969, after fixed and rotary wing flight training, he 
was designated a naval aviator and a distinguished naval 
graduate. He retired from the Navy in 1998 as a captain, 
following a career that included three operational and major 
acquisition command assignments.
    Mr. Fitch culminated his tour in the Navy as the major 
program manager for the international and joint multi-
functional information distribution system. He was a 1997 
recipient of both the David Packard Excellence in Acquisition 
Award and the Department of Defense Value Engineering Award.
    In 2001, after nearly three years in the defense industry, 
Mr. Fitch joined the faculty of the Defense Acquisition 
University, teaching and consulting acquisition executives and 
program management and other acquisition disciplines.
    In 2006, he led a major independent study of the Coast 
Guard Deep Water program. In July of 2008, he assumed the 
position of Director, Acquisition, Technology and Logistics 
(AT&L) Leadership Learning Center of Excellence after nearly 
seven years as the Dean of the Defense Systems Management 
College.
    Mr. Fitch holds degrees in business and industrial 
management from San Jose State college, an M.S. in education 
from the University of Southern California, and he is a 
graduate with highest distinction from the Naval War College in 
Newport, Rhode Island. Thank you, Mr. Fitch, for your service 
to our country, and we are glad you are with us this morning.
    Dr. Daniel Nussbaum, from 2004 to the present, has been a 
visiting professor at the Naval Postgraduate School operations 
research department. From 1999 to 2004, he was a principal at 
Booz Allen Hamilton, responsible for a broad range of costs, 
financial and economic analyses with clients across the 
government and commercial spectrum.
    From 1996 to 1999, he was the director of the Naval Center 
for Cost Analysis at the Office of the Assistant Secretary of 
the Navy for Financial Management and Comptroller here in D.C. 
From 1987 to 1996, he was division head of the Naval Center for 
Cost Analysis. From 1982 to 1986, he was Deputy Director and 
Acting Director for operations research and cost analysis 
divisions of the Naval Air Systems Command in Washington, D.C.
    Again, a very distinguished servant of our country, three 
excellent people with experience and fresh insight. We are 
happy to have you with us.
    Mr. Patterson, we would like you to begin with your 
testimony.

  STATEMENT OF J. DAVID PATTERSON, USAF (RETIRED), EXECUTIVE 
 DIRECTOR, NATIONAL DEFENSE BUSINESS INSTITUTE, UNIVERSITY OF 
                           TENNESSEE

    Mr. Patterson. Thank you, Mr. Chairman, members of the 
Defense Acquisition Reform Panel. I am very pleased to be here 
this morning to participate in a discussion of a question that 
has clearly captured the attention of the current 
Administration and Congress: How should Congress assist the 
Department of Defense in improving its acquisition of weapons 
and services so that it can meet the needs of the warfighter in 
the field while still being a good steward of the taxpayer's 
dollars?
    The first consideration for judging the success of an 
acquisition program is whether it fielded a weapons system or 
information system or service in time to make a positive impact 
for the warfighter. A system or service fielded too late to 
need may as well not have been bought at all. The phrase, ``too 
little, too late,'' can mean lost lives.
    But before we look at the measures of the acquisition 
system merit, there is another consideration central to this 
discussion. When Secretary Gates made his budget announcement 
on April 6th, 2009, I believe he was speaking from frustration 
that was as much about what has been the persistent problem 
with the acquisition system that we depend on that is simply 
not responsive to the immediate warfighter needs, as it was 
about winnowing bloated, failed and unnecessary programs.
    Implicit in that expression of frustration is a clear lack 
of confidence in a system that actually produces programs with 
uncertainty and instability. The most dramatic improvement 
metric will be when the senior leadership in the 
Administration, Congress and the Department of Defense have 
seen such improvements, results, not words, that they can say 
they have renewed confidence in the stability, predictability 
and effectiveness of the defense acquisition system.
    The Defense Acquisition Performance Assessment's Report 
contended that program stability and predictability were 
singularly and uniquely crucial to managing programs that were 
on cost, on schedule and performing. To that end, in the time I 
have, allow me to describe two areas of improvement for 
measuring program effectiveness worthy of attention.
    First, Major Defense Acquisition Programs, or MDAPs, often 
start at milestone B, the beginning of engineering and 
manufacturing development, with critical staff positions 
vacant. Percentage of critical staff positions filled at 
milestone B is an easy and important metric to be observed. It 
makes little difference to implement programs to raise the 
level of skills of the program staff if they are missing in 
action.
    Second, the acquisition strategy document, that is to lay 
out how the weapon system is to be acquired, the initial 
roadmap, if you will, is often flawed in that it focuses more 
on presenting a case for required capabilities and quantities 
than on laying out the reasoning for acquisition competition 
methodologies.
    For example, how the prime contractor participants in an 
MDAP competition will select subcontractors and how the winner 
of the competition will manage the subcontractors to gain 
improved efficiencies and effectiveness are generally given 
little consideration. Creation of the acquisition strategy 
document is one of, if not the most, important tasks the 
government acquisition program management can undertake.
    The strategy should establish the template for all the 
activities that will take place throughout the source selection 
process, engineering manufacturing development, and follow-on 
production and fielding. More important, it establishes how the 
program management team is thinking about the numerous events 
and activities that a program will encounter.
    The defense acquisition executive should establish a common 
set of strategy elements that all the military department 
service acquisition executives must include in MDAP acquisition 
strategy documents. Additionally, a set of standards or metrics 
by which the strategy elements can be evaluated as effective 
must be part of that process.
    In closing, I would be remiss if I didn't acknowledge the 
progress that has been made by the Department in improving the 
acquisition system over the last four years. Though it is the 
General Accountability Office's headline that the 96 major 
acquisition programs have grown in cost by $296 billion that 
gets attention, those numbers belie an equally worthy but 
overlooked statistic published in the very same report.
    The average increase in unit cost of the 28 MDAP programs 
with less than 5 years since development started is only 1 
percent. Compared with an average of 55 percent increase in 
acquisition unit cost of 25 programs in the group with 5 to 9 
years since program development start, there has been important 
improvement that should be recognized.
    With that, it is my privilege to be here. Thank you very 
much, and I would be happy to take any questions that you have.
    [The prepared statement of Mr. Patterson can be found in 
the Appendix on page 32.]
    Mr. Andrews. Thank you, Mr. Patterson.
    Mr. Fitch.

  STATEMENT OF DAVID P. FITCH, USN (RETIRED), DIRECTOR, AT&L 
 LEADERSHIP LEARNING CENTER OF EXCELLENCE, DEFENSE ACQUISITION 
                           UNIVERSITY

    Mr. Fitch. Chairman Andrews, Congressman Coffman, members 
of the panel, thank you for the opportunity to appear here 
today. I will address the subject of acquisition performance 
metrics and your questions about how to increase the realism of 
program baselines, making trades between affordability and 
performance and how to assess the value of systems that do not 
necessarily--please recognize that my opinions do not 
necessarily reflect the views of the Defense Acquisition 
University (DAU), the Department of Defense or the 
Administration.
    Mr. Andrews. That is why we are interested.
    Mr. Fitch. I suspected as much.
    Measurement of acquisition performance must encompass both 
strategic and tactical elements. As emphasized in a recent 
Defense Science Board report titled, ``Creating a DOD Strategic 
Acquisition Platform,'' the management, execution and oversight 
of acquisition programs is moot if we aren't spending taxpayer 
dollars to buy the right capabilities if we aren't 
demonstrating strategic choice. It is as important to decide 
what capabilities we won't buy as well as what we will buy.
    I believe one of the root causes of funding instability is 
what I described as too many programs chasing too few dollars. 
Too many programs chasing too few dollars is one of the root 
causes, I believe, of overly optimistic cost estimates. The 
recently implemented Materiel Development Decision (MDD) 
process provides a framework for making strategic investment 
decisions.
    The MDD is the formal point of inquiry into the acquisition 
process. MDD will increase integration of the three major 
acquisition support systems, requirements, resources, and 
acquisition.
    Improving requirements management, an initiative supported 
by the Congress, includes providing training to requirements 
writers and managers to ensure that they have a sufficient 
understanding of the critical elements of acquisition, such as 
systems engineering and testing. The goal is to improve 
collaboration between acquisition and requirements communities 
to exploit cost and performance trades and improve acquisition 
outcomes.
    Having a formal requirements and acquisition process is not 
unique to the DOD. We can learn important lessons from 
commercial industry. If you compare the DOD acquisition system 
with the process to develop electronic games, there are marked 
similarities and differences.
    Notably, the process to get games on the shelves for the 
December holiday season starts with a precise clarity of what 
will be developed by when, and includes a corporate commitment 
to the resources required for the project. Precise clarity is a 
result of intense interaction between the people that divine 
the capabilities of the game and the people that will design 
and test the software.
    Turning to the subject of tactical acquisition metrics, the 
most effective tools and templates incorporate metrics, both 
quantitative and qualitative. The question was raised, ``Are 
there metrics beyond cost and schedule performance that are of 
value?'' The answer is yes, and an ongoing example is a 
probability of program success metrics, POPS, that are 
currently being developed and deployed across the services and 
other federal agencies, such as the Department of Homeland 
Security.
    Starting with a blank sheet of paper, a group of DAU 
faculty comprised of experienced program managers and other 
functional experts ask themselves questions, including, ``What 
conditions facilitate the success of programs? What metrics are 
leading indicators of derailment?''
    The resulting tool, POPS, uses a structured process to 
continually assess and display key elements of planning, 
resourcing, execution and external influences that promote or 
negatively impact program success. Still evolving, POPS is 
being used in the Army, Air Force, Navy, Marine Corps and Coast 
Guard. Timely, accurate and transparent metrics integrated in a 
management and oversight process, will produce better program 
outcomes.
    Another question that has been asked, ``Can we ensure 
improved realistic baselines.'' Again, I believe the answer is 
yes, and there are ongoing initiatives that will yield more 
realistic baselines. These include increased emphasis on 
technology readiness before starting major development, 
emphasis on improved cost estimating, which I am sure that we 
will talk about today, and competitive prototyping, which is 
included in the new DOD 5000.2 and pending legislation.
    Prototyping increases the opportunity to identify and 
assess affordability and capability traits. Competitive 
prototyping also allows the government to observe the 
performance of competing industry teams before making a down-
select for engineering and manufacturing development. No matter 
how thoughtfully we plan and discipline source selection, a 
paper-only source selection process is never as good as 
demonstrated performance.
    The ultimate assessment of whether we have delivered value 
and needed capability to the warfighter is feedback from the 
field from the warfighter. At various times, I have seen photos 
of messages written with felt-tip pens on trucks and personnel 
carriers that have been returned to the government or industry 
depots. One of those signed messages--a picture is in my 
written testimony--reads, ``This truck saved my life, as well 
as five others, 2-April-2008 at 2300 Lima, Basra, Iraq.'' Those 
kinds of testimonies, those kinds of results, are what those of 
us that are in the acquisition aim to achieve on a daily basis 
by our efforts and commitment.
    Before equipment is fielded, it undergoes rigorous levels 
of development and operational testing, and the new DOD 5000 
has increased emphasis on earlier testing.
    Mr. Chairman, thank you very much for the opportunity to 
participate in today's discussion. As the Secretary of Defense 
has said, there is no silver bullet. It will be a combination 
of initiatives and collaboration amongst all the stakeholders 
to create an acquisition system that consistently produces 
successful acquisition programs.
    I look forward to your questions.
    [The prepared statement of Mr. Fitch can be found in the 
Appendix on page 57.]
    Mr. Andrews. Mr. Fitch, thank you very much for your 
testimony.
    Dr. Nussbaum.

   STATEMENT OF DR. DANIEL A. NUSSBAUM, VISITING PROFESSOR, 
  DEPARTMENT OF OPERATIONS RESEARCH, NAVAL POSTGRADUATE SCHOOL

    Mr. Nussbaum. Mr. Chairman, distinguished members of the 
panel, I would like to thank you for this opportunity to 
discuss my thoughts on how to improve acquisition and cost 
estimating processes in the Department of Defense. These ideas 
are mine alone.
    As the chairman said, I am a member of the faculty at the 
Naval Postgraduate School in Monterey, California. And I have 
spent the last 30 years mainly doing, and more recently, 
teaching and researching in the defense acquisition management 
system with a focus on cost estimating. I was a previous 
director of the Naval Center for Cost Analysis and past 
president of the Society of Cost Estimating and Analysis.
    All my experiences in cost estimating confirm that three 
things are necessary for sound cost estimating: Acceptance of 
the underlying uncertainties in predicting the future, accurate 
and plentiful historical data, and professionally trained and 
certified personnel.
    On uncertainty, there is intrinsic uncertainty in all 
estimates. It derives from several sources, mainly that we are 
usually designing or building or operating something that is 
substantively different from what we did before. The difference 
could be in the product or in the economic conditions or the 
programmatic conditions.
    An estimate reflects our knowledge at a point in time when 
we freeze the problem and base the cost on the configuration 
and programmatics as they are understood at that time. From 
that baseline, many things can change that can also change the 
cost estimate, including labor rates, overhead rates, 
schedules, enhancements of the capabilities or quantities, 
changes when a particular technical solution to a problem 
doesn't work as planned. And we need an alternative technical 
solution.
    On data, a hallmark, a necessary characteristic of a sound 
cost estimate, is that it is based on historical program 
performance from similar or related ongoing or past programs. 
Historical data is variable. Not every aircraft costs the same 
as every other aircraft.
    And the measurement of this variability is accomplished 
through statistical constructs, things like standard error of 
the estimate, things like confidence intervals. We assume in 
our community of cost estimating that the patterns of the past 
will repeat in the future. But these patterns are almost always 
statistically grounded patterns modeled with the powerful and 
subtle techniques known collectively as regression. Further, we 
know of no alternative approach to using the past as a guide to 
the future if we want a scientific that is a reproducible and 
auditable approach.
    Not all estimating is done by government employees. There 
really are three sub-communities. One is the government in-
house estimators. One is employees of the large vendors who 
also design, develop and build what we buy--Boeing, Northrop, 
Lockheed Martin, for example. And thirdly, there are support 
contractors or consultants to the government. Those are the 
three communities.
    And surely, we need to increase the capacity and the 
quality, the numbers and the training of the government 
estimators, but so do we need to enhance the professionalism of 
the other two communities.
    There are currently no undergraduate curricula in cost 
estimating, and there are only four educational institutions 
that I am aware of that teach at least one course in cost 
estimating, and those are the Naval Postgraduate School in 
Monterey, where I am; the Air Force Institute of Technology at 
Wright-Patterson in Ohio; Defense Acquisition University, 
diverse locations with a capital campus at Fort Belvoir; and 
the Massachusetts Institute of Technology, which offers one 
elective course in cost estimating within its engineering 
curriculum.
    The recent separation of the business cost estimating and 
financial management career field into two separate cost 
estimating and financial management tracks is a very welcome 
development and should be supported, but note that DAU support 
is largely limited to military and DOD, not the other two 
communities.
    The Society of Cost Estimating and Analysis--we say SCEA--
whose membership includes approximately one-third of all cost 
estimators supporting DOD, is a central and indispensable 
player in the training, initial certification and periodic 
recertification of cost estimators. I note with pleasure that 
the executive director of SCEA, Mr. Elmer Clegg, is in the 
room.
    SCEA has collected a body of cost estimating knowledge, and 
it provides the members of the cost estimating community, 
provides training in cost estimating, has developed and offers 
an examination and experience-based certification program. 
Lockheed Martin, Boeing, Northrop Grumman, other vendors use 
SCEA's training and certification as their standard.
    I appreciate very much what this committee seeks to 
accomplish, Mr. Chairman. This concludes my prepared statement, 
and I would be pleased to answer your questions.
    [The prepared statement of Dr. Nussbaum can be found in the 
Appendix on page 66.]
    Mr. Andrews. Well, thank you, Dr. Nussbaum, and I thank 
each of the three witnesses for their testimony. As I say, we 
have had the chance to review the written testimony. We are now 
going to get to the questioning phase.
    Mr. Patterson, I was intrigued by your reference in your 
written testimony to two kinds of requirements costs, which you 
express as customer requirements and derived requirements. What 
is the difference between the two?
    Mr. Patterson. Customer requirements are established in key 
performance parameters, which are requirements that are 
established for a particular weapons system that the weapons 
system then must perform against. Derived requirements are 
those requirements that the customer did not ask for but that, 
as the name would suggest, derive as a consequence of the 
design process. ``Oh, look, we could do this better if only 
we''--and of course, that is taken to the program manager.
    The program manager will say, ``Okay, we can do that. Just 
bring money.'' And they do.
    Mr. Andrews. Now, I notice on page seven of your testimony 
that you say that a recent study prepared by the Monitor 
Company Group was based on selected acquisition report data, 
estimates that approximately 33 percent of the cost growth from 
2000, 2007--I think I read this--is attributable to this second 
category of requirements that you are talking about, right, the 
derived?
    Mr. Patterson. That is correct. Yes, sir.
    Mr. Andrews. Now, how is that number reached? Where did 
that 33 percent come from?
    Mr. Patterson. That comes from the selected acquisition 
reports (SARs). In fact, they lay that out in the SARs and 
explain in each of the SARs where the----
    Mr. Andrews. Well, one of the things that we want to do is 
to make sure that we discriminate between derived changes that 
are beneficial and those that may be superfluous. How would you 
suggest that we might do that? In other words, I don't want to 
leave the impression that we are saying, or the report is 
saying, that ``Oh, that 33 percent was waste.''
    Mr. Patterson. Oh, absolutely not.
    Mr. Andrews. And how do we draw the line between 
beneficial, cost-effective derived requirements and not-so-
beneficial derived requirements?
    Mr. Patterson. Well, let me give you a real-life example. 
In 1993, I came to the C-17 program. It was a troubled, 
problem-plagued program.
    The two program managers decided that much of the problem 
was that requirements were growing in an airplane. They 
established a set of rules. The set of rules was very simple: 
If you have a requirement or an engineering change, which is 
effectively a derived requirement, then it must go to an 
engineering change board.
    The engineering change board will evaluate the change for 
its intrinsic merit. But if it doesn't meet the safety of 
flight or other driven requirements, then it has to have a 
three-to-one payback in savings and not perturbate the 
schedule.
    Mr. Andrews. Interesting.
    Mr. Patterson. Real simple.
    Mr. Andrews. Now, in terms of the scope of that C-17 
program, how much were the derived requirements overrun? In 
other words, how much was attributable to derived requirements?
    Mr. Patterson. I would only be guessing. I can certainly 
get that for the record.
    [The information referred to can be found in the Appendix 
on page 83.]
    Mr. Andrews. And with this method that you just described, 
in your judgment, what percent of the derived requirements were 
beneficial, and which failed to make that three-to-one cut and 
didn't happen?
    Mr. Patterson. Again, let me give you an example.
    At Edwards Air Force Base, they were testing the airplane. 
One of the requirements was that one airplane needed to start 
another airplane, but it started it with two hoses from the 
pneumatic system. Somebody said, ``I wonder if it will start 
with one hose.'' It didn't. ``Oh, my gosh. Well, now we have to 
go and figure out why that is the case.''
    And so, that took a considerable amount of test time and 
money, and as it turned out, they really wanted to have it 
start with one hose. I am sorry, that would have taken----
    Mr. Andrews. Who was the someone who said that? Not the 
person's name, but where were they in----
    Mr. Patterson. That was the test community out on the ramp 
at Edwards.
    Mr. Andrews. Right.
    I am going to go back for a second round, but Mr. Fitch 
mentioned this probability of success metrics program. Has 
there been built into that a litmus test for a probability, if 
you would fall below it, things stop? In other words, does that 
program have in it a built-in go or no-go line?
    Mr. Fitch. It does not have a go or no-go line. It is 
information that is updated and provided----
    Mr. Andrews. Do you think it should?
    Mr. Fitch. I think that there needs to be an informed 
review and decisions that look at each of the things that occur 
that is negative, because some of the things that are in the 
probability of success metrics and reporting are positive.
    So I think that it needs to have the program manager, when 
that information is put together on a monthly basis, needs to 
look at that. It needs to be reviewed by someone in his Program 
Executive Office (PEO), okay, and potentially even at higher 
authority when there are negative occurrences.
    Could I add something about--reporting?
    Mr. Andrews. Sure, and then we are going to go to Mr. 
Coffman. Sure.
    Mr. Fitch. Yes.
    I just wanted to explain that the derived requirement 
process is actually part of the systems engineering process so 
that if, for instance, the warfighter says in an aircraft, ``I 
need a display that has color, I need a display that has this 
amount of resolution,'' et cetera, the first pass, even by the 
warfighter or the cost estimators, may say, ``This amount of 
processing may be sufficient for that.''
    When you get into, ``Well, by the way, you are going to 
have these other software capabilities.'' You put those 
together, and they start to build on one another, you can find 
all of a sudden that the processor that was planned may be a 
commercial off-the-shelf item with X amount of memory, 
throughput, et cetera, is insufficient.
    At that point is the point where it would be very useful to 
have the requirements community have a real dialogue with the 
acquisition community to say do I now take away some of the 
requirements so I can continue to use that commercial off-the-
shelf processor that is less expensive, or are those 
requirements really important now that we have figured out what 
they are. So the point is the derived requirements is a very 
important process, and it is actually the process where you 
allocate where features will go to hardware or software.
    Mr. Andrews. Thank you very much.
    Mr. Coffman.
    Mr. Coffman. Thank you, Mr. Chairman.
    This is to all of you. It seems like part of the problem is 
sometimes we are dealing with immature technologies, and I 
think we have had previous testimony to that effect, that 
sometimes we are asking the contractors to develop something.
    Should we bifurcate the process? In other words, that you 
are contracting with one entity to develop the technology, to 
develop the--I don't know if you would go as far as a 
prototype, and then where you can go into fixed cost production 
with another--maybe that entity would be allowed to bid on it 
as well. But is it better to bifurcate the process?
    Mr. Nussbaum. You know, I would say we would do that now. 
If we have open competition, we certainly have a research and 
development (R&D) contract followed by an acquisition contract 
with--there is certainly no guarantee that the R&D contractor 
will have a follow on.
    So in some sense, we do that, but in another sense, the 
Department has now mandated technology readiness levels, TRLs 
of certain levels before a program can get beyond a milestone. 
They have narrowed that cone of uncertainty by saying you have 
to have a TRL level six. They would prefer seven, and GAO 
prefers seven, but you have to pass a TRL level of six before 
you go into the next milestone.
    Mr. Fitch. As a part of the analysis of alternatives, that 
process that immediately follows it, the technologies that 
would be appropriate for that system, that capability, are 
assessed for technology readiness, when you get to the point of 
saying who should develop it, I prefer the concept of the 
competitive prototyping, because you are going to get proposals 
from industry.
    They are not going to all have the same strengths. Some are 
going to view that I want to do this more in a hardware 
function. Others may be more software intensive. There are 
different technologies involved.
    Their proposal, and then watching them deliver on what they 
promise and they say is possible in competitive prototyping, is 
key to then being able to transition and make an award to that 
contractor who has proven that they did what they say they 
could do. So having a different contractor, if I understood 
your premise correctly, to develop technology and then award it 
to somebody differently to develop the system, I think that is 
not the intent of the competitive prototyping system.
    Mr. Patterson. I think, too, there is a problem that arises 
when we talk about spiral development. Originally, spiral 
development, evolutionary development, were designed to have a 
weapons system that effectively was fielded and then have block 
upgrades to improve the weapons system.
    In many cases, that is not how that works, because you do 
have a parallel R&D program that is working on improvements. 
And before the weapon is actually fielded, you have the attempt 
to integrate improvements.
    Well, the consequence of that often is that you have a 
stretched-out program. Costs escalate. And in the end, 
performance is degraded, and it takes a while for the--much 
longer than had been anticipated for that weapons system to be 
fielded. So I think we need to have very specific and clear 
understood standards by which we will integrate or put in 
upgrades or technology advances into weapons system. It 
certainly can't be during engineering, manufacturing, 
development.
    Mr. Coffman. I think it was mentioned, and it has been 
mentioned repeatedly, that the changes drive a lot of the cost 
as they go forward. Where did the changes emanate from? I mean, 
are some of the changes sort of broader in scope where the 
affected branch who will receive and utilize the weapons 
system, is it changes in evolving doctrine, or is it mostly at 
a very technical level with this engineering requirements?
    Mr. Nussbaum. You know, I think that changes come from all 
sorts of places, including changes in labor, overhead rates, 
schedules, requirements, quantity, absolutely everything, just 
like building a house. Everything that changes has the 
contractor saying, ``Cement has gone up. Brick has gone up. My 
subs have gone up''--or down, but they are always changing.
    And so there is a great churn. Some of it is part of life, 
and some of it we try to control by saying, ``Tell us what you 
are building, and we will cost that program.'' But I think it 
is just intrinsic in the cost estimating process.
    Mr. Coffman. Is there any way to just have more--I mean, 
what would be some of the methods for having more discipline 
over the--and I take it some of it would--having a change 
board?
    Mr. Nussbaum. There is something called a CARD, a Cost 
Analysis Requirement Document, which is, to the chagrin of all 
program managers, we say three or six months before we are 
going to a milestone, you tell us what you are building, how 
many, what the technologies are, and we will cost that program. 
It may be that things change after that, but that is the 
program we are going to cost, because otherwise we are chasing 
our tail, and we don't know until the very last moment what we 
are costing.
    And in fact, it takes time to do a cost estimate. It is a 
part of the discipline at the Office of the Secretary of 
Defense (OSD) is a Cost Analysis Requirements Document will be 
prepared six months before the milestone, and the lack of 
availability of that CARD results in a day-to-day slip in the 
milestone. So that is real discipline, but it has its obvious 
down side, too.
    Mr. Patterson. I think, too, that in dealing with the 
discipline and structure, that I must tell you, I submit that 
it requires rules. And the rules simply have to say that, after 
milestone B, you don't have any more requirements. And, I mean, 
it is somewhat draconian, but nonetheless, if it--unless of 
course it is a safety of flight or it is an obvious design 
failure that needs to be corrected.
    But those kinds of things are few and far between, quite 
frankly. And then, this idea that a requirement or a change 
would give you a three to one or four to one payback in savings 
while not perturbing the schedule is not a bad idea, either. 
And as technology moves forward, that is entirely possible over 
the course of a program.
    Mr. Fitch. There are times with the requirements--I recall 
when I was developing a black box that was going to go in 17 
different platforms. I would have this platform coming to me 
and say, ``It would really be easier if you changed it this 
way. It will make me less expensive for my integration, save my 
costs and schedule.''
    What I came to understand is the program manager has to be 
prepared to say no during the development phase. Now, the 
configuration steering boards we have in place anticipates that 
there may be compelling reasons to change a requirement, 
moderate a requirement during development, but that is the 
process of the configuration steering board, is to raise that, 
if you would, visibility to the pressures of changing 
requirements to have a senior level decision made about the 
requirement once the milestone has been approved.
    Mr. Andrews. Thank you.
    Mr. Cooper.
    Mr. Cooper. Thank you, Mr. Chairman.
    And I would especially like to welcome a fellow Tennessean, 
David Patterson, and congratulate him on starting Defense 
Business Institute at the great University of Tennessee in 
Knoxville. Thank you not only for your past service, but what 
you are doing right now.
    Can anyone provide me with enough historical perspective to 
help me understand how, during World War II, men like Henry 
Kaiser were able to produce destroyers and--I forget exactly 
who produced airplanes, but it was an amazingly productive 
period. And I don't know how--was there less bureaucracy then? 
How was so much able to be accomplished so quickly? Anybody 
know?
    Mr. Nussbaum. My reading of history is that there were a 
lot of failures, and then we tend to remember the successes, 
which were terrific. But in fact, there were false starts.
    The P-51 was wonderfully successful, but because it had a 
bigger gas tank so it could keep up with the other aircraft, it 
gave us greater range, but we didn't build it for that reason. 
Just we sort of lucked into that, if you will.
    I think that things were much simpler. We were able to turn 
technology generations around much faster and, therefore, 
absorb the failures. Today's systems are very complex. They 
take a long time, and we just won't accept failure.
    Mr. Fitch. I think that what we remember a lot is those 
ships going down the ways, okay, daily, okay? That was in the 
production phase.
    Even the P-51 that was just mentioned by Mr. Nussbaum, when 
it was first fielded, it didn't have the engine that ended up 
being what everybody remembers about the performance of that 
aircraft. There were various modifications made after it was 
fielded. It was in the field and found that it was not exactly 
was needed.
    Mr. Patterson. I think, too, you had a tremendous 
industrial base that was able to accommodate to the level of 
technology that it was asked to accommodate to. Today, I am not 
sure that we would be able to do that again in that amount of 
time.
    Dr. Ron Sega did a study not long ago when he was at DDR&E, 
Director of Defense Research and Engineering, in which he 
looked at the industrial base and what it could do, and found 
that 62 percent of all of the Ph.D candidates that are enrolled 
in disciplines critical to national security have temporary 
visas. And that tends to put us at a disadvantage, because they 
generally don't stay around and work for Lockheed or Northrop 
or Boeing, and Skunk Works, particularly, or Phantom Works.
    And if you will look at the Aerospace Industries 
Association data, you will find that in the 1990, 1991, you had 
1.3 million touch labor workers, highly skilled workers. Today, 
we have something less than 700,000. Those are statistics that 
should give us pause to consider what we are going to do in the 
future.
    Mr. Cooper. Help me understand. I know that we have had 
failures in acquisition throughout our history, and Mr. Fitch 
mentioned a success when he posted the note that was on the 
vehicle that saved lives.
    But on the front page of the Washington Post recently was a 
statement by Secretary of Defense Gates when he attended the 
return of the remains of some of our troops. He asked how they 
died, and he was told they were in an inadequate vehicle. And 
he cursed because that was symbolic of the fact that we have 
had difficulty fielding relatively simple platforms like Mine 
Resistant Ambush Protected vehicles (MRAPs) or up-armored 
Humvees.
    And at least for some period of time there was only one 
manufacturer of up-armored Humvees in America, and yet, at the 
same time, we have automobile companies going bankrupt and 
looking for vehicles to build. So there seems to be a mismatch 
somehow between pretty basic demands for troops and our ability 
to field and source those even when we have able and willing 
automobile companies who are looking for ways to keep their 
plants busy. Why the mismatch?
    Mr. Patterson. Oh, I think it is a very fundamental problem 
in that you have a Department of Defense that understands fully 
that we are at war with terrorists, and you have a country that 
doesn't.
    Mr. Cooper. So Chrysler or General Motors (GM) or Ford 
didn't want to bid on the up-armored Humvee, or we couldn't----
    Mr. Patterson. Well, actually, the subsidiaries of those 
folks did, but they really were not in the business of putting 
out those kinds of vehicles. The people who were, International 
Harvester, obviously, and--but the design, fortunately or 
unfortunately for us, that was available were foreign designs 
of up-armored vehicles that were designed to sustain the kinds 
of things that an MRAP would have to sustain.
    Mr. Fitch. And the number of companies that have as a core 
competency in the technologies for armor, okay, is not the same 
as we have for the auto industry. In fact, we are doing a lot 
of investment in the Department of Defense today to try to find 
better armor, cheaper armor, especially lighter armor, whether 
that be for the vehicles because, when you put the armor on it, 
it puts a demand on the engine. It puts a demand on the 
drivetrain. If you double the weight of the vehicle, whatever 
the percentage is, okay, it has additional tolls upon the 
reliability of the vehicle.
    So we don't have the same industrial base. In other words, 
the auto industry isn't the industrial base for our Army, 
either.
    Mr. Nussbaum. In a sense, the Humvee is the wrong vehicle 
to up-armor, but it is the only vehicle to up-armor. But it was 
built as a replacement for the Jeep, which didn't go in harm's 
way.
    So it was designed to optimize all the functions that it 
was going to perform, and we missed the fact that it was going 
to go in harm's way. So it wasn't suitable, wasn't optimal, for 
up-armoring, but it is what we have, so we are going to up-
armor it.
    Mr. Cooper. I see that my time has expired. Mr. Chairman, 
look forward to another round.
    Mr. Andrews. Thank you.
    With the consent of my colleagues, we are going to do 
another round, if that is okay with the panel as well, if it 
fits your schedule. Thank you.
    Mr. Patterson just made a suggestion that we might have a 
rule that says no new requirements after milestone B, and that 
prompted this question. Again, this piece of data that you 
cite, about 33 percent of the program cost growth being 
attributable to these requirements changes, how many of those 
requirements changes happened after milestone B? Do you know?
    Mr. Patterson. That data, as I recall, is after milestone 
B.
    Mr. Andrews. It is all after milestone B.
    Mr. Patterson. That is where the majority of the growth 
generally takes place.
    Mr. Andrews. A related question: in your oral testimony, 
you talk about the fact that, if I read this correctly, the 
average increase in unit cost of the 28 MDAP programs of less 
than 5 years since development is only 1 percent.
    Mr. Patterson. That is correct. That is----
    Mr. Andrews. But it is 55 percent from years 5 to 9. How 
many of that in the 55 percent category, the five to nine, got 
a waiver through milestone B, didn't meet the requirements to 
get to milestone B but got waived past it? Do you know?
    Mr. Patterson. No, I don't. I don't know exactly how many.
    Mr. Andrews. But, I mean, would it be accurate to say it is 
probably most of the 55 percent cost overrun comes from that?
    Mr. Patterson. I would say a significant portion of it, 
sure.
    Mr. Andrews. One of the things that is in the conference 
report that we will be looking at in the WASTE TKO bill today 
is what we call intensive care, where if a program is permitted 
to go forward, even though it didn't achieve the milestone B 
criteria, if it is waived past it, there is a whole set of 
intensive requirements that are imposed upon that program to 
try to get it back under control.
    I wanted to come back to, again, this bifurcation that you 
create between derived and customer requirements. Describe for 
us the process that you think ought to be instituted to 
determine whether a derived requirement is added to the package 
or not.
    Let's say we are at a point--assuming for a moment that we 
accept your proposition that there are none after milestone B, 
which I assume you say there should be some exceptions, now as 
you said, for true safety or emergency purposes, but let's 
assume we are living in a world where, except for those narrow 
situations, there are going to be no changes in requirements 
after we hit milestone B.
    We are now in pre-milestone B, and an ``Oh, by the way'' 
comes up, as you said earlier, ``Oh, by the way, this can do 
this.'' Who should make the decision as to whether that gets 
added to the package, and by what criteria?
    Mr. Patterson. Well, I think that the program manager 
should have the initial cut at whether or not they are going to 
include that into the program. But program managers generally 
are colonels or, in very large programs, brigadier generals who 
have significant oversight within the Department.
    I think that if they have a set of rules that say, first of 
all, if it is not a safety of flight or if it is not some sort 
of safety issue, or if it doesn't give me a return on my 
investment, then I am going to have a thumbs-down, initially.
    Mr. Andrews. How do you measure the concept of return?
    Mr. Patterson. Well, let's take, for example, you have--
again, I will turn to this C-17.
    There were parts of that airplane that were originally 
designed for aluminum lithium, for example. Well, aluminum 
lithium is strong, but it is brittle.
    So an engineering change was made to change that to a 
different alloy. We were breaking the aluminum lithium cargo 
floor guides at a regular pace, so changing that eliminated the 
problem of having to constantly replace it. That was a savings.
    And those are the kinds of things that I would suggest are 
three-to-one. And even the suppliers were given the opportunity 
to do that.
    Mr. Andrews. Got you.
    It strikes me that this is really the essence of the 20 
percent the secretary talks about in his 80 percent solution, 
that what he really is aiming to get at here is to give us an 
adjudicatory mechanism that draws the line between the 80 and 
the 20 when you get to this point, and it is your suggestion 
the program manager should be the first person to weigh in on 
this.
    Who should evaluate his or her recommendation?
    Mr. Patterson. Well, then you have an engineering change 
board that would provide a corporate view of it. And if it is a 
particularly expensive change, then you are going to go to your 
service acquisition executive, or if it is an ACAT 1-D, an 
acquisition category 1-D, then you have the Defense Department 
acquisition executive who would have a cut at that.
    Mr. Andrews. And my final question, are you confident that 
we can quantify this concept of value sufficiently to hit the 
three or four? In other words, are all of the values that we 
want to promote--you gave a great example of saving, replacing 
a piece of a plane that is going to go wrong. But do all of the 
value concepts lend themselves to that kind of quantification 
that would let us say, ``Well, this fails to meet three-to-one, 
so out?'' Pretty hard?
    Mr. Patterson. No, it is much more difficult than that. And 
that is why it takes a lot of research and study and to set up 
standards and conditions whereby you can evaluate these.
    Mr. Andrews. Frankly, and I will just conclude with this, 
it is one of the reasons why we are glad we have the three of 
you and the institutions that you represent, because we really 
do turn to institutions like yours to assemble those data, 
analyze them and give us a factual basis to draw the lines that 
my questions imply. Thank you.
    Mr. Coffman, your turn.
    Mr. Nussbaum. Mr. Chairman, is it----
    Mr. Andrews. Sure, Doctor.
    Mr. Nussbaum. Is it appropriate for me to make a remark, 
or----
    Mr. Andrews. Sure. It is okay.
    Mr. Nussbaum. I think it is not hard to measure the value 
of ideas that replace current capabilities. It is always harder 
to measure the value of things that represent new capabilities. 
They don't represent a savings for the operating and support 
detail that you were going to incur.
    But if you are replacing a current capability, then it is 
pretty easy to do an estimate, but it is still an estimate, of 
what does it cost to invest to make this thing happen, and what 
do you save over time in the operating----
    Mr. Andrews. I agree. The much more difficult proposition 
is where you have a new function that could be added by 
something that you discover. How do you measure that, and that 
requires trade-off analysis. It requires opportunity cost 
analysis. It requires a lot of broader inquiries.
    Mr. Nussbaum. And if you have that long tail, the question 
then is do you do any discounting on it, the technical issue of 
net present value and at what rates. The Office of Management 
and Budget (OMB) helps us there.
    Mr. Andrews. Not to be hyper-technical, but one of our 
problems is then matching up the federal credit scoring and net 
present value rules with the real-world ones, that very often a 
decision--a classic example is in energy. The Department has 
guidance to hit 25 percent alternative fuels by 2020. And in 
order for them to do that, they need to do multi-year 
contracts. But to do multi-year contracts, the Congressional 
Budget Office (CBO) scores that as putting the whole net 
present value into one year, which makes it almost impossible 
to do, which means we don't do much of it, which means we are 
falling backwards. So marrying the CBO criteria with the real-
world criteria is a bit of a challenge, too.
    Mr. Nussbaum. Yes, sir. And so the devil really is in the 
details on this, and----
    Mr. Andrews. Well, the devil is in the CBO in this case. 
Now, don't tell Mr. Elmendorf I said that, okay?
    Mr. Nussbaum. And one other comment, and that is, when you 
go to the Configuration Control Board or your Service 
Acquisition Executive (SAE), you are proposing to spend 
investment dollars to make this thing happen, and the promise 
is that you will return operation and support (O&S) dollars 
later on. That is a nice conversation, but it doesn't accord 
with the budgeting realities.
    Mr. Andrews. And it doesn't score in our--nor should it.
    Mr. Fitch, do you want to add one thing, then I am going to 
go to my friend from Colorado?
    Mr. Fitch. If I could. I just wanted to say, again, that I 
think it is useful to talk about the operation requirements, 
the derived requirements, but to say again that, when we get 
the requirement from the user, it is stated usually in 
operational terms, okay, the results they wanted to see.
    Mr. Andrews. Yes.
    Mr. Fitch. To deal with industry, and for industry to build 
something, those need to be translated into technical terms. We 
also use the term ``derived requirements'' for that process. So 
the derived requirements are really--there are a couple types 
we are talking about here, that which are a part of the normal 
system that you have to do.
    The other one I would just say is your question is how do 
we know we get value. As a program manager, I had an 
acquisition program baseline. That was my contract. The way I 
viewed it, it was my contract with my--decision authority and 
with the American taxpayer to produce a capability at such a 
cost with certain milestones.
    And I think that is what most of us take and go back to. 
And as we do these questions about derived requirements and 
everything else, I think we keep in mind that framework.
    Mr. Andrews. Yes. I think the panel clearly understands 
that some subset of derived requirements are quite legitimate, 
necessary and desirable. And I think the three of you have 
given us some interesting tools to discriminate between 
undesirable derived requirements and desirable ones, which is 
what we are about.
    Mr. Coffman.
    Mr. Coffman. Thank you, Mr. Chairman.
    We talked today about having discipline in terms of change 
requirements. But do we also have to have discipline when 
programs get so far out of line that they become questionable?
    And I am thinking about--and I am working if you could 
reflect on the President's--I can't remember the nomenclature 
of the follow-on helicopter. Littoral Combat Ship (LCS) is in 
question. I wonder if you all could reflect on when a program 
gets so far out of line.
    Mr. Patterson. Well, it would be helpful, I think, in the 
total scheme of things, if we never let them get out of line. 
But nonetheless, you are exactly right. They do happen. It does 
happen.
    And one of the things that I think is difficult when you 
take, for example, the VH-71, the President's helicopter, and 
that is a perfect example of where you came in with one set of 
criteria as requirements and, over the course of the time, it 
changed dramatically.
    I think that we don't have a set of standards or conditions 
that raise a budget flag that say, ``Wait a second, I am sorry, 
you are red here, and you have been red three reviews in a row, 
and we are canceling your program.'' And what we do is we put 
ourselves in a position where it is the only game in town.
    You have the President's helicopter that is arguably old, 
and we don't have an alternative. You chose a manufacturer and 
a helicopter, and there is no off-ramp. There is no plan B. And 
we do this rather consistently.
    And I am almost of the opinion that we do it by design. And 
I would offer the Marines' Expeditionary Fighting Vehicle 
(EFV), for example. We don't have a way to walk the dog back 
down the path to get an amphibious vehicle that would replace 
the EFV, so we need to make that work.
    Mr. Fitch. I don't have the particulars on either of the 
programs you asked about. I was the systems engineer for the 
VH-3D, the current--well, it is one of the two current 
presidential helicopters. It is a unique mission. I think there 
is a desire usually if the White House says, or whatever or 
whoever it is, says, ``I need a capability,'' that you find a 
way to do it.
    Going back to it, most of the changes that I saw, and the 
pressures and the surprises that I got, occur in that first--
traditionally, the first 12 to 15 months of a program when you 
go towards a Preliminary Design Review (PDR), because the 
contractor is off doing a lot of things. There isn't a lot of 
object deliverables to figure out does he get it, he doesn't 
get it.
    I think that the competitive prototyping system to focus on 
doing PDRs earlier, to have effective communications with the 
contractor teams and oversight of the contractor teams during 
that period of time will do much more to get to a stable 
baseline at milestone B, which is the actual program 
initiation.
    Mr. Nussbaum. I am going to sound like a professor on one 
hand. On the other hand, because there are some historical 
examples of things which failed, failed, failed and then were 
terrific. Aegis was one, and Tomahawk was another. They just 
took a long time to bring aboard. And for some reason, we had 
the fortitude to stick with them and not say, ``This is an A-12 
or a Gama Goat,'' and get rid of it.
    So now we come to the category of LCS. Is it a Tomahawk or 
is it a Gama Goat? We don't know. But I am taken with my 
colleague's remark that, once you get past milestone B, if you 
have three reds in a row, you have got some serious explaining 
to do, with the presumptive answer, ``You are out.'' It is a 
rule.
    But the problem is knowing the future, and that is always 
the problem. I don't know whether LCS is a Tomahawk which is 
going to be absolutely terrific after a long incubation period, 
and the same for V-22. You just don't know, so we make 
decisions as people.
    Mr. Coffman. Thank you.
    Thank you, Mr. Chairman.
    Mr. Andrews. Thank you.
    Well, gentlemen, thank you very much for your expertise. 
Your reward for doing such a good job is we will have to call 
on you again. As the committee goes forward, our intention is 
to try to make legislative proposals for the fiscal year 2011 
authorization bill that will deal with the area of the problem 
that the WASTE TKO legislation that we are dealing with today 
does not deal with.
    And I think you have given us some very intriguing ways to 
measure the gap between what we pay and what we get. It is also 
heartening to hear what I have heard this morning, a consensus 
that this panel's contribution to the WASTE TKO bill was 
essentially two concepts.
    The first was to add a whole series of reviews and a lot of 
scrutiny pre-milestone B, with particular emphasis on the 
requirements process. And that did make it into the conference 
report, and that will become the law this week, we think.
    And second, the panel was very interested in much more 
rigorous review, what we call intensive care, of systems that 
pass milestone B by waiver, that have not met the requirements, 
or that fail Nunn-McCurdy standards and get exempted from the 
penalties there and go forward anyway. And I think if you look 
at the cost overruns, a huge majority of them fall into one of 
those two categories.
    So what we wanted to do was to take the best practices that 
you very ably described this morning and engage them as 
intensely as we can in the systems that, again, never met the 
criteria to get past milestone B but get past it anyway, and 
those that fail Nunn-McCurdy but continue to live on.
    And as Mr. Patterson said a few minutes ago--I think it was 
Mr. Patterson--our ultimate goal is not to have any of those 
cases in the future by unraveling the requirements process and 
looking at it more intensely to intensify that pre-milestone B 
analysis of what is going on.
    And the other point that I would make that is more on our 
side of the table than yours, I think that the principal reason 
that we get these cost overruns is that, once something passes 
milestone B, an enormous political constituency develops around 
it. Now, there are tens of thousands of people deriving their 
paycheck from a project, hundreds or thousands of 
subcontractors, dozens or hundreds of congressional districts.
    And as Secretary Gates I think can attest, making changes 
in those programs is very politically difficult. If you get to 
these flawed programs earlier when their political 
constituencies are smaller and weaker, the chance to do the 
right thing is a lot higher.
    So not just for analytical reasons, but we think, given the 
dynamic of the way these decisions are made in the political 
world, the more precise we are in our measurements and the more 
focused we are in our evaluation, in the requirements phase and 
a little bit beyond that, we think the better job that we will 
do.
    So I would say to each of the three of you we welcome your 
continued participation and input. We are certainly going to 
call upon you for your feedback as we go forward in our 
drafting process. And thank you very, very much for your time 
and attention this morning.
    Members will have a period of time by contacting either 
majority or minority staff to supplement the record with 
written questions, and we would invite the witnesses to do the 
same thing.
    And with that, the hearing is adjourned.
    [Whereupon, at 9:09 a.m., the panel was adjourned.]



=======================================================================




                            A P P E N D I X

                              May 19, 2009

=======================================================================


              PREPARED STATEMENTS SUBMITTED FOR THE RECORD

                              May 19, 2009

=======================================================================

    [GRAPHIC(S) NOT AVAILABLE IN TIFF FORMAT]


=======================================================================


              WITNESS RESPONSES TO QUESTIONS ASKED DURING

                              THE HEARING

                              May 19, 2009

=======================================================================

      
             RESPONSE TO QUESTION SUBMITTED BY MR. ANDREWS

    Colonel Patterson. To appreciate the context in which early C-17 
cost overruns occurred, some background and explanation of a 
methodology for arriving at an estimate of how much of the cost 
overruns could be attributed to derived requirements is in order. In 
this case engineering changes were the predominant example of what I 
have described as derived requirements and that appear in the public 
record.
    The first manufactured part for the first C-17 was milled in 
November of 1987 and the first squadron was declared to have Initial 
Operating Capability in January 1995. The period between these events 
was one of significant turbulence for the C-17 program. The contract 
for the C-17 program was a fixed-price, incentive fee development 
contract, because the aircraft was to be designed using off-the-shelf 
technology and was determined to be ``low'' risk.

        At the time of contract award, both the Government and MDC 
        [McDonnell Douglas Corporation] envisioned a program based on 
        commercial practices, minimum Government involvement and 
        concurrent development/production effort. Consequently, a fixed 
        price incentive contract was used to match the perceived low 
        risk of the program.\1\
---------------------------------------------------------------------------
    \1\ Report of the Defense Science Board Task Force on C-17 Review, 
December 1993, p 3

    As the program became more complex and engineering changes more 
frequent, MDC had no recourse to recoup investments in solving the 
engineering problems and meeting the cost of addressing requirements 
that were being added during full scale engineering and development 
than to make claims to the Government against the contract.\2\ Though 
it is not feasible to determine precisely how much of the total cost 
overrun was attributable to derived requirements or engineering 
changes, it is possible to understand the relationship between the 
unanticipated cascade of engineering changes and increased requirements 
and cost overruns by virtue of the amount of the claim MDC believed 
justifiable.
---------------------------------------------------------------------------
    \2\ Ibid.
---------------------------------------------------------------------------
                 COST OVERRUNS AND ENGINEERING CHANGES
    As early as 1985 engineering changes were impacting the C-17 
program costs. The Selected Acquisition Report for December 1985 
described cost increase of $214.5M for engineering changes ``needed for 
a four-pallet ramp, a combat offload rail system, and DoD standard 
avionics racks.'' \3\ Though these modifications were not in 1985 
referred to as ``derived requirements,'' they do meet the definition 
used in the study included with the oral statement provided for the 
record on May 19, 2009.\4\ These were the typical derived requirements 
not in the original contract, but identified by the program management 
to be necessary during development and presented as engineering 
changes. These changes and the corresponding cost increase occurred in 
the very early stages of the program, In December of 1985 the Full 
Scale Development contract had been signed.\5\ The Government became 
alarmed following a 1992 Department of Defense Inspector General Report 
that identified a cost overrun of $700M on a $6.6B contract ceiling.\6\ 
McDonnell Douglas was paying for costs overruns on the fixed-price 
contract and submitted claims for approximately $300M for what MDC 
asserted were changes in scope of its initial contract agreement which 
included ``costs for engineering, development, testing and production 
of six C-17's and three test planes.'' \7\ In the end, after 
significant negotiations with the Department of Defense, McDonnell 
Douglas agreed to spend $456M in process improvements and the Defense 
Department agreed to ``provide an additional $438M for the program.''
---------------------------------------------------------------------------
    \3\ Selected Acquisition Report As of December 1985, p 10
    \4\ Derived requirements, on the other hand, are requirements that 
the customer has not specified directly as a requirement but that 
emerge or derive from the design decisions that are made. Derived 
requirements are not capabilities that the customer specifically has 
identified. Particularly troublesome is a subset of derived 
requirements that fall into the category of engineering changes--those 
changes that improve on ``good enough'' and that have a combined effect 
of driving up costs and missing schedule milestones.
    \5\ Aeroflight, Aircraft of the World, Boeing C-17 Globemaster III, 
retrieved May 27, 2009, from http://www.aeroflight.co.uk/types/usa/
boeing/c-17/c-17.htm
    \6\ Battershell, A. L., The DoD C-17 versus the Boeing 777, A 
Comparison of Acquisition and Development, National Defense University, 
Washington, D.C. 1999, p 87-88
    \7\ Adelson, A., Company News; McDonnell May Submit Big C-17 Bill, 
New York Times, January 14, 1993, retrieved May 27, 2009, from http://
www.nytimes.com/1993/01/14/business/company-news-mcdonnell-may-submit-
big-c-17-bill.html
---------------------------------------------------------------------------
    A significant portion of the cost problems centered on a cascade of 
engineering changes that typified the years and immediate several 
months leading up to the negotiated settlement between MDC and the 
Department of Defense. The magnitude of the engineering changes 
occurring during this period was described clearly in the December 1993 
Defense Science Board's report as it stated, ``In May 1993 alone, over 
1,000 changes were issued directly impacting production. The backlog of 
engineering changes at the end of May 1993 reveals 5,800 open work 
authorizations.'' \8\ Additionally, costly changes to the program were 
government-imposed. A typical example is the C-17 test program which 
was increased from an 80 aircraft month test program to a ``rebaselined 
test program of approximately 152 aircraft months'' at the 
recommendation of the Defense Science Board study, with cost split 
evenly between the Government and MDC.\9\
---------------------------------------------------------------------------
    \8\ Report of the Defense Science Board Task Force on C-17 Review, 
December 1993, p 3
    \9\ Ibid. p 12
---------------------------------------------------------------------------
                       SOME EDUCATED CONCLUSIONS
    With the data that is immediately available from the historical 
record it is reasonable to conclude that the engineering changes were 
significant and made an impact on cost, schedule and performance on the 
C-17 program. Referring again to Battershell, of the $7.3B for the 
development costs and the first six aircraft in production Lots I and 
II, MDC invested approximately $1.7B of its own funds.\10\ This amounts 
to about 26 percent of the original contract amount of $6.6B. MDC 
proposed a claim against the government of $1.2B in addition to the 
$438M it had received in the settlement \11\ intended to recover what 
it believed to be legitimate costs associated with program changes a 
significant number of which were engineering changes. Though the term 
``significant'' used above is not a precise accounting regarding ``how 
much'' of the program was ``attributable'' to derived requirements, it 
does provide a qualitative data point that is important in evaluating 
opportunities to improve acquisition programs in the future. [See page 
11.]
---------------------------------------------------------------------------
    \10\ Battershell, p 90
    \11\ Government Accountability Office Report, (GAO/NDIAD-94-141), 
Military Airlift C-17 Settlement Is Not a Good Deal, April, 1994, p 6
---------------------------------------------------------------------------

                                  
