Best Practices: An Integrated Portfolio Management Approach to
Weapon System Investments Could Improve DOD's Acquisition
Outcomes (30-MAR-07, GAO-07-388).
Over the next several years, the Department of Defense (DOD)
plans to invest $1.4 trillion in major weapons programs. While
DOD produces superior weapons, GAO has found that the department
has failed to deliver weapon systems on time, within budget, and
with desired capabilities. While recent changes to DOD's
acquisition policy held the potential to improve outcomes,
programs continue to experience significant cost and schedule
overruns. GAO was asked to examine how DOD's processes for
determining needs and allocating resources can better support
weapon system program stability. Specifically, GAO compared DOD's
processes for investing in weapon systems to the best practices
that successful commercial companies use to achieve a balanced
mix of new products, and identified areas where DOD can do
better. In conducting its work, GAO identified the best practices
of: Caterpillar, Eli Lilly, IBM, Motorola, and Procter and
Gamble.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-388
ACCNO: A67565
TITLE: Best Practices: An Integrated Portfolio Management
Approach to Weapon System Investments Could Improve DOD's
Acquisition Outcomes
DATE: 03/30/2007
SUBJECT: Accountability
Allocation (Government accounting)
Best practices
Comparative analysis
Cost analysis
Defense capabilities
Defense economic analysis
Defense procurement
Private sector practices
Procurement planning
Procurement practices
Strategic planning
Weapons systems
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-388
* [1]Results in Brief
* [2]Background
* [3]Successful Companies Take a Disciplined, Integrated Approach
* [4]Identifying and Prioritizing Market Opportunities Lays the F
* [5]Companies Follow a Disciplined Process to Identify New Produ
* [6]Successful Portfolio Management Requires Strong Governance w
* [7]Lacking an Integrated, Portfolio-Based Approach, DOD Has Too
* [8]Service-centric Structure and Fragmented Decision-making Pro
* [9]DOD Commits to a Solution Earlier and with Less Knowledge
* [10]DOD Is Piloting Several Initiatives to Address Disconnects i
* [11]Conclusions
* [12]Recommendations
* [13]Agency Comments and Our Evaluation
* [14]Motorola
* [15]International Business Machines (IBM)
* [16]Procter & Gamble (P&G)
* [17]Eli Lilly Corporation
* [18]Caterpillar Corporation
* [19]GAO's Mission
* [20]Obtaining Copies of GAO Reports and Testimony
* [21]Order by Mail or Phone
* [22]To Report Fraud, Waste, and Abuse in Federal Programs
* [23]Congressional Relations
* [24]Public Affairs
Report to the Committee on Armed Services, U.S. Senate
United States Government Accountability Office
GAO
March 2007
BEST PRACTICES
An Integrated Portfolio Management Approach to Weapon System Investments
Could Improve DOD's Acquisition Outcomes
GAO-07-388
Contents
Letter 1
Results in Brief 3
Background 5
Successful Companies Take a Disciplined, Integrated Approach to Prioritize
Market Needs and Initiate a Balanced Mix of Executable Development
Programs 7
Lacking an Integrated, Portfolio-Based Approach, DOD Has Too Many Programs
Competing for Limited Resources 18
Conclusions 32
Recommendations 32
Agency Comments and Our Evaluation 33
Appendix I Objectives, Scope, and Methodology 35
Appendix II Comments from the Department of Defense 38
Related GAO Products 42
Table
Table 1: Cost and Cycle Time Growth for 27 Weapon Systems 5
Figures
Figure 1: DOD's Weapon System Investment Process 7
Figure 2: Portfolio Management Approach to Product Investments 9
Figure 3: IBM Market Segmentation 11
Figure 4: Risk Versus Rewards Matrix 16
Figure 5: Service Allocations of DOD's Investment Budget 22
Figure 6: Governance of DOD's Investment Process 24
Figure 7: Costs Remaining Versus Annual Appropriations for Major Defense
Acquisitions 26
Abbreviations
ACAT Acquisition Category
AOA Analysis of Alternatives
AT&L Acquisition, Technology, and Logistics
DAS Defense Acquisition System
DOD Department of Defense
FY fiscal year
GAO U.S. Government Accountability Office
IBM International Business Machines
ICD Initial Capabilities Document
IPD Integrated Product Development
JCIDS Joint Capabilities Integration and Development System
JCS Joint Chiefs of Staff
JROC Joint Requirements Oversight Council
JTRS Joint Tactical Radio System
MS Milestone
NPV Net Present Value
OSD Office of the Secretary of Defense
PA&E Program Analysis and Evaluation
PPBE Planning, Programming, Budgeting, and Execution
RDT&E research, development, test, and evaluation
USD Under Secretary of Defense
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office
Washington, DC 20548
March 30, 2007
The Honorable Carl Levin
Chairman
The Honorable John McCain
Ranking Member
Committee on Armed Services
United States Senate
Although the Department of Defense (DOD) produces the best weapons in the
world, it has not been able to deliver planned systems on time and within
budget. It is not unusual to see cost increases that add up to tens or
hundreds of millions of dollars, schedule delays that add up to years, and
large and expensive programs being scrapped after years of failing to
achieve promised capabilities.1 While recent changes to DOD's acquisition
policy held the potential to improve such outcomes, programs have
continued to experience significant cost and schedule overruns and
performance shortfalls.2 Over the next several years, DOD plans to invest
$1.4 trillion in major weapon system programs--doubling what it planned to
spend on such programs 5 years ago. Continued failure to deliver weapon
systems on time and within budget not only delays providing critical
capabilities to the warfighter, but results in less funding being
available for other DOD and federal priorities.
In the commercial market, effectively developing and marketing new
products is fundamental to the continued growth and success of companies.
Without a steady stream of product innovations to meet evolving market
needs, companies are likely to see their sales and profits fall. At the
same time, if the products in development outstrip their resources or do
not meet customer needs, companies can face financial ruin. Several recent
studies issued by leading thinkers in the area of product innovation and
development have reported that leading commercial companies achieve
success in product development by following a disciplined process for
ensuring they have the right mix of new products that meet customer needs
within available resources.
1GAO, Defense Acquisitions: Actions Needed to Get Better Results on
Weapons Systems Investments, [25]GAO-06-585T (Washington, D.C.: Apr. 5,
2006).
2GAO, Defense Acquisitions: Major Weapon Systems Continue to Experience
Cost and Schedule Problems under DOD's Revised Policy, [26]GAO-06-368
(Washington, D.C.: Apr. 13, 2006).
In fiscal year 2006, the Senate Armed Services Committee raised concerns
that DOD's poor track record with acquisition programs was linked not only
to the department's Defense Acquisition System (DAS) for managing product
development, but also to the department's Joint Capabilities Integration
and Development System (JCIDS) for identifying the warfighters' needs and
the Planning, Programming, Budgeting and Execution (PPBE) process for
allocating resources. Consequently, the Committee directed GAO to examine
how DOD's needs identification and resource allocation processes can
better support program stability in major weapon systems acquisition.3
This report (1) identifies best practices of successful commercial
companies for ensuring that they pursue the right mix of programs to meet
the needs of their customers within resource constraints and (2) compares
DOD's enterprise-level processes for investing in weapon systems to these
practices.4
To identify best practices of successful companies, we reviewed related
professional and academic publications, and interviewed knowledgeable
officials from five successful commercial companies: Caterpillar, Eli
Lilly, IBM, Motorola, and Procter & Gamble. While the products developed
by these companies range from heavy construction equipment and high-end
electronics to pharmaceuticals and household items, each of the companies
manages a large diversified portfolio of products, spends billions of
dollars annually on research and development, and has thousands of
employees worldwide. To examine DOD's processes for making investment
decisions, we reviewed related legislation and DOD directives,
instructions, and guidance; conducted interviews with and received
briefings from relevant Joint Staff, Office of the Secretary of Defense
(Comptroller, Program Analysis and Evaluation, and Acquisition, Technology
and Logistics), and other government officials; reviewed current
literature assessing DOD's decision-making processes; and analyzed DOD
requirements documents. We compared DOD's enterprise-level practices to
commercial best practices to identify potential areas for improvement. For
additional details on how we performed our review, see appendix I. Our
work was conducted between March 2006 and February 2007 in accordance with
generally accepted government auditing standards.
3S.Rep.No. 109-69 at 343-346 (2005).
4The Senate mandate specifically asked GAO to review the requirements and
budgeting portions of DOD weapon system acquisition. Therefore, we do not
focus on the DAS in this report. GAO has, however, produced a body of best
practices work focusing on the DAS; many of those reports are listed in
the related GAO products section of this report.
Results in Brief
Successful commercial companies use an integrated portfolio management
approach to prioritize market needs and allocate resources; thus, they
avoid pursuing more products than their resources can support and optimize
the return on their investment. Through portfolio management, all of a
company's product investments are addressed collectively from an
enterprise level, rather than as independent and unrelated initiatives.
Potential product developments are identified and assessed through a
systematic and disciplined screening process. Companies weigh the relative
costs, benefits, and risks of each proposed product using established
criteria and methods to select the best mix of products to develop. They
not only select those products that have a sound business case to warrant
further investment, but also those that help the company balance near- and
future-term market opportunities, different product lines, and available
resources against the demand for product investments. Once initial
investment decisions are made, they are revisited at multiple stages
throughout product development in a gated review and assessment process to
ensure products are still of high value. If not, companies make tough
decisions to defer or terminate investments and rebalance their product
portfolios. To be effective, portfolio management is enabled by strong
governance with committed leadership, clearly aligned organizational roles
and responsibilities, empowered portfolio managers who determine the best
way to invest resources, and accountability at all levels of the
organization.
Although the military services fight together on the battlefield as a
joint force, they do not identify warfighting needs and make weapon system
investment decisions together in an integrated manner. DOD has taken steps
to identify warfighting needs through a joint requirements process, but
its service-centric structure and fragmented decision-making processes do
not allow for the portfolio management approach used by successful
commercial companies to make investment decisions that benefit the
organization as a whole. DOD largely continues to define warfighting needs
and make investment decisions on a service-by-service basis, an approach
that has contributed to duplication in programs and equipment that does
not operate effectively together. Also, DOD assesses warfighting needs and
their funding implications under separate decision-making processes,
impeding its ability to prioritize warfighting needs so that it pursues
not only the ones that are most important but also the ones it can afford.
While DOD's JCIDS process provides a framework for reviewing and
validating the initial need for proposed capabilities, it does not focus
on the cost and feasibility of acquiring the capability to be developed
and fielded. Instead, these considerations are addressed through separate
budgeting and acquisition processes. Moreover, although DOD policy
provides for a series of early reviews--focused on the concept refinement
and technology development phases of proposed weapon system programs--in
prior work we found that the reviews are often skipped or are not fully
implemented. Consequently, proposed programs build momentum and move
toward starting product development with little if any early
department-level assessment of the costs and feasibility. Committing to
programs before they have this knowledge contributes to poor cost,
schedule, and performance outcomes and destabilizes acquisition programs
as the department attempts to pay for poorly performing programs by taking
funds from others.
The department has begun to pilot-test several interrelated initiatives
intended to address shortfalls in its existing approach to investment
decisions. These initiatives include a new approach to an early decision
gate for reviewing proposed programs at the concept stage, testing
portfolio management approaches in selected capability areas, and setting
up capital budgeting accounts for programs in development. However, as
currently structured, the initiatives are intended to operate within DOD's
existing organizational and process framework and may not allow for
sufficient authority and control over resources to effectively influence
weapon system investments.
To improve DOD's ability to deliver a balanced mix of weapon system
programs at the right time and right cost, we are recommending the
department establish an integrated, portfolio-based approach to
investments that incorporates best practices of successful commercial
companies. To ensure the success of such an approach, we are also
recommending that DOD establish a single point of accountability at the
department level with the responsibility, authority, and accountability
for ensuring that portfolio management for weapon system investments is
effectively implemented across the department. DOD concurred with the
majority of our recommendations and partially concurred with two.
Generally, in responding to these recommendations, DOD stated that it is
undertaking several initiatives and pilot efforts to improve the
department's approach to investment and program decision making, and that
implementation of any new business rules will be contingent upon the
outcome of these initiatives. However, we believe the department's current
initiatives do not fundamentally change DOD's service-centric framework or
sufficiently integrate its decision-making processes. DOD did not provide
comments regarding our recommendation that the Secretary establish a
single point of accountability.
Background
DOD's programs for acquiring major weapon systems have taken longer, cost
more, and delivered fewer quantities and capabilities than planned. We
have documented these problems for decades. Most recently, we reported
that 27 major weapon programs we have assessed since they began product
development have experienced cost increases of nearly 34 percent over
their original research, development, test, and evaluation (RDT&E)
estimates, and increases of almost 24 percent in acquisition cycle time
(see table 1).5
Table 1: Cost and Cycle Time Growth for 27 Weapon Systems
Billions of constant 2007 dollars
First full Latest full
estimate estimate Percentage change
Total cost $506.4 $603.1 19.1
RDT&E cost $104.7 $139.7 33.5
Weighted average acquisition 137.9 months 170.2 months 23.5
cycle timea
Source: GAO analysis of DOD data.
aThis is a weighted estimate of average acquisition cycle time for the 27
programs based on total program costs at the first full and latest
estimates. The simple average for these two estimates was 98.9 months for
the first full estimate and 124.6 months for the latest estimate resulting
in a 26.1 percent change.
When cost and schedule problems occur in one program, DOD often attempts
to pay for the poorly performing program by taking funds from others.
Doing so has destabilized other programs and reduced the overall buying
power of the defense dollar as DOD and the military services are forced to
cut back on planned quantities or capabilities to stay within budget
limitations. The F-22A Raptor program is a case in point: As costs
escalated in the program, the number of aircraft the Air Force planned to
buy was drastically reduced from 648 to 183. Similarly, as the Joint
Tactical Radio System (JTRS) encountered development problems, the number
of requirements was reduced or deferred by about one-third. As a result,
several programs that were dependent on JTRS also had to make adjustments
and go forward with alternative, less capable solutions. DOD's approach to
managing weapon system investments ultimately results in less funding
being available for other competing needs in DOD as well as other federal
priorities, as the expenditure of tax dollars within DOD reduces the
amount of funding available for those priorities.
5GAO, Defense Acquisitions: Assessments of Selected Major Weapon Programs,
[27]GAO-07-406SP (Washington, D.C.: Mar. 30, 2007).
Taking into account the differences between commercial product development
and weapons acquisitions, we have recommended that DOD adopt a
knowledge-based, incremental approach to developing and producing weapon
systems. This type of an approach requires program officials to
demonstrate that critical technologies are mature, product designs are
stable, and production processes are in control at key junctures in the
acquisition process.6
DOD has three major processes involved in making weapon system investment
decisions. These processes, depicted in figure 1, are the Joint
Capabilities Integration and Development System (JCIDS), for identifying
warfighting needs; the Planning, Programming, Budgeting and Execution
(PPBE) system, for allocating resources; and the Defense Acquisition
System (DAS), for managing product development and procurement. Much of
our prior work has focused on identifying commercial best practices that
could be used to improve the Defense Acquisition System--from the point
just before product development starts onward. In this report, however, we
look at earlier stages in DOD's investment process--from the point where
gaps in warfighting capability are assessed in JCIDS through the point
where alternative solutions to resolve those gaps are analyzed under the
DAS (see fig. 1).
6GAO, Best Practices: Capturing Design and Manufacturing Knowledge Early
Improves Acquisition Outcomes, [28]GAO-02-701 (Washington, D.C.: July 15,
2002); Best Practices: Better Matching of Needs and Resources Will Lead to
Better Weapon System Outcomes, [29]GAO-01-288 (Washington, D.C.: Mar. 8,
2001); and Best Practices: Better Management of Technology Development Can
Improve Weapon System Outcomes, [30]GAO/NSIAD-99-162 (Washington, D.C.:
July 30, 1999).
Figure 1: DOD's Weapon System Investment Process
Successful Companies Take a Disciplined, Integrated Approach to Prioritize
Market Needs and Initiate a Balanced Mix of Executable Development Programs
To ensure they achieve a balanced mix of executable development programs,
the successful commercial companies we reviewed use a disciplined and
integrated approach to prioritize market needs and allocate resources.
This approach, known as portfolio management, requires companies to view
each of their investments from an enterprise level as contributing to the
collective whole, rather than as independent and unrelated. With this
enterprise viewpoint, companies can effectively (1) identify and
prioritize market opportunities and (2) apply available resources to
potential products to select the best mix of products to exploit the
highest priority--or most promising--opportunities. Ultimately, each of
the companies we reviewed seeks to achieve a balanced portfolio that
maximizes the return on investments and moves the company toward achieving
its strategic goals and objectives. This type of approach depends on
strong governance with committed leadership, clearly aligned
responsibility, and effective accountability at all levels of the
organization.
As depicted in figure 2, a portfolio management approach begins with an
enterprise-level identification and definition of market opportunities and
then the prioritization of those opportunities within resource
constraints. Once opportunities have been prioritized, companies draft
initial business cases for alternative product ideas that could be
developed to exploit each of the highest priority opportunities. Each
alternative product proposal--represented by a black dot--enters a gated
review process. At each review gate, product proposals are assessed
against corporate resources, established criteria, competing products, and
the goals and objectives of the company as a whole. As alternatives pass
through each review gate, the number is expected to decrease, until only
those alternatives with the greatest potential to succeed make it into the
product portfolio.
Figure 2: Portfolio Management Approach to Product Investments
Identifying and Prioritizing Market Opportunities Lays the Foundation for
Achieving the Right Mix of Products
To make informed decisions about what market opportunities to target, the
companies we reviewed first establish a strategy that lays out the overall
goals, objectives, and direction for the company. As part of their
strategy, companies identify enterprise-level sales and profit targets,
strategic business areas they want to focus on, the extent to which
current products and new development efforts will support their growth
objectives, and how they will allocate resources across business units and
functional areas. This strategy provides a framework for the companies'
investment decisions. Within this framework, companies conduct a series of
market analyses to develop a comprehensive understanding of the market
environment, including product trends, technology trends, and customer
needs.
IBM for example, follows a structured market planning process to identify,
prioritize, and target attractive market segments. The first phase of this
process, called Market Definition, focuses on understanding the
marketplace, including identifying potential customers and their needs.7
During this phase, IBM examines the marketplace and technology
environments and identifies attractive market segments that contain
potential market opportunities--where customer wants or needs exist. Each
segment is categorized into one of four areas based on needs of the
customers and the company's product offerings (see fig. 3): "strike zone,"
"traditional," "pushing the envelope," and "white space." The strike zone
represents IBM's core business--market segments where IBM has an
established customer base that it is successfully serving with existing
product offerings. In contrast, white space represents market segments of
new customers with wants and needs that are new and different for IBM.
White space opportunities often require discovery and innovation. The
traditional and pushing the envelope areas fall between the strike zone
and white space. Traditional opportunities exist when new customers could
be attracted to an existing market--one IBM is already active in--by
modifying or enhancing existing products or services. Pushing the envelope
opportunities exist where the needs of current customer groups move them
into a new market segment. These attractive market segments are
prioritized during the next phase of IBM's process, known as the
Capability Assessment phase. During this phase each segment's overall
attractiveness and potential profitability are assessed, along with IBM's
available resources--like capital, cash, and current products--and its
competitive position within each segment. This analysis leads to the
selection of targeted market segments.
7IBM focuses on providing its customers with what it calls "industry
integrated solutions." As a result, portfolios within IBM are structured
around the customers' and buyers' behaviors and needs and not on specific
product offerings. In other words, the company focuses on providing
functional solutions to the customers, which could be done with a number
of product offerings and services, and not simply limiting themselves to
narrowly focused, single product solutions that the customer can take or
leave, an approach that IBM believes caused problems for the company in
the past.
Figure 3: IBM Market Segmentation
Motorola emphasizes the importance of targeting the right market segments
at the enterprise level to ensure that a balanced mix of project and
resource investments is maintained. Officials noted that excessively
focusing on segments that require new and innovative products can result
in long cycle times, wasted money, and lost opportunities elsewhere.
Likewise, critical opportunities can be lost when too much emphasis is
placed on simply continuing to invest in old markets with old products.
According to the officials we spoke with, the current investment mix for
Motorola's Government and Enterprise Mobility Solutions business unit is
roughly 70-20-10, where 70 percent of its projects and resources are
dedicated to maintaining its core business, while 20 percent are invested
in pursuing new markets with existing products or introducing new or
enhanced products into existing markets, and the remaining 10 percent are
dedicated to discovering new markets and new products.
As part of their market analyses, companies increasingly refine their
understanding of who their customers are and what they need. For several
of the companies we met with, determining the needs of their customers is
complex because they have multiple groups of customers to consider. For
example, Eli Lilly has four customer groups with diverse needs: patients,
doctors, insurance companies, and government regulators. This complexity
is compounded when considering that success in a worldwide market is
critically dependent on a company's ability to operate within different
governmental systems, laws, and regulations; and regional markets. Several
of the companies we reviewed use a variety of methods--including
interviews, surveys, focus groups, and concept tests--to actively engage
their customers and help determine what they need. Some companies also
observe customer behaviors to identify unstated wants and needs that if
met--assuming corporate knowledge and resources allow--could actually
exceed customer expectations. While companies actively seek customer input
to identify products that show the most promise and satisfy customer
needs, customers generally do not identify specific products to be
developed.
Companies Follow a Disciplined Process to Identify New Products and Achieve a
Balanced Portfolio
Once companies have identified and prioritized their market opportunities,
they follow a disciplined process to assess the costs, benefits, and risks
of potential product alternatives and allocate resources to achieve a
balanced portfolio that spreads risk across products, aligns with the
company's strategic goals and objectives, and maximizes the company's
return on investment. At an early stage, each alternative product is
expected to be accompanied by an initial business case that contains
knowledge-based information on strategic relevance and estimates of cost,
technology maturity, and the cycle time for getting the product from
concept to market. To ensure comparability across alternatives, companies
require initial business case information to be developed in a transparent
manner, to use specific standards, and to report estimates within certain
levels of confidence or allowable deviations. Each of the companies we
reviewed also stressed the importance of having multiple management review
points, or gates, at early phases to assess and prioritize alternative
products. As products move through review gates, from ideas, to more
concrete concepts, to the start of development where a final business case
is made, companies expect uncertainties--which are typically inherent in
the early phases--to be addressed and estimates to become more precise.
Consequently, the number of viable alternatives tends to decrease at each
review gate as those with the lowest potential for success and least value
are terminated or deferred, while those that are poised to succeed and
providing the best value are approved to proceed (illustrated in fig. 2).
Companies emphasized that making tough go/no-go decisions is critical to
keeping a balanced portfolio. Over time, as potential new products are
identified, companies review them against other product investments
(proposed and existing) and rebalance their portfolios based on those that
add the most value.
The companies we visited each follow a disciplined, gated review process
to ensure that they commit to development programs that help balance the
portfolio and that are executable given available corporate resources.
This allows companies to avoid committing to more programs than their
resources can support and ensure stability in the programs they invest in.
Although the number of review gates prior to the start of full-scale
product development varied between companies--ranging from four at Procter
& Gamble, to eight at Motorola--they all required potential products to
follow an established, disciplined process and meet specified criteria at
each review point. For example, Caterpillar assesses product alternatives
at four review gates prior to the start of development--three of which
were recently added to enhance the rigor of its investment decision
making. Each alternative must be supported by a draft business case that
includes quantifiable data that can be compared with specific standards
and used to determine if the related product can move past that gate. At
each gate, alternatives are reviewed to ensure that knowledge about
critical technologies, life-cycle costs, product reliability, and product
affordability is being acquired and that the product contributes to
achieving the company's strategic goals and objectives.
Because developing a new drug is costly and time consuming, Eli Lilly
requires that the data supporting potential new drugs must meet high
standards to ensure that managers are informed to make sound investment
decisions.8 Each potential new drug must be supported by an initial
business case that contains information about safety and efficacy;
forecasted revenue; expected unit demand; capital, medical, supply and
material, development, and selling and marketing expenses; and general
administrative costs. The initial business case must also identify
critical success factors, state the probability of technical success, and
provide a timeline that details when major milestone events are expected
and how long it will take to get the associated drug to market. Eli Lilly
assesses, approves, and funds proposed new drugs incrementally. At each
milestone review a contract is established between the project team and a
gatekeeper committee, which contains deliverables, time frames, and the
costs to get to the next milestone. Once this contractual agreement is
reached the budget is allocated for the entire phase. The gatekeeper
committees expect each new drug proposal to achieve an 80-percent
confidence level in their cost and schedule estimates for the next phase.
This high level of confidence is achievable in large part because final
budget estimates are not developed by project teams until 2 months prior
to the milestone review. Projects are terminated at early points in the
review process when it is determined that their critical success factors
cannot be achieved. Because Eli Lilly's projects typically have a high
degree of technical risk, only about 1 percent of those that start early
development actually make it to the marketplace.
8According to Eli Lilly representatives, the average cost to discover and
develop a new drug ranges from $800 million to $1 billion, and the average
length of time to get a new drug from discovery to the market is 10 to 15
years.
Motorola officials also emphasized the importance of having sound
information when assessing potential new products. They noted that a
process without sound information will not produce good outcomes.
Therefore, Motorola's Government and Enterprise Mobility Solutions
business unit expects potential products to be supported by initial
business cases containing data that meet specific standards and levels of
confidence at each review gate. For example, cost estimates for potential
products are developed in several phases and are expected to increase in
confidence with each successive phase. Early in the investment planning,
when an initial business case is first drafted, the confidence parameters
are generous, ranging from as much as 75 percent higher to as much as 25
percent lower than what the project will likely end up costing. By the
time a product alternative reaches the beginning of product development,
when a final business case is made, Motorola expects the cost estimates to
be at confidence levels of 10 percent higher and 5 percent lower. Proposed
products that fail to meet the specified criteria at early review gates
are either terminated or sent back to further mature and reenter the
review process from the beginning.
The companies we reviewed use a variety of portfolio management tools and
methods to inform the investment and resource allocation decisions they
make at each review gate. Some companies employ scoring methods, using
experts to rate products based on a number of factors--such as strategic
fit, risk, and economic value--and use that information to prioritize
alternative products. Another common tool plots alternative products on a
decision matrix that compares factors such as costs and benefits, or risks
and rewards of competing alternatives. Using this type of matrix,
alternative products are often represented by circles, where the size of
the circles provides information about key constraints such as available
annual resources or the estimated annual costs for each alternative. For
example, figure 4 compares risk and expected rewards9 by plotting
competing alternatives on a matrix. Alternatives that fall into the upper
left quadrant are high risk and low reward, while alternatives that fall
into the lower right quadrant are low risk and high reward. By weighing
risk against rewards and considering constraints such as annual resources
or annual cost, this tool provides critical information and a structured
means to help managers make informed decisions. Company officials at
Procter & Gamble emphasized the importance of selecting a balanced mix of
products to pursue. They noted that pursuing only low-risk and high-reward
products at the expense of more innovative, higher-risk products could
cause the company to miss out on opportunities to improve their
competitive standing in the marketplace. Likewise, excessive pursuit of
higher-risk products with the potential for high returns could also result
in lost opportunities to elsewhere.
9Companies commonly measure rewards and benefits using cash flow analysis
known as net present value (NPV) analysis. NPV techniques can show, in
today's dollars, the relative net cash flow of various alternatives over a
long period of time. Simply stated, net cash flow is the amount of dollars
that is left after sales and revenues have offset expenses. In general,
the greater the net cash flow for a particular investment, the greater the
return on the investment.
Figure 4: Risk Versus Rewards Matrix
Recognizing the inherent risks in pursuing a new development program--that
overruns or underruns in one business case result in lost opportunity to
invest resources in another worthwhile project--IBM permits products to
deviate from their original business case estimates as long as the
deviation is within established limits. These limits are specified in a
contractual document resulting from negotiations between senior management
and project managers and signed at the beginning of product development.
Product development teams are expected to execute according to the
contract; if established thresholds are breached, action is taken
immediately to reassess the product within the context of the portfolio
and determine whether it is still a relevant and affordable investment to
pursue.
Successful Portfolio Management Requires Strong Governance with Committed
Leadership, Empowered Decision Makers, and Effective Accountability
Successful portfolio management requires strong governance with committed
leadership that empowers portfolio managers to make decisions about the
best way to invest resources and holds those managers accountable for the
outcomes they achieve. The companies we reviewed indicated that it is
critical to have commitment from the top leaders of the organization and
recognition at all levels that what is best for the company must be a
priority, and not simply what is best for a particular business unit or
product line. In addition, the companies emphasized that roles and
responsibilities for implementing portfolio management, including the
designation of who is responsible for product investment decisions and
oversight, must be clearly defined. Because portfolio managers are on the
front line, the companies we reviewed empower these managers to make
product investment decisions and hold them accountable for outcomes, not
just for individual products but also for the overall performance of their
portfolios. To support their portfolio managers, the companies encourage
collaboration and communication, including sharing bad news early. Several
companies also emphasized the importance of supporting their portfolio
managers with cross-functional teams, composed of representatives from the
key functional areas within the company--such as science and technology,
marketing, engineering, and finance--to ensure that they are adequately
informed when making investment decisions. To ensure accountability,
companies often use incentives and disincentives, including promotion and
termination. We have previously reported that high-performing
organizations have monetary and other rewards that clearly link employee
knowledge, skills, and contributions to achieving the organization's goals
and objectives.10 These organizations underscore the importance of holding
individuals accountable and aligning performance expectations with
organizational goals and cascade those expectations down to lower levels.
Companies stressed that the transformation to portfolio management takes
time and requires not only process changes but cultural changes throughout
the company.
Eli Lilly emphasized that a key to making its portfolio management process
work is having a single committee with a high-level official in charge
responsible for making product investment decisions. Previously, the
company had a multi-layered committee structure in place, and decisions
were made based on reaching a consensus--an approach that was viewed as
cumbersome and lengthy. Eli Lilly also ensures accountability by directly
linking management and employee bonuses to the overall success of the
company. Individual employee performance objectives are aligned with
specific company objectives, such as meeting budgetary goals, time frames,
and data quality levels for a given project. Achievement of individual
employee objectives is measured periodically to provide feedback to the
employee. Eli Lilly officials stressed that having the right performance
metrics in place is important because ultimately you get what you measure;
therefore, be sure to measure the right things.
10GAO, Results-Oriented Cultures: Creating a Clear Linkage between
Individual Performance and Organizational Success, [31]GAO-03-488
(Washington, D.C.: Mar. 14, 2003).
Motorola considers accountability to be the critical factor in making its
portfolio management process successful. In addition, Motorola's culture
is not averse to reporting bad news to management. Project managers are
encouraged to report problems early so that they can be addressed before
they get out of control. Senior managers, however, are not intimately
involved in the day-to-day decision making for individual products. That
responsibility, in nearly every case, is delegated to the business unit
general manager. The general manager of a business unit is held
accountable for ensuring that the products within his unit succeed at all
levels. The general manager is responsible for holding product managers
accountable for the attainment of critical knowledge at key points and the
performance of their individual products overall. General managers and
product managers can be fired for not meeting objectives. Motorola
believes that if managers are held accountable for results, then they have
more desire to get it right.
Lacking an Integrated, Portfolio-Based Approach, DOD Has Too Many Programs
Competing for Limited Resources
Although the military services fight together on the battlefield as a
joint force, they do not identify warfighting needs and make weapon system
investment decisions together. DOD has taken steps to identify warfighting
needs through a more joint requirements process, but the department's
service-centric structure and fragmented decision-making processes are at
odds with the integrated, portfolio management approach used by successful
commercial companies to make enterprise-level investment decisions.
Consequently, DOD has less assurance that its weapon system investment
decisions address its most important warfighting needs and are affordable
in the context of its overall fiscal resources. In addition, DOD commits
to products earlier than the companies we reviewed and with far less
knowledge about their cost and feasibility. This leads to poor program
outcomes and funding instability, as the department attempts to fix
troubled programs by taking funds from others.
Service-centric Structure and Fragmented Decision-making Processes Impede DOD's
Ability to Prioritize Warfighting Needs
Although recent DOD policy emphasizes a more joint approach to identifying
and prioritizing warfighting needs,11 DOD's service-centric structure and
fragmented decision-making processes hinder the policy's successful
implementation. This policy, which introduced the JCIDS process, calls for
a wider range of stakeholders than before, including more customer (i.e.,
combatant command) involvement; introduces new methodologies intended to
foster jointness; and groups warfighting needs into eight functional areas
based on warfighting capabilities--such as netcentric, force application,
and battlespace awareness12--that cut across the military services and
defense agencies. The JCIDS process emphasizes early attention to the
fiscal implications of newly identified needs, including identifying ways
to pay for new capabilities by divesting the department of lower priority
or redundant capabilities. Despite these provisions, assessments of
warfighting needs continue to be driven by the services and to be based on
investment decision-making processes that do not function together to
ensure that DOD pursues needs that its resources can support.
The military services identify warfighting needs individually, and
department-level organizations are not optimized to integrate the
services' results or evaluate their fiscal implications early on.
Historically, this approach has contributed to duplication in weapon
systems and equipment that does not interoperate. At the department level,
Functional Capability Boards oversee each of the eight functional areas,
reviewing the services' assessments, and providing recommendations to the
Joint Requirements Oversight Council (JROC), which leads the JCIDS
process. However, defense experts and DOD officials report that the
Functional Capability Boards do not have the staff or analytical resources
required to effectively evaluate service assessments within the context of
the broader capability portfolio and assess whether the department can
afford to address a particular capability gap. Several recent studies have
recommended that DOD increase joint analytical resources for a less
stovepiped understanding of warfighting needs.13 In addition, the boards
lack the authority to allocate resources and to make or enforce decisions
to divest their capability area of existing programs to pay for new
ones--authority successful companies provide to their portfolio managers.
Finally, some defense experts contend that the service ties of JROC's
members--that is, the services' Vice Chiefs and the Assistant Commandant
of the Marine Corps--reinforce service stovepipes. To better ensure a more
joint perspective, they recommend a more diverse JROC, with
representatives from other department-level organizations and the
combatant commands.14
11Chairman of the Joint Chiefs of Staff Instruction, Joint Capabilities
Integration and Development System, CJCSI 3170.01E (May 11, 2005). The
original instruction was CJCSI 3170.01C (June 24, 2003).
12The other capability areas are command and control, focused logistics,
force management, force protection, and joint training.
Resource allocation decisions are made through a separate process--the
Planning, Programming, Budgeting, and Execution system (PPBE)--which
hinders the department's ability to weigh the relative costs, benefits,
and risks of investing in new weapon systems early on. Within the PPBE
system, the individual military services are responsible for budgeting and
allocating resources under authority that is commonly understood to be
based on Title 10 of the United States Code.15 PPBE is structured by
military service and defense program, although the department integrates
data on the services' current and projected budget requests under 11
crosscutting mission areas called Major Force Programs. The cross-cutting
view provided by the Major Force Program structure is intended to
facilitate a strategic basis for resource allocation, allowing the
Secretary of Defense to more easily see where the greatest mission needs
are and to re-allocate funds to meet those needs regardless of which
service stands to gain or lose. However, we have reported in the past that
the Major Force Program structure has not provided sufficient visibility
in certain mission areas.16 Moreover, although they cut across the
services, the program mission areas are not consistent with the more
recently established capability areas used in the JCIDS process,17 and as
a result, it is difficult to relate resources to capabilities. For
example, in prior work, we observed that the Major Force Programs contain
large numbers of programs with varied capabilities, complicating
comparisons needed to understand defense capabilities and associated
trade-off decisions.18 We have recommended that DOD report funding levels
for defense capabilities in its Future Years Defense Program report to the
Congress, which is currently organized by the Major Force Programs.
13Institute for Defense Analyses, Improving Integration of Department of
Defense Processes for Capabilities Development and Planning (Sept. 2006);
Center for Strategic and International Studies, Beyond Goldwater-Nichols:
Defense Reform for a New Strategic Era, Phase 1 Report (Mar. 2004); and
Joint Defense Capabilities Study Team (DOD), Joint Defense Capabilities
Study: Improving DoD Strategic Planning, Resourcing and Execution to
Satisfy Joint Capabilities. Final Report (Jan. 2004).
14Assessment Panel of the Defense Acquisition Performance Assessment
Project for the Deputy Secretary of Defense, Defense Acquisition
Performance Assessment Report (Jan. 2006); Center for Strategic and
International Studies, Beyond Goldwater-Nichols: U.S. Government and
Defense Reform for a New Strategic Era, Phase 2 Report (July 2005); and M.
Thomas Davis, "The JROC: Doing What? Going Where?" National Security
Studies Quarterly (Summer 1998).
15 Sections 3013, 5013, and 8013 of Title 10 grant authority to the
Secretaries of the Army, the Navy, and the Air Force, respectively, to
conduct all affairs of their departments, including recruiting,
organizing, supplying, equipping, training, servicing, mobilizing,
demobilizing, administering, maintaining, and military construction and
maintenance.
In addition, our analysis of DOD's investment accounts--which pay for
developing, testing, and buying weapon systems and other
equipment--indicates that DOD generally does not allocate resources on a
strategic basis. Figure 5 illustrates that the service allocations as a
percentage of the department's overall investment budget have remained
relatively static for the 25-year period we examined, even though DOD's
strategic environment and warfighting needs have changed dramatically
during that time, with the demise of the cold war and the emergence of the
global war on terror.19 In contrast, successful commercial companies using
portfolio management would expect to see their resource allocations across
business areas to reflect changes in the marketplace and the competitive
environment.
16GAO, Military Transformation: Actions Needed by DOD to More Clearly
Identify Triad Spending and Develop a Long-term Investment Approach,
[32]GAO-05-540 (Washington, D.C.: June 30, 2005).
17The Major Force Programs are as follows: Strategic Forces; General
Purpose Forces; Command, Control, Communications, and Intelligence;
Mobility Forces; Guard and Reserve Forces; Research and Development;
Central Supply and Maintenance; Training, Medical, and other General
Personnel Activities; Administration and Associated Activities; Support of
Other Nations; and Special Operations Forces.
18GAO, Future Years Defense Program: Actions Needed to Improve
Transparency of DOD's Projected Resource Needs, [33]GAO-04-514
(Washington, D.C.: May 7, 2004).
19The fiscal framework of the federal government constrains DOD's ability
to totally control its investment allocations. Ultimately resource
allocation decisions are made by the Congress in the annual authorization
and appropriations process.
Figure 5: Service Allocations of DOD's Investment Budget (FY1986 through
FY2011)
PPBE and JCIDS are led by different organizations (see fig. 6), as is the
third of the three processes involved in DOD's weapon system investment
decisions, the Defense Acquisition System (DAS), making it difficult to
hold any one person or organization accountable for investment outcomes.
The 2006 Quadrennial Defense Review highlighted the need for governance
reforms,20 and a 2006 study commissioned by DOD observed that the budget,
acquisition, and requirements processes are not connected organizationally
at any level below the Deputy Secretary of Defense, concluding that this
structure induces instability and erodes accountability.21 The Under
Secretary of Defense/Acquisitions, Technology, and Logistics (USD/AT&L)
has stated that weapon system investment decisions are a shared
responsibility, and, therefore, no one individual is accountable for these
decisions. At a broader, strategic level, we have stated in prior work
that DOD has lacked sustained leadership and accountability for various
department-wide management reform efforts,22 including the establishment
of an effective risk management approach as a framework for decision
making.23 This approach would link strategic goals to plans and budgets,
assess the value and risks of various courses of action as a tool for
setting investment priorities and allocating resources at the department
level, and use performance measures to assess outcomes. To address the
lack of sustained leadership, we have supported legislation to create a
chief management official at DOD.24
20The 2006 Quadrennial Defense Review led to the Institutional Reform and
Governance project, which is focusing on (1) integrating core decision
processes, (2) aligning and focusing the department's governance and
management functions under an integrated enterprise model, as well as (3)
establishing a common and authoritative analytical framework to link
strategic decisions to execution.
21Assessment Panel of the Defense Acquisition Performance Assessment
Project for the Deputy Secretary of Defense, Defense Acquisition
Performance Assessment Report (January 2006).
22GAO, Department of Defense: Sustained Leadership Is Critical to
Effective Financial and Business Transformation, [34]GAO-06-1000T
(Washington, D.C.: Aug. 3, 2006); and Department of Defense: Further
Actions Are Needed to Effectively Address Business Management Problems and
Overcome Key Business Transformation Challenges, [35]GAO-05-140T
(Washington, D.C.: Nov. 18, 2004).
23GAO, Defense Management: Additional Actions Needed to Enhance DOD's
Risk-Based Approach for Making Resource Decisions, [36]GAO-06-13
(Washington, D.C.: Nov. 15, 2005).
24S. 780, 109th Cong. S1 (2005).
Figure 6: Governance of DOD's Investment Process
The Office of the Secretary of Defense (OSD) does not assess the funding
implications of a proposed program at the front end of the investment
process, when it is initially validated by JROC. JCIDS is a continuous,
need-driven process that unfolds in response to warfighting needs as they
are identified. However, PPBE is a calendar-driven process comprised of
phases that occur over a 2-year cycle, thus OSD's formal review of a
proposed program is not often synchronized with JROC's, and can occur
several years later.25 Nevertheless, according to Joint Staff and AT&L
officials we met with, proposed programs begin to gain momentum when they
are validated by JROC, and they become very difficult to stop. These
officials indicated that momentum begins to gather because the services
start programming and budgeting for the proposed capability right away to
secure funding, generally several years before actual product development
begins and before OSD formally reviews the services' programming and
budgeting proposals. In the interim, the services have not only budgeted
for their proposed programs, but established a program office, conducted
their Analysis of Alternatives, and identified specific user requirements.
OSD's programming and budgeting review occurs at the back end of the
investment process, when it is difficult and disruptive to make changes,
such as terminating existing programs to pay for new, higher priority
programs.
25DAS is an event-driven process structured into discrete phases separated
by major decision points called milestones that can be tailored to
individual programs.
These practices have contributed to the department starting more programs
than its resources can support. DOD defers much of the additional cost of
its programs into the future, resulting in what some have characterized as
a fiscal bow wave (illustrated in fig. 7). This bow wave has grown at a
pace that greatly exceeds DOD's annual funding increases. The cost
remaining for DOD's major weapons programs increased almost 135 percent
between 1992 and 2006, while the department's annual funding level only
increased 57 percent over that same time period. If this trend goes
unchecked, Congress will likely be faced with a difficult choice: pull
funds from other high-priority federal programs to support DOD's
acquisitions or accept less warfighting capability than originally
promised.
Figure 7: Costs Remaining Versus Annual Appropriations for Major Defense
Acquisitions
DOD Commits to a Solution Earlier and with Less Knowledge
DOD commits to a solution to address a warfighting need earlier in the
investment process than commercial companies do and before it has adequate
knowledge about cost and technical feasibility. Proposed options for
resolving a gap in military capability are submitted in an Initial
Capabilities Document (ICD). DOD guidance states that this document should
contain a range of approaches based in part on the cost and technological
feasibility posed by the approaches, laying the foundation for a more
detailed Analysis of Alternatives to be conducted under the Defense
Acquisition System. In addition, JROC is to receive a briefing on the ICD
that follows a standard format and addresses such issues as
o linkage of the proposal to strategic guidance;
o the time frame within which the capability is needed;
o the threat/operational environment;
o risks and assumptions (including the risk associated with
proceeding and not proceeding with solutions to each); and
o a description of the best materiel and non-materiel approaches
based upon cost, efficacy, performance, technology maturity, and
risk.
Although DOD guidance calls for the analysis of a solution's cost and
feasibility, we found that ICDs contained little of this type of
information. Several DOD officials we met with, who are directly involved
in the JCIDS process, did not believe cost and feasibility information was
mandated at this point. In our review of 14 unclassified ICDs approved by
JROC from 2003-06, we found that 11 did not contain acquisition cost
estimates and 12 did not contain estimates of the technical feasibility of
proposed solutions.26 We also found that JCIDS guidance does not specify
the level of accuracy sought in cost and feasibility estimates, and a
white paper that does provide recommendations in this regard is
advisory.27
We found that ICDs generally focused on the strategic, or operational,
relevance of proposed solutions, but a lack of guidance and an evolving
methodology have raised questions about the accuracy of data supporting
those assessments. JCIDS uses new joint warfighting concepts28 to
translate top-level military strategy into the capabilities a commander
might need on the battlefield. The joint concepts underpin a
capabilities-based approach29 to identifying requirements, in which
analyses are expected to focus on broad military capabilities rather than
service-specific platforms. However, the joint concepts and
capabilities-based assessments are works in progress. The concepts are
being updated due to concerns about their scope, and guidance on
conducting a capabilities-based analysis has been lacking. Several DOD
officials we met with stated that assessments vary in their rigor, and a
senior Joint Staff official said that training on requirements development
is one of three central challenges at present. In January 2007, we
reported that DOD officials described concerns about the analytical
framework for a capabilities-based assessment on joint seabasing, which
could lead to inaccurately identifying gaps in implementing the concept.30
Enhancing a seabasing capability is expected to be costly and could be the
source of billions of dollars of investment if DOD chooses an option
involving the development of new ships.
26The 14 ICDs we reviewed are related to weapon systems and were finalized
after June 24, 2003 (the publication date for the initial JCIDS
instruction). They are unclassified ACAT I, II, or III ICDs contained in
the Joint Staff's Knowledge Management Decision Support Tool database,
which serves in part as a repository for JCIDS documents.
27The white paper suggests roughly characterizing the 20-year-lifecycle
costs of proposed solutions in terms of developmental costs, facility or
infrastructure costs, per-unit and rough force-level acquisition costs,
and recurring operating costs. It states that rough estimates of the
technical feasibility of proposed solutions should be developed, not at
the engineering level because of the broad range of possibilities, but at
the least to characterize as no risk, very low risk, low risk, medium
risk, and high risk using legitimate technology experts. "Whitepaper on
Conducting a Capabilities-Based Assessment Under the Joint Capabilities
Integration and Development System," Joint Chiefs of Staff J-8/Force
Application Assessment Division (Jan. 2006).
28Joint future concepts are visualizations of future operations that
describe how a commander might employ warfighting capabilities to achieve
effects and objectives.
29The 2001 Quadrennial Defense Review directed DOD to implement a
capabilities-based planning approach. Capabilities-based planning has been
described as a framework for defense planning and decision making in a
strategic environment characterized by uncertainty and a fiscal
environment characterized by limited resources. See Paul K. Davis,
Analytic Architectures for Capabilities-Based Planning, Mission System
Analysis, and Transformation (Santa Monica, Calif.: RAND, 2002).
DOD does not consistently follow a disciplined review process to ensure
that proposed solutions are making progress toward an executable
development program, although DOD policy emphasizes that such reviews are
necessary.31 DOD's policy identifies several key decision points prior to
starting a new weapon system development program:
o an initial decision point, where the Initial Capabilities
Document is reviewed, validated, and approved by the JROC;
o a Concept Decision review, where entry into the concept
refinement phase of the Defense Acquisition System should be
authorized; and
o a Milestone A decision point, where a preferred solution and a
technology development strategy should be reviewed and approved.32
Since Initial Capabilities Documents generally do not contain information
on cost and technical feasibility, the JROC does not have a sufficient
basis for making go/no-go decisions at the initial decision point. In the
4 years since JCIDS was implemented, nearly all of the warfighting needs
identified by the services and submitted for review in an ICD have been
validated and sent into the acquisition pipeline for further analysis as
potential programs, which calls into question whether go/no-go decisions
are the point of this first key gate. Information on cost and feasibility
is generally developed after the ICD is approved and proposed solutions
undergo further refinement through an Analysis of Alternatives (AOA). An
AOA should compare alternative solutions in terms of life-cycle cost,
schedule, and operational effectiveness, leading up to the identification
of a preferred alternative. However, officials from PA&E and the Joint
Staff indicate that AOAs often make a case for a single preferred
solution. Several of them indicated other concerns about AOAs, such as not
setting up trade-off discussions, lack of analytical rigor, length, and
timeliness.
30GAO, Force Structure: Joint Seabasing Would Benefit from a Comprehensive
Management Approach and Rigorous Experimentation before Services Spend
Billions on New Capabilities, [37]GAO-07-211 (Washington, D.C.: Jan. 26,
2007). Joint seabasing is one of several evolving concepts describing how
commanders in the future will project and sustain forces for conducting
joint military operations without relying on immediate access to nearby
land bases.
31CJCSI 3170.01E , Joint Capabilities Integration and Development System;
DODI 5000.2, Operation of the Defense Acquisition System.
32The JROC has another decision point just prior to program initiation,
when it reviews, validates, and approves a Capabilities Development
Document. Approval and validation of a Capabilities Development Document
is a key entrance criteria for initiating a new development program at
Milestone B. We did not include the Capabilities Development Document
decision point in our current review because it is closely associated with
Milestone B, the focus of many of our former reviews. See the list of
related GAO products on the last pages of this report.
In any case, the next review points--the Concept Decision and Milestone
A--are often skipped; thus, the opportunity to review an evolving business
case and to make go/no-go decisions is bypassed. In prior work, we found
that 80 percent of the programs we reviewed entered the Defense
Acquisition System at Milestone B without holding any prior major reviews,
such as a Milestone A review.33 Such reviews are intended to provide
acquisition officials with an opportunity to assess whether program
officials had the knowledge needed to develop an executable business case.
Senior officials with OSD confirmed that this is a common practice among
defense acquisition programs. We concluded that this practice eliminates a
key opportunity for decision makers to assess the early product knowledge
needed to establish a business case that is based on realistic cost,
schedule, and performance expectations. In addition, we found that
programs are regularly approved to begin development even though officials
reported levels of knowledge below the criteria suggested in DOD's
acquisition policy.
There is, then, generally little department-level oversight between the
point at which an ICD is approved and when system-level requirements are
validated and product development is initiated. At this point, as we
indicated earlier, there is generally no turning back, because the
services have invested considerable time and money, established a budget,
and formed a constituency for a proposed program, and decision makers
become reluctant to terminate a program or send it back for further study.
33GAO, Defense Acquisitions: Major Weapon Systems Continue to Experience
Cost and Schedule Problems under DOD's Revised Policy, [38]GAO-06-368
(Washington, D.C.: Apr. 13, 2006). Our review focused on the Concept
Decision and Milestone A reviews points and did not assess the extent to
which JROC reviews were held.
DOD Is Piloting Several Initiatives to Address Disconnects in Investment
Decision-making
In response to the 2006 Quadrennial Defense Review and other recent
acquisition reform studies,34 DOD has undertaken several key, interrelated
initiatives intended to strengthen the department's approach to investment
decisions. The initiatives include (1) taking a new approach to reviewing
proposed concepts that will provide decision makers with an early
opportunity to evaluate trade-offs among alternative approaches to meeting
a capability need, (2) testing portfolio management approaches in selected
capability areas to facilitate more strategic choices about how to
allocate resources across programs, and (3) using capital budgeting as a
potential means to stabilize program funding. While promising, these
initiatives do not fundamentally change DOD's existing service-centric
framework for making weapon system investment decisions.
To address a perceived gap between DOD's major decision-making processes
and provide a department-level means to assess potential solutions
(materiel and non-materiel) to fill a validated capability need, DOD is
testing a new approach to a Concept Decision review, which will take place
after a warfighting need is validated by the JROC. This new approach is
intended to focus attention on the affordability and feasibility of
potential solutions and generate early cost, schedule, and performance
trade-offs prior to the point of a significant investment commitment. As
currently proposed, the Concept Decision will be informed by a newly
required Evaluation of Alternatives that will integrate the Functional
Solutions Analysis conducted under JCIDS with the Analysis of Alternatives
conducted under the acquisition system and lay out the relative merits and
limitations of potential solutions. Furthermore, concept decision reviews
will be implemented by a tri-chair board consisting of lead decision
makers from the JCIDS, PPBE, and DAS processes. While promising, the
Concept Decision review largely reinstitutes a review point that already
existed but was only intermittingly used. For Concept Decision reviews to
be effective, DOD will have to establish enforcement and accountability
mechanisms to ensure the reviews are actually implemented. In addition,
the extent to which the concept reviews can achieve desired effects will
depend on what authority Concept Decisions carry and who will be held
accountable, particularly in light of the service-dominated investment
structure that currently exists.
34Defense Acquisition Performance Assessment (January 2006); Defense
Science Board Summer Study on Transformation: A Progress Assessment
(February 2006); Beyond Goldwater-Nichols: U.S. Government and Defense
Reform for a New Strategic Era, Phase 2 Report (July 2005); and Joint
Defense Capabilities Study (January 2004).
The department has also begun to pilot-test capability-based portfolio
management, selecting four joint capability areas to focus on--joint
command and control, joint net-centric operations, battlespace awareness,
and joint logistics. The intent is to enable the department to develop and
manage capabilities, as opposed to simply individual programs, and enhance
the integration and interoperability within and across sets of
capabilities. Each portfolio is being structured somewhat differently to
help the department determine how best to proceed with portfolio
management. All, however, are intended to focus initially on existing
programs and to operate within DOD's existing decision-making framework.
The portfolios are largely advisory and will, as a first step, provide
input to decisions made through the JCIDS, PPBE, and DAS processes. At
this point, the capability portfolio managers have not been given direct
authority to manage fiscal resources and make investment decisions.
Without portfolios in which managers have authority and control over
resources, DOD is at risk of continuing to develop and acquire systems in
a stovepiped manner and of not knowing whether its systems are being
developed within available resources.
DOD is also examining the use of capital accounts as a potential means of
stabilizing program funding, which has long been cited as a significant
issue in program management. This capital budgeting pilot initiative is in
the early stages of planning, and the specifics of how such accounts will
be implemented are being developed, but the intent is for DOD to commit a
set amount of funding for the development portion of a project and hold to
that commitment by not adjusting funding up or down until the product is
delivered. In addition to resource constraints, programs would be given a
fixed amount of time to get from one milestone to the next. If successful,
this initiative could represent a step toward stabilizing long-term costs
within major defense acquisition programs, as well as a strengthening of
the ability of program managers to conduct long-term planning and control
costs. However, for this initiative to be effective, DOD will need to
overcome long-standing problems it has had in starting programs without
sufficient knowledge of the costs, requirements, and technologies needed
to develop proposed weapon systems. Unless this changes, it is unlikely
that capital accounts will lead to increased program stability.
Conclusions
While DOD has increasingly strengthened its ability to operate as a joint
force on the battlefield, the department's organizational structures,
processes, and practices for planning and acquiring weapon systems are not
similarly joint. Put simply, DOD largely continues to base its investment
decisions on service-driven analyses that do not provide an
enterprise-level understanding of overall warfighting needs and on
individual platforms rather than broader sets of capabilities. In
contrast, successful commercial companies use an integrated portfolio
management approach to focus early investment decisions on products
collectively at an enterprise level and to ensure there is a sound basis
to justify the commitment of resources. By following a disciplined,
integrated process--where the relative pros and cons of market
opportunities and competing product proposals are assessed based on
available resources and customer needs, and where tough decisions about
which investments to pursue are made--companies are able to reduce
duplication between business units, move away from organizational
stovepipes, and effectively support each new development program they
commit to. Until DOD takes a joint, portfolio management approach to
weapon system acquisition--with functionally aligned entities that have
the requisite responsibility, authority, and control over resources--it
will continue to struggle to effectively prioritize warfighting needs,
make informed trade-offs, and achieve a balanced mix of weapon systems
that are affordable, feasible, and provide the best military value to the
warfighter. Committing to more programs than the budget can support and
approving programs based on insufficient knowledge to effectively manage
risks will further delay providing critical capabilities to the warfighter
and lead to lost opportunities to address other current and emerging
needs.
Recommendations
We recommend that the Secretary of Defense implement an enterprise-wide
portfolio management approach to making weapon system investments that
integrates the assessment and determination of warfighting needs with
available resources and cuts across the services by functional or
capability area. To ensure the success of such an approach, the Secretary
should establish a single point of accountability at the department level
with the authority, responsibility, and tools to ensure that portfolio
management for weapon system investments is effectively implemented across
the department.
In addition, the Secretary should ensure that the following commercial
best practices, identified in this report, are incorporated:
o implement a review process in which needs and resources are
integrated early and in which resources are committed
incrementally based on the achievement of specific levels of
knowledge at established decision points;
o prioritize programs based on the relative costs, benefits, and
risks of each investment to ensure a balanced portfolio;
o require increasingly precise cost, schedule, and performance
information for each alternative that meets specified levels of
confidence and allowable deviations at each decision point leading
up to the start of product development;
o establish portfolio managers who are empowered to prioritize
needs, make early go/no-go decisions about alternative solutions,
and allocate resources within fiscal constraints; and
o hold officials at all levels accountable for achieving and
maintaining a balanced, joint portfolio of weapon system
investments that meet the needs of the warfighter within resource
constraints.
We also recommend that the Secretary take steps to support
department-level decision makers and portfolio managers by developing a
stronger joint analytical capability to assess and prioritize warfighting
needs.
Agency Comments and Our Evaluation
DOD provided us with written comments on a draft of this report. The
comments appear in appendix II.
DOD concurred with the majority of our recommendations and partially
concurred with two. Generally, in responding to these recommendations, DOD
stated that it is undertaking several initiatives and pilot efforts to
improve the department's approach to investment and program decision
making, and that implementation of any new business rules will be
contingent upon the outcome of these initiatives. The department also
stated that it is experimenting with portfolio management, related
authorities and organizational constructs, and integrated decision-making
processes.
We believe that these initiatives and pilot efforts may be steps in the
right direction, but we are concerned that they do not go far enough to
address the systemic cultural and structural problems identified in this
report. DOD has attempted many similar acquisition reform efforts over the
past 3 decades, including significant revisions to both defense
requirements and acquisition policy. However, despite these efforts,
weapon system acquisition programs continued to experience cost overruns,
schedule slips, and performance shortfalls. The department's current
initiatives are likely to face the same fate because they do not
fundamentally change DOD's service-centric framework or sufficiently
integrate its decision-making processes for making weapon system
investments.
DOD did not provide comments regarding our recommendation that the
Secretary establish a single point of accountability at the department
level with the authority, responsibility, and tools to ensure that
portfolio management for weapon system investments is effectively
implemented across the department. We believe that a single point of
accountability is necessary to successfully implement a portfolio
management approach and integrate DOD's fragmented decision-making
processes under one senior official who is accountable for weapon system
investment outcomes. We further believe that our recommendations would
better position DOD to make tough, knowledge-based choices among potential
weapon system investments.
We are sending copies of this report to the Secretary of Defense; the
Secretaries of the Air Force, Army, and Navy; and the Director of the
Office of Management and Budget. We will provide copies to others on
request. This report will also be available at no charge on GAO's Web site
at http://www.gao.gov.
If you have any questions about this report or need additional
information, please call me at (202) 512-4841 ([email protected]). Key
contributors to this report were John Oppenheim, Assistant Director; Lily
Chin; John Krump; Matthew Lea; Travis Masters; Sean Seales; Karen Sloan;
Susan Woodward; and Rebecca Yurman.
Michael J. Sullivan
Director, Acquisition and Sourcing Management
Appendix I: Objectives, Scope, and Methodology
This report examines the Department of Defense's (DOD) requirements
identification and resource allocation processes for major weapons
systems. The primary focus is on identifying successful private-sector
principles and practices that could be adopted by DOD to help improve
stability in weapon system acquisition programs. Specifically, our
objectives were to (1) identify best practices of successful commercial
companies for ensuring that they pursue the right mix of programs to meet
the needs of their customers within resource constraints and (2) compare
DOD's enterprise-level processes for investing in weapon systems to those
practices. Our work was conducted between March 2006 and February 2007, in
accordance with generally accepted government auditing standards.
We analyzed the outputs of DOD's investment decision-making support
processes--the requirements determination process known as JCIDS and the
resource allocation process known as PPBE--using criteria established in
DOD policy and in previous GAO reports. We identified impacts of the
existing processes by analyzing quantitative and qualitative data on DOD
spending trends, conducting interviews with DOD officials, and reviewing
previous reports by GAO and by other knowledgeable audit and research
organizations. In addition, we met with officials representing the Office
of the Secretary of Defense, Joint Staff, and military services. At each
of these locations, we conducted interviews that helped us describe the
current condition of DOD's requirements identification and resource
allocation processes. We also reviewed DOD and military service policies
and funding documents pertaining to the DOD requirements identification
process and resource allocation decisions for major weapons systems.
Specifically, we reviewed the contents of 14 unclassified Initial
Capability Documents that were finalized after June 24, 2003--the
publication date for the JCIDS instruction--to assess the extent to which
they contained cost and technical feasibility information. Those 14 ICDs
were unclassified, weapon system-related ACAT I, II, or III ICDs that were
contained in the Joint Staff requirements database. We relied on previous
GAO reports that highlight both the symptoms and causes of unstable
requirements and funding in DOD weapons acquisition programs. A list of
these reports can be found at the end of this report. In addition, we
reviewed recent key studies and reports addressing acquisition reform
issues by the Center for Strategic International Studies, Institute for
Defense Analysis, the U.S. Naval War College, the Defense Acquisition
Performance Assessment Project, the Joint Defense Capabilities Study Team,
the Joint C4ISR Decision Support Center, the Defense Science Board, and
the 2001 and 2005 Quadrennial Defense Reviews.
We also reviewed pertinent literature from authoritative corporate,
academic, and professional organizations, to identify commercial best
practices and processes that could be used by DOD to improve its weapon
system investment decision-making processes. In addition we conducted case
studies of five leading commercial companies. In selecting them, we sought
to identify companies that were recognized in the literature for best
practices, had large and diversified portfolios of products, and make
significant investments in the development and production of new products.
For each of the companies, we interviewed management officials
knowledgeable about their requirements identification and resource
allocation activities, to gather consistent information about processes,
practices, and metrics the companies use to help achieve successful
product development outcomes. Below are descriptions of the five companies
featured in this report:
Motorola
Motorola is a Fortune 100 global communications leader that provides
seamless mobility products and solutions across broadband, embedded
systems, and wireless networks. According to Motorola's 2005 Corporate
Profile, the company is the market leader in mission critical wireless
communication systems, two-way radios, embedded telematics systems,
digital set-top shipments, cable modem shipments, digital head-ends,
embedded computer systems for communication applications, CDMA
infrastructure sales (excluding the United States), and second in world
wide wireless handsets. Motorola achieved net sales of $31.323 billion and
spent $3.060 billion on research and development in 2004. The corporation
has approximately 68,000 employees, in 320 facilities, spanning 73
countries. We met with the management of Motorola's Government &
Enterprise Mobility Solutions and Global Telecom Solutions sectors in
Schaumburg, Illinois.
International Business Machines (IBM)
IBM is one of the world's largest technological companies, spending about
$3 billion annually on research and development activities. It is the
largest supplier of hardware, software, and information technology
services. With 3,248 U.S. patents, IBM earned more patents than any other
company for the 12th consecutive year in 2004. In the past 4 years, IBM
inventors received more than 13,000 patents--approximately 5,400 more than
any other patent recipient. IBM has over 329,000 employees worldwide. We
met with managers from IBM Integrated Product Development (IPD) in Somers,
New York.
Procter & Gamble (P&G)
Procter & Gamble Corp. (P&G) is a leading producer of consumer goods. It
currently leads in global sales and marketshare among all fabric care,
baby care, feminine care, and hair care products. It currently has over
130,000 employees in 80+ countries. Twenty-two of its brands have annual
gross sales exceeding $1 billion each. In fiscal year 2005/2006, P&G
invested $2.075 billion or 3 percent of net sales in research and
development (R&D). This ranks them as one of the top 20 largest research &
development investors among U.S.-based companies. P&G has more Ph.D.s
working in labs around the world than the combined science and engineering
faculties of Harvard, MIT, and Berkeley. We met with the management of
P&G's New Initiative Delivery team in Cincinnati, Ohio.
Eli Lilly Corporation
Eli Lilly is a global pharmaceutical company and one of the world's
largest corporations. It was founded over 130 years ago and currently
employs approximately 42,000 people worldwide, including 13,991 employed
at its headquarters in Indianapolis, Ind. Approximately 8,336 employees
(19 percent of the total work force) are engaged in research and
development (R&D); clinical research is conducted in over 50 countries;
there are R&D facilities in 9 countries; and manufacturing plants in 13
countries. Its products are marketed in 143 countries. Lilly's net sales
in 2005 were $14.6 billion. Eli Lilly strives to grow sales by 6 percent
to 7 percent each year. In 2005, $3 billion was spent on R&D, a $334.4
million increase from the previous year. Currently, R&D represents 20.7
percent of sales. Lilly's total R&D investment in the last 5 years from
continuing operations was $12.5 billion. We met with managers from Eli
Lilly's Corporate Headquarters in Indianapolis, Ind.
Caterpillar Corporation
Caterpillar is a technology leader and the world's leading manufacturer of
construction and mining equipment, diesel and natural gas engines, and
industrial gas turbines. In 2005, its total sales and revenues were $36.3
billion, and its total R&D expenditures exceeded $1 billion, compared with
$20.5 billion sales and $696 million R&D in 2001. Between 2001 and 2005,
the average return on equity of its stockholders' shares more than
doubled. Caterpillar has over 85,000 employees, and over 105,000 people
are employed by Caterpillar's dealers worldwide. We met with managers
responsible for Caterpillar's New Product Introduction (NPI) process in
Peoria, Illinois.
Appendix II: Comments from the Department of Defense
Related GAO Products
Best Practices: Stronger Practices Needed to Improve DOD Technology
Transition Processes. [39]GAO-06-883 . Washington, D.C.: September 14,
2006.
Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and
Schedule Problems under DOD's Revised Policy. [40]GAO-06-368 . Washington,
D.C.: April 13, 2006.
DOD Acquisition Outcomes: A Case for Change. [41]GAO-06-257T . Washington,
D.C.: November 15, 2005.
Defense Acquisitions: Stronger Management Practices Are Needed to Improve
DOD's Software-Intensive Weapon Acquisitions. [42]GAO-04-393 . Washington,
D.C.: March 1, 2004.
Best Practices: Setting Requirements Differently Could Reduce Weapon
Systems' Total Ownership Costs. [43]GAO-03-57 . Washington, D.C.: February
11, 2003.
Best Practices: Capturing Design and Manufacturing Knowledge Early
Improves Acquisition Outcomes. [44]GAO-02-701 . Washington, D.C.: July 15,
2002.
Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices.
[45]GAO-02-469T . Washington, D.C.: February 27, 2002.
Best Practices: Better Matching of Needs and Resources Will Lead to Better
Weapon System Outcomes. [46]GAO-01-288 . Washington, D.C.: March 8, 2001.
Best Practices: A More Constructive Test Approach Is Key to Better Weapon
System Outcomes. [47]GAO/NSIAD-00-199 . Washington, D.C.: July 31, 2000.
Defense Acquisition: Employing Best Practices Can Shape Better Weapon
System Decisions. [48]GAO/T-NSIAD-00-137 . Washington, D.C.: April 26,
2000.
Best Practices: DOD Training Can Do More to Help Weapon System Programs
Implement Best Practices. [49]GAO/NSIAD-99-206 . Washington, D.C.: August
16, 1999.
Best Practices: Better Management of Technology Development Can Improve
Weapon System Outcomes. [50]GAO/NSIAD-99-162 . Washington, D.C.: July 30,
1999.
Defense Acquisitions: Best Commercial Practices Can Improve Program
Outcomes. [51]GAO/T-NSIAD-99-116 . Washington, D.C.: March 17, 1999.
Defense Acquisition: Improved Program Outcomes Are Possible.
[52]GAO/T-NSIAD-98-123 . Washington, D.C.: March 17, 1998.
Best Practices: DOD Can Help Suppliers Contribute More to Weapon System
Programs. [53]GAO/NSIAD-98-87 . Washington, D.C.: March 17, 1998.
Best Practices: Successful Application to Weapon Acquisition Requires
Changes in DOD's Environment. [54]GAO/NSIAD-98-56 . Washington, D.C.:
February 24, 1998.
Best Practices: Commercial Quality Assurance Practices Offer Improvements
for DOD. [55]GAO/NSIAD-96-162 . Washington, D.C.: August 26, 1996.
(120516)
GAO's Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site. To
have GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
[email protected] Automated answering system: (800) 424-5454 or (202)
512-7470
Congressional Relations
Gloria Jarmon, Managing Director, [email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Washington,
D.C. 20548
Public Affairs
Paul Anderson, Managing Director, [email protected] (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
www.gao.gov/cgi-bin/getrpt?GAO-07-388 .
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Michael J. Sullivan at (202) 512-4841 or
[email protected].
Highlights of [63]GAO-07-388 , a report to the Committee on Armed
Services, U.S. Senate
March 2007
BEST PRACTICES
An Integrated Portfolio Management Approach to Weapon System Investments
Could Improve DOD's Acquisition Outcomes
Over the next several years, the Department of Defense (DOD) plans to
invest $1.4 trillion in major weapons programs. While DOD produces
superior weapons, GAO has found that the department has failed to deliver
weapon systems on time, within budget, and with desired capabilities.
While recent changes to DOD's acquisition policy held the potential to
improve outcomes, programs continue to experience significant cost and
schedule overruns.
GAO was asked to examine how DOD's processes for determining needs and
allocating resources can better support weapon system program stability.
Specifically, GAO compared DOD's processes for investing in weapon systems
to the best practices that successful commercial companies use to achieve
a balanced mix of new products, and identified areas where DOD can do
better. In conducting its work, GAO identified the best practices of:
Caterpillar, Eli Lilly, IBM, Motorola, and Procter and Gamble.
[64]What GAO Recommends
GAO is making several recommendations for DOD to implement an integrated
portfolio management approach to weapon system investments. DOD stated
that it is undertaking several pilot efforts to improve the department's
approach and that implementation of any new business rules will be
contingent upon the outcomes of these efforts.
To achieve a balanced mix of executable development programs and ensure a
good return on their investments, the successful commercial companies GAO
reviewed take an integrated, portfolio management approach to product
development. Through this approach, companies assess product investments
collectively from an enterprise level, rather than as independent and
unrelated initiatives. They weigh the relative costs, benefits, and risks
of proposed products using established criteria and methods, and select
those products that can exploit promising market opportunities within
resource constraints and move the company toward meeting its strategic
goals and objectives. Investment decisions are frequently revisited, and
if a product falls short of expectations, companies make tough go/no-go
decisions. The companies GAO reviewed have found that effective portfolio
management requires a governance structure with committed leadership,
clearly aligned roles and responsibilities, portfolio managers who are
empowered to make investment decisions, and accountability at all levels
of the organization.
In contrast, DOD approves proposed programs with much less consideration
of its overall portfolio and commits to them earlier and with less
knowledge of cost and feasibility. Although the military services fight
together on the battlefield as a joint force, they identify needs and
allocate resources separately, using fragmented decision-making processes
that do not allow for an integrated, portfolio management approach like
that used by successful commercial companies. Consequently, DOD has less
assurance that its investment decisions address the right mix of
warfighting needs, and, as seen in the figure below, it starts more
programs than current and likely future resources can support, a practice
that has created a fiscal bow wave. If this trend goes unchecked, Congress
will be faced with a difficult choice: pull dollars from other
high-priority federal programs to fund DOD's acquisitions or accept gaps
in warfighting capabilities.
Figure: Costs Remaining Versus Annual Appropriations for Major Defense
Acquisitions
References
Visible links
25. http://www.gao.gov/cgi-bin/getrpt?GAO-06-585T
26. http://www.gao.gov/cgi-bin/getrpt?GAO-06-368
27. http://www.gao.gov/cgi-bin/getrpt?GAO-07-406SP
28. http://www.gao.gov/cgi-bin/getrpt?GAO-02-701
29. http://www.gao.gov/cgi-bin/getrpt?GAO-01-288
30. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-99-162
31. http://www.gao.gov/cgi-bin/getrpt?GAO-03-488
32. http://www.gao.gov/cgi-bin/getrpt?GAO-05-540
33. http://www.gao.gov/cgi-bin/getrpt?GAO-04-514
34. http://www.gao.gov/cgi-bin/getrpt?GAO-06-1000T
35. http://www.gao.gov/cgi-bin/getrpt?GAO-05-140T
36. http://www.gao.gov/cgi-bin/getrpt?GAO-06-13
37. http://www.gao.gov/cgi-bin/getrpt?GAO-07-211
38. http://www.gao.gov/cgi-bin/getrpt?GAO-06-368
39. http://www.gao.gov/cgi-bin/getrpt?GAO-06-883
40. http://www.gao.gov/cgi-bin/getrpt?GAO-06-368
41. http://www.gao.gov/cgi-bin/getrpt?GAO-06-257T
42. http://www.gao.gov/cgi-bin/getrpt?GAO-04-393
43. http://www.gao.gov/cgi-bin/getrpt?GAO-03-57
44. http://www.gao.gov/cgi-bin/getrpt?GAO-02-701
45. http://www.gao.gov/cgi-bin/getrpt?GAO-02-469T
46. http://www.gao.gov/cgi-bin/getrpt?GAO-01-288
47. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-00-199
48. http://www.gao.gov/cgi-bin/getrpt?GAO/T-NSIAD-00-137
49. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-99-206
50. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-99-162
51. http://www.gao.gov/cgi-bin/getrpt?GAO/T-NSIAD-99-116
52. http://www.gao.gov/cgi-bin/getrpt?GAO/T-NSIAD-98-123
53. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-98-87
54. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-98-56
55. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-96-162
63. http://www.gao.gov/cgi-bin/getrpt?GAO-06-368
*** End of document. ***