Business Systems Modernization: DOD Needs to Fully Define
Policies and Procedures for Institutionally Managing Investments
(11-MAY-07, GAO-07-538).
In 1995, GAO first designated the Department of Defense's (DOD)
business systems modernization program as "high-risk," and
continues to do so today. In 2004, Congress passed legislation
reflecting prior GAO recommendations for DOD to adopt a corporate
approach to information technology (IT) business system
investment management. To support GAO's legislative mandate to
review DOD's efforts, GAO assessed whether the department's
corporate investment management approach comports with relevant
federal guidance. In doing so, GAO applied its IT Investment
Management framework and associated methodology, focusing on the
framework's stages related to the investment management
provisions of the Clinger-Cohen Act of 1996.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-538
ACCNO: A69545
TITLE: Business Systems Modernization: DOD Needs to Fully Define
Policies and Procedures for Institutionally Managing Investments
DATE: 05/11/2007
SUBJECT: Best practices
Defense cost control
Defense procurement
Information technology
IT acquisitions
IT investment management
Management reengineering
Program evaluation
Program management
Risk assessment
Strategic planning
Business planning
Policies and procedures
GAO High Risk Series
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-538
* [1]Results in Brief
* [2]Background
* [3]IT Investment Management Is Critical to Achieving Successful
* [4]IT Investment Management: A Brief Description
* [5]Overview of GAO's ITIM Maturity Framework
* [6]Overview of DOD's Corporate Approach for Identifying, Fundin
* [7]DOD Business System Investments Are Subject to a Fourth Mana
* [8]Business System Investment Roles and Responsibilities
* [9]Tiered Accountability
* [10]Business Investment Certification Reviews and Approvals
* [11]DOD Has Established the Structures Needed to Effectively Man
* [12]DOD Has Begun to Build a Foundation for Project-Level Invest
* [13]DOD Has Assigned Responsibility, but Has Not Defined the Pol
* [14]Conclusions
* [15]Recommendations for Executive Action
* [16]Agency Comments and Our Evaluation
* [17]GAO Contact
* [18]Staff Acknowledgments
* [19]GAO's Mission
* [20]Obtaining Copies of GAO Reports and Testimony
* [21]Order by Mail or Phone
* [22]To Report Fraud, Waste, and Abuse in Federal Programs
* [23]Congressional Relations
* [24]Public Affairs
Report to Congressional Committees
United States Government Accountability Office
GAO
May 2007
BUSINESS SYSTEMS MODERNIZATION
DOD Needs to Fully Define Policies and Procedures for Institutionally
Managing Investments
GAO-07-538
Contents
Letter 1
Results in Brief 3
Background 5
DOD Has Established the Structures Needed to Effectively Manage Business
System Investments, but Has Not Fully Defined Many of the Related Policies
and Procedures 22
Conclusions 32
Recommendations for Executive Action 32
Agency Comments and Our Evaluation 33
Appendix I Objective, Scope, and Methodology 40
Appendix II Comments from the Department of Defense 41
Appendix III GAO Contact and Staff Acknowledgments 51
Tables
Table 1: DOD Business Investment Management System Entities' Roles,
Responsibilities, and Composition 16
Table 2: DOD's Investment Tiers 18
Table 3: Stage 2 Critical Processes--Building the Investment Foundation 23
Table 4: Summary of Policies and Procedures for Stage 2 Critical
Processes--Building the Investment Foundation 26
Table 5: Stage 3 Critical Processes--Developing a Complete Investment
Portfolio 29
Table 6: Summary of Policies and Procedures for Stage 3 Critical
Processes--Developing a Complete Investment Portfolio 31
Figures
Figure 1: Simplified DOD Organizational Structure 6
Figure 2: The Five ITIM Stages of Maturity with Critical Processes 11
Figure 3: Working Relationships among DOD Business Investment Management
System Governance Entities 17
Figure 4: Simplified Process Flow of Certification Reviews and Approvals
20
Figure 5: Simplified Process Flow of Annual Reviews 21
Abbreviations
ASD(NII)/CIO Assistant Secretary of Defense (Networks and Information
Integration)/Chief Information Officer
BEA business enterprise architecture
BMA business mission area
BTA Business Transformation Agency
DAS Defense Acquisition System
DBSAE Defense Business Systems Acquisition Executive
DBSMC Defense Business Systems Management Committee
DITPR DOD Information Technology Portfolio Repository
DOD Department of Defense
IRB Investment Review Board
IT information technology
ITIM Information Technology Investment Management framework
JCIDS Joint Capabilities Integration and Development System
MAIS Major Automated Information System
MDAP Major Defense Acquisition Programs
OMB Office of Management and Budget
OSD Office of the Secretary of Defense
PCA pre-certification authority
PPBE Planning, Programming, Budgeting, and Execution
USD(AT&L) Under Secretary of Defense (Acquisition, Technology, and
Logistics)
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office
Washington, DC 20548
May 11, 2007
Congressional Committees
For decades, the Department of Defense (DOD) has been challenged in
modernizing its timeworn business systems.^1 In 1995, we designated DOD's
business systems modernization program as high risk, and we continue to
designate it as such today.^2 As our research on public and private sector
organizations shows, one essential ingredient to a successful systems
modernization program is having an effective institutional approach to
managing information technology (IT) investments.
In May 2001, we recommended that the department establish a corporate
approach to investment control and decision making.^3 Between 2001 and
2005, we reported that the department's business systems modernization
program was still not being effectively managed,^4 and we made additional
investment-related recommendations. Congress subsequently included
provisions in the Ronald W. Reagan National Defense Authorization Act for
Fiscal Year 2005^5 that reflected our recommendations, including those for
establishing and implementing effective business system investment
management structures and processes.
^1Business systems are information systems that include financial and
nonfinancial systems and support DOD's business operations, such as
civilian personnel, finance, health, logistics, military personnel,
procurement, and transportation.
^2GAO, High-Risk Series: An Update, [25]GAO-07-310 (Washington, D.C.:
January 2007).
^3GAO, Information Technology: Architecture Needed to Guide Modernization
of DOD's Financial Operations, [26]GAO-01-525 (Washington, D.C.: May 17,
2001).
^4See, for example, GAO, DOD Business Systems Modernization: Long-standing
Weaknesses in Enterprise Architecture Development Need to Be Addressed,
[27]GAO-05-702 (Washington, D.C.: July 22, 2005); DOD Business Systems
Modernization: Billions Being Invested without Adequate Oversight,
[28]GAO-05-381 (Washington, D.C.: Apr. 29, 2005); DOD Business Systems
Modernization: Limited Progress in Development of Business Enterprise
Architecture and Oversight of Information Technology Investments,
[29]GAO-04-731R (Washington, D.C.: May 17, 2004); DOD Business Systems
Modernization: Important Progress Made to Develop Business Enterprise
Architecture, but Much Work Remains, [30]GAO-03-1018 (Washington, D.C.:
Sept. 19, 2003); Business Systems Modernization: Summary of GAO's
Assessment of the Department of Defense's Initial Business Enterprise
Architecture, [31]GAO-03-877R (Washington, D.C.: July 7, 2003);
Information Technology: Observations on Department of Defense's Draft
Enterprise Architecture, [32]GAO-03-571R (Washington, D.C.: Mar. 28,
2003); DOD Business Systems Modernization: Improvements to Enterprise
Architecture Development and Implementation Efforts Needed, [33]GAO-03-458
(Washington, D.C.: Feb. 28, 2003); and [34]GAO-01-525 .
Between 2005 and 2006,^6 we reported that DOD had made important progress
in establishing and implementing these structures and processes, but that
much remained to be accomplished relative to the act's requirements. For
example, we reported that the department's business system investment
approach was not institutionalized at all levels of the department.
To support GAO's legislative mandate to review DOD's annual report on its
business systems modernization program, and as agreed with your offices,
the objective of this review was to determine whether DOD's corporate
investment management approach comports with relevant federal guidance. To
accomplish our objective, we analyzed documents and interviewed agency
officials to determine whether DOD has developed the structures, policies,
and procedures associated with executing those key practices in our IT
Investment Management (ITIM) framework that assist organizations in
complying with the investment management provisions of the Clinger-Cohen
Act of 1996.^7 This framework provides a hierarchical maturity model for
IT investment management and a method for evaluating and assessing the
maturity of an agency's investment management. We performed our work at
DOD headquarters in Arlington, Virginia, from August 2006 through April
2007 in accordance with generally accepted government auditing standards.
Details on our objective, scope, and methodology are contained in appendix
I.
^5Ronald W. Reagan National Defense Authorization Act for Fiscal Year
2005, Pub. L. No. 108-375, S 332, 118 Stat. 1811, 1851-1856 (Oct. 28,
2004) (codified in part at 10 U.S.C. S 2222).
^6GAO, Defense Business Transformation: A Comprehensive Plan, Integrated
Efforts, and Sustained Leadership Are Needed to Assure Success,
[35]GAO-07-229T (Washington, D.C.: Nov. 16, 2006); Business Systems
Modernization: DOD Continues to Improve Institutional Approach, but
Further Steps Needed, [36]GAO-06-658 (Washington, D.C.: May 15, 2006); and
DOD Business Systems Modernization: Important Progress Made in
Establishing Foundational Architecture Products and Investment Management
Practices, but Much Work Remains, [37]GAO-06-219 (Washington, D.C.: Nov.
23, 2005).
^7GAO, Information Technology Investment Management: A Framework for
Assessing and Improving Process Maturity, [38]GAO-04-394G (Washington,
D.C.: March 2004).
Results in Brief
DOD has established the management structures needed to effectively manage
its business system investments, but it has not fully defined many of the
related policies and procedures that our framework defines. Specifically,
DOD has fully defined four of nine key practices that call for
project-level policies and procedures, and one of the five practices that
call for portfolio-level policies and procedures. For example, regarding
project-level investment, the department has (1) established an
enterprisewide investment board and subordinate boards that are
responsible for business system investment governance, (2) documented
policies and procedures for ensuring that systems support ongoing and
future business needs, (3) developed procedures for identifying and
collecting information about these systems to support investment selection
and control, and (4) assigned responsibility for ensuring that the
information collected during project identification meets the needs of the
investment management process. Regarding portfolio-based investment, DOD
has assigned responsibility to the Under Secretary of Defense for
Acquisition, Technology, and Logistics for managing business system
portfolio selection criteria.
However, DOD has not fully documented business system investment policies
and procedures related to five key project-level management practices. For
example, policies and procedures do not (1) define how the investment
selection, acquisition, and funding processes are coordinated; (2) specify
how the full range of cost, schedule, and benefit data accessible by the
Investment Review Boards (IRB) are to be used in making selection (i.e.,
certification) decisions; (3) specify how reselection decisions at the
corporate level (i.e., annual review decisions) consider investments that
are in operations and maintenance; (4) describe how funding decisions are
integrated with the process of selecting an investment at the corporate
level; and (5) provide sufficient oversight and visibility into
component-level investment management activities, including component
reviews of systems in operations and maintenance. Furthermore, DOD does
not have documented policies and procedures for (1) defining the portfolio
criteria, (2) creating the portfolio, (3) evaluating the portfolio, and
(4) conducting postimplementation reviews for all business systems.
Regarding project-level investment management practices, DOD officials
stated that these are performed at the component level, and that
departmental policies and procedures established for overseeing execution
of these practices by components are sufficient. Regarding portfolio-level
practices, however, these officials stated that they intend to improve
departmental policies and procedures for business system investments by,
for example, establishing a single governance structure, but plans or time
frames for doing so have not been established. According to our ITIM
framework, adequately documenting both the policies and the associated
procedures that govern how an organization manages its IT investment
portfolio(s) is important because doing so provides the basis for having
rigor, discipline, and repeatability in how investments are selected and
controlled across the entire organization. Until DOD fully defines
departmentwide policies and procedures for both individual projects and
portfolios of projects, it risks selecting and controlling these business
system investments in an inconsistent, incomplete, and ad hoc manner,
which in turn reduces the chances that these investments will meet mission
needs in the most cost-effective manner.
To strengthen DOD's business system investment management capability, we
are recommending that the department fully define the policies and
procedures associated with project-level and portfolio-level investment
management as discussed in our guidance for IT investment management.^8
In written comments on a draft of this report, signed by the Deputy Under
Secretary of Defense (Business Transformation) and reprinted in appendix
II, the department stated that it agreed with the report's overall
conclusions, and it described efforts under way and planned that it said
would address many of the gaps identified in the report. In this regard,
the department partially concurred with five of the report's
recommendations, adding that our recommendations and feedback are helpful
in guiding DOD's business transformation and related improvement efforts.
However, the department disagreed with the remaining four recommendations
for two primary reasons. First, it stated that its existing investment
management structure already satisfies the intent of these
recommendations. For example, it stated that its policies already require
the provision of cost, schedule, and funding data as part of investment
certifications and annual reviews, and that a linkage currently exists
among the investment selection, acquisition, and funding processes. We do
not agree with this reasoning. Our recommendations are not intended to
address whether existing policies or guidance provide for the use of cost,
schedule, and funding data, or whether they state that investment
selection, acquisition, and funding decision making are linked. Rather,
our recommendations address the definitions of policy, guidance, and
supporting procedures that fall short of satisfying the best practices
embodied in our ITIM framework. In the case of the above examples, while
we do not question whether investment data are provided to investment
decision-making bodies, the department's policies and procedures do not
include specific decision criteria that explain how these data are to be
used to make consistent, repeatable selection and reselection decisions
across all investments. Furthermore, while we do not question that
existing guidance contains an illustration depicting a link between
investment certification and review and other DOD decision support
processes, including the funding process, neither this guidance nor
supporting procedures define how this linkage is executed (i.e., how
investment funding decisions are in fact integrated with investment
selection decisions).
^8 [39]GAO-04-394G .
Second, DOD stated that our recommendations contradict the department's
"tiered accountability" approach to investment management, in which
responsibility and accountability for business system investment
management is allocated between the Office of the Secretary of Defense
(corporate level) and DOD components (subsidiary levels) on the basis of
investment size and significance. We do not agree with the department's
reasoning. We support DOD's tiered accountability concept because it is
consistent with the hierarchical investment structures described in our
ITIM framework. Under the department's current policies and guidance,
however, most DOD investments are not subject to corporate visibility and
oversight, either because they do not involve development/modernization
(i.e., they are in operations and maintenance) or because they do not
exceed a certain dollar threshold. Our framework recognizes that effective
implementation of this concept should include appropriate corporate
visibility into and oversight of investments, either through review and
approval of those investments that meet certain criteria or through
awareness of a subordinate board's investment management activities.
Moreover, this visibility and oversight should extend to the entire
portfolio of investments, including those that are in operations and
maintenance. To ensure that this occurs, applicable policies and
procedures need to explicitly cover all such investments and need to
define how this is to be accomplished.
Background
DOD is a massive and complex organization. To illustrate, the department
reported that its fiscal year 2006 operations involved approximately $1.4
trillion in assets and $2.0 trillion in liabilities, more than 2.9 million
military and civilian personnel, and $581 billion in net cost of
operations. To date, for fiscal year 2007, the department received
appropriations of about $501 billion. Organizationally, the department
includes the Office of the Secretary of Defense (OSD), the Chairman of the
Joint Chiefs of Staff, the military departments, numerous defense agencies
and field activities, and various unified combatant commands that are
responsible for either specific geographic regions or specific functions.
(See fig. 1 for a simplified depiction of DOD's organizational structure.)
Figure 1: Simplified DOD Organizational Structure
^aThe Chairman of the Joint Chiefs of Staff serves as the spokesman for
the commanders of the combatant commands, especially on the administrative
requirements of their commands.
In support of its military operations, the department performs an
assortment of interrelated and interdependent business functions,
including logistics management, procurement, health care management, and
financial management. As we have previously reported,^9 the systems
environment that supports these business functions is overly complex and
error-prone, and is characterized by (1) little standardization across the
department, (2) multiple systems performing the same tasks, (3) the same
data stored in multiple systems, and (4) the need for data to be entered
manually into multiple systems. Moreover, according to DOD, this systems
environment is comprised of approximately 3,100 separate business systems.
For fiscal year 2007, Congress appropriated approximately $15.7 billion to
DOD, and for fiscal year 2008, DOD has requested about $15.9 billion in
appropriated funds to operate, maintain, and modernize these business
systems and the associated infrastructures.
^9 [40]GAO-06-658 .
As we have previously reported,^10 the department's nonintegrated and
duplicative systems impair DOD's ability to combat fraud, waste, and
abuse. In fact, DOD currently bears responsibility, in whole or in part,
for 15 of our 27 high-risk areas.^11 Eight of these areas are specific to
DOD,^12 and the department shares responsibility for 7 other
governmentwide high-risk areas.^13 DOD's business systems modernization is
one of the high-risk areas, and it is an essential enabler to addressing
many of the department's other high-risk areas. For example, modernized
business systems are integral to the department's efforts to address its
financial, supply chain, and information security management high-risk
areas.
IT Investment Management Is Critical to Achieving Successful Systems
Modernization
A corporate approach to IT investment management is characteristic of
successful public and private organizations. Recognizing this, Congress
enacted the Clinger-Cohen Act of 1996,^14 which requires the Office of
Management and Budget (OMB) to establish processes to analyze, track, and
evaluate the risks and results of major capital investments in IT systems
made by executive agencies.^15 In response to the Clinger-Cohen Act and
other statutes, OMB has developed policy and issued guidance for the
planning, budgeting, acquisition, and management of federal capital
assets.^16 We have also issued guidance in this area,^17 which defines
institutional structures, such as the IRBs; processes for developing
information on investments (such as costs and benefits); and practices to
inform management decisions (such as whether a given investment is aligned
with an enterprise architecture).
^10See, for example, GAO, DOD Travel Cards: Control Weaknesses Resulted in
Millions of Dollars of Improper Payments, [41]GAO-04-576 (Washington,
D.C.: June 9, 2004); Military Pay: Army National Guard Personnel Mobilized
to Active Duty Experienced Significant Pay Problems, [42]GAO-04-89
(Washington, D.C.: Nov. 13, 2003); and Defense Inventory: Opportunities
Exist to Improve Spare Parts Support Aboard Deployed Navy Ships,
[43]GAO-03-887 (Washington, D.C.: Aug. 29, 2003).
^11 [44]GAO-07-310 .
^12These 8 high-risk areas include DOD's (1) overall approach to business
transformation, (2) business systems modernization, (3) financial
management, (4) personnel security clearance program, (5) supply chain
management, (6) support infrastructure management, (7) weapon systems
acquisition, and (8) contract management.
^13The 7 governmentwide high-risk areas are (1) disability programs, (2)
ensuring the effective protection of technologies critical to U.S.
national security interests, (3) interagency contracting, (4) information
systems and critical infrastructure, (5) information-sharing for homeland
security, (6) human capital, and (7) real property.
^14The Clinger-Cohen Act of 1996, 40 U.S.C. SS 11101-11704. This act
expanded the responsibilities of OMB and the agencies that had been set
under the Paperwork Reduction Act with regard to IT management. See 44
U.S.C. 3504(a)(1)(B)(vi) (OMB); and 44 U.S.C. 3506(h)(5) (agencies).
^15We have made recommendations to improve OMB's process for monitoring
high-risk IT investments; see GAO, Information Technology: OMB Can Make
More Effective Use of Its Investment Reviews, [45]GAO-05-276 (Washington,
D.C.: Apr. 15, 2005).
IT Investment Management: A Brief Description
IT investment management is a process for linking IT investment decisions
to an organization's strategic objectives and business plans. Consistent
with this, the federal approach to IT investment management focuses on
selecting, controlling, and evaluating investments in a manner that
minimize risks while maximizing the return of investment.^18
o During the selection phase, the organization (1) identifies and
analyzes each project's risks and returns before committing
significant funds to any project and (2) selects those IT projects
that will best support its mission needs.
o During the control phase, the organization ensures that
projects, as they develop and investment expenditures continue,
meet mission needs at the expected levels of cost and risk. If the
project is not meeting expectations or if problems arise, steps
are quickly taken to address the deficiencies.
o During the evaluation phase, expected results are compared with
actual results after a project has been fully implemented. This
comparison is done to (1) assess the project's impact on mission
performance, (2) identify any changes or modifications to the
project that may be needed, and (3) revise the investment
management process based on lessons learned.
^16This policy is set forth and guidance is provided in OMB Circular A-11
(Nov. 2, 2005) (section 300), and in OMB's Capital Programming Guide,
which directs agencies to develop, implement, and use a capital
programming process to build their capital asset portfolios.
^17See, for example, [46]GAO-04-394G ; GAO, Information Technology: A
Framework for Assessing and Improving Enterprise Architecture Management
(Version 1.1), [47]GAO-03-584G (Washington, D.C.: April 2003); and
Assessing Risks and Returns: A Guide for Evaluating Federal Agencies' IT
Investment Decision-making, GAO/AIMD-10.1.13 (Washington, D.C.: February
1997).
^18GAO-04-394G; GAO/AIMD-10.1.13; GAO, Executive Guide: Improving Mission
Performance Through Strategic Information Management and Technology,
GAO/AIMD-94-115 (Washington, D.C.: May 1994); and Office of Management and
Budget, Evaluating Information Technology Investments, A Practical Guide
(Washington, D.C.: November 1995).
Overview of GAO's ITIM Maturity Framework
Our ITIM framework consists of five progressive stages of maturity
for any given agency relative to selecting, controlling, and
evaluating its investment management capabilities.^19 (See fig. 2
for the five ITIM stages of maturity.) This framework is grounded
in our research of IT investment management practices of leading
private and public sector organizations. The maturity stages are
cumulative; that is, to attain a higher stage, an agency must
institutionalize all of the critical processes at the lower
stages, in addition to the higher stage critical processes.
The framework can be used to assess the maturity of an agency's
investment management processes and as a tool for organizational
improvement. The overriding purpose of the framework is to
encourage investment selection and control and to evaluate
processes that promote business value and mission performance,
reduce risk, and increase accountability and transparency. We have
used the framework in several of our evaluations,^20 and a number
of agencies have adopted it.
With the exception of the first stage, each maturity stage is
composed of "critical processes" that must be implemented and
institutionalized for the organization to achieve that stage. Each
ITIM critical process consists of "key practices"--to include
organizational structures, policies, and procedures--that must be
executed to implement the critical process. It is not unusual for
an organization to perform key practices from more than one
maturity stage at the same time. However, our research shows that
agency efforts to improve investment management capabilities
should focus on implementing all lower-stage practices before
addressing higher-stage practices.
^19GAO-04-394G.
^20GAO, Information Technology: Centers for Medicare & Medicaid Services
Needs to Establish Critical Investment Management Capabilities, GAO-06-12
(Washington, D.C.: Oct. 28, 2005); Information Technology: HHS Has Several
Investment Management Capabilities in Place, but Needs to Address Key
Weaknesses, GAO-06-11 (Washington, D.C.: Oct. 28, 2005); Information
Technology: FAA Has Many Investment Management Capabilities in Place, but
More Oversight of Operational Systems Is Needed, GAO-04-822 (Washington,
D.C.: Aug. 20, 2004); Bureau of Land Management: Plan Needed to Sustain
Progress in Establishing IT Investment Management Capabilities,
GAO-03-1025 (Washington, D.C.: Sept. 12, 2003); Information Technology:
Departmental Leadership Crucial to Success of Investment Reforms at
Interior, GAO-03-1028 (Washington, D.C.: Sept. 12, 2003); United States
Postal Service: Opportunities to Strengthen IT Investment Management
Capabilities, GAO-03-3 (Washington, D.C.: Oct. 15, 2002); and Information
Technology: DLA Needs to Strengthen Its Investment Management Capability,
GAO-02-314 (Washington, D.C.: Mar. 15, 2002).
In the ITIM framework, Stage 2 critical processes lay the
foundation by establishing successful, predictable, and repeatable
investment control processes at the project level. At this stage,
the emphasis is on establishing basic capabilities for selecting
new IT projects; controlling projects so that they finish
predictably within the established cost, schedule, and performance
expectations; and identifying and mitigating exposure to risk.
Stage 3 is where the agency moves from project-centric processes
to portfolio-based processes and evaluates potential investments
according to how well they support the agency's missions,
strategies, and goals. This stage focuses on continually assessing
both proposed and ongoing projects as part of complete investment
portfolios--integrated and competing sets of investment options.
It also focuses on maintaining mature, integrated selection (and
reselection); control; and postimplementation evaluation
processes. This portfolio perspective allows decision makers to
consider the interaction among investments and the contributions
to organizational mission goals and strategies that could be made
by alternative portfolio selections, rather than to focus
exclusively on the balance between the costs and benefits of
individual investments. Organizations implementing Stages 2 and 3
practices have in place capabilities that assist in establishing
selection, control, and evaluation structures, policies,
procedures, and practices that are required by the investment
management provisions of the Clinger-Cohen Act.^21
Stages 4 and 5 require the use of evaluation techniques to
continuously improve both investment processes and portfolios to
better achieve strategic outcomes. At Stage 4, an organization has
the capacity to conduct IT succession activities and, therefore,
can plan and implement the deselection of obsolete, high-risk, or
low-value IT investments. An organization with Stage 5 maturity
conducts proactive monitoring for breakthrough technologies that
will enable it to change and improve its business performance.
^21The Clinger-Cohen Act of 1996, 40 U.S.C. SS 11311-11313.
Figure 2: The Five ITIM Stages of Maturity with Critical Processes
Overview of DOD's Corporate Approach for Identifying, Funding, and
Acquiring All System Investments
DOD's major system investments (i.e., weapon and business systems)
are governed by three management systems--the Joint Capabilities
Integration and Development System (JCIDS); the Planning,
Programming, Budgeting, and Execution (PPBE) system; and the
Defense Acquisition System (DAS).
o JCIDS is a need-driven, capabilities-based approach to identify
warfighting needs and meet future joint forces challenges. It is
intended to identify future capabilities for DOD; address
capability gaps and mission needs recognized by the Joint Chiefs
of Staff or derived from strategic guidance, such as the National
Security Strategy Report^22 or Quadrennial Defense Review;^23 and
identify alternative solutions by considering a range of doctrine,
organization, training, materiel, leadership and education,
personnel, and facilities solutions. According to DOD, the Joint
Chiefs of Staff, through the Joint Requirements Oversight Council,
has primary responsibility for defining and implementing JCIDS.
o PPBE is a calendar-driven approach that is composed of four
phases that occur over a moving 2-year cycle. The four
phases--planning, programming, budgeting, and executing--define
how budgets for each DOD component and the department as a whole
are created, vetted, and executed. As recently reported,^24 the
components start programming and budgeting for addressing a
JCIDS-identified capability gap or mission need several years
before actual product development under DAS begins, and before OSD
formally reviews the components' programming and budgeting
proposals (i.e., Program Objective Memorandums). Once reviewed and
approved, the financial details in the Program Objective
Memorandums become part of the President's budget request to
Congress. During budget execution, components may submit program
change proposals or budget change proposals, or both (e.g.,
program cost increases or schedule delays). According to DOD, the
OSD Under Secretary of Defense (Policy), the Director for Program
Analysis and Evaluation,^25 and the Under Secretary of Defense
(Comptroller) have primary responsibility for defining and
implementing the PPBE system.
^22The National Security Strategy Report required by 50 U.S.C. 404a is a
comprehensive report on the national security strategy of the United
States submitted by the President to Congress.
^23See 10 U.S.C. 118. The Quadrennial Defense Review is a comprehensive
examination of the national defense strategy, force structure, force
modernization plans, infrastructure, budget plan, and other elements of
the defense program and policies of the United States with a view toward
determining and expressing the defense strategy of the United States and
establishing a defense program for the next 20 years.
^24GAO, Best Practices: An Integrated Portfolio Management Approach to
Weapon System Investments Could Improve DOD's Acquisition Outcomes,
GAO-07-388 (Washington, D.C.: Mar. 30, 2007).
^25The Director for Program Analysis and Evaluation is the principal staff
assistant who conducts independent analysis for, and provides independent
advice on, all DOD program and evaluation matters to the Secretary and
Deputy Secretary of Defense.
o DAS is described in the DOD Directive 5000.1 and the DOD
Instruction 5000.2^26 and establishes the procedures for the
Defense Acquisition Management Framework, which consists of three
event-based milestones associated with five key program life-cycle
phases. These five phases are as follows:
1. Concept Refinement: Intended to refine the initial
JCIDS-validated system solution (concept) and create
a strategy for acquiring the investment solution. A
decision is made at the end of this phase (milestone
A decision) regarding whether to move to the next
phase (Technology Development).
2. Technology Development: Intended to determine the
appropriate set of technologies to be integrated into
the investment solution by iteratively assessing the
viability of various technologies while
simultaneously refining user requirements. Once the
technology has been demonstrated in a relevant
environment, a decision is made at the end of this
phase (milestone B decision) regarding whether to
move to the next phase (System Development and
Demonstration).
^26DOD Directive 5000.1, May 12, 2003 and DOD Instruction 5000.2, May 12,
2003.
3. System Development and Demonstration: Intended to
develop a system or a system increment and
demonstrate through developer testing that the
system/system increment can function in its target
environment. A decision is made at the end of this
phase (milestone C decision) regarding whether to
move to the next phase (Production and Deployment).
4. Production and Deployment: Intended to achieve an
operational capability that satisfies the mission
needs, as verified through independent operational
test and evaluation, and ensures that the system is
implemented at all applicable locations.
5. Operations and Support: Intended to operationally
sustain the system in the most cost-effective manner
over its life cycle.
A key principle of DAS is that investments are assigned a
category, where programs of increasing dollar value and management
interest are subject to more stringent oversight. For example,
Major Defense Acquisition Programs (MDAP)^27 and Major Automated
Information Systems (MAIS)^28 are large, expensive programs
subject to the most extensive statutory and regulatory reporting
requirements and, unless delegated, are reviewed by acquisition
boards at the DOD corporate level. Smaller and less risky
acquisitions are generally reviewed at the component executive or
lower levels. Another key principle is that DAS requires
acquisition management under the direction of a milestone decision
authority.^29 The milestone decision authority--with support from
the program manager and advisory boards, such as the Defense
Acquisition Board^30 and the IT Acquisition Board^31--determines
the project's baseline cost, schedule, and performance
commitments. The Under Secretary of Defense for Acquisition,
Technology, and Logistics (USD(AT&L)) has primary responsibility
for defining and implementing DAS.
^27A MDAP is an acquisition program that is estimated by the Under
Secretary of Defense for Acquisition, Technology, and Logistics to require
an eventual total expenditure for research, development, and test and
evaluation of more than $365 million (fiscal year 2000 constant dollars)
or, for procurement, of more than $2.190 billion (fiscal year 2000
constant dollars).
^28A MAIS is a program or initiative that is so designated by the
Assistant Secretary of Defense (Networks and Information
Integration)/Chief Information Officer or that is estimated to require
program costs in any single year in excess of $32 million (fiscal year
2000 constant dollars), total program costs in excess of $126 million
(fiscal year 2000 constant dollars), or total life-cycle costs in excess
of $378 million (fiscal year 2000 constant dollars).
DOD Business System Investments Are Subject to a Fourth Management
System
DOD's business system investments are also governed by a fourth
management system that addresses how these investments are
reviewed, certified, and approved for compliance with the business
enterprise priorities and activities outlined by the business
enterprise architecture (BEA). For the purposes of this report, we
refer to this fourth management system as the Business Investment
Management System. This fourth management system is described in
the following text in terms of governance entities, tiered
accountability, and business system investment certification
reviews and approvals. According to DOD, these four management
systems are the means by which DOD selects, controls, and
evaluates its business system investments.
Business System Investment Roles and Responsibilities
In 2005, the department reassigned responsibility for providing
executive leadership for the direction, oversight, and execution
of its business systems modernization efforts to several entities.
These entities and their responsibilities include the following:
^29According to DOD, the milestone decision authority is the designated
individual who has overall responsibility for an investment. This person
has the authority to approve an investment's progression in the
acquisition process and is responsible for reporting cost, schedule, and
performance results. For example, the milestone decision authority for a
MDAP program, when not delegated to the component level, is the Under
Secretary of Defense for Acquisition, Technology, and Logistics, and the
milestone decision authority for a MAIS system is the Assistant Secretary
of Defense (Networks and Information Integration)/Chief Information
Officer or a designee.
^30The Defense Acquisition Board, chaired by the Under Secretary of
Defense for Acquisition, Technology, and Logistics, conducts reviews for
MDAPs at major program milestones and documents the decision(s) resulting
from the review in an Acquisition Decision Memorandum.
^31The IT Acquisition Board, chaired by the Assistant Secretary of Defense
(Networks and Information Integration)/Chief Information Officer, conducts
reviews for MAIS at major program milestones and documents the decision(s)
resulting from the review in an Acquisition Decision Memorandum.
o The Defense Business Systems Management Committee (DBSMC) serves
as the highest-ranking governance body for business systems
modernization activities.
o The Principal Staff Assistants serve as the certification
authorities for business system modernizations in their respective
core business missions.
o The IRBs are chartered by the Principal Staff Assistants and are
the review and decision-making bodies for business system
investments in their respective areas of responsibility.^32
o The component pre-certification authority (PCA) is accountable
for the component's business system investments and acts as the
component's principal point of contact for communication with the
IRBs.
o The Business Transformation Agency (BTA) is responsible for
leading and coordinating business transformation efforts across
the department. The BTA is organized into seven directorates, one
of which is the Defense Business Systems Acquisition Executive
(DBSAE)--the component acquisition executive for DOD
enterprise-level (DOD-wide) business systems and initiatives. This
directorate is responsible for developing, coordinating, and
integrating enterprise-level projects, programs, systems, and
initiatives--including managing resources such as fiscal,
personnel, and contracts for assigned systems and programs.
Table 1 lists these entities and provides greater detail on their
roles, responsibilities, and composition. Figure 3 provides a
simplified illustration of the relationships among these entities.
^32The four IRBs are for (1) Financial Management, established by the
Deputy Under Secretary of Defense for Financial Management; (2) Weapon
Systems Lifecycle Management and Materiel Supply and Services Management;
(3) Real Property and Installations Lifecycle Management, both established
by the USD(AT&L); and (4) Human Resources Management, established by the
Under Secretary of Defense for Personnel and Readiness.
Table 1: DOD Business Investment Management System Entities' Roles,
Responsibilities, and Composition
Source: GAO based on DOD documentation.
^aAccording to DOD, the BMA is responsible for ensuring that capabilities,
resources, and materiel are reliably delivered to the warfighter.
Specifically, the BMA addresses areas such as real property and human
resources management.
^bDOD has five core business missions: Human Resources Management, Weapon
System Lifecycle Management, Materiel Supply and Services Management, Real
Property and Installations Lifecycle Management, and Financial Management.
Figure 3: Working Relationships among DOD Business Investment Management
System Governance Entities
Tiered Accountability
According to DOD, in 2005 it adopted a tiered accountability approach to
business transformation. Under this approach, responsibility and
accountability for business investment management is allocated between the
DOD corporate (i.e., OSD) and the components on the basis of the amount of
development/modernization funding involved and the investment's "tier."
DOD corporate is responsible for ensuring that all business systems with a
development/modernization investment in excess of $1 million are reviewed
by the IRBs for compliance with the BEA, certified by the Principal Staff
Assistants, and approved by the DBSMC. Components are responsible for
certifying development/modernization investments with total costs of $1
million or less. All DOD development and modernization efforts are also
assigned a tier on the basis of the acquisition category or the size of
the financial investment, or both. According to DOD, a system is given a
tier designation when it passes through the certification process. Table 2
describes the four investment tiers and identifies the associated
reviewing and approving entities.
Table 2: DOD's Investment Tiers
Source: DOD.
Business Investment Certification Reviews and Approvals
DOD's business investment management system includes two types of reviews
for business systems: certification and annual reviews. Certification
reviews apply to new modernization projects with total cost over $1
million. This review focuses on program alignment with the BEA and must be
completed before components obligate funds for programs. The annual review
applies to all business programs. The focus for the annual review is to
determine whether the system development effort is meeting its milestones
and addressing its IRB certification conditions.
Certification reviews and approvals: Tiers 1 through 3 business system
investments are certified at two levels--component-level precertification
and corporate-level certification and approval. At the component level,
program managers prepare, enter, maintain, and update information about
their investments in the DOD IT Portfolio Repository (DITPR),^33 such as
regulatory compliance reporting, an architectural profile, and
requirements for investment certification and annual reviews. The
component PCA validates that the system information is complete and
accessible on the IRB Portal, reviews system compliance with the BEA and
enterprise transition plan, and verifies the economic viability analysis.
The PCA asserts the status and validity of the investment information by
submitting a component precertification letter to the appropriate IRB for
its review.
^33DITPR is DOD's authoritative repository for certain information about
DOD's business systems, such as system names and the responsible DOD
components, that are required for the certification, approval, and annual
reviews of these business system investments.
At the corporate level, the IRB reviews the system information and
precertification letter submitted by the PCA to determine whether to
recommend investment certification. On completion of its review, a
certification memorandum is prepared and signed by the designated
certification authority^34 that documents the IRB's system certification
decisions and any related conditions. The memorandum is then forwarded to
the DBSMC, which either approves or disapproves the IRB's decisions and
issues a memorandum containing its decisions. If the DBSMC disapproves a
system investment, it is up to the component PCA to decide whether to
resubmit the investment after it has resolved the relevant issues. Figure
4 provides a simplified overview of the process flow of certification
reviews and approvals.
^34The certification authority is the designated Principal Staff Assistant
with responsibility for review, approval, and oversight of the planning,
design, acquisition, deployment, operation, maintenance, and modernization
of defense business systems.
Figure 4: Simplified Process Flow of Certification Reviews and Approvals
Annual reviews: Tiers 1 through 4 business system investments are annually
reviewed at two levels--the component level and the corporate level. At
the component level, program managers review and update information on all
tiers of investments, both in modernization and operations and
maintenance, on an annual basis in DITPR. The updates for Tiers 1 through
3 with system development/modernization include cost, milestone, and risk
variances and actions or issues related to certification conditions. The
PCA then verifies and submits the information for Tiers 1 through 3
systems in development/modernization for IRB review in an annual review
assertion letter. The letter addresses system compliance with the BEA and
the enterprise transition plan, and includes investment cost, schedule,
and performance information.^35
At the corporate level, the IRBs annually review certified Tiers 1 through
3 investments in development/modernization. These reviews focus on program
compliance with the BEA, program performance against cost and milestone
baselines, and progress in meeting certification conditions. The IRBs can
revoke an investment's certification when the system has significantly
failed to achieve performance commitments (i.e., capabilities and costs).
When this occurs, the component must address the IRB's concerns and
resubmit the investment for certification. Figure 5 shows a simplified
overview of the process flow of annual reviews.
Figure 5: Simplified Process Flow of Annual Reviews
^35In addition, each component PCA submits a list of system names to the
IRBs on a semiannual basis, to include Tier 4 systems and systems in
operations and maintenance that have been reviewed at the component level.
DOD Has Established the Structures Needed to Effectively Manage Business System
Investments, but Has Not Fully Defined Many of the Related Policies and
Procedures
According to our ITIM framework, organizations should establish the
management structures needed to manage their investments and build an
investment foundation by having defined policies and procedures for
selecting and controlling individual projects (Stage 2 capabilities), and
organizations also should manage projects as a portfolio of investments
according to defined policies and procedures, treating them as an
integrated package of competing investment options and pursuing those that
best meet the strategic goals, objectives, and mission of the agency
(Stage 3 capabilities). These Stages 2 and 3 capabilities assist agencies
in complying with the investment management provisions of the
Clinger-Cohen Act.
The department has defined four of nine practices that call for
project-level policies and procedures (see table 4) and one of the five
practices that call for portfolio-level policies and procedures (see table
6). Specifically, it has established the management structures contained
in our ITIM framework, but it has not fully defined many of the related
policies and procedures.
With respect to project-level investment management practices, DOD
officials stated that these are performed at the component level, and that
departmental policies and procedures established for overseeing
components' execution of these practices are sufficient. With respect to
portfolio-level practices, however, these officials stated that they
intend to improve departmental policies and procedures for business system
investments by, for example, establishing a single governance structure,
but plans or time frames for doing so have not been established. According
to our ITIM framework, adequately documenting both the policies and the
associated procedures that govern how an organization manages its IT
investment portfolio(s) is important because doing so provides the basis
for having rigor, discipline, and repeatability in how investments are
selected and controlled across the entire organization. Until DOD fully
defines departmentwide policies and procedures for both individual
projects and the portfolios of projects, it risks selecting and
controlling these business system investments in an inconsistent,
incomplete, and ad hoc manner, which in turn reduces the chances that
these investments will meet mission needs in the most cost-effective
manner.
DOD Has Begun to Build a Foundation for Project-Level Investment Management, but
Key Policies and Procedures Are Not Fully Defined
At ITIM Stage 2, an organization has attained repeatable and successful IT
project-level investment control and basic selection processes. Through
these processes, the organization can identify project expectation gaps
early and take the appropriate steps to address them. ITIM Stage 2
critical processes include (1) defining investment board operations, (2)
identifying the business needs for each investment, (3) developing a basic
process for selecting new proposals and reselecting ongoing investments,
(4) developing project-level investment control processes, and (5)
collecting information about existing investments to inform investment
management decisions. Table 3 describes the purpose of each of these Stage
2 critical processes.
Table 3: Stage 2 Critical Processes--Building the Investment Foundation
Source: GAO.
Within these five critical processes are nine key practices that call for
policies and procedures associated with effective project-level
management. DOD has fully defined the policies and procedures needed to
ensure that four of these nine practices are performed in a consistent and
repeatable manner. Specifically, DOD has established the management
structures by instituting an enterprisewide investment board--the
DBSMC--composed of senior executives, including the Deputy Secretary of
Defense, with final approval authority over associated subsidiary
investment boards. These lower-level investment boards include
representatives from combatant commands, components, and the Joint Chiefs
of Staff. In addition, DOD's business transformation and IRB guidance
define a process for ensuring that programs support the department's
ongoing and future business needs. DOD also has policies and procedures
for submitting, updating, and maintaining investment information in DITPR
and the IRB Portal. Furthermore, the department has assigned the
component's PCA the responsibility to ensure that specific investment
information contained in the portfolio repository and the IRB Portal is
accurate and complete.
However, the policies and procedures associated with the remaining five
project-level management practices are missing critical elements needed to
effectively carry out essential investment management activities. For
example:
o Policies and procedures for instituting the investment board do
not address how investments that are past the
development/modernization stage (i.e., in operations and
maintenance) are to be governed. Given that DOD invests billions
of dollars annually in operating and maintaining business systems,
this is significant. While DOD officials stated that
component-level policies and procedures address systems outside of
development/modernization, our ITIM framework emphasizes that the
corporate investment boards should continue to review important
information about an investment, such as cost and performance
baselines, throughout the investment's life cycle. In addition,
the IRB Concept of Operations and other IRB documentation do not
explicitly outline how the business investment management system
is coordinated with JCIDS, PPBE, and DAS. Without clearly defined
visibility into all investments with an understanding of decisions
reached through other management systems, inconsistent decisions
may result.
o Procedures do not specify how the full range of cost, schedule,
and benefit data is used by the IRBs in making selection (i.e.,
certification) decisions. According to BTA officials, each IRB
decides how to ensure compliance and determines additional factors
to consider when making certification decisions. However, DOD did
not provide us with any supplemental policies or procedures for
any of the four IRBs. Without documenting how IRBs consider
factors such as cost, schedule, and benefits when making selection
decisions, the department cannot ensure that the IRBs and the
DBSMC consistently and objectively select proposals that best meet
the department's needs and priorities. Furthermore, while the
procedures specify decision criteria that address statutory
requirements for alignment to the BEA, the criteria allow programs
to postpone demonstrating full compliance with several BEA
artifacts until the final phases of the acquisition process. As a
result, programs risk beginning production and deployment before
ensuring that a business system is fully aligned to the BEA.
o Policies and procedures do not specify how reselection decisions
at the corporate level (i.e., annual review decisions) consider
investments that are in operations and maintenance. Without an
understanding of how the IRBs are to consider these investments
when making reselection decisions, their ability to make informed
and consistent reselection and termination decisions is limited.
o Policies and procedures do not specify how funding decisions are
integrated with the process of selecting an investment at the
corporate level. Without considering component and corporate
budget constraints and opportunities, the IRBs risk making
investment decisions that do not effectively consider the relative
merits of various projects and systems when funding limitations
exist.
o Policies and procedures do not exist that provide for sufficient
oversight and visibility into component-level investment
management activities, including component reviews of systems in
operations and maintenance and Tier 4 investments. According to
DOD officials, investment oversight is implemented through tiered
accountability, which, among other things, allocates
responsibility and accountability for business system investments
with total costs of $1 million or less and those in operations and
maintenance to the components. However, the department did not
provide policies and procedures defining how the DBSMC and the
IRBs ensure visibility into these component processes. This is
particularly important because, according to DOD's March 15, 2007,
annual report to Congress, only 285 of approximately 3,100 total
business systems have completed the IRB certification process and
have been approved by the DBSMC. DOD officials also stated that
the remaining business systems have not been through the
certification process and have not been given a tier designation.
Without policies and procedures defining how the DBSMC and the
IRBs have visibility into and oversight of all business system
investments, DOD risks components continuing to invest in systems
that are duplicative, stovepiped, nonintegrated, and unnecessarily
costly to manage, maintain, and operate.
Table 4 summarizes our findings relative to DOD's execution of the
nine practices that call for the policies and procedures needed to
manage IT investments at the project level.
Table 4: Summary of Policies and Procedures for Stage 2 Critical
Processes--Building the Investment Foundation
Source: GAO.
According to BTA officials, the IRB Concept of Operations and the
Investment Certification and Annual Review Process User Guidance are not
intended to describe the detailed approach that each IRB will use when
making certification decisions, adding that the components are responsible
for selection, annual review, budgeting, and acquisition. While the ITIM
framework does allow for multiple entities to carry out investment
selection, control, and evaluation, building a sound investment foundation
requires that the enterprisewide investment review board has documented
criteria and decision-making procedures, clear integration among
investment decision-support systems, and policies to ensure board access
to system information throughout the life cycle for all investments. Until
DOD's documented IT investment management policies and procedures include
fully defined policies and procedures for Stage 2 activities, specify the
linkages between the various related processes, and describe how
investments are to be governed in the operations and maintenance phase,
DOD risks that investment management activities will not be carried out
consistently and in a disciplined manner. Moreover, DOD also risks
selecting investments that will not cost-effectively meet its mission
needs.
DOD Has Assigned Responsibility, but Has Not Defined the Policies and Procedures
Associated with Effective Portfolio-Level Management
At Stage 3, an organization has defined critical processes for managing
its investments as a portfolio or set of portfolios.^36 Portfolio
management is a conscious, continuous, and proactive approach to
allocating limited resources among competing initiatives in light of the
investments' relative benefits. Taking an agencywide perspective enables
an organization to consider its investments comprehensively, so that
collectively the investments optimally address the organization's
missions, strategic goals, and objectives. Managing IT investments as
portfolios also allows an organization to determine its priorities and
make decisions about which projects to fund on the basis of analyses of
the relative organizational value and risks of all projects, including
projects that are proposed, under development, and in operation. Although
investments may initially be organized into subordinate portfolios--on the
basis of, for example, business lines or life-cycle stages--and managed by
subordinate investment boards, they should ultimately be aggregated into
enterprise-level portfolios.
According to ITIM, Stage 3 involves (1) defining the portfolio criteria;
(2) creating the portfolio; (3) evaluating (i.e., overseeing) the
portfolio; and (4) conducting postimplementation reviews. Table 5
summarizes the purpose of each of these activities.
Table 5: Stage 3 Critical Processes--Developing a Complete Investment
Portfolio
Source: GAO.
^36Investment portfolios are integrated agencywide collections of
investments that are assessed and managed collectively on the basis of
common criteria.
DOD is executing one of the five practices within these four critical
processes that call for policies and procedures associated with effective
portfolio-level management. Specifically, DOD has issued departmentwide
guidance^37 that assigns responsibilities to the USD(AT&L) for managing
and establishing business system investment portfolios, including
leveraging or establishing a governance forum to oversee these business
system investment portfolio activities.
However, DOD has not fully defined the policies and procedures needed to
effectively execute the remaining four portfolio management practices
relative to business system investments. Specifically, DOD does not have
policies and procedures for defining the portfolio criteria or for
creating and evaluating the portfolio. In addition, while DOD has policies
and procedures for conducting postimplementation reviews as part of DAS,
these reviews do not address systems at all tier levels. Furthermore,
there are no procedures detailing how lessons learned from these reviews
are used during investment review as the basis for management and process
improvements.
Table 6 summarizes the rating for each critical process required to manage
investment as a portfolio and summarizes the evidence that supports these
ratings.
^37DOD Directive 8115.01, Information Technology Portfolio Management, and
DOD Instruction 8115.02, Information Technology Portfolio Management
Implementation.
Table 6: Summary of Policies and Procedures for Stage 3 Critical
Processes--Developing a Complete Investment Portfolio
Source: GAO.
According to BTA officials, while portfolio management is primarily a
component responsibility, they are working toward developing more
effective departmentwide portfolio management processes, but plans or time
frames for doing so have not been established. Without defining corporate
policies and procedures for managing business system investment
portfolios, DOD is at risk of not consistently selecting the mix of
investments that best supports the departmentwide mission needs and
ensuring that investment-related lessons learned are shared and applied
departmentwide.
Conclusions
Given the importance of business systems modernization to DOD's mission,
performance, and outcomes, it is vital for the department to adopt and
employ an effective institutional approach to managing business system
investments. While the department has established aspects of such an
approach and, thus, has a foundation on which to build, it is lacking
other important elements, such as specific policies and procedures needed
for project-level and portfolio-level investment management, including
integration with DOD's other key management systems and sufficient
oversight and visibility into operations and maintenance investments and
Tier 4 investments. This means that DOD lacks an institutional capability
to ensure that it is investing in business systems that best support its
strategic needs, and that ongoing projects meet cost, schedule, and
performance expectations. Until DOD develops this capability, the
department will be impaired in its ability to optimize business mission
area performance and accountability.
Recommendations for Executive Action
To strengthen DOD's business system investment management capability and
address the weaknesses discussed in this report, we recommend that the
Secretary of Defense direct the Deputy Secretary of Defense, as the chair
of the DBSMC, to ensure that well-defined and disciplined business system
investment management policies and procedures are developed and issued. At
a minimum, this should include project-level management policies and
procedures that address the following five areas:
o instituting the investment boards, including assigning the
investment boards responsibility, authority, and accountability
for programs throughout the investment life cycle and specifying
how the business investment management system is coordinated with
JCIDS, PPBE, and DAS;
o selecting new investments, including specifying how cost,
schedule, and benefit data are to be used in making certification
decisions; defining the criteria used to select investments as
enterprisewide; and establishing consistent and effective guidance
for BEA compliance;
o reselecting ongoing investments, including specifying how cost,
schedule, and performance data are to be used in the annual review
process and providing for the reselection of investments that are
in operations and maintenance;
o integrating funding with the process of selecting an investment,
including specifying how the DBSMC and the IRBs use funding
information in carrying out decisions on system certification and
approvals; and
o overseeing IT projects and systems, including providing
sufficient oversight and visibility into component-level
investment management activities.
These well-defined and disciplined business system investment
management policies and procedures should also include
portfolio-level management policies and procedures that address
the following four areas:
o creating and modifying IT portfolio selection criteria for
business system investments;
o analyzing, selecting, and maintaining business system investment
portfolios;
o reviewing, evaluating, and improving the performance of its
portfolio(s) by using project indicators, such as cost, schedule,
and risk; and
o conducting postimplementation reviews for all investment tiers
and directing the investment boards, which are accountable for
corporate business system investments, to consider the information
gathered and to develop lessons learned from these reviews.
Agency Comments and Our Evaluation
In written comments on a draft of this report, signed by the
Deputy Under Secretary of Defense (Business Transformation) and
reprinted in appendix II, the department stated that it agreed
with the report's overall conclusions, and it described efforts
under way and planned that it said would address many of the gaps
identified in the report. In this regard, the department partially
concurred with five of the report's recommendations, adding that
our recommendations and feedback are helpful in guiding DOD's
business transformation and related improvement efforts.
Nevertheless, the department disagreed with the remaining four
recommendations on the grounds that their intent had already been
met through DOD's existing business system investment management
structure and processes, or that they contradicted the tiered
accountability concept embedded in this structure and processes.
The department's comments relative to each of our project-level
and portfolio-level recommendations, along with our responses to
its comments, are provided below.
With respect to our five project-level recommendations, the
department stated that it partially agreed with two and disagreed
with three.
o DOD partially agreed with our recommendation to define and
implement policies and procedures that assign the investment
boards responsibility for programs throughout the investment life
cycle and specify how the business investment management system is
coordinated with JCIDS, PPBE, and DAS. In particular, it stated
that under its tiered accountability approach to business systems
investment management, the components are currently required to
review all programs throughout their investment life cycles. We do
not question this requirement, and we recognize it in our report.
However, consistent with our ITIM framework, the corporate
investment boards should continue to review investments that meet
the defined threshold criteria throughout their life cycles (i.e.,
when they are in operations and maintenance). In contrast, DOD's
corporate boards focus only on those investments that are in the
development/modernization stage. The department also stated that a
linkage is currently depicted in existing guidance among its
investment selection, acquisition, and funding processes. While we
do not question that this guidance contains an illustration
depicting such a link, neither this guidance nor supporting
procedures define how this linkage is executed (e.g., how
investment funding decisions are in fact integrated with
investment selection decisions). DOD's comments appear to
acknowledge this point by stating that the department has begun to
define and implement a Business Capability Lifecycle concept,
which is intended to integrate the investment selection and
acquisition management processes for Tier 1 and enterprise systems
into a single oversight process that leverages the existing IRB
and DBSMC oversight framework.
o DOD partially agreed with our recommendation to define and
implement policies and procedures that specify how cost, schedule,
and benefit data are to be used in making certification and annual
review decisions; define the criteria used to select investments
as enterprisewide; and establish consistent and effective guidance
for BEA compliance. In particular, the department agreed that
additional criteria are required for selecting enterprisewide
investments, noting that initial criteria have been defined and
will be incorporated in the investment management process.
However, the department did not agree that cost, schedule, and BEA
compliance information are not sufficiently used for certification
and annual review decisions, adding that such information is
required in its current policies. We do not agree. Specifically,
while we do not question whether investment data are provided to
the DBSMC and the IRBs, the department's policies and procedures
do not include specific decision criteria that explain how these
data are to be used to make consistent, repeatable selection and
reselection decisions across all investments. In addition, while
BEA compliance policies have been developed and are being used,
the guidance is not fully defined. For example, the guidance
allows programs to defer demonstrating full compliance with
important BEA artifacts until the final phases of the acquisition
process, at which time addressing instances of noncompliance would
be more expensive and difficult. Furthermore, the compliance
criteria are not consistently described in different guidance
documentation. As a result, DOD risks beginning system production
and deployment before ensuring that a system is sufficiently
aligned to the BEA.
o DOD did not agree with our recommendation to define and
implement policies and procedures that provide for the reselection
of investments that are in operations and maintenance. According
to DOD, components are required by policy to annually review all
business systems, including investments for which there is no
planned development or modernization spending. We agree that the
annual review process does require this. However, consistent with
our ITIM framework, the corporate investment boards should
continue to reselect investments that meet the defined threshold
criteria throughout their life cycles (i.e., when they are in
operations and maintenance). In contrast, DOD's corporate boards
focus only on reselecting those investments that are in the
development/modernization stage.
o DOD did not agree with our recommendation to define and
implement policies and procedures that specify how the corporate
boards use funding information in carrying out decisions on system
certification and approvals. In this regard, it stated that such
information is required in its current policies and considered
during board deliberations. We do not agree. Our recommendation
does not address whether existing policies or guidance provide for
the collection of this information; our recommendation addresses
the definition of policy, guidance, and supporting procedures that
fall short of satisfying the best practices embodied in our ITIM
framework. Specifically, while we do not question whether funding
data are provided to investment decision-making bodies, the
department's policies and procedures do not include specific
decision criteria that explain how these data are to be used to
make consistent, repeatable selection and reselection decisions
across all investments.
o DOD did not agree with our recommendation to define and
implement policies and procedures that provide for sufficient
oversight and visibility into component-level investment
management activities. In particular, it stated that this
recommendation contradicts the department's "tiered
accountability" approach to investment management. We do not
agree. Under the department's current policies and guidance, most
DOD investments are not subject to corporate visibility and
oversight, either because they do not involve
development/modernization (i.e., they are in operations and
maintenance) or because they do not exceed a certain dollar
threshold. Our framework recognizes that effective implementation
of a tiered accountability concept should include appropriate
corporate visibility into and oversight of investments, either
through review and approval of those investments that meet certain
criteria or through awareness of a subordinate board's investment
management activities. Moreover, this visibility and oversight
should extend to the entire portfolio of investments, including
those that are in operations and maintenance. To ensure that this
occurs, applicable policies and procedures need to explicitly
cover all such investments and need to define how this is to be
accomplished.
With respect to our four portfolio-level recommendations, the
department stated that it partially agreed with three and
disagreed with one.
o DOD partially agreed with our recommendation to define and
implement policies and procedures for creating and modifying
portfolio selection criteria for business system investments. In
particular, it stated that while components are responsible for
developing and managing their own portfolio management processes,
upcoming initiatives, such as the Business Capability Lifecycle
concept, will lead to revisions in the department's investment
review policies and procedures, such as including portfolio
selection criteria for enterprise systems that span components.
However, while these are important steps, the concept, as defined
by the department, does not apply to the thousands of investments
that are not enterprisewide.
o DOD partially agreed with our recommendation to define and
implement policies and procedures that address analyzing,
selecting, and maintaining business system investment portfolios.
In particular, it stated that the implementation of the Business
Capability Lifecyle concept will provide the corporate boards with
improved visibility into all investments in a given portfolio and
a broader set of criteria for analyzing, selecting, and
maintaining business system investment portfolios.
o DOD partially agreed with our recommendation to define and
implement policies and procedures that address reviewing,
evaluating, and improving the performance of its portfolio(s) by
using cost, schedule, and risk indicators. In particular, it
stated that while such indicators are part of the investment
certification and review processes, efforts are now under way to
better understand the nature and impact of program risks through
application of an Enterprise Risk Assessment Methodology. While we
recognize the role and value of such tools in understanding and
addressing program risks, this tool is program-specific and not
portfolio-focused.
o DOD did not agree with our recommendation to define and
implement policies and procedures that address conducting
postimplementation reviews and having the corporate investment
boards consider the review results and develop lessons learned
from them. In particular, it stated that this process should not
be managed by the Deputy Secretary of Defense, and also stated
that our recommendation is redundant with postimplementation
reviews currently required under OMB Circular A-130.^38 We do not
agree with DOD's statements. Our recommendation does not call for
the Deputy Secretary to manage the postimplementation review
process. Rather, it provides for developing policies and
procedures for performing postimplementation reviews for all tiers
of business systems and having the DBSMC and IRBs, which are the
corporate investment boards, consider the information gathered
from these reviews and develop lessons learned.
We are sending copies of this report to interested congressional
committees; the Director, Office of Management and Budget; the
Secretary of Defense; the Deputy Secretary of Defense; the Under
Secretary of Defense for Acquisition, Technology, and Logistics;
the Under Secretary of Defense (Comptroller); the Assistant
Secretary of Defense (Networks and Information Integration)/Chief
Information Officer; the Under Secretary of Defense (Personnel and
Readiness); and the Director, Defense Finance and Accounting
Service. Copies of this report will be made available to other
interested parties upon request. This report will also be
available at no charge on our Web site at http://www.gao.gov .
^38According to OMB Circular A-130, which establishes policy for the
management of federal information resources, as part of the capital
planning process, an agency must, among other things, conduct
postimplementation reviews of information systems and information resource
management processes to validate estimated benefits and costs; document
effective management practices for broader use; and document lessons
learned from the postimplementation reviews.
If you or your staffs have any questions on matters discussed in
this report, please contact me at (202) 512-3439 or [email protected].
Contact points for our Offices of Congressional Relations and
Public Affairs may be found on the last page of this report. GAO
staff who made major contributions to this report are listed in
appendix III.
Randolph C. Hite
Randolph C. Hite
Director, Information Technology Architecture and Systems Issues
List of Committees
The Honorable Carl Levin
Chairman
The Honorable John McCain
Ranking Member
Committee on Armed Services
United States Senate
The Honorable Daniel Inouye
Chairman
The Honorable Ted Stevens
Ranking Member
Committee on Appropriations
United States Senate
The Honorable Ike Skelton
Chairman
The Honorable Duncan Hunter
Ranking Member
Committee on Armed Services
House of Representatives
The Honorable John P. Murtha
Chairman
The Honorable C.W. Bill Young
Ranking Member
Committee on Appropriations
House of Representatives
Appendix I: Objective, Scope, and Methodology
Our objective was to determine whether the Department of Defense's
(DOD) corporate investment management approach comports with
relevant federal guidance. Our analysis was based on the best
practices contained in GAO's Information Technology Investment
Management (ITIM) framework, and the framework's associated
evaluation methodology, and focused on DOD's establishment of
departmental-level policies and procedures for business system
investments needed to assist organizations in complying with the
investment management provisions of the Clinger-Cohen Act of 1996
(Stages 2 and 3). It did not include case studies to verify the
implementation of established policies and procedures.
To address our objective, we asked DOD to complete a
self-assessment of its corporate investment management process and
provide the supporting documentation. We then reviewed the results
of the department's self-assessment of Stages 2 and 3
organizational commitment practices--meaning those practices
related to structures, policies, and procedures--and compared them
against our ITIM framework. We also validated and updated the
results of the self-assessment through document reviews and
interviews with officials, such as the Director of Investment
Management and the Defense Business Systems Acquisition Executive.
In doing so, we reviewed written policies, procedures, and
guidance and other documentation providing evidence of executed
practices, including the Defense Acquisition System guidance, the
Investment Review Board (IRB) Concept of Operations and Guidance,
the Business Enterprise Architecture Compliance Guidance, IRB
charters and meeting minutes, and the Business Transformation
Guidance.
We compared the evidence collected from our document reviews and
interviews with the key practices in ITIM. We rated the key
practices as "executed" on the basis of whether the agency
demonstrated (by providing evidence of performance) that it had
met all of the criteria of the key practice. A key practice was
rated as "not executed" when we found insufficient evidence of all
elements of a practice being fully performed or when we determined
that there were significant weaknesses in DOD's execution of the
key practice. In addition, we provided DOD with the opportunity to
produce evidence for the key practices rated as "not executed."
We conducted our work at DOD headquarters offices in Arlington,
Virginia, from August 2006 through April 2007 in accordance with
generally accepted government auditing standards.
Appendix II: Comments from the Department of Defense
Appendix III: GAO Contact and Staff Acknowledgments
GAO Contact
Randolph C. Hite, (202) 512-3439 or [email protected]
Staff Acknowledgments
In addition to the contact person named above, key contributors to
this report were Neil Doherty, Nalani Fraser, Nancy Glover,
Michael Holland, Neelaxi Lakhmani (Assistant Director), Jacqueline
Mai, Sabine Paul, Niti Tandon, and Jennifer Stavros-Turner.
GAO's Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in
meeting its constitutional responsibilities and to help improve
the performance and accountability of the federal government for
the American people. GAO examines the use of public funds;
evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at
no cost is through GAO's Web site ( www.gao.gov ). Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of
newly posted products every afternoon, go to www.gao.gov and
select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies
are $2 each. A check or money order should be made out to the
Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are
discounted 25 percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
(202) 512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: [email protected]
Automated answering system: (800) 424-5454 or (202) 512-7470
Congressional Relations
Gloria Jarmon, Managing Director, [email protected] (202)
512-4400 U.S. Government Accountability Office, 441 G Street NW,
Room 7125 Washington, D.C. 20548
Public Affairs
Paul Anderson, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW,
Room 7149 Washington, D.C. 20548
(310636)
www.gao.gov/cgi-bin/getrpt?GAO-07-538 .
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Randolph C. Hite at (202) 512-3439 or
[email protected].
Highlights of [56]GAO-07-538 , a report to congressional committees
May 2007
BUSINESS SYSTEMS MODERNIZATION
DOD Needs to Fully Define Policies and Procedures for Institutionally
Managing Investments
In 1995, GAO first designated the Department of Defense's (DOD) business
systems modernization program as "high-risk," and continues to do so
today. In 2004, Congress passed legislation reflecting prior GAO
recommendations for DOD to adopt a corporate approach to information
technology (IT) business system investment management. To support GAO's
legislative mandate to review DOD's efforts, GAO assessed whether the
department's corporate investment management approach comports with
relevant federal guidance. In doing so, GAO applied its IT Investment
Management framework and associated methodology, focusing on the
framework's stages related to the investment management provisions of the
Clinger-Cohen Act of 1996.
[57]What GAO Recommends
GAO recommends that DOD fully define the project and portfolio management
policies and procedures discussed in GAO's framework. DOD agreed with
GAO's overall conclusions and partially agreed with five of GAO's
recommendations. However, DOD disagreed with the remaining four
recommendations, stating that the department is, among other things,
already meeting the intent of these recommendations. GAO does not agree;
its recommendations focus on fully defining policies and procedures that
satisfy key practices in its framework.
DOD has established the management structures needed to effectively manage
its business system investments, but it has not fully defined many of the
related policies and procedures that GAO's IT Investment Management
framework defines. Specifically, the department has defined four of nine
practices that call for project-level policies and procedures, and one of
the five practices that call for portfolio-level policies and procedures
(see below). For example, DOD has established an enterprisewide IT
investment board responsible for defining and implementing its business
system investment governance process, documented policies and procedures
for ensuring that systems support ongoing and future business needs,
developed procedures for identifying and collecting information about
these systems to support investment selection and control, and assigned
responsibility to an individual or a group for managing the development
and modification of the business system portfolio selection criteria.
However, DOD has not fully documented business system investment policies
and procedures for directing investment board operations, selecting new
investments, reselecting ongoing investments, integrating the investment
funding and the investment selection processes, and developing and
maintaining a complete business system investment portfolio(s).
Regarding project-level investment management practices, DOD officials
said that these are performed at the component level, and that
departmental policies and procedures established for overseeing
components' execution of these practices are sufficient. For
portfolio-level practices, however, these officials stated that they
intend to improve departmental policies and procedures for business system
investments by, for example, establishing a single governance structure,
but plans or time frames for doing so have not been established. Until DOD
fully defines departmentwide policies and procedures for both individual
projects and portfolios of projects, it risks selecting and controlling
these business system investments in an inconsistent, incomplete, and ad
hoc manner, which in turn reduces the chances that these investments will
meet mission needs in the most cost-effective manner.
Policies and Procedures for Project-Level and Portfolio-Level Management
Source: GAO.
References
Visible links
25. http://www.gao.gov/cgi-bin/getrpt?GAO-07-310
26. http://www.gao.gov/cgi-bin/getrpt?GAO-01-525
27. http://www.gao.gov/cgi-bin/getrpt?GAO-05-702
28. http://www.gao.gov/cgi-bin/getrpt?GAO-05-381
29. http://www.gao.gov/cgi-bin/getrpt?GAO-04-731R
30. http://www.gao.gov/cgi-bin/getrpt?GAO-03-1018
31. http://www.gao.gov/cgi-bin/getrpt?GAO-03-877R
32. http://www.gao.gov/cgi-bin/getrpt?GAO-03-571R
33. http://www.gao.gov/cgi-bin/getrpt?GAO-03-458
34. http://www.gao.gov/cgi-bin/getrpt?GAO-01-525
35. http://www.gao.gov/cgi-bin/getrpt?GAO-07-229T
36. http://www.gao.gov/cgi-bin/getrpt?GAO-06-658
37. http://www.gao.gov/cgi-bin/getrpt?GAO-06-219
38. http://www.gao.gov/cgi-bin/getrpt?GAO-04-394G
39. http://www.gao.gov/cgi-bin/getrpt?GAO-04-394G
40. http://www.gao.gov/cgi-bin/getrpt?GAO-06-658
41. http://www.gao.gov/cgi-bin/getrpt?GAO-04-576
42. http://www.gao.gov/cgi-bin/getrpt?GAO-04-89
43. http://www.gao.gov/cgi-bin/getrpt?GAO-03-887
44. http://www.gao.gov/cgi-bin/getrpt?GAO-07-310
45. http://www.gao.gov/cgi-bin/getrpt?GAO-05-276
46. http://www.gao.gov/cgi-bin/getrpt?GAO-04-394G
47. http://www.gao.gov/cgi-bin/getrpt?GAO-03-584G
56. http://www.gao.gov/cgi-bin/getrpt?GAO-07-538
*** End of document. ***