Business Systems Modernization: DOD Needs to Fully Define	 
Policies and Procedures for Institutionally Managing Investments 
(11-MAY-07, GAO-07-538).					 
                                                                 
In 1995, GAO first designated the Department of Defense's (DOD)  
business systems modernization program as "high-risk," and	 
continues to do so today. In 2004, Congress passed legislation	 
reflecting prior GAO recommendations for DOD to adopt a corporate
approach to information technology (IT) business system 	 
investment management. To support GAO's legislative mandate to	 
review DOD's efforts, GAO assessed whether the department's	 
corporate investment management approach comports with relevant  
federal guidance. In doing so, GAO applied its IT Investment	 
Management framework and associated methodology, focusing on the 
framework's stages related to the investment management 	 
provisions of the Clinger-Cohen Act of 1996.			 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-07-538 					        
    ACCNO:   A69545						        
  TITLE:     Business Systems Modernization: DOD Needs to Fully Define
Policies and Procedures for Institutionally Managing Investments 
     DATE:   05/11/2007 
  SUBJECT:   Best practices					 
	     Defense cost control				 
	     Defense procurement				 
	     Information technology				 
	     IT acquisitions					 
	     IT investment management				 
	     Management reengineering				 
	     Program evaluation 				 
	     Program management 				 
	     Risk assessment					 
	     Strategic planning 				 
	     Business planning					 
	     Policies and procedures				 
	     GAO High Risk Series				 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-07-538

   

     * [1]Results in Brief
     * [2]Background

          * [3]IT Investment Management Is Critical to Achieving Successful

               * [4]IT Investment Management: A Brief Description

          * [5]Overview of GAO's ITIM Maturity Framework
          * [6]Overview of DOD's Corporate Approach for Identifying, Fundin
          * [7]DOD Business System Investments Are Subject to a Fourth Mana

               * [8]Business System Investment Roles and Responsibilities
               * [9]Tiered Accountability
               * [10]Business Investment Certification Reviews and Approvals

     * [11]DOD Has Established the Structures Needed to Effectively Man

          * [12]DOD Has Begun to Build a Foundation for Project-Level Invest
          * [13]DOD Has Assigned Responsibility, but Has Not Defined the Pol

     * [14]Conclusions
     * [15]Recommendations for Executive Action
     * [16]Agency Comments and Our Evaluation
     * [17]GAO Contact
     * [18]Staff Acknowledgments
     * [19]GAO's Mission
     * [20]Obtaining Copies of GAO Reports and Testimony

          * [21]Order by Mail or Phone

     * [22]To Report Fraud, Waste, and Abuse in Federal Programs
     * [23]Congressional Relations
     * [24]Public Affairs

Report to Congressional Committees

United States Government Accountability Office

GAO

May 2007

BUSINESS SYSTEMS MODERNIZATION

DOD Needs to Fully Define Policies and Procedures for Institutionally
Managing Investments

GAO-07-538

Contents

Letter 1

Results in Brief 3
Background 5
DOD Has Established the Structures Needed to Effectively Manage Business
System Investments, but Has Not Fully Defined Many of the Related Policies
and Procedures 22
Conclusions 32
Recommendations for Executive Action 32
Agency Comments and Our Evaluation 33
Appendix I Objective, Scope, and Methodology 40
Appendix II Comments from the Department of Defense 41
Appendix III GAO Contact and Staff Acknowledgments 51

Tables

Table 1: DOD Business Investment Management System Entities' Roles,
Responsibilities, and Composition 16
Table 2: DOD's Investment Tiers 18
Table 3: Stage 2 Critical Processes--Building the Investment Foundation 23
Table 4: Summary of Policies and Procedures for Stage 2 Critical
Processes--Building the Investment Foundation 26
Table 5: Stage 3 Critical Processes--Developing a Complete Investment
Portfolio 29
Table 6: Summary of Policies and Procedures for Stage 3 Critical
Processes--Developing a Complete Investment Portfolio 31

Figures

Figure 1: Simplified DOD Organizational Structure 6
Figure 2: The Five ITIM Stages of Maturity with Critical Processes 11
Figure 3: Working Relationships among DOD Business Investment Management
System Governance Entities 17
Figure 4: Simplified Process Flow of Certification Reviews and Approvals
20
Figure 5: Simplified Process Flow of Annual Reviews 21

Abbreviations

ASD(NII)/CIO Assistant Secretary of Defense (Networks  and Information
Integration)/Chief Information Officer
BEA business enterprise architecture
BMA business mission area
BTA Business Transformation Agency
DAS Defense Acquisition System
DBSAE Defense Business Systems Acquisition Executive
DBSMC Defense Business Systems Management Committee
DITPR DOD Information Technology Portfolio Repository
DOD Department of Defense
IRB Investment Review Board
IT information technology
ITIM Information Technology Investment Management framework
JCIDS Joint Capabilities Integration and Development System
MAIS Major Automated Information System
MDAP Major Defense Acquisition Programs
OMB Office of Management and Budget
OSD Office of the Secretary of Defense
PCA pre-certification authority
PPBE Planning, Programming, Budgeting, and Execution
USD(AT&L) Under Secretary of Defense (Acquisition, Technology, and
Logistics)

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office
Washington, DC 20548

May 11, 2007

Congressional Committees

For decades, the Department of Defense (DOD) has been challenged in
modernizing its timeworn business systems.^1 In 1995, we designated DOD's
business systems modernization program as high risk, and we continue to
designate it as such today.^2 As our research on public and private sector
organizations shows, one essential ingredient to a successful systems
modernization program is having an effective institutional approach to
managing information technology (IT) investments.

In May 2001, we recommended that the department establish a corporate
approach to investment control and decision making.^3 Between 2001 and
2005, we reported that the department's business systems modernization
program was still not being effectively managed,^4 and we made additional
investment-related recommendations. Congress subsequently included
provisions in the Ronald W. Reagan National Defense Authorization Act for
Fiscal Year 2005^5 that reflected our recommendations, including those for
establishing and implementing effective business system investment
management structures and processes.

^1Business systems are information systems that include financial and
nonfinancial systems and support DOD's business operations, such as
civilian personnel, finance, health, logistics, military personnel,
procurement, and transportation.

^2GAO, High-Risk Series: An Update, [25]GAO-07-310 (Washington, D.C.:
January 2007).

^3GAO, Information Technology: Architecture Needed to Guide Modernization
of DOD's Financial Operations, [26]GAO-01-525 (Washington, D.C.: May 17,
2001).

^4See, for example, GAO, DOD Business Systems Modernization: Long-standing
Weaknesses in Enterprise Architecture Development Need to Be Addressed,
[27]GAO-05-702 (Washington, D.C.: July 22, 2005); DOD Business Systems
Modernization: Billions Being Invested without Adequate Oversight,
[28]GAO-05-381 (Washington, D.C.: Apr. 29, 2005); DOD Business Systems
Modernization: Limited Progress in Development of Business Enterprise
Architecture and Oversight of Information Technology Investments,
[29]GAO-04-731R (Washington, D.C.: May 17, 2004); DOD Business Systems
Modernization: Important Progress Made to Develop Business Enterprise
Architecture, but Much Work Remains, [30]GAO-03-1018 (Washington, D.C.:
Sept. 19, 2003); Business Systems Modernization: Summary of GAO's
Assessment of the Department of Defense's Initial Business Enterprise
Architecture, [31]GAO-03-877R (Washington, D.C.: July 7, 2003);
Information Technology: Observations on Department of Defense's Draft
Enterprise Architecture, [32]GAO-03-571R (Washington, D.C.: Mar. 28,
2003); DOD Business Systems Modernization: Improvements to Enterprise
Architecture Development and Implementation Efforts Needed, [33]GAO-03-458
(Washington, D.C.: Feb. 28, 2003); and [34]GAO-01-525 .

Between 2005 and 2006,^6 we reported that DOD had made important progress
in establishing and implementing these structures and processes, but that
much remained to be accomplished relative to the act's requirements. For
example, we reported that the department's business system investment
approach was not institutionalized at all levels of the department.

To support GAO's legislative mandate to review DOD's annual report on its
business systems modernization program, and as agreed with your offices,
the objective of this review was to determine whether DOD's corporate
investment management approach comports with relevant federal guidance. To
accomplish our objective, we analyzed documents and interviewed agency
officials to determine whether DOD has developed the structures, policies,
and procedures associated with executing those key practices in our IT
Investment Management (ITIM) framework that assist organizations in
complying with the investment management provisions of the Clinger-Cohen
Act of 1996.^7 This framework provides a hierarchical maturity model for
IT investment management and a method for evaluating and assessing the
maturity of an agency's investment management. We performed our work at
DOD headquarters in Arlington, Virginia, from August 2006 through April
2007 in accordance with generally accepted government auditing standards.
Details on our objective, scope, and methodology are contained in appendix
I.

^5Ronald W. Reagan National Defense Authorization Act for Fiscal Year
2005, Pub. L. No. 108-375, S 332, 118 Stat. 1811, 1851-1856 (Oct. 28,
2004) (codified in part at 10 U.S.C. S 2222).

^6GAO, Defense Business Transformation: A Comprehensive Plan, Integrated
Efforts, and Sustained Leadership Are Needed to Assure Success,
[35]GAO-07-229T (Washington, D.C.: Nov. 16, 2006); Business Systems
Modernization: DOD Continues to Improve Institutional Approach, but
Further Steps Needed, [36]GAO-06-658 (Washington, D.C.: May 15, 2006); and
DOD Business Systems Modernization: Important Progress Made in
Establishing Foundational Architecture Products and Investment Management
Practices, but Much Work Remains, [37]GAO-06-219 (Washington, D.C.: Nov.
23, 2005).

^7GAO, Information Technology Investment Management: A Framework for
Assessing and Improving Process Maturity, [38]GAO-04-394G (Washington,
D.C.: March 2004).

Results in Brief

DOD has established the management structures needed to effectively manage
its business system investments, but it has not fully defined many of the
related policies and procedures that our framework defines. Specifically,
DOD has fully defined four of nine key practices that call for
project-level policies and procedures, and one of the five practices that
call for portfolio-level policies and procedures. For example, regarding
project-level investment, the department has (1) established an
enterprisewide investment board and subordinate boards that are
responsible for business system investment governance, (2) documented
policies and procedures for ensuring that systems support ongoing and
future business needs, (3) developed procedures for identifying and
collecting information about these systems to support investment selection
and control, and (4) assigned responsibility for ensuring that the
information collected during project identification meets the needs of the
investment management process. Regarding portfolio-based investment, DOD
has assigned responsibility to the Under Secretary of Defense for
Acquisition, Technology, and Logistics for managing business system
portfolio selection criteria.

However, DOD has not fully documented business system investment policies
and procedures related to five key project-level management practices. For
example, policies and procedures do not (1) define how the investment
selection, acquisition, and funding processes are coordinated; (2) specify
how the full range of cost, schedule, and benefit data accessible by the
Investment Review Boards (IRB) are to be used in making selection (i.e.,
certification) decisions; (3) specify how reselection decisions at the
corporate level (i.e., annual review decisions) consider investments that
are in operations and maintenance; (4) describe how funding decisions are
integrated with the process of selecting an investment at the corporate
level; and (5) provide sufficient oversight and visibility into
component-level investment management activities, including component
reviews of systems in operations and maintenance. Furthermore, DOD does
not have documented policies and procedures for (1) defining the portfolio
criteria, (2) creating the portfolio, (3) evaluating the portfolio, and
(4) conducting postimplementation reviews for all business systems.

Regarding project-level investment management practices, DOD officials
stated that these are performed at the component level, and that
departmental policies and procedures established for overseeing execution
of these practices by components are sufficient. Regarding portfolio-level
practices, however, these officials stated that they intend to improve
departmental policies and procedures for business system investments by,
for example, establishing a single governance structure, but plans or time
frames for doing so have not been established. According to our ITIM
framework, adequately documenting both the policies and the associated
procedures that govern how an organization manages its IT investment
portfolio(s) is important because doing so provides the basis for having
rigor, discipline, and repeatability in how investments are selected and
controlled across the entire organization. Until DOD fully defines
departmentwide policies and procedures for both individual projects and
portfolios of projects, it risks selecting and controlling these business
system investments in an inconsistent, incomplete, and ad hoc manner,
which in turn reduces the chances that these investments will meet mission
needs in the most cost-effective manner.

To strengthen DOD's business system investment management capability, we
are recommending that the department fully define the policies and
procedures associated with project-level and portfolio-level investment
management as discussed in our guidance for IT investment management.^8

In written comments on a draft of this report, signed by the Deputy Under
Secretary of Defense (Business Transformation) and reprinted in appendix
II, the department stated that it agreed with the report's overall
conclusions, and it described efforts under way and planned that it said
would address many of the gaps identified in the report. In this regard,
the department partially concurred with five of the report's
recommendations, adding that our recommendations and feedback are helpful
in guiding DOD's business transformation and related improvement efforts.

However, the department disagreed with the remaining four recommendations
for two primary reasons. First, it stated that its existing investment
management structure already satisfies the intent of these
recommendations. For example, it stated that its policies already require
the provision of cost, schedule, and funding data as part of investment
certifications and annual reviews, and that a linkage currently exists
among the investment selection, acquisition, and funding processes. We do
not agree with this reasoning. Our recommendations are not intended to
address whether existing policies or guidance provide for the use of cost,
schedule, and funding data, or whether they state that investment
selection, acquisition, and funding decision making are linked. Rather,
our recommendations address the definitions of policy, guidance, and
supporting procedures that fall short of satisfying the best practices
embodied in our ITIM framework. In the case of the above examples, while
we do not question whether investment data are provided to investment
decision-making bodies, the department's policies and procedures do not
include specific decision criteria that explain how these data are to be
used to make consistent, repeatable selection and reselection decisions
across all investments. Furthermore, while we do not question that
existing guidance contains an illustration depicting a link between
investment certification and review and other DOD decision support
processes, including the funding process, neither this guidance nor
supporting procedures define how this linkage is executed (i.e., how
investment funding decisions are in fact integrated with investment
selection decisions).

^8 [39]GAO-04-394G .

Second, DOD stated that our recommendations contradict the department's
"tiered accountability" approach to investment management, in which
responsibility and accountability for business system investment
management is allocated between the Office of the Secretary of Defense
(corporate level) and DOD components (subsidiary levels) on the basis of
investment size and significance. We do not agree with the department's
reasoning. We support DOD's tiered accountability concept because it is
consistent with the hierarchical investment structures described in our
ITIM framework. Under the department's current policies and guidance,
however, most DOD investments are not subject to corporate visibility and
oversight, either because they do not involve development/modernization
(i.e., they are in operations and maintenance) or because they do not
exceed a certain dollar threshold. Our framework recognizes that effective
implementation of this concept should include appropriate corporate
visibility into and oversight of investments, either through review and
approval of those investments that meet certain criteria or through
awareness of a subordinate board's investment management activities.
Moreover, this visibility and oversight should extend to the entire
portfolio of investments, including those that are in operations and
maintenance. To ensure that this occurs, applicable policies and
procedures need to explicitly cover all such investments and need to
define how this is to be accomplished.

Background

DOD is a massive and complex organization. To illustrate, the department
reported that its fiscal year 2006 operations involved approximately $1.4
trillion in assets and $2.0 trillion in liabilities, more than 2.9 million
military and civilian personnel, and $581 billion in net cost of
operations. To date, for fiscal year 2007, the department received
appropriations of about $501 billion. Organizationally, the department
includes the Office of the Secretary of Defense (OSD), the Chairman of the
Joint Chiefs of Staff, the military departments, numerous defense agencies
and field activities, and various unified combatant commands that are
responsible for either specific geographic regions or specific functions.
(See fig. 1 for a simplified depiction of DOD's organizational structure.)

Figure 1: Simplified DOD Organizational Structure

^aThe Chairman of the Joint Chiefs of Staff serves as the spokesman for
the commanders of the combatant commands, especially on the administrative
requirements of their commands.

In support of its military operations, the department performs an
assortment of interrelated and interdependent business functions,
including logistics management, procurement, health care management, and
financial management. As we have previously reported,^9 the systems
environment that supports these business functions is overly complex and
error-prone, and is characterized by (1) little standardization across the
department, (2) multiple systems performing the same tasks, (3) the same
data stored in multiple systems, and (4) the need for data to be entered
manually into multiple systems. Moreover, according to DOD, this systems
environment is comprised of approximately 3,100 separate business systems.
For fiscal year 2007, Congress appropriated approximately $15.7 billion to
DOD, and for fiscal year 2008, DOD has requested about $15.9 billion in
appropriated funds to operate, maintain, and modernize these business
systems and the associated infrastructures.

^9 [40]GAO-06-658 .

As we have previously reported,^10 the department's nonintegrated and
duplicative systems impair DOD's ability to combat fraud, waste, and
abuse. In fact, DOD currently bears responsibility, in whole or in part,
for 15 of our 27 high-risk areas.^11 Eight of these areas are specific to
DOD,^12 and the department shares responsibility for 7 other
governmentwide high-risk areas.^13 DOD's business systems modernization is
one of the high-risk areas, and it is an essential enabler to addressing
many of the department's other high-risk areas. For example, modernized
business systems are integral to the department's efforts to address its
financial, supply chain, and information security management high-risk
areas.

IT Investment Management Is Critical to Achieving Successful Systems
Modernization

A corporate approach to IT investment management is characteristic of
successful public and private organizations. Recognizing this, Congress
enacted the Clinger-Cohen Act of 1996,^14 which requires the Office of
Management and Budget (OMB) to establish processes to analyze, track, and
evaluate the risks and results of major capital investments in IT systems
made by executive agencies.^15 In response to the Clinger-Cohen Act and
other statutes, OMB has developed policy and issued guidance for the
planning, budgeting, acquisition, and management of federal capital
assets.^16 We have also issued guidance in this area,^17 which defines
institutional structures, such as the IRBs; processes for developing
information on investments (such as costs and benefits); and practices to
inform management decisions (such as whether a given investment is aligned
with an enterprise architecture).

^10See, for example, GAO, DOD Travel Cards: Control Weaknesses Resulted in
Millions of Dollars of Improper Payments, [41]GAO-04-576 (Washington,
D.C.: June 9, 2004); Military Pay: Army National Guard Personnel Mobilized
to Active Duty Experienced Significant Pay Problems, [42]GAO-04-89
(Washington, D.C.: Nov. 13, 2003); and Defense Inventory: Opportunities
Exist to Improve Spare Parts Support Aboard Deployed Navy Ships,
[43]GAO-03-887 (Washington, D.C.: Aug. 29, 2003).

^11 [44]GAO-07-310 .

^12These 8 high-risk areas include DOD's (1) overall approach to business
transformation, (2) business systems modernization, (3) financial
management, (4) personnel security clearance program, (5) supply chain
management, (6) support infrastructure management, (7) weapon systems
acquisition, and (8) contract management.

^13The 7 governmentwide high-risk areas are (1) disability programs, (2)
ensuring the effective protection of technologies critical to U.S.
national security interests, (3) interagency contracting, (4) information
systems and critical infrastructure, (5) information-sharing for homeland
security, (6) human capital, and (7) real property.

^14The Clinger-Cohen Act of 1996, 40 U.S.C. SS 11101-11704. This act
expanded the responsibilities of OMB and the agencies that had been set
under the Paperwork Reduction Act with regard to IT management. See 44
U.S.C. 3504(a)(1)(B)(vi) (OMB); and 44 U.S.C. 3506(h)(5) (agencies).

^15We have made recommendations to improve OMB's process for monitoring
high-risk IT investments; see GAO, Information Technology: OMB Can Make
More Effective Use of Its Investment Reviews, [45]GAO-05-276 (Washington,
D.C.: Apr. 15, 2005).

  IT Investment Management: A Brief Description

IT investment management is a process for linking IT investment decisions
to an organization's strategic objectives and business plans. Consistent
with this, the federal approach to IT investment management focuses on
selecting, controlling, and evaluating investments in a manner that
minimize risks while maximizing the return of investment.^18

           o During the selection phase, the organization (1) identifies and
           analyzes each project's risks and returns before committing
           significant funds to any project and (2) selects those IT projects
           that will best support its mission needs.
           o During the control phase, the organization ensures that
           projects, as they develop and investment expenditures continue,
           meet mission needs at the expected levels of cost and risk. If the
           project is not meeting expectations or if problems arise, steps
           are quickly taken to address the deficiencies.
           o During the evaluation phase, expected results are compared with
           actual results after a project has been fully implemented. This
           comparison is done to (1) assess the project's impact on mission
           performance, (2) identify any changes or modifications to the
           project that may be needed, and (3) revise the investment
           management process based on lessons learned.
		   
^16This policy is set forth and guidance is provided in OMB Circular A-11
(Nov. 2, 2005) (section 300), and in OMB's Capital Programming Guide,
which directs agencies to develop, implement, and use a capital
programming process to build their capital asset portfolios.

^17See, for example, [46]GAO-04-394G ; GAO, Information Technology: A
Framework for Assessing and Improving Enterprise Architecture Management
(Version 1.1), [47]GAO-03-584G (Washington, D.C.: April 2003); and
Assessing Risks and Returns: A Guide for Evaluating Federal Agencies' IT
Investment Decision-making,  GAO/AIMD-10.1.13 (Washington, D.C.: February
1997).

^18GAO-04-394G; GAO/AIMD-10.1.13; GAO, Executive Guide: Improving Mission
Performance Through Strategic Information Management and Technology,
GAO/AIMD-94-115 (Washington, D.C.: May 1994); and Office of Management and
Budget, Evaluating Information Technology Investments, A Practical Guide
(Washington, D.C.: November 1995).

           Overview of GAO's ITIM Maturity Framework

           Our ITIM framework consists of five progressive stages of maturity
           for any given agency relative to selecting, controlling, and
           evaluating its investment management capabilities.^19 (See fig. 2
           for the five ITIM stages of maturity.) This framework is grounded
           in our research of IT investment management practices of leading
           private and public sector organizations. The maturity stages are
           cumulative; that is, to attain a higher stage, an agency must
           institutionalize all of the critical processes at the lower
           stages, in addition to the higher stage critical processes.

           The framework can be used to assess the maturity of an agency's
           investment management processes and as a tool for organizational
           improvement. The overriding purpose of the framework is to
           encourage investment selection and control and to evaluate
           processes that promote business value and mission performance,
           reduce risk, and increase accountability and transparency. We have
           used the framework in several of our evaluations,^20 and a number
           of agencies have adopted it.

           With the exception of the first stage, each maturity stage is
           composed of "critical processes" that must be implemented and
           institutionalized for the organization to achieve that stage. Each
           ITIM critical process consists of "key practices"--to include
           organizational structures, policies, and procedures--that must be
           executed to implement the critical process. It is not unusual for
           an organization to perform key practices from more than one
           maturity stage at the same time. However, our research shows that
           agency efforts to improve investment management capabilities
           should focus on implementing all lower-stage practices before
           addressing higher-stage practices.
		   
^19GAO-04-394G.

^20GAO, Information Technology: Centers for Medicare & Medicaid Services
Needs to Establish Critical Investment Management Capabilities, GAO-06-12
(Washington, D.C.: Oct. 28, 2005); Information Technology: HHS Has Several
Investment Management Capabilities in Place, but Needs to Address Key
Weaknesses, GAO-06-11 (Washington, D.C.: Oct. 28, 2005); Information
Technology: FAA Has Many Investment Management Capabilities in Place, but
More Oversight of Operational Systems Is Needed, GAO-04-822 (Washington,
D.C.: Aug. 20, 2004); Bureau of Land Management: Plan Needed to Sustain
Progress in Establishing IT Investment Management Capabilities,
GAO-03-1025 (Washington, D.C.: Sept. 12, 2003); Information Technology:
Departmental Leadership Crucial to Success of Investment Reforms at
Interior, GAO-03-1028 (Washington, D.C.: Sept. 12, 2003); United States
Postal Service: Opportunities to Strengthen IT Investment Management
Capabilities, GAO-03-3 (Washington, D.C.: Oct. 15, 2002); and Information
Technology: DLA Needs to Strengthen Its Investment Management Capability,
GAO-02-314 (Washington, D.C.: Mar. 15, 2002).		   

           In the ITIM framework, Stage 2 critical processes lay the
           foundation by establishing successful, predictable, and repeatable
           investment control processes at the project level. At this stage,
           the emphasis is on establishing basic capabilities for selecting
           new IT projects; controlling projects so that they finish
           predictably within the established cost, schedule, and performance
           expectations; and identifying and mitigating exposure to risk.

           Stage 3 is where the agency moves from project-centric processes
           to portfolio-based processes and evaluates potential investments
           according to how well they support the agency's missions,
           strategies, and goals. This stage focuses on continually assessing
           both proposed and ongoing projects as part of complete investment
           portfolios--integrated and competing sets of investment options.
           It also focuses on maintaining mature, integrated selection (and
           reselection); control; and postimplementation evaluation
           processes. This portfolio perspective allows decision makers to
           consider the interaction among investments and the contributions
           to organizational mission goals and strategies that could be made
           by alternative portfolio selections, rather than to focus
           exclusively on the balance between the costs and benefits of
           individual investments. Organizations implementing Stages 2 and 3
           practices have in place capabilities that assist in establishing
           selection, control, and evaluation structures, policies,
           procedures, and practices that are required by the investment
           management provisions of the Clinger-Cohen Act.^21

           Stages 4 and 5 require the use of evaluation techniques to
           continuously improve both investment processes and portfolios to
           better achieve strategic outcomes. At Stage 4, an organization has
           the capacity to conduct IT succession activities and, therefore,
           can plan and implement the deselection of obsolete, high-risk, or
           low-value IT investments. An organization with Stage 5 maturity
           conducts proactive monitoring for breakthrough technologies that
           will enable it to change and improve its business performance.
		   
^21The Clinger-Cohen Act of 1996, 40 U.S.C. SS 11311-11313.

           Figure 2: The Five ITIM Stages of Maturity with Critical Processes
		   
		   Overview of DOD's Corporate Approach for Identifying, Funding, and
		   Acquiring All System Investments

           DOD's major system investments (i.e., weapon and business systems)
           are governed by three management systems--the Joint Capabilities
           Integration and Development System (JCIDS); the Planning,
           Programming, Budgeting, and Execution (PPBE) system; and the
           Defense Acquisition System (DAS).

           o JCIDS is a need-driven, capabilities-based approach to identify
           warfighting needs and meet future joint forces challenges. It is
           intended to identify future capabilities for DOD; address
           capability gaps and mission needs recognized by the Joint Chiefs
           of Staff or derived from strategic guidance, such as the National
           Security Strategy Report^22 or Quadrennial Defense Review;^23 and
           identify alternative solutions by considering a range of doctrine,
           organization, training, materiel, leadership and education,
           personnel, and facilities solutions. According to DOD, the Joint
           Chiefs of Staff, through the Joint Requirements Oversight Council,
           has primary responsibility for defining and implementing JCIDS.
           o PPBE is a calendar-driven approach that is composed of four
           phases that occur over a moving 2-year cycle. The four
           phases--planning, programming, budgeting, and executing--define
           how budgets for each DOD component and the department as a whole
           are created, vetted, and executed. As recently reported,^24 the
           components start programming and budgeting for addressing a
           JCIDS-identified capability gap or mission need several years
           before actual product development under DAS begins, and before OSD
           formally reviews the components' programming and budgeting
           proposals (i.e., Program Objective Memorandums). Once reviewed and
           approved, the financial details in the Program Objective
           Memorandums become part of the President's budget request to
           Congress. During budget execution, components may submit program
           change proposals or budget change proposals, or both (e.g.,
           program cost increases or schedule delays). According to DOD, the
           OSD Under Secretary of Defense (Policy), the Director for Program
           Analysis and Evaluation,^25 and the Under Secretary of Defense
           (Comptroller) have primary responsibility for defining and
           implementing the PPBE system.
		   
^22The National Security Strategy Report required by 50 U.S.C. 404a is a
comprehensive report on the national security strategy of the United
States submitted by the President to Congress.

^23See 10 U.S.C. 118. The Quadrennial Defense Review is a comprehensive
examination of the national defense strategy, force structure, force
modernization plans, infrastructure, budget plan, and other elements of
the defense program and policies of the United States with a view toward
determining and expressing the defense strategy of the United States and
establishing a defense program for the next 20 years.

^24GAO, Best Practices: An Integrated Portfolio Management Approach to
Weapon System Investments Could Improve DOD's Acquisition Outcomes,
GAO-07-388 (Washington, D.C.: Mar. 30, 2007).

^25The Director for Program Analysis and Evaluation is the principal staff
assistant who conducts independent analysis for, and provides independent
advice on, all DOD program and evaluation matters to the Secretary and
Deputy Secretary of Defense.
		   
           o DAS is described in the DOD Directive 5000.1 and the DOD
           Instruction 5000.2^26 and establishes the procedures for the
           Defense Acquisition Management Framework, which consists of three
           event-based milestones associated with five key program life-cycle
           phases. These five phases are as follows:

                        1. Concept Refinement: Intended to refine the initial
                        JCIDS-validated system solution (concept) and create
                        a strategy for acquiring the investment solution. A
                        decision is made at the end of this phase (milestone
                        A decision) regarding whether to move to the next
                        phase (Technology Development).
                        2. Technology Development: Intended to determine the
                        appropriate set of technologies to be integrated into
                        the investment solution by iteratively assessing the
                        viability of various technologies while
                        simultaneously refining user requirements. Once the
                        technology has been demonstrated in a relevant
                        environment, a decision is made at the end of this
                        phase (milestone B decision) regarding whether to
                        move to the next phase (System Development and
                        Demonstration).
						
^26DOD Directive 5000.1, May 12, 2003 and DOD Instruction 5000.2, May 12,
2003.						

                        3. System Development and Demonstration: Intended to
                        develop a system or a system increment and
                        demonstrate through developer testing that the
                        system/system increment can function in its target
                        environment. A decision is made at the end of this
                        phase (milestone C decision) regarding whether to
                        move to the next phase (Production and Deployment).
                        4. Production and Deployment: Intended to achieve an
                        operational capability that satisfies the mission
                        needs, as verified through independent operational
                        test and evaluation, and ensures that the system is
                        implemented at all applicable locations.
                        5. Operations and Support: Intended to operationally
                        sustain the system in the most cost-effective manner
                        over its life cycle.

           A key principle of DAS is that investments are assigned a
           category, where programs of increasing dollar value and management
           interest are subject to more stringent oversight. For example,
           Major Defense Acquisition Programs (MDAP)^27 and Major Automated
           Information Systems (MAIS)^28 are large, expensive programs
           subject to the most extensive statutory and regulatory reporting
           requirements and, unless delegated, are reviewed by acquisition
           boards at the DOD corporate level. Smaller and less risky
           acquisitions are generally reviewed at the component executive or
           lower levels. Another key principle is that DAS requires
           acquisition management under the direction of a milestone decision
           authority.^29 The milestone decision authority--with support from
           the program manager and advisory boards, such as the Defense
           Acquisition Board^30 and the IT Acquisition Board^31--determines
           the project's baseline cost, schedule, and performance
           commitments. The Under Secretary of Defense for Acquisition,
           Technology, and Logistics (USD(AT&L)) has primary responsibility
           for defining and implementing DAS.
		   
^27A MDAP is an acquisition program that is estimated by the Under
Secretary of Defense for Acquisition, Technology, and Logistics to require
an eventual total expenditure for research, development, and test and
evaluation of more than $365 million (fiscal year 2000 constant dollars)
or, for procurement, of more than $2.190 billion (fiscal year 2000
constant dollars).

^28A MAIS is a program or initiative that is so designated by the
Assistant Secretary of Defense (Networks and Information
Integration)/Chief Information Officer or that is estimated to require
program costs in any single year in excess of $32 million (fiscal year
2000 constant dollars), total program costs in excess of $126 million
(fiscal year 2000 constant dollars), or total life-cycle costs in excess
of $378 million (fiscal year 2000 constant dollars).

           DOD Business System Investments Are Subject to a Fourth Management
		   System

           DOD's business system investments are also governed by a fourth
           management system that addresses how these investments are
           reviewed, certified, and approved for compliance with the business
           enterprise priorities and activities outlined by the business
           enterprise architecture (BEA). For the purposes of this report, we
           refer to this fourth management system as the Business Investment
           Management System. This fourth management system is described in
           the following text in terms of governance entities, tiered
           accountability, and business system investment certification
           reviews and approvals. According to DOD, these four management
           systems are the means by which DOD selects, controls, and
           evaluates its business system investments.

             Business System Investment Roles and Responsibilities

           In 2005, the department reassigned responsibility for providing
           executive leadership for the direction, oversight, and execution
           of its business systems modernization efforts to several entities.
           These entities and their responsibilities include the following:
		   
^29According to DOD, the milestone decision authority is the designated
individual who has overall responsibility for an investment. This person
has the authority to approve an investment's progression in the
acquisition process and is responsible for reporting cost, schedule, and
performance results. For example, the milestone decision authority for a
MDAP program, when not delegated to the component level, is the Under
Secretary of Defense for Acquisition, Technology, and Logistics, and the
milestone decision authority for a MAIS system is the Assistant Secretary
of Defense (Networks and Information Integration)/Chief Information
Officer or a designee.

^30The Defense Acquisition Board, chaired by the Under Secretary of
Defense for Acquisition, Technology, and Logistics, conducts reviews for
MDAPs at major program milestones and documents the decision(s) resulting
from the review in an Acquisition Decision Memorandum.

^31The IT Acquisition Board, chaired by the Assistant Secretary of Defense
(Networks and Information Integration)/Chief Information Officer, conducts
reviews for MAIS at major program milestones and documents the decision(s)
resulting from the review in an Acquisition Decision Memorandum.

           o The Defense Business Systems Management Committee (DBSMC) serves
           as the highest-ranking governance body for business systems
           modernization activities.
           o The Principal Staff Assistants serve as the certification
           authorities for business system modernizations in their respective
           core business missions.
           o The IRBs are chartered by the Principal Staff Assistants and are
           the review and decision-making bodies for business system
           investments in their respective areas of responsibility.^32 
           o The component pre-certification authority (PCA) is accountable
           for the component's business system investments and acts as the
           component's principal point of contact for communication with the
           IRBs.
           o The Business Transformation Agency (BTA) is responsible for
           leading and coordinating business transformation efforts across
           the department. The BTA is organized into seven directorates, one
           of which is the Defense Business Systems Acquisition Executive
           (DBSAE)--the component acquisition executive for DOD
           enterprise-level (DOD-wide) business systems and initiatives. This
           directorate is responsible for developing, coordinating, and
           integrating enterprise-level projects, programs, systems, and
           initiatives--including managing resources such as fiscal,
           personnel, and contracts for assigned systems and programs.

           Table 1 lists these entities and provides greater detail on their
           roles, responsibilities, and composition. Figure 3 provides a
           simplified illustration of the relationships among these entities.

^32The four IRBs are for (1) Financial Management, established by the
Deputy Under Secretary of Defense for Financial Management; (2) Weapon
Systems Lifecycle Management and Materiel Supply and Services Management;
(3) Real Property and Installations Lifecycle Management, both established
by the USD(AT&L); and (4) Human Resources Management, established by the
Under Secretary of Defense for Personnel and Readiness.

Table 1: DOD Business Investment Management System Entities' Roles,
Responsibilities, and Composition

Source: GAO based on DOD documentation.

^aAccording to DOD, the BMA is responsible for ensuring that capabilities,
resources, and materiel are reliably delivered to the warfighter.
Specifically, the BMA addresses areas such as real property and human
resources management.

^bDOD has five core business missions: Human Resources Management, Weapon
System Lifecycle Management, Materiel Supply and Services Management, Real
Property and Installations Lifecycle Management, and Financial Management.

Figure 3: Working Relationships among DOD Business Investment Management
System Governance Entities

  Tiered Accountability

According to DOD, in 2005 it adopted a tiered accountability approach to
business transformation. Under this approach, responsibility and
accountability for business investment management is allocated between the
DOD corporate (i.e., OSD) and the components on the basis of the amount of
development/modernization funding involved and the investment's "tier."
DOD corporate is responsible for ensuring that all business systems with a
development/modernization investment in excess of $1 million are reviewed
by the IRBs for compliance with the BEA, certified by the Principal Staff
Assistants, and approved by the DBSMC. Components are responsible for
certifying development/modernization investments with total costs of $1
million or less. All DOD development and modernization efforts are also
assigned a tier on the basis of the acquisition category or the size of
the financial investment, or both. According to DOD, a system is given a
tier designation when it passes through the certification process. Table 2
describes the four investment tiers and identifies the associated
reviewing and approving entities.

Table 2: DOD's Investment Tiers

Source: DOD.

  Business Investment Certification Reviews and Approvals

DOD's business investment management system includes two types of reviews
for business systems: certification and annual reviews. Certification
reviews apply to new modernization projects with total cost over $1
million. This review focuses on program alignment with the BEA and must be
completed before components obligate funds for programs. The annual review
applies to all business programs. The focus for the annual review is to
determine whether the system development effort is meeting its milestones
and addressing its IRB certification conditions.

Certification reviews and approvals: Tiers 1 through 3 business system
investments are certified at two levels--component-level precertification
and corporate-level certification and approval. At the component level,
program managers prepare, enter, maintain, and update information about
their investments in the DOD IT Portfolio Repository (DITPR),^33 such as
regulatory compliance reporting, an architectural profile, and
requirements for investment certification and annual reviews. The
component PCA validates that the system information is complete and
accessible on the IRB Portal, reviews system compliance with the BEA and
enterprise transition plan, and verifies the economic viability analysis.
The PCA asserts the status and validity of the investment information by
submitting a component precertification letter to the appropriate IRB for
its review.

^33DITPR is DOD's authoritative repository for certain information about
DOD's business systems, such as system names and the responsible DOD
components, that are required for the certification, approval, and annual
reviews of these business system investments.

At the corporate level, the IRB reviews the system information and
precertification letter submitted by the PCA to determine whether to
recommend investment certification. On completion of its review, a
certification memorandum is prepared and signed by the designated
certification authority^34 that documents the IRB's system certification
decisions and any related conditions. The memorandum is then forwarded to
the DBSMC, which either approves or disapproves the IRB's decisions and
issues a memorandum containing its decisions. If the DBSMC disapproves a
system investment, it is up to the component PCA to decide whether to
resubmit the investment after it has resolved the relevant issues. Figure
4 provides a simplified overview of the process flow of certification
reviews and approvals.

^34The certification authority is the designated Principal Staff Assistant
with responsibility for review, approval, and oversight of the planning,
design, acquisition, deployment, operation, maintenance, and modernization
of defense business systems.

Figure 4: Simplified Process Flow of Certification Reviews and Approvals

Annual reviews: Tiers 1 through 4 business system investments are annually
reviewed at two levels--the component level and the corporate level. At
the component level, program managers review and update information on all
tiers of investments, both in modernization and operations and
maintenance, on an annual basis in DITPR. The updates for Tiers 1 through
3 with system development/modernization include cost, milestone, and risk
variances and actions or issues related to certification conditions. The
PCA then verifies and submits the information for Tiers 1 through 3
systems in development/modernization for IRB review in an annual review
assertion letter. The letter addresses system compliance with the BEA and
the enterprise transition plan, and includes investment cost, schedule,
and performance information.^35

At the corporate level, the IRBs annually review certified Tiers 1 through
3 investments in development/modernization. These reviews focus on program
compliance with the BEA, program performance against cost and milestone
baselines, and progress in meeting certification conditions. The IRBs can
revoke an investment's certification when the system has significantly
failed to achieve performance commitments (i.e., capabilities and costs).
When this occurs, the component must address the IRB's concerns and
resubmit the investment for certification. Figure 5 shows a simplified
overview of the process flow of annual reviews.

Figure 5: Simplified Process Flow of Annual Reviews

^35In addition, each component PCA submits a list of system names to the
IRBs on a semiannual basis, to include Tier 4 systems and systems in
operations and maintenance that have been reviewed at the component level.

DOD Has Established the Structures Needed to Effectively Manage Business System
Investments, but Has Not Fully Defined Many of the Related Policies and
Procedures

According to our ITIM framework, organizations should establish the
management structures needed to manage their investments and build an
investment foundation by having defined policies and procedures for
selecting and controlling individual projects (Stage 2 capabilities), and
organizations also should manage projects as a portfolio of investments
according to defined policies and procedures, treating them as an
integrated package of competing investment options and pursuing those that
best meet the strategic goals, objectives, and mission of the agency
(Stage 3 capabilities). These Stages 2 and 3 capabilities assist agencies
in complying with the investment management provisions of the
Clinger-Cohen Act.

The department has defined four of nine practices that call for
project-level policies and procedures (see table 4) and one of the five
practices that call for portfolio-level policies and procedures (see table
6). Specifically, it has established the management structures contained
in our ITIM framework, but it has not fully defined many of the related
policies and procedures.

With respect to project-level investment management practices, DOD
officials stated that these are performed at the component level, and that
departmental policies and procedures established for overseeing
components' execution of these practices are sufficient. With respect to
portfolio-level practices, however, these officials stated that they
intend to improve departmental policies and procedures for business system
investments by, for example, establishing a single governance structure,
but plans or time frames for doing so have not been established. According
to our ITIM framework, adequately documenting both the policies and the
associated procedures that govern how an organization manages its IT
investment portfolio(s) is important because doing so provides the basis
for having rigor, discipline, and repeatability in how investments are
selected and controlled across the entire organization. Until DOD fully
defines departmentwide policies and procedures for both individual
projects and the portfolios of projects, it risks selecting and
controlling these business system investments in an inconsistent,
incomplete, and ad hoc manner, which in turn reduces the chances that
these investments will meet mission needs in the most cost-effective
manner.

DOD Has Begun to Build a Foundation for Project-Level Investment Management, but
Key Policies and Procedures Are Not Fully Defined

At ITIM Stage 2, an organization has attained repeatable and successful IT
project-level investment control and basic selection processes. Through
these processes, the organization can identify project expectation gaps
early and take the appropriate steps to address them. ITIM Stage 2
critical processes include (1) defining investment board operations, (2)
identifying the business needs for each investment, (3) developing a basic
process for selecting new proposals and reselecting ongoing investments,
(4) developing project-level investment control processes, and (5)
collecting information about existing investments to inform investment
management decisions. Table 3 describes the purpose of each of these Stage
2 critical processes.

Table 3: Stage 2 Critical Processes--Building the Investment Foundation

Source: GAO.

Within these five critical processes are nine key practices that call for
policies and procedures associated with effective project-level
management. DOD has fully defined the policies and procedures needed to
ensure that four of these nine practices are performed in a consistent and
repeatable manner. Specifically, DOD has established the management
structures by instituting an enterprisewide investment board--the
DBSMC--composed of senior executives, including the Deputy Secretary of
Defense, with final approval authority over associated subsidiary
investment boards. These lower-level investment boards include
representatives from combatant commands, components, and the Joint Chiefs
of Staff. In addition, DOD's business transformation and IRB guidance
define a process for ensuring that programs support the department's
ongoing and future business needs. DOD also has policies and procedures
for submitting, updating, and maintaining investment information in DITPR
and the IRB Portal. Furthermore, the department has assigned the
component's PCA the responsibility to ensure that specific investment
information contained in the portfolio repository and the IRB Portal is
accurate and complete.

However, the policies and procedures associated with the remaining five
project-level management practices are missing critical elements needed to
effectively carry out essential investment management activities. For
example:

           o Policies and procedures for instituting the investment board do
           not address how investments that are past the
           development/modernization stage (i.e., in operations and
           maintenance) are to be governed. Given that DOD invests billions
           of dollars annually in operating and maintaining business systems,
           this is significant. While DOD officials stated that
           component-level policies and procedures address systems outside of
           development/modernization, our ITIM framework emphasizes that the
           corporate investment boards should continue to review important
           information about an investment, such as cost and performance
           baselines, throughout the investment's life cycle. In addition,
           the IRB Concept of Operations and other IRB documentation do not
           explicitly outline how the business investment management system
           is coordinated with JCIDS, PPBE, and DAS. Without clearly defined
           visibility into all investments with an understanding of decisions
           reached through other management systems, inconsistent decisions
           may result.
           o Procedures do not specify how the full range of cost, schedule,
           and benefit data is used by the IRBs in making selection (i.e.,
           certification) decisions. According to BTA officials, each IRB
           decides how to ensure compliance and determines additional factors
           to consider when making certification decisions. However, DOD did
           not provide us with any supplemental policies or procedures for
           any of the four IRBs. Without documenting how IRBs consider
           factors such as cost, schedule, and benefits when making selection
           decisions, the department cannot ensure that the IRBs and the
           DBSMC consistently and objectively select proposals that best meet
           the department's needs and priorities. Furthermore, while the
           procedures specify decision criteria that address statutory
           requirements for alignment to the BEA, the criteria allow programs
           to postpone demonstrating full compliance with several BEA
           artifacts until the final phases of the acquisition process. As a
           result, programs risk beginning production and deployment before
           ensuring that a business system is fully aligned to the BEA.
           o Policies and procedures do not specify how reselection decisions
           at the corporate level (i.e., annual review decisions) consider
           investments that are in operations and maintenance. Without an
           understanding of how the IRBs are to consider these investments
           when making reselection decisions, their ability to make informed
           and consistent reselection and termination decisions is limited.
           o Policies and procedures do not specify how funding decisions are
           integrated with the process of selecting an investment at the
           corporate level. Without considering component and corporate
           budget constraints and opportunities, the IRBs risk making
           investment decisions that do not effectively consider the relative
           merits of various projects and systems when funding limitations
           exist.
           o Policies and procedures do not exist that provide for sufficient
           oversight and visibility into component-level investment
           management activities, including component reviews of systems in
           operations and maintenance and Tier 4 investments. According to
           DOD officials, investment oversight is implemented through tiered
           accountability, which, among other things, allocates
           responsibility and accountability for business system investments
           with total costs of $1 million or less and those in operations and
           maintenance to the components. However, the department did not
           provide policies and procedures defining how the DBSMC and the
           IRBs ensure visibility into these component processes. This is
           particularly important because, according to DOD's March 15, 2007,
           annual report to Congress, only 285 of approximately 3,100 total
           business systems have completed the IRB certification process and
           have been approved by the DBSMC. DOD officials also stated that
           the remaining business systems have not been through the
           certification process and have not been given a tier designation.
           Without policies and procedures defining how the DBSMC and the
           IRBs have visibility into and oversight of all business system
           investments, DOD risks components continuing to invest in systems
           that are duplicative, stovepiped, nonintegrated, and unnecessarily
           costly to manage, maintain, and operate.

           Table 4 summarizes our findings relative to DOD's execution of the
           nine practices that call for the policies and procedures needed to
           manage IT investments at the project level.

Table 4: Summary of Policies and Procedures for Stage 2 Critical
Processes--Building the Investment Foundation

Source: GAO.

According to BTA officials, the IRB Concept of Operations and the
Investment Certification and Annual Review Process User Guidance are not
intended to describe the detailed approach that each IRB will use when
making certification decisions, adding that the components are responsible
for selection, annual review, budgeting, and acquisition. While the ITIM
framework does allow for multiple entities to carry out investment
selection, control, and evaluation, building a sound investment foundation
requires that the enterprisewide investment review board has documented
criteria and decision-making procedures, clear integration among
investment decision-support systems, and policies to ensure board access
to system information throughout the life cycle for all investments. Until
DOD's documented IT investment management policies and procedures include
fully defined policies and procedures for Stage 2 activities, specify the
linkages between the various related processes, and describe how
investments are to be governed in the operations and maintenance phase,
DOD risks that investment management activities will not be carried out
consistently and in a disciplined manner. Moreover, DOD also risks
selecting investments that will not cost-effectively meet its mission
needs.

DOD Has Assigned Responsibility, but Has Not Defined the Policies and Procedures
Associated with Effective Portfolio-Level Management

At Stage 3, an organization has defined critical processes for managing
its investments as a portfolio or set of portfolios.^36 Portfolio
management is a conscious, continuous, and proactive approach to
allocating limited resources among competing initiatives in light of the
investments' relative benefits. Taking an agencywide perspective enables
an organization to consider its investments comprehensively, so that
collectively the investments optimally address the organization's
missions, strategic goals, and objectives. Managing IT investments as
portfolios also allows an organization to determine its priorities and
make decisions about which projects to fund on the basis of analyses of
the relative organizational value and risks of all projects, including
projects that are proposed, under development, and in operation. Although
investments may initially be organized into subordinate portfolios--on the
basis of, for example, business lines or life-cycle stages--and managed by
subordinate investment boards, they should ultimately be aggregated into
enterprise-level portfolios.

According to ITIM, Stage 3 involves (1) defining the portfolio criteria;
(2) creating the portfolio; (3) evaluating (i.e., overseeing) the
portfolio; and (4) conducting postimplementation reviews. Table 5
summarizes the purpose of each of these activities.

Table 5: Stage 3 Critical Processes--Developing a Complete Investment
Portfolio

Source: GAO.

^36Investment portfolios are integrated agencywide collections of
investments that are assessed and managed collectively on the basis of
common criteria.

DOD is executing one of the five practices within these four critical
processes that call for policies and procedures associated with effective
portfolio-level management. Specifically, DOD has issued departmentwide
guidance^37 that assigns responsibilities to the USD(AT&L) for managing
and establishing business system investment portfolios, including
leveraging or establishing a governance forum to oversee these business
system investment portfolio activities.

However, DOD has not fully defined the policies and procedures needed to
effectively execute the remaining four portfolio management practices
relative to business system investments. Specifically, DOD does not have
policies and procedures for defining the portfolio criteria or for
creating and evaluating the portfolio. In addition, while DOD has policies
and procedures for conducting postimplementation reviews as part of DAS,
these reviews do not address systems at all tier levels. Furthermore,
there are no procedures detailing how lessons learned from these reviews
are used during investment review as the basis for management and process
improvements.

Table 6 summarizes the rating for each critical process required to manage
investment as a portfolio and summarizes the evidence that supports these
ratings.

^37DOD Directive 8115.01, Information Technology Portfolio Management, and
DOD Instruction 8115.02, Information Technology Portfolio Management
Implementation.

Table 6: Summary of Policies and Procedures for Stage 3 Critical
Processes--Developing a Complete Investment Portfolio

Source: GAO.

According to BTA officials, while portfolio management is primarily a
component responsibility, they are working toward developing more
effective departmentwide portfolio management processes, but plans or time
frames for doing so have not been established. Without defining corporate
policies and procedures for managing business system investment
portfolios, DOD is at risk of not consistently selecting the mix of
investments that best supports the departmentwide mission needs and
ensuring that investment-related lessons learned are shared and applied
departmentwide.

Conclusions

Given the importance of business systems modernization to DOD's mission,
performance, and outcomes, it is vital for the department to adopt and
employ an effective institutional approach to managing business system
investments. While the department has established aspects of such an
approach and, thus, has a foundation on which to build, it is lacking
other important elements, such as specific policies and procedures needed
for project-level and portfolio-level investment management, including
integration with DOD's other key management systems and sufficient
oversight and visibility into operations and maintenance investments and
Tier 4 investments. This means that DOD lacks an institutional capability
to ensure that it is investing in business systems that best support its
strategic needs, and that ongoing projects meet cost, schedule, and
performance expectations. Until DOD develops this capability, the
department will be impaired in its ability to optimize business mission
area performance and accountability.

Recommendations for Executive Action

To strengthen DOD's business system investment management capability and
address the weaknesses discussed in this report, we recommend that the
Secretary of Defense direct the Deputy Secretary of Defense, as the chair
of the DBSMC, to ensure that well-defined and disciplined business system
investment management policies and procedures are developed and issued. At
a minimum, this should include project-level management policies and
procedures that address the following five areas:

           o instituting the investment boards, including assigning the
           investment boards responsibility, authority, and accountability
           for programs throughout the investment life cycle and specifying
           how the business investment management system is coordinated with
           JCIDS, PPBE, and DAS;
           o selecting new investments, including specifying how cost,
           schedule, and benefit data are to be used in making certification
           decisions; defining the criteria used to select investments as
           enterprisewide; and establishing consistent and effective guidance
           for BEA compliance;
           o reselecting ongoing investments, including specifying how cost,
           schedule, and performance data are to be used in the annual review
           process and providing for the reselection of investments that are
           in operations and maintenance;
           o integrating funding with the process of selecting an investment,
           including specifying how the DBSMC and the IRBs use funding
           information in carrying out decisions on system certification and
           approvals; and
           o overseeing IT projects and systems, including providing
           sufficient oversight and visibility into component-level
           investment management activities.

           These well-defined and disciplined business system investment
           management policies and procedures should also include
           portfolio-level management policies and procedures that address
           the following four areas:

           o creating and modifying IT portfolio selection criteria for
           business system investments;
           o analyzing, selecting, and maintaining business system investment
           portfolios;
           o reviewing, evaluating, and improving the performance of its
           portfolio(s) by using project indicators, such as cost, schedule,
           and risk; and
           o conducting postimplementation reviews for all investment tiers
           and directing the investment boards, which are accountable for
           corporate business system investments, to consider the information
           gathered and to develop lessons learned from these reviews.
		   
		   Agency Comments and Our Evaluation

           In written comments on a draft of this report, signed by the
           Deputy Under Secretary of Defense (Business Transformation) and
           reprinted in appendix II, the department stated that it agreed
           with the report's overall conclusions, and it described efforts
           under way and planned that it said would address many of the gaps
           identified in the report. In this regard, the department partially
           concurred with five of the report's recommendations, adding that
           our recommendations and feedback are helpful in guiding DOD's
           business transformation and related improvement efforts.
           Nevertheless, the department disagreed with the remaining four
           recommendations on the grounds that their intent had already been
           met through DOD's existing business system investment management
           structure and processes, or that they contradicted the tiered
           accountability concept embedded in this structure and processes.
           The department's comments relative to each of our project-level
           and portfolio-level recommendations, along with our responses to
           its comments, are provided below.

           With respect to our five project-level recommendations, the
           department stated that it partially agreed with two and disagreed
           with three.

           o DOD partially agreed with our recommendation to define and
           implement policies and procedures that assign the investment
           boards responsibility for programs throughout the investment life
           cycle and specify how the business investment management system is
           coordinated with JCIDS, PPBE, and DAS. In particular, it stated
           that under its tiered accountability approach to business systems
           investment management, the components are currently required to
           review all programs throughout their investment life cycles. We do
           not question this requirement, and we recognize it in our report.
           However, consistent with our ITIM framework, the corporate
           investment boards should continue to review investments that meet
           the defined threshold criteria throughout their life cycles (i.e.,
           when they are in operations and maintenance). In contrast, DOD's
           corporate boards focus only on those investments that are in the
           development/modernization stage. The department also stated that a
           linkage is currently depicted in existing guidance among its
           investment selection, acquisition, and funding processes. While we
           do not question that this guidance contains an illustration
           depicting such a link, neither this guidance nor supporting
           procedures define how this linkage is executed (e.g., how
           investment funding decisions are in fact integrated with
           investment selection decisions). DOD's comments appear to
           acknowledge this point by stating that the department has begun to
           define and implement a Business Capability Lifecycle concept,
           which is intended to integrate the investment selection and
           acquisition management processes for Tier 1 and enterprise systems
           into a single oversight process that leverages the existing IRB
           and DBSMC oversight framework.
           o DOD partially agreed with our recommendation to define and
           implement policies and procedures that specify how cost, schedule,
           and benefit data are to be used in making certification and annual
           review decisions; define the criteria used to select investments
           as enterprisewide; and establish consistent and effective guidance
           for BEA compliance. In particular, the department agreed that
           additional criteria are required for selecting enterprisewide
           investments, noting that initial criteria have been defined and
           will be incorporated in the investment management process.
           However, the department did not agree that cost, schedule, and BEA
           compliance information are not sufficiently used for certification
           and annual review decisions, adding that such information is
           required in its current policies. We do not agree. Specifically,
           while we do not question whether investment data are provided to
           the DBSMC and the IRBs, the department's policies and procedures
           do not include specific decision criteria that explain how these
           data are to be used to make consistent, repeatable selection and
           reselection decisions across all investments. In addition, while
           BEA compliance policies have been developed and are being used,
           the guidance is not fully defined. For example, the guidance
           allows programs to defer demonstrating full compliance with
           important BEA artifacts until the final phases of the acquisition
           process, at which time addressing instances of noncompliance would
           be more expensive and difficult. Furthermore, the compliance
           criteria are not consistently described in different guidance
           documentation. As a result, DOD risks beginning system production
           and deployment before ensuring that a system is sufficiently
           aligned to the BEA.
           o DOD did not agree with our recommendation to define and
           implement policies and procedures that provide for the reselection
           of investments that are in operations and maintenance. According
           to DOD, components are required by policy to annually review all
           business systems, including investments for which there is no
           planned development or modernization spending. We agree that the
           annual review process does require this. However, consistent with
           our ITIM framework, the corporate investment boards should
           continue to reselect investments that meet the defined threshold
           criteria throughout their life cycles (i.e., when they are in
           operations and maintenance). In contrast, DOD's corporate boards
           focus only on reselecting those investments that are in the
           development/modernization stage.
           o DOD did not agree with our recommendation to define and
           implement policies and procedures that specify how the corporate
           boards use funding information in carrying out decisions on system
           certification and approvals. In this regard, it stated that such
           information is required in its current policies and considered
           during board deliberations. We do not agree. Our recommendation
           does not address whether existing policies or guidance provide for
           the collection of this information; our recommendation addresses
           the definition of policy, guidance, and supporting procedures that
           fall short of satisfying the best practices embodied in our ITIM
           framework. Specifically, while we do not question whether funding
           data are provided to investment decision-making bodies, the
           department's policies and procedures do not include specific
           decision criteria that explain how these data are to be used to
           make consistent, repeatable selection and reselection decisions
           across all investments.
           o DOD did not agree with our recommendation to define and
           implement policies and procedures that provide for sufficient
           oversight and visibility into component-level investment
           management activities. In particular, it stated that this
           recommendation contradicts the department's "tiered
           accountability" approach to investment management. We do not
           agree. Under the department's current policies and guidance, most
           DOD investments are not subject to corporate visibility and
           oversight, either because they do not involve
           development/modernization (i.e., they are in operations and
           maintenance) or because they do not exceed a certain dollar
           threshold. Our framework recognizes that effective implementation
           of a tiered accountability concept should include appropriate
           corporate visibility into and oversight of investments, either
           through review and approval of those investments that meet certain
           criteria or through awareness of a subordinate board's investment
           management activities. Moreover, this visibility and oversight
           should extend to the entire portfolio of investments, including
           those that are in operations and maintenance. To ensure that this
           occurs, applicable policies and procedures need to explicitly
           cover all such investments and need to define how this is to be
           accomplished.

           With respect to our four portfolio-level recommendations, the
           department stated that it partially agreed with three and
           disagreed with one.

           o DOD partially agreed with our recommendation to define and
           implement policies and procedures for creating and modifying
           portfolio selection criteria for business system investments. In
           particular, it stated that while components are responsible for
           developing and managing their own portfolio management processes,
           upcoming initiatives, such as the Business Capability Lifecycle
           concept, will lead to revisions in the department's investment
           review policies and procedures, such as including portfolio
           selection criteria for enterprise systems that span components.
           However, while these are important steps, the concept, as defined
           by the department, does not apply to the thousands of investments
           that are not enterprisewide.
           o DOD partially agreed with our recommendation to define and
           implement policies and procedures that address analyzing,
           selecting, and maintaining business system investment portfolios.
           In particular, it stated that the implementation of the Business
           Capability Lifecyle concept will provide the corporate boards with
           improved visibility into all investments in a given portfolio and
           a broader set of criteria for analyzing, selecting, and
           maintaining business system investment portfolios.
           o DOD partially agreed with our recommendation to define and
           implement policies and procedures that address reviewing,
           evaluating, and improving the performance of its portfolio(s) by
           using cost, schedule, and risk indicators. In particular, it
           stated that while such indicators are part of the investment
           certification and review processes, efforts are now under way to
           better understand the nature and impact of program risks through
           application of an Enterprise Risk Assessment Methodology. While we
           recognize the role and value of such tools in understanding and
           addressing program risks, this tool is program-specific and not
           portfolio-focused.
           o DOD did not agree with our recommendation to define and
           implement policies and procedures that address conducting
           postimplementation reviews and having the corporate investment
           boards consider the review results and develop lessons learned
           from them. In particular, it stated that this process should not
           be managed by the Deputy Secretary of Defense, and also stated
           that our recommendation is redundant with postimplementation
           reviews currently required under OMB Circular A-130.^38 We do not
           agree with DOD's statements. Our recommendation does not call for
           the Deputy Secretary to manage the postimplementation review
           process. Rather, it provides for developing policies and
           procedures for performing postimplementation reviews for all tiers
           of business systems and having the DBSMC and IRBs, which are the
           corporate investment boards, consider the information gathered
           from these reviews and develop lessons learned.

           We are sending copies of this report to interested congressional
           committees; the Director, Office of Management and Budget; the
           Secretary of Defense; the Deputy Secretary of Defense; the Under
           Secretary of Defense for Acquisition, Technology, and Logistics;
           the Under Secretary of Defense (Comptroller); the Assistant
           Secretary of Defense (Networks and Information Integration)/Chief
           Information Officer; the Under Secretary of Defense (Personnel and
           Readiness); and the Director, Defense Finance and Accounting
           Service. Copies of this report will be made available to other
           interested parties upon request. This report will also be
           available at no charge on our Web site at http://www.gao.gov .
		   
^38According to OMB Circular A-130, which establishes policy for the
management of federal information resources, as part of the capital
planning process, an agency must, among other things, conduct
postimplementation reviews of information systems and information resource
management processes to validate estimated benefits and costs; document
effective management practices for broader use; and document lessons
learned from the postimplementation reviews.

           If you or your staffs have any questions on matters discussed in
           this report, please contact me at (202) 512-3439 or [email protected].
           Contact points for our Offices of Congressional Relations and
           Public Affairs may be found on the last page of this report. GAO
           staff who made major contributions to this report are listed in
           appendix III.

           Randolph C. Hite

           Randolph C. Hite
		   Director, Information Technology Architecture and Systems Issues 

           List of Committees

           The Honorable Carl Levin
		   Chairman
		   The Honorable John McCain
           Ranking Member
		   Committee on Armed Services
		   United States Senate

           The Honorable Daniel Inouye
		   Chairman
		   The Honorable Ted Stevens
           Ranking Member
		   Committee on Appropriations
		   United States Senate

           The Honorable Ike Skelton
		   Chairman
		   The Honorable Duncan Hunter
           Ranking Member
		   Committee on Armed Services
		   House of Representatives

           The Honorable John P. Murtha
		   Chairman
		   The Honorable C.W. Bill Young
		   Ranking Member
		   Committee on Appropriations
		   House of Representatives
		   
		   Appendix I: Objective, Scope, and Methodology 

           Our objective was to determine whether the Department of Defense's
           (DOD) corporate investment management approach comports with
           relevant federal guidance. Our analysis was based on the best
           practices contained in GAO's Information Technology Investment
           Management (ITIM) framework, and the framework's associated
           evaluation methodology, and focused on DOD's establishment of
           departmental-level policies and procedures for business system
           investments needed to assist organizations in complying with the
           investment management provisions of the Clinger-Cohen Act of 1996
           (Stages 2 and 3). It did not include case studies to verify the
           implementation of established policies and procedures.

           To address our objective, we asked DOD to complete a
           self-assessment of its corporate investment management process and
           provide the supporting documentation. We then reviewed the results
           of the department's self-assessment of Stages 2 and 3
           organizational commitment practices--meaning those practices
           related to structures, policies, and procedures--and compared them
           against our ITIM framework. We also validated and updated the
           results of the self-assessment through document reviews and
           interviews with officials, such as the Director of Investment
           Management and the Defense Business Systems Acquisition Executive.
           In doing so, we reviewed written policies, procedures, and
           guidance and other documentation providing evidence of executed
           practices, including the Defense Acquisition System guidance, the
           Investment Review Board (IRB) Concept of Operations and Guidance,
           the Business Enterprise Architecture Compliance Guidance, IRB
           charters and meeting minutes, and the Business Transformation
           Guidance.

           We compared the evidence collected from our document reviews and
           interviews with the key practices in ITIM. We rated the key
           practices as "executed" on the basis of whether the agency
           demonstrated (by providing evidence of performance) that it had
           met all of the criteria of the key practice. A key practice was
           rated as "not executed" when we found insufficient evidence of all
           elements of a practice being fully performed or when we determined
           that there were significant weaknesses in DOD's execution of the
           key practice. In addition, we provided DOD with the opportunity to
           produce evidence for the key practices rated as "not executed."

           We conducted our work at DOD headquarters offices in Arlington,
           Virginia, from August 2006 through April 2007 in accordance with
           generally accepted government auditing standards.
		   
		   Appendix II: Comments from the Department of Defense
		   
		   Appendix III: GAO Contact and Staff Acknowledgments
		   
		   GAO Contact

           Randolph C. Hite, (202) 512-3439 or [email protected]
		   
		   Staff Acknowledgments

           In addition to the contact person named above, key contributors to
           this report were Neil Doherty, Nalani Fraser, Nancy Glover,
           Michael Holland, Neelaxi Lakhmani (Assistant Director), Jacqueline
           Mai, Sabine Paul, Niti Tandon, and Jennifer Stavros-Turner.
		   
		   GAO's Mission

           The Government Accountability Office, the audit, evaluation and
           investigative arm of Congress, exists to support Congress in
           meeting its constitutional responsibilities and to help improve
           the performance and accountability of the federal government for
           the American people. GAO examines the use of public funds;
           evaluates federal programs and policies; and provides analyses,
           recommendations, and other assistance to help Congress make
           informed oversight, policy, and funding decisions. GAO's
           commitment to good government is reflected in its core values of
           accountability, integrity, and reliability.
		   
		   Obtaining Copies of GAO Reports and Testimony

           The fastest and easiest way to obtain copies of GAO documents at
           no cost is through GAO's Web site ( www.gao.gov ). Each
           weekday, GAO posts newly released reports, testimony, and
           correspondence on its Web site. To have GAO e-mail you a list of
           newly posted products every afternoon, go to www.gao.gov and
           select "Subscribe to Updates."
		   
		   Order by Mail or Phone

           The first copy of each printed report is free. Additional copies
           are $2 each. A check or money order should be made out to the
           Superintendent of Documents. GAO also accepts VISA and Mastercard.
           Orders for 100 or more copies mailed to a single address are
           discounted 25 percent. Orders should be sent to:

           U.S. Government Accountability Office 441 G Street NW, Room LM
           Washington, D.C. 20548

           To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
           (202) 512-6061
		   
		   To Report Fraud, Waste, and Abuse in Federal Programs

           Contact:

           Web site: www.gao.gov/fraudnet/fraudnet.htm
		   E-mail: [email protected]
		   Automated answering system: (800) 424-5454 or (202) 512-7470
		   
		   Congressional Relations

           Gloria Jarmon, Managing Director, [email protected] (202)
           512-4400 U.S. Government Accountability Office, 441 G Street NW,
           Room 7125 Washington, D.C. 20548
		   
		   Public Affairs

           Paul Anderson, Managing Director, [email protected] (202)
           512-4800 U.S. Government Accountability Office, 441 G Street NW,
           Room 7149 Washington, D.C. 20548

(310636)

www.gao.gov/cgi-bin/getrpt?GAO-07-538 .

To view the full product, including the scope
and methodology, click on the link above.

For more information, contact Randolph C. Hite at (202) 512-3439 or
[email protected].

Highlights of [56]GAO-07-538 , a report to congressional committees

May 2007

BUSINESS SYSTEMS MODERNIZATION

DOD Needs to Fully Define Policies and Procedures for Institutionally
Managing Investments

In 1995, GAO first designated the Department of Defense's (DOD) business
systems modernization program as "high-risk," and continues to do so
today. In 2004, Congress passed legislation reflecting prior GAO
recommendations for DOD to adopt a corporate approach to information
technology (IT) business system investment management. To support GAO's
legislative mandate to review DOD's efforts, GAO assessed whether the
department's corporate investment management approach comports with
relevant federal guidance. In doing so, GAO applied its IT Investment
Management framework and associated methodology, focusing on the
framework's stages related to the investment management provisions of the
Clinger-Cohen Act of 1996.

[57]What GAO Recommends

GAO recommends that DOD fully define the project and portfolio management
policies and procedures discussed in GAO's framework. DOD agreed with
GAO's overall conclusions and partially agreed with five of GAO's
recommendations. However, DOD disagreed with the remaining four
recommendations, stating that the department is, among other things,
already meeting the intent of these recommendations. GAO does not agree;
its recommendations focus on fully defining policies and procedures that
satisfy key practices in its framework.

DOD has established the management structures needed to effectively manage
its business system investments, but it has not fully defined many of the
related policies and procedures that GAO's IT Investment Management
framework defines. Specifically, the department has defined four of nine
practices that call for project-level policies and procedures, and one of
the five practices that call for portfolio-level policies and procedures
(see below). For example, DOD has established an enterprisewide IT
investment board responsible for defining and implementing its business
system investment governance process, documented policies and procedures
for ensuring that systems support ongoing and future business needs,
developed procedures for identifying and collecting information about
these systems to support investment selection and control, and assigned
responsibility to an individual or a group for managing the development
and modification of the business system portfolio selection criteria.
However, DOD has not fully documented business system investment policies
and procedures for directing investment board operations, selecting new
investments, reselecting ongoing investments, integrating the investment
funding and the investment selection processes, and developing and
maintaining a complete business system investment portfolio(s).

Regarding project-level investment management practices, DOD officials
said that these are performed at the component level, and that
departmental policies and procedures established for overseeing
components' execution of these practices are sufficient. For
portfolio-level practices, however, these officials stated that they
intend to improve departmental policies and procedures for business system
investments by, for example, establishing a single governance structure,
but plans or time frames for doing so have not been established. Until DOD
fully defines departmentwide policies and procedures for both individual
projects and portfolios of projects, it risks selecting and controlling
these business system investments in an inconsistent, incomplete, and ad
hoc manner, which in turn reduces the chances that these investments will
meet mission needs in the most cost-effective manner.

Policies and Procedures for Project-Level and Portfolio-Level Management

Source: GAO.

References

Visible links
  25. http://www.gao.gov/cgi-bin/getrpt?GAO-07-310
  26. http://www.gao.gov/cgi-bin/getrpt?GAO-01-525
  27. http://www.gao.gov/cgi-bin/getrpt?GAO-05-702
  28. http://www.gao.gov/cgi-bin/getrpt?GAO-05-381
  29. http://www.gao.gov/cgi-bin/getrpt?GAO-04-731R
  30. http://www.gao.gov/cgi-bin/getrpt?GAO-03-1018
  31. http://www.gao.gov/cgi-bin/getrpt?GAO-03-877R
  32. http://www.gao.gov/cgi-bin/getrpt?GAO-03-571R
  33. http://www.gao.gov/cgi-bin/getrpt?GAO-03-458
  34. http://www.gao.gov/cgi-bin/getrpt?GAO-01-525
  35. http://www.gao.gov/cgi-bin/getrpt?GAO-07-229T
  36. http://www.gao.gov/cgi-bin/getrpt?GAO-06-658
  37. http://www.gao.gov/cgi-bin/getrpt?GAO-06-219
  38. http://www.gao.gov/cgi-bin/getrpt?GAO-04-394G
  39. http://www.gao.gov/cgi-bin/getrpt?GAO-04-394G
  40. http://www.gao.gov/cgi-bin/getrpt?GAO-06-658
  41. http://www.gao.gov/cgi-bin/getrpt?GAO-04-576
  42. http://www.gao.gov/cgi-bin/getrpt?GAO-04-89
  43. http://www.gao.gov/cgi-bin/getrpt?GAO-03-887
  44. http://www.gao.gov/cgi-bin/getrpt?GAO-07-310
  45. http://www.gao.gov/cgi-bin/getrpt?GAO-05-276
  46. http://www.gao.gov/cgi-bin/getrpt?GAO-04-394G
  47. http://www.gao.gov/cgi-bin/getrpt?GAO-03-584G
  56. http://www.gao.gov/cgi-bin/getrpt?GAO-07-538
*** End of document. ***