Information Technology: DLA Needs to Strengthen Its Investment
Management Capability (15-MAR-02, GAO-02-314).
The Defense Logistics Agency (DLA) relies extensively on
information technology (IT) to carry out its logistics support
mission. This report focuses on DLA's processes for making
informed IT investment decisions. Because IT investment
management has only recently become an area of management focus
and commitment at DLA, the agency's ability to effectively manage
IT investments is limited. The first step toward establishing
effective investment management is putting in place foundational,
project-level control and selection processes. The second step
toward effective investment management is to continually assess
proposed and ongoing projects as an integrated and competing set
of investment options. Accomplishing these two steps requires
effective development and implementation of a plan, supported by
senior management, which defines and prioritizes investment
process improvements. Without a well-defined process improvement
plan and controls for implementing it, it is unlikely that the
agency will establish a mature investment management capability.
As a result, GAO continues to question DLA's ability to make
informed and prudent investment decisions in IT.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-02-314
ACCNO: A02900
TITLE: Information Technology: DLA Needs to Strengthen Its
Investment Management Capability
DATE: 03/15/2002
SUBJECT: Civilian employees
Information technology
Investment planning
Military forces
Military personnel
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-02-314
United States General Accounting Office
GAO
Report to Congressional Committees
March 2002
INFORMATION
TECHNOLOGY
DLA Needs to Strengthen Its Investment Management Capability
GAO-02-314
Contents
Letter
Results in Brief
Background
Scope and Methodology
DLA's Capabilities to Effectively Manage IT Investments Are
Limited DLA Lacks a Plan to Guide Improvement Efforts Conclusions
Recommendations Agency Comments and Our Evaluation 1
2 3 7
9 23 24 24 25
Appendix I GAO Contact and Staff Acknowledgments 28
GAO Contact 28 Acknowledgments 28
Tables
Table 1: Stage 2 Critical Processes: Building the Investment
Foundation Table 2: Status of Stage 2 Critical Processes Table 3: IT
Investment Board Operation Table 4: IT Project Identification Table 5: IT
Project Oversight Table 6: Business Needs Identification for IT Projects
Table 7: Proposal Selection Table 8: Stage 3 Critical Processes 10 11 13 15
17 19 21 22
Figure
Figure 1: The Five Stages of Maturity within ITIM
United States General Accounting Office Washington, DC 20548
March 15, 2002
The Honorable Carl Levin
Chairman
The Honorable John Warner
Ranking Minority Member
Committee on Armed Services
United States Senate
The Honorable Bob Stump
Chairman
The Honorable Ike Skelton
Ranking Minority Member
Committee on Armed Services
House of Representatives
The Defense Logistics Agency (DLA) plays a critical role in supporting
America's military forces worldwide. To fulfill this role, DLA employs
about 28,000 civilian and military workers, located at about 500 sites in
all
50 states and in 28 countries; in round numbers, it manages 4 million
supply items and processes 30 million annual supply distribution actions.
In fiscal year 2001, DLA reported that these operations resulted in sales to
the military services of about $15.2 billion. To perform its logistics
support
mission, the agency relies extensively on information technology (IT). For
fiscal year 2002, DLA's IT budget is about $654 million.
This report is one in a series of products to respond to the fiscal year
2001
Defense Authorization Act.1 That act directs that GAO review DLA's
efficiency and effectiveness in meeting customer requirements, application
of best business practices, and opportunities for improving agency
operations.
This report focuses on DLA's processes for making informed IT
investment decisions. As agreed with your offices, our objectives were to
determine (1) whether DLA has effective IT investment management
processes needed to modernize and maintain systems and (2) what actions
the agency has planned to improve these processes.
1P.L. 106-398, Floyd D. Spence National Defense Authorization Act for Fiscal
Year 2001, section 917.
Results in Brief
Because IT investment management has only recently become an area of
management focus and commitment at DLA, the agency's capability to
effectively manage its IT investments is limited. DLA has recognized the
need to strengthen its processes in this area. For example, the agency
recently began introducing basic project selection and control activities
into its longstanding budget-driven, decisionmaking process. Nevertheless,
DLA has much more to accomplish. Until DLA fully implements an effective IT
investment management process, it will not know whether its mix of
investments best meets its mission and business priorities.
* The first step toward establishing effective investment management is
putting in place foundational, project-level control and selection
processes. These foundational processes allow the agency to identify
variances in project cost, schedule, and performance expectations; to take
corrective action, if appropriate; and to make informed, project-specific
selection decisions. Although DLA has made progress toward establishing such
foundational processes, key practices still need to be implemented. For
example, the business needs for IT projects are not always clearly
identified and defined, an IT investment selection process has not been
fully implemented, and policies and procedures for project oversight are not
documented. With such weaknesses, executives cannot be assured that they are
consistently selecting and managing IT investments that meet agency needs
and priorities.
* The second major step toward effective investment management is to
continually assess proposed and ongoing projects as an integrated and
competing set of investment options. This portfolio management approach
enables the organization to consider the relative costs, benefits, and risks
of new and previously funded investments and thereby identify the mix that
best meets its mission, strategies, and goals. DLA officials acknowledge
that the agency has not implemented the processes associated with managing
investments as a complete portfolio (that is, an integrated, enterprisewide
collection of investments). As a result, DLA executives are unable to
adequately assess the relative merits of investment proposals and make
trade-offs among options.
Accomplishing these two major steps requires effective development and
implementation of a plan, supported by senior management, which defines and
prioritizes investment process improvements. DLA does not have such a plan.
Without a well-defined process improvement plan and controls for
implementing it, it is unlikely that the agency will establish a mature
investment management capability. As a result, DLA will continue to be
challenged in its ability to make informed and prudent investment decisions
in managing its annual multimillion dollar IT budget.
Background
To strengthen DLA's investment management capability, we are recommending
that DLA implement and develop a plan aimed at addressing the weaknesses
discussed in this report.
In commenting on a draft of this report, DOD concurred with our
recommendations and described efforts under way and planned to implement
them. Additionally, it recommended that two report captions be changed to
more accurately reflect, in DOD's view, the contents of the report and to
eliminate false impressions. We do not agree. Both of the captions cited are
fully consistent with the evidence presented in the report, and thus are
completely accurate and appropriate.
DLA is DOD's logistics manager for all departmental consumable items and
some repair parts. Its primary business function is materiel management:
providing supply support to sustain military operations and readiness. In
addition, DLA performs five other supply-related business functions:
distributing materiel from DLA and service-owned inventories, purchasing
fuels for DOD and the U.S. government, storing strategic materiel, marketing
surplus DOD materiel for reuse and disposal, and providing numerous
information services, such as item cataloging, for DOD and the U.S.
government, as well as selected foreign governments. These six business
functions are managed by field commands that report to and support the
agency's central command authority.
In 2000, DLA refocused its logistics mission from that of a supplier of
materiel to a manager of supply chain relationships. To support this
transition, the agency developed a strategic plan (known as DLA 21) to
reengineer and modernize its operations.2 Among the goals of DLA 21 are to
optimize inventories, improve efficiency, increase effectiveness through
organizational redesign, reduce inventories, and modernize business systems.
DLA relies on over 650 systems to support warfighters by allowing access to
global inventories. Whether it is ensuring that there is enough fuel to
service an aircraft fleet, providing sufficient medical supplies to protect
and treat military personnel, or supplying ample food rations to our
soldiers on the frontlines, information technology plays a key role in
ensuring that Defense Department agencies are prepared for their
2Defense Logistics Agency, Strategic Plan 2000: DLA 21 (September 1999).
missions. Because of its heavy reliance on IT to accomplish its mission, DLA
invests extensively in this area. For fiscal year 2002, DLA's IT budget is
about $654 million.
Prior Reviews Identified IT Management Weaknesses
Our recent reviews of DLA's IT management have identified weaknesses in such
important areas as enterprise architecture management, incremental
investment management, and software acquisition management.
In June 2001, we reported that DLA did not have an enterprise architecture
to guide the agency's investment in its Business Systems Modernization (BSM)
project-the agency's largest IT project.3 The use of an enterprise
architecture, which describes an organization's mode of operation in useful
models, diagrams, and narrative, is required by the OMB guidance that
implements the Clinger-Cohen Act of 19964 and is a commercial best practice.
Such a "blueprint" can help clarify and optimize the dependencies and
relationships among an agency's business operations and the IT
infrastructure and applications supporting them. An effective architecture
describes both the environment as it is and the target environment that an
organization is aiming for (as well as a plan for the transition from one to
the other). We concluded that without this architecture, DLA will be
challenged in its efforts to successfully acquire and implement BSM.
Further, we reported that DLA was not managing its investment in BSM in an
incremental manner, as required by the Clinger-Cohen Act of 1996 and OMB
guidance and in accordance with best commercial practices. An incremental
approach to investment helps to minimize the risk associated with such
large-scale projects as BSM. Accordingly, we recommended that DLA make the
development, implementation, and maintenance of an enterprise architecture
an agency priority and take steps to incrementally justify and validate its
investment in BSM. According to DLA officials, the agency is addressing
these issues.
3Further details on BSM are provided in the section on scope and
methodology.
4The Clinger-Cohen Act of 1996 was enacted to address longstanding problems
related to federal IT management. Among other things, it requires agency
heads to implement a process for maximizing the value and assessing and
managing the risks of its acquisitions. A key goal of the Clinger-Cohen Act
is that agencies have processes and information in place to help ensure that
IT projects are being implemented at acceptable costs, within reasonable and
expected time frames, and are contributing to tangible, observable
improvements in mission performance.
In January 2002, we reported a wide disparity in the rigor and discipline of
software acquisition processes between two DLA systems.5 Such inconsistency
in processes for acquiring software (the most costly and complex component
of systems) can lead to the acquisition of systems that do not meet the
information needs of management and staff, do not provide support for
necessary programs and operations, and cost more and take longer than
expected to complete.
We also reported that DLA did not have a software process-improvement
program in place to effectively strengthen its corporate software
acquisition processes, having eliminated the program in 1998. Without a
management-supported software process-improvement program, it is unlikely
that DLA can effectively improve its institutional software acquisition
capabilities, which in turn means that the agency's software projects will
be at risk of not delivering promised capabilities on time and within
budget. Accordingly, we recommended that DLA institute a software
process-improvement program and correct the software acquisition process
weaknesses that we identified. According to DLA officials, the agency is
addressing each of these issues.
Information Technology Investment Management (ITIM) Maturity Framework
In May 2000, we issued the Information Technology Investment Management
(ITIM) maturity framework,6 which identifies critical processes for
successful IT investment and organizes these processes into an assessment
framework comprising five stages of maturity. This framework supports the
fundamental requirements of the Clinger-Cohen Act of 1996, which requires IT
investment and capital planning processes and performance measurement.
Additionally, ITIM can provide a useful roadmap for agencies when they are
implementing specific, fundamental IT capital planning and investment
management practices. The federal Chief Information Officers Council has
favorably reviewed the framework, and it is also being used by a number of
executive agencies and organizations for designing related policies and
procedures and self-led or contractor-based assessments.
5U.S. General Accounting Office, Information Technology: Inconsistent
Software Acquisition Processes at the Defense Logistics Agency Increase
Project Risks, GAO-02-9 (Washington, D.C.: January 10, 2002).
6U.S. General Accounting Office, Information Technology Investment
Management: A Framework for Assessing and Improving Process Maturity,
Exposure Draft, GAO/AIMD-10-1.23 (Washington, D.C.: May 2000).
ITIM establishes a hierarchical set of five different maturity stages. Each
stage builds upon the lower stages and represents increased capabilities
toward achieving both stable and effective (and thus mature) IT investment
management processes. Except for the first stage-which largely reflects ad
hoc, undefined, and undisciplined decision and oversight processes-each
maturity stage is composed of critical processes essential to satisfy the
requirements of that stage. These critical processes are defined by core
elements that include organizational commitment (for example, policies and
procedures), prerequisites (for example, resource allocation), and
activities (for example, implementing procedures). Each core element is
composed of a number of key practices. Key practices are the specific tasks
and conditions that must be in place for an organization to effectively
implement the necessary critical processes.
Figure 1 shows the five ITIM stages and a brief description of each stage.
Figure 1: The Five Stages of Maturity within ITIM
Source: GAO.
Scope and Methodology
Using ITIM, we assessed the extent to which DLA satisfied the five critical
processes in stage 2 of the framework. Based on DLA's acknowledgment that it
had not executed any of the key practices in stage 3, we did not
independently assess the agency's capabilities in this stage or stages 4 and
5. To determine whether DLA had implemented the stage 2 critical processes,
we compared relevant DLA policies, procedures, guidance, and documentation
associated with investment management activities to the key practices and
critical processes in ITIM. We rated the key practices as "executed" based
on whether the agency demonstrated (by providing evidence of performance)
that it had met the criteria of the key practice. A key practice was rated
as "not executed" when we found insufficient evidence of a practice during
the review, or when we determined that there were significant weaknesses in
DLA's execution of the key practice.
As part of our analysis, we selected four IT projects as case studies to
verify application of the critical processes and practices. We selected
projects that (1) supported different DLA business areas (such as materiel
management), (2) were in different lifecycle phases (for example,
requirements definition, design, operations and maintenance), (3)
represented different levels of risk (such as low or medium) as designated
by the agency, and (4) included at least one investment that required
funding approval by a DOD authority outside of DLA (for example, the Office
of the Secretary of Defense (OSD)). The four projects are the following:
* Business Systems Modernization: This system, which supports DLA's materiel
management business area, is in the concept demonstration phase of
development. DLA reported that it spent about $136 million on this system in
fiscal year 2001, and it has budgeted about $133 million for fiscal year
2002. BSM is intended to modernize DLA's materiel management business
function, replacing two of its standard systems (the Standard Automated
Materiel Management System and the Defense Integrated Subsistence Management
System). The project is also intended to enable the agency to reengineer its
logistics practices to reflect best commercial business practices. For
example, in support of DLA's goal of reducing its role as a provider and
manager of materiel and increasing its role as a manager of supply chain
relationships, BSM is to help link customers with appropriate suppliers and
to incorporate commercial business practices regarding physical distribution
and financial management. The agency has classified this project as high
risk, and OSD has funding approval authority for this project.
* Hazardous Materials Information System (HMIS): This system, which supports
DLA's logistics operations function, was implemented in 1978. In
fiscal year 2001, DLA reported that it spent about $1 million on this system
and budgeted about $2.4 million for fiscal year 2002. In 1999 DLA began a
redesign effort to transform HMIS into a Web-based system with a direct
interface to the manufacturers and suppliers of hazardous material. The
project is in the development stage. It contains data on the chemical
composition of materials classified as "hazardous" for the purposes of
usage, storage, and transportation. The system is used by Emergency Response
Teams whenever a spill or accident occurs involving hazardous materials. The
agency classified this project as low risk, and funding approval occurs
within DLA.
* The Defense Reutilization and Marketing Automated Information System
(DAISY): This system, which supports DLA's materiel reuse and disposal
mission, is in the operations and maintenance lifecycle phase. The agency
reported that it spent approximately $4.4 million on DAISY in fiscal year
2001, and it has budgeted about $7 million for fiscal year 2002. This system
is a repository for transactions involving the reutilization, transfer,
donation, sale, or ultimate disposal of excess personal property from DOD,
federal, and state agencies. The excess property includes spare and repair
parts, scrap and recyclable material, precious metals recovery, hazardous
material, and hazardous waste disposal. Operated by the Defense
Reutilization and Marketing Service, the system is used at 190 locations
worldwide. The agency classified this project as low risk, and funding
approval occurs within DLA.
* Standard Automated Materiel Management System (SAMMS): This system, which
supports DLA's materiel management business area, is 30 years old and
approaching the end of its useful life. The agency reports that investment
in SAMMS (budgeted at approximately $19 million for fiscal year 2002) is
directed toward keeping the system operating until its replacement, BSM,
becomes fully operational (scheduled for fiscal year 2005). This system
provides the Inventory Control Points with information regarding stock
levels, as well as with the capabilities required for (1) acquisition and
management of wholesale consumable items, (2) direct support for processing
requisitions, (3) forecasting of requirements, (4) generation of purchase
requests, (5) maintenance of technical data, (6) financial management, (7)
identification of items, and (8) asset visibility. The agency has classified
the maintenance of SAMMS as a low risk effort, and funding approval occurs
within DLA.
For these projects, we reviewed project management documentation, such as
mission needs statements, project plans, and status reports. We also
analyzed charters and meeting minutes for DLA oversight boards, DLA's draft
Automated Information System Emerging Program Life Management (LCM) Review
and Milestone Approval Directive and Portfolio
Management and Oversight Directives, and DOD's 5000 series guidance on
systems acquisition.7 In addition, we reviewed documentation related to the
agency's self-assessment of its IT investment operations.
To supplement our document reviews, we interviewed senior DLA officials,
including the vice director (who sits on the Corporate Board, DLA's highest
level investment decisionmaking body), the chief information officer (CIO),
the chief financial officer, and oversight board members. We also
interviewed the program managers of our four case study projects, as well as
officials responsible for managing the IT investment process 8 and other
staff within Information Operations.
To determine what actions DLA has taken to improve its IT investment
management processes, we interviewed the CIO and officials of the Policy,
Plans, and Assessments and the program executive officer (PEO) operations
groups within the Information Operations Directorate. These groups are
primarily responsible for implementing investment management process
improvements. We also reviewed a draft list of IT investment management
improvement tasks.
We conducted our work at DLA headquarters in Fort Belvoir, Virginia, from
June 2001 through January 2002, in accordance with generally accepted
government auditing standards.
In order to have the capabilities to effectively manage IT investments, an
agency should (1) have basic, project-level control and selection practices
in place and (2) manage its projects as a portfolio of investments, treating
them as an integrated package of competing investment options and pursuing
those that best meet the strategic goals, objectives, and mission of the
agency.
7The 5000 series on systems acquisition includes three directives: The
Defense Acquisition System, 5000.1 (October 23, 2000); Operation of the
Defense Acquisition System, 5000.2 (January 4, 2001); and Mandatory
Procedures for Major Defense Acquisition Programs and Major Automated
Information System Acquisition Programs, 5000.2-R (June 2001).
8These officials include the enterprise portfolio manager, portfolio
managers, and staff of the program executive officer. The enterprise
portfolio manager and the individual portfolio managers are responsible for
overseeing and managing DLA's various groups of systems, which the agency
refers to as "portfolios." The PEO is the senior official who has
responsibility for directing major IT acquisition programs.
DLA's Capabilities to Effectively Manage IT Investments Are Limited
DLA has a majority of the project-level practices in place. However, it is
missing several crucial practices, and it is not performing portfolio-based
investment management. According to the CIO, the evolving state of its
investment management capabilities is the result of agency leadership's
recently viewing IT investment management as an area of management focus and
priority. Without having crucial processes and related practices in place,
DLA lacks essential management controls over its sizable IT investments.
Many Stage 2 Strengths, but Key Weaknesses Exist in Project-Level Control
and Selection Processes
At ITIM stage 2 maturity, an organization has attained repeatable,
successful IT project-level investment control processes and basic selection
processes. Through these processes, the organization can identify
expectation gaps early and take appropriate steps to address them. According
to ITIM, critical processes at stage 2 include (1) defining investment
board9 operations, (2) collecting information about existing investments,
(3) developing project-level investment control processes, (4) identifying
the business needs for each IT project, and (5) developing a basic process
for selecting new IT proposals. Table 1 discusses the purpose for each of
the stage 2 critical processes.
Table 1: Stage 2 Critical Processes: Building the Investment Foundation
Critical process Purpose
IT investment board To define and establish the governing board(s)
responsible operation for selecting, controlling, and evaluating IT
investments
IT project identification To create and maintain an IT project inventory to
assist in managerial decisionmaking
IT project oversight To regularly determine each IT project's progress
toward cost and schedule milestones, using established criteria, and to take
corrective actions when milestones are not achieved
Business needs To ensure that each IT project supports the organization's
identification for IT projects business needs and meets users' needs
Proposal selection To ensure that an established, structured process is used
to
select new IT proposals
Source: GAO.
To its credit, DLA has put in place about 75 percent of the key practices
associated with stage 2 critical processes. For example, DLA has oversight
9An investment board is a decisionmaking body made up of senior program,
financial, and information managers that is responsible for making decisions
about investments or projects.
boards to perform investment management functions, and it has basic
project-level control processes to help ensure that IT projects are meeting
cost and schedule expectations.
However, DLA has not executed several crucial stage 2 investment practices.
For example, the business needs for IT projects are not always clearly
identified and defined, basic investment selection processes are still being
developed, and policies and procedures for project oversight are not
documented. Table 2 summarizes the status of DLA's stage 2 critical
processes, showing how many associated key practices the agency has
executed. DLA's actions in each of the critical processes are discussed in
the sections that follow.
Table 2: Status of Stage 2 Critical Processes Critical process
Key practices executed
Key practices not executed Overall total
1. IT investment board operation 4 2
2. IT project identification 5 2
3. IT project oversight 9 2
4. Business needs identification 7 1
5. Proposal selection 4 2
Totals 29 9
Boards Established, but Policies and Procedures to Guide Board Operations
Are Lacking
Source: GAO.
To help ensure executive management accountability for IT capital planning
and investment decisions, an organization should establish a governing board
or boards responsible for selecting, controlling, and evaluating IT
investments. According to ITIM, effective IT investment board operations
require, among other things, that (1) board membership have both IT and
business knowledge, (2) board members understand the investment board's
policies and procedures and exhibit core competencies in using the agency's
IT investment policies and procedures, (3) the organization's executives and
line managers support and carry out board decisions, (4) the organization
create organization-specific process guidance that includes policies and
procedures to direct the board's operations, and (5) the investment board
operate according to written policies and procedures. (The full list of key
practices is provided in table 3.)
DLA has established several oversight boards that perform IT investment
management functions.10 These boards include the following:
* The DLA Investment Council, which is intended to review, evaluate, and
approve new IT and non-IT investments between $100,000 and $1,000,000.
* The Program Executive Officer Review Board, which is intended to review
and approve the implementation of IT investments that are budgeted for over
$25 million in all or over $5 million in any one year.
* The Corporate Board, which is intended to review, evaluate, and approve
all IT and non-IT investments over $1 million.
DLA is executing four of the six key practices needed for these boards to
operate effectively.11 For example, the membership of these boards
integrates both IT and business knowledge. In addition, board members
informed us of their understanding of their board's informal practices.
Further, according to IT investment officials, project managers, and agency
documentation, the boards have a process for ensuring that their decisions
are supported and carried out by organization executives and line managers.
This process involves documenting board decisions in meeting minutes,
assigning staff to carry out the decisions, and tracking the actions taken
on a regular basis until the issues are addressed.
Nonetheless, DLA is missing the key ingredient associated with two of the
board oversight practices that are needed to operate effectively-
organization-specific guidance. This guidance, which serves as official
operations documentation, should (1) clearly define the roles of key people
within its IT investment process, (2) delineate the significant events and
decision points within the processes, (3) identify the external and
environmental factors that will influence the processes (that is, legal
constraints, the behavior of key subordinate agencies and military
customers, and the practices of commercial logistics that DLA is trying to
emulate as part of DLA 21); and (4) explain how IT investment-related
processes will be coordinated with other organizational plans and processes.
DLA does not have guidance that sufficiently addresses these issues.
Policies and procedures governing operations are in draft for one board and
have not been developed for the two other boards. Without this guidance
governing the operations of the investment boards, the agency is
10We did not address the alignment of these boards, as this is a stage 3
critical process.
11Because the DLA Investment Council was only recently established and is
still defining its role, we did not talk to members about their
implementation of stage 2 critical processes.
at risk of performing key investment decisionmaking activities
inconsistently. Such guidance would also provide a degree of transparency
that is helpful in both communicating and demonstrating how these decisions
are made.
Table 3 summarizes the ratings for each key practice and the specific
findings supporting the ratings.
Table 3: IT Investment Board Operation Type of process Process Rating
Summary of evidence
IT Project Inventory Established, but Not Maintained According to Policies
and Procedures
Source: GAO.
An IT project inventory provides information to investment decision-makers
to help evaluate the impacts and opportunities created by proposed or
continuing investments. This inventory (which can take many forms) should,
at a minimum, identify the organization's IT projects (including new and
existing systems) and a defined set of relevant investment management
information about them (for example, purpose,
owner, lifecycle stage, budget cost, physical location, and interfaces with
other systems). Information from the IT project inventory can, for example,
help identify systems across the organization that provide similar functions
and help avoid the commitment of additional funds for redundant systems and
processes. It can also help determine more precise development and
enhancement costs by informing decisionmakers and other managers of
interdependencies among systems and how potential changes in one system can
affect the performance of other systems.
According to ITIM, effectively managing an IT project inventory requires,
among other things, (1) identifying IT projects, collecting relevant
information about them, and capturing this information in a repository, (2)
assigning responsibility for managing the IT project inventory process to
ensure that the inventory meets the needs of the investment management
process, (3) developing written policies and procedures for maintaining the
IT project inventory, (4) making information from the inventory available to
staff and managers throughout the organization so they can use it, for
example, to build business cases and to support project selection and
control activities, and (5) maintaining the IT project inventory and its
information records to contribute to future investment selections and
assessments. (The full list of key practices is provided in table 4.)
DLA has executed many of the key practices in this critical process. For
example, according to DLA's CIO, IT projects are identified and specific
information about them is entered into a central repository called the DLA
Profile System (DPS). DPS includes, among other things, project
descriptions, key contact information, lifecycle stage, and system
interfaces. In addition, the CIO is responsible for managing the IT project
identification process to ensure that DPS meets the needs of the investment
management process. However, DLA has not defined written policies and
procedures for how and when users should add to or update information in the
DPS. In addition, DLA is not maintaining DPS records, which would be useful
during future project selections and investment evaluations, and for
documenting the evolution of a project's development. Without appropriate
policies and procedures in place to describe the objectives and information
requirements of the inventory, DPS is not being maximized as an effective
tool to assist in the fundamental analysis essential to effective
decisionmaking.
Table 4 summarizes the ratings for each key practice and the specific
findings supporting the ratings.
Table 4: IT Project Identification
Type of process Process Rating Summary of evidence
Prerequisites 1. Adequate resources are provided for Executed According to
the officials responsible for managing DPS, identifying IT projects and
collecting adequate resources are provided for identifying IT projects
relevant information into an inventory. and collecting relevant information
into an inventory.
Key Project Oversight Practices Occurring, but IT Inventory Not Being Used
Source: GAO.
Investment review boards should effectively oversee IT projects throughout
all lifecycle phases (concept, design, development, testing, implementation,
and operations/maintenance). At stage 2 maturity, investment review boards
should review each project's progress toward predefined cost and schedule
expectations, using established criteria and performance measures, and
should take corrective actions to address cost and milestone variances.
According to ITIM, effective project oversight requires, among other things,
(1) having written polices and procedures for project management, (2)
developing and maintaining an approved management plan for each IT project,
(3) having written policies and procedures for oversight of IT projects, (4)
making up-to-date cost and schedule data for each project available to the
oversight boards, (5) reviewing each project's performance by regularly
comparing actual cost and schedule data to expectations, (6) ensuring that
corrective actions for each under-
performing project are documented, agreed to, implemented, and tracked until
the desired outcome is achieved, and (7) using information from the IT
project inventory. (The complete list of key practices is provided in table
5.)
DLA has executed most of the key practices in this area. In particular, DLA
relies on the guidance in the Department of Defense 5000 series directives
for project management and draft guidance in an Automated Information System
(AIS) Emerging Program Life-Cycle Management (LCM) Review and Milestone
Approval Directive for specific IT project management. In addition, for each
of the four projects we reviewed, a project management plan had been
approved, and cost and schedule controls were addressed during project
review meetings. Further, based on our review of project documentation and
in discussion with project managers, up-to-date cost and schedule project
data were provided to the PEO Review Board. This board oversees project
performance regularly by comparing actual cost and schedule data to
expectations and has a process for ensuring that, for underperforming
projects, corrective actions are documented, agreed to, and tracked.
Notwithstanding these strengths, DLA has some weaknesses in project
oversight. Specifically, although the Corporate Board and the Investment
Council have written charters, there are no written policies or procedures
that define their role in collectively overseeing IT projects. Without these
policies and procedures, project oversight may be inconsistently applied,
leading to the risk that performance problems, such as cost overruns and
schedule slippages, may not be identified and resolved in a timely manner.
In addition, according to representatives from the oversight boards, they do
not use information from the IT project inventory to oversee projects
because they are more comfortable using more traditional methods of
obtaining and using information (that is, informally talking with subject
matter experts and relying on experience). The inventory is of value only to
the extent that decisionmakers use it. As discussed earlier, while the
inventory need not be the only source of information, it should nevertheless
serve as a reliable and consistent tool for understanding project and
overall portfolio decisions.
Table 5 summarizes the ratings for each key practice and the specific
findings supporting the ratings.
Table 5: IT Project Oversight
Type of
process Process Rating Summary of evidence
2. Using established criteria, the IT Executed Based on our review of status
briefings, the PEO Review investment board oversees each IT Board, which is
responsible for overseeing IT project project's performance monthly and
performance, does so regularly by comparing actual cost and quarterly by
comparing actual cost schedule data to expectations using established
criteria. and schedule data to expectations.
3. The IT investment board Executed DOD directives and DLA's draft Automated
Information performs special reviews of System (AIS) Emerging Program
Life-cycle Management projects that have not met (LCM) Process, Review, and
Milestone Approval Directive predetermined performance specify when special
reviews are to occur. According to IT standards. investment officials, DLA
has not had to perform any of these
reviews. Based on our review of project documentation, special reviews were
not necessary for the four projects we reviewed.
Type of
process Process Rating Summary of evidence
4. Appropriate corrective actions Executed According to IT investment
officials and agency
for each underperforming project documentation, DLA has a process for
ensuring that
are defined, documented, and corrective actions for underperforming projects
are defined,
agreed to by the IT investment documented, and agreed to by the IT
investment board and
board and the project manager. the project manager.
5. Corrective actions are implemented and tracked until the desired outcome
is achieved.
Executed According to IT investment officials and agency documentation, DLA
has a process for ensuring that corrective actions are implemented and
tracked until the desired outcome is achieved.
Practices to Identify Business Needs Are Occurring with Most, but Not All,
Projects
Source: GAO.
Defining business needs for each IT project helps ensure that projects
support the organization's mission goals and meets users' needs. This
critical process creates the link between the organization's business
objectives and its IT management strategy. According to ITIM, effectively
identifying business needs requires, among other things, (1) defining the
organization's business needs or stated mission goals, (2) identifying users
for each project who will participate in the project's development and
implementation, (3) training IT staff adequately in identifying business
needs, and (4) defining business needs for each project. (The complete list
of key practices is provided in table 6.)
DLA has executed all but one of the key practices associated with
effectively defining business needs for IT projects. For example, DLA's
mission goals are described in DLA's strategic plan. In addition, according
to IT investment management officials, the IT staff is adequately trained in
identifying business needs because they generally have prior functional unit
experience. In addition, according to DLA directives, IT projects are
assigned an Integrated Process Team (IPT) to guide and direct the project
through the development lifecycle. The IPTs are composed of IT and
functional staff. Moreover, DOD and DLA directives require that business
requirements and system users be identified and that users participate in
the lifecycle management of the project. According to an IT investment
official, each IT project has a users' group that meets throughout the
lifecycle to discuss problems and potential changes related to the system.
We verified that this was the case for the four projects we reviewed.
While the business needs for three of the four projects we reviewed were
clearly identified and defined, 12 DLA has reported that this has not been
12DLA has determined that the fourth project we reviewed-SAMMS-is no longer
meeting the agency's business needs.
consistently done for all IT projects. According to IT investment management
officials, this inconsistency arose because policies and procedures for
developing business needs were not always followed or required.
DLA officials have stated that they are developing new guidance to address
this problem. However, until this guidance is implemented and enforced, DLA
cannot effectively demonstrate that priority mission and business
improvement needs are forming the basis for all its IT investment decisions.
Table 6 summarizes the ratings for each key practice and the specific
findings supporting the ratings.
Table 6: Business Needs Identification for IT Projects
Type of process Process Rating Summary of evidence
Organizational 1. The organization has DOD and DLA directives include
written policies Executed policies and procedures
commitment and procedures for for identifying the business
identifying the needs and associated users
business needs (and the of IT systems.
associated
users) of each IT project.
Type of process Process Rating Summary of evidence
IT Investment Selection Process Not Yet Established
aDLA has determined that the fourth project we reviewed-SAMMS-is no longer
meeting the agency's business needs.
Source: GAO.
Selecting new IT proposals requires an established and structured process to
ensure informed decisionmaking and infuse management accountability.
According to ITIM, this critical process requires, among other things, (1)
making funding decisions for new IT proposals according to an established
process, (2) providing adequate resources for proposal selection activities,
(3) using an established proposal selection process, (4) analyzing and
ranking new IT proposals according to established selection criteria,
including cost and schedule criteria, and (5) designating an official to
manage the proposal selection process. (The complete list of key practices
is provided in table 7.)
DLA has executed some of the key practices for investment proposal
selection. For example, DLA executives make funding decisions for IT
investments using DOD's Program Objective Memorandum (POM) process,13 which
is part of DOD's annual budgeting process. Through this process, proposals
for new projects or enhancements to ongoing projects are evaluated by DLA's
IT and financial groups and submitted to OSD through DLA's Corporate Board
with recommendations for funding approval. In addition, according to the
CIO, adequate resources have been provided to carry out activities related
to the POM process.
13The POM serves as the basis for the services' budget estimates. It shows
program needs for at least the next 5 years and includes cost estimates of
manpower, force levels, procurement, facilities, and research and
development.
Nonetheless, DLA has yet to execute some of the critical practices related
to this process area. Specifically, DLA acknowledges that the agency is not
analyzing and prioritizing new IT proposals according to established
selection criteria. Instead, the Corporate Board uses the expertise from the
IT organization and its own judgment to analyze and prioritize projects. To
its credit, DLA recognizes that it cannot continue to rely solely on the POM
process to make sound IT investment selection decisions. Therefore, the
agency has been working to establish an IT selection process over the past
two budget cycles that is more investment-focused and includes increased
involvement from IT Operations staff, necessary information, and established
selection criteria.
Until DLA implements an effective IT investment selection process that is
well established and understood throughout the agency, executives cannot be
adequately assured that they are consistently and objectively selecting
proposals that best meet the needs and priorities of the agency.
Table 7 summarizes the ratings for each key practice and the specific
findings supporting the ratings.
Table 7: Proposal Selection
Type of process Process Rating Summary of evidence
Prerequisites 1. Adequate resources are provided Executed The CIO has stated
that DLA has adequate resources for for proposal selection activities.
activities related to the POM process.
Source: GAO.
IT Investments Are Not Being Managed as a Complete Portfolio
An IT investment portfolio is an integrated, enterprisewide collection of
investments that are assessed and managed collectively based on common
criteria. Managing investments within the context of such a portfolio is a
conscious, continuous, and proactive approach to expending limited resources
on an organization's competing initiatives in light of the relative benefits
expected from these investments. Taking an enterprisewide perspective
enables an organization to consider its investments comprehensively so that
the collective investments optimally address its mission, strategic goals,
and objectives. This portfolio approach also allows an organization to
determine priorities and make decisions about which projects to fund based
on analyses of the relative organizational value and risks of all projects,
including projects that are proposed, under development, and in operation.
According to ITIM, stage 3 maturity includes (1) defining portfolio
selection criteria, (2) engaging in project-level investment analysis, (3)
developing a complete portfolio based on the investment analysis, (4)
maintaining oversight over the investment performance of the portfolio, and
(5) aligning the authority of IT investment boards.
Table 8 describes the purposes for the critical processes in stage 3.
Table 8: Stage 3 Critical Processes
Critical process Purpose
Authority alignment of IT To ensure that IT investments are selected and
investment boards managed by the appropriate investment board.
Portfolio selection criteria To ensure that the organization develops and
maintains definition IT portfolio selection criteria that support its
mission, organizational strategies, and business priorities.
Investment analysis To ensure that all IT investments are consistently
analyzed and prioritized according to the organization's portfolio selection
criteria.
Portfolio development To ensure that an optimal IT investment portfolio with
manageable risks and returns is selected and funded.
Portfolio performance To ensure that each IT investment portfolio achieves
its oversight cost, benefit, schedule, and risk expectations.
Source: GAO.
According to DLA officials, they are currently focusing on implementing
stage 2 processes and have not implemented any of the critical processes in
stage 3. Until the agency fully implements both stage 2 and 3 processes, it
cannot consider investments in a comprehensive manner and determine whether
it has the appropriate mix of IT investments to best meet its mission needs
and priorities.
DLA Lacks a Plan to Guide Improvement Efforts
DLA recognizes the need to improve its IT investment processes, but it has
not yet developed a plan for systematically correcting weaknesses. To
properly focus and target IT investment process improvements, an
organization should fully identify and assess current process strengths and
weaknesses (that is, create an investment management capability baseline) as
the first step in developing and implementing an improvement plan. As we
have previously reported,14 this plan should, at a minimum, (1) specify
measurable goals, objectives, milestones, and needed resources, and (2)
clearly assign responsibility and accountability for accomplishing
well-defined tasks. The plan should also be documented and approved by
agency leadership. In implementing the plan, it is important that DLA
measure and report progress against planned commitments, and that
appropriate corrective action be taken to address deviations.
DLA does not have such a plan. In March 2001, it attempted to baseline
agency IT operations by reviewing its project-level investment management
practices using ITIM. This effort identified practice strengths and
weaknesses, but DLA considered the assessment to be preliminary (to be
followed by a more comprehensive assessment at an unspecified later date)
and limited in scope. DLA used the assessment results to establish broad
milestones for strengthening its investment management process. The agency
did not, however, develop a complete process improvement plan. For example,
it did not (1) specify required resources to accomplish the various tasks,
(2) clearly assign responsibility and accountability for accomplishing the
tasks, (3) obtain support from senior level officials, and (4) establish
performance measures to evaluate the effectiveness of the completed tasks.
At the same time, the agency has separately begun other initiatives to
improve its investment management processes, but these initiatives are not
aligned with the established milestones or with each other. The DLA CIO
characterizes the agency's approach to its various process improvement
efforts as a necessary progression that includes some inevitable "trial and
error" as it moves toward a complete process improvement plan.
Without such a plan that allows the agency to systematically prioritize,
sequence, and evaluate improvement efforts, DLA jeopardizes its ability to
14U.S. General Accounting Office, Air Traffic Control: Immature Software
Acquisition Processes Increase FAA System Acquisition Risks, GAO/AIMD-97-47
(Washington, D.C.: March 21, 1997).
Conclusions
Recommendations
establish a mature investment process that includes selection and control
capabilities that result in greater certainty about future IT investment
outcomes.
Until recently, IT investment management has not been an area of DLA
management attention and focus. As a result, DLA currently finds itself
without some of the capabilities that it needs to ensure that its mix of IT
investments best meets the agency's mission and business priorities. To its
credit, DLA now recognizes the need to strengthen its IT investment
management and has taken positive steps to begin doing so. However, several
critical IT investment management capabilities need to be enhanced before
DLA can have reasonable assurance that it is maximizing the value of its IT
investment dollar and minimizing the associated risks. Moreover, DLA does
not yet have a process improvement plan that is endorsed and supported by
agency leadership. The absence of such a plan limits DLA's prospects for
introducing the management capabilities necessary for making prudent
decisions that maximize the benefits and minimize the risks of its IT
investment.
To strengthen DLA's investment management capability and address the
weaknesses discussed in this report, we recommend that the secretary of
defense direct the DLA director to designate the development and
implementation of effective IT investment management processes as an
agencywide priority.
Further, we recommend that the secretary of defense have the DLA director do
the following:
* Develop a plan, within 6 months, for implementing IT investment management
process improvements that is based on GAO's ITIM stage 2 and 3 critical
processes.
* Ensure that the plan specifies measurable goals and time frames, defines a
management structure for directing and controlling the improvements, and
establishes review milestones.
* Ensure that the plan focuses first on correcting the weakness in the ITIM
stage 2 critical processes, because these processes collectively provide the
foundation for building a mature IT investment management process.
Specifically:
* Develop and issue guidance covering the scope and operations of DLA's
investment review boards. Such guidance should include, at a
minimum, specific definitions of the roles and responsibilities within the
IT investment process; an outline of the significant events and decision
points within the processes; an identification of the external and
environmental factors that will influence the processes (for example, legal
constraints, the behavior of key suppliers or customers, or industry norms),
and the manner in which IT investment-related processes will be coordinated
with other organization plans and processes.
* Develop and issue policies and procedures for maintaining DLA's IT
projects inventory for investment management purposes.
* Finalize and issue policies and procedures (including the use of
information from the IT systems and project inventory) for the PEO Review
Board's oversight of IT projects.
* Develop and issue similar policies and procedures for the other investment
boards.
* Finalize and issue guidance supporting the identification of business
needs and implementing management controls to ensure that proposals
submitted to DLA for review clearly identify and define business
requirements.
* Develop and issue guidance for the proposal selection process in such a
way that the criteria for selection are clearly set forth, including
formally assigning responsibility for managing the proposal selection
process and establishing management controls to ensure that the proposal
selection process is working effectively.
* Ensure that the plan next focuses on stage 3 critical processes, which are
necessary for portfolio management, because along with the stage 2
foundational processes, these processes are necessary for effective
management of IT investments.
* Implement the approved plan and report on progress made against the
Agency Comments
and Our Evaluation
plan's goals and time frames to the secretary of defense every 6 months.
DOD provided what it termed "official oral comments" from the director for
acquisition resources and analysis on a draft of this report. In its
comments, DOD concurred with our recommendations and described efforts under
way and planned to implement them. However, it recommended that two report
captions be changed to more accurately reflect, in DOD's view, the contents
of the report and to eliminate false impressions.
Specifically, DOD recommended that we change one caption from "DLA's
Capabilities to Effectively Manage IT Investments Are Limited" to "DLA's
Capabilities to Effectively Manage IT Investments Should Be Improved."
DOD stated that this change is needed to recognize the fact that DLA has
completed about 75 percent of the practices associated with stage 2 critical
processes. We do not agree. As stated in our report, to effectively manage
IT investments an agency should (1) have basic, project-level control and
selection practices in place (stage 2 processes) and (2) manage its projects
as a portfolio of investments (stage 3 processes). Although DLA has executed
most of the key practices associated with stage 2 processes, the agency
acknowledges that it has not implemented any of the stage 3 processes.
Therefore, our caption as written describes DLA's IT investment management
capabilities appropriately.
In addition, DOD recommended that we change the caption "DLA Lacks a Plan to
Guide Improvement Efforts" to "DLA Lacks a Published Plan to Guide
Improvement Efforts." DOD stated that this change is needed because DLA has
developed some elements of an implementation plan. We do not agree. Our
point is that DLA did not have a complete process improvement plan, not that
it has yet to publish the plan that it has. As we describe in the report, a
complete plan should, at a minimum, (1) be based on a full assessment of
process strengths and weaknesses, (2) specify measurable goals, objectives,
milestones, and needed resources, (3) clearly assign responsibility and
accountability for accomplishing well-defined tasks, and (4) be documented
and approved by agency leadership.
In contrast, DLA's planning document was based on a preliminary assessment
of only stage 2 critical processes and lacked several of the critical
attributes listed above. Moreover, DOD stated in its comments that DLA has
not completed a formally documented and prioritized implementation plan to
resolve stage 2 and 3 practice weaknesses and has yet to complete the
self-assessment and gap analysis necessary to define planned action items.
Accordingly, it is clear that DLA has not satisfied the tenets of a complete
plan, and thus our caption is accurate as written.
DOD provided additional comments that we have incorporated as appropriate in
the report.
We are sending copies of this report to the chairmen and ranking minority
members of the Subcommittee on Defense, Senate Committee on Appropriations;
the Subcommittee on Readiness and Management Support, Senate Committee on
Armed Services; the Subcommittee on Defense, House Committee on
Appropriations; and the Subcommittee on Military Readiness, House Committee
on Armed Services. We are also sending copies to the director, Office of
Management and Budget; the
secretary of defense; the under secretary of defense for acquisition,
technology, and logistics; the deputy under secretary of defense for
logistics and materiel readiness; and the director, Defense Logistics
Agency. Copies will be made available to others upon request.
If you have any questions regarding this report, please contact us at
(202) 512-3439 and (202) 512-7351, respectively, or by e-mail at
[email protected] and [email protected]. An additional GAO contact and
staff acknowledgments are listed in appendix II.
Randolph C. Hite
Director, Information Technology Architecture
and Systems Issues
David L. McClure
Director, Information Technology Management Issues
Appendix I: GAO Contact and Staff Acknowledgments
GAO Contact Margaret Davis (202) 512-6398
Acknowledgments In addition to the individual named above, key contributors
to this report were Barbara Collier, Lester Diamond, Gregory Donnellon,
Sabine Paul, and Eric Trout.
GAO's Mission
Obtaining Copies of GAO Reports and Testimony
The General Accounting Office, the investigative arm of Congress, exists to
support Congress in meeting its constitutional responsibilities and to help
improve the performance and accountability of the federal government for the
American people. GAO examines the use of public funds; evaluates federal
programs and policies; and provides analyses, recommendations, and other
assistance to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents is through the
Internet. GAO's Web site (www.gao.gov) contains abstracts and full-text
files of current reports and testimony and an expanding archive of older
products. The Web site features a search engine to help you locate documents
using key words and phrases. You can print these documents in their
entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its Web
site daily. The list contains links to the full-text document files. To have
GAO e-mail this list to you every afternoon, go to www.gao.gov and select
"Subscribe to daily e-mail alert for newly released products" under the GAO
Reports heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out to
the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. General Accounting Office P.O. Box 37050 Washington, D.C. 20013
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
Visit GAO's Document GAO Building
Distribution Center Room 1100, 700 4th Street, NW (corner of 4th and G
Streets, NW) Washington, D.C. 20013
To Report Fraud, Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm,
Waste, and Abuse in E-mail: [email protected], or
Federal Programs 1-800-424-5454 or (202) 512-7470 (automated answering
system).
Jeff Nelligan, Managing Director, [email protected] (202) 512-4800
Public Affairs U.S. General Accounting Office, 441 G. Street NW, Room 7149,
Washington, D.C. 20548
*** End of document. ***