Information Technology: DLA Should Strengthen Business Systems
Modernization Architecture and Investment Activities (29-JUN-01,
GAO-01-631).
The Defense Logistics Agency (DLA) plays a critical role in
supporting America's military forces worldwide. DLA employs about
28,000 civilian and military workers, located at about 500 sites
in all 50 states and 28 countries; in round numbers, it manages 4
million supply items and processes 30 million annual supply
distribution actions. This report reviews DLA's efficiency and
effectiveness in managing it is Business Systems Modernization
(BSM) acquisition program. Specifically GAO determines (1)
whether DLA is using an enterprise architecture to guide and
constrain its investment in (BSM) and (2) whether DLA is
investing in BSM in an incremental manner. GAO found that DLA
does not have an enterprise architecture to guide its investment
in BSM. DLA plans call for creating an architecture as a
byproduct of BSM's implementation. In addition, GAO found that
DLA has not been managing its investment in this program in an
incremental manner; that is, DLA has not treated the first of its
four planned incremental releases of BSM as a separate investment
decision. Instead, DLA has thus far treated the entire BSM
program as a single investment decision, according to BSM
officials. DLA stated that it now plans to take an incremental
approach to future releases. This change would ensure that DLA's
investment decisions are consistent with statutory requirements
and federal guidance.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-01-631
ACCNO: A01033
TITLE: Information Technology: DLA Should Strengthen Business
Systems Modernization Architecture and Investment Activities
DATE: 06/29/2001
SUBJECT: Information resources management
Information technology
Logistics
Systems conversions
Systems design
DLA Business Systems Modernization
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-01-631
Report to Congressional Committees
United States General Accounting Office
GAO
June 2001 INFORMATION TECHNOLOGY
DLA Should Strengthen Business Systems Modernization Architecture and
Investment Activities
GAO- 01- 631
Page i GAO- 01- 631 Information Technology Letter 1
Results in Brief 2 Background 4 DLA Has Developed a Strategic Plan and
Logistics Transformation
Plan 9 BSM Is Not Being Guided by Either a DLA Architecture or a DODwide
Logistics Architecture 14 DLA Has Not Invested Incrementally in BSM 26
Conclusions 31 Recommendations 31 Agency Comments and Our Evaluation 33
Appendix I Objectives, Scope, and Methodology 38
Appendix II Comments From the Department of Defense 40
Appendix III GAO Contact and Staff Acknowledgment 59
Tables
Table 1: DLA Field Command Management Responsibilities 6 Table 2: Materiel
Management Functions Performed by Legacy
Systems 7 Table 3: BSM Investment According to Fiscal Year 2001 Budget
Submission 11 Table 4: Seven Essential Products for the DOD C4ISR
Architecture
Framework 19
Figures
Figure 1: BSM Timeline 12 Figure 2: BSM Within DLA?s Organization and
Oversight Structure 14 Figure 3: C4ISR Architecture Framework: Views and
Description 17 Figure 4: DLA?s Approach to Creating an Enterprise
Architecture 20 Contents
Page ii GAO- 01- 631 Information Technology Abbreviations
BSM Business Systems Modernization C4ISR Command, Control, Communications,
Computers,
Intelligence, Surveillance, and Reconnaissance CIO Chief Information Officer
COTS commercial, off- the- shelf DISMS Defense Integrated Subsistence
Management System DLA Defense Logistics Agency DLA 21 Strategic Plan 2000:
DLA 21
DOD Department of Defense DPACS Defense Pre- award Contracting System DT
developmental test ERP enterprise resource planning FOC full operating
capability IOC initial operating capability IOT& E initial operational test
and evaluation IT information technology NATO North Atlantic Treaty
Organization OA operational assessment OMB Office of Management and Budget
REL release SAMMS Standard Automated Materiel Management System SPEDE SAMMS
Procurement by Electronic Data Exchange TRR test readiness review Y2K Year
2000
Page 1 GAO- 01- 631 Information Technology
June 29, 2001 The Honorable Carl Levin Chairman The Honorable John Warner
Ranking Minority Member Committee on Armed Services United States Senate
The Honorable Bob Stump Chairman The Honorable Ike Skelton Ranking Minority
Member Committee on Armed Services House of Representatives
The Defense Logistics Agency (DLA) plays a critical role in supporting
America?s military forces worldwide. To fulfill this role, DLA employs about
28,000 civilian and military workers, located at about 500 sites in all 50
states and 28 countries; in round numbers, it manages 4 million supply items
and processes 30 million annual supply distribution actions. In fiscal year
2000, DLA reported that these operations resulted in sales to the military
services of about $13 billion.
This report is one in a series of products to satisfy our mandate under the
fiscal year 2001 Defense Authorization Act. 1 The act directed that we
review DLA?s efficiency and effectiveness in meeting customer requirements,
application of best business practices, and opportunities for improving DLA
operations. As agreed with your offices, our first review of DLA?s
information technology (IT) management practices focused on DLA?s $900
million Business Systems Modernization (BSM) acquisition. As further agreed
with your offices, our objectives were to determine (1) whether DLA is using
an enterprise architecture to guide and constrain its investment in BSM and
(2) whether DLA is investing in BSM in an incremental manner. Using
enterprise architectures- institutional blueprints for business and
technological change- and investing incrementally in large modernization
programs are legislative and federal
1 P. L. 106- 398, Floyd D. Spence National Defense Authorization Act for
Fiscal Year 2001, section 917.
United States General Accounting Office Washington, DC 20548
Page 2 GAO- 01- 631 Information Technology
requirements 2 and best industry practices. This review did not address
other system modernization best practices, such as whether DLA is employing
effective system acquisition process controls, and did not evaluate the
specific commercial, off- the- shelf product that DLA chose as its BSM
system solution. Details on our objectives, scope, and methodology are in
appendix I.
DLA does not have an enterprise architecture to guide its investment in BSM,
even though the law and Office of Management and Budget (OMB) guidance
recognize the importance of enterprise architectures and Department of
Defense (DOD) policy requires their use. Rather, DLA plans call for creating
an architecture as a by- product of BSM?s implementation. Moreover, DLA?s
architecture development plans address only one, albeit the largest, of its
six primary business areas- materiel management (also called supply- chain
management). According to DLA?s plans, its architectural products will not
be extended to its other business areas until 5 years from now. This
nonagencywide approach to developing and implementing an enterprise
architecture is not consistent with federal guidance, and it increases the
risk that DLA will modernize in a way that optimizes an individual business
area but does not optimize agencywide logistics management performance and
accountability. Further, DLA does not have effective management structures
and processes to support architecture development, as defined in Chief
Information Officers (CIO) Council guidance. 3 At the completion of our
review, DLA officials stated that they would modify their plans and
expeditiously pursue an agencywide architecture development and
implementation approach, consistent with federal and DOD policies and
guidance.
Equally if not more important, DOD has not developed a DOD- wide logistics
management enterprise architecture that would promote interoperability and
avoid duplication among the logistics modernization efforts now under way in
DOD component organizations, such as DLA and the services. To its credit,
the Office of the Deputy Under Secretary of Defense for Logistics and
Materiel Readiness (the office responsible for DOD- wide logistics policy
and strategy) has begun efforts to correct this
2 Clinger- Cohen Act of 1996, P. L. 104- 106; Management of Federal
Information Resources,
Office of Management and Budget Circular A- 130 (November 30, 2000). 3
Architecture Alignment and Assessment Guide, Chief Information Officers
Council (October 2000). Results in Brief
Page 3 GAO- 01- 631 Information Technology
architectural void, and it has initiated interim steps intended to
coordinate the new logistics management system investments that DLA and the
services have under way. However, according to officials in this office,
plans for completing this architecture are not sufficiently advanced to be
provided to us for evaluation. Moreover, the interim steps that these
officials described do not provide the management control rigor and
discipline that well- defined and effectively implemented enterprise
architectures provide. By allowing the services and DLA (through BSM) to
proceed separately with new logistics management systems in the absence of a
DOD- wide enterprise architecture, DOD will not be in a position to optimize
logistics operations and system performance across the department, and thus
is unlikely to successfully meet its strategic logistics management goals.
At the conclusion of our review, officials in the Deputy Under Secretary?s
office acknowledged the need to move swiftly in developing and using a DOD-
wide logistics management architecture.
In addition to the enterprise architecture issues affecting BSM, DLA has not
been managing its investment in this program in an incremental manner; that
is, DLA has not treated the first of its four planned incremental releases
of BSM as a separate investment decision, justified on the basis of release-
specific analyses of costs, benefits, and risks. Instead, DLA has thus far
treated the entire BSM program as a single investment decision, justified by
a single, ?all or nothing? economic analysis, because doing so, according to
BSM officials, was consistent with DOD policy then in place for major system
acquisitions. 4 At the completion of our review, DLA officials told us that
they now plan to take an incremental investment approach for future
releases: that is, they plan to measure the actual accrual of benefits
versus costs on each incremental release (including the first) and to use
this information to make investment decisions on future increments. This
change in BSM investment strategy is important because an ?all or nothing?
approach to investment decisions is not consistent with statutory
requirements and federal guidance for incremental investment management and
has led agencies to invest huge sums in systems without a commensurate
return.
To strengthen DLA management of BSM, we are making recommendations
(consistent with officials? stated commitments) concerning the effective
development and use of the DOD- wide and DLA architectural context needed
for guiding BSM, and concerning the need for incremental
4 DOD Instruction 5000. 2, Operation of the Defense Acquisition System.
Page 4 GAO- 01- 631 Information Technology
investment in BSM based on economic justification and validation of expected
return on investment.
In commenting on a draft of this report, DOD agreed with the importance of
using enterprise architectures and making BSM investment decisions
incrementally. DOD also acknowledged that the absence of these IT management
controls increased program risk. Further, DOD agreed with eight of our ten
recommendations. However, DOD disagreed with our recommendations to limit
investment in BSM beyond the first incremental release and to use actual
benefit accrual on the first incremental release in deciding on investment
in the next release, noting that this would delay BSM progress. According to
DOD, our findings and conclusions regarding the overall level of risk to the
successful achievement of BSM and other DOD logistics modernization program
objectives are not supported by the single risk factor addressed in this
report- the use of enterprise architectures. DOD also contended that all
program risks are being effectively mitigated.
Notwithstanding the considerable level of agreement between DOD and us on
this report, we have a differing view on the level of risk facing DOD
logistics modernization programs in general, and BSM in particular, arising
from the two risk factors addressed in this report: (1) the lack of
effective enterprise architecture development and implementation and (2) the
lack of effective incremental investment management. Our experience
reviewing other federal agency modernization programs is that the use of
enterprise architectures, employed in concert with other important IT
management controls (such as incremental investment management), greatly
increases the chances of successful modernization. Accordingly, their
absence greatly reduces the chance of success. While we have modified our
recommendations slightly and have clarified appropriate sections in the
report to address DOD?s concerns about delaying BSM progress until these IT
management controls are in place, the substance of the recommendations is
unchanged.
DLA is DOD?s logistics manager for all DOD consumable items 5 and some DOD
repair items; 6 its primary business function is providing supply
5 Consumable items include such commodities as subsistence (food), fuels,
medical supplies, clothing, and construction equipment. 6 These repair items
are spare and repair parts that support about 1, 400 DOD weapon systems.
Each of the military services also manages its own service- unique repair
items. Background
Page 5 GAO- 01- 631 Information Technology
support to sustain military operations and readiness. In addition to this
primary function, which DLA refers to as either materiel management or
supply- chain management, DLA performs five other major business functions:
distributing materiel ordered from its inventory, purchasing fuels for DOD
and the U. S. government, storing strategic materiel, 7 marketing surplus
DOD materiel for reuse and disposal, and providing numerous information
services, such as item cataloging, 8 for DOD and the U. S. and selected
foreign governments.
DLA consists of a central command authority supported by a number of field
commands that manage the agency?s six business functions. Table 1 shows
DLA?s field commands, their respective business functions, and examples of
the consumable and/ or repair items each manages.
7 Strategic materiel is defined as any item or materials needed to sustain
the United States in the event of a national emergency. 8 DLA defines item
cataloging as including all activities that describe the technical
characteristics and data for an individual item of supply.
Page 6 GAO- 01- 631 Information Technology
Table 1: DLA Field Command Management Responsibilities Field command
Business function Examples of items managed
Defense Supply Centers a Materiel management: Procuring consumable items
(except fuels) and some DOD repair parts supporting weapons systems
Food, medical supplies, clothing, construction equipment, spare and repair
parts Defense Distribution Center Distribution: Receiving, storing,
packaging, and shipping
consumable items (and some DOD repair parts supporting weapons systems)
procured by the Defense Supply Centers
(Same as above) Defense Energy Support Center Energy services: Procuring,
storing, and shipping fuel and
other energy sources for the federal government Petroleum, jet fuel, coal
Defense National Stockpile Center Strategic materiel management: Procuring,
selling, and
storing strategic and critical materiel Metals, minerals, and ores (e. g.,
aluminum, chromium, diamonds)
Defense Reutilization and Marketing Services Materiel reuse: Managing the
reuse and disposal of excess
DOD materiel Aircraft parts, automobiles, clothing, typewriters, furniture
Defense Logistics Information Service Information services: Providing
services, such as item
cataloging, for DOD, the U. S. government, and international organizations
(NATO and allied militaries)
(All of the above) a The three Defense Supply Centers are in Columbus,
Philadelphia, and Richmond. Source: DLA Customer Assistance Handbook, 1998.
To support the current materiel management business function, DLA reports
that it relies on a collection of mainframe- based Cobol systems, which are
not integrated and are more than 30 years old. These legacy systems support
DLA?s traditional mission as a manager of physical inventory; that is, they
support materiel management functions such as acquisition/ procurement,
inventory control, requirements forecasting, requisition processing,
technical data maintenance, and financial management. The systems consist of
two primary systems- the Standard Automated Materiel Management System
(SAMMS) and the Defense Integrated Subsistence Management System (DISMS)-
and two subsidiary systems- SPEDE (SAMMS Procurement by Electronic Data
Exchange) and DPACS (Defense Pre- award Contracting System). The functions
of these systems are summarized in table 2.
Page 7 GAO- 01- 631 Information Technology
Table 2: Materiel Management Functions Performed by Legacy Systems Materiel
management function SAMMS a DISMS b SPEDE c DPACS d
Acquisition/ procurement x x x x Inventory control x x x Requirements
forecasting x x Requisition processing x x x x Technical data maintenance x
x Financial management x x x x a Primary system supporting inventory
management; provides information regarding stock levels, acquisition and
management of wholesale consumable items, direct support for processing
requisitions, generation of purchase requests, identification of items,
asset visibility, and maintenance of an audit trail of transactions
processed. b Primary system supporting the worldwide wholesale food business
in support of troop feeding and
commissary resale; in supporting these commodities, performs the same
functions as SAMMS. c Subsystem giving DLA Supply Centers the capability to
interface with vendors electronically; allows
electronic transfer of requests for quotation, vendor responses, invoices,
and purchase orders. d Subsystem performing precontracting functions;
provides purchase request management data that buyers need to complete a
solicitation and award package, including item descriptions, applicable
specifications/ drawings, synopsis information, histories of past
procurements, and other purchase requests and/ or contracts open for the
same National Stock Number.
Source: DLA Y2K Program Software Development Plan.
DLA reports that these legacy systems are the product of decades of
accumulated and divergent business practices and use obsolete technology
that is no longer supported by the original equipment manufacturers and the
software support provider. Further, DLA maintains that these systems,
consisting of several million lines of unstructured and expensive- to-
maintain code, provide inadequate analytical capability and no real- time
data access.
DOD?s logistics management strategy is set forth in two strategic planning
documents: Joint Vision 2020 9 and 21st Century Logistics: DOD Logistics
Strategic Plan. 10 Joint Vision 2020 is DOD?s primary strategic plan,
covering all missions and business areas. This plan introduced what DOD
calls focused logistics, described as the ability to provide the joint
9 Joint Vision 2020, published in June 2000, was preceded by Joint Vision
2010, published in 1996. It was issued by the Chairman, Joint Chiefs of
Staff. 10 21st Century Logistics, published in August 1999, defines more
specifically the strategic vision and objectives for logistics management.
It was issued by the Deputy Under Secretary of Defense for Logistics and
Materiel Readiness. DOD Has Issued Strategic
Plans for Modernizing Logistics Management
Page 8 GAO- 01- 631 Information Technology
force 11 the right assets (personnel, equipment, and supplies) in the right
place, at the right time, and in the right quantity. To achieve this goal,
DOD envisions a real- time, Web- based 12 information system providing both
logisticians and field commanders with total asset visibility within a given
theater of operations. Additionally, this system is to include decision
support tools to improve requirements analysis and planning, as well as
provide real- time control of the logistics supply chain, regardless of
whether the requirements are to be fulfilled by the commercial sector or
within DOD.
21st Century Logistics calls for the military services and DLA to pursue
system modernization efforts to bring about the focused logistics goals
described in Joint Vision 2020. The systems are to be an integrated
collection of applications sharing a common data environment, which are to
be deployed to all forces by fiscal year 2006.
To fulfill Joint Vision 2020, the Deputy Secretary of Defense issued a
memorandum in March 2000 to the military services and DLA requiring
submission of logistics transformation plans, documenting, on an annual
basis, planned actions and related resources for implementing focused
logistics, as well as any other logistics initiatives. The military services
and DLA were directed to satisfy the 21st Century Logistics objectives by
fiscal year 2006, including developing logistics process and system
modernization plans by the end of fiscal year 2001 and implementing these
plans by fiscal year 2006. DLA was directed to submit its transformation
plan to the Under Secretary of Defense for Acquisition, Technology, and
Logistics by July 1, 2000.
11 Joint force is a general term applied to a force composed of significant
elements, assigned or attached, of two or more military departments,
operating under a single joint force commander.
12 According to DLA, real- time, Web- based means immediate computer
responsiveness through the Internet or an intranet.
Page 9 GAO- 01- 631 Information Technology
To fulfill its direction under DOD?s 21st Century Logistics plan, DLA has
outlined strategic goals and objectives in its Strategic Plan 2000: DLA 21
13 and Logistics Transformation Plan. 14 Under Goal 2 of DLA 21, DLA plans
to reduce costs, improve efficiency, and increase effectiveness through
organizational redesign, business systems modernization, strategic sourcing,
infrastructure consolidation, and optimally sized inventories.
To achieve this goal, DLA is first focusing on transforming its materiel
management (supply- chain management) function, because according to the DLA
Director, this is DLA?s most critical business function. More specifically,
it plans to implement a Web/ network- based logistics system using
commercial, off- the- shelf (COTS) products.
To select the most appropriate COTS products, DLA evaluated potential
solutions against business processes for certain of its commodities
(including clothing, aircraft parts, and medical supplies). According to
DLA, it chose important, widely differing commodities so that its analysis
would be sufficiently comprehensive to be extrapolated to the rest of the
commodities within the materiel management business function. Further, DLA
chose to evaluate solutions against business processes that had been
previously reengineered, so that the system solution chosen would be based
on modernized business processes. Through this analysis, which used scripted
demonstrations of the reengineered requirements, DLA sought to ensure that
the chosen applications could effectively meet DLA?s materiel management/
supply- chain management requirements.
The resulting COTS- based system is to be used to reengineer the agency?s
materiel management business processes. DLA refers to its acquisition and
implementation of this COTS- based system, and the associated business
process reengineering, as Business Systems Modernization (BSM).
DLA?s BSM program is intended to modernize DLA?s materiel management
business function, changing DLA from solely a provider and manager of
physical inventory to primarily a manager of supply chains. In this role,
DLA would link customers with appropriate suppliers and track physical and
financial assets. DLA expects this approach to enhance supply- chain
13 DLA refers to this plan, published in September 1999, as DLA 21.
14 DLA?s Logistics Transformation Plan was approved by the Under Secretary
of Defense for Acquisition, Technology, and Logistics on July 11, 2000. DLA
Has Developed a
Strategic Plan and Logistics Transformation Plan
DLA Has Begun Implementing its Business Systems Modernization Program
Page 10 GAO- 01- 631 Information Technology
visibility and capabilities, resulting in reduced logistics cycle times,
increased customer satisfaction, reduced customer wait time, and decreased
materiel costs. In short, DLA?s goal is ?to do business as business does
business.?
According to BSM program documents, DLA?s transformation from its current to
its future role requires modernization of the IT systems that now support
DLA?s materiel management business function, subfunctions, and processes.
The BSM system solution is envisioned as consisting of Web- and COTS- based
applications supported by an enterprise application tool. 15 Through the use
of COTS products, DLA expects to reduce materiel management systems
operating costs and take advantage of the continuous technology insertion
and continuous functional improvement that COTS packages allow. The BSM COTS
will consist of enterprise resource planning (ERP) 16 software.
DLA plans to acquire and deploy its BSM COTS system solution through a
series of four system releases/ increments. First, it plans to demonstrate
successful application of its new concept of doing business to selected
commodities, namely, earth- moving equipment, medical/ pharmaceutical
supplies, and F/ A- 18 engine components. These commodities were chosen for
the first release because each is an important responsibility of one of the
three Defense Supply Centers. Thus, the first release will be deployed to
all three Defense Supply Centers at once. Once this first release is
successfully demonstrated, DLA plans to expand the system solution to other
commodities and to other users within the materiel management business
function.
According to DLA?s fiscal year 2001 budget submission, DLA will invest
approximately $897 million to acquire and implement BSM from fiscal years
2000 through 2007 (see table 3). Thus far, DLA has obligated about $150
million for BSM. From this total, about $56 million was obligated in
15 According to DOD, this enterprise application tool is an integration tool
that effects application- to- application interfacing, supports new
?business- to- business? relationships, and mitigates the risk of
introducing new business functions into the BSM architecture.
16 ERP software packages provide a suite of software applications and
modules, usually employing a common relational database, that provide
functionality for managing a range of tasks, such as planning, purchasing,
maintaining inventories, interacting with suppliers, providing customer
service, and tracking orders. These packages are intended to provide a
generic set of enterprise management applications that can be used to
reengineer business operations.
Page 11 GAO- 01- 631 Information Technology
fiscal year 2000 to prepare and validate functional requirements, conduct
pre- award contract functions for selection of the systems integration
contractor, perform initial work on the program by awarding the integration
contract, and begin detailed business process reengineering planning for the
first increment of the program.
Table 3: BSM Investment According to Fiscal Year 2001 Budget Submission
Fiscal year Dollars a
(in millions) Cumulative percentage of total
2000 $56 6 2001 93 17 2002 104 28 2003 151 45 2004 176 65 2005 116 78 2006
101 89 2007 100 100
Total $897
a Costs include $183 million for maintenance. Source: DLA budget for fiscal
year 2001.
The remaining approximately $93 million was obligated in fiscal year 2001
for the development of a materiel management enterprise architecture
(referred to by DLA as the BSM Business Architecture Blueprint) and for the
initial implementation, including testing, of the BSM concept demonstration.
Of the unobligated portion of the BSM budget, about $549 million is to fund
implementation and sustainment of the COTS solution (system releases 1
through 4) within the materiel management business area, including its
extension to the commodities not covered by the concept demonstration. DLA
officials told us that the remaining unobligated funds for fiscal years 2006
and 2007, about $201 million, are to be used for extending the BSM ERP
software to the distribution and cataloging business areas.
Only about 1 percent of the BSM budget is for new network and computing
platform infrastructure (hardware and systems software). This is because
DLA?s current and future investments in IT infrastructure, which will
support ongoing DLA operations as well as the BSM program, are funded
separately. The infrastructure investments in fiscal year 2001 amount to
about 45 percent of DLA?s approximately $680 million IT budget.
Page 12 GAO- 01- 631 Information Technology
BSM is expected to take 6 years to acquire and implement for materiel
management (fiscal years 2000 through 2005). DLA has structured the BSM life
cycle to comply with DOD 5000.2 17 milestone phases for major systems
acquisitions. Figure 1 summarizes the BSM development timeline.
Figure 1: BSM Timeline
Legend IOC: initial operating capability FOC: full operating capability REL:
release OA: operational assessment TRR: test readiness review DT:
developmental test IOT& E: initial operational test and evaluation
Source: BSM program documents.
The first milestone phase (phase 0, December 1999 through July 2000) of the
BSM program focused on evaluation of alternative concepts and potential
system integrators.
The second and current milestone phase (phase I/ IIA 18 ) covers program
definition (design), risk reduction, and concept demonstration. The current
phase includes implementation and testing/ evaluation of the system solution
for certain commodities, as well as such implementation
17 DOD Instruction 5000. 2, Operation of the Defense Acquisition System.
18 BSM is following a modified acquisition process, as allowed by DOD
Instruction 5000. 2. The traditional acquisition process consists of four
discrete phases: 0 (concept exploration), I (program definition and risk
reduction), II (engineering and manufacturing development), and III
(production, fielding/ deployment, and operational support).
FY 2000 Quarters: Process setting:
Audits/ reviews: 2 123 3 12 3 123 12 3 123 4 4 4 4 4 4 BSM major
milestones Concept Demo (CD) Milestone
II/ III Milestone I/ IIA
Contract award Milestone 0 FOC
Design TRR OA
IOC REL 2 REL 3 REL 4
FY 2001 FY 2002 FY 2003 FY 2004 FY 2005 REL 1
REL 4 test cycle IOT& E REL 3 test cycle CD DT
REL 2 test cycle
Page 13 GAO- 01- 631 Information Technology
activities as training and site preparation. This phase is scheduled to run
from September 2000 through September 2002.
The third phase (phase II/ III) is referred to as initial fielding/
deployment and operational support. This phase focuses on implementing the
system beyond the scope of the concept demonstration, operating and
maintaining the system, and modifying and upgrading the system as required.
This phase is scheduled to run from October 2002 through March 2005.
Investment decisionmaking for BSM (and other DLA major modernization
programs) is vested in the DLA 21 Executive Board, chaired by the Chief
Acquisition Executive of DLA, who is the Vice Director. BSM life- cycle
management is the responsibility of the BSM Program Manager, who runs the
BSM program office and reports to the DLA Program Executive Officer, 19 who
oversees the program in coordination with the DLA Chief Information Officer
(CIO). Figure 2 shows this BSM program management and oversight structure.
19 Under DOD 5000 regulations, Program Executive Officers are to oversee
major systems acquisitions.
Page 14 GAO- 01- 631 Information Technology
Figure 2: BSM Within DLA?s Organization and Oversight Structure
Source: BSM program documents.
Using enterprise architectures to guide and constrain information technology
investments is called for by law, OMB, and DOD, and is an industry best
practice. An enterprise architecture is a blueprint for guiding and
constraining business and technological change for the enterprise, which can
be an organization (e. g., DLA or the services) or a functional or mission
area spanning more than one organization (e. g., logistics management or
financial management). In some cases, both organizational and functional/
mission area architectures are appropriate, because organizations
interrelate closely, sharing functional and mission area responsibilities.
This is the case for DOD and its component organizations.
DLA is managing its BSM program without having either a DLA enterprise
architecture or a DOD- wide logistics management enterprise architecture. In
doing so, DLA risks investing in significant business process and system
change that does not optimally support enterprise (DLA and DOD) requirements
(operational, system, and technical). To their credit, officials from both
DLA and the Office of the Deputy Under Secretary for Logistics BSM Is Not
Being
Guided by Either a DLA Architecture or a DOD- wide Logistics Architecture
Business process reengineering Transformation management Knowledge transfer
System deployment Acquisition planning
Requirements management Contractor management Testing
Logistics Operations
Business Modernization BSM
Stakeholders Group: DLA Senior Executives Primary External Stakeholders
CIO/ Information Operations
Program Executive Officer
DLA- 21 Executive Board Program Manager, BSM
DLA Director Vice Director Executive Director
DLA Field Commands
Organizations within grey box are represented on the DLA- 21 Executive
Board. Dotted lines represent communication and coordination.
Financial Operations
Page 15 GAO- 01- 631 Information Technology
and Materiel Readiness recognize the need to take an enterprise approach to
transforming logistics management, and at the conclusion of our work, they
committed to doing so.
Enterprise architectures are essential tools for effectively and efficiently
reengineering business processes and for implementing and evolving their
supporting systems. Enterprise architectures systematically capture- in
useful models, diagrams, and narrative- the full breadth and depth of the
mission- based mode of operation for a given enterprise, which can be (1) a
single organization or (2) a functional or mission area that transcends more
than one organizational boundary (e. g., financial management, acquisition
management, logistics management). An architecture describes the
enterprise?s operations in both (1) logical terms, such as interrelated
business processes, information needs and flows, work locations, and system
applications, and (2) technical terms, such as hardware, software, data,
communications, and security attributes and performance standards. It
provides these perspectives both for the enterprise?s current or ?as is?
environment and for its target or ?to be? environment, as well as an IT
capital investment road map for moving between the two environments.
The development, implementation, and maintenance of enterprise architectures
are recognized hallmarks of successful public and private sector
organizations. Managed properly, an enterprise architecture can clarify and
help optimize the interdependencies and interrelationships among an
organization?s business operations and the underlying IT infrastructure and
applications that support these operations. Employed in concert with other
important IT management controls, such as portfolio investment management
(selection, control, and evaluation) practices 20 and continuous information
security management practices, 21 enterprise architectures can greatly
increase the chances that modernization programs will succeed. Our
experience with federal agencies has shown that attempting to define and
build major IT systems without first completing an enterprise architecture
often results in IT systems that are duplicative, are not well integrated,
are unnecessarily costly to maintain and interface, and do not effectively
optimize mission performance.
20 Information Technology Investment Management: A Framework for Assessing
and Improving Process Maturity (Exposure Draft) (GAO/ AIMD- 10. 1. 23, May
2000). 21 Executive Guide: Information Security Management (GAO/ AIMD- 98-
68, May 1998). Enterprise Architectures:
A Cornerstone of Successful Modernization Programs
Page 16 GAO- 01- 631 Information Technology
Congress, OMB, and the federal CIO Council have recognized the importance of
enterprise architectures. The Clinger- Cohen Act, for example, requires that
agency CIOs develop, maintain, and facilitate the implementation of
enterprise architectures as a means of integrating business processes and
agency goals with IT. Further, OMB has issued guidance on the development
and implementation of agency IT architectures. Among other things, OMB
guidance directs that agency investments in information systems be based on
these architectures. 22 Similarly, the CIO Council has issued guidance
providing (1) a federal framework for the content and structure of an
enterprise architecture, 23 (2) a process for assessing investment
compliance with an enterprise architecture, 24 and (3) a set of management
controls for developing, implementing, and maintaining an enterprise
architecture. 25
According to CIO Council guidance, it is critically important that an
enterprise architecture be derived through a ?top- down? incremental
approach, consistent with the hierarchical architectural views that are the
building blocks of published architecture frameworks, including DOD?s. It is
equally important, according to this guidance, that the higher level views
span the entire enterprise. Only through such an approach can an
organization develop enterprisewide understanding of the interrelationships
and interdependencies among business operations and supporting technology.
Such understanding is vital for informed decisionmaking about whether the
enterprise, and thus the enterprise architecture, can be divided into
segments without sacrificing the goal of optimizing enterprisewide
performance and accountability.
DOD has also issued enterprise architecture policy, including a framework
defining an architecture?s content and structure. Specifically, in February
22 Funding Information Technology Systems Investments, OMB Memorandum M- 97-
02 (October 25, 1996); Information Technology Architectures, OMB Memorandum
M- 97- 16 (June 18, 1997).
23 Federal Architecture Framework, Version 1. 1, Chief Information Officers
Council (September 1999). 24 Architecture Alignment and Assessment Guide,
Chief Information Officers Council (October 2000). 25 A Practical Guide to
Federal Enterprise Architecture, Version 1.0, Chief Information Officers
Council (February 2001).
Page 17 GAO- 01- 631 Information Technology
1998, 26 DOD directed its components and activities to use the C4ISR
(Command, Control, Communications, Computers, Intelligence, Surveillance,
and Reconnaissance) Architecture Framework, Version 2.0. According to DOD,
this framework is a critical tool in achieving its strategic direction, and
all DOD components and activities should use the framework for all
functional areas and domains within the department. This framework is also
recognized in the CIO Council published guide as a model architecture
framework. Figure 3 shows the views required by the C4ISR framework and
their relationships.
Figure 3: C4ISR Architecture Framework: Views and Description
Source: C4ISR Architecture Framework, Version 2.
26 The February 28, 1998, memorandum was jointly signed by the Under
Secretary of Defense for Acquisition and Technology, the Acting Assistant
Secretary of Defense for Command, Control, Communications, and Intelligence,
and the Director for C4 Systems, Joint Chiefs of Staff.
Technical view
Specific capabilities identified to satisfy information exchange levels and
other operational requirements
Technical criteria governing interoperable implementation/ procurement of
the selected system capabilities Relates capabilities and characteristics
to operational requirements
Systems view
Processing and internodallevels of information exchange requirements
Systems associations to nodes, activities, needlines and requirements
Processing and levels ofinformation exchange requirements
Basic technologysupportability and new capabilities Identifies warfighter
relationships and information needs
Operational view
Prescribes standards and conventions
Page 18 GAO- 01- 631 Information Technology
Briefly, the C4ISR Architecture Framework decomposes an enterprise
architecture into three primary views (windows into how the enterprise
operates): the operational, systems, and technical views. According to DOD,
these three interdependent views are needed to ensure that IT systems are
developed and implemented in an interoperable and costeffective manner. Each
of these views is summarized below.
The operational architecture view defines the operational elements,
activities, tasks, and information flows required to accomplish or support
an organizational mission or business function. According to DOD, it is
useful for facilitating a number of actions and assessments across DOD, such
as examining business processes for reengineering or defining operational
requirements to be supported by physical resources and systems.
The systems architecture view defines the systems and their
interconnections supporting the organizational or functional mission,
including how multiple systems link and interoperate, and may describe the
internal construction and operations of particular systems. According to
DOD, this view has many uses, such as helping managers to evaluate
interoperability improvement and to make investment decisions concerning
cost- effective ways to satisfy operational requirements.
The technical architecture view defines a minimum set of standards and
rules governing the arrangement, interaction, and interdependence of system
applications and infrastructure. It provides the technical standards,
criteria, and reference models upon which engineering specifications are
based, common building blocks are established, and applications are
developed.
Within the three architectural views, the C4ISR Architecture Framework
identifies 26 graphical, textual, and tabular architectural artifacts or
products. Of the 26 products, DOD specifies that 7 are essential and must be
developed for each enterprise architecture. Table 4 briefly describes the
content of each essential product.
Page 19 GAO- 01- 631 Information Technology
Table 4: Seven Essential Products for the DOD C4ISR Architecture Framework
Essential product Description
Overview and summary information Serves as planning guide and summarizes
?who, what, when, why, and how? for architecture to be developed Integrated
dictionary Provides a central source for definitions of all terms used in
all architecture products High- level operational concept graphic Shows a
high- level graphic description of operational concept, including
organizations,
missions, and geographic distribution of assets Operational node
connectivity description Identifies organizational elements that produce,
process, and consume information; need
to exchange information between elements; and characteristics of information
exchanged (content, media, volume requirements, security classification,
timeliness, and interoperability requirements) Operational information
exchange matrix Provides information exchange requirements, identifying who
exchanges what information
with whom, why information is necessary, and how it is needed System
interface description Links operational and systems architecture views by
depicting information systems and
their interfaces to organizational elements that produce, process, and
consume information Technical architecture profile Establishes a set of
rules governing system implementation and operation; normally,
references existing technical guidance and discusses how that guidance has
been or needs to be implemented
Source: C4ISR Architecture Framework, Version 2.
BSM officials, including the BSM program manager and chief architect, have
acknowledged that DLA does not have an enterprise architecture. However,
these officials also maintain that BSM still has had enterprisewide
architectural direction, because although the program did not have C4ISR-
compliant architectural artifacts, the strategic direction of the agency (as
defined in strategic plans) was the primary focus of BSM. 27 Nevertheless,
these officials also stated that they planned to develop a C4ISR- compliant
DLA enterprise architecture.
Subsequently, the BSM program office prepared a draft enterprise
architecture development plan that focused first on developing a materiel
management architecture, which DLA calls the BSM Business Architecture
Blueprint, as a by- product of acquiring and implementing its BSM COTS
solution. According to the draft plan, once BSM is completed (scheduled for
fiscal year 2005), DLA will add other business functions to the BSM
27 Strategic Plan 2000: DLA 21 (September 1999); Logistics Transformation
Plan (June 28, 2000). DLA?s Approach to
Developing an Architecture Is Not Enterprise Focused
Page 20 GAO- 01- 631 Information Technology
Business Architecture Blueprint so that the end result is an enterprisewide
architecture (see fig. 4). 28
Figure 4: DLA?s Approach to Creating an Enterprise Architecture
Note: Base support, although not included by DLA among its major business
functions, is included in this diagram because it is supported by a legacy
system that would have to be modernized.
Source: Enterprise Information Architecture Plan (draft, 28 July 2000), DLA.
DLA?s approach to developing and implementing a C4ISR- based, agencywide
enterprise architecture is thus one of compliance after the fact (that is,
after acquiring and implementing its system solution), rather than
development before the fact (and application during the fact). DLA plans to
fill out the essential products of the C4ISR framework as by- products of
implementing its COTS solution, rather than developing these products
beforehand, so that they could be used as the basis for guiding and
constraining its acquisition and implementation of system solutions.
28 DLA?s plan shows each of the business functions being incorporated
separately into the enterprise architecture; however, for materiel
management operations that are to be fulfilled through commercial supply
chains, some aspects of other functions (such as distribution) will be
included in the BSM Business Architecture Blueprint.
Materiel management Fuels Distribution Cataloging Base support Reutilization
Hazardous material
a To be determined. BSM
FAS CRS
Phased development
Systems: BSM = Business Systems Modernization FAS = Fuels Automated System
DSS = Distribution Standard System CRS = Cataloging Reengineering System
TBD a DSS
TBD TBD
DLA functions:
BSM Baseline 1 BSM Baseline 4
BSM Baseline 3 BSM Baseline 2
Page 21 GAO- 01- 631 Information Technology
Although DLA has not yet formally developed the seven essential products in
the C4ISR format, BSM program officials point out that some of the
information needed for these products is available, and DLA is relying on
that information in its direction of BSM. For example, BSM officials
acknowledged as enterprise requirements the following: technology
infrastructure standards, a transition strategy, technical reference models,
standards profiles, and information assurance policies at the enterprise
level. In our view, such information would be valuable and useful in
building a DLA enterprise architecture.
However, DLA nevertheless does not have a complete enterprise architecture
as defined by DOD?s C4ISR framework, and its plans for developing one are
not consistent with a fundamental best practice in CIO Council guidance.
That is, DLA?s approach is focused on only part of its enterprise (albeit
the largest), whereas CIO Council guidance promotes a process to ensure
development of an enterprisewide perspective. According to this guidance,
the scope of the architecture for the operational view needs to encompass
the entire enterprise. Only then can the agency understand the relationships
and dependencies among its business areas and position itself to make
informed decisions about the level of depth and detail to include in the
architecture. Necessarily, this level of depth and detail will be tailored
to each enterprise based on its size and complexity, as well as the purpose
of the architecture and the riskiness of the modernization effort to be
undertaken.
Although DLA?s approach does not preclude success, it does introduce a risk
of misalignment between the COTS solution and the enterprise?s strategic
operational vision. To manage this risk, the enterprise would have to fully
evaluate the COTS solution against the business requirements for the whole
enterprise, not just a portion of it. Such an enterprisewide perspective is
what Strategic Plan 2000: DLA 21 and the Logistics Transformation Plan
envision. (For instance, DLA 21 refers to logistics solutions that are
integrated from two standpoints: throughout the supply chain and throughout
the agency.) An approach that lacks an enterprisewide perspective does not
adequately ensure that the COTS system solution being acquired and
implemented will optimally support all logistics business functions.
Despite the existence of some enterprisewide architectural information,
DLA?s plan for building an architecture for its future operational, systems,
and technical environment does not have an enterprisewide perspective.
Instead, DLA is defining and implementing its architecture by first focusing
on one of its six business functions (materiel management)
Page 22 GAO- 01- 631 Information Technology
separately from the others. For DLA, this is particularly problematic
because DLA?s six business functions are interrelated and interdependent.
For example:
Materiel management and cataloging are related, because how an item is
procured is determined in part by how an item is classified during the
cataloging process.
Materiel management and distribution are related, because decisions about
whether DLA will distribute the materiel (or the vendor will deliver
directly to the requisitioner) are made at the same time the materiel is
bought.
Distribution and cataloging are related, because how materiel is stored
and transported is determined by how it is classified during cataloging (e.
g., hazardous materiel has special storage and transportation needs).
DLA adopted its approach to developing and implementing an enterprise
architecture because the agency believed that using materiel management- its
largest business process- as a starting point would lead to a solution with
a baseline functionality that could be augmented as DLA evolves. Further,
DLA expected that addressing materiel management would lead to the inclusion
of some aspects of its other business functions, such as distribution and
cataloging, for that part of materiel management operations to be performed
via private sector supply chains.
However, DLA also recognized that postponing consideration of some business
functions might lead to disconnects later in the development. According to
DLA, if the later DLA business functions could not be incorporated into the
BSM Business Architecture Blueprint without major redesign, then DLA would
construct linkages between BSM and the other business areas to produce an
enterprisewide architecture. This approach introduces the risk that DLA will
sacrifice optimizing enterprisewide performance and accountability in order
to optimize the performance and accountability of its individual components-
the risk that well developed and implemented enterprise architectures are
intended to prevent.
At the conclusion of our review, DLA officials stated that they intend to
expand the scope of their enterprise architecture program to provide for an
agencywide, C4ISR- compliant architecture.
Page 23 GAO- 01- 631 Information Technology
CIO Council guidance defines a set of recognized key practices (management
structures and processes) for developing and implementing an enterprise
architecture that are hallmarks of successful public and private sector
organizations. Among other things, these practices include the following:
Because the enterprise architecture is a corporate asset for
systematically managing institutional change, the head of the enterprise
should support and sponsor the architecture, giving it a clear mandate in
the form of an enterprise policy statement. Such support is crucial to
gaining the commitment of all organizational components of the enterprise,
all of whom should participate in developing and implementing the enterprise
architecture.
The enterprise architecture effort should be directed and overseen by an
executive body, empowered by the head of the enterprise, with members who
represent all stakeholder organizations and have the authority to commit
resources and to make and enforce decisions for their respective
organizations.
The enterprise architecture effort should be led by a Chief Architect who
reports to the enterprise CIO, and it should be managed as a formal program.
A formal program entails creating a program office, committing core staff,
implementing a program management plan that details work breakdown structure
and schedules, allocating resources and tools, performing basic program
management functions (e. g., risk management, change control, quality
assurance, configuration management), and tracking and reporting progress
against measurable goals.
The enterprise architecture should conform to a specified framework. DLA
is following some of these recognized key practices: it plans to follow the
DOD C4ISR Architecture Framework, and a BSM Chief Architect has been
designated. However, DLA is not following most key practices. It is not, for
example,
approaching the architecture as a corporate endeavor, with explicit
support and sponsorship from the DLA Director in the form of an agency
policy statement;
using an executive body consisting of stakeholders from across DLA to
direct and oversee the architecture effort; or
managing the architecture effort as a formal program. Additionally, while
an architecture development plan was drafted following our inquiries,
neither the DLA CIO nor any DLA executive body has taken any action to
approve the draft plan (dated July 2000), and DLA Does Not Have
Effective Management Structures and Processes for Developing and
Implementing an Enterprise Architecture
Page 24 GAO- 01- 631 Information Technology
thereby demonstrate their commitment and accountability for the
architecture.
Moreover, the Chief Architect is assigned to the BSM program rather than to
an organization with authority and responsibility for DLA- wide logistics
modernization, and the draft architecture development plan focuses solely on
materiel management, only providing a notional sequence for possibly
incorporating DLA?s five other business functions at some time after 2005.
Further, no work breakdown structures and schedules are provided for adding
these other business functions, no estimates of resources are provided for
implementing the plan, no measures are provided for managing progress in
developing the architecture, and no enterprise stakeholders outside the BSM
program participated in drafting the plan.
At the conclusion of our review, DLA officials stated that they will expand
the architecture program to provide an agencywide focus, assign
responsibility and accountability for the program accordingly, and formalize
their approach to its management.
Beyond the need for a DLA organization- based enterprise architecture, DLA
needs a DOD- wide logistics enterprise architecture, which would also serve
DOD?s other component organizations that have interrelated and
interdependent logistics management missions. However, DOD does not yet have
such an architecture, although it has begun efforts to develop one.
Despite the lack of a DOD- wide logistics architecture, in March 2000, the
former Deputy Secretary of Defense directed DLA and the military services to
develop plans for modernizing their respective logistics processes and
systems by July 2001 and to implement their respective plans by September
2006. No DOD- wide architectural blueprint to promote operational and system
commonality and integration accompanied this direction.
According to the Director of Logistics Systems Modernization within the
Office of the Deputy Under Secretary of Defense for Logistics and Materiel
Readiness (the DOD organization responsible for overseeing the
implementation of systems modernization efforts in 21st Century Logistics),
DOD needs a departmental logistics management enterprise architecture to
achieve its logistics management vision and optimize DODwide logistics
management performance and accountability. Without such an architecture, the
Director stated that the services? and DLA?s DOD Does Not Have a
Departmental Logistics Management Enterprise Architecture
Page 25 GAO- 01- 631 Information Technology
investments in new logistics management systems would result in operations
and systems that, although modernized, would continue to be
?stovepiped,? rather than optimized and integrated across DOD. Accordingly
and as an outgrowth of Year 2000 testing, efforts were started to develop a
DOD- wide logistics enterprise architecture concurrently with DOD
components? efforts to modernize their respective operations and systems.
According to the Director, modernization of component operations and systems
is the single most important factor in being able to effectively and
efficiently optimize and integrate operations and systems across DOD. The
Director?s position is that components? modernization efforts and DOD- wide
logistics architecture efforts should proceed in parallel.
To date, according to the Director,
a plan has been drafted for creating a departmental logistics management
enterprise architecture;
a description of the department?s current or ?as is? logistics management
enterprise architecture is being developed; and
descriptions of both near- term and long- term future (? to be?) logistics
management enterprise architectures are being developed.
However, the Director would not provide us a copy of the enterprise
architecture development plan, noting that it has yet to be approved and
that a milestone for completing the enterprise architecture was not
available. He also would not provide us copies of any ?as is? architectural
products, acknowledging that these products were based on data obtained
under DOD?s Year 2000 Conversion Program and thus needed to be modified in
light of the ongoing modernization activities in the military service and
DLA. He added that the near- term and long- term ?to be? architecture
efforts are being managed by two different organizations within the Office
of the Deputy Under Secretary for Logistics and Materiel Readiness, and that
these architectures are to be based on yet- to- bedeveloped business rules
and operational and technical agreements among the services and DLA. The
Director specifically cited efforts under way to develop an architecture for
DOD- wide management of conventional ammunition, DOD?s most critical
commodity for the warfighter, as the prototype effort for developing a
logistics management architecture and policies for architecture- based
portfolio investment management.
The Director also told us that the Office of the Deputy Under Secretary of
Defense for Logistics and Materiel Readiness would control these investments
using traditional means, such as exercising oversight of the
Page 26 GAO- 01- 631 Information Technology
acquisition and budget processes. This oversight role will be augmented,
according to the Director, by a DOD enterprise integration consortium,
established in December 2000, made up of component organizations that are
pursuing logistics enterprise resource planning system solutions, and led by
the Director. The purpose of this consortium is to update plans for
community services and DOD- wide logistics rules, develop enterprisewide
interfaces between logistics and other functional areas, and develop
mechanisms for sharing development plans and technical artifacts among
programs and for sharing lessons learned.
These controls, however, are not sufficient substitutes for having a
welldefined enterprise architecture. As discussed earlier, an enterprise
architecture defines the business and technical rules, standards, and
protocols that govern how the entire enterprise will operate in the future,
and provides a common road map for getting to this future operational state.
It thus serves as the explicit, documented change management tool that the
Office of the Deputy Under Secretary for Logistics and Materiel Readiness
needs to effectively discharge the very acquisition and budget oversight
role that the Director cited. Without an enterprise architecture to guide
and constrain the components? modernization efforts, there is not adequate
assurance of common understanding of the nature and content of the change.
At the conclusion of our review, an official in the Deputy Under Secretary?s
office acknowledged that DOD needs to move swiftly to develop and use a DOD-
wide logistics management architecture.
Incremental investment management involves three fundamental components: (1)
developing/ acquiring a large system in a series of smaller projects or
system increments; (2) individually justifying investment in each separate
increment on the basis of costs, benefits, and risks; and (3) monitoring
actual benefits achieved and costs incurred on ongoing increments and
modifying subsequent increments/ investments to reflect lessons learned. 29
Using this approach prevents agencies from discovering too late that their
systems are not cost beneficial and allows them to reduce the enormous risks
associated with large, expensive projects. Such an approach does not
preclude overlap and smooth transition among
29 Customs Service Modernization: Serious Management and Technical
Weaknesses Must Be Corrected (GAO/ AIMD- 99- 41, February 26, 1999). DLA Has
Not Invested
Incrementally in BSM
Page 27 GAO- 01- 631 Information Technology
increments, because lessons learned from the actual results of ongoing
increments should be monitored and evaluated continuously so that these
results are available for use in defining and justifying future increments.
DLA has not yet followed an incremental approach to investment in BSM. To
date, DLA has treated the entire BSM program as one investment decision,
justified by a single economic analysis, because this approach was
consistent with DOD policy for major system acquisitions 30 that were in
effect until January 2001. DLA invested in its preliminary work (the
analysis of candidate COTS systems against previously reengineered business
processes) and in the initial release of BSM on the basis of this economic
analysis. As a result, DLA did not justify and decide on investing about
$150 million thus far in BSM on the basis of release- specific analyses of
costs, benefits, and risks.
This kind of approach to making investment decisions has historically
resulted in agencies investing huge sums of money in systems that do not
provide commensurate benefits, and thus has been abandoned by successful
organizations. The need to avoid this pitfall was a major impetus for the
Clinger- Cohen Act investment management reforms.
At the conclusion of our review, DLA officials told us that in future, they
plan to make investment decisions on BSM incrementally.
The Clinger- Cohen Act of 1996 requires agencies to follow, to the maximum
extent practicable, an incremental approach to investing in IT development/
acquisition projects. Additionally, OMB policy requires that investments in
major information systems be implemented in increments, with each increment
delivering measurable benefits. 31 More specifically, OMB?s Capital
Programming Guide 32 describes the use of modular contracting or incremental
investment, including its application and benefits. In particular, OMB
states that project increments should provide for the following:
30 DOD Instruction 5000. 2, Operation of the Defense Acquisition System.
31 Memorandum for Heads of Executive Departments and Agencies, OMB
Memorandum M- 97- 02 (October 25, 1996); Capital Programming Guide, Version
1. 0, OMB Circular A- 11, part 3 (July 1997), pp. 545- 572.
32 Capital Programming Guide, Version 1. 0, OMB Circular A- 11, part 3,
Supplement,
Planning, Budgeting, and Acquisition of Capital Assets (July 2000), pp. 35-
37. Incremental Investment
Management Spreads the Risk of Large Programs Across Smaller, Incremental
Parts
Page 28 GAO- 01- 631 Information Technology
Separability: Each increment should be fully funded, have substantial
programmatic use that is not dependent on any subsequent increment, and be
capable of performing its principal functions even if no subsequent
increments are acquired.
Interoperability: Each increment should comply with a common architecture
or commercially acceptable technology standards and should be compatible
with and capable of being integrated with other increments.
Performance requirements: Each increment?s performance requirement should
be consistent with the performance requirements of the completed overall
system and should address interface requirements with other increments.
In short, incremental investment helps to mitigate the risks inherent in
large IT acquisitions/ developments by breaking apart a single large project
into smaller, independently useful components with known and defined
relationships and dependencies. Making investment decisions up front on
large- scale, long- term projects is generally risky: their economic
justification is based on costs, benefits, and risks that are difficult to
forecast reliably, partially because later increments are not well
understood or defined, and partially because they are subject to change in
light of experience on nearer term increments and changing business needs.
Through incremental investment management, organizations can
reduce the level of project risk and complexity faced at any one time by
spreading this risk and complexity across a series of smaller investments;
permit the delivery of some part of the expected business value earlier
rather than waiting until later for the total, but more uncertain, business
value;
continuously monitor and evaluate the delivery of cost and benefit
expectations on ongoing increments and use this information to better define
and economically justify these increments; and
permit later increments to exploit technology advances or accommodate
evolving business needs.
In January 2001, DOD issued a change to its major system acquisition policy
33 requiring incremental investment management; specifically, the policy
notes that a program?s milestone decision authority must verify that each
increment meets part of the mission need and delivers a measurable benefit,
independent of future increments.
33 DOD Instruction 5000. 2, Change 1, Operation of the Defense Acquisition
System.
Page 29 GAO- 01- 631 Information Technology
Although DLA plans to acquire and implement its BSM system solution in four
increments, it has not so far managed BSM investments incrementally.
Specifically, (1) DLA has not justified proposed investment in the first BSM
increment on the basis of costs and benefits, (2) the DLA 21 Executive Board
34 has not made decisions about whether to invest in each discrete BSM
increment, and (3) the BSM program office has not measured actual return on
investment from each increment and used the results to better inform
decisionmaking about future increments.
DLA?s position is that BSM was being managed according to DOD Instruction
5000.2, which until January 2001 required a single economic analysis of a
system?s life- cycle costs and benefits before the system entered the
concept demonstration phase. Accordingly, DLA prepared a program life- cycle
economic analysis for BSM, according to which the program is expected to
produce a positive net present value over its 15- year life cycle and
improve military readiness. As a result, DLA officials stated that investing
about $900 million to acquire and implement the BSM system solution is
prudent, and DLA committed itself to funding, acquiring, and implementing
all four BSM phases or increments.
Relying on a single economic analysis to make an investment decision for a
program that is large and risky, involving many things to be done over many
years, is neither prudent nor consistent with the principles of incremental
investment management embodied in the Clinger- Cohen Act and OMB
requirements. Approaching the BSM investment decision in this way was
especially risky, since at the time that DLA prepared the BSM program life-
cycle economic analysis, it had defined BSM business requirements for only
the reengineered part of the materiel management business area. Thus, DLA
was not only attempting the daunting task of accurately estimating the costs
and benefits of many things over many years, it was doing so without knowing
its BSM requirements beyond the reengineered parts of the materiel
management business area. Our experience in reviewing IT investments across
the federal government has
34 This board is DLA?s senior investment decisionmaking body. It is
responsible for investment selection, control, and evaluation decisions for
all investments. DLA Has Not Yet Made
Investment Decisions Incrementally
Page 30 GAO- 01- 631 Information Technology
shown that such estimating does not produce reliable results and cannot
provide a sufficient basis for informed investment decisionmaking. 35
To date, the BSM program office has not analyzed the costs, benefits, and
risks of what DLA defines as its first BSM release. Program officials told
us that they justified this release, which is to cost $93 million (not
including the $56 million that was spent on BSM before the first release),
on the basis of the program life- cycle economic analysis.
At the conclusion of our review, however, DLA officials informed us that
they do now plan to make decisions on the last three releases incrementally:
they plan to justify release 2 on the basis of its costs, benefits, and
risks, and to verify that the first release produced benefits commensurate
with costs before deciding to invest in release 2. They also stated that
they would follow this approach to investing in later releases. According to
program officials, because DLA is still working on its first release
(concept demonstration), the investment decision for release 2 will not be
made until fiscal year 2002.
However, the program office does not yet have plans or measures to
determine, once an increment is implemented, whether its expected value is
actually accruing. The implementation plans for the first BSM increment
(release 1) were completed in April 2001, but no specific plans or measures
were developed to determine whether benefit versus cost expectations would
be met for this increment. Clinger- Cohen requires that agencies identify
quantifiable measurements for determining costs and benefits. Until the
program office has some means of measuring expected value, DLA will not be
able to determine whether this first increment is performing at a level
equal to or better than its current materiel management systems- SAMMS and
DISMS.
According to the BSM program manager, the investment management plans and
measures are not in place because DLA?s policies and procedures on
investment portfolio management are still in draft. When
35 Air Traffic Control: Complete and Enforced Architecture Needed for FAA
Systems Modernization (GAO/ AIMD- 97- 30, February 3, 1997); Air Traffic
Control: FAA?s Modernization Investment Management Approach Could Be
Strengthened (RCED/ AIMD99- 88, April 1999); Customs Service Modernization:
Architecture Must Be Complete and Enforced to Effectively Build and Maintain
Systems (GAO/ AIMD- 98- 70, May 5, 1998);
Information Technology: INS Needs to Strengthen Its Investment Management
Capability
(GAO- 01- 146, December 2000); Information Technology Management: Coast
Guard Practices Can Be Improved (GAO- 01- 190, December 2000).
Page 31 GAO- 01- 631 Information Technology
these are completed, they should govern, for example, (1) how investment
increments are to be defined, prioritized, and sequenced, (2) how
performance criteria are to be applied to each increment, and (3) how
accrued earned value from each investment increment will be measured.
Until these investment management plans and measures are in place, the
program manager stated that incremental investment will be accomplished
through a series of 11 planned contract task orders, consisting of 105 more
specific subtasks, which are to span the four BSM increments. However,
merely structuring contract task orders into incremental pieces is not a
sufficient substitute for making investment approval and funding decisions
incrementally.
DLA has already invested significant sums of money in its BSM program, and
its plans call for investing hundreds of millions of dollars more. To invest
this money wisely and in a way that minimizes risks, DLA will need to
acquire BSM within the context of an enterprise architecture and on an
incremental basis. To date, DLA has not used an enterprise architecture or
applied management controls for developing, implementing, and maintaining
one, as described in CIO Council guidance. Additionally, DLA has not
employed incremental investment practices.
Officials of DLA and the Office of the Secretary of Defense have committed
to correcting these management weaknesses. We view these commitments as
positive first steps for two primary reasons. First, if DLA continues to
invest large sums of money in BSM without a DLA- or DODwide logistics
enterprise architecture, it runs the serious risk that it will acquire a
system solution that focuses narrowly on DLA materiel management performance
and accountability at the expense of DLA- and DOD- wide performance and
accountability. Second, until DLA begins justifying the return on its BSM
investment incrementally, making BSM investment decisions incrementally, and
measuring actual return on this investment incrementally, it will not be
able to make well- informed and prudent investment decisions, and it will
not know whether its BSM system solution is cost- effective until it has
already spent hundreds of millions of dollars.
To address DLA?s need for an enterprise architecture to guide and constrain
its IT capital investments, including BSM, we recommend that the Secretary
of Defense direct the DLA Director to designate and treat Conclusions
Recommendations
Page 32 GAO- 01- 631 Information Technology
development, implementation, and maintenance of a DLA enterprise
architecture as an agency priority.
We recommend that in fulfilling this direction from the Secretary, the DLA
Director (1) issue a policy governing development, implementation, and
maintenance of an enterprise architecture and (2) establish a DLA enterprise
architecture steering committee, chaired by the DLA Vice Director, to
provide a DLA- wide direction and focus to the enterprise architecture, and
to ensure that one is developed and maintained in a manner consistent with
the CIO Council published guide on managing enterprise architectures.
To provide for the effective development and maintenance of the DLA
enterprise architecture, we recommend that the DLA Director make the CIO
accountable to the enterprise architecture steering committee for developing
and maintaining the agency?s enterprise architecture. We recommend that in
fulfilling this responsibility, the CIO appoint a Chief Architect for DLA
and establish a program office responsible for developing and maintaining a
DLA- wide enterprise architecture. We further recommend that the CIO direct
the Chief Architect to work collaboratively with the appropriate offices
within the Office of the Deputy Under Secretary for Logistics and Materiel
Readiness to appropriately align the DLA enterprise architecture with this
office?s ongoing efforts to develop a DOD- wide logistics enterprise
architecture. Additionally, we recommend that the CIO have the Chief
Architect, as appropriate, follow the steps outlined in the CIO Council?s
published guide for managing enterprise architectures, including
obtaining executive buy- in and support,
establishing architecture management structure and controls,
defining the architecture process and approach,
developing the baseline architecture, the target architecture, and the
sequencing plan,
facilitating the use of the architecture, and
maintaining the architecture. To ensure that, once developed, the
enterprise architecture is effectively implemented, we also recommend that
the Director require the DLA 21 Executive Board and its subsidiary
investment review boards to ensure that only architecturally compliant IT
investments are approved and funded, unless the investment decisionmaking
authority issues a written waiver in response to a written justification.
Page 33 GAO- 01- 631 Information Technology
Until a DLA enterprise architecture is developed and can be used to
effectively guide and constrain DLA IT investment, we also recommend that
the DLA Director limit future investment in BSM to only its first
incremental release.
To address DLA?s need to incrementally invest in BSM, we recommend that the
DLA Director impose three further conditions on investment in BSM beyond its
first increment. Specifically, such investment should not occur until plans
have been developed and processes implemented for (1) measuring and
validating whether ongoing BSM increments are actually producing benefits
commensurate with costs, (2) developing economic justifications for each
subsequent increment that consider the actual performance of ongoing
increments, and (3) ensuring that decisions on each subsequent increment are
based on these incremental economic justifications.
To ensure that the approach to DLA?s logistics modernization (as well as
that of other DOD component organizations) recognizes the logistics
management interdependencies and interrelationships among DOD components and
aims to optimize departmentwide performance and accountability, we recommend
that the Secretary of Defense direct the Under Secretary of Defense for
Acquisition, Technology, and Logistics to (1) expedite development of a DOD-
wide logistics management enterprise architecture in a manner consistent
with CIO Council published guidance and (2) establish effective controls for
ensuring that component investments in modernized logistics systems are
compliant with the architecture or are otherwise granted an explicit waiver
to this requirement on the basis of analytical justification.
In written comments on a draft of this report (reprinted in appendix II),
the acting Deputy Under Secretary of Defense for Logistics and Materiel
Readiness agreed with us on the importance of developing and maintaining
enterprise architectures and making BSM investment decisions incrementally,
concurring with eight of our ten recommendations. DOD also acknowledged that
the absence of these IT management controls increased program risk. Despite
these areas of substantial agreement, the Deputy Under Secretary disagreed
with our recommendations that the DLA Director limit future investment in
BSM to the first incremental release until (1) a ?full- blown? (as DOD
termed it) DLA enterprise architecture is developed and (2) the BSM program
manager has the means in place to measure actual benefit accrual on the
Agency Comments
and Our Evaluation
Page 34 GAO- 01- 631 Information Technology
first incremental release and to use this information in deciding on
investment in the next release.
According to DOD, our findings and conclusions regarding the overall level
of risk to the successful achievement of BSM and other DOD logistics
modernization program objectives are not supported by the single risk factor
addressed in this report- the use of enterprise architectures. DOD?s stated
view is that the risks associated with delaying BSM are greater than the
risks of modernizing while concurrently developing the architecture. DOD
also stated that while the conditions that we recommended regarding
measurement and use of actual benefit accrual are valuable and will be
performed by DOD, they should not be performed by the program manager and
should not be on the critical path for undertaking future incremental
releases.
While DOD agrees that the absence of an ?end- to- end? enterprise
architecture for DLA is a risk, DOD strongly believes that this risk is
amply mitigated by measures already taken. As examples, the Deputy Under
Secretary?s comments refer to the preparation of an enterprisewide logistics
implementation plan for community data management services that defines the
future information and business rules interaction among DOD components; the
existence of an ?as is? logistics architecture (documented during the Y2K
campaign); the recent establishment of an OSD- led enterprise integration
consortium that, according to the comments, will ensure optimal
collaboration among programs; and the commercial architecture and process
reengineering procedures being followed by DLA?s BSM program.
Notwithstanding the considerable level of agreement between DOD and us on
this report, we have a differing view on the level of risk facing DOD
logistics modernization programs in general, and BSM in particular, in the
absence of the two risk mitigators addressed in this report: (1) effective
enterprise architecture development and implementation and (2) effective
incremental investment management.
Our experience in reviewing other federal agency modernization programs has
shown that the absence of these program management controls are two of the
root causes for other programs? lack of success. 36 The two recommendations
that DOD disagrees with are intended to strike a
36 GAO/ AIMD- 97- 30, RCED/ AIMD- 99- 88, GAO/ AIMD- 98- 70, GAO- 01- 146,
and GAO- 01- 190.
Page 35 GAO- 01- 631 Information Technology
reasonable balance between the risk that DOD will suffer the same fate as
these other agencies and the risk that DOD points out of delaying BSM. These
recommendations permit DLA to make progress on the first BSM increment while
developing and implementing the two missing management controls and thus
becoming better equipped to move forward on subsequent increments.
Therefore, we continue to believe that DOD should limit investment in BSM
beyond the first increment until certain conditions are met.
We agree with the Deputy Under Secretary that the risk mitigation strategies
that DOD has followed can lower the risk associated with the lack of a
complete enterprise architecture. However, we continue to believe that these
efforts are not a sufficient substitute for having an explicit enterprise
architecture and effectively using it to guide and constrain modernization
investment decisions. At most, these mitigation strategies should be viewed
as temporary controls until the architecture is available, as well as useful
starting points for the development and use of an enterprise architecture.
With regard to risk mitigation, the Deputy Under Secretary comments further
that commercial implementation of enterprise resource planning (ERP) is a
proven means for achieving an enterprise business strategy, adding that a
key requirement of DLA?s BSM program is to move to a commercial approach for
achieving its end- state enterprise architecture. We agree that the use of
commercial products can be a less risky approach than the development of
custom applications, but we believe that success still depends on effective
implementation and in particular on ensuring that the chosen products meet
the needs of the enterprise; this requirement is best fulfilled by having
and using an enterprise architecture. The main point of our recommendations
is that DLA has yet to define its end- state enterprise architecture and
therefore cannot know whether its chosen ERP solution will satisfy
requirements beyond the materiel management business area. Accordingly, we
stand by our recommendation in this area, which is intended to significantly
lower the risk for the BSM program in relation to its role in developing
DLA?s enterprise architecture, not just in achieving DLA?s requirements for
modernizing its materiel management systems.
We would add that DOD?s characterization of the enterprise architecture that
we recommend as ?full- blown? and ?end- to- end? suggests that we are
prescribing the full depth and detail of this architecture. We are not. In
fact, effective architecture management practices recognize that there is no
one- size- fits- all architecture solution and that the driving goal in
Page 36 GAO- 01- 631 Information Technology
developing an architecture is to make it useful for the task at hand,
meaning that its depth and detail will vary from enterprise to enterprise.
The Deputy Under Secretary also disagreed with our recommendation to impose
additional conditions on investment in BSM beyond its first increment,
specifically concerning the development of processes for performing cost/
benefit analyses of ongoing and completed increments. According to the
Deputy Under Secretary, the BSM program does plan to prepare a business case
to support each of four scheduled releases and to include in each of these
increments a projection of business process performance and an evaluation of
empirical results of the program to that point. These are the very
activities that we recommend as conditions for investing beyond the first
increment. The Deputy Under Secretary?s disagreement with our recommendation
appears to center on who should be required to measure and use actual
benefit accrual in making decisions on continuing the program to the next
increment. It is not our intent to suggest that only the BSM program manager
should measure and use actual benefit accrual or that one increment must be
fully completed and fully evaluated before another can begin. In fact,
effective incremental investment management necessitates some overlap among
increments, and our recommendations do not contradict this. On the matter of
who should measure and validate actual benefit accrual, we are not opposed
to DLA assigning this responsibility to someone other than the program
manager, and we acknowledge that the program manager should not be solely
responsible for ensuring that investment decisions on subsequent increments
are based on economic justifications. Accordingly, we have modified the
recommendation to reflect this.
DOD also provided specific comments on other aspects of the report. These
comments have been incorporated throughout the report where appropriate.
We are sending copies of this report to the Chairmen and Ranking Minority
Members of the Senate Appropriations Subcommittee on Defense; the
Subcommittee on Readiness and Management Support, Senate Committee on Armed
Services; the House Appropriations Subcommittee on Defense; and the
Subcommittee on Readiness, House Committee on National Security. We are also
sending copies to the Director, Office of Management and Budget, and the
Director, Defense Logistics Agency. Copies will be made available to others
upon request.
Page 37 GAO- 01- 631 Information Technology
If you have any questions regarding this report, please contact me at (202)
512- 6256 or Carl M. Urie, Assistant Director, at (202) 512- 6231. We can
also be reached by e- mail at hiter@ gao. gov and uriec@ gao. gov. Other key
contributors to this report are listed in appendix III.
Randolph C. Hite Director, Information Technology Systems Issues
Appendix I: Objectives, Scope, and Methodology
Page 38 GAO- 01- 631 Information Technology
Our objectives were to determine (1) whether DLA is using an enterprise
architecture to guide and constrain its investment in its Business Systems
Modernization (BSM) program and (2) whether DLA is investing in BSM in an
incremental manner. Using enterprise architectures- institutional blueprints
for business and technological change- and investing incrementally in large
modernization programs are legislative and federal requirements and best
industry practices. This review did not address other system modernization
best practices, such as whether DLA is employing effective system
acquisition process controls, and did not evaluate the specific commercial,
off- the- shelf product that DLA chose as its BSM system solution.
To determine whether DLA has an enterprise architecture to guide BSM, we
identified the DOD organizations involved in efforts to reform and modernize
DOD logistics management operations and systems, as well as organizations
responsible for DOD policy and guidance on enterprise architectures,
including the Office of the Under Secretary of Defense (Acquisition,
Technology, and Logistics); the Director of Logistics Systems Modernization;
the DLA Logistics Operations Directorate; and the DLA Director of
Information Operations, who is in charge of corporate information policy.
From each of these organizations, we solicited information on plans and
activities that defined the form and content of these reform and
modernization efforts. We then questioned officials from each organization
about planned and existing architectural artifacts and obtained copies of
all such plans and artifacts. Next, we analyzed the information provided,
including DOD?s Joint Vision 2020, 21st Century Logistics: DOD Logistics
Strategic Plan, and available DLA enterprise architecture artifacts, against
DOD?s C4ISR Architecture Framework to determine the extent to which these
organizations individually or collectively had produced architectural
artifacts that satisfied DOD requirements. In our analysis, we also
considered other published architectural guidance, including Office of
Management and Budget memorandums and the Federal Enterprise Architecture
Framework of the Chief Information Officers Council. 1
We also obtained and reviewed the federal CIO Council?s A Practical Guide to
Federal Enterprise Architecture, Version 1.0, as well as
1 Funding Information Systems Investments, OMB Memorandum M- 97- 02 (October
25, 1996); Information Technology Architectures, OMB Memorandum M- 97- 16
(June 18, 1997);
Federal Enterprise Architecture Framework, Version 1. 1, Chief Information
Officers Council (September 1999). Appendix I: Objectives, Scope, and
Methodology
Appendix I: Objectives, Scope, and Methodology
Page 39 GAO- 01- 631 Information Technology
published information on the enterprise architecture best practices that the
guide is based upon. We compared DLA?s Enterprise Information Architecture
Plan (draft, 28 July 2000) with this guidance.
To determine whether DLA is following an incremental investment strategy for
BSM, we compared DLA?s Single Acquisition Management Plan, Operational
Requirements Document, and other program- related material against the
Clinger- Cohen Act, OMB?s Capital Programming Guide, and the associated
assessment method. We evaluated policies, procedures, and guidance related
to DLA?s IT modular contracting activities. We evaluated task order plans
and performance data against commonly accepted IT investment methodologies.
Additionally, we reviewed economic analysis of DLA?s BSM program and the
related cost justifications. We also used DOD?s Logistics Transformation
guide and the Defense Fiscal Year 2001 Budget to obtain information about
other DOD and military service supply- chain logistics initiatives.
We conducted our work at DLA headquarters, located at Fort Belvoir, VA, from
February to April 2001 in accordance with generally accepted government
auditing standards.
Appendix II: Comments From the Department of Defense
Page 40 GAO- 01- 631 Information Technology
Appendix II: Comments From the Department of Defense
See comment 2. See comment 1.
Appendix II: Comments From the Department of Defense
Page 41 GAO- 01- 631 Information Technology
See comment 5. See comment 4.
See comment 3.
Appendix II: Comments From the Department of Defense
Page 42 GAO- 01- 631 Information Technology
See comment 7. See comment 6.
Appendix II: Comments From the Department of Defense
Page 43 GAO- 01- 631 Information Technology
See comment 8.
Appendix II: Comments From the Department of Defense
Page 44 GAO- 01- 631 Information Technology
See comment 9.
Appendix II: Comments From the Department of Defense
Page 45 GAO- 01- 631 Information Technology
See comment 10.
Appendix II: Comments From the Department of Defense
Page 46 GAO- 01- 631 Information Technology
See comment 13. See comment 12.
See comment 11.
Appendix II: Comments From the Department of Defense
Page 47 GAO- 01- 631 Information Technology
See comment 15. See comment 14.
Appendix II: Comments From the Department of Defense
Page 48 GAO- 01- 631 Information Technology
See comment 17. See comment 16.
Appendix II: Comments From the Department of Defense
Page 49 GAO- 01- 631 Information Technology
See comment 19. See comment 18.
Appendix II: Comments From the Department of Defense
Page 50 GAO- 01- 631 Information Technology
See comment 21. See comment 20.
Appendix II: Comments From the Department of Defense
Page 51 GAO- 01- 631 Information Technology
See comment 24. See comment 23.
See comment 22.
Appendix II: Comments From the Department of Defense
Page 52 GAO- 01- 631 Information Technology
See comment 26. See comment 25.
Appendix II: Comments From the Department of Defense
Page 53 GAO- 01- 631 Information Technology
See comment 28. See comment 27.
Appendix II: Comments From the Department of Defense
Page 54 GAO- 01- 631 Information Technology
1. We agree that the BSM program should not be held ?hostage? to the
architecture and that the program and the architecture can proceed in
parallel, but only to a point. As we have recommended, this point is the
start of the next BSM increment, which according to DLA is August/ September
2002. Our recommendation affords DLA ample time to develop and use the
missing architectural definition without delaying the program.
2. We disagree that this report draws conclusions about the program?s
?overall? risk based on a single risk factor. Our report is not intended to
and does not provide a comprehensive BSM risk profile. Rather, it identifies
important risks associated with the two IT management controls that were the
focus of our work: (1) the use of an enterprise architecture and (2) the use
of incremental investment controls. Further, while we acknowledge that DLA
has taken risk mitigation measures, as cited in the draft report provided to
DOD for comment, these measures are not a sufficient substitute for having
an explicit enterprise architecture and effectively using it to guide and
constrain modernization investment decisions. At most, they should be viewed
as temporary controls until the architecture is available. In fact, while
DOD believes that it cannot delay modernization, it nevertheless states in
its comments that it supports the need for enterprise architectures and the
integration of these architectures into its investment management processes.
3. We agree that one should not conclude from this report that there is
unacceptable risk associated with modernizing systems while developing an
enterprise architecture. We also agree that there are risks associated with
delaying modernization. Accordingly, our recommendation strikes a reasonable
balance between the need for an architecture and the need to modernize
legacy systems by allowing DOD to continue modernization while it develops
an architecture. However, it does not allow the two to proceed in parallel
indefinitely, because the longer DLA proceeds without an architecture, the
greater the risks will become.
4. We do not disagree and our report does not question the use of an ERP
solution. Our point is that DLA has yet to define its end- state enterprise
architecture and therefore cannot be sure that it is implementing its chosen
ERP solution in a manner that will not suboptimize DLA enterprisewide
performance and accountability, in order to optimize performance and
accountability of one business area. GAO Comments
Appendix II: Comments From the Department of Defense
Page 55 GAO- 01- 631 Information Technology
5. See comments 1 and 2. Also, we do not agree that BSM is currently guided
by sound architectural discipline. Such discipline would be evidenced by the
existence and use of architectural artifacts developed in accordance with
the practices outlined in CIO Council guidance. 1 While we acknowledge the
existence of some enterprise architecture information, as cited in the draft
report provided to DOD for comment, and state in this report that that
information would be valuable and useful in building a DLA- wide
architecture, the fact remains that DLA does not currently have such an
architecture.
6. We acknowledge that the Deputy Under Secretary of Defense for Logistics
and Materiel Readiness has pursued risk- reducing strategies, as noted in
this report. However, they are not a sufficient substitute for a DOD- wide
logistics architecture.
7. We agree. The incremental investment actions described, if effectively
implemented, are consistent with our recommendations.
8. See comments 1 and 2. 9. See comment 1. 10. We generally agree. The
activities described in the comment are the
very activities that we recommend as conditions for investing beyond the
first increment. Further, it was not our intent to suggest that only the BSM
program manager should measure and use actual benefit accrual or that one
increment must be fully completed and fully evaluated before another can
begin. In fact, effective incremental investment management necessitates
some overlap among increments, and our recommendations do not preclude this.
On the matter of who should measure and validate actual benefit accrual, we
are not opposed to DLA assigning this responsibility to someone other than
the program manager, and we acknowledge that the program manager is not the
appropriate person to ensure that investment decisions on subsequent
increments are based on economic justifications. Accordingly, we have
slightly modified the recommendation and clarified language elsewhere in the
report to reflect this.
11. We agree. We have changed the title of the report. 1 A Practical Guide
to Federal Enterprise Architecture, Version 1. 0, Chief Information Officers
Council (February 2001).
Appendix II: Comments From the Department of Defense
Page 56 GAO- 01- 631 Information Technology
12. We disagree. The statement in our report is supported by DLA?s draft
Enterprise Information Architecture Plan, which clearly shows that the
initial component of DLA?s enterprise architecture will be, by design,
limited to the materiel management business area. The plan also shows that
DLA?s other business areas are to be incorporated into the enterprise
architecture following completion of the BSM program, which, according to
BSM documents, is 5 years from now.
13. Our report has given the department credit for its DOD Logistics
Strategic Plan, as well as Joint Vision 2020 and other strategic planning
directions. However, as previously stated, these plans are not a sufficient
substitute for an effective DOD- wide logistics management enterprise
architecture. Moreover, in its comments, DOD states that having an
architecture- based approach to investing in modernized systems is a more
efficient means to optimizing enterprise performance and accountability. DOD
also states that having an architecture would provide a more efficient and
effective means for developing and overseeing the implementation of a common
vision.
14. We disagree. While we have recognized these ?interim measures? in our
report, our position is that these are just what DOD terms them- temporary
controls until DOD has in place the architecture- based approach to
modernization that its comments state are needed to achieve end- to- end
optimization. See also comment 3.
15. We disagree. Notwithstanding DLA?s recent commitments to incrementally
invest in BSM, DLA has so far treated BSM as a single investment decision.
The economic analysis used to justify DLA?s decision to invest in BSM is
dated April 2000, and this analysis was for the entire program. We requested
any further economic analyses developed to justify the ongoing BSM increment
as well as any plans to produce such analyses. DLA did not provide either.
In June 2001, DLA provided an updated economic analysis, dated March 30,
2001, which continued to treat BSM as a whole. DOD?s comments are consistent
with statements made by BSM officials at the conclusion of our review, which
were cited in the draft report. We are encouraged by DOD?s comment that DLA
will prepare a business case for each release and will include a projection
of business process performance as well as a retrospective evaluation of
empirical results of the program to that point.
16. We have modified our report to reflect this change.
Appendix II: Comments From the Department of Defense
Page 57 GAO- 01- 631 Information Technology
17. We disagree. As we state in the report, enterprise architectures are
essential tools for effectively and efficiently reengineering business
processes and for implementing and evolving their supporting systems. Their
development, implementation, and maintenance are recognized hallmarks of
successful public and private sector organizations. Congress, OMB, and the
federal CIO Council have recognized the importance of enterprise
architectures.
18. We disagree. See comments 2 and 6. Further, our report does not state
that the lack of an enterprise architecture will result in ?bad? effects.
Our report concludes that this lack increases the probability of such
effects, and our report recognizes that other IT management weaknesses can
also be contributing factors.
19. See comment 5. 20. See comment 12. 21. We disagree that our report
implies that there is little interaction and
consideration of the enterprise cataloging requirements. Our report refers
to DLA?s expectation that some aspects of other business functions would be
included in the BSM blueprint. At the time of our review, however, DLA
recognized that postponing some business functions might lead to
disconnects, requiring later linkages between BSM and other business areas
to produce an architecture with an enterprisewide scope. Our point is that
because these functions are interrelated, it makes sense to construct an
enterprise architecture to guide and constrain BSM implementation and
thereby optimize DLAwide performance and accountability.
22. See comments 2 and 21. 23. See comment 13. 24. We have modified the
report to reflect this comment. 25. We have modified the report to
incorporate the additional context
provided in this comment and to recognize when efforts to develop a DOD-
wide logistics enterprise architecture began.
26. We disagree that we have been provided a copy of the document defining
the mission- critical logistics ?threads? used in DOD?s Year 2000 testing
?in their current status.? While we did receive in 1999 a
Appendix II: Comments From the Department of Defense
Page 58 GAO- 01- 631 Information Technology
copy of the document defining these mission- critical logistics threads, the
Director also told us that these were being modified in light of ongoing
modernizations, and DOD?s comments acknowledge that they have since been
modified.
27. We have modified the report to reflect this comment. 28. See comment 23.
Also, we have slightly modified the language of our
report.
Appendix III: GAO Contact and Staff Acknowledgment
Page 59 GAO- 01- 631 Information Technology
Carl Urie, (202) 512- 6231 In addition to the person named above, Katherine
Chu- Hickman, John Christian, Barbara Collier, and Greg Donnellon made key
contributions to this report. Appendix III: GAO Contact and Staff
Acknowledgment GAO Contact Acknowledgments
(310206)
The first copy of each GAO report is free. Additional copies of reports are
$2 each. A check or money order should be made out to the Superintendent of
Documents. VISA and MasterCard credit cards are also accepted.
Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.
Orders by mail:
U. S. General Accounting Office P. O. Box 37050 Washington, DC 20013
Orders by visiting:
Room 1100 700 4 th St., NW (corner of 4 th and G Sts. NW) Washington, DC
20013
Orders by phone:
(202) 512- 6000 fax: (202) 512- 6061 TDD (202) 512- 2537
Each day, GAO issues a list of newly available reports and testimony. To
receive facsimile copies of the daily list or any list from the past 30
days, please call (202) 512- 6000 using a touchtone phone. A recorded menu
will provide information on how to obtain these lists.
Orders by Internet
For information on how to access GAO reports on the Internet, send an email
message with ?info? in the body to:
Info@ www. gao. gov or visit GAO?s World Wide Web home page at: http:// www.
gao. gov
Contact one:
Web site: http:// www. gao. gov/ fraudnet/ fraudnet. htm
E- mail: fraudnet@ gao. gov
1- 800- 424- 5454 (automated answering system) Ordering Information
To Report Fraud, Waste, and Abuse in Federal Programs
*** End of document. ***