Defense Computers: LSSC Needs to Confront Significant Year 2000 Issues
(Letter Report, 09/26/97, GAO/AIMD-97-149).
Pursuant to a congressional request, GAO reviewed: (1) the status of the
Logistics Systems Support Command's (LSSC) efforts to correct Commodity
Command Standard System (CCSS) Year 2000 systems problems; and (2) the
appropriateness of LSSC's strategy and actions for ensuring that CCSS
Year 2000 issues will be successfully addressed.
GAO noted that: (1) the Year 2000 problem is one of the most
comprehensive and complex information management projects ever faced by
LSSC; (2) if not successfully completed, the procurement of weapon
systems and their spare parts, accounting for the sales of Army
equipment and services to allies, and the financial management of $9
billion of inventory could be disrupted; (3) as a result, it could be
extremely difficult to efficiently and effectively equip and sustain the
Army's forces around the world; (4) LSSC has completed several actions
to address the CCSS Year 2000 problem; (5) a Year 2000 project manager
and management staff have been designated, a project manager charter and
schedule were developed, and supplementary contractor support was
acquired to assist with assessment tasks; (6) regularly scheduled
quarterly meetings are held by the Army Materiel Command (AMC)
headquarters to report LSSC Year 2000 status; (7) these steps are
compatible with the Department of Defense's (DOD) suggested approach and
consistent with those found in GAO's five-phased approach for planning,
managing, and evaluating Year 2000 projects; (8) although LSSC commenced
its Year 2000 project over a year ago, there are several issues facing
LSSC that, if not completely addressed, may result in the failure of
CCSS to successfully operate at the year 2000; (9) LSSC has yet to
completely address: (a) competing workload priorities and staffing
issues; (b) the appropriate mix and scheduling of needed testing data
and expertise as well as the development of test plans; (c) the scope
and substance of written interface agreements with system interface
partners to ensure that CCSS subsystems will be capable of exchanging
data at the year 2000; and (d) contingency plan development to help
assure that Army missions will be accomplished if CCSS is not fully
available to users by the year 2000; (10) LSSC's risk of failure is
increased because the agency has not attained the level of software
development and maintenance maturity that would provide the foundation
needed for successful management of large-scale projects such as the
Year 2000 initiative; and (11) because CCSS is used to support military
readiness, these critical elements must be resolved and aggressively
pursued to enable LSSC to achieve a Year 2000 compliant environment
prior to the year 2000.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: AIMD-97-149
TITLE: Defense Computers: LSSC Needs to Confront Significant Year
2000 Issues
DATE: 09/26/97
SUBJECT: Computer software verification and validation
Strategic information systems planning
Military inventories
Management information systems
Computer software
Systems conversions
Logistics
Inventory control systems
Human resources utilization
IDENTIFIER: Army Commodity Command Standard System
Army Continuing Balance System-Expanded
Army Materiel Command Year 2000 Action Plan
Software Capability Maturity Model
DOD Year 2000 Management Plan
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Director, Army Logistics Systems Support Center
September 1997
DEFENSE COMPUTERS - LSSC NEEDS TO
CONFRONT SIGNIFICANT YEAR 2000
ISSUES
GAO/AIMD-97-149
Army LSSC Year 2000
(511627)
Abbreviations
=============================================================== ABBREV
AMC - Army Materiel Command
BRAC - Base Realignment and Closure
CBSX - Continuing Balance System-Expanded
CCSS - Commodity Command Standard System
CDA - central design activities
CECOM - Communications-Electronics Command
CIM - Corporate Information Management
CMM - Capability Maturity Model
COBOL - Common Business Oriented Language
DOD - Department of Defense
LAISO - Lead AMC Integration Support Office
LSSC - Logistics Systems Support Center
MOA - memorandum of agreement
MSC - major subordinate command
SEI - Software Engineering Institute
Letter
=============================================================== LETTER
B-277732
September 26, 1997
Mr. Michael A. Whitelaw
Director, Logistics Systems Support Center
1222 Spruce Street
St. Louis, Missouri 63103
Dear Mr. Whitelaw:
On August 22, 1997, we briefed you on the results of our review to
date of the Logistics Systems Support Center's (LSSC) program for
solving the Commodity Command Standard System (CCSS) Year 2000
computer system problem.\1 The problem results from the inability of
computer programs at the year 2000 to interpret the correct century
from a recorded or calculated date having only two digits to indicate
the year. Unless corrected, this problem could cause the CCSS system
to malfunction or produce incorrect information when the year 2000 is
encountered during automated data processing. Since CCSS supports
the Army's wholesale logistics supply management business area, which
procures supplies and equipment totaling over $23 billion annually
for Army forces around the world, LSSC's inability to ensure that
CCSS is Year 2000 compliant could result in a loss of operational
support that would be widespread, costly, and potentially
debilitating to important Army and other DOD agency missions.
Our briefing was based on work we performed as part of our review of
the Department of Defense's (DOD) Year 2000 computer systems effort
for the Chairman, Senate Governmental Affairs Committee; the Chairman
and Ranking Minority Member of the Subcommittee on Government
Management, Information and Technology, House Committee on Government
Reform and Oversight; and the Honorable Thomas M. Davis, III, House
of Representatives. During our review, we concentrated on
determining (1) the status of LSSC's efforts to correct CCSS Year
2000 system problems and (2) the appropriateness of LSSC's strategy
and actions for ensuring that CCSS Year 2000 issues will be
successfully addressed. This letter summarizes the concerns we
raised and provides recommendations for addressing these issues.
--------------------
\1 Although CCSS is considered to be one system, LSSC reports that it
actually consists of 561 subsystems and supporting applications which
contain about 10.2 million lines of code--about 5.5 million of which
are being reported by LSSC as potentially requiring renovation.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
The Year 2000 problem is one of the most comprehensive and complex
information management projects ever faced by LSSC. If not
successfully completed, the procurement of weapon systems and their
spare parts, accounting for the sales of Army equipment and services
to allies, and the financial management of $9 billion of inventory
could be disrupted. As a result, it could be extremely difficult to
efficiently and effectively equip and sustain the Army's forces
around the world.
To its credit, LSSC has completed several actions to address the CCSS
Year 2000 problem. For example, LSSC has inventoried software lines
of code, established a Year 2000 task force, and acquired automated
software assessment tools. In addition, a Year 2000 project manager
and management staff have been designated, a project charter and
schedule were developed, and supplementary contractor support was
acquired to assist with assessment tasks. Further, regularly
scheduled quarterly meetings are held by Army Materiel Command (AMC)
headquarters to report LSSC Year 2000 status. These steps are
compatible with DOD's suggested approach and consistent with those
found in our five-phased approach for planning, managing, and
evaluating Year 2000 projects.
Although LSSC commenced its Year 2000 project over a year ago, there
are several critical issues facing LSSC that, if not completely
addressed, may result in the failure of CCSS to successfully operate
at the year 2000. Specifically, LSSC has yet to completely address
(1) competing workload priorities and staffing issues, (2) the
appropriate mix and scheduling of needed testing data and expertise
as well as the development of test plans, (3) the scope and substance
of written interface agreements with system interface partners to
ensure that CCSS subsystems will be capable of exchanging data at the
year 2000, and (4) contingency plan development to help assure that
Army's missions will be accomplished if CCSS is not fully available
to users by the year 2000. LSSC's risk of failure is increased
because the agency has not attained the level of software development
and maintenance maturity that would provide the foundation needed for
successful management of large-scale projects such as the Year 2000
initiative. Because CCSS is used to support military readiness,
these critical elements must be resolved and aggressively pursued to
enable LSSC to successfully achieve a Year 2000 compliant environment
prior to the year 2000.
SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :2
In conducting our review, we compared LSSC's progress in planning and
managing its Year 2000 project to our Year 2000 Assessment Guide.\2
We also reviewed DOD's Year 2000 Management Plan,\3 Department of the
Army and Army Materiel Command (AMC) Year 2000 guidance,\4 and
private industry Year 2000 guidance. We focused our review on Year
2000 work performed by (1) LSSC--the designer, developer, and
maintainer of CCSS, and (2) AMC--the Army major command responsible
for promulgating Year 2000 policy and guidance and providing
assistance to its major subordinate commands and central design
activities.
To determine the status of LSSC's Year 2000 project and the
appropriateness of its strategy and actions for ensuring successful
completion, we interviewed LSSC's Year 2000 Project Manager, Project
Officer, and Focal Point who are responsible for project management,
direction, and reporting. We also interviewed the AMC Year 2000
project team and the AMC Year 2000 Logistics Systems Chair. We
obtained and analyzed Year 2000 guidance as well as documentation
from the CCSS Configuration Control Board, AMC quarterly progress
reviews, and AMC Year 2000 Logistics Task Force meetings to determine
Year 2000 plans, strategy, and status for each of the five phases.
We obtained and discussed software change schedules, workload
statistics, and testing procedures and testing resource information
with LSSC's Quality Assurance Division Chief. We discussed software
change procedures with the Technical Data Systems Division Chief and
impact and workload issues with the Asset Management Systems Chief.
To compare LSSC's workload with its available staff resources, we
obtained and discussed staffing information with the agency's
Resources Management Director, Budget and Manpower Division Chief,
and the Business Information Systems Director. To determine cost
estimates for the project, we interviewed the Year 2000 Project
Manager and obtained and analyzed LSSC documents pertaining to cost.
We also discussed LSSC's software maturity capability and efforts to
improve its maturity level with the Year 2000 Project Manager and the
computer specialist tasked with improving LSSC's software maturity
capability.
We conducted our work primarily at the Logistics Systems Support
Center in St. Louis, Missouri, and at the U.S. Army Materiel
Command in Alexandria, Virginia. Our audit work was performed from
December 1996 through August 1997 in accordance with generally
accepted government auditing standards.
The Department of Defense provided written comments on a draft of
this report. These comments are discussed in the "Agency Comments
and Our Evaluation" section and are reprinted in appendix I.
--------------------
\2 Year 2000 Computing Crisis: An Assessment Guide (Exposure Draft)
(GAO/AIMD-10.1.14, February 1997).
\3 Department of Defense Year 2000 Management Plan (Version 1.0,
April 1997).
\4 Army Project Change of Century Action Plan (Revision I, October
1996) and the U.S. Army Materiel Command Year 2000 Action Plan (May
1997).
BACKGROUND
------------------------------------------------------------ Letter :3
LSSC is one of several central design activities (CDAs) for the Army
Materiel Command (AMC).\5 LSSC's major responsibility is to design,
develop, deploy, integrate, and maintain the Commodity Command
Standard System (CCSS), a standard automated wholesale logistics
system supporting AMC and other Army and DOD organizations. CCSS
performs stock control, supply management, cataloging, provisioning,
procurement, maintenance, security assistance, and financial
management over an inventory of supply items for these organizations.
It is the business automation core of AMC's commodity commands and is
linked to other Army logistics systems, such as the Continuing
Balance System-Expanded (CBSX).\6 CCSS' financial module also
provides general accounting, inventory accounting, billing support,
and general ledger and financial reports for both reimbursable and
non-reimbursable issues. As one of the world's largest integrated
business systems, CCSS comprises 561 separate subsystems that contain
10.2 million lines of program code in about 5000 programs. These
subsystems work collectively to process an annual procurement budget
for supplies and equipment of over $23 billion.
The Year 2000 problem is rooted in the way dates are recorded and
computed in automated information systems. For the past several
decades, systems have typically used two digits to represent the
year, such as "97" representing 1997, in order to conserve electronic
data storage and reduce operating costs. However, with this
two-digit format, the year 2000 is indistinguishable from 1900, as is
2001 indistinguishable from 1901. As a result of this ambiguity,
system or application programs that use dates to perform
calculations, comparisons, or sorting may generate incorrect results
when working with years after 1999.
LSSC staff recognized the significance of this anomaly in 1991, and
at an estimated cost of $5.9 million, recommended making all CCSS
date fields Year 2000 compliant in accordance with Army Regulation
25-9.\7 The regulation specified that date fields were to be 8
positions in length (YYYYMMDD) including two century positions to be
populated with normal values of "19" or "20." LSSC has requested
funding for the recommended changes every year since 1991; however,
funding was denied because CCSS was a legacy system designated for
replacement by other systems under the DOD Corporate Information
Management (CIM) initiative. By 1994, however, it became apparent
that emergency system changes would be necessary to allow certain
CCSS subsystems to continue forecasting requirements beyond 1999.
LSSC reports that, since 1994, it has renovated at least 3.8 million
lines of code to accommodate the year 2000. However, initial funding
for completing the CCSS Year 2000 effort, now estimated at over $12
million, was not approved until January 1997.
The impact of the year 2000 on CCSS is substantial since CCSS is
heavily date dependent. Date fields are used in nearly all CCSS
subsystems, files, databases, and data used for status accounting,
computations, forecasting, financial accounting, and requisition
processing. Consequently, faulty turn of the century date processing
would significantly impair the Army's ability to order, manage, sell,
and account for commodities such as ammunition, communications, and
electronics. In turn, through its other logistics systems
connections, it could also impair the Army's ability to track and
manage major end items such as aircraft, missiles, and tanks, as well
as the many thousands of repair parts that support them.
Because CCSS is the Army's wholesale logistics system, a loss of CCSS
operational support to AMC and other DOD agencies poses a serious
threat to overall mission capability. For example, if dates are not
processed accurately in CCSS applications that support inventory
management and requisition processing, items ordered on or after
January 1, 2000, could be identified as 99-year old excess inventory
and become candidates for disposal. The cost of such faulty date
processing would be great considering the (1) cost of the inventory
item, (2) administrative costs involved in requisitioning, shipping,
handling, and accounting for the item in the various financial,
inventory, and transportation subsystems, and (3) costs associated
with designating the item as excess inventory for disposal and the
subsequent physical disposal of the item. Such an occurrence could
severely impair overall military readiness since the necessary items
would not be available for the soldier in the field. More
importantly, soldiers and military civilians may not be able to
properly maintain or replace weapon systems components, which could
result in injury or death. Also, military equipment maintenance and
overhaul facilities could be temporarily closed for lack of spare
parts.
In February 1997, we published the Year 2000 Computing Crisis: An
Assessment Guide, which addresses common issues affecting most
federal agencies and presents a structured approach and checklist to
aid in the planning, managing, and implementing of Year 2000
projects. The guide describes five phases---supported by program and
project management activities---with each phase representing a major
Year 2000 program activity or segment. The guidance draws heavily on
the work of the Best Practices Subcommittee of the Interagency Year
2000 Committee, and incorporates guidance and practices identified by
leading organizations in the information technology industry. The
five phases are consistent with those prescribed by DOD in its Year
2000 Management Plan. The phases and a description of each phase
follows:
-- Awareness--Define the Year 2000 problem and gain executive-level
support and sponsorship. Establish a Year 2000 program team and
develop an overall strategy. Ensure that everyone in the
organization is fully aware of the issue.
-- Assessment--Assess the Year 2000 impact on the enterprise.
Identify core business areas and processes, inventory and
analyze systems supporting the core business areas, and rank
their conversion or replacement. Develop contingency plans to
handle data exchange issues, lack of data, and bad data.
Identify and secure the necessary resources.
-- Renovation--Convert, replace, or eliminate selected platforms,
applications, databases, and utilities. Modify interfaces.
-- Validation--Test, verify, and validate converted or replaced
platforms, applications, databases, and utilities. Test the
performance, functionality, and integration of converted or
replaced platforms, applications, databases, utilities, and
interfaces in an operational environment.
-- Implementation--Implement converted or replaced platforms,
applications, databases, utilities, and interfaces. Implement
data exchange contingency plans, if necessary.
In addition to following the five phases described above, the Year
2000 program should also be planned and managed as a single large
information system development effort. Agencies should promulgate
and enforce good management practices on the program and project
levels.
--------------------
\5 AMC is headquartered in Alexandria, Virginia, and is responsible
for developing, buying, and maintaining equipment and supplies for
U.S. soldiers and allies worldwide.
\6 The Continuing Balance System-Expanded (CBSX) is the Army's
central logistics system for reporting the types, quantities, and
locations of equipment, generally major end items such as helicopters
and tanks.
\7 Army Data Management and Standards Program, AR 25-9, effective
October 24, 1989.
CURRENT STATUS OF LSSC'S YEAR
2000 PROJECT
------------------------------------------------------------ Letter :4
LSSC is the Army component responsible for applying the Year 2000
five-phased resolution process to CCSS. As such, in July 1996, LSSC
initiated a project to address CCSS Year 2000 processing issues. As
of July 1997, LSSC has completed a number of activities associated
with the awareness and assessment phases of the process, including
identifying its inventory, establishing a Year 2000 project team, and
assessing the date impact on CCSS' 10.2 million lines of code. LSSC
has identified that as much as 54 percent or 5.5 million lines of
code may be impacted by the year 2000 due to the fact that entire
applications may need to be corrected to accommodate the date change.
LSSC officials stated that they still need to determine how specific
code will be changed in affected applications. LSSC also reported
that an additional 3.8 million lines of code have already been
renovated but still need to undergo integrated and regression
testing. LSSC plans to implement the Year 2000-compliant CCSS by
November 1998 at a cost of over $12 million.
Prior to receiving funding in January 1997, the Year 2000 project
remained in the awareness phase. During the awareness phase, LSSC
completed tasks such as assembling technical and functional
representatives into a Year 2000 task force, evaluating automated
software assessment tools, and identifying the number of software
lines of code. Once the project was officially funded and entered
the assessment phase, LSSC officials appointed the project manager
and management staff. Also, the Year 2000 project team prepared a
project charter and schedule, secured contractor support to assist
with assessment tasks, and began to determine the date impact on CCSS
program code. As project activity proceeded, project staff routinely
reported Year 2000 progress to the AMC Deputy Commanding General, AMC
Year 2000 Logistics Task Force, Communications-Electronics Command
(CECOM) Year 2000 Project Office, and the CCSS Configuration Control
Board.
To support project management, LSSC's Year 2000 project manager
drafted a plan which initially did not conform to DOD's recommended
Year 2000 five-phased approach, although the plan did identify some
tasks typically associated with Year 2000 projects. For example, the
plan included such tasks as beginning risk assessment and contingency
plan development, providing assessment tool training, conducting an
inventory of CCSS applications, and obtaining contractor support for
date impact assessment. As a result of our concerns that the plan
did not clearly specify or identify key Year 2000 phases and
associated tasks, LSSC's Year 2000 project manager later revised the
plan in an attempt to better identify phases and tasks in accordance
with DOD's five-phased approach.
In addition to assessing the lines of code for CCSS, LSSC reported
that it had cataloged the applications, modules, functional areas
served, and languages used. LSSC had also determined that all the
source code for CCSS was available and matched production code. In
addition, LSSC had acquired automated assessment tools to help
identify affected and obsolete code and trained LSSC staff to use
these tools for the assessment, renovation, and validation phases.
Since the exchange of data with other systems through external
interfaces creates the potential to introduce or propagate errors
from one system to another, LSSC identified 57 other systems which
interface with CCSS and is in the process of confirming data exchange
requirements with the external system owners. LSSC also developed a
standard memorandum of agreement (MOA) to document coordination of
data exchange requirements.\8 Since CCSS and its interfacing partners
plan to use procedural code\9 and sliding window techniques\10
to correct the Year 2000 problem, any date formats exchanged would be
properly converted through internal program coding changes rather
than changes to date formats.
As part of its assessment of the level of date impact on CCSS, LSSC
assessed the risk of not preparing CCSS for the year 2000. LSSC
reported that CCSS, as a whole, is not Year 2000 compliant and that a
catastrophic failure of the Army wholesale logistics mission would
occur if CCSS is not made compliant. LSSC further reported that no
known commercial or government replacements exist for CCSS
functionality and that renovation of existing CCSS code was essential
to mitigate the risk of failure.
In May 1997, LSSC was still addressing the assessment phase
activities of identifying a renovation strategy and developing a
validation strategy and schedule for testing. According to DOD's
Year 2000 Management Plan and AMC's Year 2000 Action Plan, the
validation strategy should identify the general time frames for the
validation of all information resources and include consideration of
hardware concerns such as availability of processing cycles and
storage as well as human resource issues. In addition, efforts were
ongoing to contract with a vendor to perform automated code
correction on some CCSS subsystems.
--------------------
\8 LSSC's approach to data exchange and its proposal to the external
system owners consists of keeping date formats in the current
two-digit year configuration in accordance with existing military
data formatting standards. CCSS and nearly all DOD logistics systems
conform with military data formatting standards which require year
dates to be input, stored, and transmitted in a two-digit
configuration. While use of the four-digit year field is now the
preferred standard for system interfaces, DOD recognizes that a
two-digit year field is an acceptable configuration when accompanied
by signed MOAs documenting the data exchange requirements of the
interfacing partners.
\9 Procedural code is code which derives the correct century based on
the two-digit year (e.g., any year smaller than year 50 is a 2000
date, and any year 50 or larger is a 1900 date).
\10 Sliding windows are similar to procedural code in that they
derive the correct century based on the two-digit year, but the
numeric constant used to determine the century changes each year.
Using the procedural code example above, in the current year, 50 or
larger would be a 1900 date, while next year, 51 or larger would be a
1900 date.
LSSC MUST ADDRESS KEY YEAR 2000
ISSUES TO INCREASE CHANCES OF
SUCCESS
------------------------------------------------------------ Letter :5
To its credit, LSSC recognized the problems inherent in the century
date change and began seeking funding to address Year 2000 issues
years ago. However, although some progress has been made, several
key project management actions associated with the assessment phase
have not been completed. As a result, LSSC is not presently
well-positioned to move forward to the more difficult phases of
renovation, validation, and implementation in the Year 2000
process--phases that industry experts estimate could consume as much
as three-fourths of Year 2000 project time and resources. LSSC still
needs to take a number of actions to increase its chances of success,
including (1) managing competing workload priorities, (2) planning
for testing, (3) clarifying and coordinating written systems
interface agreements, and (4) developing a contingency plan. To
increase its chances of successfully managing its Year 2000 program,
LSSC will also need to institutionalize a repeatable software change
process that can be used from project to project. If these areas are
not addressed soon, LSSC could find itself limited in its ability to
meet the turn of century date. Given the prominence of date
processing in CCSS and its central mission of sustaining the soldier
in the field, LSSC cannot afford to delay any longer, and needs to
demonstrate that it will perform, all the key actions associated with
sound Year 2000 planning and management.
LSSC SHOULD INITIATE ACTIONS
TO IMPROVE ITS SOFTWARE
CAPABILITY MATURITY
---------------------------------------------------------- Letter :5.1
In 1991, the Software Engineering Institute (SEI)\11 introduced the
Capability Maturity Model (CMM) to assist organizations in assessing
the maturity level of their software development and maintenance
processes. In general, software process maturity serves as an
indicator of the likely range of cost, schedule, and quality of
results that can be expected to be achieved by projects within a
software organization. Our Year 2000 Assessment Guide points out
that few activities within federal agencies operate above CMM level
1, and as a result, organizations lack the basic policies, tools, and
practices necessary to successfully manage large-scale efforts. A
CMM level 1 is the lowest level and is characterized by a software
process that is ad hoc, and occasionally even chaotic. Few processes
are defined and success depends on individual effort. We have
recommended that federal agency information technology organizations
be at least a CMM level 2 which is characterized by an established
software development process discipline that is repeatable from
project to project.
In 1994, LSSC's software development process was assessed by a team
of LSSC and SEI-licensed contract staff. Using the CMM methodology,
the team determined that LSSC should be ranked at a level 1 maturity.
The assessment results concluded that LSSC lacked the basic software
management practices necessary for repeatable software project
success. The team also indicated that level 2 maturity could be
attainable with a modest effort. Accordingly, the assessment team
made recommendations that, if implemented, could provide the basis
for LSSC's attainment of a level 2 maturity. Based on the team's
assessment, LSSC developed an action plan to address the identified
deficiencies. However, according to LSSC officials, the action plan
was never implemented due to the reassignment of LSSC assessment
staff, agency staff reductions, and lack of funding.
After a period of nearly 3 years, LSSC resurrected the CMM assessment
project in December 1996 to, once again, review the assessment team's
findings and recommendations and to propose follow-on actions to
address the deficiencies. The review concluded that a project
management system which would allow LSSC to better plan, estimate,
and track software projects on an enterprise-wide basis was essential
for LSSC to mature to a CMM level 2. While LSSC has an automated
project management system under development, a member of the LSSC
review said the system is inadequate because it is unable to track
all software projects and may not address all level 2 requirements.
This information was presented to LSSC's Executive Steering Council
in March 1997. However, at the time of our review, LSSC had made
little progress in correcting the software process deficiencies and
was still ranked at CMM level 1. Until LSSC moves on to the next CMM
level, its ability to contend with the later stages of the Year 2000
effort will be constrained. Recently, LSSC officials informed us of
their intent to obtain a CMM level 2 certification following
completion of the Year 2000 project.
--------------------
\11 The Software Engineering Institute (SEI) is a federally funded
research and development center sponsored by DOD and operated by
Carnegie Mellon University in Pittsburgh, Pennsylvania.
ACTIONS NEEDED TO MITIGATE
IMPACT OF COMPETING WORKLOAD
PRIORITIES
---------------------------------------------------------- Letter :5.2
In addition to lacking a mature software development and maintenance
process, LSSC now has 42 percent fewer staff available to make the
needed renovations to CCSS than it had in fiscal year 1990.
Moreover, since fiscal year 1990, LSSC's workload has increased,
showing a notable jump in fiscal years 1997 and 1998--the 2 years
when the majority of Year 2000 actions need to be performed to enable
agencies to have a realistic chance of meeting the turn of century
date time frame. At the same time that its staff is decreasing and
its workload is increasing, LSSC continues to be tasked with other
software projects by the Lead AMC Integration Support Office (LAISO).
Despite these indicators of potential problems, LSSC has only
recently begun to take the steps necessary to augment its staff with
contract support for the renovation phase and has yet to fully
resolve staffing issues concerning the development of test plans. In
addition, LSSC has not prioritized its software project schedule to
provide the structure needed to keep the Year 2000 project on
schedule and within cost estimates. Until these issues are
addressed, they pose unnecessary risk to the success of LSSC's Year
2000 project.
As of June 1997, LSSC reported that it had devoted 7 of its 315 total
staff to the Year 2000 project full-time. While four contract
support staff had been retained to train LSSC staff to use the
automated software assessment tools and help with impact assessment,
these contractor staff have since been released. As of August 1997,
no contract staff were on board to augment LSSC staff during the
renovation phase, although steps were underway to obtain additional
contract support and to obtain an automated code correction
solution.\12
Also, LSSC reported that staff would be tasked to work exclusively on
the Year 2000 renovation phase after completing an ongoing major
systems change project related to a Base Realignment and Closure
(BRAC) decision.\13 LSSC officials stated that as the BRAC-related
renovation begins to diminish in September 1997, both LSSC and
contract staff would be transferred to the Year 2000 project. While
we do not question the appropriateness of performing the BRAC-related
work prior to Year 2000 work, we are concerned that LSSC's Year 2000
project approach does not provide for alternatives should the BRAC
target completion schedule slip and the subsequent LSSC staff and
contractors not become available.
Further, an examination of CCSS software release schedules since
fiscal year 1990 shows that the number of projects has increased as
much as five-fold. At the same time as the majority of CCSS Year
2000 actions are to be performed, LSSC's schedule calls for 10
software change projects to be fielded in fiscal year 1997 and 8 in
fiscal year 1998. These projects range in terms of complexity and
magnitude from routine systems maintenance, which may require minimal
effort, to the Year 2000 and BRAC projects, which call for
comprehensive changes in many of CCSS' subsystems. In past years,
LSSC routinely accomplished two to four software change projects a
year. This significant increase in workload will undoubtedly impact
the CCSS Year 2000 project schedule for several reasons. First,
LAISO, the workload manager for CCSS, has not ensured that competing
projects do not adversely affect LSSC's ability to complete the Year
2000 effort. Prioritization of projects could result in the
postponement or cancellation of some of the competing projects.
Second, LSSC has little historical experience dealing with a workload
of this magnitude which is compounded by a workforce that has
diminished significantly in recent years.
--------------------
\12 LSSC is planning to utilize the services of a private contractor
that provides an automated code conversion service to organizations
seeking an automated method for correcting Common Business Oriented
Language (COBOL) software code. Under this arrangement, LSSC will
provide its software code to the contractor's off-site facility where
Year 2000 windowing logic is inserted using the contractor's
proprietary automated software. The converted code will then be
returned to LSSC for compiling and testing.
\13 The BRAC decision entails reducing the size of DOD and saving
money by closing or consolidating DOD facilities. Since one of AMC's
major subordinate commands (MSC), located in St. Louis, Missouri,
was dissolved and its functions moved to other MSCs as a result of
the 1995 BRAC decision, it has been necessary for LSSC to develop
conversion programs to extract aviation and troop support information
and transfer the data to the MSCs now responsible for those
functions.
LSSC SHOULD BE PLANNING FOR
THE VALIDATION PHASE
---------------------------------------------------------- Letter :5.3
Prior to commencing the validation (testing) phase of its Year 2000
effort, LSSC needs to fully address two key issues regarding its
testing requirements and capabilities. Specifically, LSSC should be
planning now to (1) assure that enough staff with the appropriate
background and experience are available to develop Year 2000 test
data and transactions and to review test results and (2) assess
whether enough time has been scheduled to perform Year 2000 testing.
Without planning how it will address these issues now, LSSC is
increasing the risk that CCSS will not be fully validated in time for
the change of century date.
According to AMC's Year 2000 Action Plan, many agencies will need to
establish test environments which are specific to future date testing
and which have no possibility of corrupting or destroying production
data.\14 Since the current CCSS test files do not contain the
necessary Year 2000 test conditions and data, LSSC will need to
establish Year 2000-specific test files to certify that CCSS is Year
2000 compliant. Such test data and transactions are typically
designed by functional staff knowledgeable of the CCSS business
processes. These staff review the testing results to ensure that
Year 2000 software changes have processed data correctly and that
other data and processes have not been inadvertently changed during
testing. According to a LSSC official and LSSC staffing statistics,
however, there are fewer functional staff now available to identify
the date fields in the test transactions or test data needed to
ensure that CCSS business processes are not adversely affected by the
Year 2000 software changes. Also, LSSC officials stated that they
expect the availability of these staff to continue to decrease over
the next few years as staff retire and agency staff reductions
continue.
Further, LSSC is not allowing enough time for Year 2000 testing.
While LSSC officials assert that the complexity and scope of the Year
2000 project is about the same as the BRAC project, LSSC's June 1997
systems change release schedule calls for far less time to test Year
2000 changes than it does for BRAC changes. For example, BRAC
testing began in February 1997 and is scheduled for completion in
September 1997. Year 2000 testing is scheduled to begin in September
1998 and end almost 8 weeks later in November 1998. A LSSC official
acknowledged that the amount of time scheduled for Year 2000 testing
is insufficient, but stated that the schedule will be revised once
ongoing negotiations to acquire an automated code correction service
are resolved. The official also stated that he fully expects the
Year 2000 test schedule to greatly increase beyond the currently
scheduled 8 weeks but that the increased test time should be offset
by the reduced renovation time expected to be garnered by using the
automated code correction service. Although LSSC believes that the
automated code correction service should provide increased Year 2000
testing time, it could not provide documented analysis to support
this conclusion.
While LSSC believes it can increase its testing time without
increasing the overall Year 2000 project time, we are not as
confident given LSSC's CMM level 1 ranking. Trying to compensate for
unrealistic time schedules by either shortening earlier phases of a
software change project or by lengthening overall project time is
characteristic of level 1 organizations. Until LSSC realistically
assesses its testing requirements, capabilities, and time schedules,
effective Year 2000 project management will become increasingly
difficult to achieve, and LSSC will increase the risk that it may be
unable to meet the demand imposed by Year 2000 testing.
--------------------
\14 Production data are existing data and transactions processed by
or generated from everyday, operational use of a computer system.
Preparing and using production data for testing does not require
knowledge of the internal logic of the software. As a result,
production data may not test all the functions or logic paths desired
and could produce inconclusive results.
INTERFACE AGREEMENTS LACK
BASIC INFORMATION FOR
EFFECTIVE MANAGEMENT AND
IMPLEMENTATION
---------------------------------------------------------- Letter :5.4
CCSS' ability to successfully operate at the year 2000 hinges on the
proper and timely exchange of data with other systems, both within
the Army and with external Defense components. It is critically
important during the Year 2000 effort that agencies ensure that
interfacing systems have the ability to exchange data throughout the
transition period and protect against the potential for introducing
and propagating errors from one organization to another and ensure
that interfacing systems have the ability to exchange data through
the transition period. This potential problem may be mitigated
through formal agreements between interface partners that describe
the method of interface and assign responsibility for accommodating
the exchange of data. Both the DOD Year 2000 Management Plan and AMC
Year 2000 Action Plan place responsibility on component heads or
their designated Year 2000 contact points to document and obtain
system interface agreements in the form of memorandums of agreement
(MOA) or the equivalent. Further, to help assure that interfaces
continue to properly exchange data after systems are renovated for
the year 2000, AMC has issued minimum MOA documentation requirements
designed to produce consistency, assign accountability, and recognize
a level of detail necessary for effective interface renovation among
data exchange partners.
While LSSC has developed MOAs to document interface specifics between
CCSS and its interfacing systems and is in the process of finalizing
those agreements with system owners, nearly all the MOAs lack basic
information necessary for effective management and implementation of
the interfaces. According to AMC Year 2000 guidance and the
accompanying requirements of the standard MOA, the agreements are to
specify the (1) points of contact for reporting progress and
coordinating schedules and (2) date the agreement becomes effective.
To successfully implement interface changes, these agreements should
also communicate the type, form, and frequency of transactions
exchanged, the windowing technique that is being used at each end of
the interface, and the review process for monitoring interface
renovation progress and reconciling differences. However, our review
disclosed that 39 of 41 MOAs that LSSC had finalized as of July 1997
failed to fully follow AMC's guidance or include other information
necessary to ensure that LSSC can successfully communicate with
interface partners. Our Year 2000 Assessment Guide stresses the
importance of adequately addressing interface and data exchange
issues. Without such information, the MOAs do not serve to
communicate and coordinate the actions designed to help assure that
Year 2000 changes are made properly and promptly by LSSC and its
interfacing partners.
Timely and complete information on all systems interfaces that may be
affected by Year 2000 changes is essential to the success of the LSSC
Year 2000 compliance program. The amount of work required to
coordinate the data being exchanged between systems must be known as
early as possible and documented in written MOAs so that LSSC may
complete renovation schedules, allocate resources, plan testing, and
schedule implementation.
YEAR 2000 CONTINGENCY PLAN
NEEDED FOR CCSS
---------------------------------------------------------- Letter :5.5
The year 2000 represents a great potential for operational failure to
CCSS that could adversely impact core business processes as well as
those of entities that depend on the CCSS system for information. To
mitigate this risk of failure, our Year 2000 Assessment Guide, DOD's
Year 2000 Management Plan, and the Army's Project Change of Century
Action Plan suggest that agencies perform risk assessments and
prepare realistic contingency plans that identify alternatives to
ensure the continuity of core business processes in the event of
operational failure. These alternatives could include performing
automated functions manually or using the processing services of
contractors.
While LSSC has taken the first steps toward development of a
contingency plan by assessing the level of risk to each business area
that could be affected by processing errors and by determining how
that risk can be mitigated or reduced, at the completion of our
review, LSSC had not yet developed a contingency plan. Further,
despite explicit guidance from DOD and the Army to develop
contingency plans should Year 2000 corrections to CCSS not be
completed in time, LSSC officials stated that no contingency plan
would be developed for CCSS. They maintained that AMC does not
require a contingency plan for CCSS because CCSS is not scheduled for
replacement prior to the advent of the year 2000. While AMC's Year
2000 Action Plan states that contingency plan development is only
required for replacement systems and implies that all other systems
are exempt, the AMC plan also states that the guidance, policy, and
responsibilities identified in the Army's Project Change of Century
Action Plan are mandatory and are the basis of the AMC plan.
Nevertheless, despite LSSC's and AMC's position that a contingency
plan is not needed for CCSS because the system is not being replaced
prior to the year 2000, the system still risks unanticipated
operational failure. Without a contingency plan that identifies
specific actions to be taken if CCSS fails at the year 2000, the
procurement of weapon systems and their spare parts, accounting for
the sale of Army equipment and services to allies, and the financial
management of $9 billion of inventory could be disrupted. As a
result, the Army could be unable to efficiently and effectively equip
and sustain its forces around the world. Given the dangers
associated with an operational failure of this magnitude, LSSC needs
the protection provided by good contingency planning to ensure that
options are available if CCSS is not able to operate at the year
2000. Recently, LSSC officials stated that they have begun preparing
an initial contingency plan, which they estimate will be completed by
September 30, 1997.
CONCLUSIONS
------------------------------------------------------------ Letter :6
If CCSS cannot correctly process dates on and after January 1, 2000,
military equipment, such as tanks, artillery, aircraft, missiles,
munitions, trucks, electronics, and other supporting materials for
the soldier, in all likelihood, will not be ordered, stored,
transported, issued, paid for, or maintained. Mobilization plans and
contingencies would be significantly impaired if materiel is delayed.
However, LSSC has yet to resolve several critical problems associated
with the assessment phase to ensure that (1) systems are adequately
tested, (2) contingency plans are developed, and (3) interface
partners are fully aware of LSSC's Year 2000 plans. Furthermore,
during the same time that LSSC is addressing the Year 2000 issue, the
agency is also working to implement considerably more software
projects than it has in the past. This unprecedented workload is
compounded by a reduced staff level and LSSC's basic lack of a mature
software development and maintenance process. Together, these
factors raise the risk level of the Year 2000 project beyond what is
normally expected of a software modification effort of this
magnitude. Until these problems are resolved, LSSC is not
well-positioned to move forward into the more time-consuming phases
of renovation, validation, and implementation. As a result, we
believe LSSC will find it increasingly difficult to prepare CCSS in
time for the arrival of the year 2000.
RECOMMENDATIONS
------------------------------------------------------------ Letter :7
We recommend that you:
-- Act to improve LSSC's software development process that will
provide the basis for achieving CMM level 2 maturity.
-- Immediately assess the impact of competing workload and staffing
demands on the CCSS Year 2000 project. Based on this
assessment, consider (1) canceling or deferring less critical
software projects until after the Year 2000 project is
substantially completed and (2) augmenting the Year 2000 project
with staff having the necessary skills to ensure timely
completion of the project.
-- Ensure that LSSC has the capability to complete the testing of
all CCSS subsystems and programs. Specifically, LSSC should (1)
determine test requirements, (2) identify the testing staff
needed, (3) finalize Year 2000 test plans describing how the
testing staff will be acquired and scheduled for developing Year
2000 compliant test scenarios and data, and (4) revise the Year
2000 test schedule to assure that enough time is available to
meet Army-mandated deadlines for Year 2000 implementation.
-- Ensure that written interface agreements describe the method of
data exchange between interfacing systems, name the entity
responsible for performing the system interface modification,
and state the completion date.
-- Develop a contingency plan that includes specific actions for
ensuring that the Army's logistic functions continue to operate
at appropriate levels if all or part of CCSS fails to work at
the year 2000.
AGENCY COMMENTS AND OUR
EVALUATION
------------------------------------------------------------ Letter :8
In written comments on a draft of this report, the Office of the
Under Secretary of Defense (Acquisition and Technology) concurred
with all of our recommendations to improve the Army's LSSC Year 2000
program. Specifically, DOD agreed that a contingency plan would be
developed by September 30, 1997, to ensure continuity of operations
if all or part of CCSS fails to operate by the year 2000.
DOD also outlined a number of actions that have recently been
initiated that are aimed at reducing and prioritizing LSSC's current
workload, and increasing staff with the necessary skills to help
ensure the timely completion of the Year 2000 project. In addition,
DOD pointed to several actions, both taken and planned, to improve
its capability to complete Year 2000 testing of CCSS subsystems and
programs. While we have not reviewed LSSC's latest actions, if
properly implemented, we believe they could help resolve the workload
and testing issues we identified.
In concurring with our recommendation regarding the need to initiate
actions to improve LSSC's software development process, DOD
recognized the value of achieving a CMM level 2 maturity and agreed
that LSSC does not have all configuration management procedures in
place to reach CMM level 2 at this time. However, DOD stated that
LSSC's history indicates that it can accomplish large projects
successfully and that LSSC will meet the mandated dates for the BRAC
and Year 2000 projects without achieving CMM level 2. After
completion of these projects, LSSC plans to resume its efforts to
achieve a CMM level 2 maturity. We believe LSSC's position comes at
some risk. The discipline derived from reaching a CMM level 2
maturity can greatly enhance LSSC's ability to address the Year 2000
challenge. This higher level of maturity is key to reducing the risk
of schedule slippage, cost overruns, and poor software quality. As
our report states, we have recommended that information technology
organizations be at least a CMM level 2 to successfully manage
large-scale projects such as the Year 2000 project. Our Year 2000
Assessment Guide provides interim actions that level 1 organizations
can take prior to the year 2000 to minimize the risk of failure, such
as training staff on proven industry system development and program
management practices and soliciting assistance from organizational
entities experienced in performing or managing major software
conversions. LSSC could benefit from these interim actions.
In concurring with our recommendation on strengthening written
interface agreements, DOD stated that LSSC will formalize MOAs
between interface partners. It also agreed to include specific
detailed information in MOAs, but only when appropriate. As our
report stated, we believe that, at a minimum, MOAs should also
contain essential information for effective management of system
interfaces, such as the type, form, and frequency of transactions
exchanged, the windowing technique to be used, and the review process
for monitoring interface renovation progress and reconciling
differences. This additional information would help to ensure that
interface partners are sufficiently prepared to handle unforeseen
problems that may occur and to plan for contingencies. The full text
of DOD's comments is provided in appendix I.
---------------------------------------------------------- Letter :8.1
This report contains recommendations to you. Within 60 days of the
date of this letter, we would appreciate receiving a written
statement on actions taken to address these recommendations.
We appreciate the courtesy and cooperation extended to our audit team
by LSSC officials and staff. We are providing copies of this letter
to the Chairmen and Ranking Minority Members of the Senate Committee
on Governmental Affairs; the Subcommittee on Oversight of Government
Management, Restructuring and the District of Columbia, Senate
Committee on Governmental Affairs; and the Subcommittee on Government
Management, Information and Technology, House Committee on Government
Reform and Oversight; the Honorable Thomas M. Davis, III, House of
Representatives; the Secretary of Defense; the Deputy Secretary of
Defense; the Acting Under Secretary of Defense (Acquisition and
Technology); the Acting Under Secretary of Defense (Comptroller); the
Acting Assistant Secretary of Defense (Command, Control,
Communications and Intelligence); the Secretary of the Army;
Commanders of the Army Materiel Command and Communications-
Electronics Command; the Director of the Office of Management and
Budget; and other interested parties. Copies will be made available
to others upon request.
---------------------------------------------------------- Letter :8.2
If you have any questions on matters discussed in this letter, please
call me at (202) 512-6240, or John B. Stephenson, Assistant
Director, at (202) 512-6225. Major contributors to this report are
listed in appendix II.
Sincerely yours,
Jack L. Brock, Jr.
Director, Defense Information and
Financial Management Systems
(See figure in printed edition.)Appendix I
COMMENTS FROM THE DEPARTMENT OF
DEFENSE
============================================================== Letter
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
The following is GAO's comment on the Department of Defense's letter
dated September 12, 1997.
GAO COMMENT
1. DOD provided a number of clarifications to the report that we
have incorporated as appropriate.
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II
ACCOUNTING AND INFORMATION
MANAGEMENT DIVISION, WASHINGTON,
D.C.
Ronald B. Bageant, Assistant Director
Carl M. Urie, Technical Advisor
Brenda A. James, Senior Information Systems Analyst
Cristina T. Chaplain, Communications Analyst
KANSAS CITY FIELD OFFICE
Denice M. Millett, Evaluator-in-Charge
Michael W. Buell, Staff Member
*** End of document. ***