Defense Computers: U.S. Transportation Command's Management of Y2K
Operational Testing (Letter Report, 11/15/1999, GAO/AIMD-00-21).
Complete and thorough Year 2000 end-to-end testing is essential to
ensuring that new or modified systems used to collectively support a
core business function or mission operation will not jeopardize an
organization's ability to deliver products and services on or after
January 1, 2000. This is especially true for the Defense Department,
which relies on a complex array of interconnected computer systems --
including weapons, command and control, satellite, inventory management,
transportation management, health care, financial, personnel, and
payments systems -- to carry out its operations. This report (1)
assesses the U.S. Transportation Command's management of its end-to-end
test of its ability to plan and execute joint major theater war
deployment operations and (2) determines what the results of this test
show with respect to operational risks and readiness.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: AIMD-00-21
TITLE: Defense Computers: U.S. Transportation Command's
Management of Y2K Operational Testing
DATE: 11/15/1999
SUBJECT: Y2K
Systems conversions
Interagency relations
Combat readiness
Defense contingency planning
Mobilization
Strategic information systems planning
Computer software verification and validation
Systems compatibility
Information resources management
IDENTIFIER: Y2K
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
GAO/AIMD-00-21
Report to the Chairman of the Subcommittee on Defense, Committee on
Appropriations, House of Representatives
November 1999
DEFENSE COMPUTERS
U.S. Transportation Command's Management of Y2K Operational
Testing
*****************
*****************
GAO/AIMD-00-21
Letter 3
Appendixes
Appendix I:Briefing on Results of GAO Review of TRANSCOM Turbo Y2K-Part A
OPEVAL
14
Appendix II:Objectives, Scope, and Methodology
58
Appendix III:GAO Contact and Staff Acknowledgements
59
Table 1: Highlights of Key DOD-Recommended Year 2000
Testing Processes 7
Table 2: Summary of TRANSCOM Satisfaction of JCS Evaluation
Criteria on Deployment Evaluation 8
AMC Air Mobility Command
CINC Commanders-in-Chief
DOD Department of Defense
JCS Joint Chiefs of Staff
MSC Military Sealift Command
MTMC Military Traffic Management Command
MTW major theater war
OPEVAL operational evaluation
TRANSCOM U.S. Transportation Command
Y2K Year 2000
Accounting and Information
Management Division
B-282545
November 15, 1999
The Honorable Jerry Lewis
Chairman
Subcommittee on Defense
Committee on Appropriations
House of Representatives
Dear Mr. Chairman:
Complete and thorough Year 2000 (Y2K) end-to-end testing is essential to
provide reasonable assurance that new or modified systems used to
collectively support a core business function or mission operation will
not jeopardize an organization's ability to deliver products and services
as a result of the Y2K computing problem. This is especially true for the
Department of Defense (DOD) because it relies on a complex and broad array
of interconnected computer systems--including weapons, command and
control, satellite, inventory management, transportation management,
health, financial, personnel and payment systems--to carry out its
military operations and supporting business functions.
At your request, we are reviewing DOD's management of its various Year
2000-related end-to-end testing activities. As part of our efforts, we
assessed the U.S. Transportation Command's (TRANSCOM) management of its
end-to-end test of its ability to plan and execute joint major theater war
(MTW) deployment operations, and determined what the results of this test
show with respect to operational risks and readiness./Footnote1/ We
briefed TRANSCOM officials on our findings on August 24, 1999, and made
recommendations to correct the management weaknesses that we found.
TRANSCOM took immediate action to address our recommendations, and on
September 14, 1999, we briefed your office on our findings and TRANSCOM's
actions. The purpose of this letter is to summarize our briefing to your
office. The briefing slides that we presented to your office are in
appendix I, and the objectives, scope, and methodology of our review are
detailed in appendix II. TRANSCOM provided oral comments on our briefing
slides at the August 24, 1999, briefing, and we have incorporated them as
appropriate. We performed our audit work from March through September 1999
in accordance with generally accepted government auditing standards.
Results in Brief
----------------
Year 2000 end-to-end testing is an essential component of an effective
Year 2000 testing program since Y2K-related problems can affect so many of
the systems owned and operated by an entity as well as systems belonging
to business partners and infrastructure providers. Moreover, to be
effective, end-to-end testing should be approached in a structured and
disciplined fashion. Both the Joint Chiefs of Staff (JCS) guidance to its
combatant commands on managing Year 2000 operational
evaluations,/Footnote2/ (the term JCS uses to refer to Year 2000 end-to-
end testing) and our Year 2000 test guidance/Footnote3/ define a number of
key management controls to employ when planning, executing, analyzing, and
reporting on these tests and evaluations.
We found that TRANSCOM's deployment operational evaluation satisfied most
of the key processes that JCS' guidance specifies. For example, TRANSCOM
established a Y2K task force to guide the evaluation effort, which
included surface and airlift experts, test and evaluation experts from the
Joint Interoperability Test Center and the Defense Office of Test and
Evaluation, component command and service representatives, and a public
affairs representative. Further, TRANSCOM performed a rehearsal before
conducting the test to ensure that all critical systems and interfaces
were operating correctly and that all staff knew their roles and
responsibilities.
However, TRANSCOM had not satisfied key steps that are pivotal to
(1) fully disclosing and appropriately addressing risks associated with
limitations in the defined scope of the operational evaluation and
(2) accurately reporting on mission readiness and impacts. The result was
that the Year 2000 readiness of critical tasks associated with conducting
a MTW deployment--namely the use of commercial carriers and port
operations to deploy goods and people--was not known with sufficient
certainty to support TRANSCOM's May 1999 reported position that it can
conduct a MTW deployment in a Year 2000 environment with no material
impact on operations.
TRANSCOM has since either implemented or initiated appropriate actions to
address our recommendations for correcting these weaknesses. For example,
it amended its final reports on the evaluation to disclose the scope
limitations. It is also working with its component commands to identify
their major commercial carrier business partners, to assess their
readiness and risks, and to develop risk mitigation strategies. Because
TRANSCOM has initiated these and other actions, we are not making further
recommendations at this time.
Background
----------
As the single manager of Defense transportation, TRANSCOM's mission is to
coordinate the use of air, sea, and land transportation to deploy and
sustain U.S. forces. To perform this mission, TRANSCOM relies on a wide
array of information technology systems, including command and control
systems, physical and human asset visibility and tracking systems,
transportation feasibility and port analysis tools, and intelligence
systems. TRANSCOM also depends heavily on partnerships with commercial
transportation service providers such as Civil Reserve Air Fleet carriers,
the commercial maritime industry, and railroad and trucking carriers. In
total, these partners provide about 85 percent of Defense's transportation
needs.
In August 1998, the Secretary of Defense directed the Commanders-in-Chief
(CINC) to require its combatant commands, including TRANSCOM, to plan,
execute, analyze, and report on a series of simulated Year 2000
operational evaluations. The evaluations, which were to assess whether DOD
can continue to perform critical military operations in a Year 2000
environment, are one of three DOD end-to-end testing efforts./Footnote4/
The purpose of end-to-end testing is to verify that a defined set of
interrelated systems, which collectively support an organizational core
business area or function, interoperate as intended in an operational
environment (either actual or simulated). These interrelated systems
include not only those owned and managed by an organization, but also the
external systems with which they interface or that otherwise support the
business area or function. The CINC core business areas or functions are
referred to as "thin lines."
The boundaries for end-to-end tests can vary depending on a given business
function's system dependencies and criticality to the organizational
mission. Therefore, in managing end-to-end test activities, it is
important to analyze the interrelationships among core business functions
and their supporting systems and the mission impact and risk of date-
induced system failures and to use these analyses to define test
boundaries. It is also important to work early and continually with
functional partners to ensure that related end-to-end test activities are
effectively coordinated and integrated. Table 1 highlights key processes
recommended by JCS' Year 2000 operational evaluation guidance, which is
consistent with our Year 2000 test guide.
Table****Helvetica:x11****1: Highlights of Key DOD-Recommended Year
2000 Testing Processes
------------------------------------------------------------------------
| : |
|----------------------------------------------------------------------|
| Planning : Specify test assumptions and limitations |
|----------------------------------------------------------------------|
| : Establish a Year 2000 Task Force |
|----------------------------------------------------------------------|
| : Identify critical missions/tasks/systems |
|----------------------------------------------------------------------|
| : Verify that systems essential to the mission are |
| : Year 2000 compliant |
|----------------------------------------------------------------------|
| : Develop an operational evaluation plan to guide |
| : event planning and execution |
|----------------------------------------------------------------------|
| : Identify and schedule support from other commands, |
| : DOD components, etc. |
|----------------------------------------------------------------------|
| : Determine relevant and necessary resources (e.g., |
| : funding, personnel, equipment) |
|----------------------------------------------------------------------|
| : Ensure that approved Year 2000 contingency plans |
| : are prepared |
|----------------------------------------------------------------------|
| : Develop risk management plan |
|----------------------------------------------------------------------|
| : Identify simulation needs and establish supporting |
| : testing environment |
|----------------------------------------------------------------------|
| : Develop data collection and analysis plan or |
| : approaches |
|----------------------------------------------------------------------|
| Execution : Conduct operational evaluation rehearsal |
|----------------------------------------------------------------------|
| : Follow configuration management policy |
|----------------------------------------------------------------------|
| : Perform baseline test for operational evaluation |
|----------------------------------------------------------------------|
| : Execute required Year 2000 date rollover tests |
|----------------------------------------------------------------------|
| : Collect and archive all Year 2000-relevant data and |
| : ensure that systems are reset to current day |
| : operations |
|----------------------------------------------------------------------|
| Analysis : Categorize, document, and report failures |
|----------------------------------------------------------------------|
| : Determine mission impact of Year 2000 failures |
|----------------------------------------------------------------------|
| : Ensure that exit criteria are met |
|----------------------------------------------------------------------|
| Reporting : Prepare Year 2000 reports describing mission impact |
| : and readiness |
|----------------------------------------------------------------------|
| : Provide reports to Joint Staff within required time |
| : frames |
------------------------------------------------------------------------
TRANSCOM has already carried out two operational evaluations--the first,
conducted in April 1999, assessed its ability to manage a MTW deployment
operation and the second, conducted in May and June 1999, assessed its
ability to manage a MTW sustainment operation./Footnote5/ A third
evaluation, which will retest the same 23 systems included in the May and
June sustainment evaluation, is planned for October 1999. In addition to
these events, TRANSCOM has taken part in other DOD Year 2000 exercises.
TRANSCOM Implemented Many Important Management Processes on Its Deployment
Operational Evaluation
As noted in table 2, we found that TRANSCOM had satisfied the majority of
the management process controls (17 of 21) specified in JCS' operational
evaluation guidance.
Table****Helvetica:x11****2: Summary of TRANSCOM Satisfaction of JCS
Evaluation Criteria on Deployment Evaluation
------------------------------------------------------------------------
| Phases : Number of primary: Number of |
| : evaluation : primary |
| : criteria : criteria |
| : : satisfied |
|----------------------------------------------------------------------|
| Planning : 11 : 9 |
|----------------------------------------------------------------------|
| Execution : 5 : 5 |
|----------------------------------------------------------------------|
| Analysis : 3 : 2 |
|----------------------------------------------------------------------|
| Reporting : 2 : 1 |
|----------------------------------------------------------------------|
| Total : 21 : 17 |
------------------------------------------------------------------------
Consistent with JCS guidance governing operational evaluation planning,
TRANSCOM established a Year 2000 task force, which included surface and
airlift experts, test and evaluation experts, system analysts, component
and service representatives, and a public affairs representative. It
identified 15 critical tasks needed to carry out a MTW deployment mission
and identified a total of 23 "thin line" systems that support its
capability to manage the deployment mission. TRANSCOM also verified that
the 23 systems were certified as Year 2000 compliant. Further, TRANSCOM
developed a test plan that documented participant roles and
responsibilities, critical missions and tasks, test cases, and reporting
requirements.
TRANSCOM also took effective steps in executing and reporting its
evaluation. For instance, before executing the operational evaluation,
TRANSCOM performed a rehearsal to ensure that all critical systems and
interfaces were operating correctly and that all staff knew their roles
and responsibilities. Before resetting systems to present day operational
conditions, TRANSCOM officials determined that the defined "thin line" was
completely exercised, all items in the master scenario events list were
performed, and that all data needed to make an assessment of the command's
ability to perform its deployment mission were collected and archived.
Also, TRANSCOM ensured that required reports were submitted on time and
that they addressed specified reporting requirements.
TRANSCOM Did Not Satisfy Some Key Steps Related to Testing Limitations and
Deviations
---------------------------------------------------------------------------
While TRANSCOM followed most JCS-specified criteria needed to effectively
manage its evaluation, it did not adequately satisfy certain planning,
analysis, and reporting criteria associated with the evaluation's scope.
Specifically, when planning the evaluation, TRANSCOM concluded that it was
not feasible or affordable to actively engage commercial transportation
providers and ports in its evaluation. As a result, TRANSCOM decided to
rely on its commercial partners to ensure the readiness of their
respective transportation systems, and thus to assume these partners'
ability to perform their military deployment and sustainment support roles
and responsibilities. Further, while TRANSCOM officials stated that they
communicated this assumption orally to JCS, TRANSCOM's operational
evaluation plan did not specify this scope limitation nor did it address
the exclusion of these partners as a risk that could adversely affect its
ability to draw conclusions about TRANSCOM's mission readiness. Moreover,
TRANSCOM did not verify the validity of this assumption through any
assessment of its key business partners' readiness, and in fact did not
consider information available at the time of the operational evaluation
that raised concerns about the Year 2000 readiness of the nation's
transportation sector. Despite this scope limitation, TRANSCOM reported to
JCS that critical deployment tasks could be performed during calendar and
leap year date rollovers with no significant impact on its mission
readiness.
As stressed in our briefing to TRANSCOM officials, Defense transportation
services for deployment and sustainment are provided primarily by the
commercial transportation sector. For example, Civil Reserve Air Fleet
carriers account for 90 percent of long-range passenger capability and
40 percent of long-range cargo capability. The commercial maritime
industry provides nearly all wartime sustainment capability. In addition,
commercial rail and truck carriers provide virtually all continental U.S.
surface transportation capability. TRANSCOM also depends on commercial
systems that support and maintain air and water port operations to fulfill
its deployment and sustainment missions. Therefore, without knowing
commercial carrier and port Year 2000 readiness, TRANSCOM's conclusion
that all its critical missions and tasks could continue to be performed
into the Year 2000 without material disruption was not justified. Also,
without disclosing the scope limitation and related risks, TRANSCOM did
not provide DOD leadership with complete and reliable information upon
which to assess DOD's institutional Year 2000 readiness to execute
military operations.
To address these concerns, we recommended in our August 24, 1999, briefing
that TRANSCOM amend its final reports to JCS to disclose the scope
limitations and related risks. We also recommended that TRANSCOM assess
and selectively verify the readiness of transportation systems belonging
to its commercial partners.
TRANSCOM Is Addressing Our Recommendations
------------------------------------------
TRANSCOM agreed with our recommendations and is responding to them. First,
TRANSCOM amended its final reports to JCS to disclose the scope
limitations of its operational evaluation and to qualify its previous
conclusion about Year 2000 readiness. In brief, the amended report states
that TRANSCOM's ability to perform its full deployment and sustainment
missions requires a yet-to-be confirmed assumption that the commercial
transportation industry can perform its part of the missions.
Second, to assess commercial carrier readiness, TRANSCOM has been seeking
confirmation letters from commercial carriers whose systems exchange data
with TRANSCOM's Global Transportation Network. The Global Transportation
Network is a system that provides in-transit visibility of transportation
assets, passengers, and cargo. As of August 24, 1999, TRANSCOM had
received confirmations on Y2K readiness from 20 of 27 of the network's
data exchange partners.
Third, TRANSCOM has been working with other federal agencies, such as the
Department of Transportation, to assess transportation sector readiness by
reviewing industry assessments and Internet web sites. It also plans to
verify the reported readiness of selected transportation partners.
Fourth, TRANSCOM is working with its component commands (i.e., Air
Mobility Command, Military Traffic Management Command, and Military
Sealift Command)/Footnote6/ to identify their major commercial carrier
business partners, identify these partners' Year 2000 readiness and
associated risks, and develop risk mitigation strategies. The first of
these status reports was due September 15, and showed that significant
progress had been made by TRANSCOM in determining the Year 2000 readiness
of its domestic commercial transportation partners. For example, TRANSCOM
determined that 89 percent of its Civil Reserve Air Fleet carriers, all
major rail carriers, and 196 of 199 trucking carriers have reported that
they are Year 2000 compliant. In addition, most North American ports have
compliance programs in place and all military active and ready reserve
fleet ships are certified Year 2000 compliant. However, TRANSCOM is still
in the process of determining the Year 2000 readiness of international
ports, airfields, and other international commercial carriers that are key
to accomplishing its deployment and sustainment missions.
Conclusions
-----------
By acting swiftly to address our recommendations made during the
August 24, 1999, briefing, TRANSCOM is increasing the effectiveness and
value of its operational evaluation and is mitigating the risks associated
with being able to operate effectively in the year 2000. Further, it has
ensured that DOD managers have complete and reliable information to use in
making informed military decisions. As a result, they have largely
satisfied the intent of those recommendations, and we are not making any
further recommendations at this time.
We are sending copies of this report to Representative John P. Murtha,
Ranking Minority Member, Subcommittee on Defense, House Appropriations
Committee; Senator John Warner, Chairman, and Senator Carl Levin, Ranking
Minority Member, Senate Committee on Armed Services; Senator Ted Stevens,
Chairman, and Senator Daniel Inouye, Ranking Minority Member, Subcommittee
on Defense, Senate Committee on Appropriations; Representative Floyd
Spence, Chairman, and Ike Skelton, Ranking Minority Member, House
Committee on Armed Services.
We are also sending copies to the Honorable John Koskinen, Chair of the
President's Year 2000 Conversion Council; the Honorable William Cohen,
Secretary of Defense; the Honorable John Hamre, Deputy Secretary of
Defense; General Henry Shelton, Chairman of the Joint Chiefs of Staff;
Arthur Money, Assistant Secretary of Defense for Command, Control,
Communications, and Intelligence; and the Honorable Jacob Lew, Director,
Office of Management and Budget. Copies will also be made available to
others upon request.
Should you or your staff have any questions concerning this report, please
contact me at (202) 512-6240. I can also be reached by e-mail at
[email protected]. Other points of contact and key contributors to this
report are listed in appendix III.
Sincerely yours,
*****************
*****************
Jack L. Brock, Jr.
Director, Governmentwide and Defense
Information Systems
--------------------------------------
/Footnote1/-^DOD refers to its combatant commands' end-to-end tests as
operational evaluations (OPEVAL).
/Footnote2/-^Joint Staff Year 2000 Operational Evaluation Guide, Version
3.0, April 1, 1999.
/Footnote3/-^Year 2000 Computing Crisis: A Testing Guide (GAO/AIMD-
10.1.21, issued as an exposure draft in June 1998; issued in final in
November 1998).
/Footnote4/-^In addition to conducting operational evaluations, the
military services are conducting system integration testing and the
functional business areas, such as personnel and health affairs, are
conducting functional end-to-end tests. Each of these end-to-end
testing activities is discussed in detail in Defense Computers:
Management Controls Are Critical to Effective Year 2000 Testing
(GAO/AIMD-99-172, June 30, 1999).
/Footnote5/-^Deployment refers to all activities associated with
systematically transporting personnel, material, and other elements from
a home station or origin through destination. Sustainment refers to all
activities associated with maintaining or supporting the deployed
personnel, material, and other elements.
/Footnote6/-^TRANSCOM consists of three military service-level component
commands, including the Army Military Traffic Management Command (MTMC),
the Air Force Air Mobility Command (AMC), and the Navy Military Sealift
Command (MSC). MTMC manages both land (truck and rail) transportation
and water port operations. AMC provides strategic airlift and aerial
refueling services, performs aeromedical evacuation, and operates
special assignment aircraft (such as Air Force One). MSC employs a
combination of government-owned and commercial ships to carry out sea
transportation operations and also manages prepositioned ships.
BRIEFING ON RESULTS OF GAO REVIEW OF TRANSCOM TURBO Y2K-PART A OPEVAL
=====================================================================
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
*****************
OBJECTIVES, SCOPE, AND METHODOLOGY
==================================
At the request of the Chairman, Subcommittee on Defense, House Committee
on Appropriations, we selected the TRANSCOM Turbo Y2K-Part A OPEVAL for
review to determine (1) whether the OPEVAL was planned, executed, and
documented in accordance with DOD and GAO Year 2000 testing guidelines and
(2) what the OPEVAL results indicated concerning readiness and risks. This
OPEVAL was selected in collaboration with the DOD Inspector General to
ensure appropriate coverage of all CINC OPEVALs and no duplication of
effort.
To meet our first objective, we reviewed TRANSCOM's OPEVAL plan, testing
documentation and records, and test results and associated reports. We
also interviewed TRANSCOM officials responsible for Y2K OPEVAL planning,
execution, and reporting tasks. Further, we observed the century date
rollover test for the MTW deployment execution function and compared
TRANSCOM's OPEVAL planning, execution, analysis, and reporting actions
against JCS OPEVAL guidance and our Y2K testing guide.
To meet our second objective, we reviewed TRANSCOM's OPEVAL results,
including its 7- and 30-day reports and system problem tracking reports.
We also interviewed TRANSCOM officials and analysts responsible for
developing OPEVAL assessment methodologies, interpreting evaluation
metrics, and ensuring that evaluation exit criteria were met.
We requested and received comments on a draft of our briefing, which was
delivered on August 24, 1999. We incorporated those comments where
appropriate. We performed our audit work from March through September 1999
in accordance with generally accepted government auditing standards.
GAO CONTACT AND STAFF ACKNOWLEDGEMENTS
======================================
GAO Contact
Randolph C. Hite, (202) 512-6240
Acknowledgements
In addition to those named above, Ronald B. Bageant, Cristina T. Chaplain,
Richard B. Hung, Denice M. Millett, and Karen S. Sifford made key
contributions to this report.
(511660)
*** End of document. ***