Battlefield Automation: Software Problems Hinder Development of the
Army's Maneuver Control System (Letter Report, 10/16/97,
GAO/NSIAD-98-15).
GAO reviewed the Army's developmental and acquisition plans for the
Maneuver Control System (MCS), focusing on whether: (1) the current MCS
software development strategy is appropriate to overcome prior
development problems; and (2) 207 new computers for MCS-related training
should be procured as planned.
GAO noted that: (1) since its 1993 reorganization, the MCS has continued
to experience development problems; (2) the initial operational test and
evaluation of version 12.01 software has slipped 28 months, from
November 1995 to March 1998, and interim tests have shown that
significant software problems continue; (3) despite these problems, the
Army awarded a contract in September 1996 for the concurrent development
of the next software versions--12.1, 12.2, and 12.3--which are being
developed by a new contractor and may involve substantially different
software; (4) if the Army's current development strategy for the MCS is
not strengthened, development problems may continue to occur; (5)
currently, the army's strategy allows: (a) less than full operational
testing of version 12.1; and (b) development of follow-on versions 12.2
and 12.3 to start about 18 months before the operational testing of each
version's predecessor; (6) despite the fact that the MCS has yet to
undergo an initial operational test and evaluation or be approved for
production, the Army plans to acquire 207 computers in fiscal years 1997
and 1998 to increase the number of computers available for system
training; (7) program officials stated that they need to acquire the
computers before operational testing to provide not only MCS specific
training but also training for the larger Army Battle Command System, of
which the Army Tactical Command and Control System and the MCS are major
components; and (8) the 207 computers, however, are not needed to
satisfy any of the three legislated reasons for low-rate initial
production before an initial operational test and evaluation.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: NSIAD-98-15
TITLE: Battlefield Automation: Software Problems Hinder
Development of the Army's Maneuver Control System
DATE: 10/16/97
SUBJECT: Army procurement
Military training
Computer software verification and validation
Military systems analysis
Management information systems
Military communication
Defense capabilities
Computers
IDENTIFIER: Army Maneuver Control System
Army Command and Control System Common Hardware and
Software Program
Army Battlefield Command System
Army Tactical Command and Control System
ATCCS
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Secretary of Defense
October 1997
BATTLEFIELD AUTOMATION - SOFTWARE
PROBLEMS HINDER DEVELOPMENT OF THE
ARMY'S MANEUVER CONTROL SYSTEM
GAO/NSIAD-98-15
Battlefield Automation
(707242)
Abbreviations
=============================================================== ABBREV
MCS -
DOD -
CHS -
DOT&E -
Letter
=============================================================== LETTER
B-276830
October 16, 1997
The Honorable William S. Cohen
The Secretary of Defense
Dear Mr. Secretary:
The Army has spent over $765 million of the $1 billion estimated
total cost for the Maneuver Control System (MCS) which is to provide
battlefield information to maneuver commanders. Since 1980, the MCS
program has experienced numerous problems, such as fielding
inadequate computer software and canceling the development of one
software version due to design flaws, cost growth, and schedule
slips. Given the program's past difficulties and the important role
of MCS in the Army's battlefield automation efforts, we reviewed the
Army's development and acquisition plans for MCS. Specifically, our
objectives were to determine whether (1) the current MCS software
development strategy is appropriate to overcome prior development
problems and (2) 207 new computers for MCS related training should be
procured as planned.
BACKGROUND
------------------------------------------------------------ Letter :1
The goal of the Army's MCS program is to develop and field a computer
system that provides automated critical battlefield assistance to
maneuver commanders and their battle staff at the corps-to-battalion
level. MCS is intended to enable the command staff to collect,
store, process, display, and disseminate critical data to produce and
communicate battle plans, orders, and enemy and friendly situational
reports. It is a key component of the Army Tactical Command and
Control System, which is also intended to enhance the coordination
and control of combat forces through automated management of five key
battlefield areas, including maneuver control.\1 Given its role to
communicate battle plans, orders, and enemy and friendly situation
reports, MCS is also a key component of the Army's ongoing efforts to
digitize (automate) its battlefield operations.
In 1980, the Army fielded the first MCS system--with limited command,
control, and communications capabilities--to VII Corps in Europe. In
1982, the Army awarded a 5-year contract to continue MCS development,
and by 1986 MCS software had evolved to version 9, also fielded in
Europe. In 1987, the Army performed post-deployment tests on version
9 in Germany. The results of those tests led the Army Materiel
Systems Analysis Activity to conclude that MCS did not exhibit
adequate readiness for field use and recommend that further fielding
not occur until the system's problems were resolved.\2 However, the
Army awarded a second 5-year contract that resulted in version 10,
which was fielded by April 1989 and remains in the field today. In
November 1989, the Army Materiel Systems Analysis Activity reported
that MCS had met only 30 percent of its required operational
capabilities and again recommended that the system not be released
for field use. In May 1990, operational testers again questioned the
system's functional ability and effectiveness because it could not
produce timely, accurate, and useful information in a battle
environment.
While earlier versions of MCS were being fielded and withdrawn, the
development of software continued. In 1988, the Army awarded a
contract for the development of version 11. By February 1993, the
Army stopped development of version 11 software due to multiple
program slips, serious design flaws, and cost growth concerns. The
program was then reorganized with a plan approved by the Office of
the Secretary of Defense in April 1993. Under the reorganized
program, a group of contractors and government software experts have
been working to develop the next version of MCS software--version
12.01--utilizing software segments that could be salvaged from the
failed version 11 effort.
In addition to software, the MCS system consists of computers
procured under the Army's Common Hardware and Software (CHS) effort,
which was undertaken to reverse the proliferation of program-unique
computers and software. The Army planned to acquire 288 of the CHS
computers in fiscal years 1997 and 1998 to support the MCS training
base, and has already acquired 81. Those computers were used in a
training base assessment to support a decision to acquire the
remaining 207 computers.
--------------------
\1 The other battlefield functional areas are air defense, fire
support, intelligence and electronic warfare, and combat service
support.
\2 In April 1984, the Army Materiel Command designated the Army
Materiel Systems Analysis Activity as its independent evaluator for
materiel releases of major and other high-visibility systems.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :2
Since its 1993 reorganization, the Maneuver Control System has
continued to experience development problems. The initial
operational test and evaluation of version 12.01 software has slipped
28 months, from November 1995 to March 1998, and interim tests have
shown that significant software problems continue. Despite these
problems, the Army awarded a contract in September 1996 for the
concurrent development of the next software versions--12.1, 12.2, and
12.3--which are being developed by a new contractor and may involve
substantially different software. If the Army's current development
strategy for the Maneuver Control System is not strengthened,
development problems may continue to occur. Currently, the Army's
strategy allows (1) less than full operational testing of version
12.1 and (2) development of follow-on versions 12.2 and 12.3 to start
about 18 months before the operational testing of each version's
predecessor.
Despite the fact that the Maneuver Control System has yet to undergo
an initial operational test and evaluation or be approved for
production, the Army plans to acquire 207 computers in fiscal years
1997 and 1998 to increase the number computers available for system
training. Program officials stated that they need to acquire the
computers before operational testing to provide not only MCS specific
training but also training for the larger Army Battle Command System,
of which the Army Tactical Command and Control System and the
Maneuver Control System are major components. The 207 computers,
however, are not needed to satisfy any of the three legislated
reasons for low-rate initial production before an initial operational
test and evaluation.\3
--------------------
\3 Title 10 U.S.C. 2400 provides that low-rate initial production of
systems, except for ships and satellites, is to produce the minimum
quantity necessary to (1) provide production-configured or
representative articles for operational test and evaluation, (2)
establish an initial production base for the system, and (3) permit
an orderly increase in the production rate for the system sufficient
to lead to full-rate production upon the successful completion of
operational test and evaluation.
DEVELOPMENT PROBLEMS CONTINUE
------------------------------------------------------------ Letter :3
Since its reorganization in 1993, MCS program experience indicates
continuing problems in the system's development. Specifically, (1)
the MCS initial operational test and evaluation of version 12.01 has
slipped twice, (2) interim developmental level tests and a customer
test done to support a decision to award a contract to develop
follow-on software show that significant problems continue, and (3)
development of follow-on version 12.1 was begun despite the results
of the customer test and prior program history.
OPERATIONAL TESTING
SCHEDULES SLIP
---------------------------------------------------------- Letter :3.1
After the 1993 program reorganization, version 12.01 was scheduled to
undergo initial operational testing and evaluation in November 1995.
The test slipped to November 1996 and is now scheduled for March
1998. Program officials stated that the test date slipped initially
because the CHS computers to be used were not yet available.
During August and September 1996, version 12.01 underwent a system
confidence demonstration to determine whether it was ready for the
November 1996 initial operational test and evaluation. Because the
software was not ready, further work and two additional system
confidence demonstrations followed in August and September 1996.
Both demonstrations indicated that the system was not ready for
operational testing. Additionally, the software still had an open
priority one software deficiency and priority three and four
deficiencies that would have negatively impacted the conduct of the
operational test.\4
Both the Army's Operational Test and Evaluation Command and the
Department of Defense's (DOD) Director of Operational Test and
Evaluation (DOT&E) had stated that there could be no open priority
one or two software deficiencies before the operational test. They
had also stated that there could not be any open priority three and
four deficiencies that, in combination, were likely to have a
detrimental effect on the system's performance. DOT&E staff told us
that there were a number of open priority three and four software
deficiencies that they believe would have had a detrimental effect.
When MCS program officials realized that these deficiencies would not
be resolved in time for the initial operational test, they downgraded
the test 3 weeks before it was to occur to a limited user test,\5
utilizing $8.5 million appropriated for the MCS operational test in
fiscal years 1996 and 1997.\6 That test was conducted in November
1996. While the test report has not been finalized, a draft version
states that MCS--in the tested configuration--is not operationally
effective or suitable.
--------------------
\4 Software deficiencies are rated in a priority system, from
priority one--the most critical--to priority five--the least
critical. An open software deficiency is a deficiency identified
through testing that is not considered to be resolved.
\5 The limited user test involved the same testing planned for the
initial operational test and evaluation. It was limited in that it
had no pass/fail criteria and was not to be used to support a
full-rate production decision. It served as a learning experience,
providing information on the current maturity of the MCS software and
a baseline of performance by which to judge future development
efforts.
\6 MCS program officials stated that at the time it became apparent
that MCS was not ready for its initial operational test and
evaluation, it made sense to go forward with the test as a limited
user test because the user had been trained; the equipment was
instrumented for the test; and the users, equipment, and testers were
all in place. In providing technical comments on a draft of this
report, DOD stated that, given the sunk costs, the Army's decision to
go forward with the test made sense because an operational test could
provide invaluable feedback to the MCS developers that could not be
obtained through technical testing.
INTERIM DEVELOPMENT LEVEL
TESTS INDICATE CONTINUING
PROBLEMS
---------------------------------------------------------- Letter :3.2
Throughout the development of version 12.01, interim software builds
have undergone numerous performance tests to determine the current
state of software development, and build 4 was subjected to a
customer test.\7 The results of those tests identified continuing
problems as the number of builds proceeded. For example, a December
1995 performance test report on build 3.0 stated that, if the
problems found during the test were not quickly corrected in build
3.1, then the risk to the program might be unmanageable. The
follow-on April 1996 performance test report of build 3.1 stated that
significant problems in system stability prevented proper testing of
several requirements. The report further stated that messaging
between battlefield functional areas was extremely difficult and
problematic and that the system had other stability problems.
A September 1996 performance test report stated that of 568
previously open deficiency reports from builds 5.1 through 5.2c, 165,
almost 29 percent, still remained open. This report, the last
published on an MCS performance test, reflected the state of the MCS
software shortly before the downgraded limited user test, in which
MCS failed to demonstrate either operational effectiveness or
suitability. More recent performance tests of later builds have been
done; however, separate reports on those test events have not been
issued. Rather, the program office plans to prepare an integrated
test report in October or November 1997.
--------------------
\7 A software build involves additions or changes to the software to
add new functions or correct deficiencies in the prior build
software. Development of a new software version under the
evolutionary software development philosophy is accomplished by
multiple intraversion software builds.
CONCURRENT CONTRACT WAS
AWARDED FOR FOLLOW-ON
SOFTWARE DEVELOPMENT
---------------------------------------------------------- Letter :3.3
In April 1994, the MCS program office released a plan to begin
follow-on software development while version 12.01 was still in
development. In a May 1995 memorandum, the Deputy DOT&E expressed
concern regarding this plan. He stated that, because version 12.01
was being
"developed by a confederation of contractors who have built this
current version of MCS on the salvaged 'good' portions of the
abruptly terminated development of MCS Version 11, it needs to
stand the rigor of an Independent Operational Test and
Evaluation . . . before a MCS Block IV [post version 12.01
development] contract is awarded."
To help determine the level of risk in proceeding under the Army's
development strategy, DOT&E stated in a June 1995 memorandum that an
operational test of version 12.01 be conducted to measure the
software's maturity before the award of a contract for the
development of follow-on versions. As a result, an operational
assessment--called the MCS customer test--was conducted on version
12.01 in April 1996 to support the award of a $63.1 million contract
for the development of MCS Block IV software--MCS versions 12.1,
12.2, and 12.3.
No pass/fail criteria were set for the customer test. However, DOT&E
directed that four operational issues be tested. Those issues
related to (1) the capacity of the system to store and process
required types and amounts of data, including the ability of the
staff users to frequently update the information database; (2) the
capabilities of the MCS network to process and distribute current and
accurate data using the existing communications systems; (3) the
impact of computer server outages on continuity of operations; and
(4) the system administration and control capabilities to initialize
the system, become fully operational, and sustain operations.
In its report on the customer test, the Army's Test and
Experimentation Command stated that, at the time of the test, MCS was
evolving from a prototype system to one ready for initial operational
test and evaluation and, as such, possessed known limitations that
were described to the system users during training. The Command
reported that the test's major limitations included (1) software that
did not contain the full functional capability planned for the
initial operational test and evaluation; (2) a need to reboot the
system after crashes caused by the use of the computer's alternate
function key; (3) two changes in software versions during training;
and (4) the fact that 65 percent of the system manager functions had
not been implemented or trained. Table 1 provides more detail on the
customer test results.
Table 1
Customer Test Operational Issues and
Associated Army Test and Experimentation
Command Comments
Operational issue Army Test and Experimentation Command comments
---------------------- ----------------------------------------------
Capacity of the system "The system consistently locked up and had to
to store and process be rebooted while the staff user was
the required types and attempting to process
amounts of data, . . . data. This resulted in the loss of all
including the ability data in working files and any data in the
of the staff users to queues awaiting distribution or processing to
update the required a database."
database frequently.
"Staff users rated the systems capability to
process and provide . . . data, and to assist
the staff in the performance of their duties
as marginal."
"Storing and processing . . . data was
adequately demonstrated by [MCS] for only two
functions: editing specified reports and
processing specified messages.
. . . application software for the other
functions . . . performed inconsistently and
rendered the system unreliable."
The capabilities of "The [system to distribute data] and the
the MCS network to distributed computing environment did not work
process and distribute as required. The systems locked up and the
current and accurate message handler backed up (sometimes with
data using the thousands of messages). The test officer noted
existing . . . that . . . the dedicated server, had a
communications message queue backlog of 19,000 messages. This
systems. situation, combined with the necessity to
reboot the system . . . throughout the test,
caused backlogged messages to be lost. The
staff users were often unable to initiate or
complete tasks and transmit data between the
nodes."
Impact of computer The Army Test and Experimentation Command's
server outages on report indicates that the third operational
continuity of issue was met, stating that the success rate
operations. for continuity of operations was 100 percent.
System administration "Sixty five percent of the system manager
and control functions are not yet implemented, and were
capabilities to not trained."
initialize the system,
become fully "The results indicate that system
operational, and administration and control capabilities
sustain operations. functions are incomplete for this build of
software. Additionally, poor system
performance, and an immature training program
hampered the user's ability to sustain
operations."
----------------------------------------------------------------------
Source: Maneuver Control System/Phoenix: Customer Test Report,
Command, Control, and Communications Test Directorate; Army Test and
Experimentation Command, 1996-CT-1302, June 1996.
In addition to these findings, the MCS test officer stated the
following:
-- "System performance degraded over time causing message backlogs,
loss of data, and numerous system reboots. Over a period of 12
operational hours the [data distributing system] slowed down and
created message backlogs of up to 4 hours. To remain
functional, the entire network of [MCS] systems must be shut
down and reinitialized in proper sequence."
-- "The staff users had great difficulty using ... [multiple]
applications."
-- "The software pertaining to system management functions was
immature, incomplete and lacked documentation. This capability
is critical to the effective use and operation of the [MCS]
system."
Even though the customer test did not involve pass/fail criteria,
based on our review of the test report and the test officer's
comments, we believe that only the third operational issue--impact of
computer server outages on continuity of operations--was met.
Despite the results of the customer test and the program's prior
history, the Under Secretary of Defense for Acquisition and
Technology approved the Army's plan to award a concurrent contract
for MCS Block IV software development--MCS versions 12.1, 12.2 and
12.3.
In September 1996, the Army awarded a contract for the development of
MCS software versions 12.1, 12.2, and 12.3 to a different contractor
than the developers of MCS version 12.01. At that time, version
12.01 was still scheduled to undergo its initial operational testing
in November 1996. The start of the follow-on development could have
been timed to occur after version 12.01 had completed that
operational testing. At most, this action would have delayed the
contract award 2 months, assuming that the initial operational test
had occurred in November 1996 as scheduled. However, the contract
was awarded before the initial operational test, and the planned 5
month concurrency in the development of versions 12.01 and 12.1
became 18 months when the operational test slipped to March 1998.
The current program schedule indicates that (1) version 12.1 is
expected to undergo its operational assessment/test about 1 year
after the fielding of version 12.01 is started and (2) version 12.1
fielding is to be done 5 months after initial operational capability
of version 12.01 is achieved. If the scheduled version 12.01
operational test and evaluation slips again and the version 12.1
contractor is able to maintain its development schedule, version 12.1
could become available before version 12.01.
ARMY REQUESTED
FLEXIBILITY FOR
OPERATIONAL TESTING OF
FOLLOW-ON SOFTWARE
-------------------------------------------------------- Letter :3.3.1
By May 1997, the Army requested DOD approval of a revised acquisition
program baseline that changes the planned follow-on operational test
and evaluation of versions 12.1, 12.2, and 12.3 to operational
assessments/operational tests. Program officials said that, although
the name of the tests had changed, the planned scope of the tests had
not. However, the officials said that the name change complies with
guidance from DOT&E, which lists multiple levels of operational test
and evaluation (from an abbreviated assessment to full operational
test) and outlines a risk assessment methodology to be used to
determine the level of testing to be performed. The officials
further stated that the use of the generic term operational
test/operational assessment permits possible changes to the level of
testing for version 12.1 and follow-on software increments based on
the risk assessment process.
The contractors competing for the MCS Block IV (MCS versions 12.1,
12.2, and 12.3) development were given access to the government's
12.01 code and allowed to reuse as much of it as they chose. The
Block IV developer is not required to reuse any of version 12.01.
Rather, the Block IV contract requires the development of software to
provide specific functions. Given that (1) version 12.01 software
has not passed or even undergone an initial operational test and
evaluation and (2) the MCS Block IV contractor building version 12.1
is not the contractor that is building version 12.01 and is only
required to develop the version 12.1 to provide specified functions,
we believe that the version 12.1 development effort should not be
viewed as building upon a proven baseline. Instead, it should be
viewed as a new effort.
CONTINUATION OF CURRENT
DEVELOPMENT STRATEGY
COULD BE COSTLY
-------------------------------------------------------- Letter :3.3.2
The Army's current development plan for version 12.1 and beyond, as
shown in figure 1, continues an approach of building a follow-on
version of software on an incomplete and unstable baseline--the
uncompleted preceding version of software.
Figure 1: Future MCS Software
Development Schedule
(See figure in printed
edition.)
Source: Army.
Additionally, according to an official in the DOD's Office of the
Director of Test, Systems Engineering, and Evaluation, the Army's
development process allows requirements that are planned for one
software version, which cannot be accomplished in that version's
development as planned, to be deferred to a later version's
development. As a result, this process makes judging program risk
and total cost very difficult.
The MCS program has previously demonstrated the problem of deferring
requirements. For example, during MCS version 11 development, we
reported that the Army had deferred seven MCS functions that were to
have been developed by June 1992 and included in the software version
to undergo operational testing.\8 Even though the version 11
operational test had slipped twice, from May 1992 to September 1992
and then to May 1993, the Army continued to defer those functions,
and the operational test was planned for less than the complete
software package originally scheduled to be tested.
In commenting on a draft of this report, DOD said that they had made
progress not reflected in that draft. Specifically, they noted that
there were no priority one or two, and only 22 priority three
software deficiencies open as of September 11, 1997, as compared with
10 priority one, 47 priority two, and 67 priority three deficiencies
open on August 16, 1996. While we agree these results indicate that
some known problems have been fixed, they provide no indication of
the number or severity of still unknown problems. For example, MCS
version 12.01 development showed enough progress entering the
November 1996 scheduled initial operational test and evaluation to
reach a commitment of resources and personnel. However, that test
was later downgraded to a limited user test because of software
immaturity. Successful completion of an initial operational test and
evaluation should provide a more definitive indication of the MCS
program's progress.
--------------------
\8 Battlefield Automation: Planned Production Decision for Army
Control System is Premature (GAO/NSIAD-92-151, August 10, 1992).
BUYING TRAINING BASE COMPUTERS
BEFORE OPERATIONAL TESTING IS
QUESTIONABLE
------------------------------------------------------------ Letter :4
Before the slip of the MCS initial operational test and evaluation
from November 1996 to March 1998, the Army planned to acquire 288
computers--150 in fiscal year 1997 and 138 in fiscal year 1998--for
the MCS training base. These computers were to be acquired after a
full-rate production decision at a total cost of about $34.8
million--$19.1 million in fiscal year 1997 and $15.7 million in
fiscal year 1998.
After the initial operational test and evaluation slipped, DOD
approved the Army's acquisition of a low-rate initial production of
81 computers in fiscal year 1997 for a training base operational
assessment. The purpose of the assessment, which was performed from
February to May 1997, was to judge the merits of allowing the Army to
procure the remaining computers prior to successful completion of the
slipped operational test. On the basis of the results of that
assessment, the Acting Under Secretary of Defense for Acquisition and
Technology authorized the Army in July 1997 to proceed with its
acquisition plans. The Acting Under Secretary noted that the DOT&E
had reviewed the assessment and agreed that version 12.01 was
adequate for use in the training base.
The Acting Under Secretary also authorized the Army to move the
training base computer funds from the MCS budget to the Army's
automated data processing equipment program budget line. This action
was necessary because, according to both Army and DOD officials, it
was determined that the computers to be acquired do not meet the
legislated reasons in 10 U.S.C. 2400 for low-rate initial
production. That legislation allows the early acquisition of systems
to (1) establish an initial production base, (2) permit an orderly
increase in the production rate for the system that is sufficient to
lead to full-rate production upon successful completion of
operational test and evaluation, and (3) provide
production-representative items for operational test and evaluation.
Even though the Army now plans to acquire the computers under a
different budget line, the intended use of the computers remains
unchanged.
MCS program officials said that the computers are needed in the MCS
training base before operational testing to adequately support future
fielding of MCS and the larger Army Battle Command System, of which
the Army Tactical Command and Control System and MCS are key
components. This rationale is the same one the Acting Under
Secretary cited in his July 1997 memorandum. In that memorandum, he
stated that the "requirement to train Army-wide on commercial
equipment is a recognized requirement not only for MCS but for a host
of other digital . . . systems." The Acting Under Secretary
further noted that the funds to be moved were for equipment needed to
support integrated training of multiple systems throughout the Army
and concluded that "training on a digital system, even if it is not
the system that is ultimately fielded, is important to the Army in
order to assist in making the cultural change from current maneuver
control practice to a digitized approach."
MCS program officials stated that the MCS course curriculum needs to
be developed and that equipping the training base before the
completion of operational testing avoids a 2-year lag between the
completion of operational testing and the graduation of trained
students. The officials also commented that the computers could be
used elsewhere, since they would be compatible with other Army
programs.
The legislated requirement\9 that major systems, such as MCS, undergo
initial operational test and evaluation before full-rate production
serves to limit or avoid premature acquisitions. The Army has had
previous experience acquiring ineffective MCS equipment, which is
indicative of the need for adequate testing before systems are
fielded. In July 1990, the Army began withdrawing over $100 million
of militarized MCS hardware from the field due to both hardware and
software deficiencies. Additionally, the Army subsequently decided
not to deploy other MCS equipment it had procured for light divisions
at a cost of about $29 million because the equipment was too bulky
and heavy.
--------------------
\9 Title 10 U.S.C. 2399.
CONCLUSIONS AND RECOMMENDATIONS
------------------------------------------------------------ Letter :5
The MCS program's troubled development and acquisition history has
continued since the program's 1993 reorganization. However, the Army
awarded a new contract to develop future software versions and plans
to procure computers without fully resolving the problems of earlier
versions. This strategy does not minimize the possibility of future
development problems and ensure that the Army will ultimately field a
capable system. Also, since MCS software version 12.1 is being
developed concurrently by a different contractor to functional
specifications, it would be prudent to subject the version 12.1
software to the level of operational testing required to support a
full-rate production decision, as planned for version 12.01.
Accordingly, we believe a more appropriate strategy would require
that future software versions be developed using only fully tested
baselines, and that each version be judged against specific
pre-established criteria.
We recommend that you direct the Secretary of the Army to
-- set specific required capabilities for each software version
beyond version 12.01, test those versions against specific
pass/fail criteria for those capabilities, and only award
further development contracts once problems highlighted in that
testing are resolved;
-- perform a full operational test and evaluation of MCS software
version 12.1 to ensure that it provides the full capabilities of
version 12.01; and
-- procure additional MCS computers only after an initial
operational test and evaluation and a full-rate production
decision have been completed.
AGENCY COMMENTS AND OUR
EVALUATION
------------------------------------------------------------ Letter :6
In commenting on a draft of this report, DOD agreed with our
recommendation that specific required capabilities for each MCS
software version beyond version 12.01 are needed, that those versions
should be tested against specific pass/fail criteria for those
capabilities, and that the Army should not award further development
contracts until problems highlighted in prior tests are resolved.
DOD noted that the Army has already set specific required
capabilities for those software versions and will test those versions
against specific pass/fail criteria to ensure system maturity and
determine that the system remains operationally effective and
suitable. DOD further stated that it will not support the award of
further development contracts until the Army has successfully
resolved any problems identified during the testing of related,
preceding versions.
DOD partially agreed with our recommendation that the Army be
directed to perform a full-operational test and evaluation of MCS
software version 12.1 to ensure that it provides the full
capabilities of version 12.01. DOD stated that the Army will comply
with DOD regulation 5000.2R and will follow guidance from Director of
Operational Test and Evaluation, which lists multiple levels of
operational test and evaluation (from an abbreviated assessment to
full operational test) and outlines a risk assessment methodology to
be used to determine the level of testing to be performed. DOD did
not, however, indicate whether it would require the Army to conduct a
full operational test. We continue to believe that the version 12.1
development effort should not be viewed as building upon a proven
baseline. Instead, version 12.1 development should be viewed as a
new effort. As a result, we still believe that the prudent action is
to require that version 12.1 be subjected to the same level of
operational test and evaluation as version 12.01, the level required
to support a full-rate production decision.
DOD agreed with our recommendation that it direct the Army to not
procure more MCS computers until the completion of an initial
operational test and evaluation and a full-rate production decision.
It stated, however, that no further direction to the Army is needed
as it had already provided direction to the Army on this issue.
Specifically, the Department stated that it has directed the Army to
extract the training base computers from the MCS program and to not
procure or field more MCS hardware to operational units until
successfully completing an initial operational test and evaluation.
Our recommendation, however, is not limited to the hardware for
operational units, but also encompasses the computers the Army plans
to buy for the training base. Given the program's prior history and
the fact that the training base computers are not needed to satisfy
any of the legislated reasons for low-rate initial production, we
continue to believe that the Army should not be allowed to buy those
computers until MCS has successfully completed its initial
operational test and evaluation--the original plan prior to the MCS
initial operational test and evaluation's multiple schedule slips.
DOD's comments are reprinted in their entirety in appendix I, along
with our evaluation. In addition to those comments, we have revised
our report where appropriate to reflect the technical changes that
DOD provided in a separate letter.
SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :7
To determine whether the current MCS software development strategy is
appropriate to overcome prior problems and to determine whether the
Army should procure 207 new computers for the expansion of the MCS
training base, we interviewed responsible officials and analyzed
pertinent documents in the following DOD offices, all in Washington,
D.C.: Director of Operational Test and Evaluation; Director of Test,
Systems Engineering, and Evaluation; Assistant Secretary of Defense
for Command, Control, Communications, and Intelligence; Under
Secretary of Defense (Comptroller); and Defense Procurement. In
addition, we interviewed responsible officials and analyzed test
reports from the office of the Army's Project Manager, Operations
Tactical Data Systems, Fort Monmouth, New Jersey; and the Army's
Operational Test and Evaluation Command, Alexandria, Virginia. To
meet our second objective, we also interviewed responsible officials
and analyzed pertinent documents from the Army's Combined Arms
Center, Fort Leavenworth, Kansas.
We conducted our review from March to September 1997 in accordance
with generally accepted government auditing standards.
---------------------------------------------------------- Letter :7.1
We are sending copies of this report to the Chairman and Ranking
Minority Members, Senate and House Committees on Appropriations,
Senate Committee on Armed Services, and House Committee on National
Security; the Director, Office of Management and Budget; and the
Secretary of the Army. We will also make copies available to others
on request.
As you know, the head of a federal agency is required by 31 U.S.C.
720 to submit a written statement on actions taken our
recommendations to the Senate Committee on Governmental Affairs and
the House Committee on Government Reform and Oversight not later than
60 days after the date of this report. A written statement must also
be submitted to the Senate and House Committees on Appropriations
with the agency's first request for appropriations made more than 60
days after the date of the report.
Please contact me at (202) 512-4841 if you or your staff have any
questions concerning this report. Major contributors to this report
were
Charles F. Rey, Bruce H. Thomas, and Gregory K. Harmon.
Sincerely Yours,
Allen Li
Associate Director
Defense Acquisitions Issues
(See figure in printed edition.)Appendix I
COMMENTS FROM THE DEPARTMENT OF
DEFENSE
============================================================== Letter
(See figure in printed edition.)
(See figure in printed edition.)
The following are GAO's comments on the Department of Defense's (DOD)
letter dated October 2, 1997.
GAO COMMENTS
------------------------------------------------------------ Letter :8
1. In partially agreeing with this recommendation, DOD states that
the Army will comply with DOD regulation 5000.2R and will follow
guidance from Director of Operational Test and Evaluation--guidance
which lists multiple levels of operational test and evaluation (from
an abbreviated assessment to full operational test) and outlines a
risk assessment methodology to be used to determine the level of
testing to be performed. DOD does not, however, indicate how they
agree or disagree with our recommendation or state whether they will
implement the recommendation. As we stated in the body of this
report, given that a different contractor is building version 12.1
under a requirement to provide specific functionality, we believe
that this development effort should not be viewed as building upon a
proven baseline. Instead, version 12.1 development should be
considered a new effort. As a result, we continue to believe that it
is prudent to require that version 12.1 be subjected to the level of
operational test and evaluation required to support a full-rate
production decision.
2. DOD's direction to the Army only partially implements our
recommendation. Our recommendation is not limited to the hardware
for operational units, but also encompasses the computers the Army
plans to buy for the training base. We continue to believe that the
Army should not be allowed to buy the planned training base computers
until MCS has successfully completed its initial operational test and
evaluation--the original plan prior to the MCS initial operational
test and evaluation's schedule slips. The training base computers
are not required to satisfy any of the three purposes the law
indicates for low-rate initial production--to (1) establish an
initial production base, (2) permit an orderly increase in the
production rate for the system sufficient to lead to full-rate
production upon successful completion of operational test and
evaluation, and (3) provide production-representative items for
operational test and evaluation. Since the training base computers
are not needed to satisfy one of the above legislated conditions, we
continue to believe that the Army should refrain from buying any
additional MCS computers prior to a full-rate production decision.
*** End of document. ***