Defense Communications: Performance Measures Needed to Ensure DISN
Program Success (Letter Report, 11/27/96, GAO/AIMD-97-9).

The Defense Department (DOD) is now acquiring the transmission services
and switching technology needed to construct its future
telecommunications network known as the Defense Information System
Network (DISN). DOD's strategy for the network involves acquiring and
implementing DISN transmission and switching services across three
regions: the continental United States, the Pacific, and Europe. These
long-haul services will, in turn, interconnect DOD's base-level and
deployed communications networks. This report reviews the steps DOD has
taken in selecting and implementing its acquisition strategy for DISN in
the continental United States, which DOD is focusing on first. GAO
discusses whether (1) DOD considered other approaches, such as use of an
integrated bid, in its selection of an acquisition strategy and (2) the
selected acquisition strategy will yield the best value to the
government over DISN's life-cycle. GAO also examines the performance
measures that DOD managers are using or plan to use to objectively
evaluate the outcomes and effectiveness of their DISN implementation
efforts.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  AIMD-97-9
     TITLE:  Defense Communications: Performance Measures Needed to 
             Ensure DISN Program Success
      DATE:  11/27/96
   SUBJECT:  Military systems analysis
             Defense cost control
             Command/control/communications systems
             Defense contracts
             Defense procurement
             Defense communications operations
             Life cycle costs
             Procurement procedures
             Computer networks
IDENTIFIER:  Defense Information System Network
             Defense Commercial Telecommunications Network
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Report to the Chairman, Committee on Governmental Affairs, U.S. 
Senate

November 1996

DEFENSE COMMUNICATIONS -
PERFORMANCE MEASURES NEEDED TO
ENSURE DISN PROGRAM SUCCESS

GAO/AIMD-97-9

Defense Communications

(511367)


Abbreviations
=============================================================== ABBREV

  CONUS - continental United States
  DCTN - Defense Commercial Telecommunications Network
  DISA - Defense Information Systems Agency
  DISN - Defense Information Systems Network
  DOD - Department of Defense
  DTC - DISN Transition Contract
  FASA - Federal Acquisition Streamlining Act
  RFPs - Requests for Proposals

Letter
=============================================================== LETTER


B-275038

November 27, 1996

The Honorable Ted Stevens
Chairman
Committee on Governmental Affairs
United States Senate

Dear Mr.  Chairman: 

The Department of Defense (DOD) is in the process of acquiring the
transmission services and switching technology needed to construct
its future telecommunications network known as the Defense
Information Systems Network (DISN).  Defense's overall DISN strategy
calls for the Defense Information Systems Agency (DISA) to acquire
and implement DISN transmission and switching services across three
regions:  the continental United States (CONUS), the Pacific, and
Europe.  These long-haul services will, in turn, interconnect
Defense's base-level and deployed communications networks. 

This report responds to your request that we review the steps taken
by Defense in selecting and implementing its acquisition strategy for
DISN CONUS, which Defense is focusing on first.  Specifically, you
asked that we determine whether (1) Defense considered alternative
approaches, such as use of an integrated bid, in its selection of an
acquisition strategy and (2) the selected acquisition strategy will
yield the best value to the government over DISN's life-cycle.  Your
office also requested that we determine what performance measures
Defense managers are using or plan to use to objectively evaluate the
outcomes and effectiveness of their DISN implementation efforts. 


   RESULTS IN BRIEF
------------------------------------------------------------ Letter :1

Defense considered several options prior to selecting an acquisition
strategy for DISN, including an approach that would have involved
using a single comprehensive service provider to furnish an
integrated set of services to the government and another one that
involved separately acquiring component services with the government
integrating those components itself.  Defense considered the
advantages and disadvantages of each option in terms of five factors: 
requirements, technology enhancement, schedule, management, and cost. 
After evaluating its options and receiving industry comments on its
draft Request for Proposals (RFPs), Defense ultimately decided on an
approach that calls for the Defense Information Systems Agency (DISA)
to separately acquire and integrate component services itself, using
contracts awarded on a staggered schedule.  Defense believes that
this strategy will best meet national security needs at a reasonable
cost. 

In reviewing Defense's DISN efforts in 1995, we reported that Defense
had yet to define the program's minimal acceptable requirements.\1 We
also reported that Defense had not yet developed an estimate of what
it would cost to acquire, operate, and sustain the DISN
infrastructure.  Without this information, Defense has no objective
cost and performance baseline for measuring DISN's success.  Further,
without this baseline, we cannot determine whether the selected
acquisition strategy will yield the best value to the government over
the course of DISN's life cycle, which is estimated to be over 10
years. 

Once this baseline is developed, Defense must also establish
effective measures for tracking DISN's progress.  At present, Defense
is far from meeting federal requirements for establishing performance
measures.  By developing measures that focus on benefits, costs, and
risks, Defense management can target problem areas, highlight
successes, and ensure DISN meets its cost and performance goals. 


--------------------
\1 Defense Communications:  Management Problems Jeopardize DISN
Implementation (GAO/AIMD-95-136, July 13, 1995). 


   BACKGROUND
------------------------------------------------------------ Letter :2

Defense has long operated multiple telecommunications systems to meet
an array of mission needs, ranging from the command and control of
military forces to its payroll and logistics support functions. 
Because military services and other Defense agencies independently
procured and operated their own networks, Defense's communications
environment has been fragmented and redundant.  To improve the
effectiveness and efficiencies of its military communications
services, Defense began in 1991 to plan and implement DISN to serve
as the Department's primary worldwide telecommunications and
information transfer network to support national security and defense
operations. 

Defense's DISN strategy focuses on replacing its older data
communications systems, using emerging technologies and
cost-effective acquisition strategies that provide secure and
interoperable voice, data, video, and imagery communications services
in support of military operations.  Under Defense's DISN concept, the
military services and Defense agencies will still be responsible for
acquiring telecommunications services for their local bases and
installations, as well as deployed communications networks.  DISA
will be responsible for acquiring the long-haul services that will
interconnect these base-level and deployed networks within and
between the continental United States, Europe, and the Pacific. 
DISA's current efforts focus on acquiring and implementing DISN CONUS
services.  For 10 years, Defense users obtained switched voice, data,
video teleconferencing, and transmission services within the United
States through the Defense Commercial Telecommunications Network
(DCTN) contract with AT&T.  The DCTN contract expired in February
1996.  Since then, these services have been provided through a
follow-on, sole-source DISN Transition Contract (DTC) with AT&T until
Defense can fully implement its new DISN services.  Defense estimates
that DTC costs are approximately $18.5 million per month. 

In July 1995, we reported on Defense's efforts to plan and implement
DISN.\2 At that time, we recommended that Defense ensure that DISN
plans and program decisions were based on a validated statement of
DISN's operational requirements.  By defining the minimal acceptable
requirements for DISN as well as the critical technical
characteristics, the operational requirements document would provide
the basis for determining DISN's effectiveness.  We also recommended
that Defense develop an estimate of the acquisition, operations,
maintenance, and support costs for DISN over its life-cycle.  While
Defense concurred with these recommendations, it has not yet
completed either action. 

Nevertheless, given the expiration of its DCTN contract in February
1996, and its desire to limit the term of the sole-source DISN
Transition Contract, DISA is proceeding with its DISN implementation
efforts and has issued four RFPs supporting DISN's implementation: 

  -- DISN Support Services - Global, to provide engineering,
     operations, network management, and other support services
     worldwide. 

  -- DISN Switched/Bandwidth Manager Services - Continental United
     States (CONUS), to provide the capability to switch network
     traffic and provide bandwidth manager devices at designated
     service delivery points within the continental United States. 

  -- DISN Transmission Services - CONUS, to provide access
     transmission services and transmission services connecting the
     bandwidth managers and switches provided under the
     switched/bandwidth manager contract, and to connect Defense
     installations with the DISN network. 

  -- DISN Video Services - Global, to provide worldwide video
     teleconferencing through three video network hubs located in the
     continental United States. 

The timetable for receipt of proposals and contract awards is shown
in table 1. 



                                Table 1
                
                    Expected Dates for DISN Contract
                          Proposals and Awards

                                        Receive
Solicitation                            proposals       Award contract
--------------------------------------  --------------  --------------
Support services                        November 1995   June 1996

Switched/bandwidth                      January 1996    August 1996

Transmission                            March 1996      January 1997

Video services                          April 1996      February 1997
----------------------------------------------------------------------
DISA awarded the support services contract to Boeing Information
Services, Inc., in June 1996, and awarded the switched/bandwidth
manager services contract to MCI Corporation in August 1996.  The
evaluation of these proposals and subsequent contract awards
addressed four factors:  cost, technical, management, and past
performance.  DISA plans to award the video services contract on the
same basis.  Because transmission is a basic commodity service,
Defense advised that it intends to award the transmission services
contract primarily on the basis of lowest price.  Defense plans full
implementation of its DISN system within the continental United
States by July 1997. 

The switched/bandwidth manager, transmission services, and video
services acquisitions were subject to a bid protest in December 1995
by AT&T, which was adjudicated by the General Accounting Office
(GAO).  In this protest, AT&T argued that DISA arbitrarily refused to
allow offerors to submit and have evaluated a single, comprehensive
proposal, what AT&T termed an "integrated bid," as an alternative to
submitting individual proposals under each RFP.  GAO's decision,
issued on May 1, 1996, upheld the legality of the acquisition
strategy that DISA has followed.\3


--------------------
\2 GAO/AIMD-95-136, July 13, 1995. 

\3 The Comptroller General (B-270841; B-270842; B-270843, May 1,
1996) denied AT&T's protests regarding the terms of requests for
proposals issued for DISN. 


   OBJECTIVES, SCOPE, AND
   METHODOLOGY
------------------------------------------------------------ Letter :3

To obtain information about Defense's acquisition strategy, and the
steps taken by Defense in determining and selecting that strategy, we
obtained and analyzed copies of the DISN solicitations from DISA
staff in the Washington, D.C., area.  We analyzed studies prepared by
DISA staff during April and May 1995 that identified and evaluated
DISN acquisition alternatives.  We reviewed Defense's DISN
architecture and were briefed on steps taken to develop the DISN
design by engineering staff at DISA's Joint Interoperability and
Engineering Organization, Center for Systems Engineering, in Reston,
Virginia.  In addition, in conducting our review, we used supporting
documentation from our bid protest decision. 

To obtain information about the specific evaluation methods and
factors used to select a DISN acquisition strategy, we interviewed
several DISA officials including the DISN Program Manager and the
DISN Contracting Officer in Arlington, Virginia.  Our review was
conducted from August 1996 through October 1996 in accordance with
generally accepted government auditing standards. 


   DEFENSE'S ANALYSIS OF DISN
   ACQUISITION ALTERNATIVES
------------------------------------------------------------ Letter :4

In developing its DISN acquisition approach, Defense considered
several acquisition alternatives in April and May 1995 including
one--using a single contractor to furnish a comprehensive set of
services to the government--that is similar to the integrated
approach that AT&T had advocated.  Defense also evaluated the costs
and benefits of separately acquiring component services with the
government integrating those components itself, and other alternative
approaches as well. 

In reviewing Defense's analyses of alternatives, we found that
Defense evaluated the advantages and disadvantages of each
acquisition alternative in terms of relative cost and how it (1) met
DISN requirements, (2) facilitated technology insertion and
enhancement, (3) could be implemented within schedule constraints,
and (4) supported Defense's control of the network. 

DISA selected an acquisition strategy that divided the acquisition
into four components with four separately awarded contracts.  Under
this plan, DISA, with the assistance of the support services
contractor, would acquire, integrate, operate, and maintain the
separate DISN components rather than employ a comprehensive service
provider to integrate and operate DISN.  Defense believed that
breaking the program into functional components facilitated control
over network interoperability, integration, surge capacity,\4
technology insertion, and security.  It also concluded that by
breaking the program into pieces, more vendors could bid for
contracts, thus increasing competition.  Further, in Defense's view,
multiple contracts with frequent options made it easier to negotiate
technological upgrades, and created incentives for vendors to
maintain high standards of performance.  Finally, Defense believed
that the strategy encouraged vendors to offer their lowest prices on
each separate contract instead of just offering prices that were
averaged across the entire network. 

After issuing solicitations to implement this strategy, Defense
received comments from industry contending that vendors could offer
significant economies if they could submit one comprehensive, or
integrated, bid for all of the business offered under the
switched/bandwidth manager, transmission, and video services RFPs. 
Defense responded with an approach which staggers contract awards
such that a vendor who wins the switched/bandwidth manager contract
can use any economies that might accrue to its advantage when bidding
for the remaining contracts.  According to DISA, this approach
enables the government to reap the potential cost savings of an
integrated bid while maintaining maximum flexibility for
cost-effective technical enhancements and continuing competition over
the life of the program. 


--------------------
\4 DISN documentation defines surge capacity as the amount of excess
capacity required to serve DOD requirements during a crisis when
there is traffic overflow. 


   ABILITY OF DISN ACQUISITION
   STRATEGY TO YIELD BEST VALUE
------------------------------------------------------------ Letter :5

Defense believes that it has selected the acquisition strategy that
will yield the best value to the government over the course of DISN's
life cycle.  However, Defense lacks the baseline information needed
for us to ensure whether this is the case.  We recommended in July
1995 that Defense ensure that the DISN approach was based on valid
operational requirements and that it identify the additional
life-cycle acquisition, maintenance, and support costs that would be
incurred in developing and operating DISN.  In making these
recommendations, we concluded that without this important
information, Defense would lack a starting point for ensuring that
DISN facilities and services effectively and efficiently met their
requirements.  While Defense concurred with our recommendations, it
has not fully implemented them.  Given the current advanced state of
the DISN acquisition and the need to replace the high-cost transition
contract, we are not questioning the need to continue to move forward
with DISN.  However, Defense still needs this baseline information to
gauge the performance of DISN as it is being implemented. 

DISN program officials in DISA and staff from the Office of the
Assistant Secretary of Defense for Command, Control, Communications,
and Intelligence have told us that DISN's requirements are known and
documented because they are based on the requirements developed for
Defense's current communications systems.  However, we believe that
the operational requirements in the existing systems are not valid
for DISN because they do not consider several important factors. 

First, with the growth of worldwide telecommunications networks such
as the Internet, the information warfare threat to Defense, and thus
the need for security requirements, has significantly increased in
the past decade.  For example, we recently reported that Defense may
have experienced as many as 250,000 computer attacks last year and
that Defense estimates that these attacks are successful 65 percent
of the time.  We also reported that the number of attacks is doubling
each year, as Internet use increases along with the sophistication of
computer attackers and their tools.\5 Second, since the new strategy
calls for diversifying contractors, integration risks are
significantly higher than those accompanying the previous contract
and system management is much more complex.  Third, users now have
greater expectations for network services as telecommunications
technology has made significant strides in recent years.  Taken
together, these changes clearly demonstrate the need for Defense to
document and validate with DISN users the operational requirements
for the new strategy. 


--------------------
\5 Information Security:  Computer Attacks at Department of Defense
Pose Increasing Risks (GAO/AIMD-96-84, May 22, 1996). 


   DEFENSE LACKS PERFORMANCE
   MEASURES CRITICAL TO DISN'S
   SUCCESS
------------------------------------------------------------ Letter :6

By better establishing its operational requirements and life cycle
costs for DISN, Defense would lay the groundwork for assessing
whether the system is meeting its cost and performance goals.  The
next step would be to develop effective measures for tracking DISN's
progress against this baseline cost and performance information. 
Defense has not yet established any performance measures that would
allow it to track whether DISN is meeting its objectives.  Since
Defense plans to begin implementing DISN CONUS in less than 8 months,
the absence of these measures raises concerns that the Department
will not be able to effectively manage DISN's implementation and
operation. 

Establishing good performance measures is not only critical because
of the risks confronting the DISN program, it is central to the
success of any significant information system undertaking.  We have
previously reported, for example, that successful organizations rely
heavily upon performance measures to achieve mission goals and
objectives, quantify problems, evaluate alternatives, allocate
resources, track progress, and learn from mistakes.\6 For
service-oriented programs such as DISN, these may include such
measures as the percent of mission improvements resulting from the
new service in terms of cost, time, quality, and quantity; the
percent of customers satisfied with certain telecommunications
services; or the number of problems resolved within target times. 
Once the right measures are chosen, they help management target
problem areas, highlight successes, and generally increase the rate
of performance improvement through enhanced learning. 

Further, several statutory requirements call for Defense to define
cost, schedule, and performance goals for major defense acquisition
programs and for each phase of the acquisition cycle of such
programs.  These include the Federal Acquisition Streamlining Act
(FASA) of 1994 and the recently enacted Clinger-Cohen Act of 1996.\7
The requirement to establish program cost estimates and performance
measures of operational effectiveness are also embodied in Defense
acquisition guidance. 

At present, Defense is far from meeting any of these requirements. 
For example, even basic objectives, such as DISN's ability to provide
its users with the needed quality and volume of communications
services, have not been validated by users and lack evaluation
criteria upon which to measure success.  Without this type of
information, Defense has no way of knowing whether it will be
spending billions of dollars acquiring, operating, and maintaining
DISN facilities and services that efficiently and effectively meet
its needs. 


--------------------
\6 Executive Guide:  Improving Mission Performance Through Strategic
Information Management and Technology (GAO/AIMD-94-115, May 1994). 

\7 Under Section 2220 of Title 10, United States Code, added by FASA,
the Secretary of Defense is required to approve or define the cost,
performance, and schedule goals for major defense acquisition
programs and for each phase of the acquisition cycle.  Section 5123
(3) of the Clinger-Cohen Act states that agency heads shall ensure
that performance measurements are prescribed for information
technology used by or to be acquired for an agency and that the
performance measurements gauge how well the information technology
supports agency programs. 


   CONCLUSIONS
------------------------------------------------------------ Letter :7

Defense is striving to fully implement its DISN CONUS system by July
1997.  However, it has yet to establish the basic cost and
performance baseline information critical to laying the groundwork
for assessing DISN's success.  We continue to believe that Defense
should expeditiously implement our previous recommendation to develop
and document DISN operational requirements and to identify DISN life
cycle costs.\8 In addition, Defense has not established performance
measures that would determine how the implementation of this
multibillion dollar initiative measures up to its cost and
operational goals.  Establishing such measures now for DISN would
markedly improve DOD's and the Congress' ability to manage and
oversee implementation of this system by providing the basis for
independent analysis and evaluation. 


--------------------
\8 GAO/AIMD-95-136, July 13, 1995. 


   RECOMMENDATION
------------------------------------------------------------ Letter :8

We recommend that the Secretary of Defense direct the Assistant
Secretary of Defense for Command, Control, Communications, and
Intelligence to establish the objective measures needed to gauge
DISN's success.  At a minimum, these measures should include the
concerns of DISN customers and should correspond to the five
factors--requirements, technology enhancement, schedule, management,
and cost--that DISA used to select its acquisition strategy. 


   AGENCY COMMENTS AND OUR
   EVALUATION
------------------------------------------------------------ Letter :9

We obtained written comments on a draft of this report and have
incorporated those comments where appropriate.  These comments are
presented in appendix I. 

In commenting on the draft report, Defense concurred with our
recommendation.  We are encouraged that Defense intends to develop
cost estimates and performance measures for major DISN components
from this point forward.  It is likewise important that Defense does
so for the DISN-CONUS component currently being implemented.  As
stated in our report, these actions are critical in order for Defense
to have an objective cost and performance baseline for measuring the
success of this acquisition. 


---------------------------------------------------------- Letter :9.1

As agreed with your office, we will send copies of this report to the
Ranking Minority Member of the Senate Committee on Governmental
Affairs, Chairman and Ranking Minority Member of the House Committee
on Government Reform and Oversight, other interested congressional
committees, the Secretary of Defense, and the Director of the Office
of Management and Budget.  Copies will be sent to others upon
request.  Please contact me at (202) 512-6240 if you or your staff
have any questions.  Major contributors to this report are listed in
appendix II. 

Sincerely yours,

Jack L.  Brock, Jr.
Director, Defense Information and
 Financial Management Systems




(See figure in printed edition.)Appendix I
COMMENTS FROM THE DEPARTMENT OF
DEFENSE
============================================================== Letter 



(See figure in printed edition.)



(See figure in printed edition.)


MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II


   ACCOUNTING AND INFORMATION
   MANAGEMENT DIVISION,
   WASHINGTON, D.C. 
-------------------------------------------------------- Appendix II:1

Linda D.  Koontz, Associate Director
Franklin W.  Deffer, Assistant Director
Kevin E.  Conway, Senior Information Systems Analyst
Mary T.  Marshall, Information Systems Analyst
Cristina T.  Chaplain, Communications Analyst


*** End of document. ***